#Newton raphson method blacksholes update#

One thing to note about here is that in the first order methods, one is typically trying to capture 2nd order information about the gradients to update the step. This is very common in improving first order methods, i.e., ADAM. In the comments, the OP asks about using path history to inform the step size. However, generally the problems that mini-batching is useful for are the problems for which Newton Raphson cannot handle they are often highly-multimodal and very high dimensional, so forming and solving for the full Hessian (even from just subsample of the data) is prohibitive. Then, the fact that Newton Raphson is making an adapted step is actually bad you don't want to immediately optimize the approximated function because you will be jumping too far and will never converge. An interesting case of why you might want to is if you are doing something like mini-batch updates, where you are approximating the target function by only using a small portion of the full data (I mention this because the SGD tag was used). Adaptive step sizes for first order methods are strongly motivated by trying to adapt the step size by the Hessian, which Newton Raphson already does. In general, I cannot see any good reason for an adaptive step size outside using a line search.

If the function is particularly unstable (i.e., Hessian is very non-stationary), it's often a good idea to use a line-search method to, for example, check whether half stepping actually decreases the target function more than the full step proposed by vanilla Newton-Raphson. However, if the problem is convex, we can show that the algorithm will still converge. Of course, generally the target function is not quadratic, or we don't need the iterative Newton Raphson algorithm as there is a closed form solution (i.e., the first step). If the function is quadratic, this the 'optimal' update in that in converges in one step. In a sense, Newton Raphson is automatically doing the adaptive step size it's adapting the step in each dimension (which changes the direction) according to the rate of change of the gradient.

Newton raphson method blacksholes