Newton's Method

Gradient descent uses only the slope and inches along with a hand-tuned step size. Newton's method is greedier and smarter: it also uses the curvature — the second derivative — to estimate exactly how far to jump. For finding a root of f (a point where f(x)=0), you draw the tangent line at your current guess and slide to where that line crosses zero:

For optimization you want a minimum, where the derivative of the loss is zero — so you simply apply the same idea to f = L', dividing the gradient by the curvature L'' instead of by a guessed learning rate:

The payoff is quadratic convergence: once you are near the answer, the number of correct digits roughly doubles every step. The catch is that you must compute (and, in many dimensions, invert) the second-derivative information — the Hessian — which can be expensive or unstable far from the solution.

Tangent lines homing in

Watch Newton's method find a root of f(x) = x^3 - 2x - 5. From the current guess it rides the tangent line down to the axis; that crossing becomes the next guess. After just a few steps the tangent is essentially sitting on the root.

Click to choose a new starting point and watch it converge again.

See also

• Gradient Descent
• Convexity
• Momentum