Momentum

6 min read#optimization

Give the optimizer inertia — accumulate a velocity from past gradients so it powers through ravines and coasts over small bumps.

Momentum

Plain gradient descent has no memory: each step depends only on the gradient right now. In a long, narrow ravine — steep walls, gentle floor — that is a disaster. The optimizer keeps overreacting to the steep sidewalls, zig-zagging back and forth across the valley while barely creeping along its length. Momentum fixes this by giving the optimizer inertia, exactly like a heavy ball rolling downhill. Instead of jumping in the instantaneous gradient direction, you accumulate a velocity:

v \leftarrow \mu\,v - \eta\,\nabla L(\theta), \qquad \theta \leftarrow \theta + v.

The coefficient \mu (typically around 0.9) is how much of the previous velocity carries over. Oscillating components — the side-to-side bounces — point in opposite directions on alternate steps and cancel out in the running average, while the consistent downhill component along the valley floor adds up, building speed. The result is faster, smoother convergence and enough built-up momentum to coast straight over tiny bumps and shallow local dips.

Watch inertia win the ravine

Both balls below start at the same place in a stretched, ravine-shaped bowl (steep top-to-bottom, shallow left-to-right). The plain gradient-descent ball zig-zags wildly across the steep direction, wasting almost all its motion. The momentum ball lets those zig-zags cancel and glides along the floor to the bottom far sooner.

Same start, same ravine. Plain gradient descent (left color) bounces between the steep walls; momentum (right color) averages those bounces away and accelerates along the valley floor to the minimum. The run loops.

See also