Gradient Descent
The fundamental learning algorithm — repeatedly step a little way downhill along the negative gradient of a loss until you reach a valley.
Gradient Descent
Gradient descent is the single most important algorithm in machine learning, and it is almost embarrassingly simple. You are standing somewhere on a loss surface and you want to get low. The gradient \nabla L points in the direction of steepest increase, so its negative points straight downhill. Take a small step that way, recompute, and repeat:
Here \theta is everything you can tune (the weights of a model), L is the loss you want to minimize, and \eta — the learning rate — is how big a step you dare to take. Too small and you crawl; too large and you overshoot the valley and bounce out. The whole craft of training comes down to choosing a good direction and a good step size, over and over.
Why the negative gradient is "downhill"
The gradient is the vector of partial derivatives, \nabla L = \big(\partial L/\partial\theta_1,\dots\big). For a tiny step \Delta\theta, the change in loss is, to first order, the Dot Product \Delta L \approx \nabla L \cdot \Delta\theta. Among all steps of a fixed length, the one that makes \Delta L as negative as possible is the one pointing exactly opposite the gradient — that is what "steepest descent" means. Each update is therefore the greediest locally-improving move you can make.
Roll a ball down the loss surface
Below is a loss landscape drawn as contour lines — each ring is a level set, and the valleys (two of them here) glow brightest. A ball starts at a point and follows -\nabla L downhill, one step per frame, tracing its path behind it. Watch how it always cuts perpendicular to the contours, the steepest way down, and how its starting point decides which basin it falls into.
Click anywhere to drop a new ball and see where it rolls.