Backpropagation

5 min read#optimization

The chain rule run in reverse — compute the loss's gradient with respect to every weight in one efficient backward sweep through the network.

Backpropagation

A Neural Network is trained by Gradient Descent, which needs the gradient of the loss with respect to every weight — potentially billions of them. Computing each one separately would be hopeless. Backpropagation is the algorithm that gets them all in a single backward pass, and it is nothing more than the chain rule of calculus applied with ruthless efficiency.

The idea: a network is a long composition of functions, L = \ell(f_n(\cdots f_2(f_1(\mathbf{x}))\cdots)). The chain rule says the derivative of a composition is the product of the local derivatives along the way:

\frac{\partial L}{\partial \theta} = \frac{\partial L}{\partial a_n}\cdot\frac{\partial a_n}{\partial a_{n-1}}\cdots\frac{\partial a_{k}}{\partial \theta}.
(1)

Backprop computes this right-to-left. After a forward pass stores every neuron's activation, error is injected at the output and flows backward: at each layer the incoming gradient is multiplied by that layer's local derivative, handed to the layer before it, and used to update that layer's weights. One forward sweep, one backward sweep, and every gradient is known.

Error flowing backward

Below, a signal first runs forward through a chain of operations (faint pulses, left to right) to produce a loss. Then a gradient is injected at the loss and travels backward, and at each edge it is multiplied by that step's local derivative — the running product is the chain rule accumulating. Watch the gradient value change as it propagates back to the input.

The two passes of backpropagation along a chain. Forward (left→right) computes activations; backward (right→left) multiplies local derivatives via the chain rule so the gradient at every node is known after a single reverse sweep.

See also