Loss Landscape

3 min read#optimization

The terrain an optimizer walks — a surface over parameter space whose height is the loss, dotted with minima, saddles, and plateaus.

Contents

Loss Landscape

When you train a model, every possible setting of its parameters \theta earns a score: the loss L(\theta), a single number measuring how badly it does. Imagine plotting that score as a height above the space of all parameter settings. The result is a surface — the loss landscape — and optimization is nothing more than a hike across it toward the lowest valley. The objective function, the cost function, the loss surface: these are all the same terrain seen from different fields.

The landscape's shape decides whether learning is easy or agonizing. A smooth bowl is a gift; a jagged range of peaks, ditches, and dead-flat plains is a nightmare. Understanding the features below is how you reason about why descent sometimes sails to the answer and sometimes gets hopelessly stuck.

The features of the terrain

  • Global minimum — the genuinely lowest point; the best the model can possibly do.
  • Local minimum — a valley that is lowest in its neighborhood but not overall. Descent that wanders in can never climb out on gradient alone.
  • Saddle point — a spot that is downhill in one direction and uphill in another, like a mountain pass. The gradient vanishes there, so naive descent slows to a crawl even though it is not a minimum.
  • Plateau — a vast near-flat region where the gradient is tiny and progress stalls.
  • Basin of attraction — the catchment of a minimum: every start inside it rolls to the same bottom. In the language of dynamics, the minima are attractors and the basins are their domains.

A surface you can read

The plot below is a one-dimensional slice through a rugged loss landscape. Trace it with your eye: the deepest dip is the global minimum, the shallower dip is a local minimum that could trap a careless optimizer, and the gentle rise between them is a small ridge to be crossed.

A 1-D slice of a non-convex loss L(θ)
Valleys are minima, the overall upward sweep is a confining bowl, and the little wiggles are local traps. Descent from the left settles in a different valley than descent from the right.

Where the curve is a single clean bowl, optimization is trivial — that special, lucky case is convexity. Real models are usually non-convex, full of the bumps you see above, which is exactly why momentum, annealing, and clever initialization exist.

At a saddle point of a loss landscape, what is true of the gradient?

See also

Related topics

  • Gradient DescentThe fundamental learning algorithm — repeatedly step a little way downhill along the negative gradient of a loss until you reach a valley.
  • ConvexityThe property that a function is a single bowl with no false bottoms — the case where gradient descent is guaranteed to find the one true minimum.
  • Dynamical SystemA state together with a fixed rule that determines how that state evolves in time.
  • MomentumGive the optimizer inertia — accumulate a velocity from past gradients so it powers through ravines and coasts over small bumps.
  • Optimization & LearningHow a system searches a landscape of possibilities for its best point — the common thread linking gradient descent, evolution, and learning machines.
  • AttractorThe set a system settles onto after transients die away — a point, a loop, or something stranger.

Referenced by