#optimization

Topics tagged "optimization".

• Gradient Descent

  The fundamental learning algorithm — repeatedly step a little way downhill along the negative gradient of a loss until you reach a valley.

• Loss Landscape

  The terrain an optimizer walks — a surface over parameter space whose height is the loss, dotted with minima, saddles, and plateaus.

• Convexity

  The property that a function is a single bowl with no false bottoms — the case where gradient descent is guaranteed to find the one true minimum.

• Momentum

  Give the optimizer inertia — accumulate a velocity from past gradients so it powers through ravines and coasts over small bumps.

• Simulated Annealing

  Escape local minima by sometimes stepping uphill — with a probability set by a temperature that slowly cools from bold exploration to careful descent.

• Newton's Method

  Use curvature, not just slope — fit a tangent (or a parabola) and jump straight to where it predicts the answer, converging breathtakingly fast.

• Optimization & Learning

  How a system searches a landscape of possibilities for its best point — the common thread linking gradient descent, evolution, and learning machines.

• Genetic Algorithm

  Optimize with no gradient at all — keep a population of candidate solutions, let the fittest reproduce with mutation and crossover, and watch quality climb.

• Perceptron

  The simplest learning machine — a weighted sum past a threshold that learns to separate two classes by nudging its weights every time it errs.

• Neural Network

  Stack layers of simple neurons with nonlinear activations and they learn the curved decision boundaries a single perceptron never could.

• Backpropagation

  The chain rule run in reverse — compute the loss's gradient with respect to every weight in one efficient backward sweep through the network.