Perceptron

6 min read#optimization

The simplest learning machine — a weighted sum past a threshold that learns to separate two classes by nudging its weights every time it errs.

Contents

Perceptron

The perceptron is the atom of machine learning: a single artificial neuron, invented by Frank Rosenblatt in 1958. It takes an input vector \mathbf{x}, forms the weighted sum \mathbf{w}\cdot\mathbf{x} + b — a Dot Product plus a bias — and fires +1 if that exceeds zero, -1 otherwise:

\hat{y} = \operatorname{sign}(\mathbf{w}\cdot\mathbf{x} + b).

Geometrically, \mathbf{w}\cdot\mathbf{x} + b = 0 is a line (a hyperplane in higher dimensions), and the perceptron simply asks which side of that line a point falls on. The weight vector \mathbf{w} is the line's normal direction; learning means rotating and shifting that line until it cleanly separates the two classes. It is the simplest linear decision rule there is.

The learning rule

The perceptron learns from its mistakes, one at a time. Show it a labeled point (\mathbf{x}, y). If it classifies correctly, do nothing. If it errs, nudge the weights toward getting that point right:

\mathbf{w} \leftarrow \mathbf{w} + \eta\,y\,\mathbf{x}, \qquad b \leftarrow b + \eta\,y.

That is it. This is a form of Gradient Descent on a hinge-shaped error, and Rosenblatt proved the perceptron convergence theorem: if the two classes can be separated by a line, this rule is guaranteed to find one in a finite number of updates.

Watch the line learn

Two clouds of points, two classes. The decision line starts at a random angle and rotates into place as the perceptron processes points and corrects its mistakes; misclassified points are ringed until the line sweeps to the right side of them. The arrow shows \mathbf{w}, the direction the line considers "positive."

Click to add a point (the label alternates each click) and watch the boundary adjust.

A perceptron learning to separate two point clouds. The line is the decision boundary w·x+b=0; the arrow is the weight vector w. Each misclassified point nudges the weights, rotating the line into a separator. Click to add points and perturb the problem.

See also

Related topics

  • Neural NetworkStack layers of simple neurons with nonlinear activations and they learn the curved decision boundaries a single perceptron never could.
  • Gradient DescentThe fundamental learning algorithm — repeatedly step a little way downhill along the negative gradient of a loss until you reach a valley.
  • BackpropagationThe chain rule run in reverse — compute the loss's gradient with respect to every weight in one efficient backward sweep through the network.
  • Linear TransformationA way of warping space that keeps grid lines straight, evenly spaced, and pinned to a fixed origin — exactly what a matrix encodes.
  • Optimization & LearningHow a system searches a landscape of possibilities for its best point — the common thread linking gradient descent, evolution, and learning machines.
  • Systems, Signals & ComputationAn interactive field guide to how things change, oscillate, propagate, and compute — from a swinging pendulum to chaos, waves, networks, and learning machines.

Referenced by