Neural Network

5 min read#optimization

Stack layers of simple neurons with nonlinear activations and they learn the curved decision boundaries a single perceptron never could.

Contents

Neural Network

A single Perceptron can only carve space with a straight line. Stack many of them in layers, feed each layer's outputs into the next, and you get a neural network — a function flexible enough to bend boundaries into any shape and approximate essentially any mapping from inputs to outputs. This is the engine of modern deep learning, and a many-layered network is still, at heart, a multilayer perceptron.

Each layer does two things. First a linear step — a matrix multiply by the layer's weights plus a bias, \mathbf{z} = W\mathbf{x} + \mathbf{b}. Then a nonlinear activation applied elementwise, \mathbf{a} = \sigma(\mathbf{z}). That nonlinearity is the whole trick: without it, stacking linear layers would just collapse into one big linear map (and one straight boundary again). With it, depth lets the network compose simple bends into arbitrarily intricate curved boundaries.

Signals flowing forward

Computing a network's output is forward propagation: activations flow left to right, layer by layer, each neuron summing its weighted inputs and squashing the result. Below, an input layer feeds two hidden layers and an output; pulses trace the connections, blue for excitatory (positive) weights and orange for inhibitory (negative) ones, and neurons brighten as the wave of activation reaches them.

A small multilayer perceptron computing forward. Activation flows input → hidden → hidden → output; pulses ride each connection (blue = positive weight, orange = negative), and nodes glow as the signal arrives. This forward pass is what backpropagation later differentiates.

The nonlinearity is the magic

The activation function bends straight lines. A common choice is the hyperbolic tangent, which smoothly saturates large inputs toward \pm 1:

Two activation functions: tanh (S-curve) and ReLU (a hinge)
Squashing nonlinearities like these let stacked layers represent curved, even disconnected, decision regions. Remove them and the whole network collapses back to a single linear map.

Train the weights — usually by Gradient Descent guided by Backpropagation — and a network with even one hidden layer becomes a universal approximator: given enough neurons it can represent any continuous function to any accuracy. Depth makes that representation efficient, building rich features as a hierarchy of simpler ones.

Why does a neural network need nonlinear activation functions between its layers?

See also

Related topics

  • PerceptronThe simplest learning machine — a weighted sum past a threshold that learns to separate two classes by nudging its weights every time it errs.
  • Gradient DescentThe fundamental learning algorithm — repeatedly step a little way downhill along the negative gradient of a loss until you reach a valley.
  • BackpropagationThe chain rule run in reverse — compute the loss's gradient with respect to every weight in one efficient backward sweep through the network.
  • Linear TransformationA way of warping space that keeps grid lines straight, evenly spaced, and pinned to a fixed origin — exactly what a matrix encodes.
  • Optimization & LearningHow a system searches a landscape of possibilities for its best point — the common thread linking gradient descent, evolution, and learning machines.
  • Systems, Signals & ComputationAn interactive field guide to how things change, oscillate, propagate, and compute — from a swinging pendulum to chaos, waves, networks, and learning machines.

Referenced by