Entropy
The average surprise of a random source, measured in bits — and the hard floor on how far it can be compressed.
Contents
Entropy
Entropy is the average amount of surprise you get per symbol from a random source — the expected number of bits needed to name an outcome you do not yet know. For a Probability Distribution over symbols with probabilities p_1, p_2, \dots, p_n, Shannon defined it as
Each term weights a symbol's surprise -\log_2 p_i by how often it occurs. A source that always emits the same symbol has H = 0 — no surprise, nothing to learn. A source whose outcomes are spread out has high entropy, because every symbol genuinely tells you something new.
The binary entropy curve
For a single biased coin with P(\text{heads}) = p, the entropy is
It is zero at the certain extremes p = 0 and p = 1, and peaks at exactly 1 bit when p = \tfrac12. The curve's gentle dome is one of the most-drawn shapes in the field: a fair coin is maximally informative, and you lose information slowly as the coin tilts.
Feel the average surprise
Drag across the panel below to pour probability into four bins. Watch the entropy bar respond: pile everything into one bin and it collapses toward 0; spread it evenly and it climbs to its maximum of \log_2 4 = 2 bits.
Why it is the compression floor
Entropy is not just a tidy formula — it is a limit. Shannon's source-coding theorem says no lossless scheme can encode a source in fewer than H bits per symbol on average. Predictable sources (low H) compress well; truly random ones (high H) cannot be squeezed at all. Every method in Data Compression is a race to reach this floor, and Huffman Coding comes within one bit of it.
The same quantity reappears in dynamics. In a chaotic system the Lyapunov Exponent measures the rate at which stretching reveals ever-finer detail of the initial state — an entropy production rate, the speed at which the system manufactures new information.