Shannon Information

3 min read#information

The information content of a single outcome — its surprise, equal to minus the log of its probability.

Contents

The shape of surprise
From one outcome to the average
See also

Shannon Information

The Shannon information (or self-information, or surprise) of an outcome is the amount you learn when it happens. Shannon set it equal to

I(x) = -\log_2 p(x) \quad \text{bits,}

where $p(x)$ is the probability of that outcome. The rule is simple and inevitable: the rarer the event, the more information it carries. Learning that a tossed fair coin came up heads ( $p = \tfrac12$ ) gives you exactly 1 bit. Learning the outcome of one fair six-sided die ( $p = \tfrac16$ ) gives $\log_2 6 \approx 2.585$ bits. Learning something certain ( $p = 1$ ) gives 0 bits — you already knew it.

The shape of surprise

Plotted against probability, surprise sweeps from a towering spike near the impossible ( $p \to 0$ , information $\to \infty$ ) down to nothing at the certain ( $p = 1$ ). Common events are cheap; rare ones are expensive. This is exactly why a good code spends short codewords on frequent symbols and saves the long ones for rarities.

From one outcome to the average

Entropy is nothing more than the average self-information of a source — each outcome's surprise $-\log_2 p_i$ weighted by how often it occurs:

H = \sum_i p_i\,I(x_i) = -\sum_i p_i \log_2 p_i.

So self-information is the atom and entropy is the expectation built from it. A single bit is the natural unit throughout: the information in one equally likely yes/no answer. Using base- $e$ instead of base-2 logarithms measures the same thing in nats; the choice of base is just a choice of unit.

Shannon Information

The shape of surprise

From one outcome to the average

See also