Mutual Information

Mutual information measures how much learning one random variable reduces your uncertainty about another. If X and Y are two variables, their mutual information I(X;Y) is the number of bits that knowing X saves you when you try to guess Y (and, symmetrically, vice versa). It is the rigorous answer to "how related are these two things?" — sharper than correlation, because it captures any kind of dependence, not just linear.

In terms of Entropy it has a clean shape:

The first form says: the shared information is what you'd double-count if you added the two uncertainties separately. The second says: it is how much Y's uncertainty drops, from H(Y) down to the leftover H(Y\mid X), once X is revealed.

The overlap picture

Think of H(X) and H(Y) as two overlapping circles of uncertainty. Their union is the joint entropy H(X,Y); their intersection is the mutual information. As two variables become more dependent, the circles slide together and the shared sliver grows. The sketch lets you drag them.

Where it matters

Mutual information is the quantity Shannon maximizes to define Channel Capacity: the capacity of a noisy channel is the largest I(X;Y) achievable between its input X and output Y, taken over all input distributions. It is also the workhorse of machine learning and statistics — feature selection, clustering quality, and registration of medical images all hinge on measuring how much one signal tells you about another. Because it sees any dependence, it catches relationships that a plain correlation coefficient, blind to nonlinearity, would miss.

See also

• Entropy
• Channel Capacity
• Probability Distribution