#information

Topics tagged "information".

• Entropy

  The average surprise of a random source, measured in bits — and the hard floor on how far it can be compressed.

• Shannon Information

  The information content of a single outcome — its surprise, equal to minus the log of its probability.

• Data Compression

  Encoding data in fewer bits by removing redundancy — with entropy as the unbreakable floor.

• Huffman Coding

  A greedy algorithm that builds the optimal prefix code — frequent symbols get short codewords, rare ones long.

• Mutual Information

  How many bits knowing one variable tells you about another — the information they share.

• Channel Capacity

  The maximum rate at which information can cross a noisy channel with arbitrarily small error.

• Information Theory

  The mathematics of measuring information — quantifying surprise, and the hard limits it sets on compression and communication.

• Error-Correcting Code

  Codes that add structured redundancy so corrupted bits can be detected — and even located and repaired.

• Hamming Distance

  The number of positions in which two equal-length strings differ — the metric behind error correction.

• Kolmogorov Complexity

  The length of the shortest program that outputs a string — making "random" mean "incompressible."