Surprise, measured

A fair coin landing heads has probability 1/2. Ten fair, independent coins all landing heads have probability 1/1024. We want a measure that gives more information to the less likely event. Choose these properties:

1. Certainty is no surprise. If p = 1, the surprise is 0.
2. Rarer is more surprising. As p shrinks, surprise grows.
3. Independent surprises add. Two coin flips both landing heads should surprise you exactly twice as much as one. But their probabilities multiply (½ × ½ = ¼).

Look at demand 3. Probabilities multiply; we want the measure to add. Chapter one built exactly one tool for that job — the log. These properties determine the logarithmic form up to a positive scale factor:

surprise(p) = log(1/p)

Check it: p = 1 gives 0. Smaller p gives more. And log(1/(p·q)) = log(1/p) + log(1/q) — independent surprises add. Measured in base 2, the unit is the bit: a coin flip is 1 bit, a 1-in-1024 event is 10 bits.

Entropy is expected surprise

One event has a surprise. What about a whole world — a distribution? Chapter five says how to summarize anything per-draw: take the expectation. The expected surprise of a distribution is its entropy:

H = Σ p · log(1/p)

How surprised you are on an average day. A world that’s always sunny has entropy 0 — nothing ever surprises you. A fair coin: 1 bit. Four equally likely weathers: 2 bits, the maximum, because nothing is a better guess than anything else.

Top: the surprise curve — one event, one number. Bottom: shape a whole world with sliders and watch its entropy, the p-weighted average of every outcome's surprise.

Try it

Give snow probability 1%. Its information is about 6.6 bits, but its contribution to entropy is only 0.01 × 6.6 = 0.066 bits. Then select always sunny: one certain outcome gives entropy zero.

Go deeper: entropy is the price of describing the world

Shannon’s source-coding result connects entropy to lossless compression. For independent draws from a known finite distribution, encoding long blocks lets the average bits per outcome approach the entropy. A code for individual outcomes may need more: codeword lengths must be whole numbers, while log₂(1/p) need not be.

Where you’ve already used this

In next-token training, the cross-entropy loss for the observed token is −log(p), where p is the probability the model assigned to it. Training reduces this loss across the examples.

Which raises the sharp question the next chapter answers: what does average surprise look like when your probabilities are wrong?

Sources · 3

1. Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423.
2. MacKay, D. J. C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press.
3. Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley-Interscience.

Full bibliography →