The Math Beneath · Part 6 / 8

Surprise, measured

Three demands — certainty surprises nobody, rarer means more surprising, independent surprises add — leave exactly one formula. Average it and you get entropy, how hard your world is to predict.

The sun rose this morning. You felt nothing. A stranger correctly guessed your phone number. You’d fall off your chair.

Surprise clearly has a size, and the size clearly depends on probability. Can we turn it into an honest number? Write down only what surprise must obviously satisfy:

Certainty is no surprise. If p = 1, the surprise is 0.
Rarer is more surprising. As p shrinks, surprise grows.
Independent surprises add. Two coin flips both landing heads should surprise you exactly twice as much as one. But their probabilities multiply (½ × ½ = ¼).

Look at demand 3. Probabilities multiply; we want the measure to add. Chapter one built exactly one tool for that job — the . There is only one candidate left standing:

surprise(p) = log(1/p)

Check it: p = 1 gives 0. Smaller p gives more. And log(1/(p·q)) = log(1/p) + log(1/q) — independent surprises add. Measured in base 2, the unit is the bit: a coin flip is 1 bit, a 1-in-1024 event is 10 bits.

Entropy is expected surprise

One event has a surprise. What about a whole world — a distribution? Chapter five says how to summarize anything per-draw: take the . The expected surprise of a distribution is its :

H = Σ p · log(1/p)

How surprised you are on an average day. A world that’s always sunny has entropy 0 — nothing ever surprises you. A fair coin: 1 bit. Four equally likely weathers: 2 bits, the maximum, because nothing is a better guess than anything else.

Try it: in the bottom panel, give snow 1%. Its surprise is huge (~6.6 bits) — but its entropy term is tiny, because it almost never happens. That split — rare events scream loudly but count rarely — is the whole texture of entropy. Then hit always sunny and watch the world’s unpredictability drain to zero.

Go deeper: entropy is the price of describing the world

Shannon’s result, in one sentence: if you design the best possible code for a source — short codewords for common outcomes, long for rare ones — the average message length you can achieve is exactly the entropy, log(1/p) bits spent on each outcome. So “expected surprise” and “unavoidable description cost” are the same number. That equivalence is why entropy shows up wherever prediction, compression, or communication is happening — predicting well is compressing well.

Where you’ve already used this

Every time a language model is trained, its loss at one position is precisely log(1/p) — the model’s surprise at the true next word. A good model is one that is rarely surprised by real text. The whole of pretraining is: become less surprised by the world.

Which raises the sharp question the next chapter answers: what does average surprise look like when your probabilities are wrong?

Sources · 3

Full bibliography →