The cost of wrong beliefs

Last chapter’s world was honest: the probabilities weighting the average were the same ones generating the days. But a model doesn’t get that luxury. The world draws from p. You believe q.

Your surprise at each outcome is priced by your beliefs — log(1/q). How often each outcome actually arrives is set by the world — p. So your average surprise is:

H(p, q) = Σ p · log(1/q)

This is cross-entropy: expected surprise, experienced through the wrong beliefs.

The floor and the gap

Subtract the honest world’s entropy from it and watch it split:

H(p, q) = H(p) + KL(p ‖ q)

Two parts, with completely different characters:

• H(p) — the floor. The world’s own unpredictability. If tomorrow is genuinely 50/50, no model ever built will average under 1 bit of surprise. Not wrongness — weather.
• KL(p ‖ q) — the gap. The extra surprise you inflict on yourself by believing q instead of p. KL divergence is never negative, and it hits zero in exactly one place: q = p.

So “be a good model” has a precise meaning: you cannot lower the floor; you can only close the gap.

The world draws next words from p; your model believes q. The stacked bar is your average surprise: the gray floor is the world's entropy, the amber overhang is your self-inflicted penalty.

Try it

press match the world — the amber penalty vanishes and only the floor remains. Then confident & wrong — cross-entropy explodes, because paying log(1/q) with a tiny q on an outcome that actually happens is catastrophically expensive. Note that even confident & right carries a penalty: the world says 70% Paris, and claiming 97% is also a wrong belief.

Go deeper: perplexity, and why the gap can't go negative

Perplexity is cross-entropy walked back through the exponential: 2^H(p,q) — your average surprise re-expressed as “how many equally likely options it feels like you’re facing.” 1 bit of surprise ≈ choosing between 2; a strong language model’s ~3–4 bits per word ≈ perpetually facing an 8–16-way choice. It’s the standard scorecard for grading language models.

Why is KL ≥ 0? The log ruler is concave — it bulges upward, so averages taken under it can only lose. Any probability you overspend on outcomes the world under-delivers must be borrowed from outcomes the world over-delivers, and concavity prices the borrow above the lend. Honest beliefs are the unique free lunch.

This is the loss

You have now assembled, from scratch, the objective under nearly everything in this library:

• Pretraining a transformer minimizes cross-entropy against the true next word — become less surprised by real text; the irreducible entropy of language itself is the floor.
• Training the reward model and DPO minimize cross-entropy on preference verdicts — the same loss, because chapter three showed they’re logistic regression underneath.
• RLHF uses KL directly as a leash: stray from the reference model’s distribution and the divergence itself becomes a cost.

Today

One asymmetry worth tasting: KL(p‖q) punishes missing what the world does (assign tiny q where p is real, pay hugely), while KL(q‖p) punishes inventing what the world doesn’t do. Which direction you leash decides whether a model spreads itself thin or collapses onto safe answers.

The loss is now a number. The last chapter is about the only thing left: which way to move to make it smaller.

Sources · 4

1. Kullback, S., & Leibler, R. A. (1951). On Information and Sufficiency. Annals of Mathematical Statistics, 22(1), 79–86.
2. Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423.
3. Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley-Interscience.
4. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Ch. 3 (Probability and Information Theory), Ch. 4 (Numerical Computation), Ch. 8 (Optimization).

Full bibliography →