The world picks the next word after "The capital of France is" from a fixed true distribution p: Paris 70 percent, beautiful 15 percent, not 10 percent, Lyon 5 percent. Four sliders set your model's belief q over the same words, auto-normalized. Gray bars show p, amber bars show q. A stacked bar decomposes your average surprise, the cross-entropy H(p,q), into the world's own entropy H(p), about 1.32 bits, an unbeatable floor, plus your penalty KL(p parallel q), which is at least zero and vanishes only when q equals p. Readouts show H(p,q), KL, and perplexity 2 to the H(p,q). Presets: matching the world gives KL zero; a uniform hedge costs 2 bits total; being confidently wrong blows past 4 bits and pins the bar with an overflow marker.

Reading the world with the wrong model

The world draws the next word from p. You read it believing q. Your average surprise is the cross-entropy H(p,q) = Σ p·log₂(1/q) — the world's own entropy (a floor no model can beat) plus a penalty for believing the wrong thing.

Your average surprise = the world's own unpredictability (a floor no model can beat) + a penalty for believing the wrong thing (KL, always ≥ 0, gone only when your q matches the world's p). Note that "confident & right" still pays — overconfidence even in the right answer costs bits. Training a language model minimizes exactly this: average surprise at the true next word. The floor is fixed — all learning can ever do is shrink the gap.