Two panels. Panel one plots the surprise of a single event, s of p equals log base 2 of 1 over p, for p between 0.001 and 1; a slider moves an amber dot along the curve, reading 0 bits at p equals 1 (certainty), 1 bit at a fair coin flip, and about 10 bits at a 1-in-1024 event. Panel two sets tomorrow's weather with four sliders (sun, cloud, rain, snow) that auto-normalize to probabilities; each row shows its probability, its surprise in bits, and its entropy term p times surprise. Below, the entropy H is written out live as the expected-surprise sum with a 0-to-2-bit gauge and presets: always sunny gives 0 bits, a coin flip 1 bit, four equal outcomes 2 bits, and a 97-1-1-1 desert about 0.24 bits. Hovering a row highlights both its surprise and its term in the sum, showing that a rare event has huge surprise but a tiny entropy term.

Surprise, measured in bits

Certainty surprises nobody. Rarer means more surprising. Independent surprises add. Only one function does all three: s(p) = log₂(1/p). A whole distribution's entropy is its expected surprise.

Three demands — certainty surprises nobody (s(1) = 0), rarer must mean more surprising, and independent surprises add while their probabilities multiply — leave exactly one candidate: log(1/p). Entropy is just its average under the distribution itself — H = Σ p·log₂(1/p) — how hard your world is to predict. A rare disaster is enormous surprise but a small entropy term: it almost never happens.