The Math Beneath · Part 3 / 8

Evidence adds points

Independent clues multiply your odds — so on the straight ruler they add. Sum the points, take the trip back through the sigmoid, and logistic regression falls out. So does the reward model.

Something is behind a door. Cat or not-cat? Before any clue arrives, call it even: of 1:1.

Now a clue: it’s furry. Furry things are, say, four times likelier if it’s a cat. Your odds multiply by 4 → 4:1.

Another, independent clue: it meows. Ten times likelier from a cat. Multiply again → 40:1.

That’s the law of evidence: independent clues multiply the odds. Each clue scales your belief by how much better the cat-hypothesis explains it.

Multiply on one ruler, add on the other

But chapter one told us what to do with anything that multiplies: look at it on the log ruler. There, each clue becomes a fixed number of points — furry is +1.4, meows is +2.3, barks is −3.0 — and evidence simply adds:

0 + 1.4 + 2.3 = +3.7 log-odds → sigmoid(+3.7) ≈ 97.6% cat

Belief-updating becomes bookkeeping. Every clue contributes its points, positive or negative; the running total is where you stand on the straight ruler; the translates the total back into a probability whenever you want one.

Try it: turn on furry and meows, then add it barks. Watch the −3.0 points nearly cancel the evidence for cat — one strong counter-clue drags the whole total back toward unsure. The order you add clues never matters: addition doesn’t care.

You have just built logistic regression

Look at the machine you assembled: a list of features, each carrying a learned score; add up the scores of the features that are present; push the total through a sigmoid. That is logistic regression — the workhorse classifier of a century of statistics — derived from nothing but “clues multiply odds.”

And it’s why the story in the preferences chapter reuses everything here: the gap z_A − z_B is a total of evidence on the log-odds ruler, and training it uses the same loss as logistic regression, because underneath they are the same object.

Go deeper: softmax is the many-door version

Two hypotheses need one number (the log-odds). What if there are 50,000 doors — say, every word in a vocabulary? Give each candidate its own score z_i on the log ruler, undo the log to get each one’s strength e^(z_i), and divide by the total so the strengths become shares:

P(i) = e^(z_i) / Σ_j e^(z_j)

That’s — the sigmoid’s many-way generalization (the sigmoid is exactly softmax over two options with one score pinned at 0). It’s the machine at the mouth of every language model, turning scores back into words; is nothing but stretching or shrinking the score ruler before the trip back.

Evidence moves beliefs one clue at a time. But a walk is more than its steps — it matters where you started. Next: the number almost everyone forgets, and the famous “paradox” that dissolves the moment you remember it.

Sources · 3

Full bibliography →