The Math Beneath

Eight lessons on the mathematics used in these AI explanations: logarithms, odds, Bayes’ rule, expectation, entropy, cross-entropy, and gradients.

Contents

1. How many multiplies? — Some things grow by adding, some by multiplying. The log is the bridge between the two worlds — one question, not a formula — and the reason so much of ML lives on a log scale.
2. Probability is a squashed ruler — Equal steps in probability are not equal amounts of belief. Two moves — odds, then log — straighten the ruler, and the sigmoid appears as nothing more than the trip back.
3. Evidence adds points — Independent clues multiply your odds — so on the straight ruler they add. Sum the points, take the trip back through the sigmoid, and logistic regression falls out. So does the reward model.
4. Where the walk starts — A 99%-accurate test says you have a rare disease — and you’re probably fine. Paradox on the probability ruler, plain addition on the straight one. Bayes’ rule is chapter three plus a starting point.
5. What to expect — Expectation weights each possible value by its probability. A running average estimates it from repeated independent draws.
6. Surprise, measured — Three demands — certainty surprises nobody, rarer means more surprising, independent surprises add — leave exactly one formula. Average it and you get entropy, how hard your world is to predict.
7. The cost of wrong beliefs — Believe q while the world draws from p, and your average surprise splits in two — the world’s own entropy, a floor no model can beat, plus KL, the penalty for being wrong. That’s the training loss.
8. Downhill is a direction — A derivative measures how a small change affects a function. Gradient descent uses this information to adjust model weights and reduce loss.

Sources and further reading are in the bibliography.