The Math Beneath · Part 1 / 8
How many multiplies?
Some things grow by adding, some by multiplying. The log is the bridge between the two worlds — one question, not a formula — and the reason so much of ML lives on a log scale.
Some quantities grow by adding: take a step, then another — 1, 2, 3, 4. Your position is the sum of your steps.
Others grow by multiplying: a rumor doubles each day it spreads — 1, 2, 4, 8, 1024 by day ten. Each day multiplies what’s already there.
Adding-growth we’re built for. Multiplying-growth breaks our intuition: chains of multiplies explode past anything we can picture, or shrink to dust. And machine learning is full of multiply-chains — a hundred probabilities multiplied together, a loss shrinking by a factor at a time.
One question
So here is the whole idea of the , before any formula:
How many multiplies does it take to get here?
1024 is a scary number on a plain ruler. But ask the question — how many ×2s? — and the answer is a calm 10. That answer is the log: log₂(1024) = 10. Not a formula to memorize. A count of multiplies.
Put values on one ruler and counts of multiplies on another, and the same numbers live in two very different places:
Try it: press ×2 a few times. On the plain ruler each jump is twice as wide as the last — 1→2 is invisible, 512→1024 is half the ruler. On the log ruler, every ×2 is the same one-tick stride. That’s the whole trick.
Multiplying becomes adding
Look at what the question buys you. To reach a·b, you do the multiplies for a, then the multiplies for b. Counts of multiplies just add:
log(a · b) = log(a) + log(b)
That one line is why logs are everywhere. Multiplication — the operation that breaks intuition — becomes addition, the one we’re built for. The is simply the same bridge crossed the other way: given a count of multiplies, it hands back the value.
Go deeper: the base is just the tick size
log₂ counts ×2s, log₁₀ counts ×10s. Same ruler, different tick spacing — like measuring one distance in inches or centimeters, the choice changes the number but never the geometry. Machine learning usually counts in multiples of (the “natural” log, ln) for one reason only: its calculus is the cleanest. Nothing conceptual hangs on the choice.
Where you’ve already used this
Silently, everywhere. The probability of a whole sentence is many per-word probabilities multiplied — so language models add log-probabilities instead of multiplying vanishing ones. only look like clean straight lines because both axes are log rulers. And the training loss you’ll meet in chapter seven is built directly out of log(1/p).
One ruler question down. Next: the ruler that needs this fix most desperately — probability itself.
Sources · 2
- MacKay, D. J. C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Ch. 3 (Probability and Information Theory), Ch. 4 (Numerical Computation), Ch. 8 (Optimization).