The Math Beneath · Part 5 / 8

What to expect

You can't know the next draw. You can know the average of all of them. Expectation is the number randomness is forced toward — and the license to learn from samples instead of the whole world.

A mystery box: 5% of the time it pays $100, 25% of the time $5, the rest — nothing. What is one play worth?

No single play answers that. Most plays pay $0; occasionally one pays $100. The honest question is: what do I get per play, on average, if I keep playing?

Weight each outcome by how often it shows up

Play 100 times and the world roughly keeps its promises: ~5 plays pay $100, ~25 pay $5, ~70 pay nothing. Total ≈ 5×$100 + 25×$5 + 70×$0 = $625. Per play: $6.25.

Shrink that reasoning into one line and you get the :

E[X] = Σ p(x) · x = 0.05×100 + 0.25×5 + 0.70×0 = 6.25

Each outcome, weighted by how often it happens. Not a prediction of the next draw — a property of the whole distribution: the number the running average is forced toward as plays pile up.

Try it: press play once a few times and watch the average thrash — $0, then $33, then $22… Now press play 5,000. The line calms and hugs $6.25. Then drag the $100 probability up and watch the target line move before you’ve played at all: expectation is knowable in advance.

The license to sample

Flip the logic around and you get the most load-bearing trick in machine learning. If the true average is what you want but the distribution is too big to touch — every possible sentence, every possible game — you can sample: draw randomly, average what you see, and trust the long run to drag your estimate toward the truth.

That’s the entire justification for training on batches. The loss a model should minimize is an expectation over everything it could ever see; the loss it actually computes is the average over a few random examples. The second is a noisy stand-in for the first — but a stand-in that converges, so noisy steps still walk in the right direction.

Go deeper: why the average settles

Each play differs from E[X] by some random error. Errors don’t pile up the way plays do — positive and negative ones cancel — so after n plays the total error grows only like √n while the number of plays grows like n. The per-play error therefore shrinks like 1/√n. That’s the law of large numbers in one breath, and it’s why the widget’s line calms down instead of wandering off. (It also explains the diminishing returns: each digit of accuracy costs 100× more plays.)

Now combine this chapter with the log from chapter one. Weight what by how often it happens? The next chapter picks the single most interesting choice: how surprised you are.

Sources · 2

Full bibliography →