Masked Language Models · Part 2 / 5
Fill in the blank
With both directions visible, next-word guessing is trivial. So we hide a word in the middle and guess that instead.
Book 01’s model read left to right and guessed the next word. It had to — the rest of the sentence was its own future, hidden behind the .
This model sees the whole sentence at once, both directions. So the next-word game breaks.
Now next-word is cheating
the cat sat on the ___
Easy — but only because the blank is at the end. Let the model peek at both sides and the answer is just sitting there. Nothing left to learn.
New game: hide a word in the middle
Cover a word inside the sentence and ask the model to recover it, using everything on the left and the right.
the cat ___ on the mat →
sat
That’s the whole training game. It has an old name — the , a fill-in-the-blank test — and a model trained this way is doing .
Think of it as repairing a damaged sentence: we add the damage on purpose, the model learns to undo it. The fancy word is .
Try it: drag the blank to different words. A word boxed in by neighbours (cat ___ on) is easy; a wide-open one is a real guess. Same as Book 01’s training — just with the hole in the middle instead of the end.
Same scoring as before
Grading works exactly like Book 01: the model gives every word in the vocabulary a probability, and punishes it for putting low probability on the true word. The only twist — we score only the hidden positions. The untouched words are just there for context.
Go deeper: how BERT actually picks the holes
We don’t blank everything — just ~15% of the words. But there’s a catch the original recipe (BERT) had to fix.
The obvious move is to swap each chosen word for a special token. Problem: real sentences at use-time have no [MASK] in them. A model that only reacts to that flag would be useless on normal text.
So of the ~15% picked:
- 80% → replaced with [MASK] — the main fill-in-the-blank drill
- 10% → replaced with a random word — now it must double-check words that look fine
- 10% → left unchanged — so it keeps a real opinion at every position, flag or no flag
The 20% that isn’t a [MASK] is the trick: it forces the model to stay alert at every position, not just the obvious holes.
Next: why reading both directions gives a word a sharper meaning than reading one.
Sources · 4
- Taylor, W. L. (1953). "Cloze Procedure": A New Tool for Measuring Readability. Journalism Quarterly, 30(4), 415–433.
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). arXiv:1810.04805.
- Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI technical report.