principles.fyi Part 1 / 5

Masked Language Models · Part 1 / 5

Two ways to read

Book 01 read left-to-right to write the next word. There is another way — read both directions at once, to understand.

In Book 01 the model read left-to-right and wrote the next word. It hid the future with the — peeking ahead would be cheating.

That model is a writer. This book builds its sibling: a reader.

The fork

A writer reads one direction, because it only ever needs the past.

But to understand a sentence, you want the whole thing at once. “The movie was not boring” only makes sense if “not” can reach across to “boring”.

So make one change: drop the mask. Now looks left and right — it reads . That model is an . The famous one is .

Same sentence, two readers. The decoder's causal mask hides the future; the encoder removes it, so every word sees the whole sentence. The grid is the mask itself — the only thing that changed.

Try it: focus on “boring”, then flip to Encoder. Watch it reach back to “not” — the word that makes this review positive. The decoder never could.

That’s the entire architectural difference from Book 01. Same blocks, same attention — one rule deleted.

So what do we train it on?

A writer trains on “guess the next word.” But if our reader can see the whole sentence at once, that game is trivial — the answer is right there.

We need a different game. Next.

Sources · 5
  1. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). arXiv:1810.04805.
  2. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
  3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NeurIPS 2017). arXiv:1706.03762.
  4. Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J., & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663.
  5. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019. arXiv:1908.10084.
Full bibliography →