principles.fyi sources

principles.fyi · Masked Language Models

Sources

Every factual claim in this topic traces back to one of these 14 sources — the textbook spine plus the primary papers.

  1. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). arXiv:1810.04805. pt. 1, 2, 4, 5
  2. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models). pt. 1, 2, 3, 5
  3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NeurIPS 2017). arXiv:1706.03762. pt. 1
  4. Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J., & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663. pt. 1
  5. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019. arXiv:1908.10084. pt. 1, 4
  6. Taylor, W. L. (1953). "Cloze Procedure": A New Tool for Measuring Readability. Journalism Quarterly, 30(4), 415–433. pt. 2
  7. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI technical report. pt. 2
  8. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of NAACL-HLT 2018. arXiv:1802.05365. pt. 3
  9. Coenen, A., Reif, E., Yuan, A., Kim, B., Pearce, A., Viégas, F., & Wattenberg, M. (2019). Visualizing and Measuring the Geometry of BERT. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv:1906.02715. pt. 3
  10. Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of EMNLP-IJCNLP 2019 (pp. 55–65). arXiv:1909.00512. pt. 3
  11. Timkey, W., & van Schijndel, M. (2021). All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. In Proceedings of EMNLP 2021 (pp. 4527–4546). arXiv:2109.04404. pt. 3
  12. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692. pt. 4
  13. Williams, A., Nangia, N., & Bowman, S. R. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL-HLT 2018 (pp. 1112–1122). arXiv:1704.05426. pt. 4, 5
  14. Ramshaw, L. A., & Marcus, M. P. (1995). Text Chunking using Transformation-Based Learning. In Proceedings of the Third Workshop on Very Large Corpora (pp. 82–94). ACL. arXiv:cmp-lg/9505040. pt. 5