Meaning, in context

The input embedding for a token ID is a fixed lookup during inference. Both the decoder in the Transformer series and the encoder here transform that input into contextual hidden states.

One word, two meanings

But mouse the animal and mouse the gadget are different ideas wearing the same spelling. The input lookup is the same for a given token ID. Later hidden states can use the surrounding text to distinguish these senses (polysemy).

The arrow should depend on the neighbours

A bidirectional encoder begins with a fixed embedding lookup, then transforms it using the whole sentence — a contextual embedding: a fresh arrow for every occurrence.

The two occurrences can now receive different representations. Those differences can help with word-sense disambiguation, although neat, separate clusters are not guaranteed.

Invented coordinates illustrate how context can distinguish occurrences of a word. These points and clusters are not measurements from a trained encoder.

Try it

start in static mode — every mouse sits on one dot. Flip to contextual and watch the dot shatter into clusters. Hover a point to read the sentence that put it there, and find the lonely outlier sense.

Today

Contextual representations occur in both encoders and decoders. The same machinery — read both ways, blend the context — is what lets one model tell a riverbank from a savings bank without ever being told the rule. The distinct sense clusters in this diagram are illustrative; real representations need not divide into perfectly separated regions.

Go deeper: why the vectors all look alike (anisotropy)

Some contextual embedding spaces have directions that are much more common than others. The cloud isn’t a balanced ball — it’s a thin cone. This is anisotropy.

Large or strongly shared dimensions can affect similarity measurements. Centering or rescaling dimensions may help a particular comparison, but changes the geometry and does not guarantee clean sense clusters. Evaluate the method for the model, layer, and task in use.

Same spelling, different point. Next: representing sentence pairs.

Sources · 5

1. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
2. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of NAACL-HLT 2018. arXiv:1802.05365.
3. Coenen, A., Reif, E., Yuan, A., Kim, B., Pearce, A., Viégas, F., & Wattenberg, M. (2019). Visualizing and Measuring the Geometry of BERT. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv:1906.02715.
4. Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of EMNLP-IJCNLP 2019 (pp. 55–65). arXiv:1909.00512.
5. Timkey, W., & van Schijndel, M. (2021). All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. In Proceedings of EMNLP 2021 (pp. 4527–4546). arXiv:2109.04404.

Full bibliography →