principles.fyi · the brain · concept

embedding

A token's learned meaning stored as a list of numbers the model can work with.

embedding(token) = E[token_id] (one row of the table E)

An embedding is the vector (a fixed-length list of numbers) that a token gets turned into by looking up its row in the embedding table E. It works because the model can nudge these numbers during training, so tokens used in similar ways drift to nearby spots — directions in this space end up capturing real patterns of meaning. This is the transformer's entry point: it converts discrete tokens into numbers the attention and feed-forward layers can actually do math on.