principles.fyi · the brain · concept

vocabulary

The fixed list of every token the model can read or write.

The vocabulary is the complete, fixed set of tokens (word-pieces like "the", "ing", or "transform") that a model knows — often around 100k–200k these days, and it varies by model (e.g. ~128k in Llama 3, ~200k in GPT-4o). Every piece of text gets chopped into these tokens before the model sees it, and every token it generates must be one of them; the model can only ever pick from this list. It sits at both ends of the transformer: the input layer maps each token to a vector (the embedding), and the output layer produces one score for every token in the vocabulary, and the highest-scoring ones are the most likely next token.