principles.fyi · the brain · concept

token

A token is a small chunk of text the model reads or writes at a time.

text -> [piece, piece, ...] -> [id, id, ...] -> embedding vectors

A token is a piece of text — often a whole word, sometimes just a fragment like "ing" or "tion" — stored in a fixed vocabulary, where each entry has an integer id. (Some tokenizers also tack a space-marker like "▁" onto a word, so "▁the" is really the whole word " the", space included.) Text gets split into these pieces and swapped for their ids, so the model only ever sees a sequence of numbers, not letters. This middle-sized chunking is a sweet spot: smaller than full words (so rare or made-up words still fit by breaking into known pieces) but bigger than single characters (so sequences stay shorter and more meaningful). Inside a transformer, every id is looked up in an embedding table to become a vector, and those vectors are what attention and the rest of the network actually process.