principles.fyi · the brain · concept

hidden state

A token's running vector — its meaning so far, nudged layer by layer.

A hidden state is just the list of numbers (a vector) that stands in for one token at one point in the stack of layers. It starts as the word's embedding (its plain meaning, plus a hint of position). Then each layer updates it in two steps: attention mixes in information from the other tokens it attended to, and a feed-forward block reworks the vector on its own (this is where a lot of the model's stored knowledge gets applied). Each step adds to the vector rather than overwriting it — that is why it's a "running" vector — so by the top it encodes "this word, in this exact context" rather than the word alone. That contextual vector is what the next layer reads from, and at the very top the final token's hidden state is what gets turned into the next-word prediction.