principles.fyi · the brain · concept

unembedding

The final step that turns the model's thought-vector into a score per word.

logits = h @ W_unembed (W_unembed = embedding_table^T when weights are tied)

It's a big table of weights that takes the last token's vector and computes one number (a "logit") for every word in the vocabulary, by measuring how much that vector lines up with each word's direction. Higher alignment means a higher score, so the model is basically asking "which word does my final hidden state point toward?" Those scores are then run through softmax to become next-word probabilities. Often the same table used to turn words into vectors at the input is reused (transposed) here, so a word's "input" and "output" meaning share one set of weights — this is weight tying, and it saves parameters while keeping things consistent.