From vectors to token scores

After the last block, there is a vector for every processed position. To predict the next token, use the final position’s hidden state, after any final normalization, and call it h.

Our tiny dictionary

This chapter isolates the output calculation with a separate six-token example. The complete explorer uses eight tokens and keeps its own vocabulary throughout.

For the input “the cat sat”, suppose the possible next tokens are:

on · down · there · still · up · quietly

Score each candidate using h. Choosing among those scores is a separate decoding step.

One dot product per word

Each column of the unembedding matrix Wᵤ contains weights for scoring one vocabulary token. Dot h with that column. Multiplying h by the whole matrix performs all those dot products at once, producing logits: one raw score per token. Some models also add an output bias.

Click a word's score to see the dot product behind it.

Try it

the bottom row is the six scores. Click one — it lights up h and that word’s column and writes out the dot product. A larger score gives that token a larger probability after softmax.

see this in PyTorch

Multiply the final vector by the (transposed) embedding table to get one score per word — here we reuse the input table (weight tying); some models learn a separate one.

import torch

# h: the final vector for the LAST token        # h: (d_model,)
# E.weight: the embedding table, shape (vocab, d_model)
# here we TIE weights (reuse the input table); many large models learn a separate output table instead
logits = h @ E.weight.T      # (d_model,) @ (d_model, vocab) -> (vocab,)
# one raw score per word in the vocabulary; biggest = top guess
next_word_id = logits.argmax()

Embedding: word → vector, going in. Unembedding: vector → words, coming out.

Optional: share the input and output parameters

When input and output weights are tied, the output matrix is the transposed input embedding table. Weight tying shares parameters; it is optional, not an identity that every model must satisfy.

Watch the embedding table become the unembedding — same word, both ends.

A separate output table can learn different vectors for scoring. Both arrangements produce one logit per vocabulary token.

Raw scores aren’t probabilities yet. Last step, next.

Sources · 4

1. Jurafsky, Daniel, and James H. Martin. Speech and Language Processing (3rd ed. draft, 2026), Chapter 8: “Transformers.”
2. Radford, Alec, et al. “Language Models are Unsupervised Multitask Learners.” OpenAI Technical Report, 2019.
3. Touvron, Hugo, et al. “LLaMA: Open and Efficient Foundation Language Models.” 2023. arXiv:2302.13971.
4. Press, Ofir, and Lior Wolf. “Using the Output Embedding to Improve Language Models.” EACL, 2017. arXiv:1608.05859.

Full bibliography →