Transformers

Follow one next-token prediction through a decoder-only Transformer: token IDs become vectors, the vectors exchange information, and the final scores become probabilities. Training changes the weights used in those calculations.

The diagrams use small examples so you can follow the arithmetic. When an illustration uses invented values or scripted behavior, its caption says so. The optional code and derivations can wait until the main explanation makes sense.

Follow the token flow — one complete, computed block. Select a number to see its inputs and arithmetic, or read the lessons below in order.

Follow a prediction

One next-token prediction — What the model receives and what it produces.
Tokens, vectors, and similarity — Token IDs, embeddings, dot products, and position information.
One block — Work through attention weights and a weighted sum with actual numbers.
Inside the feed-forward network — Transform each position’s vector using shared weights and a nonlinear function.
Stacking blocks — Follow the shapes through residual additions and normalization.
From vectors to scores — Calculate one score for each possible next token.
From scores to a token — Compute probabilities, choose a token, and continue the sequence.
How it learns — Turn target probabilities into a loss, then compute gradients and update weights.

Two further questions

Scale and computation — What caching, model size, and smaller updates change.
Looking inside a model — What internal patterns can reveal, and what they leave uncertain.

Further reading: Recurrence: reuse the update — run the same block on a changing hidden state, and distinguish that from generating another token or training the weights.

If the arithmetic is unfamiliar, dot products and matrix multiplication cover the operations used here. Derivatives help with the training chapter.

For prompting, training data, and evaluation, continue to LLMs in the Wild. Other topics are in the AI overview. The bibliography collects the sources.