principles.fyi contents

principles.fyi · Topic

Transformers, ELI5

How a transformer language model actually works — from “guess the next word” to the whole machine, one widget at a time.

Contents

01 The one thing a language model does A language model does one thing — guess the next word, then do it again. Everything else is machinery for that one guess. 02 Meaning is a direction Turn each word into a vector and meaning becomes geometry — similar words point the same way, and scoring a word is just a dot product. 03 One block: mix, then think A block makes two moves — attention lets words look at each other, the FFN lets each think alone — and hands back the same shape it got. 04 Inside the feed-forward network Widen, drop the negatives, shrink back. A bank of pattern-detectors that stores most of the model's knowledge. 05 Stack it ~96 times A block hands back the same shape it got — so you stack it. Zoom out, and the whole model is mostly matrix multiplies adding up. 06 Turning a vector back into words One last matrix multiply scores every word in the vocabulary — the Chapter 2 dot product, run against the whole dictionary at once. 07 From scores to a word — and the loop Softmax makes probabilities, temperature and top-k/top-p shape the pick, then the chosen word is appended and the whole model runs again. 08 How it learns The weights didn't come from nowhere. The model learned them by playing one game a trillion times — guess the next word, check, nudge. 09 Making it big, affordably Bigger models are predictably better — and a few tricks (the KV cache, LoRA) keep the cost in check. 10 Peeking inside Can we see what a trained model actually does? A little — induction heads copy patterns, and the logit lens watches a guess form layer by layer.

✦ AI Written with AI, reviewed by a human. Each topic links its sources.