Transformers, ELI5

A language model like GPT or Claude does exactly one thing, over and over: guess the next word. This page builds that guess from scratch. Every step comes with the reason it exists and a small calculation you can check by hand.

Then follow every number through one block, see how the weights are learned, and why generating is fast.

One guess

Given a sequence of tokens (words, pieces of words, punctuation), the model outputs a probability for every possible next token. Given “the cat sat”, it might give on 40%, down 25%, and spread the rest over thousands of other tokens. Pick one, append it, and ask again. That is how it writes a paragraph.

So to understand a transformer, you only need to understand one guess: a sequence goes in, probabilities come out. Almost all the arithmetic inside is the dot product: multiply two lists pairwise and add them up.

A matrix multiply is a grid of dot products. A large model does billions of them for every token.

Words become vectors

Arithmetic needs numbers. A tokenizer gives each token an ID, but an ID is only an address: token 100 is not “twice” token 50. So each ID is used only to look up a row in a learned table, the embedding. Here are two invented rows of four numbers:

• cat → (0.9, 0.3, -0.2, 0.6)
• dog → (0.8, 0.4, -0.1, 0.5)

The table starts random. Training nudges it, and tokens used in similar contexts drift toward similar vectors, because similar vectors lead to similar predictions.

Similar means pointing the same way

A vector is an arrow. The dot product measures how much two arrows agree:

Here $\lVert a\rVert$ is the arrow’s length and $\theta$ the angle between the arrows. For the rows above,

a large positive number, since they point almost the same way. Dividing by both lengths leaves $\cos\theta$, the cosine similarity, which compares direction only.

Where each word sits

Attention (next section) treats the rows as a set: nothing in it looks at a row’s index. Shuffle the input and every word gets the same new vector, just in a different order. Without help, “dog bites man” and “man bites dog” would look identical to it.

The fix is to put position into the rows. The simplest version learns one vector per position and adds it:

Most current models use RoPE instead: they rotate each query and key by an angle proportional to its position, so that when position $i$ meets position $j$ only the offset $i-j$ survives.

Attention

Now “the cat sat” is a grid with one row per token. But each row describes only its own token: the row for sat doesn’t know a cat did the sitting. To predict what comes next, a position needs information from other positions, and which positions matter depends on the text. So the mixing weights must be computed from the rows themselves.

A soft lookup

Think of a dictionary. You bring a query, compare it with each entry’s key, and get back that entry’s value. Attention softens this: score every key, turn the scores into weights that sum to 1, and return the weighted average of the values. Each row $x$ makes all three with learned matrices:

• Query $q=xW_Q$: what this position is looking for.
• Key $k=xW_K$: what it offers to be matched on.
• Value $v=xW_V$: what it hands over when chosen.

Keys and values are separate because why a word is relevant differs from what it should contribute.

The recipe

For the position $i$ being updated, over every position $j$ it may read:

$d_k$ is the length of a query or key vector. The middle line is softmax: exponentiating makes every score positive, and dividing by the total makes the weights sum to 1.

Worked example. cat (position 2) may read positions 1 and 2. Say its scores are $s=(0,\ \ln 3)$ and the values are $v_1=(2,0)$, $v_2=(0,4)$.

• Exponentiate: $e^0=1$ and $e^{\ln 3}=3$.
• Divide by the total, 4: $a=(1/4,\ 3/4)$.
• Mix the values:

Queries and keys decided how much each position counts. Values decided what got mixed.

Why divide by √d

If the entries of $q$ and $k$ are random with variance 1, then $q\cdot k$ adds up $d_k$ such products, so its variance is $d_k$ and its typical size is $\sqrt{d_k}$. With $d_k=64$, scores of ±8 are common. A gap of 8 makes one weight $e^8\approx2981$ times another, so softmax picks a single position and the others get almost no gradient to learn from. Dividing by $\sqrt{64}=8$ brings scores back to about ±1.

Why mask the future

During training the model predicts every position’s next token at once. If position $i$ could read position $i+1$, it would just copy the answer. So scores for $j>i$ are set to $-\infty$, and $e^{-\infty}=0$ gives them zero weight. This is the causal mask. Generation never has future tokens anyway, so training now matches use.

All positions at once, and many heads

Stack the rows into matrices $Q, K, V$ (one row per position). Then $QK^\mathsf T$ holds every query–key score at once:

$M$ is the mask: 0 where $j\le i$, $-\infty$ where $j>i$.

One set of weights can follow one relationship. A word may need several at once: its subject, the previous word, a matching bracket. Multi-head attention runs $h$ narrower heads side by side, each with its own $W_Q, W_K, W_V$, then joins their outputs with one more matrix $W_O$. Training decides what each head tracks.

The feed-forward network

Attention moved information between positions. The feed-forward network (FFN) then processes each row on its own, with the same weights for every position:

1. Widen. $xW_1$ makes the row about 4× wider. Each new entry is a dot product with one column of $W_1$, so it measures how well the row matches that pattern.
2. Bend. $\max(0,\cdot)$, called ReLU, zeroes the negatives, so each feature switches on only when its pattern is present.
3. Narrow. $W_2$ maps back to the original width. Each active feature adds its own row of $W_2$, scaled by how strongly it fired.

Why the bend matters: without it, two linear maps in a row are just one linear map,

so the extra width would buy nothing.

Assemble the block

A block wraps each of the two layers the same way:

Normalize the input, compute an update, and add it back. The output has the same shape as the input, so blocks stack: GPT-3 has 96 of them, each with its own weights.

Why add instead of replace

1. Each layer only has to learn an edit. A layer with nothing to add can output about 0, and $x$ passes through untouched.
2. Learning signals get a direct route back. The slope of $x+f(x)$ is $1+f'(x)$. The 1 lets the training signal reach the first block even when every $f'$ is small; through 96 layers of plain $f$, those small factors would multiply toward zero.

Why normalize

The running sum drifts in scale from block to block, and softmax and ReLU behave very differently at different scales. Layer normalization rescales each row before a layer reads it:

$\mu$ and $\sigma$ are the row’s mean and standard deviation; $\gamma$ and $\beta$ are a learned scale and shift. For $x=(1,2,3,6)$, with $\gamma=1$ and $\beta=0$:

• Mean: $\mu=12/4=3$, so the deviations are $(-2,-1,0,3)$.
• Variance: $(4+1+0+9)/4=3.5$, so $\sigma\approx1.871$.
• Normalized: $(-1.069,\ -0.535,\ 0,\ 1.604)$.

(Real implementations add a tiny $\varepsilon$ under the square root to avoid dividing by zero.)

Count the parameters

With model width $d$, one block holds:

• Attention: $W_Q, W_K, W_V, W_O$, each $d\times d$, so $4d^2$ in total.
• FFN: $d\times4d$ plus $4d\times d$, so $8d^2$.

That is $12d^2$ per block. Check it on GPT-3: 96 blocks, $d=12{,}288$:

Add the embedding table ($50{,}257\times12{,}288\approx0.6$ billion) and you get the published 175 billion. Two-thirds of it sits in the FFNs.

Follow the numbers

Here is one complete block, computed. “the cat sat” enters as three rows of four numbers; the weights are invented but fixed, so every number can be checked. Select a token to follow its row, then select any number to see exactly which inputs and weights produced it.

From a vector to a word

After the last block, only the last position’s vector, call it $h$, is used for the guess. It is the only row that has read the whole input. Each vocabulary token $t$ has an output vector $w_t$, and its score is a dot product:

A token whose vector points along $h$ scores high. Stacking all the $w_t$ as columns of one matrix $W_U$ computes every score in one multiply, $z=hW_U$. These raw scores are called logits. (Some models reuse the embedding table as $W_U$; that is weight tying.)

Scores become probabilities

The same softmax as in attention:

Worked example. For scores $(0,\ \ln2,\ \ln3)$, the exponentials are $(1,2,3)$, which sum to 6, so the probabilities are $(1/6,\ 2/6,\ 3/6)$.

Only gaps matter: $p_t/p_u=e^{z_t-z_u}$, so adding a constant to every score changes nothing.

Temperature

Divide every score by a temperature $T$ first. That divides every gap by $T$. For two tokens whose scores differ by 1, the probability ratio becomes $e^{1/T}$:

Low $T$ stretches the gaps, so the top token takes over. High $T$ shrinks them, so the choice gets more adventurous. As $T\to0$ you always pick the top token, which is greedy decoding.

Pick, append, repeat

With 100,000 tokens, the long tail of unlikely ones adds up, and one bad draw can derail the text. So samplers cut the tail first, then rescale what’s left to sum to 1:

• Top-k: keep the $k$ most likely tokens.
• Top-p: keep the most likely tokens until their total reaches $p$, so the count adapts to how sure the model is.

Append the chosen token and run the model again. That loop is the whole of text generation.

How it learns

Every weight above was given. Training finds them from raw text and one question: how much probability did the model give to the token that actually came next?

Every position is a free example

the cat sat on holds three predictions: the → cat, the cat → sat, the cat sat → on. Thanks to the causal mask, one pass scores all of them at once, and the text is its own answer key. No labels are needed.

The loss: −log of the right answer’s probability

A good model makes real text likely. The probability of a whole text is the product of its per-token probabilities, $p_1p_2\cdots p_n$. Logs turn that product into a sum (and keep thousands of small numbers from underflowing), and a minus sign makes lower better:

Here $p_t$ is the probability the model gave the correct token at position $t$. This is the cross-entropy loss.

Confidently wrong is punished hardest: as $p\to0$, the loss grows without limit.

The correction is “predicted minus actual”

Let $y$ be 1 for the correct token and 0 elsewhere. The slope of the loss with respect to the scores is

Worked example. The model says $p=(0.5,\ 0.3,\ 0.2)$, and the right answer is token 2, so $y=(0,1,0)$:

Training steps against the slope. The right token’s score goes up by its shortfall (0.7), and each wrong token’s score goes down by the probability it took. Backpropagation (the chain rule, applied block by block) carries this signal back to every weight $w$, which moves a small step $\eta$ against its own slope:

Repeat over trillions of tokens and the weights end up encoding grammar, facts, and everything else that helps predict the next word.

Generating fast

Each new token needs keys and values for every earlier position. But those never change: position 3’s key depends only on tokens 1–3. So models store them in a KV cache and compute only the new token’s row. A step then costs $t$ query–key pairs instead of recomputing all $t(t+1)/2$.

The whole thing in code

import torch
import torch.nn as nn
import torch.nn.functional as F

class Block(nn.Module):
    def __init__(self, d, heads):
        super().__init__()
        self.norm1, self.norm2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.ReLU(), nn.Linear(4 * d, d))

    def forward(self, x):                              # x: (batch, n, d)
        n = x.size(1)
        future = torch.triu(torch.ones(n, n, dtype=torch.bool, device=x.device), 1)
        h = self.norm1(x)
        x = x + self.attn(h, h, h, attn_mask=future, need_weights=False)[0]
        x = x + self.ffn(self.norm2(x))
        return x

class TinyGPT(nn.Module):
    def __init__(self, vocab, d=64, heads=4, layers=4, max_len=256):
        super().__init__()
        self.tok = nn.Embedding(vocab, d)
        self.pos = nn.Embedding(max_len, d)
        self.blocks = nn.ModuleList(Block(d, heads) for _ in range(layers))
        self.norm = nn.LayerNorm(d)
        self.unembed = nn.Linear(d, vocab, bias=False)

    def forward(self, ids):                            # ids: (batch, n)
        positions = torch.arange(ids.size(1), device=ids.device)
        x = self.tok(ids) + self.pos(positions)
        for block in self.blocks:
            x = block(x)
        return self.unembed(self.norm(x))              # logits: (batch, n, vocab)

def train_step(model, optimizer, tokens):              # tokens: (batch, n + 1)
    logits = model(tokens[:, :-1])                     # predict from each prefix
    loss = F.cross_entropy(logits.flatten(0, 1), tokens[:, 1:].flatten())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss.item()

@torch.no_grad()
def generate(model, ids, steps, temperature=1.0):
    for _ in range(steps):
        context = ids[:, -model.pos.num_embeddings:]  # at most max_len tokens
        logits = model(context)[:, -1] / temperature   # last position only
        next_id = torch.multinomial(F.softmax(logits, dim=-1), 1)
        ids = torch.cat([ids, next_id], dim=1)          # append and repeat
    return ids