The math inside a transformer

1 · Dot products

Multiply matching entries, add them up

Almost every step in a transformer is a dot product. Take two lists of numbers, multiply them position by position, and add the results:

q \cdot k = q_1k_1 + q_2k_2 + \dots + q_{64}k_{64}

When the two lists agree in sign at the same places, the products are positive and the total grows. When they disagree, the products cancel out. So a dot product measures how much two vectors point the same way.

These are real: the query at the last “to” and the key at “Mary” from attention head 9 in layer 9. The running total climbs to 49.4.

2 · Scores into probabilities

A score for every word

At the end, the model takes a dot product between its final vector $h$ and every word’s row in the token table. That gives one score per word, called a logit:

z_{\text{word}} = e_{\text{word}} \cdot h

Doing all 50,257 dot products at once is a matrix multiplication, $z = Eh$ . The scores can be any number. Here they sit around −93, and Mary’s is the highest at −74.8. Only the differences matter.

Exponentiate, then divide by the total

To turn scores into probabilities we need numbers that are positive and add up to 1. Softmax does both:

p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

$e^{x}$ makes everything positive, and it turns differences into ratios. A score 1 point higher means $e \approx 2.72$ times as likely. Mary beats “them” by 1.66 points, so she’s $e^{1.66} \approx 5.3$ times as likely: 67.7% against 12.8%.

Why attention divides by √64

Attention runs the same softmax over its query·key scores. But a dot product of 64 entries is a sum of 64 terms, so its spread grows like $\sqrt{64} = 8$ . Big gaps between scores make softmax all-or-nothing.

Raw, Mary’s score is 49.4 against 39.0 for ⟨start⟩: a gap of 10, which gives Mary 99.997%. Dividing by 8 shrinks the gap to 1.3, and the head keeps a blend: 67% Mary, 18% ⟨start⟩, 13% John. A blend lets the gradient reach every word, which is what learning needs.

\text{attention} = \operatorname{softmax}\!\left(\frac{q\cdot k}{\sqrt{d}}\right)

3 · Why logs

Products get tiny; logs turn them into sums

The probability of a whole sentence is the product of each word’s probability given the words before it. For our sentence that’s 2.8 × 10⁻²⁴, too small to picture, and computers lose precision on numbers like that.

Logarithms turn multiplying into adding:

\log(ab) = \log a + \log b

So instead of multiplying 14 probabilities we add 14 values of $-\log p$ . They total 54.2. The surprising words cost the most: the first “Mary” (0.0003, cost 8.1) and “drink” (0.0005, cost 7.6).

The log scale shows what linear hides

Here are all 50,257 probabilities, sorted. On ordinary axes you see one spike for Mary and a flat line for everything else.

On a log scale each step on the axis multiplies by 10 instead of adding a fixed amount. Now the shape appears: the probabilities fall smoothly over 14 orders of magnitude, down to $3\times10^{-15}$ for the least likely word. Log scales are how you see quantities that span many sizes at once.

The loss: −log of the right answer

Training needs one number that says how wrong the model was. It uses the cross-entropy loss:

\text{loss} = -\log p_{\text{right word}}

If the model is certain and right ( $p = 1$ ), the loss is 0. Here $p = 0.677$ , so the loss is 0.39. Had the answer been John, which it gave 2.3%, the loss would be 3.75. The curve shoots up near 0: being confident and wrong costs a lot.

4 · Derivatives

A derivative is a slope

How would the loss change if Mary’s score went up a little? Nudge the score by $h$ , see how the loss changes, and divide:

\text{slope} \approx \frac{\text{loss}(h) - \text{loss}(0)}{h}

As $h$ shrinks, that line through two points turns into the tangent: the derivative. For softmax plus −log it comes out exactly as

\frac{\partial\,\text{loss}}{\partial z_{\text{Mary}}} = p_{\text{Mary}} - 1 = -0.323

Negative means raising Mary’s score lowers the loss.

Every score’s slope: p − y

The same rule gives the slope for every word’s score at once. Let $y$ be 1 for the right word and 0 for the rest:

\frac{\partial\,\text{loss}}{\partial z_i} = p_i - y_i

So Mary’s slope is $0.677 - 1 = -0.323$ (push her up). Every other word’s slope is just its probability: “them” +0.128, “the” +0.048, “John” +0.023 (push them down). The more wrongly confident the model, the bigger the push.

5 · Backpropagation

The chain rule, run backward

The loss depends on $p$ , which depends on $z$ , which depends on $h$ . The chain rule says the slope through a chain is the product of the slopes of each link:

\frac{\partial\,\text{loss}}{\partial h} = \frac{\partial z}{\partial h}\,\frac{\partial\,\text{loss}}{\partial z} = E^{\top}(p - y)

Backpropagation is just this, applied from the loss backward through every step. The result is 768 numbers: a pull toward Mary’s direction in the token table and away from “them”, “the” and “John”, each weighted by its $p - y$ .

Back through all 12 layers

The same pass keeps going: through the final norm, then each block’s feed-forward and attention layers, all the way to the token table. Each block passes on how much the loss cares about its output, and works out the slope for each of its own weights.

Because each block adds to the running vector, the slope flows straight back through those additions and doesn’t fade to nothing in 12 layers. Every one of the 124 million weights ends up with its own slope from this single sentence.

6 · Learning

Take a step downhill

Now move every weight a small amount against its slope:

\theta \leftarrow \theta - \eta\, \frac{\partial\,\text{loss}}{\partial \theta}

$\eta$ (eta) is the learning rate, here $2\times10^{-5}$ . The curve is real: it’s the loss on this sentence if we moved all 124 million weights along the slope by different amounts. One step of size $\eta$ takes the loss from 0.39 to 0.30 and P(Mary) from 67.7% to 74.3%.

Repeat, and the model learns

Recompute the slopes, step again. After 10 real steps on this one sentence, the loss falls from 0.39 to 0.09 and P(Mary) rises to 91.5%, mostly taken from “them”.

Real training does exactly this, except each step averages the slopes over millions of words from many documents, so the model gets better at text in general instead of memorizing one sentence.