Attention Is All You Need: read, then mix

When translating a sentence, the meaning of one word can depend on words far away. How can a model bring that information together?

The Transformer lets each position calculate how much to take from other positions, then combine what they contribute. That operation is attention. Vaswani and colleagues built a translation model around it in 2017. Read the original paper.

One calculation explains the central idea

Imagine three positions carrying the numbers 10, 20, and 40. To build a new number, take a quarter of the first, a quarter of the second, and half of the third:

Information	Share	Contribution
10	¼	2.5
20	¼	5
40	½	20

Add the contributions: 27.5. Giving the third position a larger share makes its contribution larger. Changing the shares changes the mixture.

In a model, each position carries a row of numbers—a vector. Apply this same arithmetic to every coordinate in the rows. Attention can therefore combine information from several positions into one updated representation.

We chose the shares here. The model needs a way to calculate them.

How the model chooses the shares

It transforms each position’s current row into three kinds of numbers:

• A query describes what this position will match against.
• A key gives another position something to be matched.
• A value supplies the information that position contributes.

Comparing a query with each key produces scores. Softmax turns those scores into positive shares that sum to one. Those shares mix the values, just as the fractions mixed 10, 20, and 40.

Queries, keys, and values come from learned transformations. They are numerical arrays; their coordinates do not come with human labels such as “grammar” or “meaning.”

Read alongside §3.2.1, equation 1: its order is compare → turn scores into shares → mix values. That is the calculation you have just followed.

What the whole Transformer adds

One mixture is only part of the model. Multiple heads make different query, key, and value transformations and combine the resulting mixtures. A feed-forward network then changes features within each position. A residual connection adds a sublayer’s update to the row it received. Position information tells the model where each piece occurs.

Figure 1 joins these operations into two stacks:

Part	Which information can it read?
Encoder: process the source sentence	All source positions
Decoder self-attention: produce the translation	Earlier available target positions
Decoder attention over the encoder	The source representations

The decoder blocks future target positions because those pieces will be unavailable when it is generating a translation. During training, the target sequence is already known, so many positions can be processed together with that blocking in place. Generation still depends on earlier outputs. Architecture, §3.

What the paper demonstrated

The authors trained and evaluated the model on translation tasks, reporting strong English–German and English–French results and comparing model variants. These experiments established the usefulness of this architecture in those settings. They do not establish that every later model needs exactly the same design. Results, §6.

Check the idea

Does giving a position half the attention mean it caused half the final answer?

No. It supplies half of this local value mixture. Other heads, residual paths, transformations, and later layers also affect the answer. Attention shares describe this operation; they are not a complete account of causal influence. See interpretability.

Read the equation one symbol at a time

$$A=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right),\qquad Y=AV.$$

This writes the paper’s attention operation with an explicit mask M:

1. QKᵀ: calculate a query–key score for every position pair. If each row has dₖ numbers, multiply matching coordinates and add them.
2. Divide by √dₖ: control the scale of those scores.
3. Add M: leave allowed scores alone; set blocked scores to −∞.
4. Softmax: exponentiate each score and divide by the row total. Blocked scores contribute zero.
5. AV: use the resulting shares to mix the value rows.

For our example, scores 0, 0, and ln(2) exponentiate to 1, 1, and 2. Dividing by their sum gives ¼, ¼, and ½.

With n query rows and n key rows, QKᵀ contains n × n scores. Three positions have nine possible pairs; a causal mask allows only six.

Try the calculation with an inspectable model

Open the small Transformer calculation. Select the first token, change the final token, then inspect the first position again. Future information cannot affect that earlier position when the causal mask is in place.

This is a fixed, untrained decoder with one head. It demonstrates the arithmetic; it is not the paper’s trained translation model.

Next: give the same update rule another turn, or make reading a long history cheaper.