Follow a token through a transformer

Each token has a row of four numbers. Select a token to follow its row; select a number to see its calculation.

One causal attention head, one pre-normalized block, eight vocabulary tokens. The fixed, invented weights have not learned language.

Change the input

Numbered columns are features, not words. Shapes describe the full matrix; the strips below follow the selected row.

1

Token and position

2

Attention mixes positions

+

Add the attention update

3

Feed-forward transforms this row

+

Add the feed-forward update

4

Scores for the next token

Choose the next token

Choosing the largest probability is a decoding rule. Appending runs the same model on the longer input; the weights stay fixed. This example holds at most eight positions.

See all token rows and fixed parameter tables

Activations change with the input. Parameters stay fixed in this example. In a trained model, training would adjust these parameters.