The whole language model, as matrix multiplications

Input text → one predicted next token. Amber steps are matrix multiplies. Tap any step; drag the layer count.

matrix multiply input / output other op (softmax, ReLU, norm, add)

Inside one block: 6 of the 12 steps are matrix multiplies (amber) — the other half are softmax, ReLU, layer-norms and adds. About half and half by step count, yet the matmuls hold essentially all of the model's learned weights.

Tap any step to see what it does. The amber left-bar marks a matrix multiplication.