causal mask

It stops each word from peeking at the words that come after it.

softmax(score + mask)_j, where mask = 0 if j <= i else -infinity

A causal mask is a triangular pattern applied to the attention scores before they're turned into weights: for any position, the scores pointing at later positions are forced to -infinity. After softmax those scores become exactly zero (since e^(-infinity) = 0), so a token's output blends in only itself and earlier tokens. This matters because the model is trained to predict the next word, and letting it glance at the future would be cheating — it could just copy the answer. The mask keeps every position honest, so the same network can generate text one token at a time during use.

Appears in

One block: mix, then think Transformers, ELI5 · pt 3
Two ways to read Masked Language Models · pt 1
Fill in the blank Masked Language Models · pt 2

Nearby in the brain

attention BERT bidirectional encoder cloze task cross-entropy loss denoising [MASK] token