Multi-head attention on “the cat chased the mouse”. Pick the word that is asking. Each of four hand-designed heads spreads that word’s attention differently over the words it may read: the previous word, the nouns, the verb, or the first word. Each head returns its own weighted average; the results are joined and mixed by W_O.

Which word is asking?

Four heads, four different weightings of the same words. Each head mixes its own values, the results are joined end to end, and W_O turns them back into one vector. These heads are hand-designed to show the idea; trained heads learn their own patterns.