dot product

Line up two lists of numbers, multiply pair by pair, add it all up.

a . b = a1*b1 + a2*b2 + ... + an*bn

A dot product turns two vectors into a single number: multiply each pair of matching entries and sum the results. That number is large and positive when the vectors point the same way (and are long), near zero when they point in unrelated, roughly perpendicular directions, and negative when they point opposite ways — so it scores how well two things "agree." In a transformer this is the core of attention: each word's query vector is dotted against every word's key vector, and a higher score means "this word should pay more attention to that one." (Those raw scores are then scaled and passed through softmax to become the actual attention weights.)

Appears in

The one thing a language model does Transformers, ELI5 · pt 1
Meaning is a direction Transformers, ELI5 · pt 2
Two sentences at once Masked Language Models · pt 4

Nearby in the brain

positional encoding BERT [CLS] token natural language inference next sentence prediction segment embedding [SEP] token autoregression