principles.fyi · the brain · concept

query, key, value

Each word asks a question, advertises what it offers, and carries its content.

attention(Q,K,V) = softmax(Q·Kᵀ / sqrt(d)) · V

Attention turns every token into three vectors by multiplying its embedding by three learned weight matrices. The query is what this token is looking for, the key is what each token advertises about itself, and the value is the information it will hand over. A token compares its query against every key with a dot product to score who matches; those scores are then turned into weights (via softmax, so they sum to one), and the weights decide how much of each value gets blended into the result. Splitting "asking" from "offering" from "content" is what lets the model learn flexible relationships rather than fixed ones.