Generating “the cat sat on the mat” after the prompt “the cat sat”. Each grid shows the query–key score pairs one step computes. Without a cache, every step recomputes the whole triangle. With a KV cache, earlier keys and values are stored, so each step computes only the new word’s row.

Rows are the word asking (its query); columns are the words it may read (their keys). Step 1 reads the whole prompt either way. After that, the cache adds one row per step instead of redoing the triangle.