Whisper encodes audio once, then uses cross-attention to write one text token at a time.
Whisper: encode once, then write
The encoder reads the audio. The decoder writes one token at a time while pointing back to useful audio positions.
Encoderlog-Mel frames → 2 convolution layers → audio states
Decoderspecial tokens + earlier text → next text token
An audio state is a list of learned clues for one shortened time position. Cross-attention lets the decoder give more weight to the states that matter for its next token.
Whisper special tokens · control markers that set the language and task
<|startoftranscript|> · begin
<|en|> · English
<|transcribe|> · task
<|notimestamps|> · no times
<|endoftext|> · stop
generated text so far
cross-attention over audio states · taller bar = more weight
AED means attention encoder-decoder. During teacher forcing, training gives the decoder the correct earlier tokens. During inference, meaning actual use, it must use its own earlier choices. Greedy search keeps one best next token; beam search keeps several drafts.