Three transformer architectures

Causal, bidirectional, and encoder–decoder attention

The Transformer series follows a causal decoder. The same basic operations can be arranged in other ways. Start by asking which information is available at each position.

Architecture	Available context	Typical use
Causal decoder	Current and preceding input tokens	Predict the next token
Bidirectional encoder	Tokens on both sides within the input	Build representations for classification or retrieval
Encoder–decoder	Encoder reads the input; decoder reads earlier output plus encoder representations	Generate an output conditioned on a separate input

A decoder’s hidden states are contextual too. At the word “boring” in “the film was not boring,” a causal decoder can use the earlier “not.” An encoder additionally lets the representation at “not” use the later “boring.”

Why have a separate encoder?

For translation, an encoder can first represent the complete source sentence. A decoder then generates the translation token by token while attending to those source representations. The original Transformer paper used this arrangement.

A decoder-only model can also translate when given a source sentence in its prompt. Architecture enables certain computations; it does not assign each task exclusively to one model family.

The remaining pages examine prompting, training data, evaluation, and failure. The masked-language-model section explores encoders in more detail.

Sources · 4

Full bibliography →