encoder

A transformer that reads a whole sentence at once to understand it, rather than write new words.

encoder(tokens) -> one context vector per token

An encoder is a stack of transformer blocks whose job is to turn text into rich, context-aware vectors — one per token — that capture meaning, not to produce the next word. It's almost the same machinery as Book 01's writer (attention, feed-forward, residual stream, layer norm), with one crucial change: it uses bidirectional attention and no causal mask, so every token sees the entire sentence. The output is a set of contextual embeddings you can hand to a small task head for things like classification or tagging.

Appears in

Two ways to read Masked Language Models · pt 1
Put it to work Masked Language Models · pt 5
Three shapes of a model LLMs in the Wild · pt 1

Nearby in the brain

attention conditional generation decoder encoder-decoder foundation model generative AI BERT bidirectional