principles.fyi · the brain · concept

pretraining

The first, giant training run where a model reads huge amounts of text to learn language in general.

huge text corpus -> next-word prediction -> a model that has 'read' the web

Pretraining is the expensive first stage from Book 01: take a fresh model and train it on a massive pile of mostly-web text with plain next-word prediction, until it has absorbed grammar, facts, and patterns of the world. It's where almost all of a model's raw knowledge comes from — everything it 'knows' is a compressed echo of what it read here. Later stages (fine-tuning, RLHF) only polish behavior on top; they add very little new knowledge. So a model is, to a first approximation, what it was pretrained on.