Likely is not good

A pretrained language model learns to assign probabilities to tokens in its training data. Some of that data may demonstrate useful answers; some may contain falsehoods, hostility, or unrelated continuations.

Accurately modeling that mixture does not by itself specify how an assistant should respond. Post-training adds examples or feedback aimed at desired behavior. Its success still needs to be evaluated.

Likely, not good

Consider an illustrative base-model continuation of this prompt:

Explain the moon landing to a six year old in a few sentences.

A helpful answer would be a simple explanation. One possible unwanted continuation is another instruction:

Explain the theory of gravity to a 6 year old.

It didn’t answer. It continued the pattern. Ask it to translate “The small dog” to French and it may just keep writing the English sentence — The small dog crossed the road — because that’s a likely next few words.

It isn’t broken. It’s doing exactly what it was trained to do. Likely and good are simply not the same target.

One prompt, two ways to read it. 'Likely' continues the text as it appears in the wild; 'good' answers the intent behind it. The replies are scripted contrasts. A base model can answer some requests; post-training aims to make desired behavior more reliable.

Try it

read each prompt as the base model does — what text usually comes next? — then read it the way you meant it. The gap between the two columns is the whole problem.

Two ways the gap shows up

Not helpful. The model ignores what you meant and just continues the words, like the examples above. The response fails to follow the intended request.

Not safe. Predicting likely text can mean producing text that is dangerous, false, or toxic — wrong medical advice, harmful suggestions, hateful or stereotyped language. Even a perfectly innocent prompt can lead there, because somewhere in all that training text, such words plausibly followed.

The root cause is one mismatch: the training goal (predict words) is misaligned with what we actually want (helpful, honest, harmless).

Closing the gap

Fixing this is post-training — extra training applied after pretraining. Its job is alignment: bending the model from likely toward good. The rest of this book builds it, in two main moves:

• Show it good answers. Fine-tune on examples of the right response to a request. This is instruction tuning (SFT).
• Teach it from judgments. Let it learn from which of two answers a human prefers. This is preference alignment (RLHF, DPO).

There’s a third lever, separate from both, applied at answer-time rather than during training — we save it for the last chapter.

Next: training on demonstrations.

Sources · 6

1. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 9 (Post-training: Instruction Tuning, Alignment, and Test-Time Compute).
2. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2203.02155.
3. Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020). RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. Findings of EMNLP 2020. arXiv:2009.11462.
4. Cheng, M., Durmus, E., & Jurafsky, D. (2023). Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models. ACL 2023. arXiv:2305.18189.
5. Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019). The Woman Worked as a Babysitter: On Biases in Language Generation. EMNLP 2019. arXiv:1909.01326.
6. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.14165.

Full bibliography →