Post-training
How instruction examples and preference judgments change a pretrained model’s responses. Covers instruction tuning, reward models, RLHF, DPO, and additional computation when answering.
Contents
Section titled “Contents”- Likely is not good — Separate the next-token objective from desired assistant behavior.
- Show it good answers — Train on instruction–response examples.
- Comparing two answers — Model a preference using the difference between two scores.
- Learn a reward model — Fit those scores from preference judgments.
- Use the reward model — Optimize a policy with reward feedback and a penalty for departing from a reference.
- Direct preference optimization — Learn from preferred/rejected pairs without a separate reward model.
- More computation at answer time — Compare longer reasoning or repeated attempts; neither guarantees correctness.
Return to the AI overview or consult the bibliography.