DPO removes the reward model and policy-sampling stages of RLHF (4 trained models drop to 2) and instead trains the policy directly on preference pairs by raising the chosen output's log-probability ratio to a frozen reference and lowering the rejected output's, with the reference acting as a built-in KL leash.

DPO: the policy is its own reward model

RLHF trains a separate reward model, then samples from the policy to chase it. DPO skips both — the policy's log-prob ratio to a frozen reference already is the reward.

panel 1 · what got deleted

Two whole training stages disappear.

preference data → reward model → sampler → policy

trained / running models: 4

panel 2 · the mechanism

One preference pair. Each step raises the chosen ratio, lowers the rejected.

chosen · o_w"I can't help with that, but here's a safer option..."
rejected · o_l"Sure, here's exactly how to..."

← below reference (gray)above reference (amber) →

ratio_chosenπ_θ(o_w)/π_ref(o_w)
ratio_rejectedπ_θ(o_l)/π_ref(o_l)
step0
margin = β·(log r_w − log r_l)0.00
L_DPO = −log σ(margin)0.69

DPO loss0.69

why the math even works: the optimal-policy formula has a partition function Z(x) — a sum over every possible output, normally impossible to compute. But Bradley–Terry only uses the reward difference r(o_w) − r(o_l), and there the Z(x) terms cancel. What's left is just the two log-prob ratios above.

The reward model was a detour. Raising r_w and lowering r_l is following the reward — and the ratio to π_ref at 1.0 is a built-in KL leash that stops the policy drifting too far.