Delete the critic

Last chapter’s recipe needed a whole pipeline: train a separate reward model to score answers, then keep sampling fresh answers from the policy and chasing that score. Four moving models. Slow. Touchy.

What if you never built the scorer at all — and trained the model straight on the preference pairs you already have?

The reward was hiding in the model

Start from the exact same goal as RLHF: get high reward, but don’t drift far from the reference you started from. That goal has a known best answer — and you can flip it inside out. Instead of reward → policy, write reward in terms of the policy:

r(x, o) = β · log( π(o|x) / π_ref(o|x) ) + β · log Z(x)

Read it plainly: a good answer is just one the trained model finds more likely than the reference does. That ratio is the reward. The ugly piece is Z(x) — a sum over every possible answer, far too big to ever compute.

The trick: it cancels

Here’s the gift. Preferences only ever compare two answers — the Bradley–Terry model cares only about the difference r(x, oᵢ) − r(x, oⱼ). Subtract the two rewards and the β · log Z(x) term, identical in both, simply cancels.

The intractable sum is gone. The chance a human prefers answer i over j becomes nothing but two log-ratios:

P(oᵢ ≻ oⱼ | x) = σ( β·log(π(oᵢ)/π_ref(oᵢ)) − β·log(π(oⱼ)/π_ref(oⱼ)) )

No scorer. No sampling. Just the model and its frozen copy.

The whole loss

So DPO trains directly on (winner, loser) pairs. For each pair, push the chosen answer’s log-ratio up and the rejected one’s down:

L = − log σ( β·log(π(o_w)/π_ref(o_w)) − β·log(π(o_l)/π_ref(o_l)) )

The loss encourages a larger preferred-versus-rejected log-probability-ratio margin. With shared model parameters, one update need not increase the preferred answer’s absolute probability and decrease the rejected answer’s in every case.

In the underlying KL-regularized reward objective, β is the penalty coefficient: larger β penalizes deviation from the reference more strongly for a fixed reward. In the DPO loss it also scales the preference margin. Its practical effect depends on optimization and the data; the reference is not a hard cap on model drift.

DPO fits the policy using these pairs without training a separate explicit reward model for this step.

Go deeper: why only two models, not four

PPO-style RLHF juggles four models at once: the policy being trained, a frozen reference for the KL term, the reward model, and a value network the critic uses to estimate advantage. DPO keeps just two — the policy and the frozen reference. The reward model is absorbed into the loss as a log-ratio; there’s no value network because there’s no on-policy rollout to estimate returns for. Removing rollout and reward-scoring stages simplifies this training loop. Implementations can share components or precompute reference scores, so these are logical roles rather than a universal count of separately resident models.

DPO removes the need for fresh policy rollouts during each preference update. Preference data still have to be collected, and reference log-probabilities may be precomputed. Resource savings depend on the implementation.

Next: what happens when the leash slips and the model starts gaming the reward instead of earning it.

Sources · 2

Full bibliography →