DPO optimizes a preference margin relative to a frozen reference. The diagram contrasts common PPO and DPO components. The bars use scripted log-ratio changes and calculate their DPO loss; they do not train a model.

DPO: comparing preference margins

A common PPO-based pipeline trains a reward model and samples policy responses during updates. DPO instead trains on preference pairs using the policy and a frozen reference.

panel 1 · training components

DPO avoids a separate reward-model fit and online response sampling during its updates.

preference data → reward model → sampler → policy

Components in one common setup: policy · reference · reward · value

panel 2 · the mechanism

One illustrative preference pair. Scripted log-ratio changes show how the margin affects loss; β = 0.1.

chosen · o_w"I can't help with that, but here's a safer option..."
rejected · o_l"Sure, here's exactly how to..."

← below reference (gray)above reference (amber) →

ratio_chosenπ_θ(o_w)/π_ref(o_w)
ratio_rejectedπ_θ(o_l)/π_ref(o_l)
step0
margin = β·(log r_w − log r_l)0.00
L_DPO = −log σ(margin)0.69

DPO loss0.69

Why the prompt-dependent term cancels: the optimal-policy formula has a partition function Z(x) — a sum over every possible output, normally impossible to compute. But Bradley–Terry only uses the reward difference r(o_w) − r(o_l), and the shared β log Z(x) term cancels. The remaining margin is β times the difference of the two log-probability ratios.

DPO favors a larger chosen-minus-rejected log-ratio margin. That does not guarantee that the chosen response’s absolute probability rises on every update. The reference defines the comparison; it does not impose a hard maximum KL distance. These bars are scripted examples, not learned policy updates.