An RLHF training simulator. A beta slider sets how tightly the policy is leashed to its frozen reference model. As training runs, three meters update: the amber proxy reward the policy earns, the blue KL distance it has drifted from the reference, and the teal true quality the proxy does not fully track. At beta=0 reward climbs but KL explodes and real quality crashes as the sample output degenerates into repetitive high-reward gibberish (reward hacking). A middle beta lifts both reward and real quality while keeping KL bounded.

The reward leash

The reward model is a noisy proxy, not the truth. Chase it freely and the policy hacks it; tether it to where it started and it improves for real.

π* = argmaxπ  E[ r(x,o) − β·DKL( πθ ‖ πref ) ]
maximise the proxy reward, minus β times the distance from the frozen reference
reward — proxy score KL — leash / drift real quality — hidden truth
leash strength β0.20
0 · no leashstrong leash
try:

step 0 / 60 · press run

REWARD r(x,o) the proxy pays out0.00
KL FROM REFERENCE how far πθ drifted from πref0.00
REAL QUALITY true goodness — proxy can't see this0.00
sample output o ∼ πθ(o·x)—

x: “Was the rollout a success?”

reward model says— actually—

Watch β=0: reward keeps climbing while real quality collapses and the text turns to “great great great…” — the proxy is being hacked. A middle β keeps the KL bounded so reward and real quality rise together. Optimise the proxy reward — but don’t trust it too far; leash the policy to where it started.