Post-training · Part 5 / 7

Nudge it with the critic, on a leash

Use the reward model to push the model toward better answers — but keep it on a leash, or it will hack the imperfect reward and drift into nonsense.

We have a critic that scores any finished answer. Now use it to make the model itself better.

Writing is a sequence of choices

The model writes one word at a time. Pick a token, add it to the context, pick the next. Each pick is an action; the words so far are the state.

Call the model that does the picking the , written π. The score the critic gives the finished answer is the , r(x, o) — prompt x in, output o out.

So the goal is simple to state: find the policy whose answers score highest, on average, across many prompts.

π* = argmax_π  E[ r(x, o) ]    x ~ prompts,  o ~ π(o | x)

Sample prompts. Let the policy write answers. Average the reward. Push the policy toward whatever raises that average. That training loop is — reinforcement learning from human feedback.

Two things make this not a normal game

In a game the reward is real — you win or you lose. Here the is a learned, noisy stand-in for human taste. It is only ever a guess, and it has flaws.

And we don’t start from a blank slate. The policy is already a pretrained, instruction-tuned model — already good. We only want to nudge it toward preferred answers, not retrain it from scratch.

The danger: it hacks the critic

Maximize the reward alone and the policy drifts. It stops sounding like the model that learned from the whole internet and starts chasing whatever scores high — exploiting the reward model’s flaws until the text is degenerate. That is : the score climbs while the writing falls apart.

Try it: drop the leash all the way and read what the model produces. The critic loves it. You won’t.

The fix: keep it close to where it started

Add a penalty that pulls the new policy back toward the frozen starting model — the π_ref, the model just before this step. Measure the gap with , which tells you how far one probability distribution has wandered from another.

π* = argmax_π  E[ r(x, o) − β · D_KL( π_θ(o | x) ‖ π_ref(o | x) ) ]

The first term still says score high. The second says don’t drift far from where you started. The knob β is the leash: small lets the policy roam and risks hacking; large keeps it honest but barely moves it.

Go deeper: what the KL term actually is

For a language-model policy the KL penalty is just a log-ratio: for each generated token, compare the probability the trained policy gave it to the probability the reference gave it, and take the log of the ratio. When the two agree, the term is zero — no penalty. When the trained policy starts putting weight on tokens the reference thought unlikely, the log-ratio grows, and β turns that into a cost the reward has to outweigh. It is the same softmax probabilities from pretraining, read as a distance.

The optimizer that performs this nudge — sample answers, score them, step the policy uphill on reward minus leash — is usually , proximal policy optimization. You don’t need its gears here. Treat it as the engine that runs this exact objective.

The spine holds: a pretrained model writes the likely answer. The critic measures good. PPO closes the gap — on a leash, so closing it doesn’t break the model that made the answer fluent in the first place.

Sources · 6

Full bibliography →