A reward model is a pretrained LLM with its vocabulary head removed and replaced by a single linear layer that outputs one scalar reward r(x,o). It is trained on preference pairs by minimizing -log sigmoid(r_chosen - r_rejected), which pulls the chosen answer's score up and the rejected answer's score down until the gap is wide and the loss approaches zero. The same scores let you pick the best of N sampled answers.

From judgment to a number

A model can't optimise "this answer is better." So we turn the judgment into a single number — a reward r(x,o) — and push it around with gradients.

1 · the head swap

embed x & o
↓
transformer block
transformer block
transformer block
→

word head → P(next token)

score head → 1 number

0.0 r(x, o)

The pretrained LLM ends in a vocabulary head: |V| bars, one probability per token. Click to replace it.

2 · train on one preference pair

prompt x

"How much garlic for one pizza?"
chosen o_w"Two cloves, minced — more if you love it."
rejected o_l"Garlic is a plant in the allium family."
r_w 0.0
r_l 0.0
gap  r_w − r_l 0.00
loss 0.69

L = −log σ(r_w − r_l)

step 0

Each step nudges r_w up and r_l down. The gap widens → σ(gap) → 1 → loss → 0.

3 · use it — best-of-N sampling

Same model, 4 samples. Score each with r(x,o), keep the highest.

The whole trick: turn a judgment (which is better) into a number (a reward) you can optimise.