A schematic reward-model head, a toy update of two independent reward scores, and an illustration of choosing the highest-scored answer. No language model runs in this widget.

From judgment to a number

A reward model learns a score from comparisons of answers. This schematic shows the output layer and the loss; its training panel adjusts two independent scores, rather than training a language model.

1 · the head swap

embed x & o
↓
transformer block
transformer block
transformer block
→

word head → P(next token)

score head → 1 number

0.0 r(x, o)

The pretrained LLM ends in a vocabulary head: |V| bars, one probability per token. Click to replace it.

2 · train on one preference pair

prompt x

"How much garlic for one pizza?"
chosen o_w"Two cloves, minced — more if you love it."
rejected o_l"Garlic is a plant in the allium family."
r_w 0.0
r_l 0.0
gap  r_w − r_l 0.00
loss 0.69

L = −log σ(r_w − r_l)

step 0

Here the two scores are independent variables, so each update raises r_w and lowers r_l. A real reward model shares weights across answers; an update need not improve every pair.

3 · use it — best-of-N sampling

These four illustrative scores show the best-of-N rule: keep the highest-scored candidate. The widget does not generate or evaluate real answers.

The whole trick: turn a judgment (which is better) into a number (a reward) you can optimise.