Learn a critic

The pairing rule needed a hidden score for each answer: P(A beats B) = sigmoid(z_A − z_B). But nobody handed us those zs. All we collected was a pile of judgments — this answer beat that one — one bit per comparison.

So we don’t look the scores up. We learn a function that makes them.

A critic that outputs one number

Take a model that already reads language well — a pretrained model. Normally its last layer turns the final vector into a probability over the whole vocabulary: a guess at the next word. We don’t want a word. We want a verdict.

So strip that last layer off and bolt on a new one: a single linear layer that collapses the final vector down to one scalar. That number is the reward. Feed in a prompt and an answer, get back r(x, o) — how good this answer is for this prompt. That whole machine is the reward model.

The new layer starts with random weights — at first the score is noise. We have to teach it.

Push the winner up, the loser down

We already know the shape the scores should obey. Reuse it, but with the learned rewards in place of the hidden ones:

P(o_w beats o_l | x) = sigmoid(r(x, o_w) − r(x, o_l))

For one preference pair — winner o_w, loser o_l — we want that probability to be high. The loss is just the negative log of it:

L = −log sigmoid(r(x, o_w) − r(x, o_l))

This is the same cross-entropy idea used elsewhere in training: increasing the preferred answer’s score relative to the rejected answer lowers this pair’s loss. Average the loss over the judgments and run gradient descent. With shared model weights and many training pairs, one update need not improve every pair or move each score in a particular direction.

This is the pairing rule turned into a teacher: it never needs a “correct score,” only which of two answers won.

Go deeper: why one number is enough

The reward model need not output a calibrated quality on an absolute 0-to-10 scale. Adding the same constant to every reward leaves the loss unchanged because the differences stay the same. The size of a difference still matters: a gap of 1 predicts a preference probability of about 73%, while a gap of 4 predicts about 98%. The offset is arbitrary; the gap is more than a ranking.

What a critic is good for

Once you have a scorer, it pays off right away. Best-of-N: sample several answers from a model, score them all, keep the highest. Data cleaning: rate your training examples and toss the low-scoring ones.

But the real prize comes next: we point this critic at the model itself and let its scores reshape what the model writes.

Sources · 6

Full bibliography →