principles.fyi · the brain · concept

reward model

A model that reads a prompt and answer and outputs one number for how good the answer is.

L = -log sigmoid(r(x,o_w) - r(x,o_l))

A reward model r(x,o) is typically an LLM with its next-token output head swapped for a single scalar head, so instead of predicting words it predicts one quality score. It is trained on preference pairs to give the chosen answer a higher score than the rejected one, using a loss that maximizes the sigmoid of their score difference — the Bradley-Terry assumption in action. In RLHF this learned scorer becomes the reward signal the policy is optimized against; its imperfections are what make reward-hacking possible.