KL divergence

A measure of how far one probability distribution sits from another.

D_KL(P || Q)

KL divergence quantifies how much one distribution P differs from a reference distribution Q — it is zero when they are identical and grows as they diverge (and it is not symmetric). In RLHF it is used as a penalty: the tuned policy's token distribution is compared against the frozen reference policy's, and straying far is punished. That penalty is the leash that stops the policy from chasing high reward into degenerate text, balancing reward against staying recognizably like the original model.

Appears in

Nudge it with the critic, on a leash Post-training · pt 5
Delete the critic Post-training · pt 6
The cost of wrong beliefs The Math Beneath · pt 7

Nearby in the brain

RLHF DPO reward policy reference policy reward model cross-entropy loss PPO