PPO (proximal policy optimization)

The reinforcement-learning algorithm commonly used to do the RLHF policy update.

PPO is a general reinforcement-learning method, and in RLHF it is the workhorse that actually updates the policy to increase reward. Its defining trick is to limit how much the policy can change in a single update step, clipping the update so training stays stable rather than lurching and collapsing. That caution pairs naturally with the KL penalty toward the reference policy, and DPO can be seen as a way to reach a similar result without running PPO at all.

Appears in

Nudge it with the critic, on a leash Post-training · pt 5

Nearby in the brain

KL divergence policy reference policy reward reward hacking reward model RLHF