Learning from Human Preferences: Reward Models and Policy Optimisation
In this approach, people compare outputs for the same prompt and a reward model learns scalar preference scores. A language-model policy is trained against that reward, with a KL penalty discouraging output-distribution drift from a fixed reference. Fresh human evaluation checks whether higher scores reflect better outputs.
What a learner can do afterwards
- Describe how pairwise comparisons become a scalar reward model
- Explain why policy training uses a fixed reference and a KL penalty, and describe risks that can grow without that penalty
- Give one way a policy scores well against a reward model while getting worse for people
1 · Read
People compare two outputs for the same prompt. A reward model scores each prompt-output pair with one number. It fits a comparison rule: equal scores predict a 50-50 choice, and the higher-scored output is predicted to win more often. Training fits human choices but may not match every label. These scores model relative preferences, not absolute quality. For goals that are hard to specify directly, they provide a learned reward signal.
Train the language-model policy to seek high fitted reward. Keep a fixed reference policy, often a supervised fine-tuned model. A KL penalty measures output-distribution divergence from that reference, not distance between weight vectors. It trades off reward gain against drift. Removing this penalty can allow greater drift and more exploitation of reward-model errors, but neither outcome is inevitable.
Suppose long, flattering answers earn high reward-model scores. A policy starts adding praise and filler. Its scores rise, but fresh human comparisons rate the answers worse. This is reward hacking: the policy exploits flaws in the learned reward instead of meeting the intended human goal. A KL penalty can discourage drift, but cannot prevent every form of reward hacking.
Evaluate new outputs with fresh human comparisons, including prompts not used to fit the reward model. Compare outputs from the new policy and the reference policy. A rising reward-model score alone does not prove improvement for people. If fitted scores and human judgments disagree, investigate the learned reward and policy rather than treating the score curve as proof.
Fit preference scores from comparisons. Use a fixed reference and KL penalty to discourage drift, without assuming they prevent hacking. Check improvement with fresh human comparisons.
Lesson, questions and flashcards written for this dataset, including original preference-comparison calculations and a hypothetical reward-hacking example.
2 · Watch
Take it off screen
Where it sits
Learn first
Optional connections
This opens up
Nothing builds on it yet.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.