--- title: "Learning from Human Preferences: Reward Models and Policy Optimisation" description: "In this approach, people compare outputs for the same prompt and a reward model learns scalar preference scores. A language-model policy is trained against that reward, with a KL penalty discouraging " canonical: https://lightmysky.com/learn/computing/learning-from-human-preferences-reward-models-and-policy-optimisation-mt_yA1ud53Oy_ source: https://lightmysky.com/learn/computing/learning-from-human-preferences-reward-models-and-policy-optimisation-mt_yA1ud53Oy_.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Learning from Human Preferences: Reward Models and Policy Optimisation In this approach, people compare outputs for the same prompt and a reward model learns scalar preference scores. A language-model policy is trained against that reward, with a KL penalty discouraging output-distribution drift from a fixed reference. Fresh human evaluation checks whether higher scores reflect better outputs. Subject: Computing · Area: Machine Learning · Ages 23 to 24 Page: https://lightmysky.com/learn/computing/learning-from-human-preferences-reward-models-and-policy-optimisation-mt_yA1ud53Oy_ ## Ready when they can - Describe how pairwise comparisons become a scalar reward model - Explain why policy training uses a fixed reference and a KL penalty, and describe risks that can grow without that penalty - Give one way a policy scores well against a reward model while getting worse for people ## Lesson: Turn human comparisons into a reward signal People compare two outputs for the same prompt. A reward model scores each prompt-output pair with one number. It fits a comparison rule: equal scores predict a 50-50 choice, and the higher-scored output is predicted to win more often. Training fits human choices but may not match every label. These scores model relative preferences, not absolute quality. For goals that are hard to specify directly, they provide a learned reward signal. *(drawing: Comparison labels train the reward model. Its scores can still disagree with fresh human judgments.)* Train the language-model policy to seek high fitted reward. Keep a fixed reference policy, often a supervised fine-tuned model. A KL penalty measures output-distribution divergence from that reference, not distance between weight vectors. It trades off reward gain against drift. Removing this penalty can allow greater drift and more exploitation of reward-model errors, but neither outcome is inevitable. **Example.** Suppose long, flattering answers earn high reward-model scores. A policy starts adding praise and filler. Its scores rise, but fresh human comparisons rate the answers worse. This is reward hacking: the policy exploits flaws in the learned reward instead of meeting the intended human goal. A KL penalty can discourage drift, but cannot prevent every form of reward hacking. Evaluate new outputs with fresh human comparisons, including prompts not used to fit the reward model. Compare outputs from the new policy and the reference policy. A rising reward-model score alone does not prove improvement for people. If fitted scores and human judgments disagree, investigate the learned reward and policy rather than treating the score curve as proof. **Recap.** Fit preference scores from comparisons. Use a fixed reference and KL penalty to discourage drift, without assuming they prevent hacking. Check improvement with fresh human comparisons. *Lesson, questions and flashcards written for this dataset, including original preference-comparison calculations and a hypothetical reward-hacking example.* ## Practice 8 questions on this page, each with its working shown. ## Needs first - [Policy Gradients and the Actor-Critic Split](https://lightmysky.com/learn/computing/policy-gradients-and-the-actor-critic-split-mt_eXm6stg8zj) ## Optional connections - [Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates](https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc)