Turn human comparisons into a reward signal

Computing · Machine Learning · ages 23-24

Name ______________________ Date ____________

What does the fitted reward model assign to a response together with its prompt?
- One written explanation of the human comparison decision
- One replacement policy trained on the preferred response
- One scalar preference score for that prompt-response pair
What data trains the comparison-based reward model in this lesson?
- Same-prompt output pairs with human preference labels
- Policy weight matrices paired with their update timestamps
- Random scalar targets paired with generated response lengths
What discourages output-distribution drift from the fixed reference policy?
- A KL penalty relative to the reference distribution
- A larger batch that keeps every output probability unchanged
- A faster optimiser that restores the original weight vector
A policy pads answers with flattery. Reward-model scores rise, but fresh human comparisons rate the answers worse. What is this?
- Baseline subtraction
- Data parallel training
- Reward hacking
With other training settings unchanged, which risk can increase when the KL penalty is removed?
- A guaranteed return to the reference policy's output probabilities
- Greater policy drift and exploitation of reward-model errors
- A requirement that the reward model stop scoring all new responses
For the same prompt and A/B pair, people pick B four times and A once. The taught comparison model exactly matches B's observed win rate of 4/5. Which score ordering follows?
- Output A has the higher score
- Output B has the higher score
- Both outputs must have equal scores
Keeping a KL penalty relative to a fixed reference guarantees that a policy cannot exploit flaws in its reward model.
Circle one: True False
A team's reward-model scores rise. Which check directly tests whether the new policy better meets human preferences?
- Increase the batch size until all training score curves remain stable
- Collect fresh human comparisons of new and reference outputs
- Reduce the KL penalty until the fitted reward scores increase faster

Answer key

For grown-ups. Fold this page away before handing over the rest.

Turn human comparisons into a reward signal W1-mt_yA1ud53Oy_-s1

One scalar preference score for that prompt-response pair · The reward model produces one number for a prompt-response pair. It is a learned preference score, not an absolute quality rating.
Same-prompt output pairs with human preference labels · People compare outputs for the same prompt. Their preferences provide the labels used to fit the reward model.
A KL penalty relative to the reference distribution · The KL penalty charges divergence between output distributions. It is not a distance between policy weight vectors.
Reward hacking · In this scenario, the policy exploits the learned reward's preference for flattery while making outputs worse according to fresh human comparisons.
Greater policy drift and exploitation of reward-model errors · Removing the KL term removes its pressure against reference-policy drift. Drift and exploitation of reward-model errors can increase, but need not.
Output B has the higher score · The assumed exact fit predicts B winning with probability 4/5, above 1/2. Under the taught comparison rule, B must have the higher score.
False · The KL term discourages distribution drift. It does not make the reward model accurate or prevent every form of reward hacking.
Collect fresh human comparisons of new and reference outputs · Fresh human comparisons test the preference goal. Changing batch size or the KL penalty does not directly measure people's judgments.