Learning from Human Preferences: Reward Models and Policy Optimisation · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Turn human comparisons into a reward signal

Computing · Machine Learning · ages 23-24
Name ______________________   Date ____________

  1. What does the fitted reward model give for a single response?

    • A paragraph of written feedback
    • A new policy network
    • A single scalar score

  2. What data trains the reward model?

    • Pairs of outputs with a human pick
    • Random numbers drawn from the policy
    • Copies of the pretrained weights

  3. What holds the policy near the pretrained model?

    • A penalty for drifting away from its starting point
    • A larger batch size
    • A faster learning rate

  4. A policy pads answers with flattery, scores rise, and readers rate it worse. What is this?

    • Baseline subtraction
    • Data parallelism
    • Reward hacking

  5. What happens when the drift penalty is removed?

    • Training runs slightly slower
    • The policy drifts into strange text that chases the score
    • The reward model stops updating entirely

  6. People pick output B over A in four out of five comparisons. After fitting, which scores higher?

    • Output A scores clearly higher
    • Output B scores higher
    • Both score exactly the same

  7. Rising reward-model scores alone prove the policy improved for people.

    Circle one:   True   False

  8. A team ships on a rising reward curve without reading outputs, and readers complain. What failed?

    • They forgot to enlarge the training batch
    • They trusted the stand-in score over fresh human re-checks
    • They set the drift penalty far too high
LightMySky · lightmysky.comW1-mt_yA1ud53Oy_-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Turn human comparisons into a reward signal W1-mt_yA1ud53Oy_-s1

  1. A single scalar score · Each response maps to one number that predicts human preference.
  2. Pairs of outputs with a human pick · Human picks on pairs are the supervision for the scalar.
  3. A penalty for drifting away from its starting point · Drift away from the start is charged, so the policy stays close.
  4. Reward hacking · The fitted score climbs while true human value falls.
  5. The policy drifts into strange text that chases the score · Nothing stops the policy from leaving sensible writing behind.
  6. Output B scores higher · The scalar follows the pattern of human picks.
  7. False · Scores can climb through gaming while readers grow unhappier.
  8. They trusted the stand-in score over fresh human re-checks · A climbing stand-in never proves people are happier.
Worksheet · LightMySky