Policy Gradients and the Actor-Critic Split · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Train the policy itself, with a critic to steady it

Computing · Machine Learning · ages 23-24
Name ______________________   Date ____________

  1. Why does learning work even though the reward is not differentiable?

    • The reward is smoothed until it differentiates
    • You differentiate action log probabilities and weight by measured return
    • The gradient skips the policy and tunes the reward

  2. What does the policy output in this lesson?

    • A distribution over actions set by parameters
    • A fixed table of rewards
    • A copy of the training data

  3. Which part holds the policy, and which holds the value estimate?

    • The critic holds the policy and the actor holds values
    • One network holds neither and both are discarded
    • The actor holds the policy and the critic holds values

  4. What does subtracting a baseline change about the gradient estimate?

    • Its average direction, which makes learning faster
    • Its spread, while the average direction stays fixed
    • Its sign on every single step

  5. An action earned a high return. How do its parameters change?

    • Toward repeating that action in similar states
    • Away from that action in similar states
    • Toward repeating every action equally

  6. Why are both actor and critic kept instead of training the actor alone?

    • The critic supplies fresh experience while the actor steadies values
    • Two networks always beat one on any task
    • The actor supplies fresh experience while the critic steadies its updates

  7. A proposed baseline trick shifts the average update direction. It is still just a baseline.

    Circle one:   True   False

  8. A teammate says a baseline speeds learning by fixing the average gradient direction. What is wrong?

    • Baselines leave the average direction alone and only shrink its spread
    • Baselines must be fixed constants to have any effect
    • Baselines remove the need for any reward signal
LightMySky · lightmysky.comW1-mt_eXm6stg8zj-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Train the policy itself, with a critic to steady it W1-mt_eXm6stg8zj-s1

  1. You differentiate action log probabilities and weight by measured return · Only the log probability of taken actions is differentiated, scaled by return.
  2. A distribution over actions set by parameters · The policy maps each state to action probabilities you tune.
  3. The actor holds the policy and the critic holds values · Actor means policy, critic means value baseline.
  4. Its spread, while the average direction stays fixed · A baseline steadies updates without biasing their mean.
  5. Toward repeating that action in similar states · High return scales a nudge that raises the taken action's probability.
  6. The actor supplies fresh experience while the critic steadies its updates · Each one supplies what the other needs to keep learning well.
  7. False · Moving the average adds bias, so it fails the baseline test.
  8. Baselines leave the average direction alone and only shrink its spread · The average was already right; the baseline only calms the noise.
Worksheet · LightMySky