Train the policy itself, with a critic to steady it

Computing · Machine Learning · ages 23-24

Name ______________________ Date ____________

Why can the sampled-return policy-gradient estimator taught here use a reward that is not differentiable?
- It replaces every measured reward with a differentiable prediction before updating
- It differentiates policy log probabilities and weights them by measured returns
- It differentiates the reward signal while keeping all policy parameters unchanged
What does the policy output in this lesson?
- A parameterised distribution over actions
- A fixed reward table for every possible state transition
- A stored copy of every episode in the training dataset
In the actor-critic setup taught here, what does each part learn?
- The critic learns a policy; the actor estimates returns
- Both parts learn only the same fixed reward lookup table
- The actor learns a policy; the critic estimates returns
Subtract a state-only baseline fitted before collecting the actor samples, and hold it fixed during actor differentiation. What follows for the sampled-return gradient estimate?
- Its expected gradient changes to favour every action equally
- Its expectation is preserved; its variance may change
- Every sampled gradient becomes identical across all episodes
A sampled action has return 8 and state baseline 5. What weight multiplies its log-probability gradient in the actor update?
- 3
- -3
- 13
A team removes the learned critic. Which statement about actor training is supported by this lesson?
- The actor cannot compute a policy gradient unless a critic predicts every return
- The actor must replace its action probabilities with a fixed state-value table
- The actor can still learn using measured returns without a learned critic
Replacing measured returns with bootstrapped targets from an inaccurate critic always preserves an unbiased actor-gradient estimate.
Circle one: True False
A team subtracts a quantity that depends on the sampled action, without a bias correction. What must it check before claiming an unbiased gradient?
- Whether the subtraction changes the expected gradient
- Whether a larger batch makes all sampled gradients exactly identical
- Whether the subtraction removes every need for measured rewards

Answer key

For grown-ups. Fold this page away before handing over the rest.

Train the policy itself, with a critic to steady it W1-mt_eXm6stg8zj-s1

It differentiates policy log probabilities and weights them by measured returns · The derivative is of the policy's log probability. Measured return supplies the weight; the estimator does not differentiate through the reward.
A parameterised distribution over actions · Policy parameters control the probabilities of selecting actions. A reward table or stored episode archive is not the policy's output.
The actor learns a policy; the critic estimates returns · The actor learns action probabilities. The critic estimates expected remaining returns under the policy, often through state values.
Its expectation is preserved; its variance may change · These baseline conditions preserve the expected gradient. The variance can decrease or increase depending on the baseline.
3 · Use return minus baseline: 8 - 5 = 3. This is the weight of this gradient contribution, not a guarantee about every action probability.
The actor can still learn using measured returns without a learned critic · The actor can weight policy log-probability gradients with measured returns. A critic is useful in many methods but is not required for this estimator.
False · Inaccurate bootstrapped targets can bias the actor gradient. Replacing a return is different from subtracting a valid baseline from a measured return.
Whether the subtraction changes the expected gradient · The state-only baseline argument does not cover action-dependent subtraction. Its expected contribution may be nonzero.