Policy Gradients and the Actor-Critic Split
Policy gradients train a differentiable action distribution using sampled returns and gradients of action log probabilities. A suitable action-independent baseline preserves the expected gradient and can reduce variance. An actor learns the policy; a critic estimates returns.
What a learner can do afterwards
- Say why the policy gradient works even though the reward is not differentiable in the parameters
- Explain the conditions under which a baseline preserves the expected sampled-return gradient, and why variance reduction is not guaranteed
- Name what the actor and the critic each learn and why both are kept
1 · Read
Consider finite episodes without discounting. A policy gives action probabilities controlled by parameters. Assume differentiable, positive probabilities and an environment independent of those parameters. Sample an episode. For each action, weight its log-probability gradient by the measured return from that action onward. Sum these contributions for gradient ascent on expected return. Only policy log probabilities are differentiated, so rewards need not be differentiable.
Subtract a baseline from each sampled return before weighting the log-probability gradient. Use a state-only baseline fitted before collecting these actor samples, and hold its value fixed during actor differentiation. The weight is return minus baseline. This preserves the expected gradient. A useful baseline can reduce variance, but a poor one can increase it. A negative weight reverses the contribution.
The actor learns the policy. The critic estimates remaining return under that policy, often as state values. In this on-policy setup, actor rollouts supply critic data. The critic can learn from sampled returns or TD targets like those in the previous lesson. Its estimates can guide actor updates. An actor can also learn from sampled returns without a critic.
At a fixed state, the mean log-probability gradient over actions drawn from the policy is zero: action probabilities sum to one, whose derivative is zero. A state-only baseline fitted before sampling therefore subtracts zero in expectation. Finite batches can still have different averages. Action-dependent subtraction needs a bias check. Replacing sampled returns with bootstrapped targets from an inaccurate critic can bias the actor gradient.
Weight log-probability gradients by returns. A valid state-only baseline preserves expectation but need not reduce variance. A critic estimates returns; an actor can also learn without one.
Lesson, questions and flashcards written for this dataset, including original policy-gradient teaching sequences and baseline calculations.
2 · Watch
En anglaisTake it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.