Policy Gradients and the Actor-Critic Split
Instead of learning values and reading a policy off them, the policy can be a parameterised distribution trained directly by gradient ascent on expected return. The gradient is noisy, so a learned value function serves as a baseline to hold the variance down.
What a learner can do afterwards
- Say why the policy gradient works even though the reward is not differentiable in the parameters
- Explain what a baseline changes about the estimator, and what it leaves alone
- Name what the actor and the critic each learn and why both are kept
1 · Read
You keep a policy that outputs a distribution over actions, with parameters you can tune. After each episode you nudge the parameters toward actions that earned high return, using the gradient of the log probability of the actions you actually took. The reward itself is never differentiated, so it can be a win, a score, or any signal you can measure.
That gradient estimate is noisy because returns bounce around from episode to episode. You subtract a baseline, usually the learned value of the state, from the return before scaling the nudge. The subtraction shrinks the spread of the updates while leaving their average direction unchanged, so learning stays unbiased but steadier.
Picture an agent learning a game. The actor holds the policy and picks each move. The critic holds a value estimate and reports how much better the result was than usual for that state. You keep both because the actor needs that steadying signal and the critic needs the actor's fresh experience to stay accurate.
When you meet a new baseline trick, ask two questions: did the average update stay the same, and did the spread shrink. If the average moved, the trick added bias. If only the spread shrank, it is a true baseline.
Nudge chosen actions by their return, steady the nudge with a value baseline, and keep both actor and critic.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.