---
title: "Policy Gradients and the Actor-Critic Split"
description: "Instead of learning values and reading a policy off them, the policy can be a parameterised distribution trained directly by gradient ascent on expected return. The gradient is noisy, so a learned val"
canonical: https://lightmysky.com/learn/computing/policy-gradients-and-the-actor-critic-split-mt_eXm6stg8zj
source: https://lightmysky.com/learn/computing/policy-gradients-and-the-actor-critic-split-mt_eXm6stg8zj.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Policy Gradients and the Actor-Critic Split

Instead of learning values and reading a policy off them, the policy can be a parameterised distribution trained directly by gradient ascent on expected return. The gradient is noisy, so a learned value function serves as a baseline to hold the variance down.

Subject: Computing · Area: Machine Learning · Ages 23 to 24
Page: https://lightmysky.com/learn/computing/policy-gradients-and-the-actor-critic-split-mt_eXm6stg8zj

## Ready when they can

- Say why the policy gradient works even though the reward is not differentiable in the parameters
- Explain what a baseline changes about the estimator, and what it leaves alone
- Name what the actor and the critic each learn and why both are kept

## Lesson: Train the policy itself, with a critic to steady it

You keep a policy that outputs a distribution over actions, with parameters you can tune. After each episode you nudge the parameters toward actions that earned high return, using the gradient of the log probability of the actions you actually took. The reward itself is never differentiated, so it can be a win, a score, or any signal you can measure.

That gradient estimate is noisy because returns bounce around from episode to episode. You subtract a baseline, usually the learned value of the state, from the return before scaling the nudge. The subtraction shrinks the spread of the updates while leaving their average direction unchanged, so learning stays unbiased but steadier.

**Example.** Picture an agent learning a game. The actor holds the policy and picks each move. The critic holds a value estimate and reports how much better the result was than usual for that state. You keep both because the actor needs that steadying signal and the critic needs the actor's fresh experience to stay accurate.

**Tip.** When you meet a new baseline trick, ask two questions: did the average update stay the same, and did the spread shrink. If the average moved, the trick added bias. If only the spread shrank, it is a true baseline.

**Recap.** Nudge chosen actions by their return, steady the nudge with a value baseline, and keep both actor and critic.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Temporal-Difference Learning and Q-Learning](https://lightmysky.com/learn/computing/temporal-difference-learning-and-q-learning-mt_Z78E2llE0C)

## Opens up

- [Learning from Human Preferences: Reward Models and Policy Optimisation](https://lightmysky.com/learn/computing/learning-from-human-preferences-reward-models-and-policy-optimisation-mt_yA1ud53Oy_)