Temporal-Difference Learning and Q-Learning
An agent that cannot see the transition probabilities can still learn a value by bootstrapping: move each estimate towards the reward plus the next estimate. Q-learning does this over state-action pairs and reaches the best policy while behaving by a different one.
What a learner can do afterwards
- Perform a temporal-difference update by hand and say which estimate moved and why
- Say what off-policy learning means and why exploring does not spoil the target
- Give one failure mode of bootstrapping with a function approximator
1 · Read
Some agents cannot see the odds of the world, so they must learn from experience. After each step you nudge one value estimate toward the reward you just got plus the next estimate. That trick is called bootstrapping: each guess leans on the next guess.
In a temporal-difference update, only the earlier estimate moves. It moves because it borrows the fresher news inside the reward plus the next estimate. Each update assumes the next estimate is closer to the truth than the one being fixed.
Q-learning stores one value for every state and action pair. It learns the best policy while following a different, more exploring one, and that split is called off-policy. Exploring cannot spoil the target because each update reaches for the best next value, not the action you happened to take.
Bootstrapping has a weak spot you should watch. When a function approximator stands in for the value table, each update can chase its own errors and the values can swing or blow up. Track the learning curve, and mistrust any run you cannot explain.
Nudge each estimate toward reward plus the next estimate, learn the best values while exploring freely, and watch for bootstrapping to chase its own errors.
2 · Watch
Take it off screen
Where it sits
This opens up
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.