---
title: "Temporal-Difference Learning and Q-Learning"
description: "An agent that cannot see the transition probabilities can still learn a value by bootstrapping: move each estimate towards the reward plus the next estimate. Q-learning does this over state-action pai"
canonical: https://lightmysky.com/learn/computing/temporal-difference-learning-and-q-learning-mt_Z78E2llE0C
source: https://lightmysky.com/learn/computing/temporal-difference-learning-and-q-learning-mt_Z78E2llE0C.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Temporal-Difference Learning and Q-Learning

An agent that cannot see the transition probabilities can still learn a value by bootstrapping: move each estimate towards the reward plus the next estimate. Q-learning does this over state-action pairs and reaches the best policy while behaving by a different one.

Subject: Computing · Area: Machine Learning · Ages 23 to 24
Page: https://lightmysky.com/learn/computing/temporal-difference-learning-and-q-learning-mt_Z78E2llE0C

## Ready when they can

- Perform a temporal-difference update by hand and say which estimate moved and why
- Say what off-policy learning means and why exploring does not spoil the target
- Give one failure mode of bootstrapping with a function approximator

## Lesson: Learning values by bootstrapping

Some agents cannot see the odds of the world, so they must learn from experience. After each step you nudge one value estimate toward the reward you just got plus the next estimate. That trick is called bootstrapping: each guess leans on the next guess.

In a temporal-difference update, only the earlier estimate moves. It moves because it borrows the fresher news inside the reward plus the next estimate. Each update assumes the next estimate is closer to the truth than the one being fixed.

Q-learning stores one value for every state and action pair. It learns the best policy while following a different, more exploring one, and that split is called off-policy. Exploring cannot spoil the target because each update reaches for the best next value, not the action you happened to take.

**Tip.** Bootstrapping has a weak spot you should watch. When a function approximator stands in for the value table, each update can chase its own errors and the values can swing or blow up. Track the learning curve, and mistrust any run you cannot explain.

**Recap.** Nudge each estimate toward reward plus the next estimate, learn the best values while exploring freely, and watch for bootstrapping to chase its own errors.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Markov Decision Processes: States, Actions and Return](https://lightmysky.com/learn/computing/markov-decision-processes-states-actions-and-return-mt_bTCO27sgc6)

## Opens up

- [Policy Gradients and the Actor-Critic Split](https://lightmysky.com/learn/computing/policy-gradients-and-the-actor-critic-split-mt_eXm6stg8zj)