Temporal-Difference Learning and Q-Learning · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Learning values by bootstrapping

Computing · Machine Learning · ages 23-24
Name ______________________   Date ____________

  1. What does off-policy mean in Q-learning?

    • Learning the best policy while exploring with another
    • Learning only from perfect play
    • Freezing the policy during training

  2. What does a temporal-difference update nudge an old guess toward?

    • The reward just got plus the next estimate
    • The average of all past rewards
    • A random nearby value

  3. With a function approximator, bootstrapping can chase its own errors.

    Circle one:   True   False

  4. An agent takes a random exploring action. Why is the learned value still safe?

    • Random actions teach nothing at all
    • Exploring actions skip the update
    • The update uses the best next value

  5. After one update, which estimate changed?

    • The later estimate
    • The earlier estimate
    • Both estimates equally

  6. Each update assumes what about the next estimate?

    • It is exact already
    • It is closer to the truth
    • It will never change

  7. A student says exploring ruins the best policy. What is the mistake?

    • Exploring ruins every method
    • Off-policy targets best-next anyway
    • Best policies never need data

  8. Values swing wildly after you swap in a function approximator. What failed?

    • Exploration ran too hot
    • Rewards arrived too rarely
    • Bootstrapping chased its own errors
LightMySky · lightmysky.comW1-mt_Z78E2llE0C-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Learning values by bootstrapping W1-mt_Z78E2llE0C-s1

  1. Learning the best policy while exploring with another · Two roles, two policies. Behaviour explores while values aim at best.
  2. The reward just got plus the next estimate · Reward plus next estimate is the teacher. The old guess moves toward it.
  3. True · Guesses lean on guesses. Small errors can feed on themselves.
  4. The update uses the best next value · The target ignores the detour. Best-next is what counts.
  5. The earlier estimate · Only the earlier one moves. The later one waits its turn.
  6. It is closer to the truth · Bootstrapping borrows trust. The next guess is the steadier leg.
  7. Off-policy targets best-next anyway · Q-learning aims at best-next. Exploring feeds it data, not damage.
  8. Bootstrapping chased its own errors · The approximator spreads each error. Check the learning curve first.
Worksheet · LightMySky