What does off-policy learning mean in Q-learning?

• The target policy may differ from the behavior policy
• The target policy must copy every exploratory behavior action
• The behavior policy must avoid collecting exploratory data

For a nonterminal transition from s to s', with reward r and discount gamma, what is the TD(0) target?

• V(s) + gamma * V(s')
• r + gamma * V(s)
• r + gamma * V(s')

Combining off-policy learning, bootstrapping and function approximation can amplify errors and cause divergence.

Use tabular Q-learning with alpha = 0.5 and gamma = 0.5. Initially Q(s,a) = 2. Reward 1 leads to a nonterminal state with next Q values 4 and 10; behavior next chooses the action valued 4. Later, update the same pair from a terminal transition with reward 6. What is Q(s,a) after both updates?

• 4.25
• 4
• 5

In tabular TD(0), V(s) = 2, reward r = 2, gamma = 0.5 and the nonterminal next-state estimate is 10. With alpha = 0.4, what is V(s) after one update? Use all pre-update values.

• 3
• 4
• 6

Does a TD update require the next-state estimate to be more accurate than the current estimate?

• No. It uses that estimate without an accuracy guarantee
• Yes. An independent test must rank the next estimate as better
• Yes. The next estimate must equal the true value before use

For the standard convergence result in finite tabular discounted Q-learning, which exploration requirement is needed?

• Every state-action pair is sampled infinitely often
• Only currently highest-valued actions are sampled infinitely often
• Each state is visited once, then exploration is permanently disabled

An off-policy learner with shared weights produces growing value estimates. Which conclusion is justified from that observation alone?

• The observation proves that every reward supplied by the environment is incorrect
• Instability is possible, but the observation does not identify its cause
• The observation proves that the next-state estimates are already more accurate