Markov Decision Processes: States, Actions and Return · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Writing a choice problem down

Computing · Machine Learning · ages 22-24
Name ______________________   Date ____________

  1. Which learning style takes actions for rewards instead of studying labeled examples?

    • Supervised learning
    • Unsupervised learning
    • Reinforcement learning

  2. Under the Markov assumption, the next state can depend on the whole history of past states.

    Circle one:   True   False

  3. What is the return in a decision problem?

    • The discounted sum of rewards being maximized
    • The list of all states visited
    • The count of actions available

  4. In a Markov model, the next state can depend on the whole history of where the agent has been, not just the current state. True or false?

    Circle one:   True   False

  5. An agent earns reward 4 now and reward 2 next step, with discount 0.5. What is the discounted return?

    Answer: ______________

  6. What does a small discount factor assume about the future?

    • Later rewards matter little; the agent looks a short way ahead
    • Later rewards matter most; the agent plans far ahead
    • Rewards stop arriving after one step

  7. A tiny robot world has two states: Sunny and Rainy. From Sunny, the chance of staying Sunny tomorrow is 0.8. From Rainy, the chance of moving to Sunny tomorrow is 0.5. Today it is Sunny for sure, so the state vector is [1, 0]. What is the probability that tomorrow is Sunny?

    • 0.5
    • 0.8
    • 1.3
    • 0.3

  8. Today is Sunny for sure, so the vector is [1, 0]. Sunny stays Sunny with chance 0.8. What is the chance tomorrow is Sunny?

    • 0.5
    • 0.3
    • 0.8

  9. How do you repair that two-rainy-days task so the assumption holds again?

    • Enlarge the state until it remembers enough, such as pairs of days
    • Lower all transition chances below 0.5
    • Remove the reward and maximize steps instead

  10. Rain falls only after two rainy days in a row. Why can Sunny and Rainy alone not model this?

    • Two states are too few for any weather task
    • Today alone cannot decide; yesterday must be remembered too
    • Transition chances above 0.5 are forbidden
LightMySky · lightmysky.comW1-mt_bTCO27sgc6-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Writing a choice problem down W1-mt_bTCO27sgc6-s1

  1. Reinforcement learning · Labels teach the first, groups form the second, rewards steer the third.
  2. False · Only the current state and action may shape what comes next.
  3. The discounted sum of rewards being maximized · Later rewards shrink by the discount, then all add into one quantity.
  4. False · The Markov assumption says the current state holds everything that matters. What happens next depends only on the current state and action, not on the path taken to get there.
  5. 5 · Four plus half of two is 5.
  6. Later rewards matter little; the agent looks a short way ahead · Heavy shrinking fades the future, so near rewards dominate.
  7. 0.8 · The state vector says today is Sunny with weight 1, so tomorrow's Sunny chance is just the Sunny-to-Sunny transition probability, 0.8.
  8. 0.8 · All weight sits on Sunny, so its outgoing chance carries over directly.
  9. Enlarge the state until it remembers enough, such as pairs of days · A richer state turns history into current fact, restoring memorylessness.
  10. Today alone cannot decide; yesterday must be remembered too · The rule needs history outside the current state, which the assumption forbids.
Worksheet · LightMySky