Markov Decision Processes: States, Actions and Return
Reinforcement learning begins by writing the problem down: states, actions, a transition rule, a reward and a discount. The return is the quantity being maximised, and the discount is the modelling choice that fixes how far ahead the agent looks.
What a learner can do afterwards
- Write a small described task as an MDP and say what each part holds
- Compute a discounted return along a short trajectory
- Say what the Markov assumption rules out and give a task where it is wrong
1 · Read
Learning from experience comes in three flavors. Supervised learning studies examples with correct answers attached, like labeled photos. Unsupervised learning groups unlabeled data on its own. Reinforcement learning takes actions in an environment for rewards or penalties, and learns to pick the actions that earn most.
Before any agent learns, write the problem as states, actions, a transition rule, a reward, and a discount. The return is the quantity being maximized: rewards added up with later ones shrunk by the discount. The discount is the modeling choice that fixes how far ahead the agent looks.
Take a tiny weather world with Sunny and Rainy. Today is Sunny for sure, written as the vector [1, 0], and Sunny stays Sunny with chance 0.8, so tomorrow is Sunny with chance 0.8. For returns, try rewards 4 then 2 with discount 0.5: the return is 4 plus half of 2, which is 5.
The Markov assumption says the current state holds everything that matters: what happens next depends only on the current state and action. A rule like rain only after two rainy days in a row breaks it, since today alone cannot decide. The fix is to enlarge the state until it remembers enough.
States, actions, transitions, reward, discount: write them down, maximize the return, and keep the state memoryless.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.