---
title: "Markov Decision Processes: States, Actions and Return"
description: "Reinforcement learning begins by writing the problem down: states, actions, a transition rule, a reward and a discount. The return is the quantity being maximised, and the discount is the modelling ch"
canonical: https://lightmysky.com/learn/computing/markov-decision-processes-states-actions-and-return-mt_bTCO27sgc6
source: https://lightmysky.com/learn/computing/markov-decision-processes-states-actions-and-return-mt_bTCO27sgc6.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Markov Decision Processes: States, Actions and Return

Reinforcement learning begins by writing the problem down: states, actions, a transition rule, a reward and a discount. The return is the quantity being maximised, and the discount is the modelling choice that fixes how far ahead the agent looks.

Subject: Computing · Area: Machine Learning · Ages 22 to 24
Page: https://lightmysky.com/learn/computing/markov-decision-processes-states-actions-and-return-mt_bTCO27sgc6

## Ready when they can

- Write a small described task as an MDP and say what each part holds
- Compute a discounted return along a short trajectory
- Say what the Markov assumption rules out and give a task where it is wrong

## Lesson: Writing a choice problem down

Learning from experience comes in three flavors. Supervised learning studies examples with correct answers attached, like labeled photos. Unsupervised learning groups unlabeled data on its own. Reinforcement learning takes actions in an environment for rewards or penalties, and learns to pick the actions that earn most.

Before any agent learns, write the problem as states, actions, a transition rule, a reward, and a discount. The return is the quantity being maximized: rewards added up with later ones shrunk by the discount. The discount is the modeling choice that fixes how far ahead the agent looks.

**Example.** Take a tiny weather world with Sunny and Rainy. Today is Sunny for sure, written as the vector [1, 0], and Sunny stays Sunny with chance 0.8, so tomorrow is Sunny with chance 0.8. For returns, try rewards 4 then 2 with discount 0.5: the return is 4 plus half of 2, which is 5.

**Tip.** The Markov assumption says the current state holds everything that matters: what happens next depends only on the current state and action. A rule like rain only after two rainy days in a row breaks it, since today alone cannot decide. The fix is to enlarge the state until it remembers enough.

**Recap.** States, actions, transitions, reward, discount: write them down, maximize the return, and keep the state memoryless.

## Practice

11 questions on this page, each with its working shown.

## Needs first

- [Dynamic Programming: Optimal Substructure and Overlapping Subproblems](https://lightmysky.com/learn/computing/dynamic-programming-optimal-substructure-and-overlapping-subproblems-mt__hWkCDnzNY)
- [Markov Chains and the Memoryless Assumption](https://lightmysky.com/learn/mathematics/markov-chains-and-the-memoryless-assumption-mt_2wDa_JxJ_o)

## Opens up

- [Temporal-Difference Learning and Q-Learning](https://lightmysky.com/learn/computing/temporal-difference-learning-and-q-learning-mt_Z78E2llE0C)