---
title: "The Memory Hierarchy and Why Caching Works"
description: "Fast memory is small and small memory is fast, so machines stack registers, caches, main memory and disk. The stack pays off only because programs reuse what they touched recently and touch neighbours"
canonical: https://lightmysky.com/learn/computing/the-memory-hierarchy-and-why-caching-works-mt_0f5ArFPlxP
source: https://lightmysky.com/learn/computing/the-memory-hierarchy-and-why-caching-works-mt_0f5ArFPlxP.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# The Memory Hierarchy and Why Caching Works

Fast memory is small and small memory is fast, so machines stack registers, caches, main memory and disk. The stack pays off only because programs reuse what they touched recently and touch neighbours next, which is what locality means.

Subject: Computing · Area: Computer Systems · Ages 18 to 19
Page: https://lightmysky.com/learn/computing/the-memory-hierarchy-and-why-caching-works-mt_0f5ArFPlxP

## Ready when they can

- Order the levels of the hierarchy by size and by access time
- Identify temporal and spatial locality in a piece of loop code
- Predict which of two loops over the same array runs faster and say why

## Lesson: Small fast memory, big slow memory

The instruction set is a promise about what a program can see, not about how memory is built underneath it. Machines use that freedom to stack layers: small fast registers, then caches, then main memory, then disk, each step bigger and slower.

The stack pays off because programs show locality. Caches keep copies of what you touched recently and the neighbours beside it. Reusing recent data is temporal locality. Touching neighbours next is spatial locality.

**Example.** Walking a two-dimensional array row by row visits neighbours in order, so each cache line is fully used. Walking the same data column by column jumps across rows, wastes each line, and can run several times slower.

To predict the faster loop, check the visit order against the layout. The loop that walks neighbours in order reuses each cache line and wins.

**Recap.** Small fast layers sit above big slow ones, and walking neighbours in order keeps every cache line paying off.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Memory and Storage: RAM, ROM and the Backing Store](https://lightmysky.com/learn/computing/memory-and-storage-ram-rom-and-the-backing-store-mt_AiXGJioO_S)
- [Instruction Set Architecture as a Contract](https://lightmysky.com/learn/computing/instruction-set-architecture-as-a-contract-mt_pbwHRePY5M)

## Opens up

- [Indexes: B-Trees and the Cost of a Lookup](https://lightmysky.com/learn/computing/indexes-b-trees-and-the-cost-of-a-lookup-mt_1Z5EQTqooI)
- [Pipelining and the Cost of a Hazard](https://lightmysky.com/learn/computing/pipelining-and-the-cost-of-a-hazard-mt_GI_fPY9d6s)
- [Paging, Page Faults and Replacement Policies](https://lightmysky.com/learn/computing/paging-page-faults-and-replacement-policies-mt_nEEuGL89Ze)
- [Processes, Context Switches and What They Cost](https://lightmysky.com/learn/computing/processes-context-switches-and-what-they-cost-mt_oiOwHKUoXG)