The Memory Hierarchy and Why Caching Works
Fast memory is small and small memory is fast, so machines stack registers, caches, main memory and disk. The stack pays off only because programs reuse what they touched recently and touch neighbours next, which is what locality means.
What a learner can do afterwards
- Order the levels of the hierarchy by size and by access time
- Identify temporal and spatial locality in a piece of loop code
- Predict which of two loops over the same array runs faster and say why
1 · Read
The instruction set is a promise about what a program can see, not about how memory is built underneath it. Machines use that freedom to stack layers: small fast registers, then caches, then main memory, then disk, each step bigger and slower.
The stack pays off because programs show locality. Caches keep copies of what you touched recently and the neighbours beside it. Reusing recent data is temporal locality. Touching neighbours next is spatial locality.
Walking a two-dimensional array row by row visits neighbours in order, so each cache line is fully used. Walking the same data column by column jumps across rows, wastes each line, and can run several times slower.
To predict the faster loop, check the visit order against the layout. The loop that walks neighbours in order reuses each cache line and wins.
Small fast layers sit above big slow ones, and walking neighbours in order keeps every cache line paying off.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.