---
title: "Attention as a Learned Lookup"
description: "Attention scores a query against every key in the sequence, turns the scores into weights, and returns a blend of the matching values. Nothing about distance is fixed in advance, so a position can rea"
canonical: https://lightmysky.com/learn/computing/attention-as-a-learned-lookup-mt_rrdtS1lHvC
source: https://lightmysky.com/learn/computing/attention-as-a-learned-lookup-mt_rrdtS1lHvC.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Attention as a Learned Lookup

Attention scores a query against every key in the sequence, turns the scores into weights, and returns a blend of the matching values. Nothing about distance is fixed in advance, so a position can read from anywhere, and the price is a comparison against everything.

Subject: Computing · Area: Machine Learning · Ages 22 to 23
Page: https://lightmysky.com/learn/computing/attention-as-a-learned-lookup-mt_rrdtS1lHvC

## Ready when they can

- Compute the weights of one attention head by hand and say what the softmax did to the raw scores
- Say what a fixed convolution window and a recurrent state each lose that attention keeps
- Give the cost of attention in the sequence length and name what that cost rules out

## Lesson: Attention reads anywhere

Attention scores your query against every key in the sequence, turns the scores into weights, and returns a blend of the matching values. The softmax turns raw scores into positive weights that sum to one, and it boosts the gaps so the top score dominates the blend.

**Example.** A head scores three positions 3, 1 and 0. After softmax the first position holds most of the weight, so the output is nearly its value. Read the head by finding the top scorer: that is the position it read from.

Fixed windows and recurrent states both lose what attention keeps. A convolution window sees only neighbours, and a recurrent state squeezes the whole past into one vector. Attention can read from anywhere, and the price is comparing everything with everything, so cost grows with the square of the length.

**Tip.** A sequence of length 4 needs 16 comparisons. That square law rules out very long sequences, which is why long inputs need sparser schemes.

**Recap.** Attention blends values by learned weights, reads from anywhere, and costs the square of the length.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Backpropagation: Assigning Blame for an Error](https://lightmysky.com/learn/computing/backpropagation-assigning-blame-for-an-error-mt_5bwaSWfF96)
- [Deep Learning: Convolution, Sequence and Scale](https://lightmysky.com/learn/computing/deep-learning-convolution-sequence-and-scale-mt_karzGN5JnI)
- [Principal Components and Dimensionality Reduction](https://lightmysky.com/learn/computing/principal-components-and-dimensionality-reduction-mt_l-9XHg_ChZ)

## Opens up

- [The Transformer Block: Heads, Residuals and Normalisation](https://lightmysky.com/learn/computing/the-transformer-block-heads-residuals-and-normalisation-mt_eKC2IehaDT)