Attention as a Learned Lookup
Attention scores a query against every key in the sequence, turns the scores into weights, and returns a blend of the matching values. Nothing about distance is fixed in advance, so a position can read from anywhere, and the price is a comparison against everything.
What a learner can do afterwards
- Compute the weights of one attention head by hand and say what the softmax did to the raw scores
- Say what a fixed convolution window and a recurrent state each lose that attention keeps
- Give the cost of attention in the sequence length and name what that cost rules out
1 · Read
Every position in the sequence makes three vectors from its own input: a query, a key and a value. Attention scores one position's query against every key in the sequence, turns the scores into weights, and returns a blend of the matching values. The softmax turns raw scores into positive weights that sum to one and stretches the gaps, so a score far above every other takes most of the weight, while near-equal scores share it almost evenly.
A head scores three positions 3, 1 and 0. Each weight is e raised to the score, divided by the total: e³ is about 20.1, e¹ is about 2.7 and e⁰ is 1, so the weights are about 0.84, 0.11 and 0.04. The output is 0.84 of the first value plus 0.11 of the second plus 0.04 of the third, so it is mostly the first value but never only that. Scores of 0.01, 0 and 0 would give weights near 0.34, 0.33 and 0.33, an almost even blend. Read a head by finding its top scorer: that is the position it read from most.
Weights for scores 3, 1 and 0, rounded to about 84, 11 and 4 percent, which is why they add to 99 rather than 100. The gaps are stretched, but every value still counts.
Fixed windows and recurrent states both lose what attention keeps. A convolution window sees only neighbours, and a recurrent state squeezes the whole past into one vector. Attention can read from anywhere, and the price is comparing everything with everything, so cost grows with the square of the length.
A sequence of length 4 needs 16 comparisons. That square law rules out very long sequences, which is why long inputs need sparser schemes.
Attention blends values by learned weights, reads from anywhere, and costs the square of the length.
2 · Watch
به انگلیسیTake it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.