Attention as a Learned Lookup
Attention scores a query against every key in the sequence, turns the scores into weights, and returns a blend of the matching values. Nothing about distance is fixed in advance, so a position can read from anywhere, and the price is a comparison against everything.
What a learner can do afterwards
- Compute the weights of one attention head by hand and say what the softmax did to the raw scores
- Say what a fixed convolution window and a recurrent state each lose that attention keeps
- Give the cost of attention in the sequence length and name what that cost rules out
1 · Read
Attention scores your query against every key in the sequence, turns the scores into weights, and returns a blend of the matching values. The softmax turns raw scores into positive weights that sum to one, and it boosts the gaps so the top score dominates the blend.
A head scores three positions 3, 1 and 0. After softmax the first position holds most of the weight, so the output is nearly its value. Read the head by finding the top scorer: that is the position it read from.
Fixed windows and recurrent states both lose what attention keeps. A convolution window sees only neighbours, and a recurrent state squeezes the whole past into one vector. Attention can read from anywhere, and the price is comparing everything with everything, so cost grows with the square of the length.
A sequence of length 4 needs 16 comparisons. That square law rules out very long sequences, which is why long inputs need sparser schemes.
Attention blends values by learned weights, reads from anywhere, and costs the square of the length.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.