Attention as a Learned Lookup · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Attention reads anywhere

Computing · Machine Learning · ages 22-23
Name ______________________   Date ____________

  1. A head returns a blend of values. What decides the blend?

    • The learned weights from query-key scores
    • The position of each token, with nearest first
    • A fixed window around the query

  2. What does the softmax do to the raw scores?

    • It picks the largest score and discards all the others
    • It turns them into positive weights summing to one
    • It rescales them so the largest becomes exactly one

  3. A recurrent state keeps direct access to every past position.

    Circle one:   True   False

  4. What does a fixed convolution window lose that attention keeps?

    • The ability to read distant positions directly
    • The ability to reuse the same weights at every position
    • The ability to see its immediate neighbours

  5. A head scores three positions 2, 0 and 0, and e² is about 7.39. What weight does the top position take after the softmax? Give your answer to two decimal places.

    Answer: ______________

  6. A sequence has length 6. How many query-key comparisons does full attention need?

    Answer: ______________

  7. Two heads show flat weights while one shows a sharp peak. Which head is selective?

    • A flat one, since even weights mean careful reading
    • The peaked one, it reads from its top scorer
    • None, since sharp weights mean the softmax failed

  8. A team wants full attention over million-token inputs. What rules it out?

    • The square-law comparison cost
    • The softmax, which cannot handle a million scores
    • The value vectors, which cannot be blended at that length
LightMySky · lightmysky.comW1-mt_rrdtS1lHvC-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Attention reads anywhere W1-mt_rrdtS1lHvC-s1

  1. The learned weights from query-key scores · Weights come from scoring the query against every key.
  2. It turns them into positive weights summing to one · Softmax normalises scores into a proper weighting.
  3. False · Recurrence squeezes the past into one vector, which attention avoids.
  4. The ability to read distant positions directly · Windows see neighbours only, while attention reaches anywhere.
  5. 0.79 · e² is about 7.39 and e⁰ is 1 for each of the other two, so the total is about 9.39 and the top weight is 7.39 divided by 9.39, about 0.79.
  6. 36 · Every position meets every other: 6 times 6 is 36.
  7. The peaked one, it reads from its top scorer · A sharp peak means one position takes most of the weight in the blend.
  8. The square-law comparison cost · Quadratic cost explodes long before a million positions.
Worksheet · LightMySky