Attention as a Learned Lookup · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Attention reads anywhere

Computing · Machine Learning · ages 22-23
Name ______________________   Date ____________

  1. A head returns a blend of values. What decides the blend?

    • The learned weights from query-key scores
    • The alphabetical order of tokens
    • The font size of the input

  2. What does the softmax do to the raw scores?

    • It sorts them alphabetically
    • It turns them into positive weights summing to one
    • It deletes the smallest score

  3. A recurrent state keeps direct access to every past position.

    Circle one:   True   False

  4. What does a fixed convolution window lose that attention keeps?

    • The ability to read distant positions directly
    • The ability to run fast
    • The ability to learn weights

  5. Scores are 3, 1 and 0. Which position did the head read from?

    • The one scoring 0
    • The one scoring 1
    • The one scoring 3

  6. A sequence has length 6. How many query-key comparisons does full attention need?

    Answer: ______________

  7. Two heads show flat weights while one shows a sharp peak. Which head is selective?

    • A flat one, it reads evenly
    • The peaked one, it reads from its top scorer
    • None, sharpness means failure

  8. A team wants full attention over million-token inputs. What rules it out?

    • The square-law comparison cost
    • The lack of training data
    • The spelling of the tokens
LightMySky · lightmysky.comW1-mt_rrdtS1lHvC-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Attention reads anywhere W1-mt_rrdtS1lHvC-s1

  1. The learned weights from query-key scores · Weights come from scoring the query against every key.
  2. It turns them into positive weights summing to one · Softmax normalises scores into a proper weighting.
  3. False · Recurrence squeezes the past into one vector, which attention avoids.
  4. The ability to read distant positions directly · Windows see neighbours only, while attention reaches anywhere.
  5. The one scoring 3 · The top scorer dominates after softmax.
  6. 36 · Every position meets every other: 6 times 6 is 36.
  7. The peaked one, it reads from its top scorer · A sharp peak means one position dominates the blend.
  8. The square-law comparison cost · Quadratic cost explodes long before a million positions.
Worksheet · LightMySky