Interpretability: Probes, Features and Circuits · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Ask what the network really uses

Computing · Machine Learning · ages 23-24
Name ______________________   Date ____________

  1. A probe succeeds. What has been proven?

    • The model uses the property when answering
    • The property is readable at that layer
    • The layer holds nothing else of interest

  2. What does a probe train a small classifier to do?

    • Read a property from a layer's activations
    • Rewrite the model's weights
    • Label brand-new training data

  3. What does feature analysis ask about a direction?

    • How fast it computes
    • How many parameters it owns
    • What human idea it stands for

  4. Which intervention tests a claim about a circuit?

    • Retrain the whole model from scratch
    • Silence the part and check the behaviour breaks, then restore it
    • Add extra layers around the part

  5. Why does a single unit rarely match one human concept?

    • Units usually fire for many unrelated inputs
    • Units never activate while answering
    • Concepts live only in the training data

  6. A claim comes with no edit that could prove it wrong. How should you treat it?

    • As a guess, not a finding
    • As proven by its elegance
    • As true until the model ships

  7. A property a probe can read is always one the model uses.

    Circle one:   True   False

  8. Silencing a suspected circuit changes nothing, yet the team still credits it. What is wrong?

    • The test needs a larger probe instead
    • Correlation alone settles circuit claims
    • The failing edit rejects the claim, so credit must go
LightMySky · lightmysky.comW1-mt_C8POKvRkq6-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Ask what the network really uses W1-mt_C8POKvRkq6-s1

  1. The property is readable at that layer · Readability is shown; use still needs an intervention.
  2. Read a property from a layer's activations · The probe only tests whether the property is readable there.
  3. What human idea it stands for · It maps activation directions to candidate meanings.
  4. Silence the part and check the behaviour breaks, then restore it · Breaking and restoring ties the part to the behaviour.
  5. Units usually fire for many unrelated inputs · Mixed firing spreads meaning across directions, not single neurons.
  6. As a guess, not a finding · Without a failing edit, the claim cannot be tested.
  7. False · Readable and used are different claims about a layer.
  8. The failing edit rejects the claim, so credit must go · A circuit that can be removed for free carries no such behaviour.
Worksheet · LightMySky