Interpretability: Probes, Features and Circuits
A trained network is a working artifact that nobody designed in detail. Probes ask whether a property is readable from an activation, feature analysis asks what a direction stands for, and circuit analysis asks which components carry a behaviour. Each answers less than it appears to.
What a learner can do afterwards
- Say what a successful probe does and does not prove about how the model works
- Explain why a single unit rarely stands for a single human-named concept
- Describe one intervention that turns a claim about a circuit into a test
1 · Read
A probe is a small classifier trained to read a property, like a topic or a language, from a layer's activations. A probe that succeeds shows the property is readable there, which is useful but weak. It does not prove the model uses that property when it answers, because readable and used are different claims.
Feature analysis asks what one direction in activation space stands for. Single units rarely match single human concepts, because a unit usually fires for many unrelated inputs at once. Meaning tends to live spread across directions, not in one tidy neuron per idea.
Suppose you suspect one small circuit moves names to the right place in a sentence task. To test it, intervene: silence or swap that component and watch whether the behaviour breaks, then restore it and watch the behaviour return. If the behaviour follows your edits both ways, the circuit claim passed a real test.
Turn every interpretability claim into a bet about an edit. Ask which intervention would prove you wrong, and run it before you believe the story. A claim with no failing edit is a guess, not a finding.
Readable is not used, units are not concepts, and only an intervention can promote a circuit story into a finding.
2 · Watch
Take it off screen
Where it sits
Learn first
This opens up
Nothing builds on it yet.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.