---
title: "Interpretability: Probes, Features and Circuits"
description: "A trained network is a working artifact that nobody designed in detail. Probes ask whether a property is readable from an activation, feature analysis asks what a direction stands for, and circuit ana"
canonical: https://lightmysky.com/learn/computing/interpretability-probes-features-and-circuits-mt_C8POKvRkq6
source: https://lightmysky.com/learn/computing/interpretability-probes-features-and-circuits-mt_C8POKvRkq6.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Interpretability: Probes, Features and Circuits

A trained network is a working artifact that nobody designed in detail. Probes ask whether a property is readable from an activation, feature analysis asks what a direction stands for, and circuit analysis asks which components carry a behaviour. Each answers less than it appears to.

Subject: Computing · Area: Machine Learning · Ages 23 to 24
Page: https://lightmysky.com/learn/computing/interpretability-probes-features-and-circuits-mt_C8POKvRkq6

## Ready when they can

- Say what a successful probe does and does not prove about how the model works
- Explain why a single unit rarely stands for a single human-named concept
- Describe one intervention that turns a claim about a circuit into a test

## Lesson: Ask what the network really uses

A probe is a small classifier trained to read a property, like a topic or a language, from a layer's activations. A probe that succeeds shows the property is readable there, which is useful but weak. It does not prove the model uses that property when it answers, because readable and used are different claims.

Feature analysis asks what one direction in activation space stands for. Single units rarely match single human concepts, because a unit usually fires for many unrelated inputs at once. Meaning tends to live spread across directions, not in one tidy neuron per idea.

**Example.** Suppose you suspect one small circuit moves names to the right place in a sentence task. To test it, intervene: silence or swap that component and watch whether the behaviour breaks, then restore it and watch the behaviour return. If the behaviour follows your edits both ways, the circuit claim passed a real test.

**Tip.** Turn every interpretability claim into a bet about an edit. Ask which intervention would prove you wrong, and run it before you believe the story. A claim with no failing edit is a guess, not a finding.

**Recap.** Readable is not used, units are not concepts, and only an intervention can promote a circuit story into a finding.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Serving a Model: Batching, the Key-Value Cache and Quantisation](https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4)
- [Fairness, Accountability and the Limits of a Model](https://lightmysky.com/learn/computing/fairness-accountability-and-the-limits-of-a-model-mt_tTh4WtAyk5)