Researcher Degrees of Freedom and the Pre-registered Analysis Plan
Every undeclared choice among exclusion rules, transformations, subgroups and stopping points widens the range of results a true null can produce. This stop writes an analysis plan that fixes those choices in advance.
What a learner can do afterwards
- List the analysis decisions in a stated study that could have reasonably gone another way
- Explain how looking at the data and deciding whether to continue raises the false positive rate
- Write a short pre-registration covering hypothesis, exclusion rules and the primary analysis
1 · Read
You map the garden of forking paths before you enter it. Outlier rules, covariates, outcomes, and stopping points each look innocent alone, yet together they offer many routes. Trying them all and reporting the prettiest is not discovery.
You learn why peeking bends error rates. Each interim look is another throw at significance, and stopping on a lucky peak freezes noise at its best moment. Extreme early results regress toward the mean on repeat, so fixed samples or honest error spending keep the rate where promised.
You run a small noisy study and only the exaggerated estimate clears the bar. You feel tempted to publish it at face value. Then you remember the filter: accurate estimates fell short and vanished, while the lucky overestimate sailed through.
You write a short pre-registration naming the hypothesis, the sample size, the exclusion rules, and the exact primary analysis. You share data and code so others can retrace your path. Exploration stays free, but you label it honestly as exploration apart from the confirmatory test.
You fence the garden beforehand, you fix the stopping rule, and you label exploration as exploration.
2 · Watch
Take it off screen
Where it sits
Where this leads
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.