Researcher Degrees of Freedom and the Pre-registered Analysis Plan
Data-dependent choices about exclusions, transformations, outcomes and stopping can invalidate an unadjusted confirmatory test. Specify the primary analysis and stopping procedure in advance, document deviations, and identify exploratory analyses.
What a learner can do afterwards
- List the analysis decisions in a stated study that could have reasonably gone another way
- Explain how repeated unadjusted efficacy tests with significance-based stopping can raise the false positive rate
- Write a short pre-registration covering hypothesis, exclusion rules and the primary analysis
1 · Read
Researcher degrees of freedom are choices that could reasonably differ: exclusions, transformations, covariates, outcomes, subgroups and stopping rules. These choices can be useful. The risk arises when results guide selection and the chosen analysis is reported as if fixed beforehand. List the decisions before collecting data.
Trying several analyses and reporting only a significant one can raise false positives above the advertised level. The analyses may be dependent, so their probabilities cannot usually be added. Pre-registration records intended choices; it does not prevent all bias or make an unsuitable analysis valid.
Repeated ordinary fixed-sample efficacy tests, with stopping at significance, can raise false positives. A fixed-sample plan avoids this route only when interim efficacy results do not determine stopping. Safety monitoring must remain possible. For interim efficacy decisions, pre-specify a sequential method with calibrated boundaries or error spending alongside the safety procedure.
Example culture plan: assign 40 cultures per group, a budget cap rather than a power claim. Compare day-5 growth using a two-sided Welch t-test at alpha 0.05, with no covariates. Exclude cultures documented as contaminated before dosing. Analyse available outcomes and report missing counts by group. Stop after 80 assignments. Record this plan before collection.
The example specifies an analysis but does not establish adequate power or remove bias from missing outcomes. Report power sensitivity and assess assumptions. Label hypotheses developed after inspecting results as exploratory. Report deviations, their timing and reasons. Independent new data can provide a confirmatory test of a later hypothesis.
Selective reporting changes what readers see. Unpublished nonsignificant findings create a file drawer problem. Selecting significant low-power findings can also favor unusually large magnitudes. Neither process proves every published finding false. Report planned outcomes and analyses, including nonsignificant results, so readers can assess more of the evidence.
Record analysis and stopping decisions before collection. Preserve safety monitoring, explain deviations, and label exploration separately from planned tests.
Original lesson, questions and flashcards about analysis planning and reporting, written for this learning dataset.
The five readings and accompanying flashcards were written for this learning dataset.
More to read
List choices before results guide them
An analysis can change when researchers alter exclusions, transformations, covariates, outcomes or subgroups. List these choices before collection. Flexible methods are not inherently invalid. The concern is selecting a favorable result and presenting its analysis as the sole pre-planned test. A dated plan makes intentions visible. It still needs suitable methods and checks of assumptions.
Distinguish monitoring from efficacy stopping
Safety observation must remain possible. The statistical risk discussed here is repeated unadjusted efficacy testing followed by stopping at significance. Ordinary fixed-sample thresholds do not automatically control that procedure's overall false positive rate. If interim efficacy decisions are needed, specify a calibrated sequential method, including its boundaries or error-spending rule. State safety procedures alongside it. Registering only the maximum sample size does not account for earlier efficacy stopping.
Report the attempted comparisons
A selected result does not reveal every comparison attempted. Report planned outcomes and analyses, including nonsignificant ones. Across studies, unpublished nonsignificant findings create a file drawer problem. Significance selection can also favor unusually large magnitudes in low-power settings. These concerns do not establish that every published result is false. Readers need the broader record and uncertainty to assess the evidence.
Keep hypothesis timing visible
A pattern can suggest a useful hypothesis. If the same results were used to suggest and test it, report that sequence. Calling the analysis exploratory describes its timing; it does not make the idea worthless. A later study can test the prediction with independent data and a plan written beforehand. Record amendments and explain when and why the analysis changed.
Make the plan specific enough to check
Name the hypothesis, primary outcome and measurement time. State the independent unit and sample rationale. Specify exclusions, transformations, model, covariates and missing-data handling. Add stopping and safety procedures where applicable. Date the plan before collection and retain amendments. A complete description lets readers compare intention with implementation, but pre-specification does not make biased exclusions or an unsuitable model acceptable.
2 · Watch
Take it off screen
Where it sits
Where this leads
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.