--- title: "Researcher Degrees of Freedom and the Pre-registered Analysis Plan" description: "Data-dependent choices about exclusions, transformations, outcomes and stopping can invalidate an unadjusted confirmatory test. Specify the primary analysis and stopping procedure in advance, document" canonical: https://lightmysky.com/learn/science/researcher-degrees-of-freedom-and-the-pre-registered-analysis-plan-mt_UJgZQ2AADm source: https://lightmysky.com/learn/science/researcher-degrees-of-freedom-and-the-pre-registered-analysis-plan-mt_UJgZQ2AADm.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Researcher Degrees of Freedom and the Pre-registered Analysis Plan Data-dependent choices about exclusions, transformations, outcomes and stopping can invalidate an unadjusted confirmatory test. Specify the primary analysis and stopping procedure in advance, document deviations, and identify exploratory analyses. Subject: Science · Area: Scientific Inquiry · Ages 22 to 24 Page: https://lightmysky.com/learn/science/researcher-degrees-of-freedom-and-the-pre-registered-analysis-plan-mt_UJgZQ2AADm ## Ready when they can - List the analysis decisions in a stated study that could have reasonably gone another way - Explain how repeated unadjusted efficacy tests with significance-based stopping can raise the false positive rate - Write a short pre-registration covering hypothesis, exclusion rules and the primary analysis ## Lesson: Locking your analysis plan in advance Researcher degrees of freedom are choices that could reasonably differ: exclusions, transformations, covariates, outcomes, subgroups and stopping rules. These choices can be useful. The risk arises when results guide selection and the chosen analysis is reported as if fixed beforehand. List the decisions before collecting data. Trying several analyses and reporting only a significant one can raise false positives above the advertised level. The analyses may be dependent, so their probabilities cannot usually be added. Pre-registration records intended choices; it does not prevent all bias or make an unsuitable analysis valid. *(drawing: Use the previous topic's power assumptions when stating the sampling rationale.)* Repeated ordinary fixed-sample efficacy tests, with stopping at significance, can raise false positives. A fixed-sample plan avoids this route only when interim efficacy results do not determine stopping. Safety monitoring must remain possible. For interim efficacy decisions, pre-specify a sequential method with calibrated boundaries or error spending alongside the safety procedure. **Example.** Example culture plan: assign 40 cultures per group, a budget cap rather than a power claim. Compare day-5 growth using a two-sided Welch t-test at alpha 0.05, with no covariates. Exclude cultures documented as contaminated before dosing. Analyse available outcomes and report missing counts by group. Stop after 80 assignments. Record this plan before collection. The example specifies an analysis but does not establish adequate power or remove bias from missing outcomes. Report power sensitivity and assess assumptions. Label hypotheses developed after inspecting results as exploratory. Report deviations, their timing and reasons. Independent new data can provide a confirmatory test of a later hypothesis. *(drawing: Changing a plan requires a record, not a claim that the change was always intended.)* Selective reporting changes what readers see. Unpublished nonsignificant findings create a file drawer problem. Selecting significant low-power findings can also favor unusually large magnitudes. Neither process proves every published finding false. Report planned outcomes and analyses, including nonsignificant results, so readers can assess more of the evidence. **Recap.** Record analysis and stopping decisions before collection. Preserve safety monitoring, explain deviations, and label exploration separately from planned tests. *Original lesson, questions and flashcards about analysis planning and reporting, written for this learning dataset.* ## Practice 8 questions on this page, each with its working shown. ## Needs first - [Power and Effect Size, Decided Before the Data](https://lightmysky.com/learn/science/power-and-effect-size-decided-before-the-data-mt_zYNmSWAJT8) ## Opens up - [Reading a Primary Paper: The Claim, the Figure and the Missing Control](https://lightmysky.com/learn/science/reading-a-primary-paper-the-claim-the-figure-and-the-missing-control-mt_DZE7z5vu9X)