Researcher Degrees of Freedom and the Pre-registered Analysis Plan · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Locking your analysis plan in advance

Science · Scientific Inquiry · ages 22-24
Name ______________________   Date ____________

  1. Why is trying many analyses and reporting the prettiest one wrong?

    • Because computers dislike extra work
    • Because some path crosses significance by luck alone
    • Because only one analysis can ever run

  2. What does a pre-registration fix in advance?

    • The hypothesis, exclusion rules, and primary analysis
    • The journal, the reviewers, and the publication date
    • The font, the margins, and the file name

  3. Pre-registration bans all exploratory analysis.

    Circle one:   True   False

  4. Which study choices belong in the pre-registration?

    • Outlier handling, covariates, outcome, and stopping rule
    • Office seating, lunch orders, and holidays
    • Citation styles and figure colors

  5. You peek at the data and stop on a lucky peak. What goes wrong?

    • Nothing; diligent peeking always helps
    • The sample size grows automatically
    • Nonexistent effects clear the bar far more often than promised

  6. A field publishes mostly striking positives from flexible analyses. What do you expect?

    • A literature full of breakthroughs that all replicate
    • Mostly circulating noise shaped by selection and hidden flexibility
    • The end of all statistical error

  7. You must peek for safety during a trial. What keeps the error rate honest?

    • Unlimited unplanned looks with quiet stopping
    • Fixed samples, or rules that spend error honestly across looks
    • Stopping the moment any subgroup looks good

  8. A colleague forms the hypothesis after seeing the results but writes it as planned. What is this?

    • Proper confirmatory testing
    • A valid stopping rule
    • Hypothesizing after results are known, which mislabels exploration
LightMySky · lightmysky.comW1-mt_UJgZQ2AADm-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Locking your analysis plan in advance W1-mt_UJgZQ2AADm-s1

  1. Because some path crosses significance by luck alone · With enough forks, chance hands you a win worth nothing.
  2. The hypothesis, exclusion rules, and primary analysis · Fencing the garden beforehand removes later temptations to wander.
  3. False · It labels exploration honestly, separate from the confirmatory test.
  4. Outlier handling, covariates, outcome, and stopping rule · Each fork can tip results, so each must be declared.
  5. Nonexistent effects clear the bar far more often than promised · Stopping on noise freezes it at its best moment.
  6. Mostly circulating noise shaped by selection and hidden flexibility · Flexible analyses feeding selective journals inflate the record mechanically.
  7. Fixed samples, or rules that spend error honestly across looks · Planned looks with budgeted error keep the false positive rate promised.
  8. Hypothesizing after results are known, which mislabels exploration · The separation between confirmatory claims and digging must stay honest.
Worksheet · LightMySky