Errors, Power and the Design of a Test · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Two ways to be wrong

Mathematics · Data & Statistics · ages 20-21
Name ______________________   Date ____________

  1. A trial declares a useless drug effective. The null says the drug has no effect. Which error is this?

    • Type I error
    • Type II error
    • Correct decision
    • Power

  2. A trial declares a useless drug effective. The null says the drug has no effect. Which error is this?

    • Type I error
    • Type II error
    • Correct decision

  3. Power is defined as which quantity?

    • The Type I chance
    • One minus the Type II chance
    • The p-value

  4. A Type I error means rejecting the null when it is actually true.

    Circle one:   True   False

  5. A test has P(Type II error) = 0.2 against a stated alternative. What is its power there?

    Answer: ______________

  6. The sample grows while the true effect size stays fixed. What happens to the power of the test?

    • It falls
    • It stays the same
    • It becomes zero
    • It rises

  7. A study reports p equal to 0.3 and the author claims this proves the treatment has no effect. Is the author right?

    Circle one:   True   False

  8. The sample grows while the true effect stays fixed. What happens to the power of the test?

    • It falls
    • It stays the same
    • It rises

  9. A test has chance of a Type II error 0.15 against a stated alternative. What is its power there? Give a decimal.

    Answer: ______________

  10. A study with 20 subjects finds nothing significant. The team wants a real effect of the stated size to be caught next time. What should they do?

    • Enlarge the sample to lift power before rerunning
    • Lower the sample to cut noise
    • Keep the sample and raise alpha to 0.5
LightMySky · lightmysky.comW1-mt_D7ZXbvSD0l-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Two ways to be wrong W1-mt_D7ZXbvSD0l-s1

  1. Type I error · The null says the drug does nothing, and rejecting that true null is the Type I mistake.
  2. Type I error · Rejecting a true null is the false alarm cell of the grid.
  3. One minus the Type II chance · Power is the chance of catching a stated real effect.
  4. True · That is the false alarm cell of the error grid.
  5. 0.8 · Power equals 1 minus the Type II chance: 1 - 0.2 = 0.8.
  6. It rises · Bigger samples shrink the standard error, so a real effect stands out more clearly and power rises.
  7. False · A nonsignificant result only fails to reject; it never proves nothing.
  8. It rises · A bigger sample shrinks the standard error, so the effect clears the bar more easily.
  9. 0.85 · Power is 1 minus the Type II chance.
  10. Enlarge the sample to lift power before rerunning · Power is set at design time, and a bigger sample is the honest fix.
Worksheet · LightMySky