Errors, Power and the Design of a Test
Two ways to be wrong, and a trade-off between them. Power is the chance of detecting a real effect, and it is decided at design time by the sample size.
What a learner can do afterwards
- Describe a type I and a type II error in the context of a study
- Explain how sample size and effect size change power
- Say why a nonsignificant result is not evidence of no effect
1 · Read
Every test ends in one of four cells: the null is true or false, crossed with reject or keep. Rejecting a true null is a Type I error, a false alarm whose chance is the significance level alpha. Keeping a false null is a Type II error, a miss whose chance is beta. A trial declaring a useless drug effective commits a Type I error.
Power is 1 minus the Type II chance: the probability of catching a stated real effect. Bigger samples raise power by shrinking the standard error, and bigger effects raise it by clearing the bar by more. A test with Type II chance 0.2 has power 0.8. Power is decided at design time, before data are collected.
A nonsignificant result never proves no effect; it only means the data cannot rule the null out, and small effects may still hide. That is why a study with 20 subjects may miss a real effect. Fix it by choosing the effect worth catching first, then sizing the sample to catch it with good power.
Name each mistake with the grid, design for power up front, and never read a miss as proof of nothing.
2 · Watch
Take it off screen
Where it sits
Learn first
This opens up
Where this leads
Jobs that lean on this skill. Follow one to see everything it is built on.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.