Errors, Power and the Design of a Test

A Type I error rejects a true H0. A valid level-α test has probability at most α of doing this when H0 is true; its actual rate can be smaller. A Type II error fails to reject a false H0. Its probability β must be stated for a specified true parameter in the alternative. Failing to reject is a test decision, not acceptance of H0 as true.

Power is the probability of rejecting H0 for a specified true parameter in the alternative. Rejection and failure to reject exhaust the possible decisions, so power = 1 - β. For β = 0.35, power is 0.65. Power describes repeated use of the procedure under those conditions; it does not guarantee rejection in one study.

For a one-sided normal-mean test, fix α, variance and sampling method. More independent observations raise power at a fixed mean in H1. With n fixed, moving farther into H1 raises power: below μ0 for a lower tail. An opposite-direction departure need not help. Planned power uses assumptions; actual power depends on true conditions and implementation.

For a fixed model and statistic, raising α enlarges a nested rejection region. The allowed Type I error rate rises and power cannot decrease; lowering α reverses this trade-off. If α and the target effect must stay fixed, increasing independent sample size can raise power. Plan before collecting new data and recalculate power under the model. Subtract old planned power from new planned power to quantify a gain.

Nonsignificance means no rejection at the planned level, not proof of zero effect. A precise interval can exclude practically important effects against a stated threshold. A p-value alone does not establish low power. Review estimates, uncertainty, target effects and design assumptions before planning another study.