One test is easy, twenty thousand are not

Science · Genetics & Evolution · ages 23-24

Name ______________________ Date ____________

Apply Bonferroni to 20,000 tests at family-wise level 0.05. Using p <= threshold for rejection, which rule is correct?
- p <= 0.05 for each test
- p <= 0.0000025
- p <= 0.5 for each test
All 20,000 tested gene null hypotheses are true. Each test has exact Type I error probability 0.05. How many false positives are expected?
- About 1,000 false positives
- About 5 false positives
- About 20,000 false positives
A valid FDR procedure at 0.05 bounds the expected false discovery proportion by 5%, defining the proportion as zero when there are no discoveries.
Circle one: True False
Four independent valid tests have sorted p-values 0.004, 0.024, 0.080 and 0.400. At level 0.05, using <=, how many hits do BH and Bonferroni report?
- One BH hit and two Bonferroni hits
- Four BH hits and one Bonferroni hit
- Two BH hits and one Bonferroni hit
With valid p-values, what does Bonferroni family-wise control at 0.05 bound?
- The probability of at least one false positive in the tested family
- The probability that each reported gene is null, given its individual p-value
- The observed false fraction to exactly 5% in every reported discovery list
Of 10,000 tested gene hypotheses, 2,000 nulls are true. Each true-null test has exact Type I error probability 0.05. What is the expected number of false positives?
Answer: ______________
A researcher interprets a Storey-style q-value of 0.05 as a 5% probability that this gene's null is true. What is the error?
- A q-value instead fixes the observed false fraction at exactly 5% for this particular discovery list
- It estimates a list-level pFDR threshold, not this gene's null probability
- A q-value instead guarantees that one particular gene out of every twenty discoveries is null
Which statement about filtering and covariate weighting correctly describes their statistical limits?
- Both must reduce test count, and selecting their settings from the smallest observed p-values is always valid
- Both guarantee more discoveries, even when their inputs reuse the same null p-value signal
- Filtering may remove tests; weighting need not. Both require valid error-control conditions

Answer key

For grown-ups. Fold this page away before handing over the rest.

One test is easy, twenty thousand are not W1-mt_mIQ27rphrC-s1

p <= 0.0000025 · Divide the level by the total test count: 0.05 / 20,000 = 0.0000025. Use the stated inclusive boundary.
About 1,000 false positives · All 20,000 nulls are true, so the expected false-positive count is 20,000 * 0.05 = 1,000. This is an expectation, not a guaranteed count.
True · FDR is the expectation of the false discovery proportion, including zero for studies with no rejections. It is not a bound for every observed list.
Two BH hits and one Bonferroni hit · BH's largest qualifying rank is 2 using thresholds 0.0125, 0.025, 0.0375 and 0.05. Only 0.004 passes Bonferroni's fixed threshold 0.0125.
The probability of at least one false positive in the tested family · Family-wise error is the probability of any false rejection in the family. A bound of 0.05 permits errors; it does not guarantee none.
100 · Only true nulls contribute false positives. Their expected count is 2,000 * 0.05 = 100.
It estimates a list-level pFDR threshold, not this gene's null probability · Under the stated convention, the q-value estimates a pFDR level for cutoffs including the gene. It is not an individual posterior null probability.
Filtering may remove tests; weighting need not. Both require valid error-control conditions · Filtering changes the test family. Weighting can retain all tests. Their validity conditions matter, and neither guarantees more discoveries.