Multiple Comparisons

Twenty hypotheses, each tested at 5%. Run them one at a time and watch where each p-value lands, then change the rule that decides which of them count as findings. Start with pure noise, then make some of the effects real.

Worked example — Multiple Comparisons

Average number of false findings, no correction	1.00
Bonferroni's level for each test	0.0025
Chance of at least one false finding, Bonferroni	4.88%

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Lesson

The theory — Multiple Comparisons

A family of tests is every hypothesis test run on one question or one data set, and its family-wise error rate is the chance that it rejects at least one true null hypothesis: 1 − (1 − α)^m₀ when m₀ independent true nulls are each tested at α. Bonferroni tests each of the m hypotheses at α/m. Holm sorts the p-values and tests the i-th smallest at α/(m − i + 1), stopping at the first that fails. Both keep the family-wise rate at or below α. Benjamini–Hochberg rejects the k smallest p-values, where k is the largest i with p_(i) ≤ iα/m, and holds the false discovery rate, the average share of rejections that are false, at or below (m₀/m)α for independent tests.

What each symbol means

m: the number of tests in the family
m₀: how many of them are true nulls; m − m₀ are real effects
α: the level each test is held to on its own, 5% here
p₍ᵢ₎: the i-th smallest p-value in the family
δ: the size of a real effect, in standard errors of its test statistic

Assumes: The tests are independent two-sided z-tests; a true null's p-value is uniform between 0 and 1; every real effect has the same size; and the family, its size and its rule are fixed before any data are seen.

Fourteen tests of nothing, and a false finding is more likely than not 🖖

Press Fourteen tests of nothing. Each test keeps its own promise: a true null passes 5% one time in twenty. The family makes no such promise. The chance that none of fourteen passes is 0.95¹⁴, so the chance that at least one does is 51.23%, and at twenty tests it is 64.15%. Thirteen give 48.67%, which makes fourteen the first family where a false finding is the likelier outcome. At twenty the results table puts the average number of false findings at 20 × 0.05 = 1, exactly, though 35.85% of such families find nothing at all. Randall Munroe drew the whole thing in xkcd strip 882, Significant: scientists test twenty colours of jelly bean against acne, green comes in under 0.05, and the newspaper prints "only 5% chance of coincidence!" For the green test alone, 5% is the rate at which a true null passes. The headline left out the other nineteen.

On pure noise the three corrections agree; with real effects they keep different promises 🖖

Press Twenty tests of nothing and read the table. Bonferroni lets a false finding through in 4.88% of families, Holm in the same 4.88%, Benjamini–Hochberg in 5.00%. When every hypothesis is a true null, any finding is a false one, so all three are guarding the same thing. Now press Five real effects in twenty. Bonferroni and Holm still keep the chance of even one false finding under 5%, at 3.69% and 4.13%, and they pay for it: each finds about two of the five real effects on average. Benjamini–Hochberg lets a false finding through in 13.99% of families and finds 2.81. What it holds down instead is the share of its findings that are false: on average at most (m₀/m) × 5%, here 15/20 × 5% = 3.75%. Yoav Benjamini and Yosef Hochberg proved that bound for independent tests in 1995, and the page's exact calculation sits on it. Move the power slider from 20% to 99% and the 3.75% stays put, while the chance of at least one false finding climbs from 4.63% to 19.84%. If a single false claim is costly, control the chance of any. If you are screening many candidates for a second, confirmatory look, the case Benjamini and Hochberg give with chemicals screened for drug development, a known small share of false leads is the price of finding more of the real ones.

Benjamini–Hochberg gets more generous the more there is to find 🖖

Bonferroni's bar is 0.05/m whatever the data say, so at twenty tests and 80% power each real effect is found 41.23% of the time, whether it is the only real one or one of fifteen. Benjamini–Hochberg holds the i-th smallest p-value to i × 0.05/m, so each strong result ahead of a test pushes it one rank further down the list, where its threshold is larger and easier to pass. With one real effect among twenty it is found 42.06% of the time, hardly better than Bonferroni; with five, 56.24% each; with fifteen, 72.68%, where Holm has crept from 41.25% to 46.07%. Press Fifteen real effects in twenty and choose Sorted by p to see why. Holm's staircase and the Benjamini–Hochberg line both start at 0.0025 for the smallest p-value and both end at 0.05 for the largest, but Holm's stays near the floor until the last few ranks, at 0.05/11 = 0.0045 for the tenth, where Benjamini–Hochberg has already reached 10 × 0.0025 = 0.025. Benjamini and Hochberg made the same point in their paper: the more of the hypotheses are false, the larger the possible gain in power over a rule that controls the chance of any false finding. On every family the three are nested, because 0.05/m ≤ 0.05/(m − i + 1) ≤ i × 0.05/m: Holm keeps everything Bonferroni keeps, and Benjamini–Hochberg everything Holm keeps.

Problem solved in full

What Bonferroni costs in sample size 3 steps
A study plans twenty comparisons and will correct them with Bonferroni. Each comparison was designed to have 80% power at 5%. How much larger must every sample be to win the 80% back? Press Five real effects in twenty first: the results table gives the effect size and its power under Bonferroni.

Open this problem in the tool
1. From the page: 80% power at 5% means an effect of δ = 2.802 standard errors, and at Bonferroni's 0.0025 the same effect is found only 41.23% of the time:
  
  $Φ (δ - 1.960) + Φ (- δ - 1.960) = 0.80 \Rightarrow δ = 2.802, Φ (2.802 - 3.023) + Φ (- 2.802 - 3.023) = 0.4123$
2. Leave out the chance of landing beyond the opposite bar, which is below 10⁻⁶ at 5% and below 10⁻¹¹ at 0.0025. Then 80% power needs δ = z + 0.8416, where z is the two-sided critical value and Φ(0.8416) = 0.80. At 5% that is 1.960 + 0.842 = 2.802, and at 0.0025 it is 3.023 + 0.842 = 3.865:
  
  $δ \approx z + 0.8416 : 1.960 + 0.842 = 2.802, 3.023 + 0.842 = 3.865$
3. A standard error shrinks as 1/√n, so δ grows as √n, and the sample has to grow by the square of the ratio:
  
  $δ \propto n \Rightarrow \frac{n ^{'}}{n} = (\frac{3 . 865}{2 . 802})^{2} = 1.903$
Answer

About 1.90 times as many observations in every comparison, since (3.865/2.802)² = 1.903. With 100 comparisons Bonferroni's bar is 0.0005 and the factor is 2.38. Holm and Benjamini–Hochberg need less, by an amount that depends on how many of the effects are real; the table on the page gives what each finds, and this calculation does not.

$1.90 \times the sample (m = 20), 2.38 \times (m = 100)$

Learning path

What p < 0.05 does not promise

References (2)

The step-up rule the page calls Benjamini–Hochberg (reject the k smallest, k the largest i with p(i) ≤ (i/m)q*), the definition of the false discovery rate as E(V/R) with V/R = 0 when nothing is rejected (the table note), Theorem 1 and its lemma's bound E(Q) ≤ (m0/m)q* for independent test statistics (the second insight's 3.75%), that the FDR equals the FWER when every null is true (the second insight's agreement on noise), that the gain in power is larger when more of the hypotheses are false (the third insight), screening chemicals for drug development as a use for FDR control (the second insight), and that the procedure has greater power than Holm's (1979): Y. Benjamini and Y. Hochberg, "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing," Journal of the Royal Statistical Society, Series B (Methodological) 57 (1), 289–300 (1995).
The first insight's comic: twenty jelly-bean colours tested against acne, green alone 'p < 0.05', and the newspaper's 'only 5% chance of coincidence': R. Munroe, "Significant," xkcd no. 882.

By the 42Math team · Reviewed against published sources. Corrections welcome.