The Winner's Curse

Choose a true effect and a sample size, then run the study again and again. Each dot is one study's estimate; the coloured ones reached p < 0.05. Watch where the coloured dots land, compared with the true effect.

Worked example — The Winner's Curse

Standard error of the estimate	0.2000
True effect in standard errors, λ	1.00
Smallest estimate that reaches p < 0.05	0.392
Power: the chance a study reaches p < 0.05	17.0%

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Lesson

The theory — The Winner's Curse

The winner's curse in statistics is the bias of estimates kept because they are statistically significant. A study's estimate d̂ of a true effect δ is significant at α = 0.05 when |d̂| > z·SE with z = 1.96, and given that, its expected size E(|d̂| | significant) exceeds δ. Andrew Gelman and John Carlin measure it with the exaggeration ratio E(|d̂| | significant)/δ (the Type M error) and with the probability that a significant estimate has the wrong sign (the Type S error). For a normal estimate with λ = δ/SE, the power is Φ(λ − z) + Φ(−λ − z) and the Type S rate is Φ(−λ − z) divided by the power.

What each symbol means

δ: the true effect, in standard deviations of the outcome
d̂: one study's estimate of δ
n: the number of people in each of the two groups
SE: the standard error of the estimate, √(2/n)
λ: the true effect measured in standard errors, δ/SE
z: the bar in standard errors, 1.96 for p < 0.05 two-sided

Assumes: Two groups of n with an outcome whose values are normal with a known standard deviation of 1, so the estimate is exactly normal, N(δ, 2/n), and the test is a two-sided z-test at α = 0.05. The true effect is positive and the same in every study, the studies are independent, and a study is kept exactly when it is significant.

Below 50% power, every significant result overstates the size of the effect 🖖

Press A small study. The true effect is 0.20 standard deviations and each group has 50 people, so the estimate has a standard error of 0.20 and a study reaches p < 0.05 only if its estimate lands more than 0.392 from zero. Nothing smaller is ever significant, so nothing smaller is ever the finding. Only 17.0% of studies clear that bar, and the ones that do average 0.498, which is 2.49 times the effect that produced them. Look at the coloured dots: not one lies between the two dashed bars, and the true effect does. The same is true whenever the true effect is below the bar, and the true effect is below the bar whenever power is 50% or less. Press Power at 50%: 192 per group brings the bar down to 0.200, level with the truth, and still every significant estimate overstates it. Past that point a significant estimate can fall short, but at least half of them overshoot at any sample size whatever, because the estimates that land near zero, the ones that would have pulled the average down, are the ones that never reach significance. At 80% power, 400 per group, it is 61.9%.

A significant estimate from a small study measures the noise 🖖

Press A small study and then An effect of 0.05. The true effect falls to a quarter of what it was, and the average significant estimate hardly moves: 0.498, then 0.471. With 50 per group, any true effect between zero and the bar gives significant estimates that average between 0.468 and 0.552. In standard errors the band is 2.34 to 2.76, and it is the same band for every design with 50% power or less, so in such a study the size of a significant result tells you mostly how large its standard error was. The study cannot see this from the inside. Put 0.498 into the power formula as if it were the truth and this design, whose power is 17.0%, appears to have 70.2%; do the same with 0.471 and the 5.7% design appears to have 65.4%. Andrew Gelman and John Carlin describe exactly this circle in their 2014 paper: any significant estimate from a small study is large, and a power analysis built on it makes the study look reasonable. Their way out is to take the effect size from outside the study, from the literature or from what is physically plausible, which is what the worked problem below does.

A significant result can point the wrong way 🖖

Press An effect of 0.05. Power is 5.7%, barely above the 5% that a true effect of exactly zero would give, and of the studies that reach significance, 23.7% find an effect in the wrong direction. Each of them came with a 95% confidence interval that excluded zero, on the wrong side. Gelman and Carlin call this a Type S (sign) error, and the overstatement a Type M (magnitude) error; here the significant estimates are 9.42 times the true effect. Their example is Satoshi Kanazawa's 2007 report that attractive parents have more daughters: a difference of 8 percentage points with p = 0.015, so a standard error of 3.3. The literature on sex ratios suggests a true difference well under one point. If it were 0.3, they calculate, a significant result would have roughly a 40% chance of the wrong sign and would overstate the effect about 25 times. The sign is far harder to get wrong than the size: in this model the wrong-sign rate stays under 1% until power falls below 16.5%, while the exaggeration at that power is already 2.5. The next rung, false findings, asks the question these three leave open: of all the significant results in a field, how many are real at all?

Problem solved in full

The study that found 0.45 4 steps
A study with 50 people in each group reports an effect of 0.45 standard deviations, p = 0.024. Similar interventions are known to move this outcome by about 0.10. If 0.10 is the truth, how much did this design have to overstate it, how likely was it to get the sign wrong, and how many people per group would give 80% power? Press The 0.45 study: the page sets δ = 0.10 and n = 50.

Open this problem in the tool
1. First the reported result. The standard error and the z statistic:
  
  $SE = 2/50 = 0.2000, z = \frac{0 . 45}{0 . 2000} = 2.25, p = 2 Φ (- 2.25) = 0.024$
2. Take δ = 0.10 from outside the study. In standard errors that is λ = 0.5, and the power is:
  
  $λ = \frac{0 . 10}{0 . 2000} = 0.5, power = Φ (0.5 - 1.96) + Φ (- 0.5 - 1.96) = Φ (- 1.46) + Φ (- 2.46) = 0.07215 + 0.00695 = 7 . 91%$
3. Of all studies, 7.91% reach significance and 0.69% are significant on the wrong side, which is 8.8% of the significant ones. The significant ones average 0.479 in size:
  
  $Type S = \frac{0 . 00695}{0 . 0791} = 8 . 8%, E (∣ \hat{d} ∣ ∣ sig) = 0.479 = 4.79 δ$
4. For 80% power, λ must reach z + Φ⁻¹(0.80), the far tail being negligible, and λ = δ√(n/2):
  
  $λ = 1.96 + 0.8416 = 2.8016 = 0.10 n /2 \Rightarrow n = 2 (\frac{2 . 8016}{0 . 10})^{2} = 1569.8$
Answer

1,570 people per group, 31.4 times as many. With 50 per group the design had 7.9% power, a significant estimate had to be at least 0.392, and the significant ones average 0.479, 4.79 times the truth, with 8.8% of them the wrong sign. The reported 0.45 is what this design produces when the true effect is 0.10, so it gives no reason to reject 0.10. Type 1570 into the page: power 80.0%, and the significant estimates average 0.112, 1.12 times the truth.

$n = 1570 per group (31.4 \times 50) : power 80 . 0%, E (∣ \hat{d} ∣ ∣ sig) = 0.112 = 1.12 δ$

Learning path

What p < 0.05 does not promise

References (1)

Type S and Type M errors and the exaggeration ratio, the formulas for power and the Type S rate under a normal estimate (note 2), the R function retrodesign() whose figures check:compute reproduces, the beauty-and-sex-ratio example the third insight quotes, and the circle the second insight describes (a significant estimate from a small study fed back into a power analysis makes the study look reasonable): A. Gelman and J. Carlin, "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors," Perspectives on Psychological Science 9(6), 641–651, 2014.

By the 42Math team · Reviewed against published sources. Corrections welcome.