Interactive Math & Science Lessons

← Statistics & Finance

False Findings

One cell is one hypothesis a field tests, and a dot means it is true. Set how many are true, how strong each study is and how much bias creeps in, then step through what the field does: run the studies, let the bias in, publish what came out significant, and run every published study again.

Loading interactive simulation...

Lesson

The theory — False Findings

The positive predictive value (PPV) of a field's findings is the share of its statistically significant results that are true. If a share π of the hypotheses tested are true, each study has power P at significance level α, and a share u of the analyses that found nothing are reported as findings anyway, then true hypotheses become findings with probability P + u(1 − P) and false ones with probability α + u(1 − α), and PPV = π[P + u(1 − P)] / (π[P + u(1 − P)] + (1 − π)[α + u(1 − α)]). This is John Ioannidis's 2005 formula, written with π in place of his odds R = π/(1 − π). An honest replication at the same power brings back PPV·P + (1 − PPV)·α of the findings.

What each symbol means

π
the share of the hypotheses a field tests that are true
P
the power of each study, 1 − β: the chance it finds a true effect
α
the significance level, 0.05: the chance a study of a false hypothesis reaches p < 0.05
u
bias: the share of analyses that found nothing but are reported as findings
PPV
the share of the field's findings that are true
Assumes
Every study of the same field has the same power; each hypothesis is tested once before publication; bias acts only on results that found nothing; every finding, and only findings, is published; a replication is the same study run again with the same power and no bias. A finding has no size or direction here: how much a significant study overstates its effect is the winner's curse, the previous rung.

The same p < 0.05 is worth less from a weaker study 🖖

Press 80% power. One hypothesis in ten is true and nobody bends the analysis, so a finding is p < 0.05 and nothing else. Per 1,000 hypotheses the field reports 80 true findings and 45 false ones: 64.0% of what it publishes is real. Now press 20% power. The false findings stay at 45, because they come from the 900 false hypotheses at α = 5% and power never touches them; in this draw they are the same 48 cells without a dot, in the same places, at both settings. The true findings fall to 20, and the share that is real falls to 30.8%. Low power is usually described as the researcher's loss, real effects missed. The reader loses too: a significant result from a weak study passed the same bar and is still worth less, because a smaller share of the results that pass it are real. John Ioannidis put the condition in one line in 2005: a finding is more likely true than false when (1 − β)R > α, where 1 − β is the power and R the odds that a tested hypothesis is true. At one in ten, R = 1/9, and the power has to be above 45%.

Power cannot buy back bias 🖖

Press Bias of 10%. The studies are as strong as before, 80% power, but one analysis in ten that found nothing is reported as a finding anyway: a different outcome measure, a subgroup, an inconvenient data point dropped. Ioannidis calls that share u. It hardly helps the true hypotheses, which were usually found anyway (80% becomes 82%). It hits the false ones, which honest testing turns away 95% of the time, and 10% of that 95% is 9.5 points: a false hypothesis now becomes a finding 14.5% of the time, nearly three times as often. The share of findings that are real drops from 64.0% to 38.6%, and the results row beneath it says what that is worth: an honest field reaches the same 38.6% with studies at 28.3% power. Drag the power to 99% and the biased field creeps up to 43.2%; the row for 100% power says 43.4%, and no study gets further. Now press Honest, 35% power: 43.8%. An honest field at 35% power publishes findings as trustworthy as a perfectly powered one with u = 10% could, and the crossing point, α/(α + u(1 − α)) = 34.5%, has no π in it. Move the first slider and it stays put.

A true finding can fail its replication, and often does 🖖

Press 80% power, then Start over and Step through to the last stage, where every published finding is run again, honestly and at the same power. A true one comes back 80% of the time and a false one 5%, so 53.0% of the published findings replicate; in this draw 69 of 129. Count the crosses sitting on a dot: 12 of the 60 failures were true hypotheses, and the exact share is 27.2%. Nobody cheated and nothing was wrong with those studies: at 80% power one real effect in five misses, the second time as the first. So a replication rate cannot be read as the share of a field that is true. It can never be higher than the power of the replications, since every true finding comes back at that rate and every false one at α, which is never higher. A field whose replications run at 50% power would see half its findings fail if every one of them were true. Press Bias of 10% and the same honest replications bring back 33.9%, because more of what they are asked to repeat was never there.

Problem solved in full

  1. What p = 0.04 promises the next study 4 steps

    A study reports p = 0.04, two-sided. Suppose the true effect is exactly the one it measured. An identical replication, same design and same sample size: what is the chance it reaches p < 0.05 in the same direction? Then put that power into the page, with one hypothesis in ten true and no bias, and read off what share of such a field's findings are real and how many replicate.

    Open this problem in the tool

    1. Turn the p-value back into the z it came from. Two-sided, so each tail holds 0.02, and the bar for p < 0.05 is the point with 0.025 above it:

    2. If the true effect is the one observed, the replication's z is normal with mean 2.054 and standard deviation 1. It succeeds when it lands above 1.960:

    3. Landing below −1.960 would be significant the other way, which is no replication, and is in any case negligible:

    4. Press A p = 0.04 field: 54% power (the slider moves in whole percents), one hypothesis in ten true, no bias.

    Answer

    53.7%, little better than a coin toss, and at p = 0.05 exactly the same steps give 50%. That is the generous case: the winner's curse, on the previous rung, says a significant study usually overstates its effect, so the replication's real power is lower still. In a field whose studies run at 54% power, 54.5% of the findings are real and 31.7% replicate. For an identical replication to succeed four times in five under the same assumption, the original needs p = 0.005, which gives 80.2%.

Learning path

What p < 0.05 does not promise

References (1)
  • The page's model and formula: the positive predictive value of a claimed finding from power, the pre-study odds R and the bias u (Tables 1 and 2), the condition that a finding is more likely true than false when (1 − β)R > α, which the first insight quotes, the definition of u the second insight paraphrases, and Table 4, whose first row is the 'Ioannidis's trial' preset and all nine of whose rows check:compute reproduces: J. P. A. Ioannidis, "Why Most Published Research Findings Are False," PLoS Medicine 2(8): e124, 2005.

· Reviewed against published sources. Corrections welcome.