Run twenty tests on nothing at all, and one of them will be significant
One test on real data gives p = 0.0002982. Twenty tests on pure noise give you a headline.
Open the t-test tool at its defaults: a sample of 30 with a mean of 78 against a hypothesised 72, standard deviation 8. It reports t = 4.1079 on 29 degrees of freedom, and p = 0.0002982. That is three chances in ten thousand of seeing a difference this large if there were no real difference at all. Reject the null, and be fairly comfortable about it.
Now set α = 0.05 and think about what that threshold actually promises.
The promise is per test
α = 0.05 says: if the null hypothesis is true, this test will wrongly reject it 5% of the time.
Run twenty independent tests on data with nothing in it. Each one behaves. Each one has a 95% chance of correctly finding nothing. But the chance that all twenty behave is
0.9520 = 0.358
so the chance that at least one of them screams is 64.2%. Not a risk — the likely outcome. Twenty tests on pure noise produce a significant result more often than not, and that significant result will have a p-value under 0.05, a confidence interval that excludes zero, and every other marking of a real finding, because it is indistinguishable from one.
Push it further: at forty tests the probability is 87%, at a hundred it is 99.4%. Anyone who runs enough comparisons is guaranteed a discovery, and the discovery is the arithmetic.
The arithmetic fix, and why it is not enough
The correction is as simple as the problem. If you are going to run m tests and want the chance of any false positive to stay at 5%, test each one at α/m instead. That is Bonferroni: for twenty tests, demand p < 0.0025 rather than p < 0.05.
Put 0.0025 into the t-test tool's α box and watch what happens to the critical value: the bar for "significant" moves out, and results that comfortably cleared the old line no longer do. That is the price, and it is a real one: a stricter threshold means genuine effects get missed. There are gentler corrections that trade the two errors differently, but every one of them costs power, because you cannot get something for nothing here.
The deeper problem is that the correction needs m, and m is usually unknowable. It is not the number of tests in the paper — it is the number of tests that were run. Every subgroup that did not make the write-up, every outcome measure that was tried and dropped, every choice about which outliers to exclude, is a comparison. A study reporting one test may have made fifty decisions, and the reader has no way to count them.
This is what Ioannidis was pointing at in 2005 with the deliberately blunt title Why Most Published Research Findings Are False. His argument is not that researchers cheat. It is that when many groups test many hypotheses, most of which are false to begin with, the significant findings that surface are dominated by the false ones, and no individual test in that pile is doing anything wrong.
What to look at instead
Three habits that survive this, all of which the tools on this site can show you.
Look at the size, not the verdict. A p-value answers "could this be noise?" and nothing else. It says nothing about how big the effect is or whether it matters. The confidence interval tool exists because a range carries both: where the effect probably sits, and how precisely you know it. A significant result with an interval running from "trivial" to "enormous" has told you almost nothing, and its p-value hides that.
Ask where the difference lives. A chi-square test that comes back significant tells you a table does not fit, not which part broke. The residuals tool makes the same point structurally: the overall statistic is a summary, and the cells are where the information is. A finding you cannot localise is usually a finding you cannot replicate.
Count the comparisons before you look. Deciding what you will test, and how many things, before seeing the data is the only version of this that works, because after the data arrives, every threshold is negotiable and the negotiation is invisible. Pre-registration is the only way to make m a known number.
The 0.0002982 the tool printed at the top is a genuinely small p-value. It is also just one number from one test, and its meaning depends entirely on something the number itself cannot tell you: how many other tests you ran to get it.
References (1)
- what this arithmetic does to a whole literature Ioannidis (2005). Why Most Published Research Findings Are False. PLoS Medicine 2(8).