Optional Stopping

The experiments here start with no effect in them. Press Next batch and watch its z-score after each batch of data: the first time it crosses the line, the experiment stops and reports p < 0.05. Then run the other 999.

Worked example — Optional Stopping

Threshold at look 1	1.960
Look 1 stops when p is below	0.0500
Threshold at look K	1.960
Look K stops when p is below	0.0500
False-positive rate with no effect, exact	19.34%
Chance of detecting the real effect, exact	86.49%
Average share of the data used, real effect	50.0%
In this draw of 1,000 experiments (draw 1)	176 of 1,000 run stopped with p < 0.05 (17.60%)

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Lesson

The theory — Optional Stopping

Optional stopping is deciding when to end an experiment by watching its results: testing after each batch of data and stopping as soon as the test is significant. With the fixed-sample threshold |z_k| ≥ 1.96 at each of K looks, the chance of a false positive is P(|z_k| ≥ 1.96 for some k ≤ K), which is more than 5% for every K ≥ 2. A group sequential boundary chooses thresholds c₁, …, c_K so that this chance is exactly 5%: Pocock's uses one constant, O'Brien and Fleming's uses C_B√(K/k).

What each symbol means

K: the number of looks, equally spaced through the data
zₖ: the z-statistic of all the data up to look k
cₖ: the threshold at look k; the experiment stops when |zₖ| ≥ cₖ
θ: the real effect, added to every batch, in standard deviations of one batch's total

Assumes: The data are normal with a known standard deviation, or numerous enough for the z-test to hold; the looks are equally spaced; every test is two-sided, each naive look at 5% and each boundary at the stricter level that holds the overall rate at 5%; and the largest sample is fixed in advance and the same for every rule, so the looks only divide it.

Each look adds less than the one before, and ten of them make 5% into 19% 🖖

Press Ten looks. Nothing is going on in these experiments, and the first look behaves: it stops 5.00% of them, as one test should. The second look catches 3.31% more, experiments that just missed at the first and drifted over at the second. Every later look adds less, 0.81 points by the tenth, which makes each one feel free, and the total after ten is 19.34%. The table shows where that leads. From 4 looks to 100, every doubling of the looks adds between 5.0 and 5.5 points, so a hundred looks reach 37.36%. Nothing on the page asks how large a batch is, because the answer does not depend on it: ten looks cost the same 19.34% whether each batch is ten patients or ten thousand. Klim McPherson put the figure in front of clinicians in the New England Journal of Medicine in 1974, ten unadjusted looks turning a nominal 5% into nearly 20%, five years after he, Peter Armitage and B. C. Rowe had tabulated the inflation in the Journal of the Royal Statistical Society. The p-value at the moment of stopping is computed as if the experiment had been planned to end there, and it was not.

Pocock can allow more than a fifth of 5% at each of five looks 🖖

Press Pocock, five looks. One threshold at every look, raised until the overall rate is exactly 5%: |z| ≥ 2.413, a p-value below 0.0158. Splitting 5% evenly would give 0.01 a look, the Bonferroni correction from <a href="/stat/multiple-comparisons">multiple comparisons</a>, and that is stricter than it needs to be here. The five tests share their data. The fourth look's data are four-fifths of the fifth look's, the correlation of the two z-scores is √(4/5) = 0.894, and an experiment that crosses at the fifth has usually come close at the fourth. Adding up the five chances, 5 × 0.0158 = 7.9%, counts those overlaps more than once. The gap grows with the looks: at 100 of them Pocock allows p < 0.0040 at each, eight times 0.05/100. Stuart Pocock worked the method out in Biometrika in 1977 for exactly this page's case, equal groups of patients and a normal response of known variance. Now press O'Brien–Fleming, five looks. Its threshold falls as the data grow, C_B√(K/k): 4.562 at the first look, where p must be below about 0.000005, and 2.040 at the fifth. The bars show the consequence. It spends four-fifths of its 5% at the last two looks.

With a real effect, O'Brien–Fleming keeps nearly all the power and Pocock loses nine points 🖖

Press Pocock, real effect. The effect is sized so that one test on all the data detects it 80% of the time. Spread over five looks with Pocock's boundary, the same data detect it 70.89% of the time. In return the experiment stops early: on average it uses 70.7% of the data. Press O'Brien–Fleming, real effect: 78.88%, using 80.1%. Its last threshold, 2.040, is barely stricter than 1.96, so an experiment that runs to the end is judged almost as a single test would judge it. The table keeps going: at 100 looks O'Brien–Fleming still has 77.85% and Pocock 60.65%. The two any-p < 0.05 columns show why stopping at any p < 0.05 looks attractive to the person doing it. It detects the effect 84.84% of the time at five looks, more than either boundary, because it would also have reported one 14.17% of the time when there was nothing there.

Problem solved in full

The fourth look of ten 4 steps
An online shop tests a new checkout page on 10,000 visitors and looks at the result after every 1,000. Under O'Brien–Fleming's boundary, how small must the p-value be at the fourth look for the test to stop there? Press Ten looks, O'Brien–Fleming first: the page gives the threshold at the first and the last look.

Open this problem in the tool
1. From the page, the threshold at the last look is C_B:
  
  $C_{B} (10) = 2.087, 2 (1 - Φ (2.087)) = 0.0369$
2. Every other threshold is C_B scaled by √(K/k). At the fourth look of ten:
  
  $c_{4} = C_{B} K / k = 2.087 \times 10/4 = 2.087 \times 1.5811 = 3.300$
3. As a two-sided p-value, from the normal distribution:
  
  $p_{4} = 2 (1 - Φ (3.300)) = 0.00097$
4. For comparison, stopping at any p < 0.05 over the same ten looks, from the table:
  
  $P (∣ z_{k} ∣ \geq 1.96 at some k \leq 10) = 0.1934$
Answer

p below 0.00097, about one in a thousand. A fourth-look result of p = 0.004, which would end a naive test with a headline, keeps O'Brien–Fleming's test running. At the tenth look the bar is p < 0.0369, a little stricter than 0.05, and the whole procedure holds the false-positive rate at 5%, where stopping at the first p < 0.05 lets 19.34% through.

$p_{4} < 0.00097, p_{10} < 0.0369$

Learning path

What p < 0.05 does not promise

References (2)

The first insight's history: Klim McPherson's 1974 note in the New England Journal of Medicine, showing that testing accumulating data ten times without adjustment raises a nominal 5% false-positive rate to nearly 20% (the page computes 19.34%), and his earlier tables of the inflation with Peter Armitage and B. C. Rowe (1969). Also the source for O'Brien and Fleming's 1979 boundary being the one that became standard: C. Gluud and K. Thorlund, "Long overdue recognition of Klim McPherson's 1974 article on sequential analysis of trial data," JLL Bulletin: Commentaries on the history of treatment evaluation, James Lind Library, 2025 (also J R Soc Med 2025;118:304–307). On K. McPherson, N Engl J Med 1974;290:501–502; P. Armitage, C. K. McPherson and B. C. Rowe, J R Stat Soc A 1969;132:235–244; P. C. O'Brien and T. R. Fleming, Biometrics 1979;35:549–556.
Pocock's group sequential design: patients enter in equal-sized groups, the trial stops or continues after each group on a repeated significance test, with exact results for two treatments and a normal response of known variance, which is the model the page computes. The constants the page prints (2.178, 2.289, 2.361, 2.413, 2.555 for 2, 3, 4, 5, 10 looks) are the module's own, asserted in check:compute: S. J. Pocock, "Group sequential methods in the design and analysis of clinical trials," Biometrika 64(2), 191–199, 1977. doi:10.1093/biomet/64.2.191

By the 42Math team · Reviewed against published sources. Corrections welcome.