Lesson
The theory — ANOVA F-Test
A one-way ANOVA does not ask whether several group means differ — measured means always differ a little. It asks whether they differ more than random sampling alone would produce. The F-statistic makes that comparable by putting the spread between the group means over the spread within them, and the p-value says how often chance alone would manage a ratio that large.
What each symbol means
MS_between- the between-groups mean square: how far the group means sit from the overall mean, per degree of freedom.
MS_within- the within-groups mean square: the ordinary scatter of observations around their own group’s mean. This is the yardstick.
F- their ratio. Around 1 when the groups are alike, because then both are estimating the same thing.
p- the chance that groups which are genuinely identical would produce an F at least this large.
Where the formula comes from
- Suppose for the moment that every group has the same mean. Then the observations are all draws from one population, and there are two independent ways to estimate that population’s variance from the data you have.
- The first ignores the grouping entirely and just measures scatter inside each group: that is
MS_within. It estimates the population variance whether or not the means differ, which is what makes it a fair yardstick. - The second uses only the group means. If the groups really are identical, the means still wobble apart by sampling alone, and the size of that wobble is itself an estimate of the same population variance: that is
MS_between. If the means are not identical, this second estimate is inflated by the real differences on top of the sampling wobble. - So the ratio
F = MS_between / MS_withincompares an estimate that grows when the means differ against one that does not. Under “all means equal” it should land near 1 and follows a known distribution — the curve drawn above, fixed entirely by the two degrees of freedom. The p-value is the area of that curve to the right of your F.
How to read what you see
The blue curve is what F would do if every group mean were identical, and it depends on nothing but the two degrees-of-freedom numbers — not on your data. The red shaded tail is the rejection region: the values so far out that they would occur less than α of the time by chance, with the dashed line at the critical value marking where it begins. The green line is where your F actually landed. The whole test is the question of which side of the dashed line the green line falls on, and the p-value is the area to the right of it.
- Assumes
- Observations independent of each other, roughly normal within each group, and — the one that bites — roughly equal variances across groups. This page begins from the two mean squares rather than from your data, so it cannot check any of the three for you. It will compute a confident-looking p-value from any pair of positive numbers you type.
- Breaks when
- Equal variances is where it goes wrong quietly.
MS_withinpools the scatter of all the groups into a single yardstick, and if one group is far more variable than the others that yardstick describes none of them; the statistic no longer follows the curve above, and the p-value is measuring an area under the wrong distribution. The usual repair is Welch’s ANOVA, which does not pool. Unequal group sizes make the same problem worse, because the pooled estimate is then dominated by whichever group happens to be largest.
Problem solved in full
-
Three groups, thirty observations, and an F of 1.33 5 steps
Three groups, thirty observations, and an F of 1.33 that is not significant. Work out where those degrees of freedom come from, and why you cannot simply run three t-tests instead.
-
The F statistic is a ratio of two variance estimates: how much the group means differ, over how much the observations scatter within groups. A value near 1 says the groups differ about as much as chance alone would produce.
-
The degrees of freedom tell you the design without being told. The first is groups minus one, so there are three groups; the second is total observations minus groups, so there are thirty observations.
-
Multiplying each mean square by its degrees of freedom recovers the sums of squares — 8 between and 81 within, 89 in total.
-
Their ratio is eta squared, the share of the total variation the grouping explains. Nine per cent.
-
The alternative is to compare the three groups pairwise, which is three t-tests. Each carries a 5% false-positive rate, so the chance of at least one false alarm is 1 − 0.95³.
Answer
The tool prints F = 1.3333, degrees of freedom (2, 27) and η² = 0.0899. Two things follow. First, the effect size is not zero — the grouping does account for 9% of the variation — but with this much within-group scatter that is unremarkable, which is exactly what a non-significant F means and exactly what people forget when they read one as "no difference". Second, the reason ANOVA exists: three pairwise t-tests carry a 14.3% chance of at least one false positive against the 5% you thought you were buying, and with five groups that is ten tests and 40%. One test on all groups at once is the fix.
-
Learning path
When is a difference real?
References (3)
- The paper that introduced the word “variance” and the decomposition the lesson derives: R. A. Fisher, "The Correlation between Relatives on the Supposition of Mendelian Inheritance." Transactions of the Royal Society of Edinburgh 52(2), 399–433, 1919 (read 1918).
- The repair when the groups do not share a variance, which is the assumption this page cannot check for you: B. L. Welch, "On the Comparison of Several Mean Values: An Alternative Approach." Biometrika 38(3/4), 330–336, 1951.
- Where small-sample testing came from, and why a RATIO of variances was the right thing to look at: G. E. P. Box, "Guinness, Gosset, Fisher, and Small Samples." Statistical Science 2(1), 45–52, 1987.