Overfitting: Fitting Well and Predicting Badly

A model that fits your data perfectly has told you nothing about anything else. The degree slider below makes that concrete: watch one error go down while the other comes back up, and notice how far apart they end.

Worked example — Overfitting: Fitting Well and Predicting Badly

Training error, over 28 points	0.1676
Held-out error, over 12 points	0.2054
The gap between them	0.0379
Held-out error, relative	1.23×
The noise nobody can predict	0.2000
True error, over the whole window	0.2293
So the held-out set is	2 too simple

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Lesson

The theory — Overfitting: Fitting Well and Predicting Badly

Overfitting is a model describing the noise in the sample it was fitted to. Fit a polynomial of degree d through n noisy points and the training error falls for every extra degree, because each one buys another way to bend towards the points you already have; at d = n − 1 the curve passes through all of them exactly and the fit is perfect and worthless. The error that matters is the one on data the fit has never seen.

What each symbol means

d: the degree of the polynomial, the model's freedom to bend
σ: the noise the points were drawn with, which no model can predict

Assumes: That the noise is independent from point to point and the underlying curve does not move. Where the errors are correlated — a sensor drifting, a market regime — held-out data is not independent of the training data either, and every number on this page becomes optimistic rather than honest.

Training error cannot tell you when to stop 🖖

Drag the degree from 0 to 12 and watch the blue curve. It never rises. Not once, at any setting of any other control: 144 combinations of truth, sample size, noise and split were swept, with zero exceptions. The reason is algebraic rather than statistical. Every degree-4 polynomial is a degree-5 polynomial with a zero coefficient, so a higher degree can always do at least as well on the points it was fitted to. It simply keeps the old answer if it cannot beat it. That makes "which degree gives the lowest training error?" a question with no information in it, since the answer is always the largest one you tried. The blue curve is not a diagnostic. It is a measure of how much freedom you gave the model.

A held-out score better than the noise is not skill 🖖

Set the true function to Straight, the sample size to 40, the noise to 0.5 and the split to 70%, then walk the degree slider from end to end. Every single degree reports a held-out error below 0.5, and at degree 1 it reports 0.2353. That should be impossible. The data is a straight line plus noise of standard deviation 0.5, so a model that had recovered the line exactly would still miss each point by about 0.5 on average, and nothing can predict noise. The green curve is that same error computed exactly instead of estimated from twelve points, and at degree 1 it reads 0.5037 — a shade above the noise, which is what a nearly perfect model looks like. It never once dipped below the noise, in 468 checks at this noise level or in 8,424 across the whole range. So the held-out number was not measuring the model. It was reporting that these particular twelve points happened to be easy, and it was wrong by more than a factor of two in the direction that flatters you.

One split gets the complexity right 42% of the time 🖖

The two triangles under the error panel are the degree the held-out set votes for and the degree that is genuinely best. Over 540 combinations of truth, sample size, noise and split they landed on the same degree in 226 of them, and within one degree in 325. The mean distance between them was 1.85 degrees, and the worst cases were five degrees too simple and eleven too complex. The direction is not symmetric either: 233 times the held-out set asked for a model more complex than the right one, against 81 times simpler, because a small test set can be flattered by a wiggle that happens to pass near it. None of this is a flaw in the arithmetic — the held-out error is computed correctly every time. It is that one number computed from a dozen points is a blurry view of the quantity you actually want, and the fix is to average several splits instead of trusting one. That is cross-validation, and this page is the argument for it.

Problem solved in full

You have heard that more data lets you use a bigger model. Find out when that is true and when it is not, using the two truths where the answer differs. Start with the Straight truth preset and the The true model, too little data preset, and compare what the sample-size slider does to each.

Open this problem in the tool
1. Take the straight line first. Set the true function to Straight, the noise to 0.6, and walk the sample size across its whole range. The degree with the lowest true error sits at 1 for eight of the eleven sizes and at 2 for the other three, and it never goes higher than 2 or lower than 1. There is no trend in it — the twos land at 15, 20 and 50 points, not at the large end. More data buys no degrees here at all.
  
  $straight truth : ar g d min risk (d) = 1 at eight of eleven sample sizes$
2. Now the cubic, which is where the slogan earns its keep. Set the true function to Cubic, the noise to 0.6, the split to 70%, and the sample size to 10. Seven points to train on. The true error at degree 3 is 0.7297 and at degree 1 it is 0.6107, so a straight line beats the correct model by a clear margin.
  
  $n = 10, n_{train} = 7 : risk (1) = 0.6107 < risk (3) = 0.7297$
3. Read that again, because it is the surprising part: the data really was generated by a cubic, and fitting a cubic is the wrong thing to do. Four coefficients estimated from seven noisy points are so badly pinned down that the error they carry is larger than the curvature they capture.
  
  $four coefficients from seven noisy points : Var [\hat{f}] \propto \frac{d + 1}{n _{train}}$
4. Leave everything else and drag the sample size to 40. Twenty-eight training points now. Degree 3 gives 0.6496 and degree 1 gives 0.6616, so the cubic has moved ahead and the lowest-true-error degree is 3.
  
  $n = 40, n_{train} = 28 : risk (3) = 0.6496 < risk (1) = 0.6616$
5. The two cases differ in one respect. For the straight line, degree 1 already contains the truth exactly, so no amount of data creates a reason to add a coefficient — there is nothing left for it to explain. For the cubic at ten points, degree 3 also contains the truth, and the sample is too small to locate it.
  
  $n 1040 risk (1) 0.6107 0.6616 risk (3) 0.7297 0.6496 best degree 13$
Answer

More data does not buy you a bigger model. It buys you the ability to afford the model you already needed. The condition is whether your current model class is missing something real: if it is, more data eventually lets you reach for the missing piece, and if it is not, more data changes nothing about the right complexity. That is why the straight-line truth sits at degree 1 from ten points to a hundred and twenty, while the cubic at high noise moves from 1 to 3 somewhere between ten points and forty. And the movement is not tidy — the seven sizes from 10 to 60 give 1, 1, 1, 4, 1, 3, 3, which is a trend with sampling variation on top rather than a staircase, because every sample size draws its own sample. A single run at a single size cannot tell you which of those two things you are looking at, which is the same lesson the two triangles under the error panel keep making.

$more data \Rightarrow a bigger model only if bias (d_{now}) > 0$

Learning path

Fitting a line to data

References (3)

Why a high-degree polynomial through evenly spaced points swings wildly between them — the effect the top panel shows at degree 11 and above: C. Runge, “Über empirische Funktionen und die Interpolation zwischen äquidistanten Ordinaten.” Zeitschrift für Mathematik und Physik 46, 224–243, 1901.
The origin of the fix this page argues for but does not implement — averaging several splits instead of trusting one: M. Stone, “Cross-Validatory Choice and Assessment of Statistical Predictions.” Journal of the Royal Statistical Society Series B 36(2), 111–133, 1974.
The standard treatment of everything on this page — the bias-variance decomposition, the shape of a test-error curve, and why a single split is a noisy estimate of it: T. Hastie, R. Tibshirani and J. Friedman, The Elements of Statistical Learning. Springer, 2009.

By the 42Math team · Reviewed against published sources. Corrections welcome.