Calibration: Saying 90% and Being Right 90% of the Time

Being accurate and being calibrated are different skills, and only one of them is what people usually check. The dial below changes what the forecaster said and never what happened, so every score you see describes the same two hundred events.

Worked example — Calibration: Saying 90% and Being Right 90% of the Time

Brier score	0.1934
Reliability, lower is better	0.0219
Resolution, higher is better	0.0718
Uncertainty, nobody’s to change	0.2458
Those three, recombined	0.1959
A perfect forecaster’s resolution	0.0718
This one against that	100.0%
Mean forecast minus outcome rate	-0.0671

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Lesson

The theory — Calibration: Saying 90% and Being Right 90% of the Time

A forecast is calibrated when the events it called 70% likely happen about 70% of the time — a property of a whole set of forecasts, never of one. Score them with the mean squared error of the probability against the outcome, the Brier score, and it splits into three: B = reliability − resolution + uncertainty. Reliability is how far each group of forecasts sits from the rate that actually followed it, resolution is how far those rates sit from the base rate, and uncertainty belongs to the world rather than to the forecaster.

What each symbol means

p: the probability a forecaster stated
o: what happened, 1 or 0
B: the Brier score, the mean of (p − o)² over every forecast

Assumes: That the events are exchangeable within a bin — that grouping all the 0.7 forecasts together and counting outcomes is a fair test. It is the assumption the bin count quietly rests on, and the reason a forecaster who says 0.7 about two very different kinds of event can look calibrated while being useless about both.

Comparing the two averages tells you almost nothing 🖖

The obvious check on a forecaster is to average what they said and compare it with how often things happened. Load the last preset and try it: sixty events, a forecaster saying exactly the true probability every time, average forecast 0.5283 against an outcome rate of 0.4167. That is eleven points of apparent overconfidence from someone who was perfectly honest, and their resolution is 100.0% of what a perfect forecaster reaches on the same events. Now go back to the first preset, same forecaster, two hundred events: the gap is −0.0671, which points the other way. Across fifty-six sample sizes an honest forecaster’s gap averages 0.0242 and reaches 0.1116 at its worst, while a badly overconfident one averages 0.0425. The two ranges overlap almost completely. The reliability diagram exists because an average over everything cannot see what the bins can: it adds the 90% claims to the 10% claims and reports the middle.

Perfectly calibrated and worth nothing, twice, for opposite reasons 🖖

Set the confidence to 0. Every forecast becomes the base rate, reliability is 0.0000 exactly, resolution is 0.0000 exactly, and the Brier score equals the uncertainty to the last decimal. This forecaster is perfectly calibrated in the strict sense — when they say 56.5%, it happens 56.5% of the time — and they have told you nothing, because they said it about everything. Now click Nothing to know: an honest forecaster on two hundred coin flips, and the scoresheet reads the same two zeros. Identical numbers, opposite diagnoses. What separates them is the one figure on this page a real forecaster could never compute: the resolution a perfect forecaster reaches, which is 0.0718 on the predictable events and 0.0000 on the coins. Without it, resolution 0 means "you added nothing" and cannot say whose fault that was.

Your calibration error depends on how you chose the bins 🖖

Leave everything alone and drag the bin count from 4 to 20. On four hundred honest forecasts the reliability runs from 0.0023 to 0.0124 — more than five times over, on forecasts that have not changed at all. Now push the confidence to 3 and drag the bins again: 0.0334 to 0.0491, a factor of 1.47. The dependence is worst exactly where you would most like to trust the number, because when the true miscalibration is near zero almost all of what you measure is the binning. Few bins hide a wiggle inside a wide band; many bins put four forecasts in a bucket and call the distance between 0 and 1 a finding. There is one forecaster whose reliability does not move by a thousandth at any bin count, and it is the one at confidence zero who said the base rate about everything.

Problem solved in full

The score says you should shade your forecasts down a little. Work out whether to believe it. Start from An honest forecaster and walk the confidence dial slowly across 1, watching the Brier score in the lower panel.

Open this problem in the tool
1. The Brier score is the average of (forecast − outcome)², so lower is better and 0 is perfect. Read it at a few settings: it falls as you approach 1 from either side, which is the shape you would expect if honesty were optimal.
  
  $B = \frac{1}{n} i = 1 \sum n (p_{i} - o_{i})^{2}, 0 \leq B \leq 1$
2. But look more carefully near the bottom. On these two hundred events the minimum does not sit at 1. Sweeping the whole dial in steps of 0.05 puts the best score at k = 0.80, at 0.1916 against honest forecasting’s 0.1934. Switch the events to Rare and it is 0.75. Shading every forecast towards the base rate scores better than telling the truth.
  
  $honest (k = 1) : B = 0.1934 k min B = 0.1916 at k = 0.80$
3. That should be alarming, because the Brier score is a proper scoring rule: it is built so that stating your true belief maximises your expected score. If shading wins, either the rule is not proper or something else is going on.
  
  $a proper rule : E [B (p)] is minimised at p = Pr (event)$
4. Change the sample size and sweep again. The winning k moves: it is not 0.85 at every size, it wanders. Over the fifty-six sample sizes between 50 and 600 the best k averages 0.968 on the predictable events and 1.021 on the rare ones, landing below 1 thirty times and above it twenty-six.
  
  $predictable events rare events best k 0.968 1.013 it beats honest by 0 . 39% 0 . 85%$
5. And measure what the winner actually wins. Averaged over those same sweeps the best k beats honest forecasting by 0.39% of the Brier score on the predictable events and 0.85% on the rare ones. The prize for finding the perfect amount of shading is under one per cent, and the amount you should shade by changes every time you look.
  
  $0.1934 - 0.1916 = 0.0018 \approx 0 . 9% of B$
Answer

No. The k that wins on your sample is noise, scattered around 1, and it beats honesty by well under a per cent. The scoring rule is doing exactly what it was designed to do: honesty is optimal in expectation, and a finite history wobbles around that expectation like anything else measured once. Choosing your confidence by which value scored best on the events that already happened is fitting the scoring rule to one history. It is the same mistake in a different costume as choosing a polynomial degree by which one minimised the error on a single held-out split. The defence is also the same: the question is not "what scored best last year" but "what would score best on average", and one year cannot answer it. Which is why a forecaster’s track record is quoted over decades, and why the reliability diagram — which shows you where the miscalibration is rather than how much it cost — survives a small sample better than the score does.

$ar g k min B on a sample \neq = 1, E [ar g k min B] = 1$

Learning path

Fitting a line to data

References (3)

The score itself, three pages long, written for weather forecasting and still the standard way to grade a probability: G. W. Brier, “Verification of forecasts expressed in terms of probability.” Monthly Weather Review 78(1), 1–3, 1950.
The split into reliability, resolution and uncertainty — the three terms this page prints, and the reason they have to be separate: A. H. Murphy, “A New Vector Partition of the Probability Score.” Journal of Applied Meteorology 12(4), 595–600, 1973.
Why honesty is the best policy under this score and not merely the decent one — the paper that named the property: L. J. Savage, “Elicitation of Personal Probabilities and Expectations.” Journal of the American Statistical Association 66(336), 783–801, 1971.

By the 42Math team · Reviewed against published sources. Corrections welcome.