Temperature and Sampling

Everything a model does between scoring the vocabulary and picking a word is arithmetic with no randomness in it. The dice are thrown once, at the very end.

Worked example — Temperature and Sampling

Lead over second place	3.08×
Candidates kept	12 / 12
Probability kept	100.0%
Distinct continuations in the corpus	1,280
Share of occurrences these 12 cover	13.6%

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Temperature is not randomness 🖖

Every step on this page except the last is arithmetic with a fixed answer. Temperature divides the scores before the exponential, top-k keeps a fixed count, top-p keeps a fixed share — run any of them twice with the same settings and you get identical numbers. The randomness enters exactly once, when Draw a token throws a single number against these probabilities. That is why "turn the temperature down to make it deterministic" is the wrong description: a low temperature makes one outcome overwhelmingly likely, which is not the same as making it certain.

Top-k and top-p hold different things fixed 🖖

Top-k keeps a fixed number of candidates and lets the captured probability float: k = 3 keeps three every time, but those three hold 40.7% after ␣a and 72.2% after ␣so. Top-p does the opposite — it fixes the share and lets the count float, so p = 50% keeps five candidates after ␣a and only one after ␣it. That is why nucleus sampling is usually preferred: the number of options a model should consider depends on how sure it is, and only one of these two settings can notice that.

One token of context, and twelve candidates out of 1,280 🖖

This page runs a bigram: it looks at exactly one token behind and no further. A transformer looks at thousands, which is the entire difference between this and something that can write. What survives the comparison is the arithmetic — scores become probabilities, temperature reshapes them, truncation cuts them, one is drawn — and the shape of the problem. After ␣the this corpus offers 1,280 distinct continuations and the twelve shown here cover 13.6% of its occurrences. The tail is most of the distribution, and truncation is what stops it leaking into the output.

Problem solved in full

Twelve candidate continuations that cover 13.6% of what actually follows 5 steps
Twelve candidate continuations cover 13.6% of what actually follows this context. Work out what the other 86% looks like, and why that shape decides how sampling has to work.
1. The corpus offers 1280 distinct continuations of this context. That is the real distribution the model is trying to imitate, and it is far wider than any shortlist.
  
  $the corpus offers 1280 distinct continuations of this context$
2. The top twelve — the ones the panel scores — account for 13.6% of all occurrences between them.
  
  $the top 12 cover 13.6% of all occurrences$
3. So the remaining 1268 continuations share 86.4%. Not one of them is likely on its own, and collectively they are most of what happens.
  
  $1280 - 12 = 1268 continuations share the remaining 86.4%$
4. That is 99.1% of the options holding 86.4% of the probability: a long tail, not a peak with noise around it.
  
  $\frac{1268}{1280} = 99.1% of the options carry 86.4% of the mass$
5. With top-k off all twelve are kept and the panel reads 100% probability kept. That 100% is measured inside the shortlist while the 13.6% is measured against the corpus, so every filter on this page cuts the head: the 1,268 tail continuations were gone before any slider moved.
  
  $top- k off \Rightarrow 12/12 kept, 100% probability retained$
Answer

The tool prints 12 / 12 kept, 100.0% of probability retained, 1,280 distinct continuations and 13.6% coverage. The tail is why sampling is hard. Always take the most likely token and the text collapses into repetition, because you are choosing from the 13.6% forever. Sample from everything and you eventually draw something absurd, because 86.4% of the mass sits on options that are individually terrible. Top-k and top-p exist to cut the tail without flattening the head, and the numbers above are the trade-off they are negotiating. Watch which way temperature moves it: set top-p to 90 and ten of the twelve survive, then raise the temperature to 1.50 and an eleventh comes back, because flattening the distribution makes the nucleus wider rather than narrower.

$a long tail: 99.1% of continuations hold 86.4% of the probability$

Learning path

How an LLM picks the next word

References (2)

Where the temperature in a softmax comes from — it is the one in the Boltzmann distribution: D. H. Ackley, G. E. Hinton and T. J. Sejnowski, "A Learning Algorithm for Boltzmann Machines." Cognitive Science 9(1), 147–169, 1985.
Why lowering the temperature is not the same as improving the text, and what nucleus sampling does instead: A. Holtzman, J. Buys, L. Du, M. Forbes and Y. Choi, "The Curious Case of Neural Text Degeneration." ICLR 2020.

By the 42Math team · Updated 2026-08-10

Reviewed against published sources. Corrections welcome. Editorial policy · Corrections

All Computing lessons →

Example problems

Least certain context - After ␣a the corpus offers 980 different continuations, and the leader ␣single takes only 24.6% of the twelve shown. These twelve cover 12.5% of everything that ever followed it.
Most certain context - After ␣so the leader ␣the takes 43.2%, and the twelve shown cover 53.9% of all occurrences — this context has made up its mind in a way ␣a has not.
Cold: temperature 0.2 - At temperature 0.2 the leader ␣same holds 99.5% and leads second place by 278×. Nothing was removed; the gaps were simply stretched.
Hot: temperature 2 - At temperature 2 the same leader ␣same holds 18.2%, and its lead over second place is down to 1.76×. It is still the leader — that never changed.
Nucleus: keep 50% - Keeping 50% of the probability leaves 3 candidates of 12, holding 54.5% between them. Renormalising hands the leader 63.6%.
Temperature 0 - Temperature 0 keeps 1 candidate of 12. This is not a very cold softmax; it is argmax, and the readout says so.