NLP Entropy Lab

Visualize token probability distributions, Shannon entropy, perplexity, cross-entropy, and KL divergence — the core mathematical metrics used to train and evaluate AI Large Language Models (LLMs).

Worked example — NLP Entropy Lab

Tokenization Mode	Words
Shannon Entropy H(P)	4.87254 bits
Perplexity PP(P)	29.2942
Laplace Smoothing (Reference B)	46
Cross-Entropy H(P, Q)	6.08388 bits
KL Divergence D_KL(P \|\| Q)	1.21134 bits

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Minimising cross-entropy is the same thing as predicting the next token 🖖

Cross-entropy is what a language model is trained on, and the reason is an identity rather than a convention. Cross-entropy equals the entropy of the data plus the KL divergence from the data to the model, so with the data fixed, driving cross-entropy down can only mean driving KL down. Load Zipf natural: entropy 4.87 bits, cross-entropy 6.08, and the 1.21 between them is exactly that KL. Perplexity is the same number on a friendlier scale, 2^4.87 = 29.29. KL turns up again in RLHF, as the penalty that keeps an aligned model near the one it started from.

What entropy really measures: surprise 🖖

Entropy scores how unpredictable a text is — the average "surprise" carried by each token. Perplexity turns that into an intuitive count: a perplexity of 8 means the model is, on average, as uncertain as if it were choosing uniformly among 8 words. Compare a repetitive prompt with varied prose and watch perplexity climb — predictable text scores low, diverse text scores high.

Shannon measured English with a parlor game 🖖

In 1951 Claude Shannon estimated the entropy of English not with a computer but with people: subjects guessed the next letter of hidden text, and their success rate bounded English at roughly 0.6 to 1.3 bits per character. That number is still the yardstick — modern LLMs are judged by how close their per-character perplexity gets to this human-derived limit, seventy years later.

Problem solved in full

Where the four entropy figures on the panel come from 5 steps
Entropy 4.87 bits, perplexity 29.29, cross-entropy 6.08, KL divergence 1.21. Four numbers on the panel and only two of them are independent. Find the two exact relations that tie them together.
1. Entropy is the average surprise of the true distribution, measured in bits: the number of yes/no questions needed per token if you encode optimally for this text.
  
  $H (P) = - i \sum p_{i} lo g_{2} p_{i} = 4.87254 bits$
2. Perplexity just puts that on a friendlier scale — two to the power of the entropy. It answers 'how many equally likely options would feel this uncertain?', and 29.29 is easier to picture than 4.87 bits.
  
  $PP (P) = 2^{H (P)} = 2^{4.87254} = 29.2942$
3. Cross-entropy is the cost of encoding the true distribution using a code built for a different one. It is never smaller than the entropy, because the optimal code for P is optimal.
  
  $H (P, Q) = - i \sum p_{i} lo g_{2} q_{i} = 6.08388 bits$
4. The excess is the KL divergence, and it is a subtraction, not a separate computation: 6.08388 − 4.87254 = 1.21134, to the last printed digit.
  
  $D_{KL} (P ∥ Q) = H (P, Q) - H (P) = 6.08388 - 4.87254 = 1.21134$
5. Exponentiating that excess says what it costs in practical terms. 2^1.21 is 2.32, so using the wrong model is like facing 2.32 times as many choices per word — 29.29 becomes 67.8.
  
  $2^{D_{KL}} = 2^{1.21134} = 2.32 ⟹ 29.29 \to 67.8 effective choices$
Answer

The tool prints H = 4.87254, PP = 29.2942, H(P,Q) = 6.08388 and D_KL = 1.21134. The two identities — perplexity is 2^H, and KL is cross-entropy minus entropy — hold for any pair of distributions, so they are a check you can run on any model report you are handed. They also explain why language models are scored by cross-entropy rather than KL: KL needs the true distribution, which nobody has, while cross-entropy needs only the model's probability of the text that actually occurred. Minimising one minimises the other, because the difference is a constant the model cannot touch.

$D_{KL} = H (P, Q) - H (P) \geq 0 : the wrong model costs 2^{1.21} = 2.32 \times$

References (3)

Entropy and cross-entropy, at the source: C. E. Shannon, "A Mathematical Theory of Communication." Bell System Technical Journal 27, 379–423, 1948.
KL divergence, in the paper that defined it: S. Kullback & R. A. Leibler, "On Information and Sufficiency." The Annals of Mathematical Statistics 22, 79–86, 1951.
Perplexity, introduced as a measure of task difficulty long before language models: F. Jelinek, R. L. Mercer, L. R. Bahl & J. K. Baker, "Perplexity — a measure of the difficulty of speech recognition tasks." Journal of the Acoustical Society of America 62, S63, 1977.

By the 42Math team · Updated 2026-05-09

Reviewed against published sources. Corrections welcome. Editorial policy · Corrections

All Computing lessons →

Example problems

Vulcan Greeting (Uniform) - Sixteen tokens over six types: entropy 2.50 bits and perplexity 5.66, the lowest here despite the name. Its comparison text is identical, so the KL divergence is exactly 0.
Trump Rhetoric (Repetitive) - Twenty-eight tokens over fifteen types gives entropy 3.59 and perplexity 12.08 - higher than Uniform words, because repetitive here means repeated phrases rather than a small vocabulary.
Star Trek Monologue (Natural) - Thirty-eight tokens over thirty-two types: entropy 4.87 bits and perplexity 29.29, the highest on the tool. This is the worked problem's state.
Sci-Fi vs Politics (Domain Shift) - Entropy 4.01 and perplexity 16.13, with a comparison text from a different domain. That mismatch is what lifts cross-entropy above the entropy, and the gap between them is the KL.