NLP Entropy Lab
Visualize token probability distributions, Shannon entropy, perplexity, cross-entropy, and KL divergence — the core mathematical metrics used to train and evaluate AI Large Language Models (LLMs).
Deep Math of Information Theory & AI LLMs 🖖
Shannon entropy, cross-entropy, perplexity, and KL divergence are the foundational mathematical metrics used to train and evaluate AI Large Language Models (LLMs) like GPT-4 and Llama. Minimizing cross-entropy is equivalent to training the model to predict the next token, while perplexity measures its evaluation performance, and KL divergence is used as a penalty in RLHF reinforcement learning alignment.
What entropy really measures: surprise 🖖
Entropy scores how unpredictable a text is — the average "surprise" carried by each token. Perplexity turns that into an intuitive count: a perplexity of 8 means the model is, on average, as uncertain as if it were choosing uniformly among 8 words. Compare a repetitive prompt with varied prose and watch perplexity climb — predictable text scores low, diverse text scores high.
Shannon measured English with a parlor game 🖖
In 1951 Claude Shannon estimated the entropy of English not with a computer but with people: subjects guessed the next letter of hidden text, and their success rate bounded English at roughly 0.6 to 1.3 bits per character. That number is still the yardstick — modern LLMs are judged by how close their per-character perplexity gets to this human-derived limit, seventy years later.
Example problems
- Vulcan Greeting (Uniform) - Uniform token frequencies produce higher entropy than repetitive prompts.
- Trump Rhetoric (Repetitive) - Repetition collapses token uncertainty and lowers perplexity.
- Star Trek Monologue (Natural) - Natural text tends to show Zipf-like rank-frequency shape.
- Sci-Fi vs Politics (Domain Shift) - Domain mismatch increases cross-entropy and KL divergence.