Half of written Japanese is 158 characters. The next 45% costs a thousand more.
100 kanji buys you 40.74% of a corpus. 872 buys 90%. The next five percentage points cost another 290.
The kanji frequency tool runs on a corpus of 709,707 kanji occurrences drawn from 60 literary works. Ask it how much of that text you can read for a given number of characters and it answers:
- the 100 most common kanji — 40.74% of all occurrences
- 158 kanji — 50.11%, half the text
- 872 kanji — 90.02%
- 1,162 kanji — 95.00%
- all 2,136 — 100%
Read that column downwards and the shape is unmistakable. The first hundred characters buy 40 percentage points. The next 772 buy 49. The 290 after that buy five. And the last 974 characters — nearly half the entire list — buy the final five per cent.
The same curve in every language
This is Zipf's law, and its persistence is the strange part. Rank the words of any large text by frequency and the frequency of the word at rank r falls roughly as 1/r: the second-commonest word appears about half as often as the first, the tenth about a tenth as often, the hundredth about a hundredth.
It holds for English, for Japanese kanji, for Latin, for languages with no written tradition until a linguist arrived with a recorder. It holds for programming languages. It holds, roughly, for city sizes and website visits and surnames.
What it does not have is an agreed explanation. Piantadosi's 2014 review is the honest survey: the pattern is beyond dispute and every proposed mechanism — optimisation of communication, random typing, preferential attachment, semantic structure — either fails to predict the details or predicts them while also predicting things that are false. It is one of the best-documented regularities in the study of language and nobody can tell you why it is there.
What it feels like from the inside
The curve has a psychological signature, and anyone who has studied a language has felt both ends of it.
The beginning is euphoric. A hundred characters — a fortnight's work — and 40% of the text stops being a wall. Progress feels like it is accelerating because each new character is genuinely common; you meet it again on the same page.
Then it flattens, brutally. The tool prints the complement as the "probability the next kanji is unknown", and that number is the better guide to how reading actually feels. At 100 known characters, one kanji in 1.7 stops you. At 872 it is one in 10. At 1,162, one in 20, which sounds like fluency and is not: at twenty characters a sentence, that is a stumble every sentence.
The last stretch is where learners quit, and the curve explains why without appealing to motivation. The final characters are rare by definition: you can study one and not meet it again for a week, which is exactly the condition under which memory fails. The effort per character does not rise. The reinforcement per character collapses.
Why machines care
The same lopsidedness decides how language models chop text up.
A vocabulary has to be finite, and a Zipf distribution means most of the probability mass sits on a handful of items with an endless tail of rare ones. Give every word its own token and the tail is unusable: millions of entries, each seen a handful of times, each impossible to learn. Give characters their own tokens and common words waste a dozen slots each.
Subword tokenisation is the compromise, and the tokenizer tool shows the result: common words survive as single tokens, rare ones shatter into fragments. That is the frequency curve made into a data structure. It is also why a model handles "the" perfectly and a rare surname badly, not because the surname is harder, but because it sits in the part of the distribution where there was never enough data.
And it sets the ceiling on compression. The entropy lab measures how surprising the next symbol is, and a skewed distribution is a low-entropy one: if a few characters dominate, you can encode them in fewer bits, which is exactly what a compressor exploits. Zipf's law is why text compresses well at all, and the coverage table above is the same fact counted in characters instead of bits.
One number worth carrying away: at 95% coverage, one kanji in twenty is still unknown. The curve is generous at the start and merciless at the end, and the two halves are the same law.
References (1)
- the law is solid; every explanation offered for it is contested Piantadosi (2014). Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review 21(5).