BPE Tokenizer Explorer

A language model never sees your letters. It sees tokens, chosen by a vocabulary fixed long before you typed. Paste text and watch that vocabulary cut it up.

Worked example — BPE Tokenizer Explorer

Tokens	13
Characters	68
Characters per token	5.23
Distinct tokens used	13
Merges applied	3,000

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

The vocabulary is somebody else's, and it is already finished 🖖

The merge list this page uses was built before you arrived, from this site's own English prose, and nothing you type changes it. That is not a shortcut — it is what a tokenizer is. The vocabulary is settled before a model's training even begins, and it cannot grow to fit you: a word that corpus never contained has no entry of its own, however ordinary it is to you. Type a name, a technical term, or a word in another language and watch it arrive in pieces.

The space is part of the token 🖖

A token that begins a word carries the space in front of it, shown here as ␣. So "the" at the start of a line and " the" in the middle of one are two different entries in the vocabulary, and a model has to learn both. This is why re-wrapping a paragraph can change its token count without changing a single letter, and why pasted text with double spaces or indentation costs more than it looks.

The model cannot see the letters inside a token 🖖

A token reaches the model as a single number. "strawberry" becomes ▁stra · w · ber · ry here — four numbers, and the word's three r's land in three different ones. Asking how many r's it contains asks the model to look inside pieces it only ever receives whole. It can often answer anyway, from having read the answer somewhere. That is recall, not counting, and it is why the failure looks so strange when it happens.

Problem solved in full

Compression when sixty-eight characters become thirteen tokens 5 steps
Sixty-eight characters become thirteen tokens. Work out the compression, and then what those five characters per token are actually worth to a model.
1. The two counts are the whole measurement: characters in, tokens out.
  
  $68 characters ⟶ 13 tokens$
2. Their ratio is a little over five, which is typical for English under byte-pair encoding — common words survive as single tokens and rare ones fragment.
  
  $\frac{68}{13} = 5.23 characters per token$
3. The comparison that matters is against the alternative. A byte-level model reads the same text as 68 tokens, more than five times the sequence length.
  
  $a byte-level model would emit 68 tokens for the same text$
4. And the attention matrix holds n² entries, so a factor of 5.23 in length is a factor of 27 in that part of the work. The rest of the network is linear in n and falls by 5.23, and which of the two dominates depends on the context length: attention only takes over past a few thousand tokens.
  
  $cost \propto (tokens)^{2} for attention ⟹ (\frac{68}{13})^{2} = 27 \times$
5. All thirteen tokens here are distinct, which tells you the text is short and varied — there is no repetition for the merges to exploit beyond what the vocabulary already encodes.
  
  $13 distinct of 13 ⟹ no token repeats here$
Answer

The tool prints 13 tokens for 68 characters, 5.23 characters per token. That compression is why tokenization exists at all and why it is not a detail: at a fixed context length it is the difference between five pages of text and one. It is also why the same model is cheaper in English than in languages the vocabulary was not built for — the merges were learned on a corpus, so a script under-represented there fragments toward the byte level and pays in both terms, linearly everywhere and quadratically in attention. Paste in some text in another language and watch the ratio fall.

$5.23 chars/token : a 5 \times shorter sequence, and 27 \times less attention work$

Learning path

How an LLM picks the next word

Leads to token-embeddings the vocabulary — the fixed set of entries anything downstream can attach a probability to.

References (2)

Byte-pair encoding applied to text, which is where subword tokenisation came from: R. Sennrich, B. Haddow and A. Birch, "Neural Machine Translation of Rare Words with Subword Units." Proceedings of the 54th Annual Meeting of the ACL (Volume 1: Long Papers), 1715–1725, 2016.
The language-independent scheme most current models actually ship: T. Kudo and J. Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing." Proceedings of EMNLP 2018: System Demonstrations, 66–71, 2018.

By the 42Math team · Updated 2026-08-09

Reviewed against published sources. Corrections welcome. Editorial policy · Corrections

All Computing lessons →

Example problems

English prose - 68 characters become 13 tokens. Common words like "of" and "a" each survive whole, while "quickly" splits after "quick".
The same sentence, in German - 69 characters — one more than the English sentence — become 36 tokens. The vocabulary was trained on English, and German pays for it.
Rare and long words - 52 characters become 23 tokens. "strawberry" arrives as four pieces, so the model is never shown its three separate r's.
Digits and dates - 50 characters become 31 tokens. Digits are split into whatever pairs the corpus happened to contain, which is why arithmetic on long numbers is hard for a model.
English, small vocabulary - The same English sentence as the first sample, with 200 merges instead of 3000: 33 tokens instead of 13.