Why a language model struggles to count the letters in a word
The word "strawberry" reaches GPT-4 as three pieces — str, aw, berry. The letters are gone before it has read anything.
Ask a language model how many times the letter "r" appears in "strawberry" and it may well tell you two. This is treated as evidence of some deep deficiency. It is closer to asking someone to count the brushstrokes in a word they have only ever heard spoken.
What the model actually receives
Before any model sees text, the text is cut into tokens. Not letters, not words — chunks chosen by an algorithm that has looked at a large corpus and merged whichever adjacent pairs occurred most often, over and over, until it has a vocabulary of some tens of thousands of pieces.
Common words end up as a single token. Rarer ones get split, and where the splits land is determined by corpus statistics rather than by meaning or spelling. "strawberry" may arrive as two or three pieces, and the model's view of it is those pieces as atomic symbols, each an arbitrary index into a vocabulary, with no more internal structure available to the model than a house number has.
So the letters are genuinely not there. A model can learn facts about spelling — that certain tokens tend to co-occur with certain claims about letters — but it is reasoning from hearsay about its own input rather than from the input. You can watch the splitting happen in the Tokenizer. Its vocabulary came from this site's own prose rather than a web crawl, so it cuts strawberry into four pieces where GPT's cuts three.
What else this explains
Arithmetic on long numbers. Numbers are tokenised too, and not always digit by digit. A number may be split into chunks that cut across place value, so a model doing column arithmetic is working with pieces that do not line up with the columns. Performance that degrades sharply with digit count is a clue that the problem is representational.
Rhymes, anagrams, acrostics. Every task defined over letters rather than meaning runs into the same wall.
Costs and context limits in languages other than English. Tokenisers are trained on corpora that are overwhelmingly English, so English gets efficient tokens and other languages get fragmented into more pieces per word. The same sentence in a less-represented language can consume several times as many tokens, which means it costs more to process and consumes more of a fixed context window. That is an equity issue hiding inside a preprocessing step.
Tokens are not the only place the surprise lives
Meaning is geometric. Words become vectors, and closeness in that space is learned from co-occurrence: what appears near what. This is why models capture associations so well and why they inherit the biases of the text they were trained on: if a word occurs near certain other words in the corpus, it will sit near them in the space, and nothing in the mechanism distinguishes a fact from a stereotype. The Token Embeddings tool builds a small version of that space so you can see what the geometry is made of.
The output is sampled, not chosen. A model produces a probability distribution over the next token, and something has to pick one. Turn the temperature down and it takes the most likely option every time, becoming repetitive and predictable. Turn it up and it takes low-probability options more often, becoming inventive and unreliable. Much of what is described as a model being "creative" is a sampling parameter. Hallucination is not: a model at temperature zero still invents things, and the sampler only changes how often. Temperature Sampling makes the distribution and the pick visible side by side.
Which layer a behaviour comes from
It converts a mystery into a specification. "The model is unreliable" is not actionable. "The model cannot see letters, so route spelling and character-counting tasks to code" is. So is "this language costs three times as many tokens, so budget accordingly", and "this output was sampled at a temperature that will not give reproducible results".
Almost every practical decision about using these systems well comes from knowing which layer a behaviour comes from: the tokeniser, the embedding geometry, or the sampler. None of the three is intelligence, and all three are inspectable.
How the vocabulary gets built
The usual method is byte pair encoding, and it is simpler than its reputation. Start with every character as its own token. Count all adjacent pairs across the corpus, find the most frequent pair, and merge it into a single new token. Repeat some tens of thousands of times.
What emerges is not a linguistic analysis. It is a frequency ranking. Common English words survive whole because they were common in the corpus; morphology is captured only where it happens to coincide with frequency. The tokeniser has no notion that "running" contains "run"; it will represent it that way only if that split happened to win the counting.
The vocabulary is a fossil of one particular corpus at one particular moment. Change the training text and the splits move. Two models with different tokenisers do not merely disagree about output; they disagree about what the input is.
Why it is not simply fixed
The obvious response is to feed the model characters instead. Some models do, and the trade is not free.
Tokens exist to shorten the sequence. A page of English is perhaps 250 tokens or 1,200 characters, and the cost of the attention mechanism grows with the square of the sequence length, so working in characters costs roughly twenty times more compute for the same passage. Tokenisation is a compression step, and it is subject to exactly the trade-off in the compression article: you buy shorter sequences by discarding access to structure below the token.
So it is a deliberate exchange of character-level visibility for sequence length, made before the model sees anything, and its costs land precisely where character-level detail was the point.
A compression trick from 1994
The splitting scheme has an odd history. Byte pair encoding was published by Philip Gage in 1994 as a data compression algorithm, in a programming magazine, for squeezing files. Its rule was to find the commonest adjacent pair of bytes and replace it with a byte that is not in use, then repeat.
In 2016 Sennrich, Haddow and Birch borrowed it for machine translation, not to compress anything but to build a vocabulary that could represent rare words as pieces rather than giving up on them.
So a thirty-year-old compression trick, designed when the concern was disk space, decides what today's largest models are able to perceive.
Watch
References (2)
- byte pair encoding, as compression Gage (1994). A New Algorithm for Data Compression. The C Users Journal 12(2).
- and as a tokenizer Sennrich, Haddow & Birch (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016.