Tokens: the alphabet an LLM actually reads

The one component of a modern language model that never sees a gradient is a greedy compression heuristic from the 1990s, and it explains your Hindi bill exactly — your letter-counting failures only partly.

Understanding LLMs From the Inside — part 1 of 5.

Next: The forward pass: what actually happens to a token, and what it costs →

A 71% improvement that left Hindi 63% more expensive

On 4 November 2025 a writer at kathane.substack.com ran matched text through a commercial tokenizer in English and in Hindi and found Hindi still cost roughly 63% more, after a claimed 71% improvement in Hindi tokenization efficiency. Treat both figures as one person's measurement rather than a controlled result: the 71% has no baseline stated in the retrieved post, so we do not know which two vocabularies were compared. What survives is the shape of the finding. Someone made the tokenizer substantially better for Devanagari and the same paragraph still billed more than half again as much.

Bigger claims circulate. A post on tianpan.co dated 20 April 2026 puts the production penalty at 3–8× for CJK, Arabic and Devanagari scripts. That is a blog claim with no method in the snippet, and it disagrees with the 63% figure by roughly an order of magnitude. Both numbers can be honest, because they are almost certainly not measuring the same text through the same vocabulary. That disagreement is the first argument for measuring your own ratio instead of quoting anyone's.

The disparity itself is not new. Ahia and co-authors framed it in May 2023 in *Do All Languages Cost the Same?*, asking directly what users pay when the same information takes a different number of tokens in different languages. What is new is that between January and mid-May 2026 the premium acquired per-model numbers and an engineering target.

The component responsible is byte-pair encoding, introduced to NLP by Sennrich, Haddow and Birch in a paper submitted on 31 August 2015. Read that date as a live specification, not a historical footnote: BPE and its byte-level descendants are what today's production tokenizers are still built on, which is why the January 2026 work on Bangla, Hindi and Urdu premiums treats the problem as current engineering rather than archaeology. Which exact vocabulary each frontier model ships — sizes, merge rules, byte-fallback behaviour — is not established by the sources here, so I will not name specifics. It is the only major part of the stack that was fitted once, before training began, and then frozen for the model's life. This piece was assembled on 26 July 2026 from a research sweep that ran through mid-May; nothing here speaks to the last eight weeks.

Four words you need, and the one part of the model that never sees a gradient

A token is an integer index into a fixed table. Text arrives as bytes, a deterministic segmenter chops it into pieces that appear in that table, and each piece becomes a number. A token is not a word. It is not a character. It is whatever the segmenter's table happens to contain, which for English is often a word plus its leading space, and for rarer strings is two or five fragments.

The vocabulary is that table: a finite set of byte strings with their indices, its size chosen by a human before training and unchanged afterwards. The embedding is one learned vector per vocabulary entry, the model's only channel for knowing anything about a token beyond context. The geometry of those vectors is learned by gradient descent. The partition into tokens that produced them is not. The context window is the maximum number of tokens the model can attend to in a single forward pass, counted in tokens rather than characters or words, which is why the window you were sold and the window you actually get diverge by script.

The mechanism is small enough to hold in your head. Take a corpus, split it into bytes or characters, count every adjacent pair, merge the most frequent pair into a new symbol, recount, repeat a fixed number of times. Store the merges in order. At inference, replay them greedily. That is the whole algorithm. It descends from Philip Gage's 1994 data-compression method, as the subword survey literature notes, and Sennrich et al. adopted it for a specific problem: open-vocabulary translation, on the intuition that "various word classes are translatable via smaller units than words." The evidence offered was +1.1 BLEU on WMT15 English–German and +1.3 on English–Russian against a back-off dictionary baseline. It worked. Nothing in that objective rewards knowing how many rs a string contains, or which digit sits in the hundreds place.

So here is the claim I will defend, stated with its asymmetry rather than smoothed. For money, effective context window and latency, tokenization is a complete and arithmetically checkable explanation: token counts are measured, not inferred, and every downstream cost is linear in them. For letter-counting and arithmetic it is an established contributing mechanism whose sufficiency is unresolved. The most-cited evidence shows tokenization setting the difficulty of the task, not obviously imposing a ceiling on it, and anyone telling you it is purely tokenization or purely reasoning is ahead of the data. The v2 preprint *From Tokenizer Bias to Backbone Capability*, updated 6 March 2026 and accepted to KDD 2026 in Jeju (9–13 August 2026), argues in a narrower domain that those two things are separable and have to be separated by controlled study. That is the right instinct even where the study hasn't been run.

The cost claim is arithmetic, and you can check it in a browser tab

Do this before you believe anyone, including me. Take a real paragraph from your own traffic, not a sample sentence. Tokenize it with the actual tokenizer of the model you are billed on, in English and in your target language. Divide. You now have a number that belongs to your workload.

Then multiply it through, because it lands in three places at once. It scales the input line on your invoice directly. It also eats the window: a 200k-token context holds noticeably less Hindi content than English content, so your retrieval budget and your truncation point both move without anyone changing a config. And it shows up in latency, because prefill work scales with the number of input tokens and each decode step emits one token, so a 1.6× token ratio is a 1.6× tail on the generation phase. Why that is true mechanically belongs to Part 2, which owns the forward pass and the KV cache.

The 2026 literature has started putting per-model numbers on this. *Reducing Tokenization Premiums for Low-Resource Languages*, posted 19 January 2026, states that the penalty for Bangla, Hindi and Urdu "can easily be three to five times that of English" and enumerates the downstream costs. I could not open its per-model penalty table, so I am citing it as evidence that the field's attention moved rather than as a measurement to quote — the same caution applies to the January 2026 cross-tokenizer study *Is Sanskrit the most token-efficient language?*. Sebastian Pokutta's *The Hidden Cost of Tokenization*, 14 May 2026, makes the argument that matters most: the gap is a design choice, not a property of the languages.

Notice what makes this half of the thesis strong. No capability is being inferred. There is no benchmark to overfit, no held-out set to worry about, no distribution shift. You count tokens.

The better objection: the tokenizer is downstream of the data mixture

Blaming the tokenizer is convenient, and partly a dodge. BPE picks merges by frequency in a corpus. Point it at a corpus that is overwhelmingly English and it will spend its merge budget on English morphology by construction, not by prejudice. The vocabulary is a readout of the data mixture. Change the mixture and a large share of the premium moves with it, which means "the tokenizer is unfair to Hindi" quietly absolves the data-collection decision that made it so.

The 2026 work concedes this in its architecture. TREX, January 2026, treats tokenizer choice as a regression problem over the data mixture rather than as a fixed preprocessing step. Once you are optimising the tokenizer as a function of the mixture, you have accepted that it is not an independent component with its own bias to fix.

There is also a floor that no tokenizer design removes, and it is worth doing the arithmetic yourself because I am about to do it loosely. Devanagari codepoints live in U+0900–U+097F. In UTF-8 that range encodes to three bytes per codepoint, against one byte for ASCII. So a hypothetical perfect byte-level model, with no vocabulary at all, still reads roughly three times the bytes for the same number of characters. This is my arithmetic, not a cited result, and it needs two caveats: bytes-per-character is not bytes-per-meaning, since scripts differ in how much they pack into a character, and Hindi and English do not use the same number of characters to say the same thing. The defensible version is that the floor is nonzero and script-dependent, not that it is exactly 3×.

Going byte-level to escape the vocabulary has its own bill. A November 2025 preprint titled *UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8* makes the point in its title: a model that emits bytes can emit byte sequences that are not valid text. A BPE vocabulary structurally prevents that, because every unit in the table is a well-formed string. That is a real property you give up.

So the claim narrows, and should. BPE amplifies a Unicode floor by a factor set by the training mixture. The floor is physics plus a 1990s encoding decision. The amplification is the design choice, and it is the part that is fixable.

Counting: tokenization sets the difficulty, and whether it also sets a ceiling is unresolved

The strongest objection to my hedge is that my own best source doesn't hedge. *Counting Ability of Large Language Models and Impact of Tokenization*, October 2024, is titled and argued as tokenization-as-cause, and it reports that "even with CoT enabled, significant failures in counting are still observed in modern LLMs like GPT-4, which consistently make errors when counting letters in words as short as 3 to 10 characters such as Strawberry." A fair reader draws the obvious conclusion: if the failure survives generous inference-time compute on short strings, that is a ceiling, not a difficulty setting. I retrieved the paper's framing but not its full eval tables, so I quote nothing else from it.

Here is why I still think sufficiency is open. By 12 August 2025, Max Woolf was writing about the letter-counting meme in the past tense and re-running it with "blueberry" against then-current models to see whether they now passed. If models improved on this task while the vocabulary under them stayed in the same family, the vocabulary cannot be the whole ceiling. I did not verify that the vocabularies were unchanged across those releases, and that verification is exactly what a serious version of this argument requires.

Two tests you can run today. The substitution test, for any claim that counting is fixed: change the word. A model that handles strawberry and fails on an unfamiliar string of the same shape has memorised the famous example. That is overfitting to the benchmark everyone tweets, not a repaired mechanism.

The triage rule, for your own failures: if the behaviour changes when you change the surface form of the input — spacing the digits, hyphenating the word, switching script — you have a tokenizer problem and the fix lives upstream in preprocessing or vocabulary. If it changes when you give the model more inference-time compute or a tool call, it is a compute or harness problem and the tokenizer is a red herring. Most teams reach for prompt engineering on the first category and for tokenizer folklore on the second. The rule is diagnostic, not a proof of mechanism.

Part 4 owns why more thinking tokens help at all, and Part 3 owns what the next-token objective does and does not teach. What tokenization gives you here is a mechanism with no stake in the answer: nothing in a frequency-driven merge table rewards character-level or digit-level faithfulness.

The ablation that would falsify this, and where the tokenizer-free line stands as of July 2026

For cost: a matched-compute, matched-data comparison of a tokenizer-free model against a BPE model on a held-out multilingual eval, reporting bytes per parameter update and ill-formed-output rate next to quality. Without that, "tokenizer-free closes the multilingual gap" is a claim about where the cost was moved, not whether it was removed.

For counting: hold inference-time compute constant and generous, then evaluate on strings under deliberate distribution shift from the famous examples. Whatever failure survives is the ceiling. Nobody in these sources has published either experiment.

The tokenizer-free line is real but small. H-Net, released 10 July 2025, processes raw bytes with a vocabulary of 256. H-Net++, 7 August 2025, reports a 0.159 bits-per-byte improvement over a BPE-based GPT-2-fa baseline and +5.4 points on ParsGLUE — on a 1.4B-token Persian corpus, single language, with a 1.9M-parameter context mixer. That is a promising result at small scale, not a general one. A May 2026 preprint extends the dynamic-chunking idea further.

My verification debts, so you can price this piece correctly: Meta's Byte Latent Transformer, the o200k_base vocabulary specification, the arithmetic-tokenization literature and the glitch-token work were not retrieved, and no June–July 2026 sweep was run at all. If something settled either half of this in the last eight weeks, I don't know about it.

What I would still bet on: the segmenter that decides what your model can see was fitted by frequency counting in 2015, on somebody else's corpus, and has never once been re-derived from an objective. Go run your own ratio.

---

The series

1. Tokens: the alphabet an LLM actually reads (you are here)
2. The forward pass: what actually happens to a token, and what it costs
3. Pretraining: the law everyone quotes governs a number nobody optimizes anymore
4. Post-training: where behavior comes from
5. Making it yours: distillation and fine-tuning a domain model