Removable and Irreducible: A Token-Cost Ledger for the Multilingual Tokenization Tax

# A Token-Cost Ledger for the Multilingual Tokenization Tax
Source: [https://arxiv.org/html/2609.00378](https://arxiv.org/html/2609.00378)
## Removable and Irreducible: A Token\-Cost Ledger for the Multilingual Tokenization Tax

###### Abstract

Large language models pay a well\-documented tax on non\-English text: the same content costs several times more tokens, and because attention is quadratic in sequence length, far more compute\. We ask how much of this tax is removable\. Framing the token layer as source coding – transformer compute is monotone in sequence length, whose per\-atom floor is the Shannon rateH/log2⁡VH/\\log\_\{2\}V, an object already applied to tokenizers in prior work – we assemble a token\-cost ledger that splits each language’s cost, at fixed parallel content, into a removable coding redundancy, a residual coding slack, an intrinsic\-content term, and an orthogonal, irreducible grapheme\-to\-phoneme term that governs the multimodal rather than the text cost\. On FLORES\-200 across eight languages, a production tokenizer costs up to8\.9×8\.9\\timesmore tokens for Indic scripts than for English; a script\-matched code trained on1,0121\{,\}012sentences removes a median64%64\\%of that excess \(bootstrap 95% CI\[0\.638,0\.647\]\[0\.638,0\.647\]\), and a script\-fair information floor shows the intrinsic content differs by under6%6\\%– the tax is representational, not informational\. A constructed code removes98%98\\%of a controlled source’s redundancy, and the token tax implies up to79×79\\timesattention cost\. We are explicit about scope and failure: this is compute\-and\-memory accounting, not a model\-quality claim; we neither measure nor claim the cross\-lingual direction of the orthographic term; and our matched code is a conservative small\-data demonstration\. We contribute the unifying ledger, the removable\-versus\-intrinsic attribution, and an open one\-command harness\.

## 1Introduction

It costs more to say the same thing to a language model in Telugu than in English\. On identical, professionally\-translated content, a widely\-used production tokenizer emits up to8\.9×8\.9\\timesas many tokens for an Indic script as for English \(Section[4](https://arxiv.org/html/2609.00378#S4)\); an older English\-centric tokenizer reaches1616–20×20\\timesper word\. Because self\-attention isΘ​\(N2\)\\Theta\(N^\{2\}\)in sequence lengthNN\(Vaswaniet al\.,[2017](https://arxiv.org/html/2609.00378#bib.bib20)\), ak×k\\timestoken inflation is ak2×k^\{2\}\\timesattention\-compute inflation and ak×k\\timesshrink of the effective context window\. This multilingual “token tax” is well documented as a phenomenon and a fairness problem\(Ahiaet al\.,[2023](https://arxiv.org/html/2609.00378#bib.bib10); Petrovet al\.,[2023](https://arxiv.org/html/2609.00378#bib.bib11); Lundin and others,[2025](https://arxiv.org/html/2609.00378#bib.bib12)\)\.

The question this paper asks is narrower and, we think, more useful:*how much of the tax is removable by choosing a better code, and how much is intrinsic to the language?*We answer it by assembling a*token\-cost ledger*\. Holding semantic content fixed with a parallel corpus, we decompose each language’s normalized sequence length into \(i\) a*removable*coding redundancy – the penalty for using a code fit to the wrong \(English\-dominated\) distribution; \(ii\) a residual coding slack from a finite, finitely\-trained vocabulary; \(iii\) an*intrinsic*\-content term measured by a script\-fair compressor; and, on an orthogonal axis, \(iv\) an irreducible grapheme\-to\-phoneme term that does not affect text sequence length at all but governs the multimodal \(speech\) ledger\. Terms \(i\)–\(iii\) live on the text\-compute axis; \(iv\) is the organic cost of a deep orthography\.

None of the ledger’s individual pieces is new\. The fertility floorF⋆=H/log2⁡VF^\{\\star\}=H/\\log\_\{2\}Vis Shannon source coding\(Shannon,[1948](https://arxiv.org/html/2609.00378#bib.bib2); Cover and Thomas,[2006](https://arxiv.org/html/2609.00378#bib.bib1)\), applied to tokenizers as an “efficiency” or “capacity\-utilization” metric byZouharet al\.\([2023](https://arxiv.org/html/2609.00378#bib.bib4)\)andErdoganet al\.\([2026](https://arxiv.org/html/2609.00378#bib.bib5)\), the latter of whom also give the additive floor\-plus\-redundancy split we use\. Orthographic depth as an information\-theoretic quantity is established\(Torres and Futrell,[2025](https://arxiv.org/html/2609.00378#bib.bib14)\)\. A compute\-optimal vocabulary size is derived byTaoet al\.\([2024](https://arxiv.org/html/2609.00378#bib.bib13)\)\. Our contribution is \(a\) to*unify*the removable text term and the irreducible orthographic term into one cost ledger; \(b\) to*attribute*the observed multilingual tax empirically to removable versus intrinsic components on real parallel data, with a constructed code that drives the removable term to zero; and \(c\) to keep the whole thing scoped strictly to*compute and memory*, which is what makes the accounting well\-posed\.

### Scope fence \(stated once, up front\)\.

This is a cost accounting, not a quality claim\. We do*not*assert that fewer tokens make a better model;Schmidtet al\.\([2024](https://arxiv.org/html/2609.00378#bib.bib9)\)and others show it need not, and we agree\. Everything here concerns FLOPs, KV\-cache bytes, and context\-window occupancy, quantities that are monotone in sequence length regardless of downstream accuracy\.

### Contributions\.

- •A unified token\-cost ledger\(Section[2](https://arxiv.org/html/2609.00378#S2)\) that puts the removable coding redundancy and the irreducible orthographic term on one accounting identity, with a single estimand – the removable fractionρ\\rho– for “how synthetic is this language’s tax?”
- •An empirical attribution\(Section[4](https://arxiv.org/html/2609.00378#S4)\) on FLORES\-200: a script\-matched code trained on1,0121\{,\}012sentences removes a medianρ=0\.64\\rho=0\.64of the production\-tokenizer tax \(CI\[0\.638,0\.647\]\[0\.638,0\.647\]\), and the script\-fair information floor shows intrinsic content varies by<6%<6\\%across Indic languages – the tax is representational\.
- •A constructed floor\-approaching codeand the quadratic\-cost consequence \(Section[5](https://arxiv.org/html/2609.00378#S5)\): a matched code removes98%98\\%of a controlled source’s redundancy, sitting0\.0360\.036bits above the entropy floor; the token tax implies up to79×79\\timesattention cost\.
- •An open, one\-command harnessand an honest limitations section \(Section[7](https://arxiv.org/html/2609.00378#S7)\), including a pre\-registered control that we report as*not*established\.

### Cost is monotone in sequence length\.

A decoder\-only transformer of widthddandLLlayers processingNNtokens paysΘ​\(L​N2​d\)\\Theta\(LN^\{2\}d\)attention FLOPs,Θ​\(L​N​d2\)\\Theta\(LNd^\{2\}\)feed\-forward FLOPs,Θ​\(L​N​d\)\\Theta\(LNd\)KV\-cache memory, and anO​\(V​d\)O\(Vd\)vocabulary term\(Vaswaniet al\.,[2017](https://arxiv.org/html/2609.00378#bib.bib20)\)\.

###### Proposition 1\(cost monotonicity\)\.

For fixed\(d,L,V\)\(d,L,V\), transformer compute and KV memory for a fixed piece of content are non\-decreasing inNN, and strictly increasing onceN≳dN\\gtrsim d\(the attention term dominates\)\.

Every term is non\-decreasing inNN; theN2N^\{2\}term’s derivative2​L​N​d2LNdovertakes the linear terms onceN\>O​\(d\)N\>O\(d\)\. The consequence is the objective:*minimize expectedNNfor fixed content*\. In the quadratic regime a sequence\-length ratiorris an attention\-cost ratior2r^\{2\}\.

### The floor \(prior art\)\.

Model content as atoms drawn from a sourceppwith entropyH​\(p\)H\(p\)bits/atom, encoded to tokens over aVV\-ary alphabet\.

###### Lemma 1\(fertility floor;Cover and Thomas,[2006](https://arxiv.org/html/2609.00378#bib.bib1); Zouharet al\.,[2023](https://arxiv.org/html/2609.00378#bib.bib4); Erdoganet al\.,[2026](https://arxiv.org/html/2609.00378#bib.bib5)\)\.

Any uniquely\-decodableVV\-ary code has expected tokens per atomF≥H\(p\)/log2V=:F⋆F\\geq H\(p\)/\\log\_\{2\}V=:F^\{\\star\}, with equality approached within one token by a matched Huffman code and exactly in the block limit\. If the code is optimal for a wrong distributionqq, thenF=F⋆\+RF=F^\{\\star\}\+RwithR=DKL​\(p∥q\)/log2⁡V≥0R=D\_\{\\mathrm\{KL\}\}\(p\\,\\\|\\,q\)/\\log\_\{2\}V\\geq 0\.

We state Lemma[1](https://arxiv.org/html/2609.00378#Thmlemma1)for self\-containedness and*attribute*it;η=H/log2⁡V\\eta=H/\\log\_\{2\}Vis[Erdoganet al\.](https://arxiv.org/html/2609.00378#bib.bib5)’s capacity utilization, and theF⋆\+RF^\{\\star\}\+Rsplit is their Appendix\-C decomposition\.RRis the*removable*part: refit the code toppandR→0R\\to 0\.

### The ledger \(our synthesis\)\.

Hold content fixed with a parallel corpus and normalize every quantity to a reference language \(English=1=1\)\. WriteNSLE​\(L\)\\mathrm\{NSL\}\_\{E\}\(L\)for the total tokens of languageLLunder encoderEEdivided by English’s under the same encoder\. Then the excess of a production encoder decomposes additively into individually measurable terms that are non\-negative for every language whose tax we attribute:111The identity telescopes exactly for all languages; the residual\-slack termNSLmatch−NSLfloor\\mathrm\{NSL\}\_\{\\mathrm\{match\}\}\-\\mathrm\{NSL\}\_\{\\mathrm\{floor\}\}can nonetheless turn marginally negative for a shallow\-orthography Latin control whose intrinsic content already sits near the English baseline\. German is the one such case here \(NSLmatch=1\.13<NSLfloor=1\.15\\mathrm\{NSL\}\_\{\\mathrm\{match\}\}=1\.13<\\mathrm\{NSL\}\_\{\\mathrm\{floor\}\}=1\.15\): the small matched BPE and the LZMA content proxy are distinct instruments, each normalized to English under itself, and agree to within noise this close to11\. For all five Indic languages—our focus—every term is strictly positive \(Table[1](https://arxiv.org/html/2609.00378#S4.T1)\)\.

NSLprod​\(L\)−1⏟observed tax=\[NSLprod−NSLmatch\]⏟removable​R​\(vocab mismatch\)\+\[NSLmatch−NSLfloor\]⏟residual coding slack\+\[NSLfloor−1\]⏟intrinsic content,\\underbrace\{\\mathrm\{NSL\}\_\{\\mathrm\{prod\}\}\(L\)\-1\}\_\{\\text\{observed tax\}\}=\\underbrace\{\[\\mathrm\{NSL\}\_\{\\mathrm\{prod\}\}\-\\mathrm\{NSL\}\_\{\\mathrm\{match\}\}\]\}\_\{\\text\{removable \}R\\text\{ \(vocab mismatch\)\}\}\+\\underbrace\{\[\\mathrm\{NSL\}\_\{\\mathrm\{match\}\}\-\\mathrm\{NSL\}\_\{\\mathrm\{floor\}\}\]\}\_\{\\text\{residual coding slack\}\}\+\\underbrace\{\[\\mathrm\{NSL\}\_\{\\mathrm\{floor\}\}\-1\]\}\_\{\\text\{intrinsic content\}\},\(1\)and, on an orthogonal axis, an irreducible termG​\(L\)=H​\(phoneme∣grapheme\)G\(L\)=H\(\\text\{phoneme\}\\mid\\text\{grapheme\}\)that leaves textNNuntouched but governs the speech ledger\. The unification of the removable text term with the irreducible orthographic term is the contribution; neither half is ours \(the removable term isErdoganet al\.,[2026](https://arxiv.org/html/2609.00378#bib.bib5); orthographic depth as algorithmic mutual compressibility isTorres and Futrell,[2025](https://arxiv.org/html/2609.00378#bib.bib14), whose Kolmogorov instrument we replace with a Shannon one in the LLM\-cost setting\)\. The headline estimand is the*removable fraction*

ρ​\(L\)=1−NSLmatch​\(L\)−1NSLprod​\(L\)−1,\\rho\(L\)=1\-\\frac\{\\mathrm\{NSL\}\_\{\\mathrm\{match\}\}\(L\)\-1\}\{\\mathrm\{NSL\}\_\{\\mathrm\{prod\}\}\(L\)\-1\},\(2\)the share of the production tax a language\-matched code removes:ρ→1\\rho\\\!\\to\\\!1means the tax is a code artifact \(synthetic\),ρ→0\\rho\\\!\\to\\\!0means it is intrinsic \(organic\)\.

### A one\-line remark on vocabulary\.

SubstitutingN=A​H/log2⁡VN=A\\,H/\\log\_\{2\}Vinto the cost model gives aCost​\(V\)\\mathrm\{Cost\}\(V\)with an interior minimizerV⋆V^\{\\star\}balancing shrinking sequence terms against a growing vocab term – but this compute\-optimal vocabulary is already established, empirically, byTaoet al\.\([2024](https://arxiv.org/html/2609.00378#bib.bib13)\)andLimisiewiczet al\.\([2026](https://arxiv.org/html/2609.00378#bib.bib6)\); we cite it and claim nothing here\.

## 3Experimental setup

Real parallel data\.FLORES\-200\(NLLB Teamet al\.,[2022](https://arxiv.org/html/2609.00378#bib.bib19)\)\(CC\-BY\-SA\), professionally translated and sentence\-aligned, so lineiiof every language file is the same sentence\. We report on eight languages: a Latin control \(English, Spanish, German\), Devanagari \(Hindi\), Bengali, and three Dravidian abugidas \(Telugu, Tamil, Kannada\)\. We measure on the1,0121\{,\}012\-sentencedevtestsplit\.Encoders\.UTF\-8 bytes; Unicode extended grapheme clusters \(\\X, the akshara\-preserving atom\); three production byte\-level BPE tokenizers \(GPT\-2,cl100k\_base,o200k\_base\); and a per\-language*matched*BPE trained*held out*on the FLORESdevsplit and evaluated ondevtest\.Information floor\.A script\-fair content estimate: LZMA over the grapheme\-cluster\-ID stream \(not UTF\-8 bytes\), so a script is not charged for its 3\-byte\-per\-codepoint UTF\-8 assignment\.Constructed source\.A Zipf\(s=1\.1s\{=\}1\.1\) source over256256concepts with a known entropy\.Apparatus\.Single\-thread Python;tiktokenand the standalonetokenizerslibrary \(no GPU, no model weights\); deterministic seeds; one command regenerates every number and figure\.

### Pre\-registration\.

Hypotheses, decision rules, and four negative controls were frozen before the confirmatory run\. The calibration control \(NC2\) requires the stack to recover a*known*entropy: on a uniform\-32 source it returnsH^=5\.0000\\hat\{H\}=5\.0000bits and a Huffman length in\[H,H\+1\)\[H,H\{\+\}1\); it passed before any language number was read\.

## 4Results: the removable tax

Table[1](https://arxiv.org/html/2609.00378#S4.T1)and Figure[1](https://arxiv.org/html/2609.00378#S4.F1)report the ledger\. Undercl100k\_basethe production tax reaches8\.29×8\.29\\times\(Telugu\),8\.86×8\.86\\times\(Kannada\), and4\.76×4\.76\\times\(Hindi\); under the older GPT\-2 tokenizer, fertility per word reaches16\.416\.4–20\.3×20\.3\\timesfor Dravidian scripts\. A*script\-matched*code trained on only1,0121\{,\}012sentences \(learned vocabulary2,0002\{,\}000–3,0003\{,\}000\) brings these to2\.892\.89,2\.952\.95, and2\.54×2\.54\\times; and the script\-fair information floor sits at1\.021\.02–1\.06×1\.06\\times– the intrinsic content of the same sentences is within6%6\\%across all Indic languages \(this is our pre\-registered content\-invariance control, NC3: no language exceeded the1\.5×1\.5\\timesthreshold, so no part of the tax is re\-attributed to intrinsic content\)\. Bootstrapping sentences, the median removable fraction across the five Indic languages isρ=0\.642\\rho=0\.642with a95%95\\%CI of\[0\.638,0\.647\]\[0\.638,0\.647\]:a matched code removes about two\-thirds of the production tax, and the information floor shows nearly all of the rest is coding slack rather than content\. That production tokenizers themselves disagree by4×4\\timeson the same Telugu content \(8\.29×8\.29\\timesundercl100kvs\.1\.93×1\.93\\timesunder the more multilingualo200k\) is independent evidence that the tax is a property of the code, not the language\.

Table 1:The token\-cost ledger on FLORES\-200devtest\(1,0121\{,\}012parallel sentences\), production tokenizercl100k\_base\.NSL\\mathrm\{NSL\}= normalized sequence length \(English=1=1\)\.NSLfloor\\mathrm\{NSL\}\_\{\\mathrm\{floor\}\}is the script\-fair LZMA content ratio\.ρ\\rhois the removable fraction \(Eq\.[2](https://arxiv.org/html/2609.00378#S2.E2)\); the matched code is trained held\-out on∼\\sim1k sentences \(a conservative demonstration\)\.![Refer to caption](https://arxiv.org/html/2609.00378v1/fig1_token_tax.png)Figure 1:The multilingual token tax and how much of it is removable\. Production BPE \(red\) vs\. a script\-matched code \(orange\) vs\. the script\-fair information floor \(blue\), normalized to English\.### Bytes\-per\-char predicts the tax \(H2, a replication\)\.

Across the eight languages, UTF\-8 bytes\-per\-character correlates with the production tax at Spearmanρs=0\.83\\rho\_\{s\}=0\.83\(p=0\.01p=0\.01\), replicatingAhiaet al\.\([2023](https://arxiv.org/html/2609.00378#bib.bib10)\)andPetrovet al\.\([2023](https://arxiv.org/html/2609.00378#bib.bib11)\); our addition is the decomposition, not the correlation\.

## 5Decomposition, a constructed code, and quadratic cost

Figure[2](https://arxiv.org/html/2609.00378#S5.F2)decomposes the Indic excess of Eq\.[1](https://arxiv.org/html/2609.00378#S2.E1)\. The removable term \(vocabulary mismatch\) dominates; the intrinsic\-content sliver is≤0\.06\\leq 0\.06in every case\. The residual coding slack – the gap between our small matched code and the floor – is itself removable in principle \(a better\-trained code closes it\), so it is a lower bound on removability, not a second intrinsic term; that a200200k\-vocabulary production tokenizer \(o200k\) already reaches1\.93×1\.93\\timeson Telugu confirms the slack is training, not content\.

![Refer to caption](https://arxiv.org/html/2609.00378v1/fig2_decomposition.png)Figure 2:Ledger decomposition of the Indic token tax \(Eq\.[1](https://arxiv.org/html/2609.00378#S2.E1)\): removable coding redundancy \(red\) dominates; residual coding slack \(orange\) is removable in principle; intrinsic content \(blue\) is a sliver\.### A constructed code hits the floor\.

On the controlled Zipf source \(entropyH=5\.77H=5\.77bits/concept\), a mismatched fixed\-width code pays8\.08\.0bits/concept; the matched Huffman code\(Huffman,[1952](https://arxiv.org/html/2609.00378#bib.bib3)\)\(“Silicon Vernacular”\) pays5\.805\.80–0\.0360\.036bits above the floor, within Lemma[1](https://arxiv.org/html/2609.00378#Thmlemma1)’s one\-bit guarantee – removing98%98\\%of the redundancy \(Figure[3](https://arxiv.org/html/2609.00378#S5.F3), left\)\. Our pre\-registered no\-sub\-floor control \(NC1\) requires that no code we report encodes belowHH, and the Kraft sum of every code we build satisfies∑2−ℓi≤1\\sum 2^\{\-\\ell\_\{i\}\}\\leq 1\. We therefore deliberately do*not*report an empirical block code beating the floor: at block sizes\>1\>1thekk\-gram alphabet is undersampled and an in\-sample Huffman would appear to beatHHfrom finite\-sample bias – a dishonest number\. The residual sub\-bit gap is closed by block/arithmetic coding as a theorem, not a measurement\. We position this construction as the discrete\-code limit of byte\-entropy patching\(Pagnoniet al\.,[2024](https://arxiv.org/html/2609.00378#bib.bib15)\), not as a proposed human language\.

### Quadratic amplification\.

Because attention isΘ​\(N2\)\\Theta\(N^\{2\}\), the token tax is amplified in compute: the same content costs up to79×79\\times\(Kannada\) and69×69\\times\(Telugu\) the attention work of English undercl100k, collapsing to88–11×11\\timesunder the matched code \(Figure[3](https://arxiv.org/html/2609.00378#S5.F3), right\)\.

![Refer to caption](https://arxiv.org/html/2609.00378v1/fig3_constructed_compute.png)Figure 3:Left: a constructed matched code removes98%98\\%of a controlled source’s redundancy, landing0\.0360\.036bits above the entropy floor\. Right: theΘ​\(N2\)\\Theta\(N^\{2\}\)attention\-cost proxy \(English=1=1\), production vs\. matched – the token tax is quadratically amplified in compute\.

## 6Related work

Information theory of tokenizers\.Zouharet al\.\([2023](https://arxiv.org/html/2609.00378#bib.bib4)\)frame the tokenizer as a channel and define an entropy\-to\-max\-entropy efficiency \(the parent ofH/log2⁡VH/\\log\_\{2\}V\);Erdoganet al\.\([2026](https://arxiv.org/html/2609.00378#bib.bib5)\)define capacity utilizationη=H1/log2⁡K\\eta=H\_\{1\}/\\log\_\{2\}Kand the additive floor\-plus\-redundancy decomposition we build on;Rajaramanet al\.\([2024](https://arxiv.org/html/2609.00378#bib.bib7)\)andGastaldiet al\.\([2025](https://arxiv.org/html/2609.00378#bib.bib8)\)give complementary theories of tokenization\. We contribute neither the floor nor the split – we contribute their*use*as a multilingual accounting\.The multilingual tax\.Ahiaet al\.\([2023](https://arxiv.org/html/2609.00378#bib.bib10)\),Petrovet al\.\([2023](https://arxiv.org/html/2609.00378#bib.bib11)\), andLundin and others \([2025](https://arxiv.org/html/2609.00378#bib.bib12)\)document the cross\-language cost disparity; Indic\-specific tokenizers\(Ranaet al\.,[2025](https://arxiv.org/html/2609.00378#bib.bib18)\)reduce fertility\. We explain the disparity as a removable KL redundancy above a script\-fair floor – and note that[Petrovet al\.](https://arxiv.org/html/2609.00378#bib.bib11)’s residual byte\-level disparity is exactly the intrinsic \+ orthographic remainder our ledger predicts\.Byte\-level models\.MEGABYTE\(Yuet al\.,[2023](https://arxiv.org/html/2609.00378#bib.bib16)\)and the Byte Latent Transformer\(Pagnoniet al\.,[2024](https://arxiv.org/html/2609.00378#bib.bib15)\)remove the discrete code and let entropy set patch boundaries; our ledger*explains*what they can remove \(the redundancyRR\) and cannot \(intrinsic content, orthographicGG\)\.Compression is not quality\.Schmidtet al\.\([2024](https://arxiv.org/html/2609.00378#bib.bib9)\)andBostrom and Durrett \([2020](https://arxiv.org/html/2609.00378#bib.bib17)\)show fewer tokens need not improve models; our scope fence makes this orthogonal – we account for cost, not quality\.Orthographic depth\.Torres and Futrell \([2025](https://arxiv.org/html/2609.00378#bib.bib14)\)formalize transparency as Kolmogorov mutual compressibility; we borrow the quantity \(as ShannonH​\(phoneme∣grapheme\)H\(\\text\{phoneme\}\\mid\\text\{grapheme\}\)\) for the irreducible axis\.Optimal vocabulary\.Taoet al\.\([2024](https://arxiv.org/html/2609.00378#bib.bib13)\)andLimisiewiczet al\.\([2026](https://arxiv.org/html/2609.00378#bib.bib6)\)derive the compute\-optimal vocabulary/granularity we merely cite\.

## 7Limitations and honest negatives

\(1\)We did not establish the orthographic direction\.We pre\-registered a control \(NC4\) that English’s grapheme\-to\-phoneme ambiguity exceeds shallow Indic scripts’\. We measure only the English side \(CMUdict homograph entropy0\.0700\.070bits/type\); we have no Indic pronunciation lexicon, so the cross\-lingual direction is literature\-consistent but*not*established by our instrument\. Per the pre\-registration we report this as a negative and future work, not a result\. \(2\)The matched code is a small\-data demonstration, trained on∼\\sim1k sentences \(vocabulary22–33k\); it understates removability, makingρ=0\.64\\rho=0\.64a lower bound\. \(3\)The information floor is an LZMA estimate, an upper bound on intrinsic content; a tighter estimator would shrink the intrinsic term further, again in the direction of “more removable\.” \(4\)This is not a quality claim\.We measure compute and memory; we make no statement about accuracy or loss, and explicitly do not contradictSchmidtet al\.\([2024](https://arxiv.org/html/2609.00378#bib.bib9)\)\. \(5\)Scope: eight languages, one parallel benchmark, text only; speech/multimodal cost is argued, not measured\. \(6\) We prove no new theorem: the floor and its redundancy split are cited, not claimed\.

## 8Conclusion

The multilingual tokenization tax is, in the compute ledger, mostly*removable*: on real parallel text a script\-matched code trained on a thousand sentences erases about two\-thirds of it, and a script\-fair floor shows the intrinsic content of the same sentences differs by under six percent\. What remains – the organic grapheme\-to\-phoneme cost of a deep orthography – is real but lives on a different \(multimodal\) axis, and flips the ranking\. We contribute the unifying ledger, the removable\-versus\-intrinsic attribution, a constructed code that reaches the entropy floor, and an open harness; we claim neither the floor, nor the optimal vocabulary, nor a model\-quality benefit\. The larger program these results open –*design principles of a near\-optimal language for foundational models*, spanning grammar and attention routing, acoustic isomorphism, and human learnability – we name as future work and a thesis, not a claim of this paper\.

### Reproducibility\.

## References

- O\. Ahia, S\. Kumar, H\. Gonen, J\. Kasai, D\. R\. Mortensen, N\. A\. Smith, and Y\. Tsvetkov \(2023\)Do all languages cost the same? tokenization in the era of commercial language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p1.8),[§4](https://arxiv.org/html/2609.00378#S4.SS0.SSS0.Px1.p1.2),[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- K\. Bostrom and G\. Durrett \(2020\)Byte pair encoding is suboptimal for language model pretraining\.InFindings of the Association for Computational Linguistics: EMNLP 2020,Note:arXiv:2004\.03720Cited by:[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- T\. M\. Cover and J\. A\. Thomas \(2006\)Elements of information theory\.2nd edition,Wiley\-Interscience\.Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p3.1),[Lemma 1](https://arxiv.org/html/2609.00378#Thmlemma1)\.
- M\. Erdogan, A\. Gorle, S\. Chandak, M\. Pilanci, and T\. Weissman \(2026\)An information\-theoretic perspective on llm tokenizers\.arXiv preprint arXiv:2601\.09039\.Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p3.1),[§2](https://arxiv.org/html/2609.00378#S2.SS0.SSS0.Px2.p2.5),[§2](https://arxiv.org/html/2609.00378#S2.SS0.SSS0.Px3.p1.6),[§6](https://arxiv.org/html/2609.00378#S6.p1.5),[Lemma 1](https://arxiv.org/html/2609.00378#Thmlemma1)\.
- J\. L\. Gastaldi, J\. Terilla, L\. Malagutti, B\. DuSell, T\. Vieira, and R\. Cotterell \(2025\)The foundations of tokenization: statistical and computational concerns\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2407\.11606Cited by:[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- D\. A\. Huffman \(1952\)A method for the construction of minimum\-redundancy codes\.Proceedings of the IRE40\(9\),pp\. 1098–1101\.Cited by:[§5](https://arxiv.org/html/2609.00378#S5.SS0.SSS0.Px1.p1.10)\.
- T\. Limisiewicz, A\. Pagnoni, S\. Iyer,et al\.\(2026\)Compute optimal tokenization\.arXiv preprint arXiv:2605\.01188\.Cited by:[§2](https://arxiv.org/html/2609.00378#S2.SS0.SSS0.Px4.p1.3),[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- J\. Lundinet al\.\(2025\)The token tax: systematic bias in multilingual tokenization\.arXiv preprint arXiv:2509\.05486\.Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p1.8),[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- NLLB Team, M\. R\. Costa\-jussà,et al\.\(2022\)No language left behind: scaling human\-centered machine translation\.arXiv preprint arXiv:2207\.04672\.Note:FLORES\-200 evaluation benchmarkCited by:[§3](https://arxiv.org/html/2609.00378#S3.p1.4)\.
- A\. Pagnoni, R\. Pasunuru, P\. Rodriguez,et al\.\(2024\)Byte latent transformer: patches scale better than tokens\.arXiv preprint arXiv:2412\.09871\.Cited by:[§5](https://arxiv.org/html/2609.00378#S5.SS0.SSS0.Px1.p1.10),[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- A\. Petrov, E\. La Malfa, P\. H\.S\. Torr, and A\. Bibi \(2023\)Language model tokenizers introduce unfairness between languages\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2305\.15425Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p1.8),[§4](https://arxiv.org/html/2609.00378#S4.SS0.SSS0.Px1.p1.2),[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- N\. Rajaraman, J\. Jiao, and K\. Ramchandran \(2024\)Toward a theory of tokenization in LLMs\.arXiv preprint arXiv:2404\.08335\.Cited by:[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- S\. Rana, A\. Menezes, A\. Kulkarni, C\. Khatri, and S\. Agarwal \(2025\)IndicSuperTokenizer: an optimized tokenizer for indic multilingual LLMs\.arXiv preprint arXiv:2511\.03237\.Cited by:[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- C\. W\. Schmidt, V\. Reddy, H\. Zhang, A\. Alameddine, O\. Uzan, Y\. Pinter, and C\. Tanner \(2024\)Tokenization is more than compression\.arXiv preprint arXiv:2402\.18376\.Cited by:[§1](https://arxiv.org/html/2609.00378#S1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2609.00378#S6.p1.5),[§7](https://arxiv.org/html/2609.00378#S7.p1.5)\.
- C\. E\. Shannon \(1948\)A mathematical theory of communication\.Bell System Technical Journal27,pp\. 379–423\.Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p3.1)\.
- C\. Tao, Q\. Liu, L\. Dou, N\. Muennighoff, Z\. Wan, P\. Luo, M\. Lin, and N\. Wong \(2024\)Scaling laws with vocabulary: larger models deserve larger vocabularies\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2407\.13623Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p3.1),[§2](https://arxiv.org/html/2609.00378#S2.SS0.SSS0.Px4.p1.3),[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- D\. Torres and R\. Futrell \(2025\)Clarifying orthography: orthographic transparency as compressibility\.arXiv preprint arXiv:2505\.13657\.Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p3.1),[§2](https://arxiv.org/html/2609.00378#S2.SS0.SSS0.Px3.p1.6),[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p1.8),[§2](https://arxiv.org/html/2609.00378#S2.SS0.SSS0.Px1.p1.7)\.
- L\. Yu, D\. Simig, C\. Flaherty, A\. Aghajanyan, L\. Zettlemoyer, and M\. Lewis \(2023\)MEGABYTE: predicting million\-byte sequences with multiscale transformers\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2609.00378#S6.p1.5)\.
- V\. Zouhar, C\. Meister, J\. L\. Gastaldi, L\. Du, M\. Sachan, and R\. Cotterell \(2023\)Tokenization and the noiseless channel\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2306\.16842Cited by:[§1](https://arxiv.org/html/2609.00378#S1.p3.1),[§6](https://arxiv.org/html/2609.00378#S6.p1.5),[Lemma 1](https://arxiv.org/html/2609.00378#Thmlemma1)\.