Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter

# Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter
Source: [https://arxiv.org/html/2608.11361](https://arxiv.org/html/2608.11361)
## Lifecycle\-Optimal Tokenization: Vocabulary Size as a Deployment\-Regime\-Dependent Infrastructure Parameter

Ankit GubraniAffiliation:ankit\.gubrani90@gmail\.comSatyanarayana KakolluAffiliation:kakollu@icloud\.com

###### Abstract

Tokenizer vocabulary size is a foundational design choice in large language model \(LLM\) infrastructure, yet it is typically fixed at training time based on convention rather than deployment analysis\. We show that the cost\-optimal vocabulary is not a constant but a function of the serving regime\. We formalize total deployment cost asClifecycle​\(V\)=Ctrain​\(V\)\+λ⋅Cinfer​\(V,B\)C\_\{\\text\{lifecycle\}\}\(V\)=C\_\{\\text\{train\}\}\(V\)\+\\lambda\\cdot C\_\{\\text\{infer\}\}\(V,B\), whereλ\\lambdais inference volume andBBis the serving batch size\. Through controlled experiments on two GPU families spanning the memory\-bound to compute\-bound regimes \(A10G, ridge≈\\approx117 FLOP/byte; A100, ridge≈\\approx183 FLOP/byte\), we demonstrate: \(1\) the inference\-optimal vocabulary shifts 16×\\timeswith serving batch, from 32k atB=1B=1to 524k atB=64\+B=64\+, driven by amortization of theV×dV\\times dunembedding matrix read; \(2\) at 1\.3–2\.3B model scale, quality \(bits per byte, BPB\) is optimized atV=65V=65k, confirming scale\-dependent vocabulary preference; \(3\) the lifecycle\-optimal vocabulary diverges from training\-optimal by up to 16×\\timesfor production deployments\. Quality is approximately invariant across the optimal range \(<<2% BPB spread\), making vocabulary a pure systems optimization with no quality penalty in the measured range\. Our results provide actionable capacity planning guidance: on\-device deployments \(B=1B=1\) should useV≈32V\\approx 32k; datacenter serving \(B≥64B\\geq 64,λ≥10\\lambda\\geq 10\) should useV≈131V\\approx 131–262k\.

###### Index Terms:

tokenization, vocabulary optimization, LLM inference, capacity planning, roofline model, lifecycle cost, serving infrastructure

## IIntroduction

Every large language model deployed in production carries a vocabulary – a fixed set of subword tokens used to encode all input and output text\. Vocabulary sizeVVis set once at tokenizer training time, before any inference infrastructure is planned\. YetVVhas first\-order effects on both training throughput and inference cost: it determines the size of the unembedding matrixW∈ℝV×dW\\in\\mathbb\{R\}^\{V\\times d\}that must be read from high\-bandwidth memory \(HBM\) at every decode step\.

At each autoregressive step, the model produces add\-dimensional hidden vector and projects it throughWWto obtain logits over allVVtokens\. This projection dominates decode\-time memory traffic: for vocabularyV=262144V=262144and hidden dimensiond=512d=512in fp16, the weight matrix alone is262144×512×2=256262144\\times 512\\times 2=256MB read per step\. The cost of this read depends critically on the serving batch sizeBB\.

The key physical insight follows from roofline analysis\[[5](https://arxiv.org/html/2608.11361#bib.bib5)\]\. The unembedding matmul\[𝐡\]B×d⋅\[WT\]d×V\[\\mathbf\{h\}\]\_\{B\\times d\}\\cdot\[W^\{T\}\]\_\{d\\times V\}has arithmetic intensityI≈BI\\approx BFLOP/byte \(fp16\)\. AtB=1B=1\(on\-device\),I=1≪ρI=1\\ll\\rhowhereρ\\rhois the GPU’s ridge point, so the operation is memory\-bound: cost scales linearly withVVand the weight matrix is streamed at near\-peak bandwidth\. AtB=256B=256\(datacenter\),I=256\>ρI=256\>\\rho, the operation is compute\-bound: the sameV×dV\\times dread is amortized across 256 sequences, reducing per\-sequence cost by∼\\sim256×\\times\. The*optimal vocabulary therefore depends on the serving batch size*– a deployment parameter determined long after the tokenizer is frozen\.

This observation has direct capacity planning implications\. Infrastructure teams provisioning GPU clusters for LLM serving must account for how vocabulary choice affects memory bandwidth requirements, throughput per accelerator, and the number of accelerators needed to meet latency SLAs\. A model with a suboptimally small vocabulary wastes GPU cycles at datacenter batch; a model with a suboptimally large vocabulary degrades on\-device latency\. Neither is captured by existing vocabulary selection methods, which optimize for training quality\[[1](https://arxiv.org/html/2608.11361#bib.bib1)\]or single\-sample inference\[[2](https://arxiv.org/html/2608.11361#bib.bib2)\]in isolation\.

Current practice selectsVVfrom convention \(32k in Llama\-2/3\[[6](https://arxiv.org/html/2608.11361#bib.bib6)\], 100k\+ in GPT\-4\[[7](https://arxiv.org/html/2608.11361#bib.bib7)\], 256k in Gemma\-2\[[8](https://arxiv.org/html/2608.11361#bib.bib8)\]\) or from training\-time quality metrics\[[1](https://arxiv.org/html/2608.11361#bib.bib1)\]\. Neither accounts for how the cost of the unembedding step changes with the serving regime\.

### I\-AContributions

1. 1\.Alifecycle cost modelClifecycle​\(V,B,λ\)C\_\{\\text\{lifecycle\}\}\(V,B,\\lambda\)that jointly captures training and inference costs over vocabulary size, parameterized by serving batchBBand inference volumeλ\\lambda\. Both cost components share physical units \(ms/byte\), makingλ\\lambdaa dimensionless, physically interpretable ratio\.
2. 2\.Empirical measurementof the inference\-optimal vocabulary across batch sizesB∈\{1,16,64,256,1024,4096\}B\\in\\\{1,16,64,256,1024,4096\\\}on two GPU ridge points \(A10G≈\\approx117, A100≈\\approx183 FLOP/byte\), with kernel\-launch overhead eliminated via CUDA graph capture and isolated head measurement\. The 16×\\timesshift in optimalVVis confirmed overhead\-free\.
3. 3\.Scale\-dependent quality measurementat 1\.3–2\.3B parameters using Fully Sharded Data Parallel \(FSDP\) with bf16 precision, showing the quality\-optimal vocabulary shifts from 16k \(at 100M\) to 65k \(at 1\.5B\), consistent with Tao et al\.’sV∗∝N0\.5V^\{\*\}\\propto N^\{0\.5\}scaling law\[[1](https://arxiv.org/html/2608.11361#bib.bib1)\]\.
4. 4\.Alifecycle sweep table\(Table[VI](https://arxiv.org/html/2608.11361#S5.T6)\) andprovisioning recommendationsproviding direct guidance as a function ofλ\\lambdaandBB, suitable for infrastructure capacity planning decisions\.

## IIBackground and Related Work

### II\-APrior Work on Vocabulary Optimization

Tao et al\. \(NeurIPS 2024\)\[[1](https://arxiv.org/html/2608.11361#bib.bib1)\]derive that the loss\-optimal vocabulary scales asV∗∝N0\.5V^\{\*\}\\propto N^\{0\.5\}whereNNis model parameters\. Their analysis is FLOPs\-only at training time; no inference cost, batch dependence, or hardware regime is modeled\. We extend their quality insight with an inference cost model parameterized by serving batch\.

Length\-MAX \(2025\)\[[2](https://arxiv.org/html/2608.11361#bib.bib2)\]builds inference\-efficient tokenizers by optimizingscore​\(t\)=freq​\(t\)⋅\|t\|\\text\{score\}\(t\)=\\text\{freq\}\(t\)\\cdot\|t\|to balance token frequency and length\. Benchmarks are on A100 atB=1B=1only; no batch sweep, no lifecycle framework, and the cost function contains no bytes\-moved term\.

Compute\-Optimal Tokenization \(2026\)\[[3](https://arxiv.org/html/2608.11361#bib.bib3)\]addresses training\-time FLOPs combined with compression and loss \(BPB\)\. Their Section 3\.5 explicitly states “the inference\-optimal frontier \[is\] underexplored\.” We address this gap directly\.

Hardware Co\-Design Scaling Laws \(2026\)\[[4](https://arxiv.org/html/2608.11361#bib.bib4)\]co\-designs depth, width, and quantization against a roofline model on Jetson Orin, but holds vocabulary constant\. We add vocabulary as the co\-design variable and sweep across hardware ridge points\.

Getting the Most Out of Your Tokenizer \(2024\)\[[10](https://arxiv.org/html/2608.11361#bib.bib10)\]provides an analytic batch\-dependent model for vocabulary cost and introduces BPB as a cross\-tokenizer quality metric\. This is the closest prior art on the cost side; we extend it with empirical measurement across hardware regimes and a full lifecycle framework\.

### II\-BRoofline Model

The roofline model\[[5](https://arxiv.org/html/2608.11361#bib.bib5)\]bounds the achievable performance of any operation:

performance=min⁡\(BW⋅I,peak\_FLOPS\)\\text\{performance\}=\\min\(\\text\{BW\}\\cdot I,\\ \\text\{peak\\\_FLOPS\}\)\(1\)whereII\(FLOP/byte\) is arithmetic intensity\. The ridge pointρ=peak\_FLOPS/BW\\rho=\\text\{peak\\\_FLOPS\}/\\text\{BW\}separates memory\-bound \(I<ρI<\\rho\) from compute\-bound \(I\>ρI\>\\rho\) operation\. We measureρ\\rhoempirically at runtime: peak bandwidth via a large memory copy \(64M fp16 elements, 128 MB\), and peak compute via a large square GEMM \(4096×40964096\\times 4096, fp16\)\. This yieldsρA10G≈117\\rho\_\{\\text\{A10G\}\}\\approx 117andρA100≈183\\rho\_\{\\text\{A100\}\}\\approx 183FLOP/byte, which are used to label every measurement as memory\-bound or compute\-bound\.

### II\-CBPB as a Quality Metric

Bits per byte \(BPB\) normalizes model quality across tokenizers:

BPB=ℒln⁡2⋅r⁡\(V\)\\text\{BPB\}=\\frac\{\\mathcal\{L\}\}\{\\ln 2\}\\cdot r\(V\)\(2\)whereℒ\\mathcal\{L\}is cross\-entropy loss in nats per token andr⁡\(V\)r\(V\)is tokens per byte\. Unlike raw loss, BPB is comparable across tokenizers of different vocabulary sizes because it expresses predictive uncertainty per byte of original text, independent of how that text was tokenized\[[10](https://arxiv.org/html/2608.11361#bib.bib10)\]\.

## IIICost Model

### III\-AInference Cost

Each autoregressive decode step produces one token via:

𝐥𝐨𝐠𝐢𝐭𝐬=𝐡​WT,𝐡∈ℝB×d,W∈ℝV×d\\mathbf\{logits\}=\\mathbf\{h\}W^\{T\},\\quad\\mathbf\{h\}\\in\\mathbb\{R\}^\{B\\times d\},\\ W\\in\\mathbb\{R\}^\{V\\times d\}\(3\)
The bytes moved from HBM are dominated by the weight matrix read:bytes≈V​d⋅bpp\\text\{bytes\}\\approx Vd\\cdot\\text\{bpp\}where bpp = 2 for fp16\. The FLOPs are2​B​V​d2BVd\. Arithmetic intensity \(simplified forV≫BV\\gg B\):

I=2​B​V​dV​d⋅bpp=2​Bbpp=B\(fp16\)I=\\frac\{2BVd\}\{Vd\\cdot\\text\{bpp\}\}=\\frac\{2B\}\{\\text\{bpp\}\}=B\\quad\\text\{\(fp16\)\}\(4\)
Per\-token head cost follows the roofline:

h⁡\(V,B\)=1B⋅max⁡\(V​d⋅bppBW,2​B​V​dpeak\)h\(V,B\)=\\frac\{1\}\{B\}\\cdot\\max\\\!\\left\(\\frac\{Vd\\cdot\\text\{bpp\}\}\{\\text\{BW\}\},\\ \\frac\{2BVd\}\{\\text\{peak\}\}\\right\)\(5\)
WhenB<ρB<\\rho\(memory\-bound\):h⁡\(V,B\)∝V/Bh\(V,B\)\\propto V/B, falling linearly with batch\. WhenB\>ρB\>\\rho\(compute\-bound\):h⁡\(V,B\)∝V/peakh\(V,B\)\\propto V/\\text\{peak\}, independent ofBB\.

The transformer body \(attention \+ FFN, no head\) contributes a V\-independent costcbody​\(B\)c\_\{\\text\{body\}\}\(B\)measured via CUDA graph capture\. Total inference cost per character:

Cinfer​\(V,B\)=cbody​\(B\)\+h​\(V,B\)p⁡\(V\)C\_\{\\text\{infer\}\}\(V,B\)=\\frac\{c\_\{\\text\{body\}\}\(B\)\+h\(V,B\)\}\{p\(V\)\}\(6\)wherep⁡\(V\)=1/r⁡\(V\)p\(V\)=1/r\(V\)is chars per token \(compression, from Phase A\)\.

This expression has a U\-shape inVV: smallVVforces many decode steps \(smallp⁡\(V\)p\(V\)\); largeVVincurs largeh⁡\(V,B\)h\(V,B\)per step\. The minimum – the inference\-optimal vocabulary – shifts rightward asBBgrows andh⁡\(V,B\)h\(V,B\)shrinks\.

### III\-BTraining Cost

Training processes all tokens in the corpus once\. The per\-byte training cost \(ms/byte\):

Ctrain​\(V\)=r⁡\(V\)τ⁡\(V\)×1000C\_\{\\text\{train\}\}\(V\)=\\frac\{r\(V\)\}\{\\tau\(V\)\}\\times 1000\(7\)wherer⁡\(V\)r\(V\)is tokens per byte andτ⁡\(V\)\\tau\(V\)is training throughput \(tok/s\)\. Both decrease withVV: largerVVcompresses better \(fewer tokens per byte\) but also slows each step \(larger embedding and head layers\)\. Their ratio has a minimum atV≈16V\\approx 16k in our experiments \(Section[IV\-D](https://arxiv.org/html/2608.11361#S4.SS4)\)\.

### III\-CLifecycle Cost and Optimal Vocabulary

The total deployment cost over the model’s lifetime:

Clifecycle​\(V,B,λ\)=Ctrain​\(V\)\+λ⋅Cinfer​\(V,B\)C\_\{\\text\{lifecycle\}\}\(V,B,\\lambda\)=C\_\{\\text\{train\}\}\(V\)\+\\lambda\\cdot C\_\{\\text\{infer\}\}\(V,B\)\(8\)
λ=inference\_bytes/training\_bytes\\lambda=\\text\{inference\\\_bytes\}/\\text\{training\\\_bytes\}is a dimensionless ratio capturing inference volume:λ=1\\lambda=1denotes one inference pass per training byte;λ=1000\\lambda=1000is typical of a production API;λ=106\\lambda=10^\{6\}represents a high\-traffic service \(e\.g\., a public chatbot\)\. BothCtrainC\_\{\\text\{train\}\}andCinferC\_\{\\text\{infer\}\}are in ms/byte, makingλ\\lambdaa direct weight between the two costs\.

The lifecycle\-optimal vocabulary:

V∗​\(B,λ\)=arg⁡minV⁡Clifecycle​\(V,B,λ\)V^\{\*\}\(B,\\lambda\)=\\mathop\{\\arg\\min\}\_\{V\}\\ C\_\{\\text\{lifecycle\}\}\(V,B,\\lambda\)\(9\)
Since both cost curves are convex \(verified empirically\),V∗​\(B,λ\)V^\{\*\}\(B,\\lambda\)interpolates betweenVtrain∗V^\{\*\}\_\{\\text\{train\}\}\(atλ=0\\lambda=0\) andVinfer∗​\(B\)V^\{\*\}\_\{\\text\{infer\}\}\(B\)\(atλ→∞\\lambda\\to\\infty\)\. The lifecycle optimum never falls outside\[Vtrain∗,Vinfer∗​\(B\)\]\[V^\{\*\}\_\{\\text\{train\}\},\\,V^\{\*\}\_\{\\text\{infer\}\}\(B\)\]– but this interval is itselfBB\-dependent, widening dramatically at datacenter batch\.

## IVExperimental Setup

### IV\-AHardware and Calibration

TABLE I:GPU hardware used in experiments\. Ridge point measured empirically\.Peak bandwidth and compute are measured at startup viacalibrate\(\): a 64M\-element fp16 memory copy \(128 MB, too large for L2 cache\) for bandwidth, and a4096×40964096\\times 4096fp16 GEMM \(intensity≈1365≫ρ\\approx 1365\\gg\\rho\) for peak compute\. Achieved values are 67–79% of theoretical peak due to memory refresh cycles, cache pressure, and thermal state\. Using measured rather than spec\-sheet values yields ridge\-point predictions that match observed crossover batches to within 1\.6×\\times\.

### IV\-BPhase A: Tokenizer Training and Compression Curve

We trained BPE tokenizers using SentencePiece\[[11](https://arxiv.org/html/2608.11361#bib.bib11)\]on 500 MB of FineWeb\-Edu\[[9](https://arxiv.org/html/2608.11361#bib.bib9)\], a curated English educational web corpus, at eight vocabulary sizes:V∈\{8192,16384,32768,65536,131072,262144,524288,1048576\}V\\in\\\{8192,16384,32768,65536,131072,262144,524288,1048576\\\}\. Training used the default BPE algorithm: starting from individual characters, iteratively merging the most frequent adjacent token pair until reaching the targetVV\.

Compressionp⁡\(V\)p\(V\)\(chars per token\) was evaluated on a held\-out 10k\-line validation split\. Key result:p⁡\(V\)p\(V\)grows from 4\.00 atV=8V=8k to 5\.19 atV=262V=262k, then saturates – only 0\.6% additional gain from 262k to 524k, and 0\.3% from 524k to 1M\. This saturation is critical: it creates the right arm of the U\-shapedCinferC\_\{\\text\{infer\}\}curve at high batch, bounding the inference optimum at≈\\approx524k rather than infinity\.

### IV\-CPhase B: Inference Benchmarks

Head measurement\.We measure the unembedding matmul𝐡​WT\\mathbf\{h\}W^\{T\}in isolation using CUDA events \(50 timed iterations, 15 warmup, median of 3 independent runs\)\. SweepingV∈\{8kV\\in\\\{8\\text\{k\}–1M\}1\\text\{M\}\\\}andB∈\{1,16,64,256,512,1024,4096,16384\}B\\in\\\{1,16,64,256,512,1024,4096,16384\\\}on a single GPU\. Measuring in isolation is essential: a full decode step launches∼\\sim100 GPU kernels \(one per operation in 12 transformer layers\), each incurring∼\\sim15μ\\mus of kernel\-launch overhead\. AtB=1B=1, this overhead \(∼\\sim1\.5 ms total\) dwarfs the head cost \(∼\\sim0\.015–0\.76 ms depending onVV\), making the vocabulary signal unmeasurable\. Isolating the single head kernel removes all launch overhead\.

For each \(V, B\) pair, we record: total latency \(ms\), per\-token latency \(μ\\mus/tok\), achieved bandwidth \(GB/s\), achieved compute \(TFLOP/s\), arithmetic intensity \(FLOP/byte\), regime label \(mem/compute\), bandwidth utilization, and FLOP utilization\.

Body measurement\.The transformer body \(12 layers,d=512d=512, 8 heads, no LM head\) is timed via three methods: \(1\)*eager mode*\(baseline with full kernel\-launch overhead\), \(2\)*torch\.compile*\(fused kernels, reduced overhead\), and \(3\)*CUDA graph capture*\(gold standard – the decode step is recorded once and replayed with zero launch overhead\)\. The gold\-standard body cost atB=1B=1, ctx=512 iscbody=0\.667c\_\{\\text\{body\}\}=0\.667ms/tok\. An analytical roofline estimate gives 0\.065 ms/tok, implying the actual kernels operate at≈\\approx10% of theoretical efficiency – typical for small\-batch transformer inference without custom kernels\.

### IV\-DPhase C: Training and Quality Measurement

100M\-scale run\.We trained 6 GPT models with identical architecture \(d=512d=512, 12 layers, 8 heads, ctx=2048\) atV∈\{8192,16384,32768,65536,131072,262144\}V\\in\\\{8192,16384,32768,65536,131072,262144\\\}for 10k steps each, batch=16, AdamW \(lr=3e\-4, cosine schedule\), on 50M tokens of FineWeb\-Edu\. All 6 models share the same transformer body \(37\.7M parameters\); only the embedding and head layers grow withVV\. Hardware: 8×\\timesA100 with DataParallel\.

A key implementation challenge: the standard DataParallel approach gathers the full\[batch×ctx×V\]\[\\text\{batch\}\\times\\text\{ctx\}\\times V\]logit tensor on GPU 0 before computing the cross\-entropy loss\. AtV=65536V=65536, this tensor alone requires16×2048×65536×4=816\\times 2048\\times 65536\\times 4=8GB, causing out\-of\-memory errors\. We resolve this by wrapping the model in aGPTWithLossmodule that computes the loss inside each GPU shard and gathers only the scalar loss to GPU 0, reducing cross\-GPU communication from gigabytes to bytes\.

1\.3–2\.3B scale run \(scale test\)\.To test whether quality differentiates acrossVVat larger scale, we trained 3 models atV∈\{16384,65536,262144\}V\\in\\\{16384,65536,262144\\\}with architectured=2048d=2048, 24 layers, 16 heads, ctx=1024, for 5k steps, batch=32 total across 8×\\timesA100\.

At this scale, DataParallel is insufficient: the 2\.3B model atV=262V=262k requires≈\\approx19 GB of AdamW optimizer states \(momentum \+ variance \+ master weights\), all of which DataParallel places on GPU 0, saturating its 40 GB HBM\. We switch toFully Sharded Data Parallel \(FSDP\), which shards parameters, gradients, and optimizer states evenly across all 8 GPUs, reducing per\-GPU optimizer memory from∼\\sim19 GB to∼\\sim2\.4 GB\.

Training was initially attempted with fp16 mixed precision\. AtV=262144V=262144, the raw unembedding logits can exceed fp16’s maximum representable value \(≈\\approx65504\), causing overflow and training divergence \(loss increasing monotonically from∼\\sim9 to\>\>18 over 2k steps\)\. Switching tobf16\(same exponent range as fp32, maximum≈3\.4×1038\\approx 3\.4\\times 10^\{38\}\) eliminates the overflow\. A learning rate of 6e\-5 \(vs\. 3e\-4 at 100M scale\) is used to stabilize 2B\+ parameter training\.

## VResults

### V\-ARegime Crossover Confirms Roofline Prediction

Table[II](https://arxiv.org/html/2608.11361#S5.T2)shows the batch at which the unembedding transitions from memory\-bound to compute\-bound on each GPU\. The measured crossover matches the theoretical ridge point to within a factor of 1\.5–2×\\times\(the small discrepancy arises because the full bytes\-moved includes activations and logits, not just the weight matrix\)\.

TABLE II:Measured memory\-to\-compute crossover batch vs\. predicted from ridge point\.Fig\.[1](https://arxiv.org/html/2608.11361#S5.F1)shows per\-token head cost vs\. batch on A100\. AtB=1B=1, bandwidth utilization is 82–99% and FLOP utilization is<<1% \(memory\-bound\)\. ByB=512B=512, FLOP utilization reaches 49–97% \(compute\-bound\)\. The crossover shifted fromB≈128B\\approx 128–256 on A10G toB≈256B\\approx 256–512 on A100, tracking the ridge ratio183/117≈1\.6×183/117\\approx 1\.6\\times\. This confirms the effect is roofline physics, not an artifact of one platform\.

![Refer to caption](https://arxiv.org/html/2608.11361v1/figures/fig2_head_amortization.png)Fig\. 1:Per\-token unembedding cost vs\. batch on A100 for four vocabulary sizes\. Gray band marks the memory\-to\-compute crossover\. FromB=1B=1toB=16384B=16384, cost drops 143–377×\\timesdepending onVV\.
### V\-BInference\-Optimal Vocabulary Shifts 16×\\timeswith Batch

Table[III](https://arxiv.org/html/2608.11361#S5.T3)shows per\-token head cost at key \(V, B\) pairs on A100\. The amortization ratio \(column 5\) ranges from 143×\\timesto 377×\\timesfromB=1B=1toB=16384B=16384, confirming the theoretical∝B\\propto Bscaling\.

TABLE III:Per\-token head cost \(μ\\mus\) on A100 and amortization ratioB=1→B=16384B=1\\to B=16384\.Usingcbody=0\.667c\_\{\\text\{body\}\}=0\.667ms/tok \(CUDA graph, ctx=512,B=1B=1\) and real BPE compression from Phase A, the cost\-optimal vocabulary from Eq\. \([6](https://arxiv.org/html/2608.11361#S3.E6)\):

- •B=1B=1:Vinfer∗=32768V^\{\*\}\_\{\\text\{infer\}\}=32768\. Head cost atV=262V=262k is 203μ\\mus/tok vs\. body 667μ\\mus/tok\. Head contributes 23%, creating a real penalty for largeVV\.
- •B=64B=64:Vinfer∗=524288V^\{\*\}\_\{\\text\{infer\}\}=524288\. Head atV=262V=262k is now 61μ\\mus/tok vs\. body 75μ\\mus/tok\. Head is 45% of total but compression gain past 262k is only 0\.6%, so the minimum is at 524k where compression saturates\.

The 16×\\timesshift \(3232k→\\to524524k\) is shown in Fig\.[2](https://arxiv.org/html/2608.11361#S5.F2)\. The saturation of the BPE compression curve at≈\\approx524k \(Phase A\) naturally bounds the optimum, preventing it from running to infinity at high batch\.

![Refer to caption](https://arxiv.org/html/2608.11361v1/figures/fig3_optimal_v_vs_batch.png)Fig\. 2:Inference\-cost\-optimal vocabulary vs\. serving batch \(A100, real BPE compression,cbody=0\.667c\_\{\\text\{body\}\}=0\.667ms/tok\)\. The blue dashed line showsVtrain∗=16V^\{\*\}\_\{\\text\{train\}\}=16k\.
### V\-CTraining Results and Training\-Optimal Vocabulary

Table[IV](https://arxiv.org/html/2608.11361#S5.T4)summarizes Phase C results at 100M scale\.

TABLE IV:Training results at 100M scale \(d=512, 12L, 10k steps, 8x A100\)\.Two observations: \(1\) BPB is flat across the fullVVrange \(<<2% spread, best atV=16V=16k\), confirming that vocabulary is a pure systems decision at this model scale with negligible quality impact\. \(2\) Training throughput drops 2\.4×\\timesfromV=8V=8k toV=262V=262k, reflecting the larger embedding/head layers\. The training\-cost minimum isVtrain∗=16384V^\{\*\}\_\{\\text\{train\}\}=16384: below this, poor compression forces more tokens per byte; above this, the throughput penalty outweighs compression gains\.

### V\-DQuality Shifts Rightward at Scale

At 1\.3–2\.3B scale \(FSDP, bf16, 5k steps\), BPB results are:

TABLE V:Quality at 1\.3–2\.3B scale \(d=2048, 24L, FSDP, bf16, 5k steps\)\.The quality optimum shifted fromV=16V=16k \(at 100M\) toV=65V=65k \(at 1\.5B\): BPB improves from 1\.399 to 1\.387 \(0\.9% gain\), consistent with Tao et al\.’s prediction ofV∗=k​N≈63V^\{\*\}=k\\sqrt\{N\}\\approx 63k atN=1\.5N=1\.5B\.V=262V=262k is slightly worse thanV=65V=65k, likely because at 2\.3B total parameters the embedding and head account for∼\\sim1\.1B of 2\.3B parameters, leaving less capacity for the transformer body that performs language modeling\. Fig\.[3](https://arxiv.org/html/2608.11361#S5.F3)shows BPB vs\. vocabulary at both scales\.

![Refer to caption](https://arxiv.org/html/2608.11361v1/figures/fig4_bpb_vs_vocab_scale.png)Fig\. 3:BPB vs\. vocabulary at two scales\. Left \(100M\): BPB is flat; quality does not constrain vocabulary choice\. Right \(1\.3–2\.3B\): the optimum shifts toV=65V=65k, confirming scale\-dependent vocabulary preference\.
### V\-ELifecycle Sweep

Table[VI](https://arxiv.org/html/2608.11361#S5.T6)presentsV∗​\(B,λ\)V^\{\*\}\(B,\\lambda\)from Eq\. \([9](https://arxiv.org/html/2608.11361#S3.E9)\), computed using measuredCtrainC\_\{\\text\{train\}\}\(Table[IV](https://arxiv.org/html/2608.11361#S5.T4)\) andCinferC\_\{\\text\{infer\}\}\(Section IV\-B\)\. Atλ=0\\lambda=0\(training only\),V∗=16V^\{\*\}=16k regardless of batch\. Any serving volume \(λ≥1\\lambda\\geq 1\) immediately shiftsV∗V^\{\*\}away from the training optimum\. The transition is gradual: atB=256B=256,λ=1\\lambda=1givesV∗=65V^\{\*\}=65k but reachingV∗=262V^\{\*\}=262k requiresλ≥100\\lambda\\geq 100, corresponding to serving at least 100 bytes of inference per byte of training data\.

TABLE VI:Lifecycle\-optimal vocabularyV∗​\(B,λ\)V^\{\*\}\(B,\\lambda\)\. Values in thousands \(e\.g\., 262 = 262k tokens\)\.λ=1000\\lambda=1000is typical production\.

## VIInfrastructure Implications

### VI\-ACapacity Planning Impact

Vocabulary size directly affects the number of accelerators required to meet a throughput SLA\. The inference cost per character \(Eq\.[6](https://arxiv.org/html/2608.11361#S3.E6)\) determines how many decode steps the cluster must execute per unit of output text\. Moving from the conventionalV=32V=32k to the lifecycle\-optimalV=262V=262k at datacenter batch \(B=256B=256,λ≥100\\lambda\\geq 100\) reduces per\-character inference cost by≈\\approx2×\\times, directly halving the number of GPU\-hours required for a fixed output volume\.

Concretely, atB=64B=64:V=524V=524k reduces head cost from 61μ\\mus/tok \(V=262V=262k\) and provides 0\.6% more compression\. The combined effect on cost\-per\-character is a≈\\approx10% reduction vs\.V=262V=262k\. While modest, at hyperscale inference volumes this translates to substantial cluster cost savings\.

Vocabulary also interacts with KV\-cache utilization\. LargerVVproduces shorter token sequences for the same text \(better compression\), which directly reduces KV\-cache size: a 16×\\timesmore compressed tokenizer halves sequence length, halving KV\-cache memory requirements\. For long\-context serving, this can be the dominant capacity driver\.

### VI\-BTheoretical Extension to Quantization

While we did not run quantization experiments, the roofline model predicts the effect analytically\. int4 quantization of the unembedding \(bpp = 0\.5 instead of 2\) would reduce bytes read from HBM by 4×\\times, shifting arithmetic intensity fromI≈BI\\approx BtoI≈4​BI\\approx 4Band raising the effective ridge point 4×\\times\. The memory\-to\-compute crossover batch would shift fromB≈256B\\approx 256toB≈64B\\approx 64on A100, meaning the head amortizes at lower batch sizes and the lifecycle optimum moves toward largerVVeven at modest batch\. Empirical validation of this prediction is left as future work\.

### VI\-CProvisioning Recommendations

For infrastructure teams making vocabulary selection decisions:

- •On\-device / edge \(B=1B=1\):UseV≈32V\\approx 32k\. The head is memory\-bound and scales linearly withVV; the optimal vocabulary minimizes memory traffic per step while maintaining sufficient compression\. Quality is unaffected \(<<1% BPB difference across 32k–262k at 1\.5B scale\)\.
- •API servers \(B=16B=16–64\):UseV≈65V\\approx 65–131k\. The head begins to amortize; compression reduces total step count\. The lifecycle optimum atλ=10\\lambda=10\(moderate serving\) falls in this range\.
- •Datacenter / high\-throughput \(B≥256B\\geq 256,λ≥100\\lambda\\geq 100\):UseV≈262V\\approx 262k\. Full head amortization; maximum compression gain; no quality penalty at≥\\geq1B model scale\. This is 8–16×\\timeslarger than the training convention of 16–32k\.

These recommendations apply at model scales≥\\geq1B where quality weakly favors largerVV\. At<<300M parameters, quality is flat and the choice reduces to the training\-inference cost trade\-off alone\.

## VIILimitations

Model scale\.Our largest training run is 2\.3B parameters\. Tao’s scaling lawV∗∝N0\.5V^\{\*\}\\propto N^\{0\.5\}predicts the quality benefit of largeVVcontinues to grow with scale; at 7B\+ the quality optimum would be≈\\approx150k\+ \(untested here\)\.

Training budget\.We trained 1\.3–2\.3B models for only 5k steps\. TheV=262V=262k model may be undertrained relative toV=65V=65k at an equal compute budget \(FLOPs\), since the larger embedding/head consume parameters that would otherwise contribute to the transformer body\.

Corpus\.Tokenizers were trained on English FineWeb\-Edu\. Compression curves differ for code \(higher compression from repeated patterns\), multilingual text \(lower per\-character compression for non\-Latin scripts\), and mathematical notation\. The optimalVVvalues are corpus\-dependent\.

Architecture\.All results use decoder\-only GPT\. Mixture\-of\-experts \(MoE\) models, state\-space models, and encoder\-decoder architectures have different body/head cost ratios and may exhibit different optimal vocabulary sizes\.

## VIIIConclusion

We have shown that the cost\-optimal tokenizer vocabulary is not a constant but a deployment\-regime\-dependent infrastructure parameter\. The lifecycle frameworkClifecycle​\(V,B,λ\)C\_\{\\text\{lifecycle\}\}\(V,B,\\lambda\)quantifies a three\-way tension: training cost is minimized atV≈16V\\approx 16k; quality \(BPB\) is minimized atV≈65V\\approx 65k at 1\.5B scale; and inference cost at serving batch is minimized atV≈262V\\approx 262k\+ forB≥64B\\geq 64\.

The key practical finding: for any production LLM deployment withλ≥100\\lambda\\geq 100andB≥64B\\geq 64, the lifecycle\-optimal vocabulary is 8–16×\\timeslarger than training convention suggests, with<<2% quality cost in the measured parameter range\. The effect is roofline physics – confirmed on two GPU ridge points and predictable from hardware specs – making it applicable to future GPU generations without re\-measurement\.

For infrastructure teams, this result is directly actionable: vocabulary selection should be treated as a capacity planning input alongside batch size, serving regime, and quantization strategy, rather than a fixed convention inherited from training\.

## Acknowledgment

The authors thank the open\-source communities behind PyTorch, SentencePiece, and HuggingFace Datasets\. Experiments were conducted on AWS EC2 p4d\.24xlarge \(8×\\timesA100\-SXM4\-40GB\) and g5\.4xlarge \(1×\\timesA10G\-24GB\) instances\.

AI Tool Disclosure \(IEEE Policy\):This work made substantial use of Claude \(Anthropic\) as an AI assistant throughout the research process\. The authors originated the core research idea – that vocabulary size should be treated as a deployment\-regime parameter rather than a training\-time constant – and defined the experimental questions and hypotheses\. Claude was used to: \(1\) conduct the literature survey, identifying and summarizing relevant prior work \(Tao 2024, Length\-MAX, Compute\-Optimal Tokenization, Hardware Co\-Design Scaling Laws, and others\) and assessing novelty and differentiation of our contribution; \(2\) implement the experimental harness \(bench\_head\.py,bench\_body\.py,train\_models\.py,lifecycle\_sweep\.pyand related scripts\) from author\-provided specifications and seed ideas; \(3\) debug training instabilities \(fp16 overflow, DataParallel OOM, FSDP configuration\); \(4\) assist with data analysis and result interpretation; and \(5\) draft and revise sections of this manuscript\. All experimental design decisions, scientific claims, and final interpretations are the authors’ own\. The authors verified all code, reviewed all outputs, and take full responsibility for the content of this paper\.

## References

- \[1\]C\. Tao, Q\. Liu, L\. Dou, N\. Muennighoff, Z\. Wan, P\. Luo, M\. Lin, and N\. Wong, “Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies,”Advances in Neural Information Processing Systems \(NeurIPS\), 2024, arXiv:2407\.13623\.
- \[2\]D\. Dong and W\. Su, “Length\-MAX Tokenizer for Language Models,” 2025, arXiv:2511\.20849\.
- \[3\]T\. Limisiewicz, A\. Pagnoni, S\. Iyer, M\. Lewis, S\. Mehta, A\. Liu, M\. Li, G\. Ghosh, and L\. Zettlemoyer, “Compute Optimal Tokenization,” 2025, arXiv:2605\.01188\.
- \[4\]L\. Sun, J\. Jiang, Y\. Ding, F\. Li, Y\. Song, H\. Zhang, J\. Ying, L\. Ren, K\. Zhan, W\. Chen, Y\. Xie, and C\. Deng, “Hardware Co\-Design Scaling Laws via Roofline Modelling for On\-Device LLMs,” 2026, arXiv:2602\.10377\.
- \[5\]S\. Williams, A\. Waterman, and D\. Patterson, “Roofline: An Insightful Visual Performance Model for Floating\-Point Programs and Multicore Architectures,”Communications of the ACM, vol\. 52, no\. 4, pp\. 65–76, Apr\. 2009\.
- \[6\]H\. Touvron, L\. Martin, K\. Stoneet al\., “Llama 2: Open Foundation and Fine\-Tuned Chat Models,” 2023, arXiv:2307\.09288\.
- \[7\]OpenAI, J\. Achiam, S\. Adleret al\., “GPT\-4 Technical Report,” 2023, arXiv:2303\.08774\.
- \[8\]Gemma Team: M\. Riviere, S\. Pathak, P\. G\. Sessaet al\., “Gemma 2: Improving Open Language Models at a Practical Size,” 2024, arXiv:2408\.00118\.
- \[9\]G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. Wolf, “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale,” 2024, arXiv:2406\.17557\.
- \[10\]G\. Dagan, G\. Synnaeve, and B\. Rozière, “Getting the Most Out of Your Tokenizer for Pre\-Training and Domain Adaptation,” 2024, arXiv:2402\.01035\.
- \[11\]T\. Kudo and J\. Richardson, “SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing,”Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\): System Demonstrations, 2018, arXiv:1808\.06226\.