Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

# Learning Per-Group Precision for “Bigger-but-Smaller” Language Models
Source: [https://arxiv.org/html/2607.02893](https://arxiv.org/html/2607.02893)
## Variable Bit\-width Quantization: Learning Per\-Group Precision for “Bigger\-but\-Smaller” Language Models

###### Abstract

Low\-bit quantization shrinks language models but treats precision as a single global hyper\-parameter: every weight is forced to the same bit\-width\. We introduceVariable Bit\-width Quantization\(VBQ\), a training\-time method in which each contiguous group of 64 weights*learns*its own resolution from\{1,2,4,8\}\\\{1,2,4,8\\\}bits through a Gumbel\-Softmax relaxation, trained jointly with the network by an alternating optimization that gives the precision logits a clean, task\-aligned signal\. Across decoder\-only transformers,VBQdiscovers a consistent and strongly heterogeneous allocation,*within*individual projection types and not merely across layers, that is impossible to express with per\-layer methods:69%69\\%of groups collapse to1 bit, the language\-model head averages1\.091\.09bits, while the first MLP block keeps∼2\.5\\sim\\\!2\.5bits\. The discovered pattern is stable enough to be frozen into a fixed*recipe*and reused without any further search\. Using this recipe we demonstrate a “bigger\-but\-smaller” regime: a 131M\-parameter model at1\.821\.82mean bits reaches validation perplexity4\.24\.2on TinyStories,*beating*a 55M FP16 model \(PPL4\.44\.4\) while using3\.8×3\.8\\timesless storage, and the same recipe lets a1\.461\.46B\-parameter model on FineWeb\-Edu match a593593M FP16 control at∼3\.7×\\sim\\\!3\.7\\timesless storage \(2\.5×2\.5\\timesmore parameters,∼3\.7×\\sim\\\!3\.7\\timessmaller footprint\)\. Measured as quality\-per\-byte,VBQis3\.93\.9–8\.4×8\.4\\timesmore efficient than FP16\. Crucially, the recipe maps directly to packed low\-bit storage, so it*also accelerates inference*: with custom fused dequantize\-and\-multiply kernels, autoregressive decode, which is memory\-bandwidth bound, is faster at equal output, and because the benefit is bandwidth\-driven the speedup*grows with scale*\(parity at131131M,1\.9×1\.9\\timesat1\.01\.0B,4\.7×4\.7\\timesat99B on Apple silicon\)\. We complement perplexity with a distributional analysis \(KL divergence and argmax\-flip rate\) that reveals a striking mechanism: deeper layers progressively*self\-heal*the quantization error injected by early layers\. We delimit the method’s scope honestly: its win is a from\-scratch, train\-time phenomenon, and scaling the search economically beyond1\.51\.5B parameters and adapting it to pretrained checkpoints remain open\.VBQreframes precision as a learnable, non\-uniform resource and shows that spending a fixed bit budget unevenly is materially better than spending it uniformly\.

## 1Introduction

The dominant lever for deploying large language models \(LLMs\) under tight memory budgets is quantization: storing weights at fewer than 16 bits\. Almost all deployed schemes \(post\-training methods such as GPTQ\[[13](https://arxiv.org/html/2607.02893#bib.bib13)\]and AWQ\[[18](https://arxiv.org/html/2607.02893#bib.bib18)\], and quantization\-aware training such as BitNet\[[34](https://arxiv.org/html/2607.02893#bib.bib34),[21](https://arxiv.org/html/2607.02893#bib.bib21)\]\) fix a*single*bit\-width for \(almost\) all weights\. The schemes that do vary it still fix the allocation by a predetermined rule rather than learning it\. Production toolchains such as Unsloth’s dynamic GGUF quants\[[33](https://arxiv.org/html/2607.02893#bib.bib33)\]keep salient layers at higher precision as a per\-layer, post\-training heuristic; NVIDIA’s NVFP4\[[23](https://arxiv.org/html/2607.02893#bib.bib23)\]goes finer, mixing precision*within*a tensor through 16\-element micro\-blocks \(and reserving a few hand\-picked layers for higher precision\), and is used both for post\-training quantization and for 4\-bit pretraining\[[24](https://arxiv.org/html/2607.02893#bib.bib24)\]\. In every case the bit allocation follows a fixed format or heuristic; it is not a per\-group choice the optimization*learns*during training\.

This is a convenient, yet suboptimal modeling choice\. Weights in an LLM are not equally important: attention output projections tolerate aggressive compression, while the first feed\-forward block and a small set of frequent output\-vocabulary rows are far more sensitive\. A method that could*allocate*bits where they matter, at fine granularity and decided by the optimization itself, should dominate any uniform assignment on the quality/size frontier\.

A second observation motivates*how much*precision is actually needed, and is what set this work in motion\. In production neural retrieval, the neural hashing that compresses embedding vectors in Algolia’s search stack\[[29](https://arxiv.org/html/2607.02893#bib.bib29)\]was originally trained on FP32 vectors and compressed them roughly10×10\\times, to about3\.23\.2bits per dimension, with a change in ranking quality \(nDCG\) smaller than the noise of the embedding model itself; we later relaxed this to44bits, but pushing*below*that range degraded quality quickly\. We had treated this as a practical rule of thumb: a trained representation seems to carry on the order of33–44useful bits per value, a limit we hit in production, and one that has since surfaced repeatedly in the LLM literature\. Apple’s on\-device foundation model is independent, production\-scale evidence of the same limit: it compresses to a*mixed*2\-/4\-bit scheme averaging3\.73\.7bits\-per\-weight to*match*the uncompressed model \(and3\.53\.5bits without significant quality loss\)\[[1](https://arxiv.org/html/2607.02893#bib.bib1)\]\. The same threshold recurs inDettmers and Zettlemoyer\[[6](https://arxiv.org/html/2607.02893#bib.bib6)\], who find44\-bit Pareto\-optimal in theirkk\-bit inference scaling laws\. That the largest deployers either still pay for 16\-bit weights or accept lossy quantization, yet a carefully mixed scheme reaches∼4\\sim\\\!4bits without measurable quality loss, is what made the opportunity concrete: there was clearly room to operate near44bits losslessly; the open question was*how*to get there\. We read this convergence across retrieval, on\-device LLMs, and scaling\-law analyses as an empirical floor: below roughly3\.53\.5–44bits per value, information is genuinely lost\.VBQ’s “bigger\-but\-smaller” result is consistent with this floor, but with a crucial twist about*how*to reach it\. The way to hit the floor is*not*to round every weight to four bits: uniform 4\-bit quantization forces a lossy approximation of weights that were never trained for it\. It is to spend a comparable total budget*differently*: distribute it across more, lower\-precision weights \(in our recipe, an average of∼1\.8\\sim\\\!1\.8bits\) so the network learns a representation that lives natively at low bits\. The information budget is conserved; the layout is what changes, and, as we show, layout is decisive\.

We proposeVariable Bit\-width Quantization\(VBQ\)\. Each group of 64 contiguous weights is given a categorical variable over the candidate bit\-widths\{1,2,4,8\}\\\{1,2,4,8\\\}, relaxed with Gumbel\-Softmax\[[16](https://arxiv.org/html/2607.02893#bib.bib16),[22](https://arxiv.org/html/2607.02893#bib.bib22)\]so it is differentiable\. The central difficulty is that the cross\-entropy gradient, flowing through the quantizer, overwhelms the bit\-budget penalty by two\-to\-three orders of magnitude, freezing the precision logits at their initialization\. We resolve this with two design choices:*gradient isolation*\(the reconstructed weights are detached when computing the bit\-selection path, so cross\-entropy cannot back\-propagate into the precision logits\) and an*alternating optimization*\(weight steps and precision steps are interleaved, in the spirit of DARTS\[[19](https://arxiv.org/html/2607.02893#bib.bib19)\]\)\. Together these let the logits actually move and commit\.

Our key empirical findings are:

1. 1\.A learnable, heterogeneous precision hierarchy emerges\(Section[5\.1](https://arxiv.org/html/2607.02893#S5.SS1)\)\. The network drives69%69\\%of groups to1 bitand reserves44/8 bitfor a minority, producing a1\.781\.78\-bit mean\. The allocation is heterogeneous*within*projection types, structurally impossible for per\-layer mixed\-precision methods\.
2. 2\.The recipe is the product, not the search\(Section[3\.4](https://arxiv.org/html/2607.02893#S3.SS4)\)\. Bit assignments lock in by∼20%\\sim\\\!20\\%of training; freezing them into a fixed recipe \(uniform2 bitblocks, a44\-bit first block, and a frequency\-tiered language\-model head averaging1\.181\.18bits\) reproduces the adaptive result with no Gumbel machinery at deployment\.
3. 3\.Bigger\-but\-smaller\(Section[5\.2](https://arxiv.org/html/2607.02893#S5.SS2)\)\. A 131M model at1\.821\.82bits \(29 MB\)*beats*a 55M FP16 model \(109 MB\) on perplexity, and the same recipe lets a1\.461\.46B model \(2\.5×2\.5\\timesmore parameters\) match a593593M FP16 control on FineWeb\-Edu at∼3\.7×\\sim\\\!3\.7\\timesless storage\.
4. 4\.Depth self\-heals quantization error\(Section[5\.3](https://arxiv.org/html/2607.02893#S5.SS3)\)\. A KL\-divergence decomposition shows that the representational gap injected by the early 4\-bit block is progressively reduced by later 2\-bit blocks; the per\-position argmax\-flip rate vs\. FP16 falls monotonically from71%71\\%to28%28\\%across depth\.
5. 5\.The compression is real at inference time\(Section[6](https://arxiv.org/html/2607.02893#S6)\)\. The recipe maps to packed\{1,2,4,8\}\\\{1,2,4,8\\\}\-bit storage that our fused dequantize\-and\-multiply kernels consume directly, never materializing FP16 weights\. Because autoregressive decode is memory\-bandwidth bound, the storage win becomes a speed win that grows with model size: from parity at131131M to4\.7×4\.7\\timesfaster decode at99B, at bit\-exact agreement, on Apple silicon\.

VBQ’s win is a*from\-scratch, train\-time*phenomenon rather than a free lunch for compressing arbitrary pretrained checkpoints; we make this scope explicit and report the relevant open problems \(chiefly economical scaling of the search beyond1\.51\.5B parameters\) in Section[7](https://arxiv.org/html/2607.02893#S7)\.

## 2Related Work

#### Low\-bit and ternary LLMs\.

BitNet\[[34](https://arxiv.org/html/2607.02893#bib.bib34)\]and BitNet b1\.58\[[21](https://arxiv.org/html/2607.02893#bib.bib21)\]train transformers with 1\-bit / ternary weights from scratch, fixing the same precision everywhere\. ParetoQ\[[20](https://arxiv.org/html/2607.02893#bib.bib20)\]unifies 1\- to 4\-bit QAT and finds that most of the training budget is best spent in full precision with a short low\-bit fine\-tune, a different regime from our purely from\-scratch QAT\. Low\-bit models have also gained traction at the deployment frontier: the Bonsai family\[[26](https://arxiv.org/html/2607.02893#bib.bib26)\]ships open\-weight 1\-bit and ternary \(1\.58\-bit\) models up to88B parameters with strong reported accuracy\-per\-byte for on\-device inference, obtained by quantization\-aware training to a single*uniform*low\-bit format\. We note it as evidence of real appetite for the “bigger\-but\-smaller” regime;VBQdiffers in*learning*a heterogeneous per\-group allocation, of which a uniform budget is a special case, so a successfully\-learned non\-uniform allocation can only match or improve on a uniform one at equal bytes \(we do not attempt a head\-to\-head comparison, as the scale, tokenizer, and evaluation suite differ\)\.VBQdiffers from all of these in that the bit\-width is*not*a global constant but a per\-group learned quantity\.

#### Post\-training quantization \(PTQ\)\.

GPTQ\[[13](https://arxiv.org/html/2607.02893#bib.bib13)\], AWQ\[[18](https://arxiv.org/html/2607.02893#bib.bib18)\], SmoothQuant\[[37](https://arxiv.org/html/2607.02893#bib.bib37)\], QuIP\#\[[32](https://arxiv.org/html/2607.02893#bib.bib32)\]and AQLM\[[10](https://arxiv.org/html/2607.02893#bib.bib10)\]compress an already\-trained model using calibration data, second\-order information, or learned codebooks\. LLM\.int8\(\)\[[7](https://arxiv.org/html/2607.02893#bib.bib7)\]keeps a few outlier channels in higher precision via a fixed threshold\. Production toolchains increasingly ship*per\-layer*dynamic mixed precision in this PTQ regime: Unsloth’s dynamic GGUF quants\[[33](https://arxiv.org/html/2607.02893#bib.bib33)\]leave salient layers at higher precision via an importance heuristic while pushing the rest to low bit\-widths: a post\-training, per\-tensor analogue of the per\-group allocationVBQlearns during training\. These are complementary toVBQ: they operate*after*training and at \(mostly\) fixed, per\-layer bit\-widths, whereas we learn the allocation*during*training at per\-group granularity\.

#### Mixed\-precision search\.

Learning or searching per\-layer bit\-widths is well studied for CNNs: HAQ\[[35](https://arxiv.org/html/2607.02893#bib.bib35)\]\(RL\), HAWQ\[[8](https://arxiv.org/html/2607.02893#bib.bib8)\]\(Hessian\), and the differentiable DNAS / EdMIPS family\[[36](https://arxiv.org/html/2607.02893#bib.bib36),[3](https://arxiv.org/html/2607.02893#bib.bib3)\]\. For LLMs, MixLLM\[[38](https://arxiv.org/html/2607.02893#bib.bib38)\]and SliM\-LLM\[[15](https://arxiv.org/html/2607.02893#bib.bib15)\]allocate precision across output features or groups, but as*post\-training*salience heuristics\. The closest in spirit is the differentiable, Gumbel\-based CNN search ofWu et al\.\[[36](https://arxiv.org/html/2607.02893#bib.bib36)\];VBQbrings this idea to decoder\-only LLMs, at*per\-group*\(64\-weight\) granularity, with gradient isolation and alternating optimization that are necessary to make training\-time selection work at this scale\. To our knowledge, no prior work learns per\-group bit\-widths jointly during training for LLMs\.

#### Concurrent work on differentiable quantization\.

Two recent papers apply differentiable discrete relaxations to LLM precision in*post\-training*settings, and clarify what is distinct aboutVBQ\. GSQ\[[5](https://arxiv.org/html/2607.02893#bib.bib5)\]uses a Gumbel\-Softmax relaxation to jointly learn per\-coordinate scalar\-grid assignments and per\-group scales of an*already\-trained*model, matching trellis\-quantized accuracy while staying in a deployment\-friendly scalar format; it optimizes a layer\-wise reconstruction loss rather than the task loss, and selects grid points rather than power\-of\-two bit\-widths\. dMX\[[12](https://arxiv.org/html/2607.02893#bib.bib12)\]learns*per\-layer*floating\-point \(OCP MX\) formats via a continuous offset annealed to discrete formats\.VBQis complementary on three axes: it learns*integer*bit\-widths at*per\-group*\(64\-weight\) granularity,*from scratch*against the task loss, and distills the result into a fixed recipe; the shared lesson across all three is that annealed differentiable relaxations are an effective way to make discrete precision decisions trainable\.

#### Scaling laws and the practical precision floor\.

Kumar et al\.\[[17](https://arxiv.org/html/2607.02893#bib.bib17)\]andCao et al\.\[[4](https://arxiv.org/html/2607.02893#bib.bib4)\]formalize how parameter count trades against precision, andDettmers and Zettlemoyer\[[6](https://arxiv.org/html/2607.02893#bib.bib6)\]studykk\-bit inference scaling and find44\-bit Pareto\-optimal\. This∼4\\sim\\\!4\-bit regime is borne out in production: Apple’s on\-device foundation model\[[1](https://arxiv.org/html/2607.02893#bib.bib1)\]compresses to a*mixed*2\-/4\-bit scheme averaging3\.73\.7bits\-per\-weight that matches the uncompressed model \(and3\.53\.5bits without significant quality loss\), a deployed mixed\-precision system landing at exactly the precision floor that also motivatesVBQ\(Section[3](https://arxiv.org/html/2607.02893#S3)\)\. Hardware is converging on the same regime: NVIDIA’s NVFP4\[[23](https://arxiv.org/html/2607.02893#bib.bib23)\]is a44\-bit floating\-point format with two\-level micro\-block scaling \(a per\-1616\-value FP8 scale and a per\-tensor FP32 scale\) that holds under1%1\\%accuracy loss versus FP8 on some models, but it fixes a single44\-bit format across the network rather than learning where precision is needed\. Our “bigger\-but\-smaller” results are a concrete instantiation of this trade in the per\-group mixed\-precision setting: rather than round each weight to the∼4\\sim\\\!4\-bit floor, spend a comparable budget on more, lower\-precision parameters\.

#### Evaluation beyond perplexity\.

Dutta et al\.\[[9](https://arxiv.org/html/2607.02893#bib.bib9)\]show that accuracy alone hides large behavioral changes from compression, and advocate KL divergence and*flips*\(answers changing\) as distance metrics\. We adopt both to auditVBQ\(Section[5\.3](https://arxiv.org/html/2607.02893#S5.SS3)\)\.

## 3Method

### 3\.1Per\-group quantization

A weight matrixWWis partitioned into contiguous groups ofG=64G=64weights\. Each groupgghas a symmetric absmax scalesgs\_\{g\}and a bit\-widthbgb\_\{g\}\. For a target bit\-widthbb, the quantizer is

sg=maxi∈g⁡\|wi\|2b−1−1,w^i=sg⋅clip​\(round​\(wi/sg\),−\(2b−1−1\),2b−1−1\)\.s\_\{g\}=\\frac\{\\max\_\{i\\in g\}\|w\_\{i\}\|\}\{2^\{\\,b\-1\}\-1\},\\qquad\\hat\{w\}\_\{i\}=s\_\{g\}\\cdot\\mathrm\{clip\}\\\!\\Big\(\\mathrm\{round\}\(w\_\{i\}/s\_\{g\}\),\\,\-\(2^\{\\,b\-1\}\-1\),\\,2^\{\\,b\-1\}\-1\\Big\)\.\(1\)The 1\-bit case is handled as sign with a per\-group absmean scale,w^i=sign​\(wi\)⋅meanj∈g​\|wj\|\\hat\{w\}\_\{i\}=\\mathrm\{sign\}\(w\_\{i\}\)\\cdot\\mathrm\{mean\}\_\{j\\in g\}\|w\_\{j\}\|\. Gradients pass through the round/clip via the straight\-through estimator\[[2](https://arxiv.org/html/2607.02893#bib.bib2)\]\.

#### Why group size matters\.

WithG=1G\{=\}1the scheme is degenerate: for any single weight,w^=round​\(w/s\)⋅s=w\\hat\{w\}=\\mathrm\{round\}\(w/s\)\\cdot s=wregardless ofbb, so all bit\-widths produce identical outputs and there is nothing to choose\. A shared per\-group scale \(G=64G\{=\}64\) is what creates a genuine quality/precision trade\-off and, empirically, the heterogeneous allocation we observe \(Section[5\.1](https://arxiv.org/html/2607.02893#S5.SS1)\)\.

### 3\.2Differentiable bit selection

Each group carries logitsℓg∈ℝK\\bm\{\\ell\}\_\{g\}\\in\\mathbb\{R\}^\{K\}over theK=4K\{=\}4candidatesℬ=\{1,2,4,8\}\\mathcal\{B\}=\\\{1,2,4,8\\\}\. During the selection path we draw a Gumbel\-Softmax sample𝝅g=softmax​\(\(ℓg\+𝒆g\)/τ\)\\bm\{\\pi\}\_\{g\}=\\mathrm\{softmax\}\(\(\\bm\{\\ell\}\_\{g\}\+\\bm\{e\}\_\{g\}\)/\\tau\)with Gumbel noise𝒆g\\bm\{e\}\_\{g\}and temperatureτ\\tauannealed from5\.05\.0to0\.50\.5, and form the soft\-quantized weight as a convex combination of the candidate reconstructions,w^isoft=∑kπg,k​w^i\(ℬk\)\\hat\{w\}^\{\\text\{soft\}\}\_\{i\}=\\sum\_\{k\}\\pi\_\{g,k\}\\,\\hat\{w\}^\{\(\\mathcal\{B\}\_\{k\}\)\}\_\{i\}\. The expected bit cost of the network isb¯=1N​∑g∑kπg,k​ℬk\\bar\{b\}=\\frac\{1\}\{N\}\\sum\_\{g\}\\sum\_\{k\}\\pi\_\{g,k\}\\,\\mathcal\{B\}\_\{k\}, and the training objective isℒ=ℒCE\+λ​b¯\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{CE\}\}\+\\lambda\\,\\bar\{b\}\.

#### What the objective optimizes \(and whyVBQis not lossless at fixed size\)\.

ℒ\\mathcal\{L\}is a*penalty*\(Lagrangian\) objective, not a quality\-constrained one:λ\\lambdais an exchange rate that trades cross\-entropy for bits, and each group is compressed until the marginal quality cost of one fewer bit exceedsλ\\lambda\. We deliberately chooseλ\\lambdaso the mean lands at∼1\.8\\sim\\\!1\.8bits,*below*the3\.53\.5–44\-bit information floor discussed in the introduction\. At a*fixed*parameter countVBQis therefore*expected*to be worse than an FP16 model: pushing precision below the floor genuinely discards information, and this is by design, not a mis\-tuned selection\. The capacity is recovered not by squeezing each weight but by*layout*: spending the same total bit budget across more, lower\-precision weights that the network learns to use natively \(the “bigger\-but\-smaller” result\)\. This depends on weights*co\-adapting*to their assigned bit\-widths*during*training; it cannot be obtained by lowering the bits\-per\-weight of an already\-trained model, which merely forces a lossy approximation of weights that were never trained for it\. In other words,VBQdoes not compress a trained network to fewer bits; it starts from a more efficient bit*layout*chosen for the target accuracy and storage\.

### 3\.3Gradient isolation and alternating optimization

The naive objective does not work: the cross\-entropy gradient reachingℓg\\bm\{\\ell\}\_\{g\}throughw^soft\\hat\{w\}^\{\\text\{soft\}\}is∼16×\\sim\\\!16\\timeslarger than the penalty gradient*at every temperature*, and Adam’s normalization entrenches whichever logit started largest\. The logits never move\.

We make two changes\. \(i\)Gradient isolation: in the bit\-selection path the candidate reconstructions are detached from the weight values \(w^\(ℬk\)=detach​\(⋅\)\\hat\{w\}^\{\(\\mathcal\{B\}\_\{k\}\)\}=\\texttt\{detach\}\(\\cdot\)\), so cross\-entropy cannot back\-propagate intoℓg\\bm\{\\ell\}\_\{g\}; only the bit penalty trains the precision decision\. \(ii\)Alternating optimization\(cf\. DARTS\[[19](https://arxiv.org/html/2607.02893#bib.bib19)\]\): we interleave a*weight step*\(updateWWonℒCE\\mathcal\{L\}\_\{\\text\{CE\}\}with logits frozen\) and a*logit step*\(updateℓ\\bm\{\\ell\}onℒCE\+λ​b¯\\mathcal\{L\}\_\{\\text\{CE\}\}\+\\lambda\\bar\{b\}withWWfrozen\)\. This gives the logits a clean, task\-aligned signal about how precision affects loss, rather than the contaminated gradient that froze them\. With both changes the bit distribution collapses from near\-uniform to a committed \(entropy<1%<\\\!1\\%\) heterogeneous allocation\. Algorithm[1](https://arxiv.org/html/2607.02893#alg1)makes the two steps precise\.

Algorithm 1VBQalternating optimization \(one logit update everymmweight steps\)1:weights

WW, precision logits

ℓ\\bm\{\\ell\}, candidates

ℬ=\{1,2,4,8\}\\mathcal\{B\}=\\\{1,2,4,8\\\}, penalty

λ\\lambda, temperature

τt\\tau\_\{t\}, interval

mm
2:foreach training step

ttdo

3:

w^\(ℬk\)←quant​\(W,ℬk\)\\hat\{w\}^\{\(\\mathcal\{B\}\_\{k\}\)\}\\leftarrow\\texttt\{quant\}\(W,\\mathcal\{B\}\_\{k\}\)for each

kk⊳\\trianglerightper\-group reconstructions, Eq\. \(1\)

4:if

tmodm≠0t\\bmod m\\neq 0then⊳\\trianglerightweight step: logits frozen

5:

𝝅←detach​\(GumbelSoftmax​\(ℓ,τt\)\)\\bm\{\\pi\}\\leftarrow\\texttt\{detach\}\\big\(\\mathrm\{GumbelSoftmax\}\(\\bm\{\\ell\},\\tau\_\{t\}\)\\big\)
6:

w^←∑kπk​w^\(ℬk\)\\hat\{w\}\\leftarrow\\sum\_\{k\}\\pi\_\{k\}\\,\\hat\{w\}^\{\(\\mathcal\{B\}\_\{k\}\)\};

W←W−η​∇WℒCE​\(w^\)\\;W\\leftarrow W\-\\eta\\,\\nabla\_\{W\}\\,\\mathcal\{L\}\_\{\\text\{CE\}\}\(\\hat\{w\}\)
7:else⊳\\trianglerightlogit step: weights frozen,

w^\(ℬk\)\\hat\{w\}^\{\(\\mathcal\{B\}\_\{k\}\)\}detached

8:

𝝅←GumbelSoftmax​\(ℓ,τt\)\\bm\{\\pi\}\\leftarrow\\mathrm\{GumbelSoftmax\}\(\\bm\{\\ell\},\\tau\_\{t\}\);

b¯←1N​∑g∑kπg,k​ℬk\\bar\{b\}\\leftarrow\\tfrac\{1\}\{N\}\\sum\_\{g\}\\sum\_\{k\}\\pi\_\{g,k\}\\,\\mathcal\{B\}\_\{k\}
9:

ℓ←ℓ−η​∇ℓ\(ℒCE​\(sg​\[w^\]\)\+λ​b¯\)\\bm\{\\ell\}\\leftarrow\\bm\{\\ell\}\-\\eta\\,\\nabla\_\{\\bm\{\\ell\}\}\\big\(\\mathcal\{L\}\_\{\\text\{CE\}\}\(\\texttt\{sg\}\[\\hat\{w\}\]\)\+\\lambda\\,\\bar\{b\}\\big\)
10:endif

11:endfor

12:returnhardened bit\-widths

bg=ℬarg⁡maxk⁡ℓg,kb\_\{g\}=\\mathcal\{B\}\_\{\\arg\\max\_\{k\}\\ell\_\{g,k\}\}

Heresg​\[⋅\]\\texttt\{sg\}\[\\cdot\]\(≡detach\\equiv\\texttt\{detach\}\) is the stop\-gradient that implements gradient isolation: in the logit step the reconstructions are held constant so the only path intoℓ\\bm\{\\ell\}is throughb¯\\bar\{b\}and the explicit dependence of𝝅\\bm\{\\pi\}onℓ\\bm\{\\ell\}\.

### 3\.4From discovery to a fixed recipe

Because the allocation locks early \(Section[5\.1](https://arxiv.org/html/2607.02893#S5.SS1)\), production training does not need the Gumbel machinery at all\. We distill the discovered pattern into a fixedVBQrecipe:

- •First block MLP \(gate/up/down\):4 bit\(most sensitive layer\)\.
- •All other attention and MLP projections:2 bit\.
- •Language\-model head: frequency\-tiered per\-row mixed precision\. We count token frequencies in the training corpus, rank the vocabulary in descending order of frequency, and assign bit\-widths by rank percentile: top0\.5%0\.5\\%of tokens at8 bit, next1\.5%1\.5\\%at4 bit, next10%10\\%at2 bit, bottom88%88\\%at1 bit\(mean1\.181\.18bits\)\. These four cutoffs are not tuned per run; they are chosen once to reproduce the per\-row allocation that the adaptive search discovers for the head \(Section[5\.1](https://arxiv.org/html/2607.02893#S5.SS1)\)\.

The recipe is the single most transferable artifact of the method: it requires only a token\-frequency table for the head \(a single pass over the corpus\) and a one\-line projection\-name map for a new architecture\.

## 4Experimental Setup

#### Models and data\.

The discovery and ablation studies use nanoGPT\-style decoder\-only transformers \(RoPE\[[31](https://arxiv.org/html/2607.02893#bib.bib31)\], SwiGLU\[[30](https://arxiv.org/html/2607.02893#bib.bib30)\], weight\-tied head\) trained on TinyStories\[[11](https://arxiv.org/html/2607.02893#bib.bib11)\]\(200M tokens, GPT\-2 tokenizer\[[28](https://arxiv.org/html/2607.02893#bib.bib28)\]\)\. Scale\-up experiments use FineWeb\-Edu\[[25](https://arxiv.org/html/2607.02893#bib.bib25)\]\(2B tokens, Qwen tokenizer\[[27](https://arxiv.org/html/2607.02893#bib.bib27)\]\)\. Sizes range from 55M to 1\.46B parameters\.

#### Training\.

AdamW \(β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), weight decay0\.10\.1\), cosine schedule3×10−4→3×10−53\\\!\\times\\\!10^\{\-4\}\\\!\\to\\\!3\\\!\\times\\\!10^\{\-5\}with warmup, gradient clip1\.01\.0\. TinyStories runs use3535K iterations\. Storage is reported as the packed weight size at the realized mean bit\-width \(embedding/head included\); “FP16” baselines store every weight in 16 bits\. We report validation perplexity \(PPL\), and for the distributional study, mean token\-level KL divergence to an FP16 reference and the argmax\-flip rate\.

## 5Results

### 5\.1A learned, heterogeneous precision hierarchy

The adaptive pilot \(58M, 8 layers, TinyStories\) converges to a mean of1\.781\.78bits with a sharply heterogeneous distribution \(Figure[1](https://arxiv.org/html/2607.02893#S5.F1)a\):69\.3%69\.3\\%of groups choose1 bit,20\.5%20\.5\\%2 bit,6\.7%6\.7\\%4 bit,3\.5%3\.5\\%8 bit, at near\-zero selection entropy\. The allocation follows a stable*projection\-type hierarchy*\(Figure[1](https://arxiv.org/html/2607.02893#S5.F1)b\): the language\-model head averages1\.091\.09bits \(92\.5%92\.5\\%of its rows are pure1 bit\), attention output projections are the cheapest in\-block projection, and the SwiGLU gate/up projections of the*first*block are the hungriest \(up to2\.52\.5bits with∼10%\\sim\\\!10\\%8 bitgroups\)\. Crucially, the heterogeneity exists*within*a projection type, not only across layers, a degree of freedom no per\-layer method can use\.

The head’s allocation is semantically interpretable:4\.8%4\.8\\%of the10001000most\-frequent vocabulary rows contain8 bitgroups versus0\.06%0\.06\\%of the rarest, and the high\-precision rows are dominated by function words and punctuation\. The same frequency structure is what motivates the recipe’s frequency\-tiered head \(Section[3\.4](https://arxiv.org/html/2607.02893#S3.SS4)\)\. Finally, the allocation*locks*by∼10\\sim\\\!10K of5050K steps; the remaining80%80\\%of training only tunes weights within fixed precision, empirical justification for prescribing the recipe directly\.

#### The discovery transfers to a different architecture \(DeltaNet\)\.

To test whether this hierarchy is an artifact of standard transformers, we re\-ran the same Gumbel search on Qwen3\.5’s hybrid architecture, in which75%75\\%of the token\-mixing layers are Gated DeltaNet \(linear attention\) with non\-standard projections \(in\_proj\_qkv,in\_proj\_z,linear\_attn\.out\_proj\) rather than softmax attention\. The same projection\-type hierarchy emerges, and, counter to our worry that an untested layer family might need*more*precision, the DeltaNet projections turn out to be the*most*compressible in the model:in\_proj\_qkvselects1 bitfor95\.7%95\.7\\%of its groups, while the precision\-hungry projections are again the value and MLP up/down projections \(∼70%\\sim\\\!70\\%1 bit,∼30%\\sim\\\!30\\%2 bit\)\. The ranking is stable across checkpoints and no group demands44/8 bitby mid\-training\. The practical payoff is that the recipe needs*no*architecture\-specific retuning: extending it to the new projection names and placing DeltaNet layers at2 bitis, if anything, conservative\. This discovery run fixes the*layout*; its*quality*is then validated by the controlled1\.01\.0B from\-scratch comparison against a matched FP16 control \(Section[7](https://arxiv.org/html/2607.02893#S7)\), and the same packed model supplies the Qwen3\.5 inference measurements \(Section[6](https://arxiv.org/html/2607.02893#S6)\)\.

![Refer to caption](https://arxiv.org/html/2607.02893v1/x1.png)Figure 1:VBQlearns a heterogeneous precision allocation\.\(a\) Global distribution of learned bit\-widths \(mean1\.781\.78, selection entropy<1%<1\\%\)\. \(b\) Mean bit\-width per projection type: the language\-model head collapses to near\-binary while the first\-block MLP stays\>2\>2bits\. The hierarchy is discovered, not hand\-set\.

### 5\.2From adaptive to prescribed, and “bigger\-but\-smaller”

Table[1](https://arxiv.org/html/2607.02893#S5.T1)shows the TinyStories scaling ladder\. Three points stand out\. First,*the search is a one\-time cost, not a per\-model one*: the discovered layout, frozen into the fixed recipe, equals or beats the adaptive pilot with no Gumbel machinery at all\. This is the practical crux of the method: the expensive part is*discovering where the bits should go*, and that needs to happen only once\. Once the allocation pattern is known, any new model can be trained at low precision by simply reading off the recipe \(a per\-projection bit map plus a token\-frequency table for the head\); no search, no relaxation, no extra hyper\-parameters at train time\. Every result below the pilot row in Table[1](https://arxiv.org/html/2607.02893#S5.T1), and the entire1\.461\.46B FineWeb\-Edu scale\-up, uses the prescribed recipe directly rather than re\-running the search\. Second, the*frequency\-tiered head is the single most impactful component*: switching the head from uniform1 bit\(008a\) to frequency\-tiered \(008f\) improves PPL by0\.60\.6for only\+0\.09\+0\.09mean bits\. Third, and headline: scaling the model up2\.4×2\.4\\timeswhile staying at1\.821\.82bits yields PPL4\.24\.2, which*beats*the 55M FP16 baseline \(PPL4\.44\.4\) at3\.8×3\.8\\timesless storage \(Figure[2\(a\)](https://arxiv.org/html/2607.02893#S5.F2.sf1)\)\. EveryVBQpoint lies on or below the FP16 Pareto frontier\.

Table 1:TinyStories scaling\.A 131MVBQmodel at1\.821\.82bits beats a 55M FP16 model on perplexity at3\.8×3\.8\\timesless storage\. “Δ\\DeltaPPL” is relative to the 55M FP16 baseline\.ModelParamsMean bitsVal PPLStorageΔ\\DeltaPPL vs\. 55M FP1655M FP16 \(baseline\)55M16\.04\.4109 MBn/a131M FP16 \(quality ceiling\)131M16\.03\.9262 MB−11%\-11\\%VBQadaptive pilot55M1\.785\.112 MB\+16%\+16\\%VBQuniform2 bit55M1\.505\.510 MB\+25%\+25\\%VBQmixed head55M1\.594\.911 MB\+11%\+11\\%VBQ1\.3×1\.3\\timesscale73M1\.664\.915 MB\+11%\+11\\%VBQ2\.4×2\.4\\timesscale131M1\.824\.229 MB−𝟓%\\bm\{\-5\\%\}![Refer to caption](https://arxiv.org/html/2607.02893v1/x2.png)\(a\)Quality vs\. storage frontier\.
![Refer to caption](https://arxiv.org/html/2607.02893v1/x3.png)\(b\)Training crossover \(real eval logs\)\.

Figure 2:Bigger\-but\-smaller\.\(a\) Validation perplexity vs\. packed storage on TinyStories: everyVBQmodel \(crimson stars\) sits on or below the FP16 baselines \(blue circles\); the131131MVBQmodel at2929MB beats the5555M FP16 model at109109MB\. \(b\) Validation perplexity over training for three fixed\-architecture3535K\-iteration runs\. Faint markers are the raw per\-eval points; solid lines trace the best perplexity reached so far \(a running minimum\), which is robust to the transient single\-eval spikes visible in the scatter\. The131131MVBQmodel \(1\.821\.82bits\) converges more slowly than FP16 but keeps improving after the5555M FP16 model plateaus, crossing its quality line \(dotted\) at∼46%\\sim\\\!46\\%of training and finishing at PPL4\.24\.2vs\.4\.44\.4\.#### Quantization as regularization\.

Figure[2\(b\)](https://arxiv.org/html/2607.02893#S5.F2.sf2)plots the real validation curves\.VBQconverges*slower*than FP16 but keeps improving after FP16 plateaus, crossing the 55M\-FP16 quality line near46%46\\%of training\. We attribute this to the low\-bit constraint acting as a regularizer: with dropout0, the 131MVBQmodel shows a smaller train/val gap than its FP16 twin\. A practical consequence is thatVBQruns must not be judged early\.

#### Scaling to 1\.46B\.

On FineWeb\-Edu with the Qwen tokenizer, the same recipe applied to a1\.461\.46B\-parameter model \(1\.771\.77mean bits\) reaches a best validation loss of2\.9642\.964\(PPL19\.419\.4\), matching the593593M FP16 control’s best PPL of19\.419\.4\. The match holds at∼3\.7×\\sim\\\!3\.7\\timesless storage \(0\.320\.32vs1\.191\.19GB\) with2\.5×2\.5\\timesmore parameters\. The recipe keeps the frequency\-tiered head \(mean1\.181\.18bits\) here too, and its importance is quantified by the TinyStories ablation above: replacing it with a uniform1 bithead costs0\.60\.6PPL for a saving of only\+0\.09\+0\.09mean bits\. What matters is not the head’s*average*bit budget but*where*those bits go\.

#### Quality\-per\-byte\.

Summarizing with an efficiency metric\(1/PPL\)/storage \(GB\)\(1/\\text\{PPL\}\)/\\text\{storage\\,\(GB\)\},VBQmodels are3\.93\.9–8\.4×8\.4\\timesmore efficient than FP16 across the ladder \(Figure[3](https://arxiv.org/html/2607.02893#S5.F3)\); the mixed\-head 55M model is the most efficient point overall, and there is no overlap between theVBQand FP16 clusters\.

![Refer to caption](https://arxiv.org/html/2607.02893v1/x4.png)Figure 3:Quality\-per\-byte\(1/PPL\)/storage \(GB\)\(1/\\text\{PPL\}\)/\\text\{storage\\,\(GB\)\}\.VBQmodels \(crimson\) and FP16 baselines \(blue\) do not overlap; the iso\-parameter 131M comparison is8\.4×8\.4\\times\.

### 5\.3Distributional analysis: depth self\-heals quantization error

Perplexity can hide behavioral change\[[9](https://arxiv.org/html/2607.02893#bib.bib9)\]\. We therefore measure mean token\-level KL divergence and argmax\-flip rate of the 131MVBQmodel against a 131M FP16 reference\. The full model has mean KLD0\.210\.21nats and an18\.7%18\.7\\%flip rate: at roughly one position in fiveVBQselects a different top\-1 token, a behavioral cost invisible to PPL, which we report for honesty\.

Two findings refine the picture\. First, swapping components between theVBQand FP16 models makes things*worse*\(an FP16 head onVBQblocks yields KLD0\.620\.62,3×3\\timesthe fullVBQmodel\), showing the blocks and head*co\-adapt*: the low\-bit head works precisely because the blocks learned to produce representations robust to sign\-only quantization\. Second, a depth probe that projects each block’s hidden state through a single shared FP16 head \(excludingVBQ’s own co\-adapted head, so blocks are comparable across depth\) shows the representational gap is injected early: the44\-bit first block contributes the largest KLD \(0\.830\.83,71%71\\%flips\) and is then*progressively reduced by later22\-bit blocks*, the probe’s flip rate falling monotonically from71%71\\%to28%28\\%at the final block \(Figure[4](https://arxiv.org/html/2607.02893#S5.F4)\)\. This probe sits above the full model’s18\.7%18\.7\\%end\-to\-end flip rate precisely because it bypassesVBQ’s co\-adapted head; restoring that head recovers the18\.7%18\.7\\%\.VBQnetworks effectively learn an error\-correcting code over depth\. This also explains a negative ablation: adding precision to individual blocks barely moves KLD \(<4\.4%<\\\!4\.4\\%for\+0\.34\+0\.34bits\), because the co\-adaptation absorbs whatever precision it is given, making*more blocks*, not more bits per block, the natural lever\.

![Refer to caption](https://arxiv.org/html/2607.02893v1/x5.png)Figure 4:Self\-healing through depth\.Cumulative KL divergence to FP16 \(crimson\) and argmax\-flip rate \(blue\) measured by projecting each block’s hidden state through a shared head\. The early 4\-bit block injects the largest gap; later 2\-bit blocks progressively correct it\.
### 5\.4Depth vs\. width

The self\-healing mechanism suggests depth should be a cheap quality lever, since each extra22\-bit block is nearly free in storage\. We tested this directly at fixed width \(d=512d\{=\}512\): adding depth helps consistently but*sub\-linearly*\. A 12\-layer model improves PPL by55–6%6\\%over 8 layers \(4\.9→4\.64\.9\\\!\\to\\\!4\.6\), but a 26\-layer model plateaus at4\.774\.77, far short of the∼4\.3\\sim\\\!4\.3a naive depth extrapolation predicts, and worse per parameter than going*wider*\. Pure depth is therefore not a substitute for width: the headline131131M result wins by being both wider*and*deeper, not deeper alone\.

## 6Inference: storage savings become speed

A learned bit budget is only useful if it survives to deployment\. Because autoregressive decode at batch size 1 is*memory\-bandwidth bound*\(each generated token must stream every weight from memory through the arithmetic units\), a model that stores its weights in fewer bits also*reads*fewer bytes per token, and decodes faster in direct proportion, provided the weights are never expanded to FP16 in memory\.

![Refer to caption](https://arxiv.org/html/2607.02893v1/x6.png)Figure 5:The inference win compounds with scale\.Measured single\-stream decode speedup of the packedVBQmodel over the same model dequantized to dense FP16 \(Apple M4 Max, batch 1\), plotted against parameter count on a log axis\. At131131M the model is too small for weight traffic to dominate andVBQruns at FP16 parity \(0\.95×0\.95\\times\); by1\.01\.0B it is1\.86×1\.86\\timesand by99B it is4\.68×4\.68\\timesfaster\. The dashed line is an illustrative log\-linear guide through the three measured points \(not a fitted scaling law\), included only to show the direction of the trend\. The mechanism is memory bandwidth: as models grow, decode spends proportionally more time streaming weights, so reading∼6×\\sim\\\!6\\timesfewer bytes per token converts more directly into wall\-clock speed\. This suggests that the combinationVBQtargets \(more parameters,∼3\.7×\\sim\\\!3\.7\\timessmaller storage,*and*faster decode\) becomes more attractive precisely at the larger scales where inference cost matters most, though we caution that we measure only up to99B and do not claim a specific extrapolated multiplier beyond that range\.#### Fused kernels over packed weights\.

We implement custom fused*dequantize\-and\-multiply*GEMV kernels \(Metal for Apple silicon, CUDA for NVIDIA\) with a specialization per bit bucket\{1,2,4,8\}\\\{1,2,4,8\\\}\. Each kernel loads the packed low\-bit bytes for a group, expands them to FP16*in registers*using the per\-group scale, and accumulates the matrix–vector product, so the full\-precision weight matrix is never materialized in memory and the bandwidth saving is realized end to end\. The mixed\-precision head is stored as a small number of frequency\-grouped row buckets\.

#### The speedup grows with model size\.

Because the benefit is bandwidth\-driven, it appears only once weight traffic dominates; i\.e\., it*scales with model size*\(Figure[5](https://arxiv.org/html/2607.02893#S6.F5)\)\. Table[2](https://arxiv.org/html/2607.02893#S6.T2)reports the same\-weights comparison \(identical model and runtime; the FP16 baseline is produced by dequantizing the VBQ weights into dense FP16, so only the weight representation differs\) at batch 1 on an Apple M4 Max, as the median of repeated runs\. At131131M the model is small enough that per\-token overhead, not bandwidth, dominates, soVBQruns at parity \(0\.95×0\.95\\times\) and the win is purely the∼2×\\sim\\\!2\\timessmaller memory footprint\. By1\.01\.0B decode is1\.86×1\.86\\timesfaster, and by99B it is4\.68×4\.68\\timesfaster \(135135vs\.2929tok/s\), while also fitting in3\.83\.8GB instead of1717GB of peak memory, i\.e\. running comfortably where the FP16 model is far heavier\. The1\.01\.0B/99B runs use mlx\-lm’sgenerate\_step\(the realasync\_evaldecode path\); the131131M run uses our fused\-kernel greedy decode\.VBQreproduces the dense\-FP16 model’s outputs to numerical parity \(verified separately: Pearson correlation1\.000001\.00000, identical top\-1 token\), so these are speedups at equal output, not an accuracy/speed trade\.

Table 2:Compressed decode on an Apple M4 Max\(batch 1, median of repeats\)\. The FP16 baseline is the*same model*with its quantized linear layers dequantized to dense FP16, isolating the cost of the packed\-vs\-dense weight representation \(the token embedding is left quantized in both\)\. “Linear weights” is the byte size the decode kernels must stream; decode is memory\-bandwidth bound, so the speedup tracks that saving and grows with model size\.The99B row benchmarks decode speed only; the quality of that particular from\-scratch run is a separate, unresolved matter \(Section[7](https://arxiv.org/html/2607.02893#S7)\), but the packed model is architecturally representative, so it is a valid measurement of how the inference benefit scales\.

#### Granularity is deployment\-friendly\.

We quantize*contiguous*6464\-weight groups \(rather than per\-channel or per\-tensor\) because a single shared scale over a short contiguous run is what creates a local quality/precision trade\-off \(Section[3](https://arxiv.org/html/2607.02893#S3)\) while still packing into aligned byte boundaries\. Two practical details keep the packed model fast\. First, the deployed artifact is the*recipe*, which is near\-uniform per projection type \(mostly2 bit, a4 bitfirst block, a tiered head\) rather than a fully heterogeneous per\-group map, so a kernel dispatches one bit bucket per projection, not one per group\. Second, the fastest decode GEMV requires the contracted dimension to be a multiple of512512; we zero\-pad MLP intermediates to satisfy this \(the padded channels are mathematically inert:silu​\(0\)⋅u=0\\mathrm\{silu\}\(0\)\\cdot u=0\), recovering a1\.67×1\.67\\timeskernel speedup that would otherwise be lost on awkward shapes such as Qwen’s89608960\-wide MLP\. An export\-time check validates this contract\.

## 7Limitations and Open Problems

We state the method’s scope plainly, and separate what we have*shown*from what remains*open*\.

#### Economical scaling beyond 1\.5B is the main open problem\.

Our positive quality results run from 55M to1\.461\.46B parameters; economically validating the recipe beyond that remains open\. A separate controlled1\.01\.0B gate run on Qwen3\.5’s hybrid architecture \(1006\.71006\.7M total parameters\) used a different tokenizer, a matched FP16 control, and≥99%\\geq\\\!99\\%quantization coverage, but was deliberately short at0\.370\.37tokens/parameter, about4×4\\timesfewer than the∼1\.4\\sim\\\!1\.4used in the matched FineWeb runs\. This was by design: it is a cheap*transfer probe*to check that the recipe carries to a hybrid architecture at all, not a run to convergence; full matched runs at this scale are precisely the economics we flag as unsolved\. Even so, at the stopped checkpointVBQwas\+0\.28\+0\.28nats behind FP16 \(1\.33×1\.33\\timesPPL\) with the gap still narrowing over the final third \(0\.36→0\.280\.36\\\!\\to\\\!0\.28nats\), consistent withVBQ’s known slower convergence and showing no sign of a scaling ceiling\. This1\.01\.0B gate is not numerically comparable to the1\.461\.46B FineWeb\-Edu standard\-transformer result \(architecture, tokenizer, and token budget all differ\), so it probes recipe transfer to a harder hybrid setting rather than contradicting the standard\-transformer parity result\. We also attempted a single99B from\-scratch knowledge\-distillation\[[14](https://arxiv.org/html/2607.02893#bib.bib14)\]run, but it was confounded and we do not draw conclusions from it: a coverage bug left18%18\\%of parameters \(non\-standard hybrid\-attention projections\) unquantized so the realized precision was not the intended1\.771\.77bits, and it was trained at only0\.270\.27tokens/parameter \(about5×5\\timesunder the∼1\.4\\sim\\\!1\.4used by our successful runs\) on an untested architecture family\. A properly\-tokened99B run \(estimated∼12\\sim\\\!12B tokens\) was beyond our compute budget\. Nothing in our results suggests the method*stops*scaling; closing the≥9\\geq\\\!9B regime economically, and clearing the strong\-baseline bar at that scale \(a44\-bit PTQ of a same\-storage teacher\), is future work rather than a counter\-result\.

#### Frozen\-base depth\-boosting does not add useful capacity\.

Motivated by self\-healing, we tried to*add*low\-bit correction blocks onto a converged 12\-layer model with the base frozen \(a gradient\-boosting analogy with near\-identity initialization\)\. Training loss improved but validation did not \(a classic overfit signature, with intermittent instability\); the added blocks fit batch idiosyncrasies rather than generalizable residual signal\. At least with a small frozen base, post\-hoc correction depth is not a shortcut to the from\-scratch win\. We suspect a larger pretrained base would*not*rescue this either: the “bigger\-but\-smaller” win comes from weights co\-adapting to low bit\-widths*during*training, which a frozen or otherwise pretrained base cannot inherit; and indeed, separate attempts to search per\-group bit\-widths post\-hoc on*frozen*pretrained weights were dominated by trivial uniform 4\-bit\. The more promising direction is therefore to scale the from\-scratch “bigger\-but\-smaller” recipe itself, not to adapt pretrained bases\.

#### Scope\.

Our positive results are on English next\-token modeling up to1\.51\.5B parameters, on two corpora, with perplexity and KL/flip metrics; we do not report downstream task suites \(MMLU/GSM8K/etc\.\) at scale\. The method targets the*train\-from\-scratch*setting; it is not, on current evidence, a superior*post\-training*compressor competing with GPTQ, AWQ, QuIP\#, or AQLM at44bits\.

## 8Conclusion

VBQtreats numerical precision as a per\-group, learnable resource rather than a global constant\. A Gumbel\-Softmax relaxation, made trainable by gradient isolation and alternating optimization, discovers a strongly heterogeneous bit\-width hierarchy that is stable enough to freeze into a reusable recipe\. The central lesson is that spending a fixed bit budget*unevenly*is materially better than spending it uniformly: with the recipe, models become simultaneously*larger*in parameter count and*smaller*in storage than their FP16 equivalents, and3\.93\.9–8\.4×8\.4\\timesmore efficient in quality\-per\-byte\. This is consistent with the∼4\\sim\\\!4\-bit compression floor that recurs across independent settings\[[6](https://arxiv.org/html/2607.02893#bib.bib6),[1](https://arxiv.org/html/2607.02893#bib.bib1),[29](https://arxiv.org/html/2607.02893#bib.bib29)\]:VBQreaches that floor not by rounding each weight to four bits, but by redistributing a comparable budget across more, lower\-precision weights\. This is not only a storage result: the same packed weights decode faster through fused low\-bit kernels, increasingly so with scale \(up to4\.7×4\.7\\timesat99B\), because decode is memory\-bandwidth bound\. The three wins compound rather than trade off: aVBQmodel can hold*more*parameters, occupy*about a quarter*of the storage, and decode*several times faster*than its FP16 counterpart at once, and because the speedup is bandwidth\-driven it grows with model size, so the combination is most valuable exactly where serving cost dominates\. A KL\-divergence analysis reveals*why*the aggressive bit budget is survivable: depth self\-heals the error injected by early layers, so the network effectively learns an error\-correcting code that lets a non\-uniform allocation pay off\. The remaining frontier is to push the from\-scratch “bigger\-but\-smaller” recipe economically past1\.51\.5B parameters\. Precision should not be treated merely as a post\-training tuning knob, but as a structural design input to training itself: choosing the bit layout up front yields faster, more efficient models that can operate closer to the quality/size Pareto frontier out of the box\.

#### Reproducibility\.

All figures are regenerated from the released training logs and tabulated results by the script inpaper/figures/\.

## References

- Apple \[2024\]Apple\.Introducing apple’s on\-device and server foundation models\.Apple Machine Learning Research,[https://machinelearning\.apple\.com/research/introducing\-apple\-foundation\-models](https://machinelearning.apple.com/research/introducing-apple-foundation-models), 2024\.On\-device model compressed via mixed 2\-/4\-bit palettization averaging 3\.7 bits\-per\-weight \(3\.5 bits\-per\-weight without significant quality loss\)\.
- Bengio et al\. \[2013\]Yoshua Bengio, Nicholas Léonard, and Aaron Courville\.Estimating or propagating gradients through stochastic neurons for conditional computation\.*arXiv preprint arXiv:1308\.3432*, 2013\.
- Cai and Vasconcelos \[2020\]Zhaowei Cai and Nuno Vasconcelos\.Rethinking differentiable search for mixed\-precision neural networks\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2020\.
- Cao et al\. \[2024\]Zeyu Cao, Cheng\-Chang Kao, Chao Wang, Tianhua Xiao, Ling Zhu, Yibo Wang, Yiren Lin, and Nicholas D\. Lane\.Scaling laws for mixed quantization in large language models\.*arXiv preprint arXiv:2410\.06722*, 2024\.
- Dadgarnia et al\. \[2026\]Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtic, and Dan Alistarh\.GSQ: Highly\-accurate low\-precision scalar quantization for LLMs via gumbel\-softmax sampling\.*arXiv preprint arXiv:2604\.18556*, 2026\.
- Dettmers and Zettlemoyer \[2023\]Tim Dettmers and Luke Zettlemoyer\.The case for 4\-bit precision: k\-bit inference scaling laws\.*International Conference on Machine Learning \(ICML\)*, 2023\.
- Dettmers et al\. \[2022\]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer\.LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Dong et al\. \[2019\]Zhen Dong, Zhewei Yao, Amir Gholami, Michael W\. Mahoney, and Kurt Keutzer\.HAWQ: Hessian aware quantization of neural networks with mixed\-precision\.In*IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2019\.
- Dutta et al\. \[2024\]Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee\.Accuracy is not all you need\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.arXiv:2407\.09141\.
- Egiazarian et al\. \[2024\]Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh\.Extreme compression of large language models via additive quantization\.*International Conference on Machine Learning \(ICML\)*, 2024\.
- Eldan and Li \[2023\]Ronen Eldan and Yuanzhi Li\.TinyStories: How small can language models be and still speak coherent english?*arXiv preprint arXiv:2305\.07759*, 2023\.
- Franco et al\. \[2026\]Giuseppe Franco, Ian Colbert, Pablo Monteagudo\-Lago, Felix Marty, and Nicholas Fraser\.dMX: Differentiable mixed\-precision assignment for low\-precision floating\-point formats\.*arXiv preprint arXiv:2606\.04115*, 2026\.
- Frantar et al\. \[2023\]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.GPTQ: Accurate post\-training quantization for generative pre\-trained transformers\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Hinton et al\. \[2015\]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- Huang et al\. \[2024\]Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi\.SliM\-LLM: Salience\-driven mixed\-precision quantization for large language models\.*arXiv preprint arXiv:2405\.14917*, 2024\.
- Jang et al\. \[2017\]Eric Jang, Shixiang Gu, and Ben Poole\.Categorical reparameterization with Gumbel\-Softmax\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- Kumar et al\. \[2025\]Tanishq Kumar, Zachary Ankner, Benjamin F\. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan\.Scaling laws for precision\.*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Lin et al\. \[2024\]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.AWQ: Activation\-aware weight quantization for on\-device LLM compression and acceleration\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2024\.
- Liu et al\. \[2019\]Hanxiao Liu, Karen Simonyan, and Yiming Yang\.DARTS: Differentiable architecture search\.In*International Conference on Learning Representations \(ICLR\)*, 2019\.
- Liu et al\. \[2025\]Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, Lin Xiao, Yuandong Tian, Bilge Soran, Raghuraman Krishnamoorthi, Tijmen Blankevoort, and Vikas Chandra\.ParetoQ: Improving scaling laws in extremely low\-bit LLM quantization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.arXiv:2502\.02631\.
- Ma et al\. \[2024\]Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei\.The era of 1\-bit LLMs: All large language models are in 1\.58 bits\.*arXiv preprint arXiv:2402\.17764*, 2024\.
- Maddison et al\. \[2017\]Chris J\. Maddison, Andriy Mnih, and Yee Whye Teh\.The concrete distribution: A continuous relaxation of discrete random variables\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- NVIDIA \[2025a\]NVIDIA\.Introducing NVFP4 for efficient and accurate low\-precision inference\.NVIDIA Technical Blog,[https://developer\.nvidia\.com/blog/introducing\-nvfp4\-for\-efficient\-and\-accurate\-low\-precision\-inference/](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/), 2025a\.4\-bit floating\-point format \(E2M1\) with two\-level micro\-block scaling: a per\-16\-value FP8 \(E4M3\) scale plus a per\-tensor FP32 scale; reports under 1% accuracy loss vs FP8 on some models via post\-training quantization\.
- NVIDIA \[2025b\]NVIDIA\.Pretraining large language models with NVFP4\.arXiv:2509\.25149, 2025b\.4\-bit pretraining of a 12B hybrid Mamba\-Transformer on 10T tokens; keeps a hand\-chosen subset of layers \(first two plus final eight\) in BF16, i\.e\. a fixed mixed\-precision layout rather than a learned per\-group allocation\.
- Penedo et al\. \[2024\]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf\.The FineWeb datasets: Decanting the web for the finest text data at scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track*, 2024\.arXiv:2406\.17557; introduces FineWeb\-Edu\.
- PrismML \[2026\]PrismML\.Bonsai: 1\-bit and ternary \(1\.58\-bit\) language models for on\-device inference\.[https://prismml\.com/news/ternary\-bonsai](https://prismml.com/news/ternary-bonsai), 2026\.Open\-weight 1\-bit and 1\.58\-bit ternary models up to 8B parameters; whitepapers at[https://github\.com/PrismML\-Eng/Bonsai\-demo](https://github.com/PrismML-Eng/Bonsai-demo)\.
- Qwen Team \[2025\]Qwen Team\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388, 2025\.
- Radford et al\. \[2019\]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language models are unsupervised multitask learners\.*OpenAI Technical Report*, 2019\.
- Search\.io \[2022\]Search\.io\.Neural hashing: The future of search\.Search\.io whitepaper,[https://cdn\.prod\.website\-files\.com/61830fa9d578e127186810ed/6238c27c15df074466448996\_neuralhashing\-wp\-searchio\.pdf](https://cdn.prod.website-files.com/61830fa9d578e127186810ed/6238c27c15df074466448996_neuralhashing-wp-searchio.pdf), 2022\.Neural hashing compresses embedding vectors into compact binary hashes using a fraction of the storage while retaining up to 99% of the vector information \(Search\.io was acquired by Algolia in 2022\)\.
- Shazeer \[2020\]Noam Shazeer\.GLU variants improve transformer\.*arXiv preprint arXiv:2002\.05202*, 2020\.
- Su et al\. \[2024\]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, 568, 2024\.
- Tseng et al\. \[2024\]Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa\.QuIP\#: Even better LLM quantization with hadamard incoherence and lattice codebooks\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Unsloth \[2025\]Unsloth\.Unsloth dynamic GGUF quants\.Unsloth Documentation,[https://unsloth\.ai/docs/models/glm\-5\.2](https://unsloth.ai/docs/models/glm-5.2), 2025\.Post\-training, per\-layer dynamic mixed\-precision GGUF quantization \(UD\-\*series\): salient layers are retained at higher precision via an importance heuristic while the rest are pushed to low bit\-widths\.
- Wang et al\. \[2023\]Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei\.BitNet: Scaling 1\-bit transformers for large language models\.*arXiv preprint arXiv:2310\.11453*, 2023\.
- Wang et al\. \[2019\]Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han\.HAQ: Hardware\-aware automated quantization with mixed precision\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2019\.
- Wu et al\. \[2018\]Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer\.Mixed precision quantization of ConvNets via differentiable neural architecture search\.*arXiv preprint arXiv:1812\.00090*, 2018\.
- Xiao et al\. \[2023\]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han\.SmoothQuant: Accurate and efficient post\-training quantization for large language models\.In*International Conference on Machine Learning \(ICML\)*, 2023\.
- Zheng et al\. \[2026\]Zhen Zheng, Xiaonan Song, and Chuanjie Liu\.MixLLM: LLM quantization with global mixed\-precision between output\-features and highly\-efficient system design\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2026\.arXiv:2412\.14590\.