Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters

# A Controlled Re-Examinationof Ternary Language Models at 60K Parameters
Source: [https://arxiv.org/html/2609.29397](https://arxiv.org/html/2609.29397)
## Baseline Shape Decides the Verdict: A Controlled Re\-Examination of Ternary Language Models at 60K Parameters

###### Abstract

Ternary \(1\.58\-bit\) weights are attractive for microcontroller\-class language models, but the sub\-1M\-parameter regime is supported mainly by isolated, single\-seed comparisons\. A prominent example reports that a routed ternary block \(local convolution, diagonal state\-space model and sparse attention mixed by a per\-token router\) beats a parameter\-matched full\-precision transformer by 22% in perplexity at 60K parameters, and attributes this to architectural inductive bias\. We re\-run that comparison under one fixed recipe with three seeds per cell, 98 runs in total, on a single laptop\. The models are byte\-level, so budgets below are byte counts, and every relative gap is divided by the reference system named beside it\. Four results follow\. \(i\) Baseline*shape*dominates the comparison: at a 16M\-byte budget, param\-matched transformers span 2\.835–3\.477 nats/byte purely by depth/width choice \(a 22\.6% span, far larger than any architecture effect we measure at that budget\), and at this controlled budget the routed model’s advantage disappears against the best\-shaped transformer, so the published margin is at least partly a baseline\-shape effect; the ordering of shapes reverses with budget, so a single fixed shape cannot be trusted\. \(ii\) With a 130M\-byte budget the routed model does win, by 22\.2–24\.0% over each of the three transformer shapes we evaluate there, but a plain gated diagonal\-SSM block beats it by a further 9\.1% in full precision, and the routed model’s own router puts most of its weight on its recurrent pathway, so the gain does not require routing\. \(iii\) The ternary penalty differs sharply by architecture at the larger budget \(\+5\.3% for the best transformer shape against \+19\.5% routed and \+28\.1% gated SSM\), but we cannot attribute that difference to architecture alone: our transformers keep learned positional embeddings in full precision, 11–22% of their parameters, so they are less quantized than the models they are compared with\. \(iv\) A 90/10 full\-precision\-then\-ternary schedule beats all\-ternary training here, but only with a stage\-2 learning rate about 10×\\timesthe pretraining peak; at a conventional fine\-tuning rate the same schedule looks 15\.3% worse, reversing the conclusion\. The from\-scratch baseline was not itself learning\-rate tuned, which bounds \(iii\) and \(iv\)\. All code, corpus manifests and run logs are released\.111[https://github\.com/veldanda/ByteLM](https://github.com/veldanda/ByteLM), release tagp1\-v1\. The tag contains the training code, the parameter\-matched configs, the corpus manifest with SHA\-256, and one directory of logs and checkpointed metrics per run\.

## 1Introduction

Running a generative language model on a microcontroller \(MCU\) leaves a budget of tens to hundreds of kilobytes of flash and SRAM\. Ternary weights,w∈\{−α,0,\+α\}w\\in\\\{\-\\alpha,0,\+\\alpha\\\}atlog2⁡3≈1\.58\\log\_\{2\}3\\approx 1\.58bits, are a natural fit: they shrink storage by up to∼\\sim20×\\timesrelative to FP32 and replace multiplications with additions\[[10](https://arxiv.org/html/2609.29397#bib.bib10)\]\. Whether ternary training is*competitive*at these sizes, however, is poorly understood\. The most careful low\-bit scaling studies \(ParetoQ\[[9](https://arxiv.org/html/2609.29397#bib.bib9)\]and Spectra\[[8](https://arxiv.org/html/2609.29397#bib.bib8),[14](https://arxiv.org/html/2609.29397#bib.bib14)\]\) begin at 125M and 99M parameters respectively, and the recent sub\-100M study of[Thomassen \[13\]](https://arxiv.org/html/2609.29397#bib.bib13)considers integer bit\-widths \(8/6/4\) on transformers only\.

Below 10M parameters, the evidence is largely anecdotal\. Atome\-LM\[[1](https://arxiv.org/html/2609.29397#bib.bib1)\]reports that a routed ternary hybrid beats a parameter\-matched FP32 transformer by 22% perplexity at 60K parameters, but loses by∼\\sim11% at 944K, and interprets this as the architecture’s inductive bias “substituting for capacity at small scale and constraining it at larger scale”\. If true, this would be an important design rule for edge models: use strong structural priors only in the smallest tier\. But the claim rests on two single\-seed comparisons, and, as we show in §[4](https://arxiv.org/html/2609.29397#S4), the 60K and 944K comparisons were run under very different training regimes\.

Re\-running that comparison under one recipe with three seeds gives a different picture, and the first reason is mundane\. At a fixed 60K parameter budget, a transformer’s depth/width ratio moves validation loss by 22\.6%, further than replacing the architecture does, so the published result depends on which single transformer shape is used as the baseline\. Once the strongest shape is used at each budget, two things remain: at a larger byte budget the routed block genuinely wins, but a plain gated recurrence wins by more; and the measured cost of ternary weights differs by architecture, shrinking with training for transformers and not for recurrent models, though that difference is confounded by unequal quantization and by an untuned from\-scratch ternary baseline, as we set out in §[7](https://arxiv.org/html/2609.29397#S7)\. We therefore ask:*at microcontroller scale, what actually accounts for reported architectural advantages, and how does the ternary penalty behave as the training budget grows?*

#### Contributions\.

1. 1\.An audit of the published evidence\(§[4](https://arxiv.org/html/2609.29397#S4)\): the 60K and 944K comparisons differ by∼650×\{\\sim\}650\\timesin training tokens and in corpus, schedule, optimizer settings and baseline FFN width, and the 944K comparison changes precision and architecture at once\.
2. 2\.A controlled study at 60K, 98 runs, three seeds per cell, one fixed recipe, token\-matched full\-precision anchors and a2​σ2\\sigmaseed\-noise criterion fixed before the results were inspected \(§[5](https://arxiv.org/html/2609.29397#S5)\): baseline shape sensitivity, a three\-family architecture comparison, the budget dependence of the ternary penalty, and a tuned FP\-init recipe \(§[6](https://arxiv.org/html/2609.29397#S6)\)\.
3. 3\.Two methodological warningsthat each reverse a headline: a fixed baseline shape \(which flips the architecture verdict at 16M bytes\), and an untuned stage\-2 learning rate \(which flips the FP\-init verdict\)\.
4. 4\.An open, laptop\-scale release: an identity quantizer that enables the first full\-precision run of the routed architecture, plus exact re\-implementations of two of its layers that make the study affordable \(Appendix[A](https://arxiv.org/html/2609.29397#A1)\); 98 runs in∼40\{\\sim\}40device\-hours on one MacBook Air\.

## 2Related Work

#### Low\-bit training and scaling\.

BitNet b1\.58\[[10](https://arxiv.org/html/2609.29397#bib.bib10)\]popularized ternary weights with per\-tensor absmean scaling and the straight\-through estimator\[[2](https://arxiv.org/html/2609.29397#bib.bib2)\]\. Spectra\[[8](https://arxiv.org/html/2609.29397#bib.bib8)\]trains ternary transformers from scratch from 99M to 3\.9B parameters; Spectra 1\.1\[[14](https://arxiv.org/html/2609.29397#bib.bib14)\]finds that ternary models “benefit more from increasing training data than from scaling model parameters”\. ParetoQ\[[9](https://arxiv.org/html/2609.29397#bib.bib9)\]compares 1–4\-bit quantization\-aware training \(QAT\) from 125M to 8B parameters and finds that spending∼\\sim90% of the budget in full precision before switching to QAT outperforms low\-bit training from scratch, with≤\\leq2\-bit models requiring more tokens to saturate\. None of these studies go below∼\\sim100M parameters or vary architecture\.

#### Sub\-100M quantization\.

[Thomassen \[13\]](https://arxiv.org/html/2609.29397#bib.bib13)runs a 720\-run factorial of INT8/INT6/INT4 QAT on 3M–350M transformers and finds the optimal learning\-rate warmdown to be bit\-width\-agnostic, with an INT4 regime boundary at 30–50M parameters and an INT6 penalty that*grows*with training length\. Their study uses transformers only, integer bit\-widths and QAT from initialization; we study ternary weights, three architecture families, and FP\-initialized vs\. from\-scratch training, down to 60K parameters\. TernaryLM\[[11](https://arxiv.org/html/2609.29397#bib.bib11)\]trains a 132M ternary transformer on TinyStories with full\-precision embeddings and head\.

#### Non\-transformer low\-bit models\.

Ternary Mamba\[[5](https://arxiv.org/html/2609.29397#bib.bib5)\]applies grouped QAT to a pretrained 1\.3B Mamba\-2\[[6](https://arxiv.org/html/2609.29397#bib.bib6)\]\. Hybrid attention/SSM designs such as Hymba\[[3](https://arxiv.org/html/2609.29397#bib.bib3)\]target the≥\\geq100M regime in full precision\. Atome\-LM\[[1](https://arxiv.org/html/2609.29397#bib.bib1)\]is the only publicly released routed ternary hybrid at MCU scale we are aware of\.

#### Depth, width and the shape of a small model\.

At a fixed parameter budget, depth and width trade off against each other, and the optimum moves with the compute or data budget: this is standard for large models\[[7](https://arxiv.org/html/2609.29397#bib.bib7),[12](https://arxiv.org/html/2609.29397#bib.bib12)\]but is rarely controlled in small\-model comparisons, where a single “param\-matched baseline” is usually reported\. Our shape sweep \(§[6](https://arxiv.org/html/2609.29397#S6)\) shows the effect is large enough at 60K parameters to determine the verdict of an architecture comparison, and that the best shape at one budget is the worst at another\.

#### Inductive bias and scale\.

[Tay et al\. \[12\]](https://arxiv.org/html/2609.29397#bib.bib12)show across ten architectures that “the best performing model can fluctuate at different scales”\. They vary model size in full precision; we vary training budget and precision as well, and ask whether the fluctuation reported at MCU scale is a size effect at all\.

## 3Background: the Atome MCU block

Each block applies LayerNorm, then three pathways in parallel \(a depthwise causal convolution \(k=5k\{=\}5\), a diagonal SSMht=a⊙ht−1\+b⊙xth\_\{t\}=a\\odot h\_\{t\-1\}\+b\\odot x\_\{t\},yt=c⊙hty\_\{t\}=c\\odot h\_\{t\}witha=tanh⁡\(araw\)a=\\tanh\(a\_\{\\text\{raw\}\}\), and single\-head top\-kk\(k=4k\{=\}4\) causal attention\), mixed by a per\-token softmax router, followed by a residual connection\. There is no feed\-forward layer; channel mixing happens only in the attention value projection and the router\. Tokens are raw bytes \(vocabulary 256\) and there is no positional embedding\. All linear, convolution, embedding, unembedding and router weights are ternarized with a single per\-tensor absmean scale,W~=α​clip​\(round⁡\(W/α\),−1,1\)\\tilde\{W\}=\\alpha\\,\\mathrm\{clip\}\(\\mathrm\{round\}\(W/\\alpha\),\-1,1\),α=mean​\|W\|\\alpha=\\mathrm\{mean\}\|W\|, recomputed every forward pass with STE; the SSM’s per\-channel vectors stay in FP32\. Embedding and head are 13\.9% of parameters at 944K\.

## 4Auditing the published comparison

Table[1](https://arxiv.org/html/2609.29397#S4.T1)summarizes the conditions behind the two published data points, reconstructed from the released run artifacts \(ab\_results\.json,\*\.train\.json\) and scripts at commitbe12186\.

Table 1:Conditions of the two published Atome\-LM comparisons\.Three observations follow\. \(i\) Scale is confounded with training budget and recipe: a strong structural prior that helps in a 3M\-token, constant\-LR regime and is overtaken when a generic model receives 650×\\timesmore data would produce the same “reversal” with no size effect at all \(*H\-budget*\)\. \(ii\) Neither comparison isolates precision from architecture, because no full\-precision routed model was trained; the released quantizer registry has no identity option\. \(iii\) The 60K transformer baseline uses FFN width1×d1\\times d, while the 944K baseline uses4×d4\\times d\. A fourth difference is worth naming: at 3M tokens over a 0\.5 MB corpus the 60K runs make roughly six passes over their data, a regime in which memorization contributes to validation loss, whereas our budgets stay under 0\.07 epochs\. Their reported 6\.31 perplexity \(≈1\.84\\approx 1\.84nats/byte\) is far below anything we reach at 16M bytes \(2\.7–3\.5\), which is consistent with that reading\. We do not re\-run their setup, so we cannot separate memorization from architecture there; it is a further reason the two published points are not comparable to each other\. We also note that the documented corpus\-construction command produces at most 1 MB of the TinyStories V2 validation file, whereas the results document states the full train\+valid set was used; we therefore build our own corpus with a checksum manifest and re\-run every comparison rather than relying on reported numbers\. We stress that this reconstruction was only possible because the authors released their run artifacts, configurations and training scripts, which is more than most work at this scale provides; the conditions above are stated in their own repository\. Our point is not that their measurements are wrong \(we do not re\-run their exact 60K setup, and have no reason to doubt the numbers\), but that the two data points cannot be compared to each other, and that the 60K comparison rests on one baseline shape \(§[6](https://arxiv.org/html/2609.29397#S6)\)\.

## 5Experimental design

#### Hypotheses\.

\(Outcomes in §[6](https://arxiv.org/html/2609.29397#S6); a further hypothesis on quantization granularity, namely full\-precision embedding/head and per\-row scales, was specified but not run, and is left to future work\.\)

H\-budget\.The routed architecture’s advantage over a transformer at fixed size shrinks or reverses as the training budget grows, independent of precision\.

H\-recipe\.Ternary models initialized from a full\-precision checkpoint of the same architecture \(ParetoQ\-style\) close part of the ternary–FP gap at sub\-10M scale\.

H\-interaction\.The ternary penalty differs between architectures \(precision×\\timesarchitecture interaction\), beyond seed noise\.

#### What was run \(98 runs, 3 seeds per cell unless noted\)\.

All at 60K parameters, byte\-level, on TinyStories V2\.

- •Main grid \(24\):\{routed, transformer3×323\\times 32\}×\\times\{FP32, ternary\}×\\times\{16M, 130M bytes\}\.
- •Shape sweep \(36\):five param\-matched transformer shapes \(1×521\\times 52,2×362\\times 36,3×323\\times 32,4×284\\times 28,6×246\\times 24; all within 3% of 60,800 parameters\) at 16M, and the three strongest at 130M, in both precisions\.
- •Third architecture \(12\):gated diagonal\-SSM block, both budgets, both precisions\.
- •FP\-init \(18 \+ 10\):stage 1 in FP32 for 117M bytes, stage 2 ternary for 13M \(90/10 of the 130M budget\), at stage\-2 LR×10−41\\\!\\times\\\!10^\{\-4\}and at the swept value; plus a five\-point stage\-2 LR sweep per architecture at seed 0\. The transformer arm here is the1×521\\times 52shape, not the pre\-specified3×323\\times 32, and its from\-scratch reference uses the same shape\.

Parameter\-matched widths come from a depth schedule fixed before any results were seen \(configs/scales\.json\); the shape sweep then tests that choice directly\.

Table 2:Run accounting\. Two of the tuned FP\-init arms’ seed\-0 runs are the corresponding sweep runs reused rather than retrained, so 100 run directories correspond to 98 distinct trainings\.Table 3:Exact configurations\. Every model ternarizes its token embedding, unembedding and all weight matrices \(plus the depthwise conv and router, for the routed block\)\. “FP kept” is the share of parameters that remain full precision: per\-channel SSM scalars and LayerNorms everywhere, plus the transformers’ learned positional embeddings, which dominate that column\.The transformers are within 3% of the routed model’s parameter count; the gated SSM is 5\.6% larger, which flatters it and should be read as a caveat on §6\.2\. The1×521\\times 52shape usesdff=2​dd\_\{\\text\{ff\}\}\{=\}2dbecause4​d4dat that width overshoots the budget\.

#### Fixed recipe\.

All runs use the published 944K recipe: AdamW \(β1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.95\\beta\_\{2\}\{=\}0\.95, weight decay 0\.1 on all parameters\), peak LR×10−43\\\!\\times\\\!10^\{\-4\}with cosine decay to×10−53\\\!\\times\\\!10^\{\-5\}, gradient clipping 1\.0, sequence length 256, batch64×464\\times 4\(65,280 predicted tokens per step\)\. Warmup is 3\.33% of steps \(the published1000/300001000/30000\), and each budget is a separate run whose schedule ends at that budget\.

#### Data\.

TinyStories V2\[[4](https://arxiv.org/html/2609.29397#bib.bib4)\]train and validation files concatenated as raw bytes: 2,250,255,763 bytes \(SHA\-256 recorded in a manifest in the repository\), cut into 256\-byte chunks, 10% held out, leaving 2\.03B training tokens\. Both budgets are therefore well under one epoch \(16M==0\.008 epochs, 130M==0\.064\), so no result here is affected by repeated data\. The held\-out chunks are drawn at random from the same corpus, so near\-duplicate stories could straddle the split; since every system is evaluated on the same slice this cannot favour one architecture, but absolute losses may be slightly optimistic\.

#### Metric and decision rule\.

The primary metric is validation loss in nats/byte of the*last*checkpoint on 256 held\-out batches \(best\-on\-validation selection is available in the released logs\)\. For each comparison we report the mean over seeds, the relative gap, and a noise floorσ=σa2\+σb2/μb\\sigma=\\sqrt\{\\sigma\_\{a\}^\{2\}\+\\sigma\_\{b\}^\{2\}\}/\\mu\_\{b\}; a gap is called real only if it exceeds2​σ2\\sigma, and comparisons with fewer than two seeds per group are reported as undetermined\. Every ternary run is compared to a full\-precision anchor trained on the same total number of bytes\. Initialization and data\-order seeds are controlled separately\.

#### Diagnostics\.

For all routed runs we log per\-layer router entropy, maximum router probability and mean pathway weights; for all quantized runs we log the relativeL1L\_\{1\}change of shadow weights from their reference, the fraction of ternary levels that changed, and the zero fraction\. These are used for the router analysis in §[6\.5](https://arxiv.org/html/2609.29397#S6.SS5)and are otherwise available in the released logs; we do not report them run by run\.

#### Availability\.

Training code, configs, the corpus manifest and the per\-run logs are at[https://github\.com/veldanda/ByteLM](https://github.com/veldanda/ByteLM)under release tagp1\-v1; every table and figure in this paper is regenerated from those logs byexperiments/analyze\_all\.py\.

#### Compute\.

Every run was trained on one Apple MacBook Air \(M2, 16 GB, PyTorch 2\.8, MPS backend\),∼40\{\\sim\}40device\-hours in total, at zero marginal cost\.

## 6Results

All numbers are validation loss in nats/byte on 256 held\-out batches of the last checkpoint, mean±\\pmsd over three seeds\. The models are byte\-level, so a training “token” is one byte and we state budgets in bytes: 16M and 130M bytes, i\.e\.∼260\{\\sim\}260and∼2,140\{\\sim\}2\{,\}140bytes per parameter\. Budgets are matched within every comparison\. Note that changing the budget changes three things at once: data seen, optimizer steps \(245 vs\. 1,991\) and schedule length, so “budget” here means all three together, not data quantity alone\.

Table 4:All systems at 60K parameters\. Ternary penalty = \(ternary−\-FP\)/FP\.11223344662\.62\.62\.82\.8333\.23\.23\.43\.43\.63\.6routedgated SSMtransformertransformer depth \(layers\)val loss \(nats/byte\)16M bytes

1122331\.61\.61\.81\.8222\.22\.2routedgated SSMtransformertransformer depth \(layers\)130M bytes

Figure 1:Full\-precision validation loss at 60K parameters, three seeds \(bars: sd\)\. Every transformer point is within 3% of the same parameter count; depth, width, FFN ratio and positional\-embedding share vary together along thexx\-axis \(Table[3](https://arxiv.org/html/2609.29397#S5.T3)\)\.Left:at a short budget the best shape \(1 layer, width 52\) ties the routed block while the worst \(6 layers, width 24\) is 22\.6% worse than the best and 19\.2% worse than the routed block\.Right:at 130M bytes the ordering inverts \(depth now helps\) and both recurrent architectures pull clearly ahead, the gated SSM furthest\. Ternary runs \(Table[4](https://arxiv.org/html/2609.29397#S6.T4)\) show the same shape ordering\.### 6\.1Baseline shape decides the verdict

At 16M bytes, param\-matched transformer shapes span 2\.835–3\.477 nats/byte in full precision: the worst shape is 22\.6% worse than the best \(equivalently, the best is 18\.5% better\), a span far wider than any architecture effect we measure at this budget: the routed model is 0\.95% and the gated SSM 4\.6% better than the best shape \(Figure[1](https://arxiv.org/html/2609.29397#S6.F1)\)\. Against the best shape \(1×521\\times 52\) the routed model is0\.95%0\.95\\%better in full precision \(2​σrel=4\.5%2\\sigma\_\{\\text\{rel\}\}=4\.5\\%\) and4\.0%4\.0\\%better in ternary \(2​σrel=4\.3%2\\sigma\_\{\\text\{rel\}\}=4\.3\\%\), both inside the noise band\. Against the worst shape \(6×246\\times 24\) it is19\.2%19\.2\\%and21\.0%21\.0\\%better, far outside it\. A study that fixes one shape can therefore report either verdict\.

One caveat on the best shape itself:1×521\\times 52differs from the other shapes in more than depth and width\. Its feed\-forward ratio is2​d2drather than4​d4d\(4​d4dovershoots the parameter budget at that width\), and its learned positional embedding occupies 22% of its parameters against 11–16% for the deeper shapes \(Table[3](https://arxiv.org/html/2609.29397#S5.T3)\)\. “Shape” here therefore means the whole depth/width/FFN/position\-budget package, not depth alone\.

Two of these comparisons are exploratory rather than confirmatory: routed vs\. the best observed shape, in full precision and in ternary\. The pre\-specified baseline was3×323\\times 32, from a depth schedule fixed before any runs; the other four shapes were added after seeing that result, and “routed vs\. the best observed shape” is therefore a post\-hoc descriptive comparison, reported as such\.

Crucially the ordering*reverses with budget*: at 130M bytes the shallow\-wide1×521\\times 52shape is the worst transformer \(2\.195\) and the deeper3×323\\times 32the best \(2\.143\)\. Shape must be swept per budget; it cannot be chosen once\.

### 6\.2At a larger budget the gain is real, but it does not require routing

At 130M bytes the routed model beats each of the three transformer shapes evaluated at that budget \(1×521\\times 52,2×362\\times 36,3×323\\times 32\) by 22\.2–24\.0% in full precision and 13\.7–14\.4% in ternary, all outside the noise band\. The two deepest shapes were not rerun at 130M, so this is a claim about the three strongest shapes at 16M, not about the whole shape family\.

Our gated diagonal\-SSM block \(a single recurrent pathway with a GELU feed\-forward layer and no router\) beats the routed model by 9\.1% in full precision \(2​σrel=3\.2%2\\sigma\_\{\\text\{rel\}\}=3\.2\\%\) and matches it in ternary \(−2\.6%\-2\.6\\%,2​σrel=3\.7%2\\sigma\_\{\\text\{rel\}\}=3\.7\\%, within the band\)\. Routing is therefore not required for the advantage: a much simpler gated recurrence reaches comparable or better loss at the same parameter and byte budget\. We do*not*claim to isolate recurrence as the causal ingredient \(our block also differs from the routed one in having a feed\-forward layer and denser channel mixing\), only that the router is not what buys the gain\.

### 6\.3The ternary penalty differs by architecture, with caveats we cannot yet remove

At 130M bytes the ternary penalty is\+5\.3%\+5\.3\\%for the best transformer shape,\+19\.5%\+19\.5\\%for the routed model and\+28\.1%\+28\.1\\%for the gated SSM; in absolute terms0\.1160\.116,0\.3260\.326and0\.4250\.425nats/byte\. The transformer’s penalty falls sharply with budget \(\+20\.9%→\+5\.3%\+20\.9\\%\\to\+5\.3\\%\), matching what ParetoQ reports at≥\\geq125M parameters, while the recurrent models’ do not \(\+24\.9%→\+28\.1%\+24\.9\\%\\to\+28\.1\\%and\+17\.2%→\+19\.5%\+17\.2\\%\\to\+19\.5\\%\)\. Because the 16M penalties sit inside noise bands of2​σrel=7\.5%2\\sigma\_\{\\text\{rel\}\}=7\.5\\%and6\.4%6\.4\\%, the defensible statement is that the recurrent penalties*do not shrink*, not that they grow\. The architecture gap consequently narrows without closing: the routed model is22\.2%22\.2\\%better than the best transformer shape in full precision and13\.7%13\.7\\%better in ternary\.

Three alternative explanations are not excluded, and we flag them rather than claim the result is clean\.Unequal quantization\.Our transformers keep learned positional embeddings in full precision \(1111–22%22\\%of their parameters, largest for the shallow\-wide shape\) against1\.61\.6–2\.3%2\.3\\%for the recurrent models \(Table[3](https://arxiv.org/html/2609.29397#S5.T3)\)\. A model that quantizes less of itself should lose less, so some unknown share of the difference is bookkeeping rather than architecture\. A replication with a zero\-parameter position scheme \(RoPE, ALiBi, sinusoidal\) would settle it\.An untuned ternary baseline\.Every from\-scratch ternary run uses the full\-precision recipe’s peak learning rate; §[6\.4](https://arxiv.org/html/2609.29397#S6.SS4)shows the ternary stage of the FP\-init arm prefers a rate10×10\\timeslarger, so the from\-scratch penalties reported here may be inflated, possibly unequally across architectures\.Loss level rather than architecture\.At 130M bytes the recurrent models sit at 1\.52–1\.67 nats/byte and the transformers at 2\.14–2\.20, so penalty and loss level covary; at 16M, where the routed model \(2\.808\) and the1×521\\times 52transformer \(2\.835\) reach similar loss, their penalties are also similar \(17\.2% vs\. 20\.9%\)\. The penalty may track how much fine structure a model has already fitted rather than what kind of model it is\. A loss\-matched comparison would separate these, and we have not run one\.

### 6\.4A 90/10 FP\-then\-ternary schedule helps, at a learning rate 10×\\timesthe pretraining peak

Following ParetoQ we ran 90% of the budget in full precision and the last 10% in ternary, token\-matched to the from\-scratch arm \(Table[5](https://arxiv.org/html/2609.29397#S6.T5)\)\. With a conventional fine\-tuning rate \(×10−41\\\!\\times\\\!10^\{\-4\}\) the recipe looks clearly*worse*than training ternary from scratch\. A five\-point stage\-2 sweep \(seed 0\) shows loss falling monotonically up to×10−33\\\!\\times\\\!10^\{\-3\}, an order of magnitude above the pretraining peak of×10−43\\\!\\times\\\!10^\{\-4\}; at that rate the same recipe*wins*\. The sweep had not saturated at its upper end, so×10−33\\\!\\times\\\!10^\{\-3\}is a lower bound on the useful rate, not an estimated optimum; we did not extend it further, and where the recipe becomes unstable is untested\. Note also what this comparison is and is not: it tests a 90/10 allocation of one byte budget between full\-precision and ternary optimization, not quantization\-aware fine\-tuning in general\. That so short an adaptation phase \(10% of the budget\) prefers so large a step size suggests the ternary stage is doing something closer to re\-solving the weights than to fine\-tuning them, which is consistent with the “reconstruction” behaviour ParetoQ reports at larger scale\.

Table 5:FP\-init \(90% FP→\\to10% ternary\) vs\. ternary from scratch, token\-matched at 130M\. Left: stage\-2 LR sweep, seed 0\. Right: three\-seed arms\. The transformer arm is the1×521\\times 52shape in both the FP\-init runs and the from\-scratch reference they are compared against, so the comparison holds shape fixed\.
### 6\.5What the router actually does

The routing diagnostics logged for every routed run support the same conclusion from inside the model\. At the end of a 130M\-byte ternary run, the mean per\-token mixing weight on the state \(SSM\) pathway is 0\.49–0\.73 across the four layers, against 0\.09–0\.25 for the local convolution and 0\.19–0\.28 for sparse attention; the full\-precision run is more extreme still \(state 0\.54–0\.79\)\. Per\-layer router entropy stays at 0\.67–0\.87 nats against a maximum ofln⁡3=1\.099\\ln 3=1\.099, so the router neither collapses to one pathway nor stays uniform: it settles on a mixture dominated by the recurrence\. A model whose router spends most of its weight on the SSM pathway is consistent with a plain gated SSM matching it\.

### 6\.6On the original H\-budget hypothesis

We had hypothesised that the published 60K advantage would shrink as the training budget grew \(the two published points differ by∼650×\{\\sim\}650\\timesin training bytes\)\. It does not: against the shape the original work used, the routed model’s advantage*grows*from13\.3%13\.3\\%at 16M bytes to22\.2%22\.2\\%at 130M bytes\. The confound we identified is real, but it is not what explains the result; at our two controlled budgets, baseline shape is\.

### 6\.7What the experiments do and do not establish

Table 6:Claims against the evidence that supports them\.

## 7Limitations

One parameter count\.Every result is at 60K parameters\. We cannot say whether the architecture ordering or the ternary penalties change at 1M or 10M, and we therefore do not address the 944K reversal that motivated the audit\. Training a 944K grid to a comparable bytes/parameter budget is roughly two orders of magnitude more compute than a laptop allows\.

One corpus, and a byte\-level one\.TinyStories favours local, repetitive structure, and byte\-level input makes short\-range dependencies dominant; both plausibly flatter convolutional and recurrent pathways relative to attention\. The architecture×\\timesprecision interaction we report is established on TinyStories V2 at byte level; whether it persists on less locally structured corpora \(code, multilingual text, subword tokenization\) is open, and the specific gap percentages should not be carried over\. The tokenization itself is part of this: English text is roughly four bytes per subword token, so a byte\-level model sees sequences several times longer at the same context width, which weights local sequential structure more heavily and plausibly favours the convolutional and recurrent pathways over attention\. A subword\-tokenized replication could narrow the architecture gaps we report even if their ordering holds\. A wider\-domain corpus is the single most valuable follow\-up\.

Two budgets, low bytes/parameter at the top end\.16M and 130M bytes \(∼260\{\\sim\}260and∼2,140\{\\sim\}2\{,\}140bytes/parameter\); the shortest budget is only 245 optimizer steps\.

Scope of the baselines\.The transformer shapes are param\-matched within 3% but share one recipe; we did not tune learning rate per shape\. The gated SSM is deliberately simple and is not a Mamba\-class model\. Loss is the only quality metric\. Wall\-clock numbers come from a laptop, not an MCU; deployment cost is left to follow\-up work\.

Unequal quantization across architectures\.Our transformers keep their learned positional embeddings in full precision \(1111–22%22\\%of their parameters\) against1\.61\.6–2\.3%2\.3\\%for the recurrent models, so the “ternary” arms are not equally quantized\. Any architecture\-dependent ternary penalty we report is therefore an upper bound on the architectural component; a replication with a zero\-parameter position scheme is needed before the difference can be attributed to architecture\.

The from\-scratch ternary baseline is untuned\.Every from\-scratch ternary run reuses the full\-precision peak learning rate\. The FP\-init sweep shows that ternary optimization prefers a rate an order of magnitude larger, so the from\-scratch ternary losses \(and hence every ternary penalty in this paper\) may be inflated, possibly by different amounts across architectures\. A per\-architecture ternary learning\-rate sweep is the cheapest experiment that would tighten §[6](https://arxiv.org/html/2609.29397#S6)\.

Stage\-2 LR not bracketed\.The FP\-init sweep had not saturated at×10−33\\\!\\times\\\!10^\{\-3\}, so the reported FP\-init results are a lower bound on what the recipe can do\.

## 8Conclusion

A reader might take three things from the published comparison: that the routed block beats a parameter\-matched transformer at 60K, that its router is what produces the advantage, and that the cost of ternary weights is a property of precision rather than of architecture\. At 60K parameters the first two do not survive a controlled re\-run\. The advantage at a short budget is at least partly a baseline\-shape artifact, and the advantage that remains at a longer budget does not require routing: a single gated recurrence does at least as well, though our comparison does not isolate recurrence itself as the cause\. What survives is more qualified: the cost of ternary weights differs sharply by architecture at the larger budget \(the transformer recovers with more training, the recurrent models do not\), though our transformers keep 11–22% of their parameters in full precision and our from\-scratch ternary baseline was never learning\-rate tuned, and loss level covaries with architecture here, so we report that difference as an observation with three live alternative explanations rather than as an established architectural law\. The cleanest new result is the recipe one: a 90/10 full\-precision\-then\-ternary schedule needs a learning rate far above the usual fine\-tuning range before it beats all\-ternary training at all, though it is measured against a from\-scratch baseline that was never itself learning\-rate tuned, so the size of the win, and not only its sign, is provisional\. None of this establishes behaviour beyond 60K parameters or beyond TinyStories\. Both of the methodological traps we hit \(a fixed baseline shape, an untuned stage\-2 learning rate\) individually reverse a headline result, which is worth more caution than the field currently gives them at this scale\.

What this does not settle is the scaling question that motivated the audit\. We show the published 60K comparison does not support the interpretation placed on it; the reported reversal at 944K parameters remains untested, because a comparable grid at that size is roughly two orders of magnitude beyond a laptop budget\. Whether the architecture ordering and the architecture\-dependent ternary penalty hold, shrink or invert between 60K and 1M parameters is the obvious next experiment, and the one we would run first given compute\.

## References

- \[1\]Atome LM contributors\.Atome LM: a tiny ternary language model for microcontroller deployment\.[https://github\.com/TilelliLab/atome\-lm](https://github.com/TilelliLab/atome-lm), commit be12186, v2\.0, 2026\.doi:10\.5281/zenodo\.20518644\.
- \[2\]Yoshua Bengio, Nicholas Léonard, and Aaron Courville\.Estimating or propagating gradients through stochastic neurons for conditional computation\.*arXiv preprint arXiv:1308\.3432*, 2013\.
- \[3\]Xin Dong et al\.Hymba: A hybrid\-head architecture for small language models\.*arXiv preprint arXiv:2411\.13676*, 2024\.
- \[4\]Ronen Eldan and Yuanzhi Li\.TinyStories: How small can language models be and still speak coherent English?*arXiv preprint arXiv:2305\.07759*, 2023\.
- \[5\]Ramprasath Ganesaraja, Sahil Dilip Panse, and Swathika N\.Ternary Mamba: Grouped quantization\-aware training of W1\.58A16 state space models\.*arXiv preprint arXiv:2606\.18114*, 2026\.
- \[6\]Albert Gu and Tri Dao\.Mamba: Linear\-time sequence modeling with selective state spaces\.*arXiv preprint arXiv:2312\.00752*, 2023\.
- \[7\]Jordan Hoffmann et al\.Training compute\-optimal large language models\.*arXiv preprint arXiv:2203\.15556*, 2022\.
- \[8\]Ayush Kaushal et al\.Spectra: Surprising effectiveness of pretraining ternary language models at scale\.*arXiv preprint arXiv:2407\.12327*, 2024\.ICLR 2025\.
- \[9\]Zechun Liu et al\.ParetoQ: Scaling laws in extremely low\-bit LLM quantization\.*arXiv preprint arXiv:2502\.02631*, 2025\.NeurIPS 2025\.
- \[10\]Shuming Ma et al\.The era of 1\-bit LLMs: All large language models are in 1\.58 bits\.*arXiv preprint arXiv:2402\.17764*, 2024\.
- \[11\]Nisharg Nargund and Priyesh Shukla\.TernaryLM: Memory\-efficient language modeling via native 1\.5\-bit quantization with adaptive layer\-wise scaling\.*arXiv preprint arXiv:2602\.07374*, 2026\.
- \[12\]Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q\. Tran, Dani Yogatama, and Donald Metzler\.Scaling laws vs model architectures: How does inductive bias influence scaling?*arXiv preprint arXiv:2207\.10551*, 2022\.
- \[13\]Christian Thomassen\.Mapping the schedule×\\timesbit\-width boundary in sub\-100M quantisation\-aware training\.*arXiv preprint arXiv:2605\.25966*, 2026\.
- \[14\]Tejas Vaidhya, Ayush Kaushal, Vineet Jain, Francis Couture Harpin, Prashant Shishodia, Majid Behbahani, Yuriy Nevmyvaka, and Irina Rish\.Spectra 1\.1: Scaling laws and efficient inference for ternary language models\.*arXiv preprint arXiv:2506\.23025*, 2025\.

## Appendix AExact faster kernels

#### Chunked\-scan SSM\.

The released implementation trains the diagonal SSM as a depthwise convolution with a length\-LLkernel \(O⁡\(L2\)O\(L^\{2\}\)per channel\)\. We split the sequence into chunks ofT=32T\{=\}32: within a chunk, the recurrence is a lower\-triangularT×TT\\times TmatrixMt​s=at−sM\_\{ts\}=a^\{t\-s\}applied per channel; across chunks the state is carried withL/TL/Tsequential steps,Hn=aT​Hn−1\+hnendH\_\{n\}=a^\{T\}H\_\{n\-1\}\+h^\{\\text\{end\}\}\_\{n\}, and added asat\+1​Hn−1a^\{t\+1\}H\_\{n\-1\}\. Only non\-negative powersaka^\{k\},k≤Tk\\leq T, appear, so the scheme is stable for any\|a\|<1\|a\|<1\. Forward and all gradients agree with the original to≤2×10−6\\leq 2\\times 10^\{\-6\}relative error \(also against the step\-by\-step recurrence\), at 19×\\timeslower cost on CPU \(155 ms→\\to8 ms,C=L=256C\{=\}L\{=\}256, batch 16\)\. On the MPS backend the same substitution reduced one block’s SSM from 300 ms to 14 ms, measured while another training job shared the GPU, so that ratio is indicative rather than a clean benchmark\.

#### Gather top\-kkattention\.

Entries outside the top\-kkare exactly zero after softmax, so the output equals a softmax over thekkselected scores applied to thekkgathered value rows; this avoids scattering scores back into anL×LL\\times Lmatrix\. Agreement≤1\.2×10−6\\leq 1\.2\\times 10^\{\-6\}\. End\-to\-end, a toy training pipeline gives the same ternary–FP relative gaps \(\+4\.56% vs\. \+4\.57%; \+7\.10% vs\. \+7\.11%\) under either implementation\. All runs in this paper use the fast path;\-\-exact\-kernelsrestores the upstream implementations for verification\.