Echo：面向大语言模型训练与推理的误差补偿 4 比特张量并行通信

# Error-Compensated 4-Bit Tensor-ParallelCommunication for LLM Training and Inference
Source: [https://arxiv.org/html/2610.14277](https://arxiv.org/html/2610.14277)
Echo: Error\-Compensated 4\-Bit Tensor\-Parallel Communication

## Echo: Error\-Compensated 4\-Bit Tensor\-Parallel Communication for LLM Training and Inference

###### Abstract

Tensor parallelism ends every layer with an all\-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows as compute outpaces interconnect bandwidth\. Sending 4\-bit instead of 16\-bit values would cut this cost by nearly 4×\\times, but the all\-reduce must add the values it carries, and rounding the sum injects an error at every call\. Model quality depends on how these errors accumulate over layers, yet existing methods treat each call on its own\. We presentEcho, a 4\-bit all\-reduce for training and inference that controls the accumulated error\.Echomoves data in 4 bits and sums in high precision, so each value is quantized at most twice per call for any number of GPUs\. The summation reveals each injected error, andEchofeeds it back into later calls, so old errors cancel instead of accumulating\. We prove that the accumulated error stays bounded independently of depth\. On A100 GPUs,Echois 2\.09×\\timesfaster than the 16\-bit NCCL all\-reduce on 1 GB messages\. It speeds up the time to first token by up to 1\.16×\\timesand the training step by up to 1\.09×\\timeson an idle interconnect, and by up to 1\.44×\\timesand 1\.38×\\timesas background traffic grows\. It keeps the average accuracy over seven benchmarks within 0\.4 points of 16\-bit inference on Qwen3\-8B\. In 10B\-token pretraining, its final loss is less than 0\.9% above that of 16\-bit training\.Echoneeds no FP8 or FP4 hardware\.

Keywords:Tensor parallelism, collective communication, low\-bit quantization, error feedback, large language models\.

## 1Introduction

Tensor parallelism splits the weight matrices of every attention and MLP layer across several GPUs, and each GPU computes a partial result from its share of the weights\([Shoeybi et al\., 2019](https://arxiv.org/html/2610.14277#bib.bib45)\)\. Turning the partial results into the layer output requires an all\-reduce, a collective operation that adds the partial results from all GPUs and returns the sum to each of them\. The all\-reduce runs between two dependent matrix multiplications, so it is a synchronization barrier on the critical path\. Unlike the gradient all\-reduce of data parallelism, which runs during the backward pass, or the point\-to\-point transfers of pipeline parallelism, which are pipelined across microbatches, it cannot be hidden behind the surrounding computation in the same way\. Overlapping can hide only part of it\([Wang et al\., 2024b](https://arxiv.org/html/2610.14277#bib.bib51)\)\. In production inference the all\-reduce contributes up to 30% of end\-to\-end latency\([Zhao et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib63)\)\. An 80\-layer model issues 160 all\-reduce calls in one forward pass, and on GPUs with limited interconnect bandwidth this communication can exceed half of the prefill cost\([Li et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib24)\)\. In large\-scale training, tensor\-parallel communication takes 50 to 60% of total communication time\([Liu et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib27)\)\. Hardware trends make the problem worse every generation, because peak compute has grown3\.0×3\.0\\timesevery two years while interconnect bandwidth has grown only1\.4×1\.4\\times\([Gholami et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib13)\)\. Kernel and decoding optimizations reduce the compute cost of serving\([Dao et al\., 2022](https://arxiv.org/html/2610.14277#bib.bib8);[Xie et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib59)\)and leave the communication untouched\. Faster collective algorithms and better use of spare links recover part of the latency but leave the transferred bytes unchanged\([Shen et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib44);[Zhao et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib63)\)\. Reducing the cost further requires sending fewer bits\.

Sending fewer bits means using a lower\-precision format, and lower precision has been adopted by every other part of the LLM stack\. Matrix computation has moved from FP16 to FP8 in production\-scale training\([DeepSeek\-AI, 2024](https://arxiv.org/html/2610.14277#bib.bib9)\)and to FP4 on recent hardware\([NVIDIA, 2025](https://arxiv.org/html/2610.14277#bib.bib32)\), with model quality preserved by careful algorithm design\. Model weights are served in 4\-bit formats\([Frantar et al\., 2023](https://arxiv.org/html/2610.14277#bib.bib11);[Lin et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib26)\)\. KV caches are compressed to 2 to 4 bits\([Liu et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib28);[Hooper et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib19)\)\. Sharded data parallelism communicates weights and gradients at close to 4 bits\([Jia et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib20)\), and production mixture\-of\-experts systems send expert\-layer activations at FP8\([DeepSeek\-AI, 2024](https://arxiv.org/html/2610.14277#bib.bib9);[Jin et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib21)\)\. Below 8 bits, tensor\-parallel communication is the exception\. The all\-reduce of most deployed systems still transfers 16\-bit values, although communication time is proportional to the bits sent and a 4\-bit all\-reduce would nearly quadruple the effective interconnect bandwidth on unchanged hardware\. The reason is that the all\-reduce must add the numbers it moves, and a sum of 4\-bit values does not fit in 4 bits\.

The same constraint appears outside communication\. Low\-precision matrix multiplication has the same problem, and tensor cores solve it inside the chip\. They multiply low\-precision operands but accumulate the products in high\-precision registers, so the wide sum is never stored and never moved\([Micikevicius et al\., 2022](https://arxiv.org/html/2610.14277#bib.bib30);[NVIDIA, 2025](https://arxiv.org/html/2610.14277#bib.bib32)\)\. The all\-reduce cannot do the same\. The values it adds start on different GPUs and must cross the interconnect before the addition, and the sum must cross the interconnect again to reach every GPU\. If both crossings are to stay at 4 bits, the collective must dequantize what arrives, add in high precision, and quantize the sum again for the way back\. Every call injects a fresh rounding error into the hidden state\. The injections are also frequent\. An LLM weight is quantized once, before deployment, and only read afterwards, while the all\-reduce quantizes at every layer of every forward pass and, in training, of every backward pass over tens of thousands of steps\. The injected errors accumulate, and published results show the consequences\. In inference, quantizing tensor\-parallel communication to 4 bits without training costs measurable accuracy\. On Gemma 2 27B, ARC\-challenge accuracy drops from 62\.4 to 57\.5 and WinoGrande from 79\.1 to 74\.3, and keeping outlier channels at 16 bits recovers only part of the loss\([Dong et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib10)\)\. In training, naive low\-bit communication destabilizes the loss curve or diverges\([Liu et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib27)\)\. The most recent training\-side work identifies error accumulation under high\-frequency communication as the central obstacle, yet its remedy is a stronger quantizer applied call by call, and it stops at FP8\([Liu et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib27)\)\. Every existing method works on each call in isolation, but low\-bit communication is a dynamic process, a sequence of rounding errors that flows through the network and through the training run\. Model quality depends on the accumulated error of this process, which a per\-call quantizer does not control\.

We control the process rather than the individual calls\. We presentEcho, a tensor\-parallel all\-reduce that sends 4\-bit integers, for both inference and training\.Echofirst restructures the collective around the constraint that addition must happen in high precision\. A quantized all\-to\-all brings each data segment to one owning GPU, the owner forms the sum in high precision, and a quantized all\-gather returns the reduced segments, so each element is quantized at most twice per call \(Sec\.[4](https://arxiv.org/html/2610.14277#S4)\)\. On top of this primitive,Echocontrols the error process\. The reduction step reveals each rounding error exactly, and a two\-timescale compensator feeds it back into subsequent calls, so the accumulated error stays bounded instead of growing with the number of calls\. The guarantee is independent of network depth and free of the exponential Lipschitz factor of worst\-case bounds \(Secs\.[5](https://arxiv.org/html/2610.14277#S5)and[6](https://arxiv.org/html/2610.14277#S6)\)\. The name comes from this mechanism, in which each injected error returns to the collective at the calls that follow, like an echo\. The compensation is fused into the entry pass of the all\-reduce and into an update that runs beside the next matrix multiplication, without changing the computed values \(Sec\.[5\.3](https://arxiv.org/html/2610.14277#S5.SS3)\)\. The complete design transports 4\-bit integers with half\-precision quantization parameters for each group and requires no FP8 or FP4 hardware support\.

Our contributions are summarized as follows\.

1. 1\.A quantized all\-reduce primitive in which each element is quantized at most twice per call, independent of the number of GPUs\. Computation and communication overlap inside the collective through a four\-stage tiled pipeline that runs asynchronously over peer\-to\-peer direct writes, entirely on the GPUs with no CPU coordination between stages\.
2. 2\.A two\-timescale feedback compensator that measures the rounding error at the reduction step and cancels its accumulation across calls, keeping the total error bounded regardless of the number of calls, with the same mechanism serving inference and training\.
3. 3\.A theory of the compensated collective\. The bounds are independent of network depth and free of the exponential Lipschitz factor of worst\-case bounds\. The source bound holds for any error sequence on the forward pass and, at every site, in mean square on the backward pass under a smoothness condition\. The propagation bound keeps the first\-order output deviation, measured against the hidden state, at the level of a single rounding error at every depth\.
4. 4\.A CUDA implementation that fuses compensation, quantization, and communication into one set of kernels\. For 1 GB messages, the compensated 4\-bit all\-reduce is 2\.09×\\timesfaster than the 16\-bit NCCL all\-reduce, and the time to first token and the training step improve by up to 1\.16×\\timesand 1\.09×\\timeson an idle interconnect and by up to 1\.44×\\timesand 1\.38×\\timesas background traffic grows\. 8\-bit inference matches 16\-bit accuracy, and 4\-bit inference stays within 0\.4 and 1\.7 points of it on average over seven tasks for Qwen3\-8B and Llama\-3\-8B\. Over 10B tokens of pretraining, 4\-bit training with a slightly larger learning rate ends less than 0\.9% above the 16\-bit loss\. To our knowledge,Echois the first to run tensor\-parallel communication with 4\-bit integers for both training and inference\.

The code will be released upon acceptance\.

## 2Related Work

We discuss three directions of related work: methods that hide or accelerate the tensor\-parallel all\-reduce, methods that quantize it, and error\-feedback methods that compensate compression errors in other forms of parallelism\. None of them controls the accumulated error of a quantized all\-reduce, which is the central object of this paper\.

The first direction hides or accelerates the all\-reduce without reducing the bits it moves\. Domino and FLUX split the computation of a transformer layer into pieces with no mutual dependence, so that one piece computes while another communicates\([Wang et al\., 2024b](https://arxiv.org/html/2610.14277#bib.bib51);[Chang et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib5)\)\. TokenWeave fuses the all\-reduce with the normalization that follows it, using multicast hardware that delivers one write to all GPUs at once\([Gond et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib14)\)\. FlexLink routes part of the traffic over otherwise idle PCIe and RDMA links\([Shen et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib44)\)\. SHARP builds the addition into the network switches, so the data is summed on its way through the network instead of on the GPUs\([Graham et al\., 2016](https://arxiv.org/html/2610.14277#bib.bib15)\)\. SwitchML, which aggregates gradients inside a programmable switch, must first convert them to a fixed\-point format with enough headroom to hold the sum\([Sapio et al\., 2021](https://arxiv.org/html/2610.14277#bib.bib42)\), the same constraint our collective is built around \(Sec\.[4](https://arxiv.org/html/2610.14277#S4)\)\. All of these methods transfer 16\-bit values\. Hiding works only when enough independent computation exists to run behind the communication, and this amount shrinks every hardware generation because peak compute grows faster than interconnect bandwidth\([Gholami et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib13)\)\. Our collective reduces the bytes instead, and a scheduler can treat it as an ordinary all\-reduce with less communication to hide\.

The second direction quantizes tensor\-parallel communication directly and is the closest to our work\. For inference, Flash Communication \(FlashComm\) uses fine\-grained group quantization and a two\-step all\-reduce\([Li et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib24)\)\. FlashComm V2 supports arbitrary bit widths through bit splitting and stores outliers in floating point\([Li et al\., 2025b](https://arxiv.org/html/2610.14277#bib.bib25)\)\. Mixed\-precision schemes keep outlier channels at 16 bits and reach an average of 4\.2 bits\([Dong et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib10)\)\. Fine\-grained quantization of the exchanged activations shortens the time to first token by up to 2×\\times\([Hansen\-Palmus et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib17)\), and EQuARX builds a block\-wise int8 all\-reduce into the XLA compiler for TPUs\([Ahmed et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib1)\)\. QuickReduce brings INT4, INT6, and INT8 all\-reduce to inference serving on AMD GPUs\([Li et al\., 2025a](https://arxiv.org/html/2610.14277#bib.bib23)\)\. For training, TACO compresses the communicated tensors to FP8 with a Hadamard transform and a two\-level scaling scheme\([Liu et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib27)\), and MegaScale\-MoE replaces the tensor\-parallel reduce\-scatter of FP8 mixture\-of\-experts training with an FP8 all\-to\-all and an FP32 sum\([Jin et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib21)\)\. All of these techniques work within a single call\. Finer groups, preserved outliers, rotations, and wider formats reduce the error of one quantization, but no method compensates the errors that earlier calls have already injected, so the accumulated error over layers and steps remains uncontrolled\.Echocontrols the accumulated error rather than the per\-call error, and since its bounds scale with the per\-call error, these techniques can be combined with it\. The compensator keeps the accumulation bounded \(Sec\.[5](https://arxiv.org/html/2610.14277#S5)\), so a standard INT4 quantizer is sufficient for both inference and training, and each element is quantized at most twice per call regardless of the number of GPUs \(Sec\.[4](https://arxiv.org/html/2610.14277#S4)\)\.

The third direction compensates quantization errors in other forms of parallelism\. Error feedback is a classical technique from signal processing and entered deep\-learning training with 1\-bit SGD, which stores the quantization error of the gradient and adds it to the gradient of the next step\([Seide et al\., 2014](https://arxiv.org/html/2610.14277#bib.bib43)\)\. Later analysis proved that this correction removes the bias of aggressive compressors\([Karimireddy et al\., 2019](https://arxiv.org/html/2610.14277#bib.bib22)\), and EF21 strengthened the guarantees with a different mechanism that communicates a compressed difference to a reference state\([Richtárik et al\., 2021](https://arxiv.org/html/2610.14277#bib.bib39)\)\. PowerSGD, 1\-bit Adam, ZeRO\+\+, and SDP4Bit bring compressed communication to large\-scale data\-parallel training\([Vogels et al\., 2019](https://arxiv.org/html/2610.14277#bib.bib49);[Tang et al\., 2021](https://arxiv.org/html/2610.14277#bib.bib48);[Wang et al\., 2024a](https://arxiv.org/html/2610.14277#bib.bib50);[Jia et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib20)\), and LoCo shows that a moving\-average estimate of the compression error, applied before quantization, keeps low\-bit gradient communication accurate at scale\([Xie et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib58)\)\. AQ\-SGD extends compensation from gradients to the activations at a pipeline\-parallel boundary, quantizing the change of each example’s activations since its previous visit\([Wang et al\., 2022](https://arxiv.org/html/2610.14277#bib.bib53)\)\. Error feedback works in all of these settings because the compressed data is a gradient, or changes slowly over training\. The training loop produces a new gradient for the same parameters at every step, so an error left uncorrected at one step can be added to the gradient of the next step, and nothing is lost\. A hidden state gives no such opportunity\. It is produced once, consumed by the next layer, and never communicated again, and at inference time there is no training loop at all\. Accordingly, error feedback applied to the activations of model\-parallel training gives no gain over plain compression\([Rudakov et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib40)\), and compression of model\-parallel activations has been studied empirically across pruning, learned, and quantized schemes\([Bian et al\., 2023](https://arxiv.org/html/2610.14277#bib.bib2)\)\. In tensor parallelism the data does not repeat, but the operation does\. The all\-reduce runs again at the next layer, and in a pre\-normalization transformer all its outputs feed the same residual stream \(Sec\.[3](https://arxiv.org/html/2610.14277#S3)\)\.Echobuilds its feedback on this repetition\. The reduction step exposes every rounding error exactly \(Sec\.[5](https://arxiv.org/html/2610.14277#S5)\), the compensator adds a correction at the following calls, and the corrections cancel the accumulated error along the residual stream, with a bound independent of depth \(Sec\.[6](https://arxiv.org/html/2610.14277#S6)\)\. To our knowledge,Echois the first to place feedback compensation inside a quantized collective for tensor parallelism\.

## 3Problem Formulation

### 3\.1The quantized all\-reduce and its error

An all\-reduce adds the contributions of thePPGPUs in the tensor\-parallel group, and this addition cannot be done in the low\-bit transport format, because a sum of low\-bit values overflows its range\. The sum is formed in high precision, and the low\-bit format only moves data between GPUs, so a quantized all\-reduce returns the exact sum plus the rounding error of the data it moved\. Section[4](https://arxiv.org/html/2610.14277#S4)states this constraint precisely and builds the decomposition that keeps the number of roundings as small as possible\.

Let𝐱k,i\\mathbf\{x\}\_\{k,i\}be the contribution held by GPUiiat thekk\-th all\-reduce of a pass, so an exact all\-reduce returns∑i𝐱k,i\\sum\_\{i\}\\mathbf\{x\}\_\{k,i\}\. The quantized all\-reduce, written𝖰𝖠𝖱\\mathsf\{QAR\}, returns

𝐲^k=𝖰𝖠𝖱⁡\(\{𝐱k,i\}i=1P\)=∑i=1P𝐱k,i\+𝜹k\.\\hat\{\\mathbf\{y\}\}\_\{k\}\\;=\\;\\mathsf\{QAR\}\\big\(\\\{\\mathbf\{x\}\_\{k,i\}\\\}\_\{i=1\}^\{P\}\\big\)\\;=\\;\\sum\_\{i=1\}^\{P\}\\mathbf\{x\}\_\{k,i\}\\;\+\\;\\bm\{\\delta\}\_\{k\}\.\(1\)The vector𝜹k\\bm\{\\delta\}\_\{k\}is the error the quantized all\-reduce injects at sitekk, and it is the only difference between𝐲^k\\hat\{\\mathbf\{y\}\}\_\{k\}and the exact sum\. Its norm is bounded by a constantqq,‖𝜹k‖≤q\\norm\{\\injnoise\_k\}\\leq q, set by the bit widthbb, the group sizegg, and the scale of the data \(Sec\.[4](https://arxiv.org/html/2610.14277#S4)\)\. Communication volume is proportional tobb, so abb\-bit all\-reduce sends about a fractionb/16b/16of the bytes of a 16\-bit all\-reduce, the most that lower precision alone can save\.

### 3\.2The communication noise process

A pre\-normalization transformer keeps a hidden state and, at each attention and MLP block, normalizes the hidden state, runs the block, and adds the result back\([Xiong et al\., 2020](https://arxiv.org/html/2610.14277#bib.bib60)\)\. In tensor\-parallel form each block produces its output in pieces, so GPUiicomputes a partial output𝐱k,i\\mathbf\{x\}\_\{k,i\}and an all\-reduce sums thePPpieces into the block output\. Writing𝐳k\\mathbf\{z\}\_\{k\}for the hidden state entering thekk\-th such block,

𝐳k\+1=𝐳k\+∑i=1P𝐱k,i\.\\mathbf\{z\}\_\{k\+1\}\\;=\\;\\mathbf\{z\}\_\{k\}\\;\+\\;\\sum\_\{i=1\}^\{P\}\\mathbf\{x\}\_\{k,i\}\.\(2\)Each transformer layer has two of these all\-reduces, one for its attention block and one for its MLP block, so a pass hasNNof them\. Qwen3\-0\.6B, with 28 layers, hasN=56N=56\.

With the quantized all\-reduce the block contributes𝐲^k=∑i𝐱k,i\+𝜹k\\hat\{\\mathbf\{y\}\}\_\{k\}=\\sum\_\{i\}\\mathbf\{x\}\_\{k,i\}\+\\bm\{\\delta\}\_\{k\}instead, so

𝐳k\+1=𝐳k\+∑i=1P𝐱k,i\+𝜹k\.\\mathbf\{z\}\_\{k\+1\}\\;=\\;\\mathbf\{z\}\_\{k\}\\;\+\\;\\sum\_\{i=1\}^\{P\}\\mathbf\{x\}\_\{k,i\}\\;\+\\;\\bm\{\\delta\}\_\{k\}\.\(3\)The useful block output∑i𝐱k,i\\sum\_\{i\}\\mathbf\{x\}\_\{k,i\}is unchanged and the quantized all\-reduce only adds the error𝜹k\\bm\{\\delta\}\_\{k\}\. Low\-bit tensor\-parallel communication is therefore a noise process on the computation graph, injecting an error at each all\-reduce site of a pass \(Fig\.[1](https://arxiv.org/html/2610.14277#S3.F1.fig1)\)\. The same process occurs every time the model runs\. A single forward pass in inference injectsNNsuch errors\. In training the same injection happens on the forward and the backward pass of every step, and over a full run it repeats across many steps, so the process acts over far more sites\. The point of injection also differs between the two passes\. On the forward pass𝜹k\\bm\{\\delta\}\_\{k\}joins the hidden state directly, while on the backward pass it enters the gradient of the hidden state after the normalization, so the forward and backward errors are corrected separately \(Sec\.[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)\)\.

hidden state𝐳\\mathbf\{z\}block00\+𝜹0\+\\,\\bm\{\\delta\}\_\{0\}\+\+block11\+𝜹1\+\\,\\bm\{\\delta\}\_\{1\}\+\+blockN−1N\\\!\-\\\!1\+𝜹N−1\+\\,\\bm\{\\delta\}\_\{N\\\!\-\\\!1\}\+\+⋯\\cdotsone all\-reduce per block, so a forward pass hasNNinjection sites

Figure 1:Low\-bit tensor\-parallel communication as a noise process\.A pre\-normalization transformer adds each attention and MLP block’s output to the hidden state\. In tensor\-parallel form the block ends in an all\-reduce, and replacing it with the quantized all\-reduce leaves the block output in place while adding an error𝜹k\\bm\{\\delta\}\_\{k\}at that site\. A forward pass hasNNsuch sites, and the backward pass injects at the same sites \(Sec\.[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)\)\.
### 3\.3Controlling the accumulated error

The output error is not set by any single𝜹k\\bm\{\\delta\}\_\{k\}\. Once injected, an error is carried to the output by the blocks that follow it, which respond to the already\-perturbed hidden state and amplify it\. The output error is built from theNNinjections after this amplification\. Its size is bounded by two factors\. The first is how much the network amplifies a perturbation, which Sec\.[6](https://arxiv.org/html/2610.14277#S6)bounds under one assumption on the trained network\. The second is how much the injections reinforce one another as they build up, measured by the largest partial sum of the injected sequence\. The amplification is a property of the network, so the factor a communication primitive can change is the partial sum\.

The partial sum can be kept from growing because the quantized all\-reduce reveals its own error\. The residual left on GPUiiafter the all\-reduce,𝐮k,i=𝐱k,i′−𝐲^k/P\\mathbf\{u\}\_\{k,i\}=\\mathbf\{x\}^\{\\prime\}\_\{k,i\}\-\\hat\{\\mathbf\{y\}\}\_\{k\}/P, with𝐱k,i′\\mathbf\{x\}^\{\\prime\}\_\{k,i\}the \(compensated\) contribution it sends into the all\-reduce \(Sec\.[5](https://arxiv.org/html/2610.14277#S5)\), sums across the GPUs to

∑i=1P𝐮k,i=−𝜹k,\\sum\_\{i=1\}^\{P\}\\mathbf\{u\}\_\{k,i\}\\;=\\;\-\\,\\bm\{\\delta\}\_\{k\},\(4\)so the group recovers the error it has just injected, at no extra communication\. The primitive feeds this measurement back: before the next quantization it adds a correction𝐂k\\mathbf\{C\}\_\{k\}, built from the errors measured at earlier sites, to the all\-reduce input\. Through the skip connection the correction enters the hidden state beside the injection, so the net error added at the site is

𝝆k=𝜹k\+𝐂k\.\\bm\{\\rho\}\_\{k\}\\;=\\;\\bm\{\\delta\}\_\{k\}\\;\+\\;\\mathbf\{C\}\_\{k\}\.\(5\)Because𝐂k\\mathbf\{C\}\_\{k\}is built from the earlier injected errors, it cancels their buildup in the running sum, so the partial sums stay bounded as the pass proceeds\. This is error feedback, and Sec\.[5](https://arxiv.org/html/2610.14277#S5)gives the specific filter\.

The factor to control is then the largest partial sum of these net errors\. Writing

𝐒t=∑k≤t𝝆k,\\mathbf\{S\}\_\{t\}\\;=\\;\\sum\_\{k\\leq t\}\\bm\{\\rho\}\_\{k\},\(6\)the requirement on the primitive is thatmaxt⁡‖𝐒t‖\\max\_\{t\}\\norm\{\\mathbf\{S\}\_t\}does not grow with the number of all\-reduces\. Without correction the partial sum is∑k≤t𝜹k\\sum\_\{k\\leq t\}\\bm\{\\delta\}\_\{k\}, whose largest value can reachN​qNqover a forward pass\. This growth causes the accuracy loss of a plain 4\-bit all\-reduce in inference and its instability in training\. Since the amplification is shared, inference and training both need a partial\-sum bound that stays flat as the number of sites grows\. Section[5](https://arxiv.org/html/2610.14277#S5)builds the correction𝐂k\\mathbf\{C\}\_\{k\}\. Section[6](https://arxiv.org/html/2610.14277#S6)proves that the partial summaxt⁡‖𝐒t‖\\max\_\{t\}\\norm\{\\mathbf\{S\}\_t\}stays bounded independently ofNNfor any error sequence on the forward pass, that its backward counterpart stays bounded in mean square at every site, and that its first\-order effect on the output, measured against the hidden state, stays at the level of a single rounding error\. Section[7](https://arxiv.org/html/2610.14277#S7)evaluates the accuracy\.

## 4Quantized P2P All\-Reduce

Echoreplaces the tensor\-parallel \(TP\) all\-reduce with a communication primitive that moves data between GPUs in low precision but forms the sum in high precision\. Two properties of this design are used throughout the paper\. First, the reduction is carried out in high precision and introduces no error of its own, so the all\-reduce output differs from the exact sum only because the transported data is quantized\. Second, every value passes through the quantizer at most twice, regardless of how many GPUs participate, so the number of roundings at each site does not grow with the tensor\-parallel degreePP\. The rest of the section describes an implementation that realizes both properties at low latency\.

### 4\.1A forced decomposition

Turning the lower precision of Sec\.[3](https://arxiv.org/html/2610.14277#S3)into a faster all\-reduce runs into a constraint\. The all\-reduce must add the contributions of thePPGPUs, and this addition cannot be carried out in the low\-bit transport format\. Two 4\-bit values can already add up to a number that no longer fits in four bits, so a low\-bit accumulator overflows\. Even where 4\-bit arithmetic is available, four bits are too few to hold a running partial sum without large rounding error, and low\-bit accumulation is in any case not widely supported across accelerators\. The sum is therefore formed in high precision, and the low\-bit format only moves data between GPUs\. Carried out naively, a quantized all\-reduce becomes a repeated cycle of dequantize, add, and requantize, and each requantization injects one of the errors𝜹k\\bm\{\\delta\}\_\{k\}of Sec\.[3](https://arxiv.org/html/2610.14277#S3)\. The rest of this section keeps the number of these requantizations as small as possible\.

We first recall the standard structure of an all\-reduce\. Each GPU, one rank of the TP group, holds a tensor split intoPPequal segments, one associated with each GPU\. A*reduce\-scatter*sums the segments across GPUs so that GPUjjis left with the sum of thejj\-th segments\. An*all\-gather*then copies every summed segment to every GPU, after which all GPUs hold the complete result\. This two\-phase form is the bandwidth\-optimal way to build an all\-reduce\([Patarasuk and Yuan, 2009](https://arxiv.org/html/2610.14277#bib.bib34)\)\. This section treats a single all\-reduce and drops the site indexkkof Sec\.[3](https://arxiv.org/html/2610.14277#S3), writing𝐱i\\mathbf\{x\}\_\{i\}for GPUii’s contribution𝐱k,i\\mathbf\{x\}\_\{k,i\}and𝐱ij\\mathbf\{x\}\_\{i\}^\{\\,j\}for itsjj\-th segment,

𝐬j\\displaystyle\\mathbf\{s\}^\{\\,j\}=∑i=1P𝐱ij,\\displaystyle=\\sum\_\{i=1\}^\{P\}\\mathbf\{x\}\_\{i\}^\{\\,j\},\(7\)𝖠𝗅𝗅𝖱𝖾𝖽𝗎𝖼𝖾⁡\(\{𝐱i\}\)\\displaystyle\\mathsf\{AllReduce\}\\big\(\\\{\\mathbf\{x\}\_\{i\}\\\}\\big\)=𝖠𝗅𝗅​\-​𝖦𝖺𝗍𝗁𝖾𝗋​\(𝐬1,…,𝐬P\)\.\\displaystyle=\\mathsf\{All\\text\{\-\}Gather\}\\big\(\\mathbf\{s\}^\{\\,1\},\\dots,\\mathbf\{s\}^\{\\,P\}\\big\)\.Because the segment sums𝐬j\\mathbf\{s\}^\{\\,j\}must be formed in high precision, we implement the reduce\-scatter with an*all\-to\-all*, the collective in which every GPU sends a different piece to every other GPU\. GPUiisends itsjj\-th segment to the owner GPUjj, which then holds thePPcopies it needs and adds them locally\. For each segment it does not own, GPUiiquantizes that segment tobbbits and writes it directly into GPUjj’s buffer over the peer\-to\-peer link\. The segment a GPU owns is never sent over the link, so it stays in full precision and enters the sum without any quantization error, and only theP−1P\-1copies arriving from peers are quantized\. GPUjjdequantizes thoseP−1P\-1copies, adds them to its own full\-precision segment in high precision to obtain𝐬j\\mathbf\{s\}^\{\\,j\}, and quantizes𝐬j\\mathbf\{s\}^\{\\,j\}once more for transport\. The all\-gather sends each quantized𝐬j\\mathbf\{s\}^\{\\,j\}back to the other GPUs, and every GPU dequantizes what it receives\. GPUjjreconstructs its own copy of𝐬j\\mathbf\{s\}^\{\\,j\}from that samebb\-bit code rather than keeping the high\-precision value, so that its result is bit\-for\-bit identical to the copies the other GPUs receive\. Figure[2](https://arxiv.org/html/2610.14277#S4.F2)\(a\) illustrates the process forP=4P=4, and Fig\.[4](https://arxiv.org/html/2610.14277#A3.F4)in Appendix[C](https://arxiv.org/html/2610.14277#A3)works the same case through with numbers\.

Among bandwidth\-optimal schemes, this decomposition also uses the fewest quantization steps\. A ring all\-reduce passes partial sums from GPU to GPU and requantizes each value once per hop,𝒪⁡\(P\)\\order\{\\TP\}times in total\. In our decomposition a value is quantized only when it crosses the link, once in the all\-to\-all and once in the all\-gather, so each value is rounded at most twice however many GPUs participate \(Table[1](https://arxiv.org/html/2610.14277#S4.T1.fig1)\)\. The two\-phase skeleton itself is standard and is shared by quantized systems such as ZeRO\+\+\([Wang et al\., 2024a](https://arxiv.org/html/2610.14277#bib.bib50)\), MegaScale\-MoE\([Jin et al\., 2026](https://arxiv.org/html/2610.14277#bib.bib21)\), and FlashComm\([Li et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib24);[Li et al\., 2025b](https://arxiv.org/html/2610.14277#bib.bib25)\)\. We adopt it because it minimizes the roundings\. On top of this, the compensator keeps the error accumulated over a whole forward pass at about twice the error of a single site, however deep the network \(Secs\.[5](https://arxiv.org/html/2610.14277#S5)and[6](https://arxiv.org/html/2610.14277#S6)\)\.

\(a\) 4\-bit P2P all\-reduce: route→\\rightarrowaccumulate→\\rightarrowshareP=4P=4shownInput on each GPUOwner\-side high\-precision sumIdentical output00112233GPU00GPU11GPU22GPU33Pushowner00Σ\\Sigmaowner11Σ\\Sigmaowner22Σ\\Sigmaowner33Σ\\Sigma00112233FPQ4Q\_\{4\}GatherD​Q4DQ\_\{4\}on all GPUsGPU00GPU11GPU22GPU33local FP segmentremote 4\-bit codeAt most 2 quantizations / element, independent ofPP\(b\) Tiled four\-stage pipelinetime→\\rightarrow\(schematic slots\)00112233445566PushQ4Q\_\{4\}\+ P2P writet0t\_\{0\}t1t\_\{1\}t2t\_\{2\}t3t\_\{3\}ReduceD​Q4DQ\_\{4\}\+ FP sum \+Q4Q\_\{4\}t0t\_\{0\}t1t\_\{1\}t2t\_\{2\}t3t\_\{3\}GatherP2P share \+ ownerD​Q4DQ\_\{4\}t0t\_\{0\}t1t\_\{1\}t2t\_\{2\}t3t\_\{3\}TailpeerD​Q4DQ\_\{4\}t0t\_\{0\}t1t\_\{1\}t2t\_\{2\}t3t\_\{3\}Cross\-GPU readiness: peer\-visible flags\|\\hskip 8\.19447pt\|\\hskip 8\.19447ptWithin\-GPU ordering: CUDA events4 tiles active concurrently

Figure 2:The quantized P2P all\-reduce\.\(a\) ForP=4P=4, each GPU sends only its non\-owned segments as 4\-bit codes to their owners\. Each owner dequantizes the received codes and sums them with its own unquantized segment in high precision\. The reduced segment is quantized once for the all\-gather, and every GPU, including the owner, dequantizes the same code to obtain identical outputs\. Each element is quantized at most twice, independently ofPP\. \(b\) Tiles are staggered across push, reduce, gather, and tail\. Peer\-visible flags signal cross\-GPU readiness and CUDA events order work within a GPU, allowing quantization and transfers of different tiles to overlap\.Table 1:Quantized P2P all\-reduce against a full\-precision ring all\-reduceonPPGPUs: how many times each element is \(re\)quantized, the source of the injected noise𝜹k\\bm\{\\delta\}\_\{k\}, and the bytes on the wire, withbbthe communication bit width\.
### 4\.2Pipelined execution and synchronization

The all\-reduce lies between two dependent matrix multiplications, so its data path cannot overlap with them, and the room for overlap on this path is inside the primitive, between its quantization work and its data transfers\. To expose it, we split each segment into small contiguous blocks, called*tiles*, and run the tiles through a four\-stage pipeline\. The stages are push, reduce, gather, and tail\. Push quantizes the remote segments and writes them to their owners\. Reduce dequantizes the received copies, adds them to the owned segment, and requantizes the result\. Gather sends this requantized segment to the other GPUs and, on the owning GPU, dequantizes it to produce that GPU’s own portion of the output\. Tail dequantizes the reduced segments arriving from the other owners\. Because the tiles pass through the stages independently, one tile can be quantized while an earlier tile is still traveling over NVLink \(Fig\.[2](https://arxiv.org/html/2610.14277#S4.F2)\(b\)\), so the quantization work of one tile is hidden behind the transfers of the tiles around it\. The stream and event schedule is given in Appendix[C](https://arxiv.org/html/2610.14277#A3)\.

The four stages run on different GPUs and must be kept in order across them\. Routing that ordering through the host CPU would add latency and serialize the pipeline, so we synchronize the GPUs directly over their peer\-to\-peer \(P2P\) links\. Each GPU keeps a small flag in memory that its peers can read over the P2P link\. A stage advances the flag after it finishes a tile, and the consuming stage on another GPU waits until the flag reaches the value it expects before it starts\. Ordering within a GPU is handled by CUDA events, and ordering across GPUs by these flags\.

Signaling after each tile, rather than after the whole tensor, lets communication and quantization use different block sizes, which they need because they favor opposite choices\. Peer\-to\-peer transfers reach their peak bandwidth with a few large blocks, whereas the quantization kernels are most efficient with many small blocks\. An implementation that exchanges the tensor only after it has been staged in full must commit to a single block size for both\. Tracking readiness at the tile level removes this coupling, because a consuming stage waits only for the specific tiles it needs rather than for the entire tensor\. The exact flag values are given in Appendix[C](https://arxiv.org/html/2610.14277#A3)\.

### 4\.3Quantization

The quantizer must not overflow thebb\-bit codes and must preserve model quality\. We quantize group\-wise\. Each group ofggconsecutive values is sent asbb\-bit codes together with a small set of per\-group parameters\.

A symmetric group tracks its absolute maximum and uses a single scaless,

q⁡\(x\)\\displaystyle q\(x\)=clip⁡\(round⁡\(x/s\),−\(2b−1−1\),2b−1−1\),\\displaystyle=\\mathrm\{clip\}\\\!\\big\(\\mathrm\{round\}\(x/s\),\\,\-\(2^\{b\-1\}\\\!\-\\\!1\),\\,2^\{b\-1\}\\\!\-\\\!1\\big\),\(8\)s\\displaystyle s=maxk⁡\|xk\|/\(2b−1−1\)\.\\displaystyle=\\max\\nolimits\_\{k\}\|x\_\{k\}\|\\,\\big/\\,\(2^\{b\-1\}\-1\)\.The scale sends the largest magnitude to the code2b−1−12^\{b\-1\}\\\!\-\\\!1, so the most negative code−2b−1\-2^\{b\-1\}is never used\. Leaving it out keeps the levels symmetric about zero, which we found to be more stable at 4 bits\.

Activations are often not centered on zero, so a symmetric grid, whose levels are symmetric about zero, spends much of its range on values that never appear\. An asymmetric group instead fits an affine map to the observed range\. This map has strictly more freedom, so it should be at least as accurate as the symmetric grid\. At 4 bits a naive asymmetric map often is not, and the reason lies in its zero\-point\. The map is set by a scalessand an integer zero\-pointzz,

s\\displaystyle s=maxk⁡xk−mink⁡xk2b−1,\\displaystyle=\\frac\{\\max\_\{k\}x\_\{k\}\-\\min\_\{k\}x\_\{k\}\}\{2^\{b\}\-1\},\(9\)q⁡\(x\)\\displaystyle q\(x\)=clip⁡\(round⁡\(x/s\)\+z,0,2b−1\),\\displaystyle=\\mathrm\{clip\}\\\!\\big\(\\mathrm\{round\}\(x/s\)\+z,\\,0,\\,2^\{b\}\-1\\big\),and dequantizes asx^=s⁡\(q−z\)\\hat\{x\}=s\\,\(q\-z\)\. The integerzzis the code that maps to00, so00is represented exactly\. A naive map instead anchors the grid at the group minimum, leaving the level nearest00at the non\-integer position−minkxk/s\-\\min\_\{k\}x\_\{k\}/s, so00falls between two codes\. Since activation distributions are sharply peaked at zero, this rounds the densest part of the distribution, and with only sixteen levels the error there outweighs the finer resolution the asymmetric map was meant to gain\.

Our policy is zero\-aware\. When a group straddles zero we round its zero\-point to the nearest integer\. When a group lies entirely on one side of zero the value00never occurs, so we keep the naive map and spend the whole grid on the values that are present\. Groups in between blend smoothly between the two\. This change alone removes the 4\-bit regression, and asymmetric quantization then outperforms the symmetric policy \(Sec\.[7](https://arxiv.org/html/2610.14277#S7)\)\.

We expose only the two parameters that matter, the bit widthb∈\{4,8\}b\\in\\\{4,8\\\}and the group sizegg, and otherwise keep both policies fixed\. The per\-group storage is compact, one value for symmetric and a scale and a zero\-point for asymmetric, and a backward\-pass scaling prevents small gradient groups from underflowing it \(Sec\.[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)\)\.

## 5Dual Error Feedback

The quantized all\-reduce injects the error𝜹k\\bm\{\\delta\}\_\{k\}of Eqn\. \([1](https://arxiv.org/html/2610.14277#S3.E1)\) at every site, with‖𝜹k‖≤q\\norm\{\\injnoise\_k\}\\leq q\. Left uncorrected, these errors accumulate, and the partial sum𝐒t\\mathbf\{S\}\_\{t\}of Eqn\. \([6](https://arxiv.org/html/2610.14277#S3.E6)\), which governs the output error, grows to orderN​qNqover a pass \(Sec\.[3\.3](https://arxiv.org/html/2610.14277#S3.SS3)\)\. We remove this growth with an error\-feedback compensator that realizes the correction𝐂k\\mathbf\{C\}\_\{k\}of Eqn\. \([5](https://arxiv.org/html/2610.14277#S3.E5)\)\. It recovers each injected error, summed over the group, from the output of the all\-reduce and feeds a running estimate of it back into the next all\-reduce, so the injected errors cancel in𝐒t\\mathbf\{S\}\_\{t\}rather than accumulate\. Section[5\.1](https://arxiv.org/html/2610.14277#S5.SS1)defines the compensator and Sec\.[5\.2](https://arxiv.org/html/2610.14277#S5.SS2)shows, term by term, why it keeps𝐒t\\mathbf\{S\}\_\{t\}bounded\.

### 5\.1Two\-timescale error\-feedback compensator

On each GPU the compensator keeps two moving averages, applies a blend of them as a correction before each all\-reduce, and refreshes them from the residual it then forms\. The update at sitekkon GPUiiruns as follows\.

It first forms a correction from its current averages and adds it to the contribution, before the all\-reduce quantizes it,

𝐜k,i=η⁡\(a​𝐦k−1,i\(1\)\+\(1−a\)​𝐦k−1,i\(2\)\),𝐱k,i′=𝐱k,i\+𝐜k,i,\\mathbf\{c\}\_\{k,i\}=\\eta\\big\(a\\,\\mathbf\{m\}^\{\(1\)\}\_\{k\-1,i\}\+\(1\-a\)\\,\\mathbf\{m\}^\{\(2\)\}\_\{k\-1,i\}\\big\),\\qquad\\mathbf\{x\}^\{\\prime\}\_\{k,i\}=\\mathbf\{x\}\_\{k,i\}\+\\mathbf\{c\}\_\{k,i\},\(10\)with blend weighta∈\[0,1\]a\\in\[0,1\]and gainη∈\[0,1\]\\eta\\in\[0,1\]\. The all\-reduce acts on the compensated contribution𝐱k,i′\\mathbf\{x\}^\{\\prime\}\_\{k,i\}and returns𝐲^k=∑i𝐱k,i′\+𝜹k\\hat\{\\mathbf\{y\}\}\_\{k\}=\\sum\_\{i\}\\mathbf\{x\}^\{\\prime\}\_\{k,i\}\+\\bm\{\\delta\}\_\{k\}\(Eqn\. \([1](https://arxiv.org/html/2610.14277#S3.E1)\)\)\. From it each GPUiiforms the residual𝐮k,i=𝐱k,i′−𝐲^k/P\\mathbf\{u\}\_\{k,i\}=\\mathbf\{x\}^\{\\prime\}\_\{k,i\}\-\\hat\{\\mathbf\{y\}\}\_\{k\}/Pof Sec\.[3\.3](https://arxiv.org/html/2610.14277#S3.SS3), whose two operands are already on the device, so it costs no communication, and whose group sum recovers the injected error,∑i𝐮k,i=−𝜹k\\sum\_\{i\}\\mathbf\{u\}\_\{k,i\}=\-\\bm\{\\delta\}\_\{k\}\(Eqn\. \([4](https://arxiv.org/html/2610.14277#S3.E4)\)\)\. The two averages are then refreshed with this residual, at decay ratesβ1,β2∈\[0,1\)\\beta\_\{1\},\\beta\_\{2\}\\in\[0,1\),

𝐦k,i\(j\)=βj𝐦k−1,i\(j\)\+\(1−βj\)𝐮k,i,j=1,2\.\\mathbf\{m\}^\{\(j\)\}\_\{k,i\}=\\beta\_\{j\}\\,\\mathbf\{m\}^\{\(j\)\}\_\{k\-1,i\}\+\(1\-\\beta\_\{j\}\)\\,\\mathbf\{u\}\_\{k,i\},\\qquad j=1,2\.\(11\)Equation \([10](https://arxiv.org/html/2610.14277#S5.E10)\) is the error\-feedback step\. Summed over the group, the averages estimate the errors injected at earlier sites, and feeding them back into the input corrects those errors rather than passing them on\. Each GPU stores and updates its own averages, so the compensator adds no communication, and summed over the group the corrections give the aggregate𝐂k=∑i𝐜k,i\\mathbf\{C\}\_\{k\}=\\sum\_\{i\}\\mathbf\{c\}\_\{k,i\}of Eqn\. \([5](https://arxiv.org/html/2610.14277#S3.E5)\)\. The averages are reset to zero at the start of each pass\. One pair serves all sites of a pass, so the state costs two tensors of the size of the contribution per pass direction\. Algorithm[1](https://arxiv.org/html/2610.14277#algorithm1)lists the update\.

Algorithm 1Compensated all\-reduce at sitekk\(GPUii\)Input:contribution𝐱k,i\\mathbf\{x\}\_\{k,i\}, state𝐦\(1\),𝐦\(2\)\\mathbf\{m\}^\{\(1\)\},\\mathbf\{m\}^\{\(2\)\}, and parametersβ1,β2,a,η\\beta\_\{1\},\\beta\_\{2\},a,\\eta

Output:block output

𝐲^k\\hat\{\\mathbf\{y\}\}\_\{k\}
1

𝐜←η⁡\(a​𝐦\(1\)\+\(1−a\)​𝐦\(2\)\)\\mathbf\{c\}\\leftarrow\\eta\\big\(a\\,\\mathbf\{m\}^\{\(1\)\}\+\(1\-a\)\\,\\mathbf\{m\}^\{\(2\)\}\\big\)//correction

2

𝐱′←𝐱k,i\+𝐜\\mathbf\{x\}^\{\\prime\}\\leftarrow\\mathbf\{x\}\_\{k,i\}\+\\mathbf\{c\}//apply

3

𝐲^k←𝖰𝖠𝖱⁡\(\{𝐱k,j′\}j\)\\hat\{\\mathbf\{y\}\}\_\{k\}\\leftarrow\\mathsf\{QAR\}\\big\(\\\{\\mathbf\{x\}^\{\\prime\}\_\{k,j\}\\\}\_\{j\}\\big\)//all\-reduce

4

𝐮←𝐱′−𝐲^k/P\\mathbf\{u\}\\leftarrow\\mathbf\{x\}^\{\\prime\}\-\\hat\{\\mathbf\{y\}\}\_\{k\}/P//residual

5

𝐦\(1\)←β1​𝐦\(1\)\+\(1−β1\)​𝐮\\mathbf\{m\}^\{\(1\)\}\\leftarrow\\beta\_\{1\}\\mathbf\{m\}^\{\(1\)\}\+\(1\-\\beta\_\{1\}\)\\,\\mathbf\{u\}//update

6

𝐦\(2\)←β2​𝐦\(2\)\+\(1−β2\)​𝐮\\mathbf\{m\}^\{\(2\)\}\\leftarrow\\beta\_\{2\}\\mathbf\{m\}^\{\(2\)\}\+\(1\-\\beta\_\{2\}\)\\,\\mathbf\{u\}//update

### 5\.2Suppression of error accumulation

We first write the correction in closed form\. Unrolling Eqn\. \([11](https://arxiv.org/html/2610.14277#S5.E11)\) from the reset gives𝐦k,i\(j\)=\(1−βj\)​∑s≤kβjk−s​𝐮s,i\\mathbf\{m\}^\{\(j\)\}\_\{k,i\}=\(1\-\\beta\_\{j\}\)\\sum\_\{s\\leq k\}\\beta\_\{j\}^\{\\,k\-s\}\\mathbf\{u\}\_\{s,i\}, so each average is a decaying sum of past residuals\. Substituting into Eqn\. \([10](https://arxiv.org/html/2610.14277#S5.E10)\), summing over the group, and using∑i𝐮s,i=−𝜹s\\sum\_\{i\}\\mathbf\{u\}\_\{s,i\}=\-\\bm\{\\delta\}\_\{s\}gives the correction as a weighted sum of the earlier injected errors,

𝐂k\\displaystyle\\mathbf\{C\}\_\{k\}=−η∑s<kwk−1−s𝜹s,\\displaystyle=\-\\eta\\sum\_\{s<k\}w\_\{k\-1\-s\}\\,\\bm\{\\delta\}\_\{s\},\(12\)wr\\displaystyle w\_\{r\}:=a⁡\(1−β1\)​β1r\+\(1−a\)​\(1−β2\)​β2r,\\displaystyle:=a\(1\-\\beta\_\{1\}\)\\beta\_\{1\}^\{\\,r\}\+\(1\-a\)\(1\-\\beta\_\{2\}\)\\beta\_\{2\}^\{\\,r\},wherewrw\_\{r\}is the weight placed on an error of agerr, and the weights sum to one,∑r≥0wr=1\\sum\_\{r\\geq 0\}w\_\{r\}=1\.

We can now read off how much of each injected error survives in the partial sum\. Putting𝝆k=𝜹k\+𝐂k\\bm\{\\rho\}\_\{k\}=\\bm\{\\delta\}\_\{k\}\+\\mathbf\{C\}\_\{k\}into𝐒t=∑k≤t𝝆k\\mathbf\{S\}\_\{t\}=\\sum\_\{k\\leq t\}\\bm\{\\rho\}\_\{k\}\(Eqn\. \([6](https://arxiv.org/html/2610.14277#S3.E6)\)\) and collecting the terms in each𝜹s\\bm\{\\delta\}\_\{s\},

𝐒t=∑s≤tϕt−s​𝜹s,ϕm:=1−η​∑r=0m−1wr\.\\mathbf\{S\}\_\{t\}=\\sum\_\{s\\leq t\}\\phi\_\{t\-s\}\\,\\bm\{\\delta\}\_\{s\},\\qquad\\phi\_\{m\}:=1\-\\eta\\sum\_\{r=0\}^\{m\-1\}w\_\{r\}\.\(13\)The coefficientϕm\\phi\_\{m\}is the fraction of an injected error that still remains in the partial summmsites after it enters\. It starts atϕ0=1\\phi\_\{0\}=1and decreases with age, since∑r=0m−1wr\\sum\_\{r=0\}^\{m\-1\}w\_\{r\}climbs toward∑r≥0wr=1\\sum\_\{r\\geq 0\}w\_\{r\}=1\. At gainη=1\\eta=1it decreases to zero\. An old error is then subtracted away and eventually leaves𝐒t\\mathbf\{S\}\_\{t\}, so the partial sum carries only a bounded window of recent errors\. As a result,maxt⁡‖𝐒t‖\\max\_\{t\}\\norm\{\\mathbf\{S\}\_t\}stays bounded as the pass lengthens, in place of the order\-N​qNqgrowth of the uncorrected sum∑k≤t𝜹k\\sum\_\{k\\leq t\}\\bm\{\\delta\}\_\{k\}\(Sec\.[3\.3](https://arxiv.org/html/2610.14277#S3.SS3)\)\. Section[6](https://arxiv.org/html/2610.14277#S6)turns Eqn\. \([13](https://arxiv.org/html/2610.14277#S5.E13)\) into the exact bound, and treats the caseη<1\\eta<1, whereϕm\\phi\_\{m\}settles at1−η1\-\\etaand a fixed share of each error persists\.

The two averages, with decay ratesβ1\\beta\_\{1\}andβ2\\beta\_\{2\}, give the correction two time scales\. A single average with decayβ\\betaweights past residuals by\(1−β\)​βr\(1\-\\beta\)\\beta^\{r\}, which shrink by the factorβ\\betaeach site, so it spans about1/\(1−β\)1/\(1\-\\beta\)sites\. The decay rateβ1\\beta\_\{1\}sets a longer memory of about1/\(1−β1\)1/\(1\-\\beta\_\{1\}\)sites, andβ2\\beta\_\{2\}sets a shorter memory of about1/\(1−β2\)1/\(1\-\\beta\_\{2\}\)sites\. The longer one estimates the part of the injected error that stays similar from site to site, and the shorter one adapts to the part that changes quickly\. One average alone cannot do both\. Combining moving averages at several time scales in this way is a known technique for accelerating first\-order optimization\([Pagliardini et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib33);[Xie et al\., 2024b](https://arxiv.org/html/2610.14277#bib.bib57)\)\. The two decay rates and the blendaaare studied in Sec\.[7](https://arxiv.org/html/2610.14277#S7)\.

### 5\.3Fusing the compensator into the all\-reduce

The compensator of Sec\.[5\.1](https://arxiv.org/html/2610.14277#S5.SS1)runs two steps at every site, the apply step𝐱′=𝐱\+𝐜\\mathbf\{x\}^\{\\prime\}=\\mathbf\{x\}\+\\mathbf\{c\}of Eqn\. \([10](https://arxiv.org/html/2610.14277#S5.E10)\) before the all\-reduce and, after it returns𝐲^\\hat\{\\mathbf\{y\}\}, the update step of Eqn\. \([11](https://arxiv.org/html/2610.14277#S5.E11)\) on the residual𝐮=𝐱′−𝐲^/P\\mathbf\{u\}=\\mathbf\{x\}^\{\\prime\}\-\\hat\{\\mathbf\{y\}\}/P\. Here and in the rest of this section we drop the site and GPU indices\. Each step reads and writes tensors as large as𝐱\\mathbf\{x\}itself, so implemented as separate kernels the two steps add full memory passes of their own\. This subsection shows that the apply step can instead run inside the entry pass of the quantized all\-reduce of Sec\.[4](https://arxiv.org/html/2610.14277#S4)and the update beside the next matrix multiplication, at a small added latency \(Table[3](https://arxiv.org/html/2610.14277#S7.T3)\)\. Section[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)then describes the deployment of the compensator on the backward pass\.

A direct implementation surrounds the all\-reduce with two kernels, an apply kernel that forms𝐱′=𝐱\+𝐜\\mathbf\{x\}^\{\\prime\}=\\mathbf\{x\}\+\\mathbf\{c\}and writes it to a buffer of its own, and an update kernel that forms𝐮=𝐱′−𝐲^/P\\mathbf\{u\}=\\mathbf\{x\}^\{\\prime\}\-\\hat\{\\mathbf\{y\}\}/Pand refreshes𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}\. The two kernels are costly in two ways\. They move about ten tensor\-sized arrays through memory, while the all\-reduce itself reads𝐱\\mathbf\{x\}once and writes𝐲^\\hat\{\\mathbf\{y\}\}once\. They also stretch the critical path: the apply kernel must deliver𝐱′\\mathbf\{x\}^\{\\prime\}before the all\-reduce can start, and the update kernel needs𝐲^\\hat\{\\mathbf\{y\}\}, so it starts only after the all\-reduce returns\. The all\-reduce is bandwidth bound, so the extra traffic cannot hide behind computation, and the two effects together place the full cost of compensation in the latency of every site\.

On entry the all\-reduce already moves the full tensor through registers\. Every element of𝐱\\mathbf\{x\}is loaded, the quantization statistics are formed, and the value is quantized for transport \(Sec\.[4\.2](https://arxiv.org/html/2610.14277#S4.SS2)\)\. The apply step needs𝐱\\mathbf\{x\}, which the entry pass is holding\. The update step needs𝐲^\\hat\{\\mathbf\{y\}\}, which the next matrix multiplication only reads, so the update can run while that multiplication does\.

The apply step is therefore fused into the entry pass\. As each element of𝐱\\mathbf\{x\}arrives in registers, the kernel also loads𝐦\(1\)\\mathbf\{m\}^\{\(1\)\}and𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}, forms𝐜=η⁡\(a​𝐦\(1\)\+\(1−a\)​𝐦\(2\)\)\\mathbf\{c\}=\\eta\\,\(a\\,\\mathbf\{m\}^\{\(1\)\}\+\(1\-a\)\\,\\mathbf\{m\}^\{\(2\)\}\)and𝐱′=𝐱\+𝐜\\mathbf\{x\}^\{\\prime\}=\\mathbf\{x\}\+\\mathbf\{c\}in registers, and only then computes the group statistics and quantizes\. The tensor𝐱′\\mathbf\{x\}^\{\\prime\}exists only in registers and is never written to memory\. This is also the only place where the step can go\. The all\-reduce quantizes𝐱′\\mathbf\{x\}^\{\\prime\}, not𝐱\\mathbf\{x\}\(Sec\.[5\.1](https://arxiv.org/html/2610.14277#S5.SS1)\), so the quantization statistics must be taken on𝐱′\\mathbf\{x\}^\{\\prime\}, and the entry load is the last moment before they are formed\.

The update step is taken off the critical path\. Once the exit pass has written𝐲^\\hat\{\\mathbf\{y\}\}, the update runs on a separate stream alongside the next matrix multiplication\. It recomputes𝐱′=𝐱\+𝐜\\mathbf\{x\}^\{\\prime\}=\\mathbf\{x\}\+\\mathbf\{c\}from𝐱\\mathbf\{x\},𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}, forms𝐮=𝐱′−𝐲^/P\\mathbf\{u\}=\\mathbf\{x\}^\{\\prime\}\-\\hat\{\\mathbf\{y\}\}/P, and applies𝐦\(j\)←βj​𝐦\(j\)\+\(1−βj\)​𝐮\\mathbf\{m\}^\{\(j\)\}\\leftarrow\\beta\_\{j\}\\,\\mathbf\{m\}^\{\(j\)\}\+\(1\-\\beta\_\{j\}\)\\,\\mathbf\{u\}in place\. The apply step of the next site waits for it, so it always reads current averages\. Every GPU, owner included, dequantizes𝐲^\\hat\{\\mathbf\{y\}\}from the same transported code \(Sec\.[4\.1](https://arxiv.org/html/2610.14277#S4.SS1)\), so the residuals across the group are formed from a common𝐲^\\hat\{\\mathbf\{y\}\}, as the identity∑i𝐮i=−𝜹\\sum\_\{i\}\\mathbf\{u\}\_\{i\}=\-\\bm\{\\delta\}of Eqn\. \([4](https://arxiv.org/html/2610.14277#S3.E4)\) requires\.

After fusion the apply kernel and the𝐱′\\mathbf\{x\}^\{\\prime\}buffer are gone, and neither step runs as a serial phase on the critical path\. The apply step inherits the pipelining of Sec\.[4\.2](https://arxiv.org/html/2610.14277#S4.SS2)and runs tile by tile inside the entry stage, overlapped with the communication of the other tiles, and the update runs beside the next matrix multiplication\. The extra traffic falls from about ten tensor\-sized arrays to about eight, reading𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}at entry, reading𝐱\\mathbf\{x\},𝐲^\\hat\{\\mathbf\{y\}\},𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}in the update, and writing𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}back\. Six of the eight are the cost any implementation pays to consult the averages and refresh them, and the update traffic overlaps with the matrix multiplication\. The added arithmetic hides behind the memory accesses of the entry pass and of the update, and Table[3](https://arxiv.org/html/2610.14277#S7.T3)measures the resulting cost\. Fusion changes when and where each step executes, not what it computes, and𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}remain the only state, updated in place\.

### 5\.4The backward pass

Training runs the all\-reduce in both directions\. The compensator is unchanged on the backward pass, with the same recurrence and the same coefficients, but its working conditions change in two ways\.

First, backward errors enter the computation graph at a different point\. A forward site delivers its output directly to the residual addition, while the output of a backward site reaches the gradient stream through the Jacobian of the normalization \(Sec\.[3\.2](https://arxiv.org/html/2610.14277#S3.SS2)\), the setting in which Appendix[A](https://arxiv.org/html/2610.14277#A1)proves the backward source bound\. Residuals of the two directions therefore have different coordinates and different magnitudes, and a single pair𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}fed by both streams would form a correction that matches neither pass\. Each direction keeps its own pair and runs the recurrence of Sec\.[5\.1](https://arxiv.org/html/2610.14277#S5.SS1)along its own sequence of sites\.

Second, gradients are small enough to underflow the storage of the quantizer’s scale\. The quantizer of Sec\.[4\.3](https://arxiv.org/html/2610.14277#S4.SS3)stores one scalessper group in half precision\. Backward gradients are orders of magnitude smaller than activations, so a group scale can fall below the smallest positive half\-precision number and load back as zero\. Every value in that group then dequantizes to zero\. The remedy is a change of units around the communication\. Before the quantized all\-reduce, every GPU multiplies its gradient by the same power\-of\-two factor, chosen from the largest gradient magnitude across the tensor\-parallel group with a one\-scalar all\-reduce\. After the all\-reduce, it divides the result by that factor\. The two steps of the compensator run outside the scaling, so𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}stay in original gradient units\. The scaling is exact and leaves the analysis of Secs\.[5\.1](https://arxiv.org/html/2610.14277#S5.SS1)and[6](https://arxiv.org/html/2610.14277#S6)unchanged, as Appendix[C](https://arxiv.org/html/2610.14277#A3)verifies\.

## 6Theoretical Guarantees

### 6\.1Decomposition of the output error

The quantized all\-reduce injects an error at each of theNNsites of a pass\. Let𝐳k\\mathbf\{z\}\_\{k\}be the hidden state carried by the exact all\-reduce and𝐳^k\\hat\{\\mathbf\{z\}\}\_\{k\}the hidden state carried by the quantized one, so that

𝐳k\+1=𝐳k\+Fk​\(𝐳k\),𝐳^k\+1=𝐳^k\+Fk​\(𝐳^k\)\+𝝆k,\\mathbf\{z\}\_\{k\+1\}=\\mathbf\{z\}\_\{k\}\+F\_\{k\}\(\\mathbf\{z\}\_\{k\}\),\\qquad\\hat\{\\mathbf\{z\}\}\_\{k\+1\}=\\hat\{\\mathbf\{z\}\}\_\{k\}\+F\_\{k\}\(\\hat\{\\mathbf\{z\}\}\_\{k\}\)\+\\bm\{\\rho\}\_\{k\},\(14\)whereFkF\_\{k\}is the block at sitekk, the map that normalizes the hidden state and applies the attention or MLP transformation, and𝝆k=𝜹k\+𝐂k\\bm\{\\rho\}\_\{k\}=\\bm\{\\delta\}\_\{k\}\+\\mathbf\{C\}\_\{k\}is the net error of Eqn\. \([5](https://arxiv.org/html/2610.14277#S3.E5)\)\. Write𝐄k=𝐳^k−𝐳k\\mathbf\{E\}\_\{k\}=\\hat\{\\mathbf\{z\}\}\_\{k\}\-\\mathbf\{z\}\_\{k\}for the deviation between the two, with𝐄0=𝟎\\mathbf\{E\}\_\{0\}=\\mathbf\{0\}\. Subtracting the recursions of Eqn\. \([14](https://arxiv.org/html/2610.14277#S6.E14)\),

𝐄k\+1=𝐄k\+𝝍k\+𝝆k,𝝍k=Fk​\(𝐳^k\)−Fk​\(𝐳k\)\.\\mathbf\{E\}\_\{k\+1\}=\\mathbf\{E\}\_\{k\}\+\\bm\{\\psi\}\_\{k\}\+\\bm\{\\rho\}\_\{k\},\\qquad\\bm\{\\psi\}\_\{k\}=F\_\{k\}\(\\hat\{\\mathbf\{z\}\}\_\{k\}\)\-F\_\{k\}\(\\mathbf\{z\}\_\{k\}\)\.\(15\)Equation \([15](https://arxiv.org/html/2610.14277#S6.E15)\) is exact, and it separates the two roles of a site\. The term𝝍k\\bm\{\\psi\}\_\{k\}propagates the deviation accumulated so far, and adds nothing of its own, since𝐄k\\mathbf\{E\}\_\{k\}determines it\. The term𝝆k\\bm\{\\rho\}\_\{k\}injects the new error of sitekk\. ExpandingFkF\_\{k\}about𝐳k\\mathbf\{z\}\_\{k\},

𝝍k=𝐉k​𝐄k\+𝐫k,‖𝐫k‖≤12​Hk​‖𝐄k‖2,\\bm\{\\psi\}\_\{k\}=\\mathbf\{J\}\_\{k\}\\mathbf\{E\}\_\{k\}\+\\mathbf\{r\}\_\{k\},\\qquad\\norm\{\\mathbf\{r\}\_k\}\\leq\\tfrac\{1\}\{2\}H\_\{k\}\\norm\{\\mathbf\{E\}\_k\}^\{2\},\(16\)with𝐉k\\mathbf\{J\}\_\{k\}the Jacobian ofFkF\_\{k\}at𝐳k\\mathbf\{z\}\_\{k\}andHkH\_\{k\}a bound on its second derivative on the segment from𝐳k\\mathbf\{z\}\_\{k\}to𝐳^k\\hat\{\\mathbf\{z\}\}\_\{k\}\. Substituting Eqn\. \([16](https://arxiv.org/html/2610.14277#S6.E16)\) into Eqn\. \([15](https://arxiv.org/html/2610.14277#S6.E15)\) and unrolling from𝐄0=𝟎\\mathbf\{E\}\_\{0\}=\\mathbf\{0\},

𝐄N\\displaystyle\\mathbf\{E\}\_\{N\}=∑t=0N−1𝚽N,t\+1​\(𝝆t\+𝐫t\),\\displaystyle=\\sum\_\{t=0\}^\{N\-1\}\\bm\{\\Phi\}\_\{N,t\+1\}\\big\(\\bm\{\\rho\}\_\{t\}\+\\mathbf\{r\}\_\{t\}\\big\),\(17\)𝚽N,j\\displaystyle\\bm\{\\Phi\}\_\{N,j\}=\(𝐈\+𝐉N−1\)⋯\(𝐈\+𝐉j\)\.\\displaystyle=\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{N\-1\}\)\\cdots\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{j\}\)\.The deviation at the output is a sum of the injected errors, each propagated there by the Jacobians of the sites that follow it, together with the second\-order remainders\. The remainders are second order in the injected errors, and Proposition[1](https://arxiv.org/html/2610.14277#Thmproposition1)in Appendix[A](https://arxiv.org/html/2610.14277#A1)quantifies them\. The first\-order terms therefore carry the analysis, and two quantities remain to be bounded, the propagation𝚽\\bm\{\\Phi\}and the accumulation of the𝝆t\\bm\{\\rho\}\_\{t\}\.

The accumulation must be measured through the partial sum𝐒t\\mathbf\{S\}\_\{t\}of Eqn\. \([6](https://arxiv.org/html/2610.14277#S3.E6)\), not through the individual errors\. A term\-by\-term triangle inequality on Eqn\. \([17](https://arxiv.org/html/2610.14277#S6.E17)\) allows‖𝝆k‖\\norm\{\\bm\{\\rho\}\_k\}as large as2​q2qand gives a bound of orderN​qNq\. This bound shows no benefit from error feedback, because feedback does not reduce any single net error\. Instead, feedback makes the errors cancel as they accumulate \(Sec\.[3\.3](https://arxiv.org/html/2610.14277#S3.SS3)\)\. Summation by parts rewrites the first\-order part of Eqn\. \([17](https://arxiv.org/html/2610.14277#S6.E17)\) so that the errors enter only through their partial sums, with coefficients formed from the propagation \(Appendix[A](https://arxiv.org/html/2610.14277#A1)\)\.

Section[6\.2](https://arxiv.org/html/2610.14277#S6.SS2)bounds the two in turn, the partial sums by a constant for any error sequence, and the propagation in mean square under one assumption on the trained network\. Neither result restates existing error feedback theory\. That theory concerns gradient compression, where the compressed errors perturb an optimization step and the outcome is a convergence rate\. Here the errors enter the forward computation and are then carried through the network itself\. This propagation has no counterpart in the gradient setting, and Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1)and Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)address it\.

### 6\.2Depth\-independent bounds

This subsection contains two results\. Theorem[1](https://arxiv.org/html/2610.14277#Thmtheorem1)bounds the source\. With compensation, the accumulated error at the sites stays below a constant, about2​q2qat gainη=1\\eta=1, for any error sequence, while without compensation it can reachN​qNq\. This requires no probabilistic model and no property of the network\. Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)bounds the output\. Under a noise model for the injected errors and one assumption on the trained network, the mean\-square first\-order deviation of the output, measured against the energy of the hidden state, stays at the level set by a single rounding error, at every depth\. Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)builds on Theorem[1](https://arxiv.org/html/2610.14277#Thmtheorem1), whose constant reappears inside its bound\. The new element is the propagation, which Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1)controls\.

We start with the source\. At gainη=1\\eta=1the corrections return each error in full\. Of an error injectedmmsites ago, the fraction∑r<mwr\\sum\_\{r<m\}w\_\{r\}has been subtracted back out by later corrections, and the fraction

Tm:=∑r≥mwrT\_\{m\}:=\\sum\_\{r\\geq m\}w\_\{r\}\(18\)remains\. These tails are exactly the coefficientsϕm\\phi\_\{m\}of Eqn\. \([13](https://arxiv.org/html/2610.14277#S5.E13)\), so each error enters the partial sum with weight11and decays toward zero, and no partial sum can exceed the total of the tails\.

###### Theorem 1\(source bound\)\.

At gainη=1\\eta=1, for every error sequence with‖𝛅k‖≤q\\norm\{\\injnoise\_k\}\\leq q,

maxt⁡‖𝐒t‖\\displaystyle\\max\_\{t\}\\norm\{\\mathbf\{S\}\_t\}≤q​∑m≥0Tm=\(1\+μw\)​q,\\displaystyle\\leq\\;q\\sum\_\{m\\geq 0\}T\_\{m\}\\;=\\;\(1\+\\mu\_\{w\}\)\\,q,\(19\)μw\\displaystyle\\mu\_\{w\}=∑r≥0r​wr,\\displaystyle=\\;\\textstyle\\sum\_\{r\\geq 0\}r\\,w\_\{r\},whereμw\\mu\_\{w\}is the mean age, in sites, at which the compensator returns an injected error\. Without compensation the same quantity reachesN​qNq\.

The theorem has one hypothesis, the boundqqon each error\. TheTmT\_\{m\}are nonnegative, so the proof is a triangle inequality, and it covers errors that are dependent, correlated with the activations, or chosen adversarially\. The constant reflects how the compensator works\. An injected error stays in the partial sum until later corrections have subtracted it back out, which takes1\+μw1\+\\mu\_\{w\}sites on average, about2\.12\.1for the decay rates and blend used in the experiments\. At any moment the partial sum therefore holds only the errors of the last few sites, about two errors in total\. Without compensation errors are never subtracted, allNNof them accumulate, and the partial sum reachesN​qNq\. Compensation reduces the accumulated source fromN​qNqto\(1\+μw\)​q\(1\+\\mu\_\{w\}\)q, a factor ofN/\(1\+μw\)N/\(1\+\\mu\_\{w\}\), about2727at theN=56N=56of Sec\.[3\.2](https://arxiv.org/html/2610.14277#S3.SS2)\.

Theorem[1](https://arxiv.org/html/2610.14277#Thmtheorem1)controls what the sites accumulate\. The deviation at the output also depends on the factor𝚽\\bm\{\\Phi\}of Eqn\. \([17](https://arxiv.org/html/2610.14277#S6.E17)\), the transformation the network applies to each error on its way to the output\. Bounds that take the worst case of every site grow with depth\. The standard estimate is‖𝚽‖≤\(1\+L\)N\\norm\{\\bm\{\\Phi\}\}\\leq\(1\+L\)^\{N\}, withLLa Lipschitz constant of the blocks\. Growth with depth is not an artifact of the estimate\. Proposition[2](https://arxiv.org/html/2610.14277#Thmproposition2)\(a\) in Appendix[A](https://arxiv.org/html/2610.14277#A1)constructs a network that amplifies one fixed direction at every site, and in it even isotropic injections produce a deviation that grows exponentially inN\\sqrt\{N\}\. A depth\-independent bound must therefore exclude networks that keep amplifying one direction\. The noise model rules out injections aligned with such a direction, and Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1)rules out the networks\. These are the two hypotheses of Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)\.

We first treat the injections\. A rounding error is fixed by the quantizer at the moment of injection, from the activations alone, with no reference to the directions the network later amplifies\. We model the errors accordingly\. Conditioned on everything injected earlier,𝜹s\\bm\{\\delta\}\_\{s\}has mean zero, no preferred direction, and norm at mostqq\([Widrow and Kollár, 2008](https://arxiv.org/html/2610.14277#bib.bib55)\)\. Under this model the contributions of different sites are uncorrelated, and the linearized deviation𝐄Nlin\\mathbf\{E\}\_\{N\}^\{\\mathrm\{lin\}\}, the first\-order part of Eqn\. \([17](https://arxiv.org/html/2610.14277#S6.E17)\), satisfies

𝔼​‖𝐄Nlin‖2≤q2d​∑s‖𝐊s‖F2,\\mathbb\{E\}\\,\\bigl\\\|\\mathbf\{E\}\_\{N\}^\{\\mathrm\{lin\}\}\\bigr\\\|^\{2\}\\;\\leq\\;\\frac\{q^\{2\}\}\{d\}\\sum\_\{s\}\\norm\{\\mathbf\{K\}\_s\}\_\{F\}^\{2\},\(20\)whereddis the dimension of the hidden state,𝐊s\\mathbf\{K\}\_\{s\}is the matrix carrying𝜹s\\bm\{\\delta\}\_\{s\}to the output, built from the tailsTmT\_\{m\}and the propagation𝚽\\bm\{\\Phi\}\(Appendix[A](https://arxiv.org/html/2610.14277#A1)\), and‖⋅‖F\\norm\{\\cdot\}\_\{F\}is the Frobenius norm\.

We next treat the networks\. The reference scale here is the hidden state itself\. Letσk2=‖𝐳k‖2/d\\sigma\_\{k\}^\{2\}=\\norm\{\\mathbf\{z\}\_k\}^\{2\}/d, the mean\-square entry of the hidden state at sitekk\. In a pre\-normalization transformerσk\\sigma\_\{k\}rises with depth, because every site adds new content to the hidden state\([Sun et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib47)\)\. The hidden state therefore gains energy at a definite rate, and Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1)allows a deviation to gain energy at most at that same rate\.

###### Assumption 1\(bounded reinforcement\)\.

There areεk≥0\\varepsilon\_\{k\}\\geq 0with∑kεk≤Λ\\sum\_\{k\}\\varepsilon\_\{k\}\\leq\\Lambda, for aΛ\\Lambdaindependent ofNN, such that for every sitekk, every earlier sitess, and every isotropic injection𝛅s\\bm\{\\delta\}\_\{s\}from the noise model, the deviation𝐯=𝚽k,s\+1​𝛅s\\mathbf\{v\}=\\bm\{\\Phi\}\_\{k,s\+1\}\\bm\{\\delta\}\_\{s\}it produces at sitekksatisfies

𝔼​‖\(𝐈\+𝐉k\)​𝐯‖2≤\(1\+εk\)​σk\+12σk2​𝔼​‖𝐯‖2\.\\mathbb\{E\}\\,\\bigl\\\|\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)\\mathbf\{v\}\\bigr\\\|^\{2\}\\;\\leq\\;\(1\+\\varepsilon\_\{k\}\)\\,\\frac\{\\sigma\_\{k\+1\}^\{2\}\}\{\\sigma\_\{k\}^\{2\}\}\\;\\mathbb\{E\}\\norm\{\\mathbf\{v\}\}^\{2\}\.Appendix[A](https://arxiv.org/html/2610.14277#A1)completes the family of deviations𝐯\\mathbf\{v\}used in the proofs\.

The assumption does not restrict what a block does to any single vector\. A block may rotate an arriving deviation, spread it across coordinates, or overwrite it\. Theεk\\varepsilon\_\{k\}give further room\. Single sites may amplify beyond the ratioσk\+12/σk2\\sigma\_\{k\+1\}^\{2\}/\\sigma\_\{k\}^\{2\}, and only their total over all sites must stay bounded independently of depth\. The assumption excludes one behavior, a network that keeps amplifying the arriving errors from site to site\. Trained transformers show the opposite tendency\. Consecutive blocks tend to cancel each other’s contributions\([Patrawala et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib35)\), and small perturbations grow with depth at the rate of the hidden state, not faster\([Budzinskiy et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib4)\)\. Such an exclusion is also necessary\. The amplifying network of Appendix[A](https://arxiv.org/html/2610.14277#A1)satisfies the noise model, so no depth\-independent bound holds without an assumption of this kind\.

Writeq¯=q/\(d​σ1\)\\bar\{q\}=q/\(\\sqrt\{d\}\\,\\sigma\_\{1\}\), the per\-entry size of one injected error relative to the scaleσ1\\sigma\_\{1\}of the hidden state after the first site\.

###### Theorem 2\(propagation\)\.

Under Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1), the noise model, and the regularity conditions of Appendix[A](https://arxiv.org/html/2610.14277#A1), the following hold\.

- \(i\)In mean square, 𝔼​‖𝐄Nlin‖2d​σN2≤2​\[τ2​σ12σN2\+\(1\+μw\)2​C0​eΛ\]​q¯2,\\frac\{\\mathbb\{E\}\\,\\\|\\mathbf\{E\}\_\{N\}^\{\\mathrm\{lin\}\}\\\|^\{2\}\}\{d\\sigma\_\{N\}^\{2\}\}\\;\\leq\\;2\\Big\[\\tau\_\{2\}\\frac\{\\sigma\_\{1\}^\{2\}\}\{\\sigma\_\{N\}^\{2\}\}\+\(1\+\\mu\_\{w\}\)^\{2\}C\_\{0\}e^\{\\Lambda\}\\Big\]\\bar\{q\}^\{2\},withτ2=∑mTm2\\tau\_\{2\}=\\sum\_\{m\}T\_\{m\}^\{2\}a constant of the kernel andC0C\_\{0\}a constant of the checkpoint defined in Appendix[A](https://arxiv.org/html/2610.14277#A1)\.
- \(ii\)For everyp∈\(0,1\)p\\in\(0,1\), with probability at least1−p1\-p, ‖𝐄Nlin‖2d​σN2≤C​q¯2​log⁡2p\+C′​q¯2​log2​2p,\\frac\{\\\|\\mathbf\{E\}\_\{N\}^\{\\mathrm\{lin\}\}\\\|^\{2\}\}\{d\\sigma\_\{N\}^\{2\}\}\\;\\leq\\;C\\,\\bar\{q\}^\{2\}\\log\\frac\{2\}\{p\}\+C^\{\\prime\}\\bar\{q\}^\{2\}\\log^\{2\}\\frac\{2\}\{p\},withCCandC′C^\{\\prime\}the depth\-independent constants of Appendix[A](https://arxiv.org/html/2610.14277#A1)\.

The termτ2​σ12/σN2\\tau\_\{2\}\\,\\sigma\_\{1\}^\{2\}/\\sigma\_\{N\}^\{2\}in \(i\) decays with depth\. No other quantity in \(i\) and \(ii\) grows withNN\.

The quantity bounded is the energy of the deviation measured against the energy of the hidden state\. The second\-order remainders are at mostC2​q¯​NC\_\{2\}\\bar\{q\}\\sqrt\{N\}times the scaleq¯​d​σN\\bar\{q\}\\sqrt\{d\}\\,\\sigma\_\{N\}of the linear bound, for as long as the deviation stays within twice its linear bound \(Proposition[1](https://arxiv.org/html/2610.14277#Thmproposition1), sharp by Proposition[2](https://arxiv.org/html/2610.14277#Thmproposition2)\(b\), see also Corollary[2](https://arxiv.org/html/2610.14277#Thmcorollary2)\)\. HereC2C\_\{2\}depends on the curvature and the energy growth of the network and grows at most aslog2⁡\(N/p\)\\log^\{2\}\(N/p\)\. Up to the logarithm inC2C\_\{2\}, the factorC2​q¯​NC\_\{2\}\\bar\{q\}\\sqrt\{N\}is small whenever the noise injected over a pass is a small fraction of the hidden state\.

In the same mean\-square terms the advantage over the uncompensated primitive is direct\. With compensation the variance of the accumulated source stays belowτ2​q2\\tau\_\{2\}q^\{2\}at every site, about1\.3​q21\.3\\,q^\{2\}for the decay rates and blend used in the experiments\. Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)turns this bounded source into a first\-order output deviation at the level of a single rounding error, independent of depth\. Without compensation each site keeps its full contribution and the same bound grows aslog⁡N\\log N\(Remark[2](https://arxiv.org/html/2610.14277#Thmremark2)\)\. Compensation multiplies the term of each site in the bound by at most2​\(1\+μw\)2​C0​Δk/σk22\(1\+\\mu\_\{w\}\)^\{2\}C\_\{0\}\\Delta\_\{k\}/\\sigma\_\{k\}^\{2\}, whereΔk=σk\+12−σk2\\Delta\_\{k\}=\\sigma\_\{k\+1\}^\{2\}\-\\sigma\_\{k\}^\{2\}is the energy that blockkkadds, so the sum stays bounded\.

The source analysis extends to the backward pass\. Its injected errors enter through the Jacobian of the normalization \(Sec\.[3\.2](https://arxiv.org/html/2610.14277#S3.SS2)\), and Appendix[A](https://arxiv.org/html/2610.14277#A1)proves the corresponding depth\-independent source bound, in mean square and under a smoothness condition on the normalization Jacobians\. Section[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)describes the backward implementation\.

### 6\.3Choice of the compensator weights

Theorems[1](https://arxiv.org/html/2610.14277#Thmtheorem1)and[2](https://arxiv.org/html/2610.14277#Thmtheorem2)fix the gain atη=1\\eta=1and take the decay ratesβ1,β2\\beta\_\{1\},\\beta\_\{2\}of Sec\.[5](https://arxiv.org/html/2610.14277#S5)as given\. Both choices affect the guarantee\.

The gain interpolates between no compensation and full compensation\. At gainη\\etathe coefficient of an injected error in the partial sum is\(1−η\)\+η​Tt−s\(1\-\\eta\)\+\\eta T\_\{t\-s\}, and the source bound follows the same split,

maxt⁡‖𝐒t‖≤\(1−η\)​N​q\+η⁡\(1\+μw\)​q\.\\max\_\{t\}\\norm\{\\mathbf\{S\}\_t\}\\;\\leq\\;\(1\-\\eta\)\\,Nq\+\\eta\\,\(1\+\\mu\_\{w\}\)\\,q\.\(21\)Theorem[1](https://arxiv.org/html/2610.14277#Thmtheorem1)is the caseη=1\\eta=1, the only gain at which the growing term vanishes\. Atη<1\\eta<1, the fraction1−η1\-\\etaof every error is never corrected and accumulates as it would without compensation\. The compensator state is reset at the start of each pass, so this growth does not carry over between passes\. In mean square this uncorrected share is small\. The net error is linear in the gain, so the first\-order output deviation is1−η1\-\\etatimes the uncompensated deviation plusη\\etatimes the deviation of Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)\(Corollary[1](https://arxiv.org/html/2610.14277#Thmcorollary1)\)\. At the gainη=0\.75\\eta=0\.75used in the experiments, the part that grows with depth is the uncompensated bound scaled by2​\(1−η\)2=1/82\(1\-\\eta\)^\{2\}=1/8, which grows only aslog⁡N\\log N\. AtN=56N=56the variance of the source is about5​q25q^\{2\}, against56​q256q^\{2\}without compensation\.

The decay rates enter the source bound only throughμw\\mu\_\{w\}, so every choice ofβ1,β2\\beta\_\{1\},\\beta\_\{2\}keeps the source below a constant independent ofNN\. The constant grows with the memory of the averages, and is about2​q2qat the values used in the experiments\. The smallest constant belongs to the limitβ1,β2→0\\beta\_\{1\},\\beta\_\{2\}\\to 0, where the compensator keeps only the error of the preceding site and reduces to single\-step error feedback\. This optimum is a worst\-case one, attained on adversarial sequences, and practical error sequences are not adversarial\. Away from that extreme, correcting with a short\-range and a long\-range average together does better than correcting with the last error alone, and Sec\.[7](https://arxiv.org/html/2610.14277#S7)reports the ablation overβ1\\beta\_\{1\},β2\\beta\_\{2\}, andη\\etaaround the values used in the experiments\.

## 7Experiments

The experiments cover four parts\. We first measure speed, including the latency of the quantized all\-reduce, the overhead of the compensator before and after fusion, and end\-to\-end speed in inference and training \(Sec\.[7\.1](https://arxiv.org/html/2610.14277#S7.SS1)\)\. We then evaluate inference accuracy under 4\-bit communication \(Sec\.[7\.2](https://arxiv.org/html/2610.14277#S7.SS2)\), compare training loss curves against the 16\-bit tensor\-parallel baseline \(Sec\.[7\.3](https://arxiv.org/html/2610.14277#S7.SS3)\), and report ablations, including the sweep overβ1\\beta\_\{1\},β2\\beta\_\{2\}, andη\\eta\(Sec\.[7\.4](https://arxiv.org/html/2610.14277#S7.SS4)\)\.

Setup\.All experiments run on a single server with eight A100 GPUs connected by NVLink\. The inference experiments serve Llama\-3\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib16)\)and Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib61)\)atP=4P=4and evaluate language modeling perplexity together with standard downstream accuracy benchmarks\. The training experiments pretrain Qwen3\-0\.6B from scratch atP=4P=4with BF16 mixed precision and sequence length 2,048\. Data and schedules are given in Sec\.[7\.3](https://arxiv.org/html/2610.14277#S7.SS3)\. End\-to\-end speed is measured on Llama\-3\-8B and Qwen3\-8B atP=8P=8\(Sec\.[7\.1](https://arxiv.org/html/2610.14277#S7.SS1)\)\. Unless stated otherwise, communication uses the 4\-bit quantized all\-reduce of Sec\.[4](https://arxiv.org/html/2610.14277#S4)with group sizeg=128g=128and the zero\-aware asymmetric quantizer, and the compensator usesη=0\.75\\eta=0\.75,a=0\.35a=0\.35,β1=0\.7\\beta\_\{1\}=0\.7, andβ2=0\.3\\beta\_\{2\}=0\.3\(Sec\.[7\.4](https://arxiv.org/html/2610.14277#S7.SS4)\)\.

### 7\.1Latency and end\-to\-end speed

Table[3](https://arxiv.org/html/2610.14277#S7.T3)reports the median latency of one all\-reduce atP=4P=4for 16\-bit messages of 64 MB to 1 GB per GPU\. Each value is the median of 80 iterations after 30 warmup iterations, and all rows come from the same run\. The fused compensator updates its state while the next matrix multiplication runs, so for the fused row we time the matrix multiplications and the all\-reduce together and subtract the time of the matrix multiplications alone\. The rows are the 16\-bit NCCL baseline, a two\-collective quantized all\-reduce assembled from DeepSpeed’s quantized all\-to\-all and quantized all\-gather primitives in the style of ZeRO\+\+\([Wang et al\., 2024a](https://arxiv.org/html/2610.14277#bib.bib50)\)at 8 and 4 bits, our all\-reduce at 8 and 4 bits without compensation, and the 4\-bit all\-reduce with the compensator attached in two forms, as separate PyTorch operations and fused as described in Sec\.[5\.3](https://arxiv.org/html/2610.14277#S5.SS3)\.

Table 2:Median all\-reduce latency\(ms\) atP=4P=4over message sizes, measured in one run\. Lower is better\. Bold marks the minimum in each column\. The last two rows attach the compensator to our 4\-bit all\-reduce, unfused and fused\. The fused row updates its state beside the next matrix multiplication and reports the time the call adds to the critical path, measured as the time of the matrix multiplications and the all\-reduce together minus that of the matrix multiplications alone\.
Table 3:End\-to\-end speedup\(×\\times\) of our fused, compensated 4\-bit all\-reduce over the 16\-bit NCCL all\-reduce on eight A100 GPUs \(P=8P=8,g=128g=128\)\. Speedup isTNCCL/ToursT\_\{\\mathrm\{NCCL\}\}/T\_\{\\mathrm\{ours\}\}\. Inference reports the time to first token of Llama\-3\-8B and Qwen3\-8B, training the mean step time\. Load is the duty cycle of background peer\-to\-peer copies\. AR slowdown isTARNCCL​\(ℓ\)/TARNCCL​\(0\)T\_\{\\mathrm\{AR\}\}^\{\\mathrm\{NCCL\}\}\(\\ell\)/T\_\{\\mathrm\{AR\}\}^\{\\mathrm\{NCCL\}\}\(0\)for a fixed\-size 1 GB 16\-bit NCCL all\-reduce, the per\-call message of the batch\-64 runs, at loadℓ\\ellagainst no load\.

On the 1 GB message, our uncompensated 8\-bit all\-reduce is 1\.63×\\timesfaster than NCCL, and our uncompensated 4\-bit all\-reduce is 2\.65×\\timesfaster and takes 37\.8% of the NCCL time\. At equal bit width, our all\-reduce is 1\.43×\\timesfaster than the two\-collective baseline at 8 bits and 1\.69×\\timesfaster at 4 bits\. The gap comes from the pipelined design of Sec\.[4](https://arxiv.org/html/2610.14277#S4), which overlaps quantization with communication instead of running two dependent collectives back to back\.

The measured 4\-bit speedup is below the4×4\\timesbyte ratio of 4\-bit against 16\-bit transport, for two reasons\. First, the payload is not the only traffic\. With group sizeg=128g=128, every 128 quantized values share one pair of half\-precision quantization parameters, so a group costs 68 bytes instead of 64, which alone lowers the ideal ratio from4×4\\timesto about3\.8×3\.8\\times\. Second, quantization adds computation on the datapath\. Each message is quantized before it is sent and dequantized after it is received, and the owner performs the sum in high precision between a dequantize and a requantize, none of which exists in the 16\-bit all\-reduce\. The pipeline hides most of this computation behind the transfers, and the remaining gap to3\.8×3\.8\\timesis the part it cannot hide\.

The two compensator rows measure the cost of compensation\. Built from separate PyTorch operations, the compensator adds two full passes over the tensor and an intermediate buffer, and on the 1 GB message it adds 14\.09 ms to the all\-reduce\. With the apply step fused into the entry pass and the update running beside the next matrix multiplication, it adds 0\.90 ms, 10% of the NCCL time\. The fused path matches a PyTorch reference implementation of the compensator exactly, with a maximum absolute difference of zero at every size\. With fused compensation, the 4\-bit all\-reduce adds 4\.30 ms to the critical path for the 1 GB message, against 9\.00 ms for NCCL, a 2\.09×\\timesreduction\.

Table[3](https://arxiv.org/html/2610.14277#S7.T3)reports end\-to\-end speed on the same server atP=8P=8with group sizeg=128g=128\. We compare the 16\-bit NCCL all\-reduce with our fused, compensated 4\-bit all\-reduce without background load and with background peer\-to\-peer copies running between the GPUs for 50%, 75%, and 100% of the time, which emulates an interconnect shared with other traffic\. In each setting both collectives run under the same load, and we report the speedupTNCCL/ToursT\_\{\\mathrm\{NCCL\}\}/T\_\{\\mathrm\{ours\}\}\.

For inference, we measure the time to first token of Llama\-3\-8B and Qwen3\-8B with batch size 64 and a 2,048\-token prefill\. For training, we measure the mean step time of Llama\-3\-8B with the same batch size and sequence length and of Qwen3\-8B with batch size 32 and sequence length 2,048\. The training measurement covers the all\-reduce calls of both the forward and the backward pass, including the gradient rescaling of the 4\-bit path\. Without background load, the speedup is 1\.16×\\timeson Llama\-3\-8B and 1\.09×\\timeson Qwen3\-8B for inference, and 1\.09×\\timesand 1\.08×\\timesfor training\. It grows with the load and reaches 1\.44×\\timesand 1\.31×\\timesfor inference and 1\.38×\\timesand 1\.34×\\timesfor training at 100% load\.

These speedups are bounded by the share of time the all\-reduce blocks the 16\-bit baseline\. On Llama\-3\-8B without background load, this share is 20\.3% of the time to first token and 18\.8% of the training step, which bounds the speedups at 1\.25×\\timesand 1\.23×\\times\. At 100% load, the shares rise to 43\.0% and 36\.2%, and the bounds to 1\.75×\\timesand 1\.57×\\times\. A prefill waits on every all\-reduce, and our all\-reduce synchronizes through flags that the GPUs set and poll themselves, tile by tile \(Sec\.[4\.2](https://arxiv.org/html/2610.14277#S4.SS2)\), so it starts on each tile as soon as its data arrives and loses little time when links are congested\. At 100% load, the fixed\-size 16\-bit NCCL all\-reduce takes 2\.73×\\timesas long as it does without background load\. The speedup thus grows with the share of the step that the all\-reduce takes\.

### 7\.2Inference accuracy

The speedups reported in Sec\.[7\.1](https://arxiv.org/html/2610.14277#S7.SS1)matter only if accuracy is preserved\. We evaluate Llama\-3\-8B and Qwen3\-8B with the 4\-bit all\-reduce at every forward site, on three groups of tasks: language modeling perplexity on WikiText\([Merity et al\., 2017](https://arxiv.org/html/2610.14277#bib.bib29)\), accuracy on seven standard benchmarks \(PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.14277#bib.bib3)\), ARC\-easy, ARC\-challenge\([Clark et al\., 2018](https://arxiv.org/html/2610.14277#bib.bib6)\), HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.14277#bib.bib62)\), WinoGrande\([Sakaguchi et al\., 2020](https://arxiv.org/html/2610.14277#bib.bib41)\), MMLU\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.14277#bib.bib18)\), MMLU\-Pro\([Wang et al\., 2024c](https://arxiv.org/html/2610.14277#bib.bib54)\)\), and reasoning on GPQA\([Rein et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib38)\)and GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.14277#bib.bib7)\)\. All measured rows, including the 16\-bit baselines, are obtained by us in a single evaluation harness\([Gao et al\., 2023](https://arxiv.org/html/2610.14277#bib.bib12)\), and each task uses the standard shot count of the harness\. We compare two rows on a task with a two\-sided test at the 5% level, using the binomial standard error of each accuracy\. The FlashComm rows are quoted from the original papers\([Li et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib24);[Li et al\., 2025b](https://arxiv.org/html/2610.14277#bib.bib25)\)and are set apart in Tables[5](https://arxiv.org/html/2610.14277#S7.T5)and[6](https://arxiv.org/html/2610.14277#S7.T6)\.

Table 4:WikiText perplexityfor Llama\-3\-8B and Qwen3\-8B\. Lower is better\. The measured rows useP=4P=4\. The FlashComm rows are quoted from\([Li et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib24)\)and set in gray\. That paper reports no 16\-bit perplexity, so these rows are compared through the change from 8 to 4 bits\. Dashes mark results not reported in the cited work\.
Table 5:Math reasoning accuracy\(%\) for Llama\-3\-8B and Qwen3\-8B atP=4P=4\. GSM8K is scored by strict\-match and flexible\-extract\. Early\-site 8\-bit promotes the all\-reduce calls of the first half of the layers to 8 bits, 32 calls on Llama\-3\-8B and 36 on Qwen3\-8B, and keeps the rest at 4 bits\. The cited work reports neither benchmark\.

Perplexity\.Table[5](https://arxiv.org/html/2610.14277#S7.T5)reports WikiText perplexity, where lower is better\. The 4\-bit path raises perplexity by 0\.29 on Llama\-3\-8B \(6\.07 to 6\.36\) and by 0\.10 on Qwen3\-8B \(9\.70 to 9\.80\), and the 8\-bit path leaves it unchanged\. An increase of this size is small at these perplexity levels, and it is much smaller than the increase that the raw quantizer error causes\. Without compensation the 4\-bit perplexity is 6\.73 and 10\.45, and the compensator closes most of the distance between those values and the 16\-bit model\. Moving from 8 to 4 bits raises the perplexity of Llama\-3\-8B by 0\.29 with our all\-reduce and by 0\.56 with FlashComm\([Li et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib24)\)\. Without compensation our 4\-bit path rises by 0\.66, close to the 0\.56 of FlashComm, so the difference in perplexity comes from the compensator\. The task benchmarks below test whether the remaining difference affects answers\.

Table 6:Downstream accuracy\(%\)\. The FlashComm rows are quoted from\([Li et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib24);[Li et al\., 2025b](https://arxiv.org/html/2610.14277#bib.bib25)\), set in gray, and are read against the 16\-bit references of those papers\. All other rows are measured in our harness atP=4P=4and are read against the 16\-bit row of the same model\. Dashes mark results not reported in the cited work\.Downstream accuracy\.Table[6](https://arxiv.org/html/2610.14277#S7.T6)reports seven standard benchmarks\. At 8 bits, our all\-reduce matches 16\-bit inference, with average scores within 0\.12 points on both models\. At 4 bits, Qwen3\-8B loses only 0\.39 points on average and scores higher than 16\-bit on PIQA and ARC\-easy, and Llama\-3\-8B loses 1\.66 points on average\. The compensator keeps these losses small\. Without it, the average drop is 3\.46 points on Llama\-3\-8B and 2\.33 on Qwen3\-8B, so compensation removes about half of the loss on Llama\-3\-8B and 83% of it on Qwen3\-8B\. Against the 16\-bit row, the uncompensated 4\-bit path loses significant accuracy on 10 of the 14 model and task pairs\. Compensation raises accuracy on all 14 pairs, and significantly on 6 of them\. On the five tasks that FlashComm also reports, moving from 8 to 4 bits costs our all\-reduce 1\.43 points on Llama\-3\-8B and 0\.22 on Qwen3\-8B on average, against 1\.42 and 0\.87 for FlashComm\.

Math reasoning\.Table[5](https://arxiv.org/html/2610.14277#S7.T5)reports GPQA and GSM8K\. For GSM8K we report both scoring modes of the harness: strict\-match, which accepts the answer only in the exact expected format, and flexible\-extract, which accepts any correctly extracted number\. On GPQA the two paths differ by less than a point on both models and in either direction, 33\.03 against 33\.71 on Llama\-3\-8B and 38\.84 against 38\.51 on Qwen3\-8B\. On GSM8K the 4\-bit path scores 46\.17 strict\-match against 49\.81 for 16\-bit, a loss of 48 of 1,319 problems\. The two tasks differ in what they ask of the model\. GPQA is answered with a single choice, while a GSM8K solution is a generated chain of several hundred tokens in which each token conditions on the ones before it, so one flipped intermediate token can make the rest of the derivation wrong\. The compensator keeps the accumulated deviation small, but at 4 bits even a single rounding error occasionally flips a token, and a long chain offers many chances\. We therefore spend more bits on a small part of the traffic\. An error injected at an early call enters the residual stream and reaches every later layer, while an error injected near the output reaches only a few\. In a long generated chain, early errors therefore have more chances to change a token\. We promote the all\-reduce calls of the first half of the layers, on each prefill and each decode forward, to 8 bits, and call this policy early\-site 8\-bit\. This recovers most of the loss\. It moves about 1\.5 times the bytes of uniform 4\-bit communication\. On Llama\-3\-8B it recovers strict\-match to 49\.81, the 16\-bit score, and reaches 50\.11 on flexible\-extract against 49\.96 for 16\-bit\. On Qwen3\-8B it reaches 81\.73 and 82\.18 against 82\.49 and 82\.56 for 16\-bit\. Early\-site 8\-bit is used only in the serving experiments of this section\. The training experiments of Sec\.[7\.3](https://arxiv.org/html/2610.14277#S7.SS3)use uniform 4\-bit communication throughout\.

### 7\.3Training

Figure 3:Training lossof Qwen3\-0\.6B atP=4P=4on 10B tokens of Dolma C4, 16\-bit at peak LR3\.0×10−43\.0\\times 10^\{\-4\}against our 4\-bit all\-reduce at peak LR3\.5×10−43\.5\\times 10^\{\-4\}\. Left: full run \(50\-step moving average, y axis clipped, gray band marks the steps on the right\)\. Right: last 1,500 steps \(100\-step average,±\\pm1 s\.d\. band\)\.Table 7:Training lossof Qwen3\-0\.6B atP=4P=4on Dolma C4\. \(a\) Full 10B\-token training runs\. \(b\) A separate 4\-bit learning\-rate ablation with batch size 32 and 4,096 steps\. Tail\-100 is the mean loss over the last 100 steps\. In \(a\),Δ\\Deltais relative to the 16\-bit run, and in \(b\),Δ3\.5\\Delta\_\{3\.5\}is relative to the run at LR3\.5×10−43\.5\\times 10^\{\-4\}\. All differences use unrounded tail\-100 losses\.*\(a\) Full training: 10B tokens, batch size 384*

*\(b\) LR ablation: batch size 32, 4,096 steps*

We pretrain Qwen3\-0\.6B\([Yang et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib61)\)from scratch atP=4P=4with the 4\-bit all\-reduce at every forward and backward site\. The main experiment trains on 10B tokens of Dolma C4\([Soldaini et al\., 2024](https://arxiv.org/html/2610.14277#bib.bib46);[Raffel et al\., 2020](https://arxiv.org/html/2610.14277#bib.bib37)\)with a global batch of 384 sequences for 12,716 steps, using AdamW, 100 warmup steps, and cosine decay \(Appendix[B](https://arxiv.org/html/2610.14277#A2)\)\. The 16\-bit and 4\-bit runs share the data, batch size, optimizer, and schedule\. Low\-bit training has been observed to benefit from a larger learning rate than its full\-precision counterpart\([Wang et al\., 2023](https://arxiv.org/html/2610.14277#bib.bib52);[Xie et al\., 2025](https://arxiv.org/html/2610.14277#bib.bib58)\), so we give the 4\-bit run a slightly larger peak learning rate,3\.5×10−43\.5\\times 10^\{\-4\}against3\.0×10−43\.0\\times 10^\{\-4\}for 16\-bit\.

Figure[3](https://arxiv.org/html/2610.14277#S7.F3)compares the two runs\. Their loss curves track each other closely over all 12,716 steps, and the right panel magnifies the last 1,500 steps\. Over the last 100 steps \(tail\-100\), the 4\-bit loss averages 3\.225 and the 16\-bit loss 3\.197, a gap of 0\.027 computed before rounding, less than 0\.9% of the 16\-bit loss\. Early in training the 4\-bit run, with its larger learning rate, has the lower loss, 4\.042 against 4\.095 at step 1,000 \(Table[7](https://arxiv.org/html/2610.14277#S7.T7)a\)\.

Table[7](https://arxiv.org/html/2610.14277#S7.T7)b supports this choice of learning rate\. In separate, shorter 4\-bit runs on Dolma C4 with a global batch of 32 sequences for 4,096 steps, the tail\-100 loss falls steadily as the peak learning rate rises, from 4\.135 at2\.0×10−42\.0\\times 10^\{\-4\}to 4\.078, 4\.016, and 3\.952 at2\.52\.5,3\.03\.0, and3\.5×10−43\.5\\times 10^\{\-4\}\. The final loss is sensitive to the learning rate\([Xie et al\., 2024a](https://arxiv.org/html/2610.14277#bib.bib56)\), and the 4\-bit run gains from the larger value\.

### 7\.4Ablations

Table 8:Component ablations, Llama\-3\-8B atP=4P=4, WikiText and C4 perplexity \(WT = WikiText\), lower is better\. Each row changes one component of the method, and theg=256g=256rows also change the group size\. Bold marks the minimum within each bit\-width column, excluding the 16\-bit reference\.
We now remove or replace one component of the method at a time and measure the effect on perplexity, where lower is better\. All runs serve Llama\-3\-8B atP=4P=4with the same evaluation as Table[5](https://arxiv.org/html/2610.14277#S7.T5), on WikiText and C4\.

Table[8](https://arxiv.org/html/2610.14277#S7.T8)evaluates every variant at 4 and at 8 bits\. At 8 bits the variants span only 0\.011 perplexity on WikiText and 0\.020 on C4\. At 4 bits they span 2\.842 and 5\.115\. At 4 bits, the full design has much lower perplexity than the naive zero\-point and no\-compensation variants\. Changing any single coefficient moves the 4\-bit perplexity by at most 0\.022 on WikiText and 0\.041 on C4, so the method is not sensitive to these coefficients\. We examine the 4\-bit columns below\.

The first factor is the group size\. Coarsening the groups from 128 to 256 raises perplexity by 0\.35 on WikiText and 0\.79 on C4 with the symmetric quantizer, and by 1\.9 and 3\.3 with the naive zero\-point\. A group has sixteen levels and one scale, so groups must stay small for the scale to resolve the values it covers\. Since each group also carries its own scale and zero\-point, finer groups add bytes to the message \(Sec\.[7\.1](https://arxiv.org/html/2610.14277#S7.SS1)\), and we useg=128g=128\.

The second factor is the zero\-point\. With a naive zero\-point the asymmetric quantizer loses to the symmetric grid by 0\.73 on WikiText and 1\.43 on C4\. As shown in Sec\.[4\.3](https://arxiv.org/html/2610.14277#S4.SS3), the naive map represents zero inexactly, and the error this injects into the densest part of the activation distribution outweighs the gain from the finer resolution of the fitted range\. The zero\-aware assignment removes this error, and the asymmetric quantizer reaches 6\.362 on WikiText and 9\.484 on C4, lower than both alternatives\.

The third factor is the compensator\. Removing it makes each call quantize independently, so the errors accumulate across sites, which is the setting of one\-shot compression in prior work\. Perplexity rises to 6\.730 on WikiText and 10\.130 on C4\. The accuracy counterpart of this row appears in Table[6](https://arxiv.org/html/2610.14277#S7.T6)\. Keeping the compensator but collapsing its two averages into one loses 0\.14 on WikiText and 0\.31 on C4, the contribution of the second time scale\. At 8 bits the same substitution moves C4 perplexity only from 8\.970 to 8\.972\.

The last rows vary the coefficients\. The decay rates and the blend enter the guarantee only through a constant \(Sec\.[6\.3](https://arxiv.org/html/2610.14277#S6.SS3)\)\. Consistent with this, everyaaand\(β1,β2\)\(\\beta\_\{1\},\\beta\_\{2\}\)row stays within 0\.022 of the full method\. The gain trades two errors against each other\. Atη=1\\eta=1nothing is left uncorrected \(Eqn\. \([21](https://arxiv.org/html/2610.14277#S6.E21)\)\), but the correction is an estimate of an error the compensator has not yet seen, and at full strength an imperfect estimate overcorrects whenever the error changes between sites\. A gain below one leaves the fraction1−η1\-\\etaof each error uncorrected in exchange for less overcorrection\. The theory is stated atη=1\\eta=1, and the sweep favors a smaller gain\. The gainη=0\.9\\eta=0\.9is worse than both 0\.7 and 0\.75 on WikiText and C4\. We useη=0\.75\\eta=0\.75,a=0\.35a=0\.35,β1=0\.7\\beta\_\{1\}=0\.7, andβ2=0\.3\\beta\_\{2\}=0\.3throughout, and nothing is tuned per model\.

The 4\-bit accuracy of Sec\.[7\.2](https://arxiv.org/html/2610.14277#S7.SS2)thus depends on both components\. The quantizer bounds the error each site injects, and the compensator keeps the remaining error from accumulating\.

## 8Conclusion

Echostarts from the arithmetic constraint of a low\-bit collective\. The all\-reduce must add the values it moves, and the sum cannot be formed in the transport format\. The collective is therefore built as low\-bit transport around a high\-precision reduction, which quantizes each element at most twice per call for any tensor\-parallel degree\. The reduction also reveals each injected error at no extra communication, and the fused compensator feeds it back into later calls, so the accumulated error stays bounded with guarantees independent of network depth\. At 8 bits, inference matches 16\-bit accuracy, and 4\-bit inference stays within 0\.4 and 1\.7 points of it on average over seven tasks for Qwen3\-8B and Llama\-3\-8B\. With a slightly larger learning rate, 4\-bit pretraining over 10B tokens ends less than 0\.9% above the 16\-bit loss\. For 1 GB messages the compensated all\-reduce is 2\.09×\\timesfaster than the 16\-bit NCCL all\-reduce\.

The end\-to\-end gains grow with the traffic on the interconnect\. The time to first token of Llama\-3\-8B improves by 1\.16×\\timeswithout background load and by 1\.44×\\timesat 100% load, because communication then takes a larger share of each step\. The compensation itself relies on two structural properties\. The collective reveals the error it injects, and its outputs join the same residual stream call after call, so a later correction can cancel an earlier error\. The reduce\-scatter of sequence parallelism and the combine step of expert parallelism share both, and are natural next targets for the compensated collective\.

## References

- Ahmed et al\. \(2025\)I\. Ahmed, C\. Schaefer, G\. Tabak, D\. Vnukov, Z\. Zhang, F\. Chern, et al\.EQuARX: Efficient quantized AllReduce in XLA for distributed machine learning acceleration\.*arXiv preprint arXiv:2506\.17615*, 2025\.
- Bian et al\. \(2023\)S\. Bian, D\. Li, H\. Wang, E\. P\. Xing, and S\. Venkataraman\.Does compressing activations help model parallel training?*arXiv preprint arXiv:2301\.02654*, 2023\.
- Bisk et al\. \(2020\)Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. Choi\.PIQA: Reasoning about physical commonsense in natural language\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 34, pages 7432–7439, 2020\.doi:10\.1609/aaai\.v34i05\.6239\.
- Budzinskiy et al\. \(2025\)S\. Budzinskiy, W\. Fang, L\. Zeng, and P\. Petersen\.Numerical stability analysis of large language models\.*arXiv preprint arXiv:2503\.10251*, 2025\.
- Chang et al\. \(2024\)L\.\-W\. Chang, W\. Bao, Q\. Hou, C\. Jiang, N\. Zheng, Y\. Zhong, et al\.FLUX: Fast software\-based communication overlap on GPUs through kernel fusion\.*arXiv preprint arXiv:2406\.06858*, 2024\.
- Clark et al\. \(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, et al\.Think you have solved question answering? Try ARC, the AI2 reasoning challenge\.*arXiv preprint arXiv:1803\.05457*, 2018\.
- Cobbe et al\. \(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, et al\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Dao et al\. \(2022\)T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré\.FlashAttention: Fast and memory\-efficient exact attention with IO\-awareness\.In*Advances in Neural Information Processing Systems*, volume 35, pages 16344–16359, 2022\.
- DeepSeek\-AI \(2024\)DeepSeek\-AI\.DeepSeek\-V3 technical report\.*arXiv preprint arXiv:2412\.19437*, 2024\.
- Dong et al\. \(2024\)H\. Dong, T\. Johnson, M\. Cho, and E\. Soroush\.Towards low\-bit communication for tensor parallel LLM inference\.In*Advances in Neural Information Processing Systems, Workshop on Efficient Natural Language and Speech Processing*, 2024\.arXiv:2411\.07942\.
- Frantar et al\. \(2023\)E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\.OPTQ: Accurate quantization for generative pre\-trained transformers\.In*International Conference on Learning Representations*, 2023\.
- Gao et al\. \(2023\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, et al\.A framework for few\-shot language model evaluation\.Zenodo, Dec\. 2023\.Version v0\.4\.0\.
- Gholami et al\. \(2024\)A\. Gholami, Z\. Yao, S\. Kim, C\. Hooper, M\. W\. Mahoney, and K\. Keutzer\.AI and memory wall\.*IEEE Micro*, 44\(3\):33–39, 2024\.
- Gond et al\. \(2026\)R\. Gond, N\. Kwatra, and R\. Ramjee\.TokenWeave: Efficient compute\-communication overlap for distributed LLM inference\.In*Proceedings of Machine Learning and Systems*, volume 8, pages 36–56, 2026\.
- Graham et al\. \(2016\)R\. L\. Graham, D\. Bureddy, P\. Lui, H\. Rosenstock, G\. Shainer, G\. Bloch, et al\.Scalable hierarchical aggregation protocol \(SHArP\): A hardware architecture for efficient data reduction\.In*Proceedings of the First International Workshop on Communication Optimizations in HPC*, pages 1–10\. IEEE, 2016\.
- Grattafiori et al\. \(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Hansen\-Palmus et al\. \(2024\)J\. Hansen\-Palmus, M\. T\. Le, O\. Hausdörfer, and A\. Verma\.Communication compression for tensor parallel LLM inference\.*arXiv preprint arXiv:2411\.09510*, 2024\.
- Hendrycks et al\. \(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, et al\.Measuring massive multitask language understanding\.In*International Conference on Learning Representations*, 2021\.
- Hooper et al\. \(2024\)C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, et al\.KVQuant: Towards 10 million context length LLM inference with KV cache quantization\.In*Advances in Neural Information Processing Systems*, volume 37, pages 1270–1303, 2024\.
- Jia et al\. \(2024\)J\. Jia, C\. Xie, H\. Lu, D\. Wang, H\. Feng, C\. Zhang, et al\.SDP4Bit: Toward 4\-bit communication quantization in sharded data parallelism for LLM training\.In*Advances in Neural Information Processing Systems*, volume 37, pages 8734–8759, 2024\.doi:10\.52202/079017\-0279\.
- Jin et al\. \(2026\)C\. Jin, Z\. Jiang, Z\. Bai, Z\. Zhong, J\. Liu, X\. Li, et al\.MegaScale\-MoE: Large\-scale communication\-efficient training of Mixture\-of\-Experts models in production\.In*Proceedings of the 21st European Conference on Computer Systems*, pages 366–382\. ACM, 2026\.doi:10\.1145/3767295\.3769325\.
- Karimireddy et al\. \(2019\)S\. P\. Karimireddy, Q\. Rebjock, S\. U\. Stich, and M\. Jaggi\.Error feedback fixes SignSGD and other gradient compression schemes\.In*International Conference on Machine Learning*, volume 97 of*Proceedings of Machine Learning Research*, pages 3252–3261\. PMLR, 2019\.
- Li et al\. \(2025a\)H\. Li, W\. Luo, X\. Niu, S\. Tiwari, K\. Wang, J\. Ren, A\. Sirasao, and D\. Lehr\.QuickReduce: Up to 3x faster all\-reduce for vLLM and SGLang\.AMD ROCm Blogs,[https://rocm\.blogs\.amd\.com/artificial\-intelligence/quick\-reduce/README\.html](https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce/README.html), 2025a\.
- Li et al\. \(2024\)Q\. Li, B\. Zhang, L\. Ye, Y\. Zhang, W\. Wu, Y\. Sun, et al\.Flash Communication: Reducing tensor parallelization bottleneck for fast large language model inference\.*arXiv preprint arXiv:2412\.04964*, 2024\.
- Li et al\. \(2025b\)Q\. Li, B\. Zhang, H\. Kang, T\. Xu, Y\. Qian, Y\. Xie, et al\.FlashCommunication V2: Bit splitting and spike reserving for any bit communication\.*arXiv preprint arXiv:2508\.03760*, 2025b\.
- Lin et al\. \(2024\)J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\.\-M\. Chen, W\.\-C\. Wang, et al\.AWQ: Activation\-aware weight quantization for on\-device LLM compression and acceleration\.In*Proceedings of Machine Learning and Systems*, volume 6, pages 87–100, 2024\.
- Liu et al\. \(2026\)M\. Liu, X\. Liu, X\. Tian, B\. Lu, S\. Lyu, S\. Yin, et al\.TACO: Efficient communication compression of intermediate tensors for scalable tensor\-parallel LLM training\.In*Proceedings of the 35th International Symposium on High\-Performance Parallel and Distributed Computing*, pages 235–248\. ACM, 2026\.doi:10\.1145/3806645\.3807584\.
- Liu et al\. \(2024\)Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, et al\.KIVI: A tuning\-free asymmetric 2bit quantization for KV cache\.In*International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 32332–32344\. PMLR, 2024\.
- Merity et al\. \(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\.Pointer sentinel mixture models\.In*International Conference on Learning Representations*, 2017\.
- Micikevicius et al\. \(2022\)P\. Micikevicius, D\. Stosic, N\. Burgess, M\. Cornea, P\. Dubey, R\. Grisenthwaite, et al\.FP8 formats for deep learning\.*arXiv preprint arXiv:2209\.05433*, 2022\.
- Nagel et al\. \(2019\)M\. Nagel, M\. van Baalen, T\. Blankevoort, and M\. Welling\.Data\-free quantization through weight equalization and bias correction\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 1325–1334, 2019\.
- NVIDIA \(2025\)NVIDIA\.Pretraining large language models with NVFP4\.*arXiv preprint arXiv:2509\.25149*, 2025\.
- Pagliardini et al\. \(2025\)M\. Pagliardini, P\. Ablin, and D\. Grangier\.The AdEMAMix optimizer: Better, faster, older\.In*International Conference on Learning Representations*, 2025\.
- Patarasuk and Yuan \(2009\)P\. Patarasuk and X\. Yuan\.Bandwidth optimal all\-reduce algorithms for clusters of workstations\.*Journal of Parallel and Distributed Computing*, 69\(2\):117–124, 2009\.
- Patrawala et al\. \(2025\)A\. Patrawala, J\. Feng, E\. Jones, and J\. Steinhardt\.LLM layers immediately correct each other\.In*Advances in Neural Information Processing Systems*, volume 38, pages 139071–139113, 2025\.
- Pinelis \(1994\)I\. Pinelis\.Optimum bounds for the distributions of martingales in Banach spaces\.*The Annals of Probability*, 22\(4\):1679–1706, 1994\.
- Raffel et al\. \(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, et al\.Exploring the limits of transfer learning with a unified text\-to\-text transformer\.*Journal of Machine Learning Research*, 21\(140\):1–67, 2020\.
- Rein et al\. \(2024\)D\. Rein, B\. L\. Hou, A\. Cooper Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, et al\.GPQA: A graduate\-level Google\-proof Q&A benchmark\.In*First Conference on Language Modeling*, 2024\.
- Richtárik et al\. \(2021\)P\. Richtárik, I\. Sokolov, and I\. Fatkhullin\.EF21: A new, simpler, theoretically better, and practically faster error feedback\.In*Advances in Neural Information Processing Systems*, volume 34, pages 4384–4396, 2021\.
- Rudakov et al\. \(2024\)M\. Rudakov, A\. Beznosikov, Y\. Kholodov, and A\. Gasnikov\.Activations and gradients compression formodel\-paralleltraining\.*arXiv preprint arXiv:2401\.07788*, 2024\.
- Sakaguchi et al\. \(2020\)K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi\.WinoGrande: An adversarial Winograd schema challenge at scale\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 34, pages 8732–8740, 2020\.doi:10\.1609/aaai\.v34i05\.6399\.
- Sapio et al\. \(2021\)A\. Sapio, M\. Canini, C\.\-Y\. Ho, J\. Nelson, P\. Kalnis, C\. Kim, et al\.Scaling distributed machine learning with in\-network aggregation\.In*USENIX Symposium on Networked Systems Design and Implementation*, pages 785–808\. USENIX Association, 2021\.
- Seide et al\. \(2014\)F\. Seide, H\. Fu, J\. Droppo, G\. Li, and D\. Yu\.1\-bit stochastic gradient descent and its application to data\-parallel distributed training of speech DNNs\.In*Interspeech*, pages 1058–1062, 2014\.
- Shen et al\. \(2025\)A\. Shen, R\. Zhang, and J\. Zhao\.FlexLink: Boosting your NVLink bandwidth by 27% without accuracy concern\.*arXiv preprint arXiv:2510\.15882*, 2025\.
- Shoeybi et al\. \(2019\)M\. Shoeybi, M\. Patwary, R\. Puri, P\. LeGresley, J\. Casper, and B\. Catanzaro\.Megatron\-LM: Training multi\-billion parameter language models using model parallelism\.*arXiv preprint arXiv:1909\.08053*, 2019\.
- Soldaini et al\. \(2024\)L\. Soldaini, R\. Kinney, A\. Bhagia, D\. Schwenk, D\. Atkinson, R\. Authur, et al\.Dolma: An open corpus of three trillion tokens for language model pretraining research\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15725–15788, Bangkok, Thailand, Aug\. 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.840\.
- Sun et al\. \(2025\)W\. Sun, X\. Song, P\. Li, L\. Yin, Y\. Zheng, and S\. Liu\.The curse of depth in large language models\.In*Advances in Neural Information Processing Systems*, volume 38, pages 181014–181046, 2025\.
- Tang et al\. \(2021\)H\. Tang, S\. Gan, A\. A\. Awan, S\. Rajbhandari, C\. Li, X\. Lian, et al\.1\-bit Adam: Communication efficient large\-scale training with Adam’s convergence speed\.In*International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, pages 10118–10129\. PMLR, 2021\.
- Vogels et al\. \(2019\)T\. Vogels, S\. P\. Karimireddy, and M\. Jaggi\.PowerSGD: Practical low\-rank gradient compression for distributed optimization\.In*Advances in Neural Information Processing Systems*, volume 32, pages 14259–14268, 2019\.
- Wang et al\. \(2024a\)G\. Wang, H\. Qin, S\. A\. Jacobs, X\. Wu, C\. Holmes, Z\. Yao, et al\.ZeRO\+\+: Extremely efficient collective communication for large model training\.In*International Conference on Learning Representations*, 2024a\.
- Wang et al\. \(2024b\)G\. Wang, C\. Zhang, Z\. Shen, A\. Li, and O\. Ruwase\.Domino: Eliminating communication in LLM training via generic tensor slicing and overlapping\.*arXiv preprint arXiv:2409\.15241*, 2024b\.
- Wang et al\. \(2023\)H\. Wang, S\. Ma, L\. Dong, S\. Huang, H\. Wang, L\. Ma, F\. Yang, R\. Wang, Y\. Wu, and F\. Wei\.BitNet: Scaling 1\-bit transformers for large language models\.*arXiv preprint arXiv:2310\.11453*, 2023\.
- Wang et al\. \(2022\)J\. Wang, B\. Yuan, L\. Rimanic, Y\. He, T\. Dao, B\. Chen, et al\.Fine\-tuning language models over slow networks using activation quantization with guarantees\.In*Advances in Neural Information Processing Systems*, volume 35, pages 19215–19230, 2022\.
- Wang et al\. \(2024c\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, et al\.MMLU\-Pro: A more robust and challenging multi\-task language understanding benchmark\.In*Advances in Neural Information Processing Systems*, volume 37, pages 95266–95290, 2024c\.doi:10\.52202/079017\-3018\.
- Widrow and Kollár \(2008\)B\. Widrow and I\. Kollár\.*Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications*\.Cambridge University Press, 2008\.
- Xie et al\. \(2024a\)X\. Xie, K\. Ding, S\. Yan, K\.\-C\. Toh, and T\. Wei\.Optimization hyper\-parameter laws for large language models\.*arXiv preprint arXiv:2409\.04777*, 2024a\.
- Xie et al\. \(2024b\)X\. Xie, P\. Zhou, H\. Li, Z\. Lin, and S\. Yan\.Adan: Adaptive Nesterov momentum algorithm for faster optimizing deep models\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 46\(12\):9508–9520, 2024b\.doi:10\.1109/TPAMI\.2024\.3423382\.
- Xie et al\. \(2025\)X\. Xie, Z\. Lin, K\.\-C\. Toh, and P\. Zhou\.LoCo: Low\-bit communication adaptor for large\-scale model training\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 47\(6\):4285–4298, 2025\.
- Xie et al\. \(2026\)X\. Xie, Z\. Yu, Y\. Liao, T\. Wang, K\.\-C\. Toh, and S\. Yan\.Slow\-fast inference: Training\-free inference acceleration via within\-sentence support stability\.*arXiv preprint arXiv:2603\.12038*, 2026\.
- Xiong et al\. \(2020\)R\. Xiong, Y\. Yang, D\. He, K\. Zheng, S\. Zheng, C\. Xing, et al\.On layer normalization in the Transformer architecture\.In*International Conference on Machine Learning*, volume 119 of*Proceedings of Machine Learning Research*, pages 10524–10533\. PMLR, 2020\.
- Yang et al\. \(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Zellers et al\. \(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\.HellaSwag: Can a machine really finish your sentence?In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 4791–4800, Florence, Italy, July 2019\. Association for Computational Linguistics\.doi:10\.18653/v1/P19\-1472\.
- Zhao et al\. \(2025\)C\. Zhao, X\. Wang, and J\. Huang\.Scaling LLM inference: Innovations in tensor parallelism, context parallelism, and expert parallelism\.Engineering at Meta blog,[https://engineering\.fb\.com/2025/10/17/ai\-research/scaling\-llm\-inference\-innovations\-tensor\-parallelism\-context\-parallelism\-expert\-parallelism/](https://engineering.fb.com/2025/10/17/ai-research/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-expert-parallelism/), Oct\. 2025\.

## Appendix AComplete Theoretical Proofs

This appendix proves the results of Sec\.[6](https://arxiv.org/html/2610.14277#S6)in the order in which the section states them: notation and regularity, Theorem[1](https://arxiv.org/html/2610.14277#Thmtheorem1)together with its gain and variance forms, the partial\-sum and coefficient forms behind Eqn\. \([20](https://arxiv.org/html/2610.14277#S6.E20)\), the completion of Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1), Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2), the constructions behind the sharpness discussion of Sec\.[6\.2](https://arxiv.org/html/2610.14277#S6.SS2), and the backward source bound\.

### A\.1Notation and regularity

Sites are indexedk=0,…,N−1k=0,\\dots,N\-1\. The clean trajectory𝐳k\\mathbf\{z\}\_\{k\}, the deviation𝐄k\\mathbf\{E\}\_\{k\}, the Jacobians𝐉k\\mathbf\{J\}\_\{k\}, and the products𝚽n,j=\(𝐈\+𝐉n−1\)⋯\(𝐈\+𝐉j\)\\bm\{\\Phi\}\_\{n,j\}=\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{n\-1\}\)\\cdots\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{j\}\), with𝚽n,n=𝐈\\bm\{\\Phi\}\_\{n,n\}=\\mathbf\{I\}, are those of Eqn\. \([17](https://arxiv.org/html/2610.14277#S6.E17)\)\. The scale isσk2=‖𝐳k‖2/d\\sigma\_\{k\}^\{2\}=\\norm\{\\mathbf\{z\}\_k\}^\{2\}/d, with incrementΔk=σk\+12−σk2\\Delta\_\{k\}=\\sigma\_\{k\+1\}^\{2\}\-\\sigma\_\{k\}^\{2\}\. Three per\-block quantities enter the constants, all uniform over the class of trained checkpoints considered, as is the budgetΛ\\Lambdaof Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1):

\(R1\)0<Δmin≤Δk≤Δmax,\\displaystyle 0<\\Delta\_\{\\min\}\\leq\\Delta\_\{k\}\\leq\\Delta\_\{\\max\},\(R2\)‖𝐉k‖F2/d≤C0​Δk/σk2,\\displaystyle\\norm\{\\mathbf\{J\}\_k\}\_\{F\}^\{2\}/d\\leq C\_\{0\}\\,\\Delta\_\{k\}/\\sigma\_\{k\}^\{2\},\(R3\)Hk≤h¯/\(σk2​d\),\\displaystyle H\_\{k\}\\leq\\bar\{h\}/\(\\sigma\_\{k\}^\{2\}\\sqrt\{d\}\),withHkH\_\{k\}the second\-derivative bound of Eqn\. \([16](https://arxiv.org/html/2610.14277#S6.E16)\)\. R1 records that every site adds a bounded, nonvanishing amount of energy to the hidden state, and R2 and R3 record the scaling that the normalization in front of each block imposes on its derivatives\. A bound is called depth independent when its constant is built fromΔmin,Δmax,C0,h¯\\Delta\_\{\\min\},\\Delta\_\{\\max\},C\_\{0\},\\bar\{h\}, the scalesσ0,σ1\\sigma\_\{0\},\\sigma\_\{1\}, the compensator constantsμw,τ2\\mu\_\{w\},\\tau\_\{2\}, the budgetΛ\\Lambdaof Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1), and possibly the hidden dimensiondd, and from nothing that grows withNN\.

Withℱs\\mathcal\{F\}\_\{s\}the history of the injections up to and including sitess, the noise model is

𝔼⁡\[𝜹s∣ℱs−1\]=𝟎,𝔼⁡\[𝜹s​𝜹s⊤∣ℱs−1\]⪯q2d​𝐈,‖𝜹s‖≤q​a\.s\.\\mathbb\{E\}\[\\bm\{\\delta\}\_\{s\}\\mid\\mathcal\{F\}\_\{s\-1\}\]=\\mathbf\{0\},\\qquad\\mathbb\{E\}\[\\bm\{\\delta\}\_\{s\}\\bm\{\\delta\}\_\{s\}^\{\\top\}\\mid\\mathcal\{F\}\_\{s\-1\}\]\\preceq\\frac\{q^\{2\}\}\{d\}\\,\\mathbf\{I\},\\qquad\\norm\{\\injnoise\_s\}\\leq q\\ \\text\{a\.s\.\}\(22\)The clean trajectory is fixed, so the𝐉k\\mathbf\{J\}\_\{k\}and𝚽n,j\\bm\{\\Phi\}\_\{n,j\}are deterministic and all probability is over the injected errors\.

### A\.2The source bound

###### Lemma 1\(tail representation\)\.

At gainη=1\\eta=1,𝐒t=∑s≤tTt−s​𝛅s\\;\\mathbf\{S\}\_\{t\}=\\sum\_\{s\\leq t\}T\_\{t\-s\}\\,\\bm\{\\delta\}\_\{s\}\.

###### Proof\.

Equation \([13](https://arxiv.org/html/2610.14277#S5.E13)\) gives the coefficientsϕm=1−η​∑r<mwr\\phi\_\{m\}=1\-\\eta\\sum\_\{r<m\}w\_\{r\}\. Atη=1\\eta=1, since∑rwr=1\\sum\_\{r\}w\_\{r\}=1,

ϕm=1−∑r<mwr=∑r≥mwr=Tm\.∎\\phi\_\{m\}=1\-\\sum\_\{r<m\}w\_\{r\}=\\sum\_\{r\\geq m\}w\_\{r\}=T\_\{m\}\.\\qed

###### Proof of Theorem[1](https://arxiv.org/html/2610.14277#Thmtheorem1)\.

TheTmT\_\{m\}are nonnegative, so for every sequence with‖𝜹k‖≤q\\norm\{\\injnoise\_k\}\\leq q,

‖𝐒t‖≤∑s≤tTt−s​‖𝜹s‖≤q​∑m≥0Tm=q​∑m≥0∑r≥mwr=q​∑r≥0\(r\+1\)​wr=\(1\+μw\)​q\.\\norm\{\\mathbf\{S\}\_t\}\\;\\leq\\;\\sum\_\{s\\leq t\}T\_\{t\-s\}\\norm\{\\injnoise\_s\}\\;\\leq\\;q\\sum\_\{m\\geq 0\}T\_\{m\}\\;=\\;q\\sum\_\{m\\geq 0\}\\sum\_\{r\\geq m\}w\_\{r\}\\;=\\;q\\sum\_\{r\\geq 0\}\(r\+1\)\\,w\_\{r\}\\;=\\;\(1\+\\mu\_\{w\}\)\\,q\.Without compensationϕm≡1\\phi\_\{m\}\\equiv 1, and for any fixed unit vector𝐞\\mathbf\{e\}the constant sequence𝜹k=q​𝐞\\bm\{\\delta\}\_\{k\}=q\\,\\mathbf\{e\}attains

‖𝐒N−1‖=‖∑k<Nq​𝐞‖=N​q\.\\norm\{\\mathbf\{S\}\_\{\\Nsite\-1\}\}=\\Bigl\\\|\\sum\_\{k<N\}q\\,\\mathbf\{e\}\\Bigr\\\|=Nq\.At gainη<1\\eta<1the coefficient isϕm=\(1−η\)\+η​Tm\\phi\_\{m\}=\(1\-\\eta\)\+\\eta T\_\{m\}, and the same estimate gives

maxt⁡‖𝐒t‖≤\(1−η\)​N​q\+η⁡\(1\+μw\)​q,\\max\_\{t\}\\norm\{\\mathbf\{S\}\_t\}\\;\\leq\\;\(1\-\\eta\)\\,Nq\+\\eta\\,\(1\+\\mu\_\{w\}\)\\,q,which is Eqn\. \([21](https://arxiv.org/html/2610.14277#S6.E21)\)\. Asβ1,β2→0\\beta\_\{1\},\\beta\_\{2\}\\to 0the kernel concentrates at age zero,T0=1T\_\{0\}=1,Tm≥1=0T\_\{m\\geq 1\}=0, and the bound isqq\. ∎

### A\.3The linearized deviation

The linearized deviation is𝐄Nlin=∑t𝚽N,t\+1​𝝆t\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}=\\sum\_\{t\}\\bm\{\\Phi\}\_\{N,t\+1\}\\bm\{\\rho\}\_\{t\}, the first\-order part of Eqn\. \([17](https://arxiv.org/html/2610.14277#S6.E17)\)\. From here on the gain isη=1\\eta=1\.

###### Lemma 2\(partial\-sum form\)\.

𝐄Nlin=𝐒N−1\+∑t=0N−2𝚽N,t\+2​𝐉t\+1​𝐒t\.\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}=\\mathbf\{S\}\_\{N\-1\}\+\\sum\_\{t=0\}^\{N\-2\}\\bm\{\\Phi\}\_\{N,t\+2\}\\,\\mathbf\{J\}\_\{t\+1\}\\,\\mathbf\{S\}\_\{t\}\.

###### Proof\.

Since𝝆t=𝐒t−𝐒t−1\\bm\{\\rho\}\_\{t\}=\\mathbf\{S\}\_\{t\}\-\\mathbf\{S\}\_\{t\-1\}with𝐒−1=𝟎\\mathbf\{S\}\_\{\-1\}=\\mathbf\{0\}\(Eqn\. \([6](https://arxiv.org/html/2610.14277#S3.E6)\)\), summation by parts with𝚽N,t\+1−𝚽N,t\+2=𝚽N,t\+2​𝐉t\+1\\bm\{\\Phi\}\_\{N,t\+1\}\-\\bm\{\\Phi\}\_\{N,t\+2\}=\\bm\{\\Phi\}\_\{N,t\+2\}\\mathbf\{J\}\_\{t\+1\}and𝚽N,N=𝐈\\bm\{\\Phi\}\_\{N,N\}=\\mathbf\{I\}gives

∑t=0N−1𝚽N,t\+1​\(𝐒t−𝐒t−1\)=𝚽N,N​𝐒N−1\+∑t=0N−2\(𝚽N,t\+1−𝚽N,t\+2\)​𝐒t\.∎\\sum\_\{t=0\}^\{N\-1\}\\bm\{\\Phi\}\_\{N,t\+1\}\(\\mathbf\{S\}\_\{t\}\-\\mathbf\{S\}\_\{t\-1\}\)=\\bm\{\\Phi\}\_\{N,N\}\\,\\mathbf\{S\}\_\{N\-1\}\+\\sum\_\{t=0\}^\{N\-2\}\\bigl\(\\bm\{\\Phi\}\_\{N,t\+1\}\-\\bm\{\\Phi\}\_\{N,t\+2\}\\bigr\)\\mathbf\{S\}\_\{t\}\.\\qed

The deviation therefore depends on the injected errors only through their partial sums, the form promised in Sec\.[6\.1](https://arxiv.org/html/2610.14277#S6.SS1)\. The final partial sum is carried by the identity path to the output, and each earlier partial sum enters through the Jacobian of the site that follows it and is then propagated forward\. This is where the compensator enters the output bound, through the partial sums𝐒t\\mathbf\{S\}\_\{t\}\.

###### Lemma 3\(coefficient matrices\)\.

𝐄Nlin=∑s=0N−1𝐊s​𝜹s,𝐊s=TN−1−s​𝐈\+∑t=sN−2Tt−s​𝚽N,t\+2​𝐉t\+1\.\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}=\\sum\_\{s=0\}^\{N\-1\}\\mathbf\{K\}\_\{s\}\\,\\bm\{\\delta\}\_\{s\},\\qquad\\mathbf\{K\}\_\{s\}=T\_\{N\-1\-s\}\\,\\mathbf\{I\}\+\\sum\_\{t=s\}^\{N\-2\}T\_\{t\-s\}\\,\\bm\{\\Phi\}\_\{N,t\+2\}\\,\\mathbf\{J\}\_\{t\+1\}\.

###### Proof\.

Substitute𝐒t=∑s≤tTt−s​𝜹s\\mathbf\{S\}\_\{t\}=\\sum\_\{s\\leq t\}T\_\{t\-s\}\\bm\{\\delta\}\_\{s\}\(Lemma[1](https://arxiv.org/html/2610.14277#Thmlemma1)\) into Lemma[2](https://arxiv.org/html/2610.14277#Thmlemma2)and collect the coefficient of𝜹s\\bm\{\\delta\}\_\{s\}: the term𝐒N−1\\mathbf\{S\}\_\{N\-1\}contributesTN−1−s​𝐈T\_\{N\-1\-s\}\\,\\mathbf\{I\}, and the term att≥st\\geq scontributesTt−s​𝚽N,t\+2​𝐉t\+1T\_\{t\-s\}\\,\\bm\{\\Phi\}\_\{N,t\+2\}\\mathbf\{J\}\_\{t\+1\}\. ∎

The first term of𝐊s\\mathbf\{K\}\_\{s\}is the tail of the error not yet returned by the end of the pass, carried by the identity path\. The second collects the corrections applied at the later sites, each entering through the block Jacobian there and then propagated to the output\.

###### Lemma 4\(mean square\)\.

Under Eqn\. \([22](https://arxiv.org/html/2610.14277#A1.E22)\), Eqn\. \([20](https://arxiv.org/html/2610.14277#S6.E20)\) holds:𝔼​‖𝐄Nlin‖2≤\(q2/d\)​∑s‖𝐊s‖F2\\;\\mathbb\{E\}\\,\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}\\leq\(q^\{2\}/d\)\\sum\_\{s\}\\norm\{\\mathbf\{K\}\_s\}\_\{F\}^\{2\}\.

###### Proof\.

The𝐊s\\mathbf\{K\}\_\{s\}are deterministic, so conditioning onℱt−1\\mathcal\{F\}\_\{t\-1\},

𝔼⁡⟨𝐊s​𝜹s,𝐊t​𝜹t⟩=𝔼⁡⟨𝐊s​𝜹s,𝐊t​𝔼​\[𝜹t∣ℱt−1\]⟩=0,s<t\.\\mathbb\{E\}\\langle\\mathbf\{K\}\_\{s\}\\bm\{\\delta\}\_\{s\},\\mathbf\{K\}\_\{t\}\\bm\{\\delta\}\_\{t\}\\rangle=\\mathbb\{E\}\\bigl\\langle\\mathbf\{K\}\_\{s\}\\bm\{\\delta\}\_\{s\},\\ \\mathbf\{K\}\_\{t\}\\,\\mathbb\{E\}\[\\bm\{\\delta\}\_\{t\}\\mid\\mathcal\{F\}\_\{t\-1\}\]\\bigr\\rangle=0,\\qquad s<t\.Hence

𝔼​‖𝐄Nlin‖2=∑s𝔼​‖𝐊s​𝜹s‖2=∑s𝔼​tr⁡\(𝐊s⊤​𝐊s​𝜹s​𝜹s⊤\)≤q2d​∑s‖𝐊s‖F2\.∎\\mathbb\{E\}\\,\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}=\\sum\_\{s\}\\mathbb\{E\}\\norm\{\\mathbf\{K\}\_s\\injnoise\_s\}^\{2\}=\\sum\_\{s\}\\mathbb\{E\}\\,\\operatorname\{tr\}\\bigl\(\\mathbf\{K\}\_\{s\}^\{\\top\}\\mathbf\{K\}\_\{s\}\\,\\bm\{\\delta\}\_\{s\}\\bm\{\\delta\}\_\{s\}^\{\\top\}\\bigr\)\\;\\leq\\;\\frac\{q^\{2\}\}\{d\}\\sum\_\{s\}\\norm\{\\mathbf\{K\}\_s\}\_\{F\}^\{2\}\.\\qed

### A\.4The completed assumption

The statement in Sec\.[6\.2](https://arxiv.org/html/2610.14277#S6.SS2)covers the deviations𝚽k,s\+1​𝜹s\\bm\{\\Phi\}\_\{k,s\+1\}\\bm\{\\delta\}\_\{s\}, with covariances proportional to\(q2/d\)​𝚽k,s\+1​𝚽k,s\+1⊤\(q^\{2\}/d\)\\,\\bm\{\\Phi\}\_\{k,s\+1\}\\bm\{\\Phi\}\_\{k,s\+1\}^\{\\top\}\. The proofs apply the same inequality to two further collections produced by the same propagation\. The completed family𝒞k\\mathcal\{C\}\_\{k\}is the closed convex hull of the normalizations of

𝚽k,ℓ​𝚽k,ℓ⊤,𝚽k,t\+2​𝐉t\+1​𝐉t\+1⊤​𝚽k,t\+2⊤,𝚽k,j\+1​𝔼​\[𝐫jτ​\(𝐫jτ\)⊤\]​𝚽k,j\+1⊤,\\bm\{\\Phi\}\_\{k,\\ell\}\\bm\{\\Phi\}\_\{k,\\ell\}^\{\\top\},\\qquad\\bm\{\\Phi\}\_\{k,t\+2\}\\mathbf\{J\}\_\{t\+1\}\\mathbf\{J\}\_\{t\+1\}^\{\\top\}\\bm\{\\Phi\}\_\{k,t\+2\}^\{\\top\},\\qquad\\bm\{\\Phi\}\_\{k,j\+1\}\\,\\mathbb\{E\}\\bigl\[\\mathbf\{r\}^\{\\tau\}\_\{j\}\(\\mathbf\{r\}^\{\\tau\}\_\{j\}\)^\{\\top\}\\bigr\]\\,\\bm\{\\Phi\}\_\{k,j\+1\}^\{\\top\},\(23\)overℓ≤k\\ell\\leq k,t\+2≤kt\+2\\leq k, andj\+1≤kj\+1\\leq k, with𝐫jτ\\mathbf\{r\}^\{\\tau\}\_\{j\}the stopped remainder of Proposition[1](https://arxiv.org/html/2610.14277#Thmproposition1)below, included at the noise levelqqin use\. Every member is the covariance of a deviation that injected noise, or its second\-order response, produces at sitekk\. The third collection varies with the truncation threshold of Proposition[1](https://arxiv.org/html/2610.14277#Thmproposition1), which depends onppandNN, and the assumption is understood to hold with the same budgetΛ\\Lambdafor every truncation threshold\. For𝚺=𝔼​𝐯𝐯⊤\\bm\{\\Sigma\}=\\mathbb\{E\}\\,\\mathbf\{v\}\\mathbf\{v\}^\{\\top\}the inequality of Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1)is

tr⁡\(\(𝐈\+𝐉k\)⊤​\(𝐈\+𝐉k\)​𝚺\)≤\(1\+εk\)​σk\+12σk2​tr⁡𝚺,𝚺∈𝒞k\.\\operatorname\{tr\}\\bigl\(\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)^\{\\top\}\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)\\bm\{\\Sigma\}\\bigr\)\\;\\leq\\;\(1\+\\varepsilon\_\{k\}\)\\,\\frac\{\\sigma\_\{k\+1\}^\{2\}\}\{\\sigma\_\{k\}^\{2\}\}\\,\\operatorname\{tr\}\\bm\{\\Sigma\},\\qquad\\bm\{\\Sigma\}\\in\\mathcal\{C\}\_\{k\}\.\(24\)
###### Lemma 5\(propagation of covariances\)\.

Let𝚺\\bm\{\\Sigma\}be a member of one of the three collections in Eqn\. \([23](https://arxiv.org/html/2610.14277#A1.E23)\) at sitejj\. Then forj≤n≤Nj\\leq n\\leq N,

tr⁡\(𝚽n,j​𝚺​𝚽n,j⊤\)σn2≤eΛ​tr⁡𝚺σj2\.\\frac\{\\operatorname\{tr\}\\bigl\(\\bm\{\\Phi\}\_\{n,j\}\\bm\{\\Sigma\}\\bm\{\\Phi\}\_\{n,j\}^\{\\top\}\\bigr\)\}\{\\sigma\_\{n\}^\{2\}\}\\;\\leq\\;e^\{\\Lambda\}\\,\\frac\{\\operatorname\{tr\}\\bm\{\\Sigma\}\}\{\\sigma\_\{j\}^\{2\}\}\.

###### Proof\.

Set𝚺m=𝚽m,j​𝚺​𝚽m,j⊤\\bm\{\\Sigma\}\_\{m\}=\\bm\{\\Phi\}\_\{m,j\}\\bm\{\\Sigma\}\\bm\{\\Phi\}\_\{m,j\}^\{\\top\}, so that𝚺m\+1=\(𝐈\+𝐉m\)​𝚺m​\(𝐈\+𝐉m\)⊤\\bm\{\\Sigma\}\_\{m\+1\}=\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{m\}\)\\bm\{\\Sigma\}\_\{m\}\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{m\}\)^\{\\top\}\. Since𝚽m,j​𝚽j,ℓ=𝚽m,ℓ\\bm\{\\Phi\}\_\{m,j\}\\bm\{\\Phi\}\_\{j,\\ell\}=\\bm\{\\Phi\}\_\{m,\\ell\}, each𝚺m\\bm\{\\Sigma\}\_\{m\}again belongs to the same collection of Eqn\. \([23](https://arxiv.org/html/2610.14277#A1.E23)\) at sitemm, so Eqn\. \([24](https://arxiv.org/html/2610.14277#A1.E24)\) applies at every step,

tr⁡𝚺m\+1σm\+12≤\(1\+εm\)​tr⁡𝚺mσm2,\\frac\{\\operatorname\{tr\}\\bm\{\\Sigma\}\_\{m\+1\}\}\{\\sigma\_\{m\+1\}^\{2\}\}\\;\\leq\\;\(1\+\\varepsilon\_\{m\}\)\\,\\frac\{\\operatorname\{tr\}\\bm\{\\Sigma\}\_\{m\}\}\{\\sigma\_\{m\}^\{2\}\},and∏m\(1\+εm\)≤e∑mεm≤eΛ\\prod\_\{m\}\(1\+\\varepsilon\_\{m\}\)\\leq e^\{\\sum\_\{m\}\\varepsilon\_\{m\}\}\\leq e^\{\\Lambda\}\. ∎

###### Lemma 6\(propagated sources\)\.

Under Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1),

∑t=0N−2‖𝚽N,t\+2​𝐉t\+1‖F2d≤eΛ​C0​σN2σ12\.\\sum\_\{t=0\}^\{N\-2\}\\frac\{\\norm\{\\bm\{\\Phi\}\_\{\\Nsite,t\+2\}\\mathbf\{J\}\_\{t\+1\}\}\_\{F\}^\{2\}\}\{d\}\\leq e^\{\\Lambda\}C\_\{0\}\\,\\frac\{\\sigma\_\{N\}^\{2\}\}\{\\sigma\_\{1\}^\{2\}\}\.

###### Proof\.

The matrix𝐉t\+1​𝐉t\+1⊤\\mathbf\{J\}\_\{t\+1\}\\mathbf\{J\}\_\{t\+1\}^\{\\top\}is the member of the second collection at sitej=t\+2j=t\+2, the casek=t\+2k=t\+2, where𝚽t\+2,t\+2=𝐈\\bm\{\\Phi\}\_\{t\+2,t\+2\}=\\mathbf\{I\}\. Lemma[5](https://arxiv.org/html/2610.14277#Thmlemma5), then R2 andΔt\+1/\(σt\+12​σt\+22\)=1/σt\+12−1/σt\+22\\Delta\_\{t\+1\}/\(\\sigma\_\{t\+1\}^\{2\}\\sigma\_\{t\+2\}^\{2\}\)=1/\\sigma\_\{t\+1\}^\{2\}\-1/\\sigma\_\{t\+2\}^\{2\}, give

‖𝚽N,t\+2​𝐉t\+1‖F2σN2≤eΛ​‖𝐉t\+1‖F2σt\+22≤eΛ​C0​d​\(1σt\+12−1σt\+22\),\\frac\{\\norm\{\\bm\{\\Phi\}\_\{\\Nsite,t\+2\}\\mathbf\{J\}\_\{t\+1\}\}\_\{F\}^\{2\}\}\{\\sigma\_\{N\}^\{2\}\}\\;\\leq\\;e^\{\\Lambda\}\\,\\frac\{\\norm\{\\mathbf\{J\}\_\{t\+1\}\}\_\{F\}^\{2\}\}\{\\sigma\_\{t\+2\}^\{2\}\}\\;\\leq\\;e^\{\\Lambda\}C\_\{0\}\\,d\\Bigl\(\\frac\{1\}\{\\sigma\_\{t\+1\}^\{2\}\}\-\\frac\{1\}\{\\sigma\_\{t\+2\}^\{2\}\}\\Bigr\),and the sum overtttelescopes to at mosteΛ​C0​d/σ12e^\{\\Lambda\}C\_\{0\}\\,d/\\sigma\_\{1\}^\{2\}\. ∎

### A\.5Proof of the propagation theorem

###### Proof of Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2), part \(i\)\.

Writeft=‖𝚽N,t\+2​𝐉t\+1‖Ff\_\{t\}=\\norm\{\\bm\{\\Phi\}\_\{\\Nsite,t\+2\}\\mathbf\{J\}\_\{t\+1\}\}\_\{F\}\. Taking Frobenius norms in Lemma[3](https://arxiv.org/html/2610.14277#Thmlemma3),

‖𝐊s‖F≤d​TN−1−s\+∑t≥sTt−s​ft\.\\norm\{\\mathbf\{K\}\_s\}\_\{F\}\\;\\leq\\;\\sqrt\{d\}\\,T\_\{N\-1\-s\}\+\\sum\_\{t\\geq s\}T\_\{t\-s\}f\_\{t\}\.Squaring, summing overss, and applying Young’s inequality‖T∗f‖ℓ2≤‖T‖ℓ1​‖f‖ℓ2\\\|T\*f\\\|\_\{\\ell\_\{2\}\}\\leq\\\|T\\\|\_\{\\ell\_\{1\}\}\\\|f\\\|\_\{\\ell\_\{2\}\}with‖T‖ℓ1=1\+μw\\\|T\\\|\_\{\\ell\_\{1\}\}=1\+\\mu\_\{w\}, then Lemma[6](https://arxiv.org/html/2610.14277#Thmlemma6),

∑s‖𝐊s‖F2\\displaystyle\\sum\_\{s\}\\norm\{\\mathbf\{K\}\_s\}\_\{F\}^\{2\}≤2​d​τ2\+2​\(1\+μw\)2​∑tft2≤2​d​τ2\+2​\(1\+μw\)2​d​eΛ​C0​σN2σ12\.\\displaystyle\\;\\leq\\;2d\\,\\tau\_\{2\}\+2\\,\(1\+\\mu\_\{w\}\)^\{2\}\\sum\_\{t\}f\_\{t\}^\{2\}\\;\\leq\\;2d\\,\\tau\_\{2\}\+2\\,\(1\+\\mu\_\{w\}\)^\{2\}\\,d\\,e^\{\\Lambda\}C\_\{0\}\\,\\frac\{\\sigma\_\{N\}^\{2\}\}\{\\sigma\_\{1\}^\{2\}\}\.Inserting into Lemma[4](https://arxiv.org/html/2610.14277#Thmlemma4)and dividing byd​σN2d\\sigma\_\{N\}^\{2\},

𝔼​‖𝐄Nlin‖2d​σN2≤q2d2​σN2​\[2​d​τ2\+2​\(1\+μw\)2​d​eΛ​C0​σN2σ12\]=2​\[τ2​σ12σN2\+\(1\+μw\)2​C0​eΛ\]​q¯2\.∎\\frac\{\\mathbb\{E\}\\,\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}\}\{d\\sigma\_\{N\}^\{2\}\}\\;\\leq\\;\\frac\{q^\{2\}\}\{d^\{2\}\\sigma\_\{N\}^\{2\}\}\\Bigl\[2d\\,\\tau\_\{2\}\+2\\,\(1\+\\mu\_\{w\}\)^\{2\}\\,d\\,e^\{\\Lambda\}C\_\{0\}\\,\\frac\{\\sigma\_\{N\}^\{2\}\}\{\\sigma\_\{1\}^\{2\}\}\\Bigr\]\\;=\\;2\\Bigl\[\\tau\_\{2\}\\frac\{\\sigma\_\{1\}^\{2\}\}\{\\sigma\_\{N\}^\{2\}\}\+\(1\+\\mu\_\{w\}\)^\{2\}C\_\{0\}e^\{\\Lambda\}\\Bigr\]\\bar\{q\}^\{2\}\.\\qed

###### Corollary 1\(general gain\)\.

Under the hypotheses of Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2), at gainη∈\[0,1\]\\eta\\in\[0,1\],

𝔼​‖𝐄Nlin‖2d​σN2≤4​η2​\[τ2​σ12σN2\+\(1\+μw\)2​C0​eΛ\]​q¯2\+2​\(1−η\)2​eΛ​q¯2​∑s=0N−1σ12σs\+12\.\\frac\{\\mathbb\{E\}\\,\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}\}\{d\\sigma\_\{N\}^\{2\}\}\\;\\leq\\;4\\eta^\{2\}\\Big\[\\tau\_\{2\}\\frac\{\\sigma\_\{1\}^\{2\}\}\{\\sigma\_\{N\}^\{2\}\}\+\(1\+\\mu\_\{w\}\)^\{2\}C\_\{0\}e^\{\\Lambda\}\\Big\]\\bar\{q\}^\{2\}\+2\(1\-\\eta\)^\{2\}e^\{\\Lambda\}\\bar\{q\}^\{2\}\\sum\_\{s=0\}^\{N\-1\}\\frac\{\\sigma\_\{1\}^\{2\}\}\{\\sigma\_\{s\+1\}^\{2\}\}\.

###### Proof\.

By Eqn\. \([12](https://arxiv.org/html/2610.14277#S5.E12)\) the correction is linear in the gain,𝐂k=η​𝐂k\(1\)\\mathbf\{C\}\_\{k\}=\\eta\\,\\mathbf\{C\}^\{\(1\)\}\_\{k\}with𝐂k\(1\)\\mathbf\{C\}^\{\(1\)\}\_\{k\}the correction atη=1\\eta=1, so𝝆k=\(1−η\)​𝜹k\+η⁡\(𝜹k\+𝐂k\(1\)\)\\bm\{\\rho\}\_\{k\}=\(1\-\\eta\)\\,\\bm\{\\delta\}\_\{k\}\+\\eta\\,\(\\bm\{\\delta\}\_\{k\}\+\\mathbf\{C\}^\{\(1\)\}\_\{k\}\)\. The linearized deviation is linear in the net errors, so it is1−η1\-\\etatimes the uncompensated deviation plusη\\etatimes the deviation atη=1\\eta=1, both driven by the same injections\. The bound follows from\(x\+y\)2≤2​x2\+2​y2\(x\+y\)^\{2\}\\leq 2x^\{2\}\+2y^\{2\}, part \(i\), and Remark[2](https://arxiv.org/html/2610.14277#Thmremark2)\. ∎

###### Proof of part \(ii\)\.

The sums𝐌n=∑s≤n𝐊s​𝜹s\\mathbf\{M\}\_\{n\}=\\sum\_\{s\\leq n\}\\mathbf\{K\}\_\{s\}\\bm\{\\delta\}\_\{s\}form a martingale inℝd\\mathbb\{R\}^\{d\}\. Its predictable quadratic variation is at most the deterministic majorant

vN:=q2d​∑s‖𝐊s‖F2,‖𝐊s​𝜹s‖≤ζN:=q​maxs​‖𝐊s‖\.v\_\{N\}:=\\frac\{q^\{2\}\}\{d\}\\sum\_\{s\}\\norm\{\\mathbf\{K\}\_s\}\_\{F\}^\{2\},\\qquad\\norm\{\\mathbf\{K\}\_s\\injnoise\_s\}\\leq\\zeta\_\{N\}:=q\\max\_\{s\}\\norm\{\\mathbf\{K\}\_s\}\.By the first display in the proof of Lemma[6](https://arxiv.org/html/2610.14277#Thmlemma6), withΔt\+1≤Δmax\\Delta\_\{t\+1\}\\leq\\Delta\_\{\\max\},

‖𝚽N,t\+2​𝐉t\+1‖≤σN​d​eΛ​C0​Δmaxσt\+1​σt\+2,\\norm\{\\bm\{\\Phi\}\_\{\\Nsite,t\+2\}\\mathbf\{J\}\_\{t\+1\}\}\\;\\leq\\;\\sigma\_\{N\}\\,\\frac\{\\sqrt\{d\\,e^\{\\Lambda\}C\_\{0\}\\,\\Delta\_\{\\max\}\}\}\{\\sigma\_\{t\+1\}\\sigma\_\{t\+2\}\},so, by Lemma[3](https://arxiv.org/html/2610.14277#Thmlemma3),Tm≤1T\_\{m\}\\leq 1,∑mTm=1\+μw\\sum\_\{m\}T\_\{m\}=1\+\\mu\_\{w\}, andσt\+1​σt\+2≥σ12\\sigma\_\{t\+1\}\\sigma\_\{t\+2\}\\geq\\sigma\_\{1\}^\{2\},

‖𝐊s‖≤1\+\(1\+μw\)​σNσ12​d​eΛ​C0​Δmax\.\\norm\{\\mathbf\{K\}\_s\}\\;\\leq\\;1\+\(1\+\\mu\_\{w\}\)\\,\\frac\{\\sigma\_\{N\}\}\{\\sigma\_\{1\}^\{2\}\}\\,\\sqrt\{d\\,e^\{\\Lambda\}C\_\{0\}\\,\\Delta\_\{\\max\}\}\.The Bennett–Bernstein inequality for Hilbert\-space martingales\([Pinelis, 1994](https://arxiv.org/html/2610.14277#bib.bib36)\), applied withvNv\_\{N\}andζN\\zeta\_\{N\}, gives

Pr\[maxn‖𝐌n‖≥2​vN​λp\+23ζNλp\]≤p,λp=log2p\.\\Pr\\Bigl\[\\max\_\{n\}\\norm\{\\mathbf\{M\}\_n\}\\geq\\sqrt\{2v\_\{N\}\\lambda\_\{p\}\}\+\\tfrac\{2\}\{3\}\\zeta\_\{N\}\\lambda\_\{p\}\\Bigr\]\\leq p,\\qquad\\lambda\_\{p\}=\\log\\tfrac\{2\}\{p\}\.On the complementary event,\(x\+y\)2≤2​x2\+2​y2\(x\+y\)^\{2\}\\leq 2x^\{2\}\+2y^\{2\}gives‖𝐄Nlin‖2≤4​vN​λp\+89​ζN2​λp2\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}\\leq 4v\_\{N\}\\lambda\_\{p\}\+\\tfrac\{8\}\{9\}\\zeta\_\{N\}^\{2\}\\lambda\_\{p\}^\{2\}\. Dividing byd​σN2d\\sigma\_\{N\}^\{2\}and using part \(i\) forvNv\_\{N\}and the display above forζN\\zeta\_\{N\},

‖𝐄Nlin‖2d​σN2≤C​q¯2​λp\+C′​q¯2​λp2,C:=8​\[τ2​σ12σN2\+\(1\+μw\)2​C0​eΛ\],\\frac\{\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}\}\{d\\sigma\_\{N\}^\{2\}\}\\;\\leq\\;C\\,\\bar\{q\}^\{2\}\\lambda\_\{p\}\+C^\{\\prime\}\\bar\{q\}^\{2\}\\lambda\_\{p\}^\{2\},\\qquad C:=8\\Bigl\[\\tau\_\{2\}\\frac\{\\sigma\_\{1\}^\{2\}\}\{\\sigma\_\{N\}^\{2\}\}\+\(1\+\\mu\_\{w\}\)^\{2\}C\_\{0\}e^\{\\Lambda\}\\Bigr\],C′:=89​\(σ1σN\+\(1\+μw\)​d​eΛ​C0​Δmaxσ1\)2,C^\{\\prime\}:=\\frac\{8\}\{9\}\\Bigl\(\\frac\{\\sigma\_\{1\}\}\{\\sigma\_\{N\}\}\+\(1\+\\mu\_\{w\}\)\\,\\frac\{\\sqrt\{d\\,e^\{\\Lambda\}C\_\{0\}\\,\\Delta\_\{\\max\}\}\}\{\\sigma\_\{1\}\}\\Bigr\)^\{2\},sinceζN/\(d​σN\)≤q¯​σ1/σN\+q¯​\(1\+μw\)​d​eΛ​C0​Δmax/σ1\\zeta\_\{N\}/\(\\sqrt\{d\}\\,\\sigma\_\{N\}\)\\leq\\bar\{q\}\\,\\sigma\_\{1\}/\\sigma\_\{N\}\+\\bar\{q\}\\,\(1\+\\mu\_\{w\}\)\\sqrt\{d\\,e^\{\\Lambda\}C\_\{0\}\\,\\Delta\_\{\\max\}\}/\\sigma\_\{1\}\. Both constants are depth independent\. The factorddinsideC′C^\{\\prime\}comes from bounding the operator norm of𝚽N,t\+2​𝐉t\+1\\bm\{\\Phi\}\_\{N,t\+2\}\\mathbf\{J\}\_\{t\+1\}by its Frobenius norm, and it multiplies only thelog2\\log^\{2\}term\.

The statement is for the fixed endpointNN\. For a prefix endpointnnthe same proof gives the constants withσn\\sigma\_\{n\}in place ofσN\\sigma\_\{N\}, and sinceσ1/σn≤1\\sigma\_\{1\}/\\sigma\_\{n\}\\leq 1, all of them are dominated by the uniform constants

C¯:=8​\[τ2\+\(1\+μw\)2​C0​eΛ\],C¯′:=89​\(1\+\(1\+μw\)​d​eΛ​C0​Δmaxσ1\)2\.\\bar\{C\}:=8\\bigl\[\\tau\_\{2\}\+\(1\+\\mu\_\{w\}\)^\{2\}C\_\{0\}e^\{\\Lambda\}\\bigr\],\\qquad\\bar\{C\}^\{\\prime\}:=\\frac\{8\}\{9\}\\Bigl\(1\+\(1\+\\mu\_\{w\}\)\\,\\frac\{\\sqrt\{d\\,e^\{\\Lambda\}C\_\{0\}\\,\\Delta\_\{\\max\}\}\}\{\\sigma\_\{1\}\}\\Bigr\)^\{2\}\.Covering all prefixes at levelp/Np/Nreplacesλp\\lambda\_\{p\}bylog⁡\(2​N/p\)\\log\(2\\Nsite/p\)via a union bound\. ∎

###### Proposition 1\(second\-order remainder\)\.

Under the hypotheses of Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2), fixp∈\(0,1\)p\\in\(0,1\), letΘn\\Theta\_\{n\}be the bound that the proof of Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)\(ii\) gives for the prefix ending at sitenn, at levelp/Np/N, letτ\\taube the first site with‖𝐄n‖\>2​Θn\\norm\{\\mathbf\{E\}\_n\}\>2\\Theta\_\{n\}\(τ=N\\tau=Nif there is none\), and set𝐫nτ=𝐫n\\mathbf\{r\}^\{\\tau\}\_\{n\}=\\mathbf\{r\}\_\{n\}forn<τn<\\tauand𝟎\\mathbf\{0\}otherwise\. Then

\(𝔼​‖∑n<N𝚽N,n\+1​𝐫nτ‖2\)1/2≤C2​q¯2​N​d​σN,\\Bigl\(\\mathbb\{E\}\\,\\Bigl\\\|\\sum\_\{n<N\}\\bm\{\\Phi\}\_\{N,n\+1\}\\mathbf\{r\}^\{\\tau\}\_\{n\}\\Bigr\\\|^\{2\}\\Bigr\)^\{1/2\}\\;\\leq\\;C\_\{2\}\\,\\bar\{q\}^\{2\}\\sqrt\{N\}\\,\\sqrt\{d\}\\,\\sigma\_\{N\},withC2C\_\{2\}growing at most aslog2⁡\(N/p\)\\log^\{2\}\(N/p\)\. The sum equals the second\-order part𝐄N−𝐄Nlin\\mathbf\{E\}\_\{N\}\-\\mathbf\{E\}\_\{N\}^\{\\mathrm\{lin\}\}whenever no site exceeds the threshold\.

###### Proof of Proposition[1](https://arxiv.org/html/2610.14277#Thmproposition1)\.

Setκp2:=C¯​log⁡\(2​N/p\)\+C¯′​log2⁡\(2​N/p\)\\kappa\_\{p\}^\{2\}:=\\bar\{C\}\\log\(2\\Nsite/p\)\+\\bar\{C\}^\{\\prime\}\\log^\{2\}\(2N/p\), the uniform constant with which part \(ii\), applied at levelp/Np/Nand extended to all prefixes, bounds every prefix deviation, so thatΘn=κp​q¯​d​σn\\Theta\_\{n\}=\\kappa\_\{p\}\\,\\bar\{q\}\\,\\sqrt\{d\}\\,\\sigma\_\{n\}, withτ\\tauand𝐫nτ\\mathbf\{r\}^\{\\tau\}\_\{n\}as in the statement\. The truncated sum∑n<N𝚽N,n\+1​𝐫nτ\\sum\_\{n<N\}\\bm\{\\Phi\}\_\{N,n\+1\}\\mathbf\{r\}^\{\\tau\}\_\{n\}has the deterministic endpointNN, and on the eventτ=N\\tau=Nit equals𝐄N−𝐄Nlin\\mathbf\{E\}\_\{N\}\-\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}, the second\-order part of Eqn\. \([17](https://arxiv.org/html/2610.14277#S6.E17)\)\.

Forn<τn<\\tau, R3 and the definition ofτ\\taugive

‖𝐫nτ‖≤12​Hn​‖𝐄n‖2≤h¯2​σn2​d​\(2​Θn\)2=2​h¯​κp2​q¯2​d\.\\norm\{\\mathbf\{r\}^\\tau\_n\}\\;\\leq\\;\\tfrac\{1\}\{2\}H\_\{n\}\\norm\{\\mathbf\{E\}\_n\}^\{2\}\\;\\leq\\;\\frac\{\\bar\{h\}\}\{2\\sigma\_\{n\}^\{2\}\\sqrt\{d\}\}\\,\(2\\Theta\_\{n\}\)^\{2\}\\;=\\;2\\bar\{h\}\\,\\kappa\_\{p\}^\{2\}\\,\\bar\{q\}^\{2\}\\sqrt\{d\}\.The covariances of the𝐫nτ\\mathbf\{r\}^\{\\tau\}\_\{n\}form the third collection of Eqn\. \([23](https://arxiv.org/html/2610.14277#A1.E23)\), so Lemma[5](https://arxiv.org/html/2610.14277#Thmlemma5), Minkowski’s inequality, and∑n1/σn\+1≤2​σN/Δmin\\sum\_\{n\}1/\\sigma\_\{n\+1\}\\leq 2\\sigma\_\{N\}/\\Delta\_\{\\min\}\(from R1\) give

‖∑n<N𝚽N,n\+1​𝐫nτ‖L2≤∑neΛ/2​σNσn\+1​‖𝐫nτ‖L2≤2​h¯​κp2​eΛ/2​q¯2​d​2​σN2Δmin\.\\displaystyle\\Bigl\\\|\\sum\_\{n<N\}\\bm\{\\Phi\}\_\{N,n\+1\}\\mathbf\{r\}^\{\\tau\}\_\{n\}\\Bigr\\\|\_\{L^\{2\}\}\\;\\leq\\;\\sum\_\{n\}e^\{\\Lambda/2\}\\,\\frac\{\\sigma\_\{N\}\}\{\\sigma\_\{n\+1\}\}\\,\\bigl\\\|\\mathbf\{r\}^\{\\tau\}\_\{n\}\\bigr\\\|\_\{L^\{2\}\}\\;\\leq\\;2\\bar\{h\}\\,\\kappa\_\{p\}^\{2\}\\,e^\{\\Lambda/2\}\\bar\{q\}^\{2\}\\sqrt\{d\}\\;\\frac\{2\\sigma\_\{N\}^\{2\}\}\{\\Delta\_\{\\min\}\}\.SinceσN≤σ1\+Δmax​N≤\(σ1\+Δmax\)​N\\sigma\_\{N\}\\leq\\sigma\_\{1\}\+\\sqrt\{\\Delta\_\{\\max\}N\}\\leq\(\\sigma\_\{1\}\+\\sqrt\{\\Delta\_\{\\max\}\}\)\\sqrt\{N\},

4​h¯​κp2​eΛ/2Δminq¯2dσN2≤4​h¯​eΛ/2​\(σ1\+Δmax\)Δmin​κp2⏟=:C2q¯2NdσN\.∎\\frac\{4\\bar\{h\}\\,\\kappa\_\{p\}^\{2\}\\,e^\{\\Lambda/2\}\}\{\\Delta\_\{\\min\}\}\\,\\bar\{q\}^\{2\}\\sqrt\{d\}\\;\\sigma\_\{N\}^\{2\}\\;\\leq\\;\\underbrace\{\\frac\{4\\bar\{h\}\\,e^\{\\Lambda/2\}\(\\sigma\_\{1\}\+\\sqrt\{\\Delta\_\{\\max\}\}\)\}\{\\Delta\_\{\\min\}\}\\,\\kappa\_\{p\}^\{2\}\}\_\{=:C\_\{2\}\}\\;\\bar\{q\}^\{2\}\\sqrt\{N\}\\,\\sqrt\{d\}\\,\\sigma\_\{N\}\.\\qed

###### Corollary 2\(second\-order budget\)\.

If in addition

∑n\(𝔼​‖𝐫nτ‖2\)1/2d​σn\+1≤Λ2​q¯2\\sum\_\{n\}\\frac\{\\bigl\(\\mathbb\{E\}\\norm\{\\mathbf\{r\}^\\tau\_n\}^\{2\}\\bigr\)^\{1/2\}\}\{\\sqrt\{d\}\\,\\sigma\_\{n\+1\}\}\\;\\leq\\;\\Lambda\_\{2\}\\,\\bar\{q\}^\{2\}\(25\)withΛ2\\Lambda\_\{2\}independent ofNN, then

\(𝔼​‖∑n<N𝚽N,n\+1​𝐫nτ‖2\)1/2≤∑neΛ/2​σNσn\+1​‖𝐫nτ‖L2≤eΛ/2​Λ2​q¯2​d​σN,\\Bigl\(\\mathbb\{E\}\\,\\Bigl\\\|\\sum\_\{n<N\}\\bm\{\\Phi\}\_\{N,n\+1\}\\mathbf\{r\}^\{\\tau\}\_\{n\}\\Bigr\\\|^\{2\}\\Bigr\)^\{1/2\}\\;\\leq\\;\\sum\_\{n\}e^\{\\Lambda/2\}\\,\\frac\{\\sigma\_\{N\}\}\{\\sigma\_\{n\+1\}\}\\,\\bigl\\\|\\mathbf\{r\}^\{\\tau\}\_\{n\}\\bigr\\\|\_\{L^\{2\}\}\\;\\leq\\;e^\{\\Lambda/2\}\\Lambda\_\{2\}\\,\\bar\{q\}^\{2\}\\sqrt\{d\}\\,\\sigma\_\{N\},with no factorN\\sqrt\{N\}\.

Condition \([25](https://arxiv.org/html/2610.14277#A1.E25)\) asks that the total second\-order response along a pass stay at the level of a single response\. Standard post\-training bias correction shows that quantization\-induced mean shifts can be compensated explicitly\([Nagel et al\., 2019](https://arxiv.org/html/2610.14277#bib.bib31)\), which motivates the mean part of this budget without proving it\.

### A\.6Sharpness

###### Proposition 2\(sharpness\)\.

\(a\) There is a network satisfying R1–R3 and the noise model in which𝔼​‖𝐄Nlin‖2/\(d​σN2\)\\mathbb\{E\}\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}/\(d\\sigma\_\{N\}^\{2\}\)grows exponentially inN\\sqrt\{N\}\. Its budgetΛ\\Lambdain Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1)grows asN\\sqrt\{N\}\. \(b\) There is a network satisfying R1–R3, the noise model, and Eqn\. \([24](https://arxiv.org/html/2610.14277#A1.E24)\) with everyεk=0\\varepsilon\_\{k\}=0for every positive semidefinite𝚺\\bm\{\\Sigma\}, in which the stopped remainder of Proposition[1](https://arxiv.org/html/2610.14277#Thmproposition1)attains the orderq¯2​N\\bar\{q\}^\{2\}\\sqrt\{N\}\.

###### Proof of \(a\)\.

Fix orthonormal vectors𝐞,𝐞0\\mathbf\{e\},\\mathbf\{e\}\_\{0\}, a constantc\>0c\>0, and the scale pathσk2=σ02\+k​Δ\\sigma\_\{k\}^\{2\}=\\sigma\_\{0\}^\{2\}\+k\\Delta, and take the affine blocks

Fk​\(𝐱\)=cσk​𝐞𝐞⊤​𝐱\+d​\(σk\+1−σk\)​𝐞0F\_\{k\}\(\\mathbf\{x\}\)=\\frac\{c\}\{\\sigma\_\{k\}\}\\,\\mathbf\{e\}\\mathbf\{e\}^\{\\top\}\\mathbf\{x\}\+\\sqrt\{d\}\\,\(\\sigma\_\{k\+1\}\-\\sigma\_\{k\}\)\\,\\mathbf\{e\}\_\{0\}on the clean ray𝐳k=d​σk​𝐞0\\mathbf\{z\}\_\{k\}=\\sqrt\{d\}\\,\\sigma\_\{k\}\\mathbf\{e\}\_\{0\}\. Then𝐳k\+1=𝐳k\+Fk​\(𝐳k\)\\mathbf\{z\}\_\{k\+1\}=\\mathbf\{z\}\_\{k\}\+F\_\{k\}\(\\mathbf\{z\}\_\{k\}\)stays on the ray,

𝐉k=cσk​𝐞𝐞⊤,\(𝐈\+𝐉k\)​𝐞=\(1\+cσk\)​𝐞,\\mathbf\{J\}\_\{k\}=\\frac\{c\}\{\\sigma\_\{k\}\}\\,\\mathbf\{e\}\\mathbf\{e\}^\{\\top\},\\qquad\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)\\mathbf\{e\}=\\Bigl\(1\+\\frac\{c\}\{\\sigma\_\{k\}\}\\Bigr\)\\mathbf\{e\},andHk=0H\_\{k\}=0, so R3 is trivial and R1 and R2 hold with constants depending oncc,Δ\\Delta, anddd\. Let the injected errors be independent and uniform on the sphere of radiusqq, so the noise model holds with conditional covariance exactly\(q2/d\)​𝐈\(q^\{2\}/d\)\\,\\mathbf\{I\}and Lemma[4](https://arxiv.org/html/2610.14277#Thmlemma4)is an equality\.

In Lemma[3](https://arxiv.org/html/2610.14277#Thmlemma3)all terms of𝐊s​𝐞\\mathbf\{K\}\_\{s\}\\mathbf\{e\}carry one sign, so for an early sitess,

‖𝐊s​𝐞‖≥T0​cσs\+1​∏ℓ=s\+2N−1\(1\+cσℓ\),∑ℓlog⁡\(1\+cσℓ\)≍2​c​N/Δ\.\\norm\{\\mathbf\{K\}\_s\\mathbf\{e\}\}\\;\\geq\\;T\_\{0\}\\,\\frac\{c\}\{\\sigma\_\{s\+1\}\}\\prod\_\{\\ell=s\+2\}^\{N\-1\}\\Bigl\(1\+\\frac\{c\}\{\\sigma\_\{\\ell\}\}\\Bigr\),\\qquad\\sum\_\{\\ell\}\\log\\Bigl\(1\+\\frac\{c\}\{\\sigma\_\{\\ell\}\}\\Bigr\)\\;\\asymp\\;2c\\sqrt\{N/\\Delta\}\.Hence

𝔼​‖𝐄Nlin‖2=q2d​∑s′‖𝐊s′‖F2≥q2d​‖𝐊s​𝐞‖2,\\mathbb\{E\}\\,\\\|\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{N\}\\\|^\{2\}=\\frac\{q^\{2\}\}\{d\}\\sum\_\{s^\{\\prime\}\}\\norm\{\\mathbf\{K\}\_\{s'\}\}\_\{F\}^\{2\}\\;\\geq\\;\\frac\{q^\{2\}\}\{d\}\\,\\norm\{\\mathbf\{K\}\_\{s\}\\mathbf\{e\}\}^\{2\},which grows exponentially inN\\sqrt\{N\}even after dividing byd​σN2d\\sigma\_\{N\}^\{2\}\.

It remains to check that the network violates Assumption[1](https://arxiv.org/html/2610.14277#Thmassumption1)\. An injection at sitessproduces at sitekkthe deviation𝐯=𝚽k,s\+1​𝜹s\\mathbf\{v\}=\\bm\{\\Phi\}\_\{k,s\+1\}\\bm\{\\delta\}\_\{s\}with covariance proportional to𝚽k,s\+1​𝚽k,s\+1⊤=𝐈−𝐞𝐞⊤\+πk2​𝐞𝐞⊤\\bm\{\\Phi\}\_\{k,s\+1\}\\bm\{\\Phi\}\_\{k,s\+1\}^\{\\top\}=\\mathbf\{I\}\-\\mathbf\{e\}\\mathbf\{e\}^\{\\top\}\+\\pi\_\{k\}^\{2\}\\,\\mathbf\{e\}\\mathbf\{e\}^\{\\top\}, whereπk=∏ℓ=s\+1k−1\(1\+c/σℓ\)→∞\\pi\_\{k\}=\\prod\_\{\\ell=s\+1\}^\{k\-1\}\(1\+c/\\sigma\_\{\\ell\}\)\\to\\infty\. Since𝐈\+𝐉k\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}fixes𝐞⟂\\mathbf\{e\}^\{\\perp\}and stretches𝐞\\mathbf\{e\}by1\+c/σk1\+c/\\sigma\_\{k\},

𝔼​‖\(𝐈\+𝐉k\)​𝐯‖2𝔼​‖𝐯‖2=\(d−1\)\+πk2​\(1\+c/σk\)2\(d−1\)\+πk2≥1\+2​cσk⋅πk2\(d−1\)\+πk2≥1\+cσk\\frac\{\\mathbb\{E\}\\\|\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)\\mathbf\{v\}\\\|^\{2\}\}\{\\mathbb\{E\}\\norm\{\\mathbf\{v\}\}^\{2\}\}=\\frac\{\(d\-1\)\+\\pi\_\{k\}^\{2\}\(1\+c/\\sigma\_\{k\}\)^\{2\}\}\{\(d\-1\)\+\\pi\_\{k\}^\{2\}\}\\;\\geq\\;1\+\\frac\{2c\}\{\\sigma\_\{k\}\}\\cdot\\frac\{\\pi\_\{k\}^\{2\}\}\{\(d\-1\)\+\\pi\_\{k\}^\{2\}\}\\;\\geq\\;1\+\\frac\{c\}\{\\sigma\_\{k\}\}onceπk2≥d\\pi\_\{k\}^\{2\}\\geq d\. The assumption then requires\(1\+εk\)​\(1\+Δ/σk2\)≥1\+c/σk\(1\+\\varepsilon\_\{k\}\)\(1\+\\Delta/\\sigma\_\{k\}^\{2\}\)\\geq 1\+c/\\sigma\_\{k\}, so at every site withπk2≥d\\pi\_\{k\}^\{2\}\\geq dandσk≥max⁡\(2​Δ/c,Δ\)\\sigma\_\{k\}\\geq\\max\(2\\Delta/c,\\sqrt\{\\Delta\}\),

εk≥c/σk−Δ/σk21\+Δ/σk2≥c4​σk\.\\varepsilon\_\{k\}\\;\\geq\\;\\frac\{c/\\sigma\_\{k\}\-\\Delta/\\sigma\_\{k\}^\{2\}\}\{1\+\\Delta/\\sigma\_\{k\}^\{2\}\}\\;\\geq\\;\\frac\{c\}\{4\\sigma\_\{k\}\}\.Both conditions hold from a fixed site on, withπk→∞\\pi\_\{k\}\\to\\inftyats=0s=0andσk→∞\\sigma\_\{k\}\\to\\infty, and the sum over the remaining sites,∑kc/\(4​σk\)≍\(c/2\)​N/Δ\\sum\_\{k\}c/\(4\\sigma\_\{k\}\)\\asymp\(c/2\)\\sqrt\{N/\\Delta\}, grows asN\\sqrt\{N\}, so noΛ\\Lambdaindependent ofNNexists\. Conversely,\(𝐈\+𝐉k\)⊤​\(𝐈\+𝐉k\)⪯\(1\+c/σk\)2​𝐈\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)^\{\\top\}\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)\\preceq\(1\+c/\\sigma\_\{k\}\)^\{2\}\\mathbf\{I\}givesεk≤2​c/σk\+c2/σk2\\varepsilon\_\{k\}\\leq 2c/\\sigma\_\{k\}\+c^\{2\}/\\sigma\_\{k\}^\{2\}, so the budget grows exactly asN\\sqrt\{N\}\. ∎

###### Proof of \(b\)\.

Take orthonormal𝐞0,𝐟\\mathbf\{e\}\_\{0\},\\mathbf\{f\},𝚷=𝐈−𝐞0​𝐞0⊤\\bm\{\\Pi\}=\\mathbf\{I\}\-\\mathbf\{e\}\_\{0\}\\mathbf\{e\}\_\{0\}^\{\\top\},n⁡\(𝐱\)=𝐱/‖𝐱‖n\(\\mathbf\{x\}\)=\\mathbf\{x\}/\\norm\{\\mathbf\{x\}\}, a constanth\>0h\>0, and the00\-homogeneous blocks

Fk​\(𝐱\)=bk​n​\(𝐱\)\+12​h​d​‖𝚷​n​\(𝐱\)‖2​𝐟,bk=d​\(σk\+1−σk\),F\_\{k\}\(\\mathbf\{x\}\)=b\_\{k\}\\,n\(\\mathbf\{x\}\)\+\\tfrac\{1\}\{2\}h\\sqrt\{d\}\\,\\norm\{\\bm\{\\Pi\}n\(\\mathbf\{x\}\)\}^\{2\}\\,\\mathbf\{f\},\\qquad b\_\{k\}=\\sqrt\{d\}\\,\(\\sigma\_\{k\+1\}\-\\sigma\_\{k\}\),on the clean ray𝐳k=d​σk​𝐞0\\mathbf\{z\}\_\{k\}=\\sqrt\{d\}\\,\\sigma\_\{k\}\\mathbf\{e\}\_\{0\},σk2=σ02\+k​Δ\\sigma\_\{k\}^\{2\}=\\sigma\_\{0\}^\{2\}\+k\\Delta\. At the clean point the quadratic term has zero first derivative, and

𝐉k=αk​𝚷,1\+αk=σk\+1σk,\(𝐈\+𝐉k\)⊤​\(𝐈\+𝐉k\)=𝐈\+Δσk2​𝚷⪯σk\+12σk2​𝐈,\\mathbf\{J\}\_\{k\}=\\alpha\_\{k\}\\bm\{\\Pi\},\\quad 1\+\\alpha\_\{k\}=\\frac\{\\sigma\_\{k\+1\}\}\{\\sigma\_\{k\}\},\\qquad\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)^\{\\top\}\(\\mathbf\{I\}\+\\mathbf\{J\}\_\{k\}\)=\\mathbf\{I\}\+\\frac\{\\Delta\}\{\\sigma\_\{k\}^\{2\}\}\\,\\bm\{\\Pi\}\\preceq\\frac\{\\sigma\_\{k\+1\}^\{2\}\}\{\\sigma\_\{k\}^\{2\}\}\\,\\mathbf\{I\},so Eqn\. \([24](https://arxiv.org/html/2610.14277#A1.E24)\) holds withεk=0\\varepsilon\_\{k\}=0for every positive semidefinite𝚺\\bm\{\\Sigma\}, and R1–R3 hold\.

Inject one error at site00, uniform on the sphere of radiusqq, and none later\. Its tangential part is carried at the critical gain,𝚽n,k\|𝚷=\(σn/σk\)​𝚷\\bm\{\\Phi\}\_\{n,k\}\\big\|\_\{\\bm\{\\Pi\}\}=\(\\sigma\_\{n\}/\\sigma\_\{k\}\)\\,\\bm\{\\Pi\}, so𝐗k:=𝚷​𝐄klin/σk\\mathbf\{X\}\_\{k\}:=\\bm\{\\Pi\}\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{k\}/\\sigma\_\{k\}satisfies𝐗k\+1=𝐗k\+𝚷​𝝆k/σk\+1\\mathbf\{X\}\_\{k\+1\}=\\mathbf\{X\}\_\{k\}\+\\bm\{\\Pi\}\\bm\{\\rho\}\_\{k\}/\\sigma\_\{k\+1\}\. For a single injection Eqn\. \([12](https://arxiv.org/html/2610.14277#S5.E12)\) gives𝝆0=𝜹0\\bm\{\\rho\}\_\{0\}=\\bm\{\\delta\}\_\{0\}and𝝆k=−wk−1​𝜹0\\bm\{\\rho\}\_\{k\}=\-w\_\{k\-1\}\\bm\{\\delta\}\_\{0\}fork≥1k\\geq 1, so𝐗k=gk​𝚷​𝜹0\\mathbf\{X\}\_\{k\}=g\_\{k\}\\,\\bm\{\\Pi\}\\bm\{\\delta\}\_\{0\}with

gk=1σ1−∑t=1k−1wt−1σt\+1=Tk−1σk\+∑t=0k−2Tt​\(1σt\+1−1σt\+2\),g\_\{k\}=\\frac\{1\}\{\\sigma\_\{1\}\}\-\\sum\_\{t=1\}^\{k\-1\}\\frac\{w\_\{t\-1\}\}\{\\sigma\_\{t\+1\}\}=\\frac\{T\_\{k\-1\}\}\{\\sigma\_\{k\}\}\+\\sum\_\{t=0\}^\{k\-2\}T\_\{t\}\\Bigl\(\\frac\{1\}\{\\sigma\_\{t\+1\}\}\-\\frac\{1\}\{\\sigma\_\{t\+2\}\}\\Bigr\),where the second form follows by summation by parts\. Sincegk\+1−gk=−wk−1/σk\+1≤0g\_\{k\+1\}\-g\_\{k\}=\-w\_\{k\-1\}/\\sigma\_\{k\+1\}\\leq 0, the sequence decreases towardg∞=∑t≥0Tt​\(1/σt\+1−1/σt\+2\)\>0g\_\{\\infty\}=\\sum\_\{t\\geq 0\}T\_\{t\}\(1/\\sigma\_\{t\+1\}\-1/\\sigma\_\{t\+2\}\)\>0, sogk≥g∞g\_\{k\}\\geq g\_\{\\infty\}for everykk\. Along𝐞0\\mathbf\{e\}\_\{0\}nothing is propagated, so only the tail of the injected error remains,⟨𝐞0,𝐄klin⟩=Tk−1​⟨𝐞0,𝜹0⟩\\langle\\mathbf\{e\}\_\{0\},\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{k\}\\rangle=T\_\{k\-1\}\\langle\\mathbf\{e\}\_\{0\},\\bm\{\\delta\}\_\{0\}\\rangle\.

The𝐟\\mathbf\{f\}\-component of the second\-order response at sitekkcollects two curvatures\. The quadratic term of the block gives

h2​σk2​d​‖𝚷​𝐄klin‖2​\(1\+O⁡\(q\)\)=h2​d​gk2​‖𝚷​𝜹0‖2​\(1\+O⁡\(q\)\),\\frac\{h\}\{2\\sigma\_\{k\}^\{2\}\\sqrt\{d\}\}\\,\\bigl\\\|\\bm\{\\Pi\}\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{k\}\\bigr\\\|^\{2\}\\bigl\(1\+O\(q\)\\bigr\)\\;=\\;\\frac\{h\}\{2\\sqrt\{d\}\}\\,g\_\{k\}^\{2\}\\,\\norm\{\\bm\{\\Pi\}\\injnoise\_0\}^\{2\}\\bigl\(1\+O\(q\)\\bigr\),terms of one sign carrying the same random factor‖𝚷​𝜹0‖2\\norm\{\\bm\{\\Pi\}\\injnoise\_0\}^\{2\}, of meanq2​\(d−1\)/dq^\{2\}\(d\-1\)/d\. The curvature ofn⁡\(𝐱\)n\(\\mathbf\{x\}\)in thebk​n​\(𝐱\)b\_\{k\}\\,n\(\\mathbf\{x\}\)term adds the cross term

−bkd​σk2​⟨𝐟,𝐄klin⟩​⟨𝐞0,𝐄klin⟩​\(1\+O⁡\(q\)\),\-\\frac\{b\_\{k\}\}\{d\\sigma\_\{k\}^\{2\}\}\\,\\langle\\mathbf\{f\},\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{k\}\\rangle\\langle\\mathbf\{e\}\_\{0\},\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{k\}\\rangle\\bigl\(1\+O\(q\)\\bigr\),of the same order inqqbut carrying the factorTk−1T\_\{k\-1\}, which decays geometrically inkk, and of mean zero\. In the sums below the cross terms therefore contributeO⁡\(1\)O\(1\)against≍N\\asymp\\sqrt\{N\}one\-sign terms, and the quadratic term dominates\. The remaining second\-order components of𝐫k\\mathbf\{r\}\_\{k\}carry an extra factor1/σk1/\\sigma\_\{k\}orTk−1T\_\{k\-1\}\. Those along𝐞0\\mathbf\{e\}\_\{0\}are orthogonal to𝐟\\mathbf\{f\}, the𝐟\\mathbf\{f\}\-part of those along𝚷​𝜹0\\bm\{\\Pi\}\\bm\{\\delta\}\_\{0\}is the cross term above, and neither changes the sums\. For eachNNtakeqqsmall enough that the deviation stays within its linear bound, so the truncation is never active and𝐫nτ=𝐫n\\mathbf\{r\}^\{\\tau\}\_\{n\}=\\mathbf\{r\}\_\{n\}\. This is possible because𝐄nlin\\mathbf\{E\}^\{\\mathrm\{lin\}\}\_\{n\}and the thresholdΘn\\Theta\_\{n\}both scale linearly inqqwhile the remainder is quadratic\. The propagated responses then add coherently along𝐟\\mathbf\{f\},

∑n\(𝔼​‖𝐫nτ‖2\)1/2d​σn\+1≍q¯2​N,1d​σN2​𝔼​‖∑n𝚽N,n\+1​𝐫nτ‖2≍q¯4​N\.\\sum\_\{n\}\\frac\{\(\\mathbb\{E\}\\norm\{\\mathbf\{r\}^\\tau\_n\}^\{2\}\)^\{1/2\}\}\{\\sqrt\{d\}\\,\\sigma\_\{n\+1\}\}\\;\\asymp\\;\\bar\{q\}^\{2\}\\sqrt\{N\},\\qquad\\frac\{1\}\{d\\sigma\_\{N\}^\{2\}\}\\,\\mathbb\{E\}\\Bigl\\\|\\sum\_\{n\}\\bm\{\\Phi\}\_\{N,n\+1\}\\mathbf\{r\}^\{\\tau\}\_\{n\}\\Bigr\\\|^\{2\}\\;\\asymp\\;\\bar\{q\}^\{4\}N\.The network satisfies every hypothesis of Theorem[2](https://arxiv.org/html/2610.14277#Thmtheorem2)withεk=0\\varepsilon\_\{k\}=0, so no argument from those hypotheses alone can remove theN\\sqrt\{N\}of Proposition[1](https://arxiv.org/html/2610.14277#Thmproposition1)\. ∎

### A\.7The backward source bound

On the backward pass the compensator runs the same recurrence along the backward site sequence \(Sec\.[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)\), so at gain one the correction applied at a site is again built from the errors injected at the sites visited before it, with the weights of Eqn\. \([12](https://arxiv.org/html/2610.14277#S5.E12)\)\. We keep the forward index throughout\. The backward pass visits the sites in decreasing order ofkk, so the error injected at sitessis corrected at the sitesk=s−1,s−2,…k=s\-1,s\-2,\\dots, whereσk≤σs\\sigma\_\{k\}\\leq\\sigma\_\{s\}\. What changes is the point of entry\. An error or a correction handled at sitekkreaches the gradient of the hidden state through the Jacobian of the normalization at that site \(Sec\.[3\.2](https://arxiv.org/html/2610.14277#S3.SS2)\)\. Write this operator at sitekkas𝐀k=𝐃k/σk\\mathbf\{A\}\_\{k\}=\\mathbf\{D\}\_\{k\}/\\sigma\_\{k\}with‖𝐃k‖≤γ¯\\norm\{\\mathbf\{D\}\_k\}\\leq\\bar\{\\gamma\}, whereγ¯\\bar\{\\gamma\}is a constant of the order of the largest magnitude among the normalization weights\. The error𝜹s\\bm\{\\delta\}\_\{s\}then enters through𝐀s\\mathbf\{A\}\_\{s\}at its own site, while the fractionws−1−k​𝜹sw\_\{s\-1\-k\}\\bm\{\\delta\}\_\{s\}subtracted at a later visited sitek<sk<senters through𝐀k\\mathbf\{A\}\_\{k\}\. Once the backward pass has reached sitett, the compensated partial sum is

𝐒tbwd=∑s≥t𝐁s,t​𝜹s,𝐁s,t=𝐀s−∑k=ts−1ws−1−k​𝐀k\.\\mathbf\{S\}^\{\\mathrm\{bwd\}\}\_\{t\}=\\sum\_\{s\\geq t\}\\mathbf\{B\}\_\{s,t\}\\,\\bm\{\\delta\}\_\{s\},\\qquad\\mathbf\{B\}\_\{s,t\}=\\mathbf\{A\}\_\{s\}\-\\sum\_\{k=t\}^\{s\-1\}w\_\{s\-1\-k\}\\,\\mathbf\{A\}\_\{k\}\.A correction now leaves through a different operator than the one its error entered through, so the cancellation is no longer exact, and the bound depends on how fast the operators vary along the trajectory\.

###### Proposition 3\(backward source\)\.

Suppose, in addition to R1, that the backward injections satisfy the noise model of Eqn\. \([22](https://arxiv.org/html/2610.14277#A1.E22)\) with the history taken in the order in which the backward pass visits the sites and withqqtheir bound in gradient units, and that the operators vary smoothly along the trajectory,‖𝐃s−𝐃k‖≤cD​\(s−k\)​Δmax/σs\\norm\{\\mathbf\{D\}\_s\-\\mathbf\{D\}\_k\}\\leq c\_\{D\}\\,\(s\-k\)\\sqrt\{\\Delta\_\{\\max\}\}/\\sigma\_\{s\}fork≤sk\\leq s\. Then

maxt⁡𝔼​‖𝐒tbwd‖2≤Cbwd​q2σ12,\\max\_\{t\}\\;\\mathbb\{E\}\\,\\bigl\\\|\\mathbf\{S\}^\{\\mathrm\{bwd\}\}\_\{t\}\\bigr\\\|^\{2\}\\;\\leq\\;C\_\{\\mathrm\{bwd\}\}\\;\\frac\{q^\{2\}\}\{\\sigma\_\{1\}^\{2\}\},withCbwdC\_\{\\mathrm\{bwd\}\}independent ofNN, built fromγ¯\\bar\{\\gamma\},cDc\_\{D\}, the kernel momentsμw\\mu\_\{w\}andτ2\\tau\_\{2\}, and the scale constantsσ0,σ1,Δmin,Δmax\\sigma\_\{0\},\\sigma\_\{1\},\\Delta\_\{\\min\},\\Delta\_\{\\max\}\.

###### Proof\.

Since∑rwr=1\\sum\_\{r\}w\_\{r\}=1,

𝐁s,t=𝐀s​Ts−t\+∑k=ts−1ws−1−k​\(𝐀s−𝐀k\)\.\\mathbf\{B\}\_\{s,t\}=\\mathbf\{A\}\_\{s\}\\,T\_\{s\-t\}\+\\sum\_\{k=t\}^\{s\-1\}w\_\{s\-1\-k\}\\,\(\\mathbf\{A\}\_\{s\}\-\\mathbf\{A\}\_\{k\}\)\.Fork<sk<s, R1 givesσs2−σk2≤\(s−k\)​Δmax\\sigma\_\{s\}^\{2\}\-\\sigma\_\{k\}^\{2\}\\leq\(s\-k\)\\Delta\_\{\\max\}, henceσs−σk≤\(s−k\)​Δmax/σs\\sigma\_\{s\}\-\\sigma\_\{k\}\\leq\(s\-k\)\\Delta\_\{\\max\}/\\sigma\_\{s\}, and withσk≥σ0\\sigma\_\{k\}\\geq\\sigma\_\{0\}and the smoothness hypothesis,

‖𝐀s−𝐀k‖≤‖𝐃s−𝐃k‖σs\+γ¯​σs−σkσs​σk≤CA​\(s−k\)σs2,CA=cD​Δmax\+γ¯​Δmaxσ0\.\\norm\{\\mathbf\{A\}\_s\-\\mathbf\{A\}\_k\}\\;\\leq\\;\\frac\{\\norm\{\\mathbf\{D\}\_s\-\\mathbf\{D\}\_k\}\}\{\\sigma\_\{s\}\}\+\\bar\{\\gamma\}\\,\\frac\{\\sigma\_\{s\}\-\\sigma\_\{k\}\}\{\\sigma\_\{s\}\\sigma\_\{k\}\}\\;\\leq\\;\\frac\{C\_\{\\mathrm\{A\}\}\\,\(s\-k\)\}\{\\sigma\_\{s\}^\{2\}\},\\qquad C\_\{\\mathrm\{A\}\}=c\_\{D\}\\sqrt\{\\Delta\_\{\\max\}\}\+\\frac\{\\bar\{\\gamma\}\\,\\Delta\_\{\\max\}\}\{\\sigma\_\{0\}\}\.The kernel has∑rwr​\(r\+1\)=1\+μw\\sum\_\{r\}w\_\{r\}\(r\+1\)=1\+\\mu\_\{w\}, the Frobenius norms carry a factord\\sqrt\{d\}, and Lemma[4](https://arxiv.org/html/2610.14277#Thmlemma4)applies with𝐁s,t\\mathbf\{B\}\_\{s,t\}in place of𝐊s\\mathbf\{K\}\_\{s\}and the history taken in visiting order, so

𝔼​‖𝐒tbwd‖2≤q2d​∑s≥t‖𝐁s,t‖F2\\displaystyle\\mathbb\{E\}\\,\\bigl\\\|\\mathbf\{S\}^\{\\mathrm\{bwd\}\}\_\{t\}\\bigr\\\|^\{2\}\\;\\leq\\;\\frac\{q^\{2\}\}\{d\}\\sum\_\{s\\geq t\}\\norm\{\\mathbf\{B\}\_\{s,t\}\}\_\{F\}^\{2\}≤2​q2​γ¯2​∑s≥tTs−t2σs2\+2​q2​CA2​\(1\+μw\)2​∑s≥t1σs4\\displaystyle\\;\\leq\\;2q^\{2\}\\,\\bar\{\\gamma\}^\{2\}\\sum\_\{s\\geq t\}\\frac\{T\_\{s\-t\}^\{2\}\}\{\\sigma\_\{s\}^\{2\}\}\+2q^\{2\}\\,C\_\{\\mathrm\{A\}\}^\{2\}\(1\+\\mu\_\{w\}\)^\{2\}\\sum\_\{s\\geq t\}\\frac\{1\}\{\\sigma\_\{s\}^\{4\}\}≤2​q2σ12​\[γ¯2​cΔ\+CA2​\(1\+μw\)2​cΔ′σ12\],\\displaystyle\\;\\leq\\;\\frac\{2q^\{2\}\}\{\\sigma\_\{1\}^\{2\}\}\\Bigl\[\\bar\{\\gamma\}^\{2\}\\,c\_\{\\Delta\}\+C\_\{\\mathrm\{A\}\}^\{2\}\(1\+\\mu\_\{w\}\)^\{2\}\\,\\frac\{c\_\{\\Delta\}^\{\\prime\}\}\{\\sigma\_\{1\}^\{2\}\}\\Bigr\],wherecΔ:=supt∑s≥tTs−t2​σ12/σs2c\_\{\\Delta\}:=\\sup\_\{t\}\\sum\_\{s\\geq t\}T\_\{s\-t\}^\{2\}\\,\\sigma\_\{1\}^\{2\}/\\sigma\_\{s\}^\{2\}andcΔ′:=∑sσ14/σs4c\_\{\\Delta\}^\{\\prime\}:=\\sum\_\{s\}\\sigma\_\{1\}^\{4\}/\\sigma\_\{s\}^\{4\}\. Both are finite with explicit bounds:σs≥σ0\\sigma\_\{s\}\\geq\\sigma\_\{0\}givescΔ≤\(σ12/σ02\)​τ2c\_\{\\Delta\}\\leq\(\\sigma\_\{1\}^\{2\}/\\sigma\_\{0\}^\{2\}\)\\,\\tau\_\{2\}, andσs4≥\(σ02\+s​Δmin\)2\\sigma\_\{s\}^\{4\}\\geq\(\\sigma\_\{0\}^\{2\}\+s\\Delta\_\{\\min\}\)^\{2\}givescΔ′≤σ14/σ04\+σ14/\(Δmin​σ02\)c\_\{\\Delta\}^\{\\prime\}\\leq\\sigma\_\{1\}^\{4\}/\\sigma\_\{0\}^\{4\}\+\\sigma\_\{1\}^\{4\}/\(\\Delta\_\{\\min\}\\sigma\_\{0\}^\{2\}\)by comparison with the integral\. Neither depends onNN\. ∎

The backward pass therefore has the analogous source guarantee, in mean square and under a smoothness condition on the normalization Jacobians\. The guarantee follows from the same cancellation mechanism, with constants degraded by the operator mismatch between a site and the sites that correct it\.

## Appendix BTraining Configuration

The training runs of Sec\.[7\.3](https://arxiv.org/html/2610.14277#S7.SS3)use the NeMo Qwen3\-0\.6B configuration, trained from scratch on Dolma C4: 28 layers, hidden size 1,024, FFN size 3,072, 16 attention heads with 8 query groups, RMSNorm, QK layer normalization, RoPE base10610^\{6\}, and no dropout\. The 151,669\-token vocabulary of the Qwen tokenizer is padded to 151,936\. Sequences have 2,048 tokens\. The main run uses a micro batch of 24 and a global batch of 384 sequences, 786,432 tokens per step, for 12,716 steps, about 10\.0B tokens in total\. Parallelism isP=4P=4without pipeline, context, data, or sequence parallelism\. Training uses BF16 mixed precision with FP32 master weights, gradients, and optimizer states\. The optimizer is AdamW withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95,ϵ=10−5\\epsilon=10^\{\-5\}, weight decay0\.10\.1, and gradient clipping at1\.01\.0\. The learning rate warms up over 100 steps to its peak,3\.0×10−43\.0\\times 10^\{\-4\}for the 16\-bit run and3\.5×10−43\.5\\times 10^\{\-4\}for the 4\-bit run, and follows a cosine decay to3×10−53\\times 10^\{\-5\}\. The learning\-rate ablation of Table[7](https://arxiv.org/html/2610.14277#S7.T7)b uses the same configuration with a global batch of 32 sequences for 4,096 steps, about 0\.27B tokens\. The reported loss is the training cross\-entropy averaged over tokens\.

## Appendix CImplementation Details

The pipeline of Sec\.[4\.2](https://arxiv.org/html/2610.14277#S4.SS2)keeps four stages busy at once on each of thePPGPUs, and a tile has to move from one stage to the next without the host taking part\. Three things have to be in place beforehand\. Each GPU has to know where to write inside its peers\. The stages within a GPU have to overlap across tiles while the stages of any one tile stay in order\. Finally, a GPU has to detect when the data it needs from a peer has arrived\. The rest of this appendix describes these in turn, then the cache that keeps their setup out of the per\-call cost, and finally the exactness of the backward rescaling of Sec\.[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)\.

GPU 00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0GPU 10\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0GPU 20\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0GPU 30\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0segment 0segment 1segment 2segment 3each GPU quantizes the three segments it does not ownall\-to\-allΣ\\Sigma0\.000\.000\.000\.00Σ\\Sigma0\.000\.000\.000\.00Σ\\Sigma0\.000\.000\.000\.00Σ\\Sigma0\.000\.000\.000\.00the owner adds four copiesin high precisionthe solid copy is the owner’s own,the dashed ones crossed a linkall\-gather0\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.00Echo outputexact sumerror𝜹\\bm\{\\delta\}the same codes reach every GPU, so this row is what every GPU holdsFigure 4:Worked example atP=4P=4\.The quantizer is Eqn\. \([8](https://arxiv.org/html/2610.14277#S4.E8)\) withb=4b=4and a group equal to the two values of one segment\. Top left: each GPU quantizes the three segments it does not own and keeps its own segment in full precision\. Top right: the owner adds the four copies in high precision, three of them dequantized frombb\-bit codes and one its own\. Bottom: the all\-gather sends one quantized copy of each segment sum, so every GPU holds the same values\. The last row is the error this all\-reduce injects, the𝜹k\\bm\{\\delta\}\_\{k\}of Fig\.[1](https://arxiv.org/html/2610.14277#S3.F1.fig1)\. Inputs are shown to one decimal and sums to two\.Workspace layout\.Before a GPU can write into a peer, it must hold that peer’s addresses\. A call uses three kinds of memory on every GPU: a buffer that receives thebb\-bit codes its peers send, a buffer that receives the scale and zero\-point that decode those codes \(Sec\.[4\.3](https://arxiv.org/html/2610.14277#S4.SS3)\), and an array of signal words that peers write to announce that the data has arrived\. Each GPU allocates its own three and shares their addresses with the others when the tensor\-parallel group is created, which gives one table of3​P\+13P\{\+\}1addresses: thePPcode buffers, thePPparameter buffers, thePPsignal arrays, and one flag base\. The table is built once and stays fixed, and every later call reads the same entries, so a kernel reaches any peer by indexing the table with its rank\. The compensator state𝐦\(1\),𝐦\(2\)\\mathbf\{m\}^\{\(1\)\},\\mathbf\{m\}^\{\(2\)\}of Sec\.[5\.3](https://arxiv.org/html/2610.14277#S5.SS3)is kept out of this table and owned by the compensation path, so the interface of the collective is the same with and without compensation\.

Stream and event schedule\.Push, reduce, gather, and tail each get their own CUDA stream\. Work issued to different streams runs concurrently, so putting the stages on separate streams lets push quantize tile33while gather is still sending tile11\. Streams do not enforce the order within a tile, and tile22must not be reduced before it has been pushed\. CUDA events enforce this order\. When push finishes tile22it records an event, and reduce waits on that event before it starts tile22\. Since a stage waits only for the stage directly before it, the event of a tile is no longer needed once the next stage has started on the tile, and two event slots used in turn suffice for the whole pipeline \(Fig\.[2](https://arxiv.org/html/2610.14277#S4.F2)\(b\)\)\. The caller is synchronized at the two ends\. It records an event before the first push, so the pipeline starts only once the input is ready, and waits on one after the last tail, so its next kernel starts only once the output is complete\. The compensator update is issued on a fifth stream after the last tail and runs alongside the caller’s next matrix multiplication, and the entry stage of the next call waits on an event recorded at the end of the update\. The host launches the kernels and returns\. It performs no synchronization while the tiles are in flight\.

Flag values\.Events do not cross GPUs\. A stage that needs a segment from a peer therefore polls the flag of Sec\.[4\.2](https://arxiv.org/html/2610.14277#S4.SS2)and waits until it reads the value that stands for the data it needs\. Each stage publishes a value one higher than the stage before it\. Push publishes the base value, reduce publishes the base plus one, and gather publishes the base plus two, which is the value tail polls for\. The host raises the base by three at every call, so no value is used twice in the life of the group, and no flag needs to be cleared\. A consumer that arrives early reads a smaller number and keeps polling, and a flag still holding a value from the previous call is below anything the current call polls for, so it cannot release a consumer by mistake\. No slot is cleared between calls and no CPU round trip takes place on the readiness path\.

Schedule cache\.The tiling, the launch order, and the streams and events above depend only on the shape of the call, so they are computed once and reused, with a cache keyed by the data type, bit width, policy, group size, and segment size\. The cache stores the tiling and launch table together with the streams and events for each device\. Only the first call for a new shape, or the first after a rebuild, pays this build and warm\-up cost, which the latency measurements of Sec\.[7](https://arxiv.org/html/2610.14277#S7)exclude, so all reported numbers are steady state\.

Backward rescaling\.Section[5\.4](https://arxiv.org/html/2610.14277#S5.SS4)multiplies the gradient by a shared power\-of\-two factor before the quantized all\-reduce and divides the result by the same factor after it\. Three choices make the scaling exact\. The factor is a power of two, so multiplying and dividing shift only the floating\-point exponent and add no rounding of their own\. Its exponent is derived from the largest gradient magnitude across the tensor\-parallel group, obtained with amax\\maxall\-reduce of a single scalar, so every GPU applies the same factor and the all\-reduce computes the same sum, only rescaled\. Finally, the two steps of the compensator run outside the scaling\. The compensator forms𝐱′=𝐱\+𝐜\\mathbf\{x\}^\{\\prime\}=\\mathbf\{x\}\+\\mathbf\{c\}before the multiplication and𝐮=𝐱′−𝐲^/P\\mathbf\{u\}=\\mathbf\{x\}^\{\\prime\}\-\\hat\{\\mathbf\{y\}\}/Pafter the division, so𝐦\(1\)\\mathbf\{m\}^\{\(1\)\},𝐦\(2\)\\mathbf\{m\}^\{\(2\)\}stay in original gradient units and the analysis of Secs\.[5\.1](https://arxiv.org/html/2610.14277#S5.SS1)and[6](https://arxiv.org/html/2610.14277#S6)applies unchanged\.