Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability

# Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
Source: [https://arxiv.org/html/2606.28116](https://arxiv.org/html/2606.28116)
Ruixuan Huang1, Yipei Wang2, Wenyi Fang2, Hantao Huang3, Yifan Huang3, Ansheng You3, Zhenxing Zhang3, Shuai Wang1, Fan Wu2, Yang Zheng2 1HKUST2Huawei3Independent Researcher

###### Abstract

Frontier large language model training consumes massive accelerator fleets and long wall\-clock computation, making stability failures costly when they occur\. After a numerical or a hyperparameter fault has already destabilized the training dynamics, it may continue for thousands of steps while loss and gradient norms still appear normal\. We study mechanism\-driven detection of training instability by deriving internal monitors from the functional role of each critical module and from the earliest computational sites where failures are expected to produce measurable signatures\. For low\-precision flash attention, we monitor the spectral entropy of a QK bilinear decomposition, whose first\-order term becomes abnormal before the loss fully collapses\. For MoE routers, we derive indicators from their role in expert selection\. Our fault\-injection experiments on low\-precision attention, large learning\-rate, and combined faults show that these signals provide distinct signatures for different failures, triggering thousands of steps before loss divergence\.

## 1Introduction

Frontier large language model \(LLM\) training typically occupies thousands of accelerators for weeks to months\(Chowdheryet al\.,[2022](https://arxiv.org/html/2606.28116#bib.bib19); Smithet al\.,[2022](https://arxiv.org/html/2606.28116#bib.bib20)\)\. Parameter counts now reach hundreds of billions to over a trillion\(Yanget al\.,[2025](https://arxiv.org/html/2606.28116#bib.bib23); Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17); DeepSeek\-AI,[2026](https://arxiv.org/html/2606.28116#bib.bib26)\); pre\-training corpora span tens of trillions of tokens\(GLM\-4\.5 Team,[2025](https://arxiv.org/html/2606.28116#bib.bib24); Meituan LongCat Team,[2025](https://arxiv.org/html/2606.28116#bib.bib41); Qwen Team,[2026](https://arxiv.org/html/2606.28116#bib.bib25)\)\. DeepSeek\-V3 gives an explicit cost accounting, where 14\.8T pre\-training tokens required 2\.788M H800 GPU\-hours and a reported $5\.576M in direct rental cost, excluding prior research and ablation experiments\(DeepSeek\-AI,[2024](https://arxiv.org/html/2606.28116#bib.bib21)\)\. At this scale, training stability has become an engineering concern of the training system\. GLM\-130B describes unexpected 100B\-scale training challenges, especially loss spikes and divergence\(Zenget al\.,[2023](https://arxiv.org/html/2606.28116#bib.bib27)\); DeepSeek\-V3 highlights FP8 mixed\-precision training and explicitly reports no irrecoverable loss spikes or rollbacks\(DeepSeek\-AI,[2024](https://arxiv.org/html/2606.28116#bib.bib21)\); and Kimi K2 introduces MuonClip with QK\-clip to address training instability and reports 15\.5T\-token pre\-training with zero loss spike\(Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17)\)\.

The risk of training instability often comes from two sources\. The first is numerical precision error\. For example, flash attention \(FA\) exhibits substantially larger BF16 numeric deviation than baseline attention in isolated forward passes\(Goldenet al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib16)\), and low\-precision FA can corrupt weight updates through biased rounding errors and gradually derail training dynamics\(Qiu and Yao,[2026](https://arxiv.org/html/2606.28116#bib.bib1)\)\. The second is hyperparameter interaction, such as the coupling among global batch size \(GBS\), learning rate schedule and MoE auxiliary loss\. However, before the global symptoms appear, a training run may already have entered an unstable state in its weights or optimizers, while training silently continues for thousands of steps before the symptoms become visible\. Exhaustive ablation only increases these sunk costs\. A useful monitor should therefore identify which subsystem has been destabilized before loss divergence appears\.

Current training stability monitoring mainly relies on global training curves and symptom\-level indicators\. Loss, gradient norms, and weight norms are the most delayed indicators\. Once a loss spike or divergence appears, the fault may already have been written into weights or optimizer state\. Attention entropy, maximum attention logit, and spectral indicators further characterize attention\-side instability symptoms\(Zhaiet al\.,[2023](https://arxiv.org/html/2606.28116#bib.bib14); Takaseet al\.,[2025](https://arxiv.org/html/2606.28116#bib.bib15); Goldenet al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib16); Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17)\)\. Edge\-of\-stability analysis explains high\-level loss dynamics, but does not identify which module failed first\(Cohenet al\.,[2021](https://arxiv.org/html/2606.28116#bib.bib32)\)\. Max\-logit signals are difficult to expose in production FA because they require kernel modification and recomputation\(Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17)\)\. Hessian or curvature diagnostics can provide finer geometric information, but are too expensive to run as routine online checks at frontier scale\(Yaoet al\.,[2020](https://arxiv.org/html/2606.28116#bib.bib49); Kalraet al\.,[2026](https://arxiv.org/html/2606.28116#bib.bib50)\)\.

![Refer to caption](https://arxiv.org/html/2606.28116v1/hero/p1372b45b_a_lm_loss_first25k.png)\(a\)LM loss
![Refer to caption](https://arxiv.org/html/2606.28116v1/hero/p1372b45b_b_qkv_weight_norm_first25k.png)\(b\)\|WQ​K​V\|F\|W\_\{QKV\}\|\_\{F\}
![Refer to caption](https://arxiv.org/html/2606.28116v1/hero/p1372b45b_c_grad_norm_first25k.png)\(c\)Gradient norm
![Refer to caption](https://arxiv.org/html/2606.28116v1/hero/p1372b45b_d_qkv_weight_dw_singular_spectrum_first25k.png)\(d\)Δ​WQ​K​V\\Delta W\_\{QKV\}spectrum

Figure 1:Monitoring signals over the first 25,000 steps of a training run\. \(a\)–\(c\) are standard symptom\-level indicators: LM loss, QKV weight norm, and gradient norm\. \(d\) shows an internal update monitor used in this paper\. In \(b\) and \(d\), the solid curve is the layer\-wise average and the shaded band spans the 10th–90th percentile across all layers\.Our core idea is mechanism\-driven monitoring\. For each critical module, we ask what is this module supposed to compute, and where would a malfunction first leave an attributable trace\. We apply this principle to two modules\. For low\-precision FA, we monitor weight updates, where low\-precision backward errors first enter the model state \(Section[3](https://arxiv.org/html/2606.28116#S3)\)\. We decompose the two\-snapshot increment of the QK operator and monitor the spectral entropy ofΔ​W\\Delta W\. Figure[1](https://arxiv.org/html/2606.28116#S1.F1)shows an example under low\-precision FA training, where theΔ​W\\Delta Wspectrum collapses thousands of steps earlier than loss, gradient norms and weight norms\.

For MoE routers, the intended computation is discriminative and non\-collapsed expert selection \(Section[4](https://arxiv.org/html/2606.28116#S4)\)\. Therefore, we monitor router weight similarity and centered conditioning as weight indicators that characterize whether the effective expert\-selection axes become redundant\. For the behavior, we monitor per\-token routing entropy\. It reads the full softmax distribution and can therefore capture the collapse of routing behavior before downstream discrete quantities such as top\-kkcounts, capacity overflow, or load\-balance statistics change\.

We further analyze how learning rate and GBS interact through stable\-winner reinforcement\. A larger learning rate amplifies coherent margin growth, while a smaller batch size increases margin noise; both can reduce router entropy and accelerate expert\-use collapse\. Our fault\-monitoring experiments demonstrate the separate roles of the two monitor families, and combined faults inherit both signatures without obscuring their attribution\.

## 2Related Work

#### Training\-stability monitors\.

Existing work has proposed monitors to detect or mitigate training instability\. On the attention side, max\-logit clipping, introduced for ViT\-22B\(Dehghaniet al\.,[2023](https://arxiv.org/html/2606.28116#bib.bib12)\)and studied through small\-scale Transformer proxies\(Wortsmanet al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib13)\), catches softmax explosion directly\. Kimi K2 adds QK\-clipping and per\-head MuonClip\(Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17)\)\. Other approaches target attention\-entropy collapse viaσ\\sigmaReparam\(Zhaiet al\.,[2023](https://arxiv.org/html/2606.28116#bib.bib14)\), loss spikes via spectral\-norm control\(Takaseet al\.,[2025](https://arxiv.org/html/2606.28116#bib.bib15)\), or Flash\-Attention output distributions\(Goldenet al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib16)\)\. On the MoE router side, LongCat\-Flash monitors the average cosine similarity among expert router weights and the gradient\-norm ratio between the load\-balancing objective and the language\-modeling objective on average expert probabilities\(Meituan LongCat Team,[2025](https://arxiv.org/html/2606.28116#bib.bib41)\)\.

#### Attention Circuit Analyses\.

The attention QK\-circuit has been analyzed primarily as a static object\.Baoet al\.\([2024](https://arxiv.org/html/2606.28116#bib.bib3)\)characterize attention localization through the eigenspectrum variance ofWq⊤​WkW\_\{q\}^\{\\top\}W\_\{k\}, andPanet al\.\([2024](https://arxiv.org/html/2606.28116#bib.bib4)\)examine singular\-vector correspondence on the QK kernel for vision transformers\. Researches show that attention maps and QK kernels can exhibit strong low\-rank structure\.Bhojanapalliet al\.\([2020](https://arxiv.org/html/2606.28116#bib.bib5)\)study the rank\-deficiency bottleneck ofWq​Wk⊤W\_\{q\}W\_\{k\}^\{\\top\}at smalldkd\_\{k\}, whileDonget al\.\([2021](https://arxiv.org/html/2606.28116#bib.bib11)\)prove doubly\-exponential rank collapse in pure self\-attention with depth\. Recent works also show that weight updates contain informative low\-rank structure\. LoRA\(Huet al\.,[2022](https://arxiv.org/html/2606.28116#bib.bib28)\), GaLore\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib29)\), and the Muon optimizer family\(Liuet al\.,[2025](https://arxiv.org/html/2606.28116#bib.bib30)\)exploit low\-rank or spectral structure in updates for parameter\-efficient adaptation or optimization, andYuniset al\.\([2024](https://arxiv.org/html/2606.28116#bib.bib31)\)survey spectral evolution of weights as a window onto training dynamics\. Mechanistically,Qiu and Yao \([2026](https://arxiv.org/html/2606.28116#bib.bib1)\)identify low\-precision FA failure as biased rounding accumulating into similar low\-rank update directions\. These works motivate usingΔ​W\\Delta Witself as an analysis object\.

## 3Attention Updates Monitoring

Flash Attention\(Goldenet al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib16)\)fuses the softmax\-scaledQ​K⊤QK^\{\\top\}computation in on\-chip memory and not materializes the fullN×NN\\times Nlogit matrix\. This brings dramatic memory and throughput gains, and modern LLM training and inference now rely on it as the prevalent attention implementation\. However, FA is reported as a source of training instability, where low\-precision arithmetic in its backward pass can deposit persistent, biased errors into weight updates\(Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17); Qiu and Yao,[2026](https://arxiv.org/html/2606.28116#bib.bib1)\)\. Moreover, the fused implementation blocks the most natural symptom\-level monitor used at scale, namely tracking the maximum attention logit \(max\-logit\) to catch softmax explosion\(Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17)\)\. Reading max\-logits out of a production FA kernel requires either invasive kernel modification or a recompute pass, both unacceptable in a large training run\.

Runtime training monitors are viable only for quantities that require no kernel modification or activation recomputation\. For instance, use gradients or weightsWWdirectly\(Fanget al\.,[2023](https://arxiv.org/html/2606.28116#bib.bib51)\)\. However, in practice, gradient\-based indicators are dominated by mini\-batch noise across consecutive steps, andWW\-based indicators are diluted by initialization energy\. The natural remaining target is the parameter updateΔ​W\\Delta Witself, which is exactly the level at which low\-precision FA faults have been shown to deposit their persistent damage\(Qiu and Yao,[2026](https://arxiv.org/html/2606.28116#bib.bib1)\)\.

In fact, modern LLMs are severely over\-parameterized, with intrinsic dimension far below parameter count\(Jacotet al\.,[2018](https://arxiv.org/html/2606.28116#bib.bib34); Chizatet al\.,[2019](https://arxiv.org/html/2606.28116#bib.bib35); Leeet al\.,[2019](https://arxiv.org/html/2606.28116#bib.bib36)\), so a fault that perturbs along currently low\-impact directions is absorbed silently—loss lags not by how long corruption takes to occur, but by how long it takes the corrupted directions to become task\-loaded\. Among parameter\-side quantities, the updateΔ​W\\Delta Wis preferable to the raw weightWWon signal\-to\-noise grounds: singular\- value statistics ofWtW\_\{t\}are diluted by initialization energy \(Appendix[A](https://arxiv.org/html/2606.28116#A1)\), whereas the incrementΔ​Wt,δ=Wt−Wt−δ\\Delta W\_\{t,\\delta\}=W\_\{t\}\-W\_\{t\-\\delta\}removes this background and exposes the update geometry directly\.

### 3\.1The Intrinsic Low\-Precision Issue of Flash Attention

As low\-precision arithmetic becomes standard in LLM training, Qiu and Yao\(Qiu and Yao,[2026](https://arxiv.org/html/2606.28116#bib.bib1)\)show that low\-precision FA induces biased scalar errors inδ=rowsum⁡\(d​O⊙O\)\\delta=\\operatorname\{rowsum\}\(dO\\odot O\), and that these biased scalars multiply structurally coherent rank\-one update atoms\. We adopt their per\-step source model as the basis for update\-side monitoring\. Following and simplifying their notation, letX∈ℝN×dX\\in\\mathbb\{R\}^\{N\\times d\}be the hidden\-state matrix entering the query projection,K=X​WkK=XW\_\{k\}, andP=softmax⁡\(Q​K⊤/dk\)P=\\operatorname\{softmax\}\(QK^\{\\top\}/\\sqrt\{d\_\{k\}\}\)be the attention probability matrix\. For a token\-step samplejj, letXjX\_\{j\}and\(P​K\)j\(PK\)\_\{j\}denote the corresponding rows\. Following their mechanism, index token\-step samples byjjand write the update\-side source asej​Rje\_\{j\}R\_\{j\}, whereeje\_\{j\}is the biased scalar error induced throughδ\\deltaandRj=Xj⊤​\(P​K\)jR\_\{j\}=X\_\{j\}^\{\\top\}\(PK\)\_\{j\}is the associated rank\-one update atom, up to the attention scale and sign\. Hereeje\_\{j\}corresponds to Qiu and Yao’s biased coefficient\(δl​p−δh​p\)​\[T\]\(\\delta\_\{lp\}\-\\delta\_\{hp\}\)\[T\], andRjR\_\{j\}to their common low\-rank error direction𝐑≈\(𝐏𝐊\)​\[T\]⊤​X​\[T\]\\mathbf\{R\}\\approx\(\\mathbf\{PK\}\)\[T\]^\{\\top\}X\[T\]\(their Claim 2, Equation 3\)\. The monitoring premise is that biased scalar coefficients and coherent atoms produce a low\-rank mean component in accumulated update windows:

> Observation 1 \(accumulation consequence of Qiu–Yao\)\.Index token\-step samples byjjand writeRj=Xj⊤​\(P​K\)j∈ℝd×dkR\_\{j\}=X\_\{j\}^\{\\top\}\(PK\)\_\{j\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\}\. IfM=𝔼​\[ej​Rj\]M=\\mathbb\{E\}\[e\_\{j\}R\_\{j\}\]has effective rankr≪dkr\\ll d\_\{k\}\(this is the substantive premise; sinceM=𝔼​\[ej​Rj\]∈ℝd×dkM=\\mathbb\{E\}\[e\_\{j\}R\_\{j\}\]\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\},rank⁡\(M\)≤dk\\operatorname\{rank\}\(M\)\\leq d\_\{k\}automatically, supported by the empirical finding inQiu and Yao \([2026](https://arxiv.org/html/2606.28116#bib.bib1)\)that the atomsRjR\_\{j\}share common column structure across tokens and training steps; see their Figure 4\) and the centered fluctuationsej​Rj−Me\_\{j\}R\_\{j\}\-Mare independent \(or martingale\-difference\) with bounded second moment, then overnnsamples ∑j=1nej​Rj=n​M\+Op​\(n\)\.\\sum\_\{j=1\}^\{n\}e\_\{j\}R\_\{j\}=nM\+O\_\{p\}\(\\sqrt\{n\}\)\.\(1\)The coherent low\-rank component grows linearly innn, while zero\-mean residuals grow sublinearly\. Oncen​‖M‖2n\\\|M\\\|\_\{2\}dominates the residual, the singular spectrum of the accumulated update is controlled byMM\.

Observation 1 is a concentration restatement of Qiu–Yao’s accumulation mechanism for the windowed setting: it turns their per\-step source model into a prediction that accumulatedΔ​W\\Delta Wspectra should develop a low\-rank component\. Its short proof and a biased\-rounding perturbation note are in Appendix[B](https://arxiv.org/html/2606.28116#A2)\. Observation 1 predicts that spectral concentration will eventually emerge but does not predict the detection\-onset step; see Section[6](https://arxiv.org/html/2606.28116#S6)for the quantitative gap\.

### 3\.2Δ​W\\Delta WSpectral Indicators

LetΔ​W=Wt−Wt−δ\\Delta W=W\_\{t\}\-W\_\{t\-\\delta\}for sampling intervalδ\\delta\. The structural state ofΔ​W\\Delta Wcan be summarized by various mathematical quantities\. Given its singular valuesσ1≥⋯≥σr\\sigma\_\{1\}\\geq\\cdots\\geq\\sigma\_\{r\}, the*stable rank*srank​\(Δ​W\)=‖Δ​W‖F2/‖Δ​W‖22\\mathrm\{srank\}\(\\Delta W\)=\\\|\\Delta W\\\|\_\{F\}^\{2\}/\\\|\\Delta W\\\|\_\{2\}^\{2\}\(Ipsen and Saibaba,[2024](https://arxiv.org/html/2606.28116#bib.bib33); Roy and Vetterli,[2007](https://arxiv.org/html/2606.28116#bib.bib43)\)measures the ratio of the squared Frobenius norm to the squared spectral norm, which is the inverse of how much the top\-1 singular value dominates the spectrum\. However, this metric loses information about the rest of the spectrum, and thus lacks interpretability – reaching full stable rank requires all singular values to be equal, which is not the case in practice\. On the other hand, effective rank𝒮α​\(Δ​W\)=exp⁡\(−∑ipi​log⁡pi\)\\mathcal\{S\}\_\{\\alpha\}\(\\Delta W\)=\\exp\\bigl\(\-\\sum\_\{i\}p\_\{i\}\\log p\_\{i\}\\bigr\)withpi=σiα/∑jσjαp\_\{i\}=\\sigma\_\{i\}^\{\\alpha\}/\\sum\_\{j\}\\sigma\_\{j\}^\{\\alpha\}is another way of evaluating the state of the spectrum\. Empirically, to balance sensitivity and noise, we useα=2\\alpha=2, which is also known as*singular spectrum*\(Alteret al\.,[2000](https://arxiv.org/html/2606.28116#bib.bib44)\)\.

![Refer to caption](https://arxiv.org/html/2606.28116v1/FA/p1372b45b_s3_w_all_metrics_first25k.png)\(a\)Monitored metrics of the weight matrixWW, including \(1\) norm, \(2\) stable rank, \(3\) singular spectrum
![Refer to caption](https://arxiv.org/html/2606.28116v1/FA/p1372b45b_s3_dw_all_metrics_first25k.png)\(b\)Monitored metrics of the weight updateΔ​W\\Delta W, including \(1\) norm, \(2\) stable rank, \(3\) singular spectrum

Figure 2:Weight\-side spectral monitors under the low\-precision FA fault, over the first25,00025\{,\}000steps\. \(top\) Metrics of the weight matrixWWand \(bottom\) metrics of the weight incrementΔ​W=Wt−Wt−δ\\Delta W=W\_\{t\}\-W\_\{t\-\\delta\}\. The rawWWstatistics are diluted by initialization energy and reveal little, whereas theΔ​W\\Delta Wincrement exposes the update geometry\.
### 3\.3TheΔ​W\\Delta WMonitor in Practice

Following the low\-precision FA mechanism ofQiu and Yao \([2026](https://arxiv.org/html/2606.28116#bib.bib1)\), we compare the training of LLMs between baseline and a biased low\-precision fault injection described in Appendix[B](https://arxiv.org/html/2606.28116#A2)\. As shown in Figure[1\(a\)](https://arxiv.org/html/2606.28116#S1.F1.sf1), the loss curve of the low\-precision run diverges at∼22,000\\sim 22\{,\}000, while the baseline run remains stable\. Traditionally, LLM training practitioners monitor weight\-related metrics such as‖W‖F\\\|W\\\|\_\{F\}andstable​\_​rank⁡\(W\)\\operatorname\{stable\\\_rank\}\(W\), etc\. to detect potential instability\. However, Figure[2\(a\)](https://arxiv.org/html/2606.28116#S3.F2.sf1)shows that because of the lazy regime, analyzing matrix properties ofWWyields limited insight into the instability of the low\-precision run\. We stress that the fault certificate is the*deviation*of theΔ​W\\Delta Wspectrum from the healthy baseline trajectory–not low\-rankness per se–since healthy updates already carry low\-rank structure that LoRA, GaLore, and the Muon family exploit; this is why we compare the baseline and fault runs rather than reading low rank off a single trace\.

The weight metrics forΔ​W\\Delta Ware plotted in Figure[2\(b\)](https://arxiv.org/html/2606.28116#S3.F2.sf2)\. The singular spectrum ofΔ​W\\Delta Wshows observable spectrum collapse at10,000∼14,00010\{,\}000\\sim 14\{,\}000, thousands of steps before the loss diverges\. The stable rank ofΔ​W\\Delta Wdoes show some early signals, but the instability nature of stable rank adds noise to the signal\. Apart from the singular\-value\-based metrics, the update norm‖Δ​W‖F\\\|\\Delta W\\\|\_\{F\}does not show a clear signal of instability until the loss diverges\. This suggests that the low\-rank structure ofΔ​W\\Delta Wis not explained by a massive energy blow\-up or a few overwhelmingly large update entries, but rather a global low\-rank structure\. Overall, the singular spectrum ofΔ​W\\Delta Wis a more sensitive and explainable metric for detecting early signs of instability in the low\-precision FA module\.

### 3\.4The Bilinear DecompositionΔ1=Δ2\+Δ3\\Delta\_\{1\}=\\Delta\_\{2\}\+\\Delta\_\{3\}

TheΔ​W\\Delta Wmonitor of the previous section treatsWqW\_\{q\}andWkW\_\{k\}as independent matrices and computes spectral metrics onΔ​Wq\\Delta W\_\{q\},Δ​Wk\\Delta W\_\{k\}, or their concatenation\[Δ​Wq,Δ​Wk\]\[\\Delta W\_\{q\},\\Delta W\_\{k\}\]\. However, in FA, the attention scoreQ​K⊤=X​\(Wq​Wk⊤\)​X⊤QK^\{\\top\}=X\(W\_\{q\}W\_\{k\}^\{\\top\}\)X^\{\\top\}depends onWqW\_\{q\}andWkW\_\{k\}only through the*bilinear form*F​\(Wq,Wk\)=Wq​Wk⊤F\(W\_\{q\},W\_\{k\}\)=W\_\{q\}W\_\{k\}^\{\\top\}, so monitoring the factors separately can miss correlated drifts that cancel or amplify in the product\. A natural alternative is to trackFFitself, but direct spectral analysis ofWq​Wk⊤W\_\{q\}W\_\{k\}^\{\\top\}across snapshots offers limited discriminability: slow secular trends dominate, and the signal of interest is buried\. This motivates decomposing the*increment*Δ1=Ft−Ft−δ\\Delta\_\{1\}=F\_\{t\}\-F\_\{t\-\\delta\}into components with distinct physical and spectral signatures\. This increment ofFFadmits an exact decomposition into a first\-order termΔ2\\Delta\_\{2\}and a second\-order termΔ3\\Delta\_\{3\}\.

> Proposition 2 \(bilinear decomposition\)\.ForWq,t,Wk,tW\_\{q,t\},W\_\{k,t\}at two time points andΔ​Wq=Wq,t−Wq,t−δ\\Delta W\_\{q\}=W\_\{q,t\}\-W\_\{q,t\-\\delta\},Δ​Wk=Wk,t−Wk,t−δ\\Delta W\_\{k\}=W\_\{k,t\}\-W\_\{k,t\-\\delta\}, with the un\-subscripted factors evaluated at the base pointt−δt\-\\delta\(i\.e\.Wq:=Wq,t−δW\_\{q\}:=W\_\{q,t\-\\delta\}andWk:=Wk,t−δW\_\{k\}:=W\_\{k,t\-\\delta\}\), Δ1:=Wq,t​Wk,t⊤−Wq,t−δ​Wk,t−δ⊤=Δ2\+Δ3,\\Delta\_\{1\}:=W\_\{q,t\}W\_\{k,t\}^\{\\top\}\-W\_\{q,t\-\\delta\}W\_\{k,t\-\\delta\}^\{\\top\}=\\Delta\_\{2\}\+\\Delta\_\{3\},\(2\)whereΔ2=Δ​Wq​Wk⊤\+Wq​Δ​Wk⊤\\Delta\_\{2\}=\\Delta W\_\{q\}W\_\{k\}^\{\\top\}\+W\_\{q\}\\Delta W\_\{k\}^\{\\top\}andΔ3=Δ​Wq​Δ​Wk⊤\\Delta\_\{3\}=\\Delta W\_\{q\}\\Delta W\_\{k\}^\{\\top\}\.

This follows immediately from the bilinearity ofFF\. In particular,D​F​\[\(Δ​Wq,Δ​Wk\)\]=Δ2DF\[\(\\Delta W\_\{q\},\\Delta W\_\{k\}\)\]=\\Delta\_\{2\}and12​D2​F​\[\(Δ​Wq,Δ​Wk\)⊗2\]=Δ3\\tfrac\{1\}\{2\}D^\{2\}F\[\(\\Delta W\_\{q\},\\Delta W\_\{k\}\)^\{\\otimes 2\}\]=\\Delta\_\{3\}, with all higher derivatives vanishing identically\.

![Refer to caption](https://arxiv.org/html/2606.28116v1/FA/p1372b45b_s3_d1_all_metrics_first25k.png)\(a\)Δ1\\Delta\_\{1\}: exact QK\-product increment
![Refer to caption](https://arxiv.org/html/2606.28116v1/FA/p1372b45b_s3_d2_all_metrics_first25k.png)\(b\)Δ2\\Delta\_\{2\}: first\-order QK\-product increment
![Refer to caption](https://arxiv.org/html/2606.28116v1/FA/p1372b45b_s3_d3_all_metrics_first25k.png)\(c\)Δ3\\Delta\_\{3\}: second\-order Q/K update interaction

Figure 3:QK\-product increment monitors under the low\-precision FA fault\.Δ1\\Delta\_\{1\}is the exact increment ofWq​Wk⊤W\_\{q\}W\_\{k\}^\{\\top\},Δ2\\Delta\_\{2\}is its first\-order term, andΔ3\\Delta\_\{3\}is the second\-order interaction betweenΔ​Wq\\Delta W\_\{q\}andΔ​Wk\\Delta W\_\{k\}\.#### Magnitude regime\.

In the early\-to\-mid training regime where‖W‖F≫‖Δ​W‖F\\\|W\\\|\_\{F\}\\gg\\\|\\Delta W\\\|\_\{F\}, and absent cancellation between the two first\-order termsΔ​Wq​Wk⊤\\Delta W\_\{q\}W\_\{k\}^\{\\top\}andWq​Δ​Wk⊤W\_\{q\}\\Delta W\_\{k\}^\{\\top\}, we have‖Δ2‖F≫‖Δ3‖F\\\|\\Delta\_\{2\}\\\|\_\{F\}\\gg\\\|\\Delta\_\{3\}\\\|\_\{F\}by a factor of order‖W‖F/‖Δ​W‖F\\\|W\\\|\_\{F\}/\\\|\\Delta W\\\|\_\{F\}\. We therefore monitor*shape*, not magnitude: the singular\-spectrum entropy ofΔ2\\Delta\_\{2\}, the dominant first\-order signal\. The second\-order termΔ3\\Delta\_\{3\}remains part of the exact decomposition, but its spectral shape can still expose Q/K update coupling once the interaction becomes coherent\.

#### Exact low\-rank spectral computation\.

AlthoughΔ1\\Delta\_\{1\},Δ2\\Delta\_\{2\}, andΔ3\\Delta\_\{3\}are formallyd×dd\\times dQK\-product increments, their nonzero singular spectra can be computed exactly from small cores\. For anyA,B∈ℝd×rA,B\\in\\mathbb\{R\}^\{d\\times r\}with thin decompositionsA=QA​RAA=Q\_\{A\}R\_\{A\}andB=QB​RBB=Q\_\{B\}R\_\{B\},

A​B⊤=QA​\(RA​RB⊤\)​QB⊤,AB^\{\\top\}=Q\_\{A\}\(R\_\{A\}R\_\{B\}^\{\\top\}\)Q\_\{B\}^\{\\top\},\(3\)so the nonzero singular values ofA​B⊤AB^\{\\top\}are those of ther×rr\\times rcoreRA​RB⊤R\_\{A\}R\_\{B\}^\{\\top\}\. Applied to a single attention head,

Δ3\\displaystyle\\Delta\_\{3\}=Δ​Wq​Δ​Wk⊤,\\displaystyle=\\Delta W\_\{q\}\\Delta W\_\{k\}^\{\\top\},\(4\)Δ2\\displaystyle\\Delta\_\{2\}=\[Δ​Wq,Wq\]​\[Wk,Δ​Wk\]⊤,\\displaystyle=\[\\Delta W\_\{q\},W\_\{q\}\]\[W\_\{k\},\\Delta W\_\{k\}\]^\{\\top\},Δ1\\displaystyle\\Delta\_\{1\}=\[Wq,t,Wq,t−δ\]​\[Wk,t,−Wk,t−δ\]⊤,\\displaystyle=\[W\_\{q,t\},W\_\{q,t\-\\delta\}\]\[W\_\{k,t\},\-W\_\{k,t\-\\delta\}\]^\{\\top\},with ranks at mostdkd\_\{k\},2​dk2d\_\{k\}, and2​dk2d\_\{k\}, respectively\. Thus monitoring does not require materializing a densed×dd\\times dproduct whend≫dkd\\gg d\_\{k\}; it only requires the singular spectrum of a head\-dimensional core\. The architectural rank cap itself is not the anomaly–the signal is spectral concentration among the nonzero singular modes\. On a single Ascend 910B NPU, the compressed\-core computation gives large speedups at realistic hidden sizes while preserving the spectrum to small relative error \(See Table[1](https://arxiv.org/html/2606.28116#S3.T1)\)\.

Table 1:Single Ascend 910B NPU timing for full\-matrix eigendecomposition versus compressed\-core computation of the same singular\-spectrum quantities\.
#### Empirical ordering of the QK\-product increments\.

Empirically,Δ1\\Delta\_\{1\}andΔ2\\Delta\_\{2\}detect the low\-precision FA fault almost simultaneously, whileΔ3\\Delta\_\{3\}deviates later; all three precede the rawΔ​W\\Delta Wspectrum\. This ordering is consistent with the scale separation above \(Figure[3](https://arxiv.org/html/2606.28116#S3.F3)\)\. Since

Δ1=Δ2\+Δ3,‖Δ2‖F=O​\(‖W‖F​‖Δ​W‖F\),‖Δ3‖F=O​\(‖Δ​W‖F2\),\\Delta\_\{1\}=\\Delta\_\{2\}\+\\Delta\_\{3\},\\qquad\\\|\\Delta\_\{2\}\\\|\_\{F\}=O\(\\\|W\\\|\_\{F\}\\\|\\Delta W\\\|\_\{F\}\),\\qquad\\\|\\Delta\_\{3\}\\\|\_\{F\}=O\(\\\|\\Delta W\\\|\_\{F\}^\{2\}\),\(5\)the early\-to\-mid training regime‖Δ​W‖F≪‖W‖F\\\|\\Delta W\\\|\_\{F\}\\ll\\\|W\\\|\_\{F\}impliesΔ1≈Δ2\\Delta\_\{1\}\\approx\\Delta\_\{2\}\. Thus the exact QK\-product increment and its first\-order part expose the fault at nearly the same time\. The interaction termΔ3\\Delta\_\{3\}is weaker because it is second order, but it still lives directly in Q/K update\-coupling space and can become visible before spectral concentration is obvious in the separate factor updatesΔ​Wq,Δ​Wk\\Delta W\_\{q\},\\Delta W\_\{k\}\. This also explains why the QK\-product increment curves are smoother: they aggregate the update through the functional QK product, suppressing factor\-wise noise while preserving coherent Q/K drift\.

## 4MoE Router Monitoring

The MoE module is central to the frontier transformer architecture; most\>\>30B LLMs are now MoE\-based\(Meituan LongCat Team,[2025](https://arxiv.org/html/2606.28116#bib.bib41); Kimi Team,[2025](https://arxiv.org/html/2606.28116#bib.bib17); Wanget al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib42)\)\. In a top\-kkMoE layer\(Jacobset al\.,[1991](https://arxiv.org/html/2606.28116#bib.bib37); Shazeeret al\.,[2017](https://arxiv.org/html/2606.28116#bib.bib38)\), a lightweight router gating function selects which experts process each token\. Concretely, with the router weight matrixWR=\[w1,…,wn\]∈ℝd×nW\_\{R\}=\[w\_\{1\},\\ldots,w\_\{n\}\]\\in\\mathbb\{R\}^\{d\\times n\}, each tokenxxreceives expert scoress=WR⊤​xs=W\_\{R\}^\{\\top\}x, a softmax produces a routing distribution, and the top\-kkentries dictate which of thennexperts are actually invoked\. AlthoughWRW\_\{R\}typically holds well below0\.1%0\.1\\%of the parameters of a single MoE layer, the choices it makes determine which of the remaining99\.9%99\.9\\%are exercised on any given token\. This asymmetry between trivial parameter count and outsized influence on capacity utilization makes the router the natural place to look for MoE\-specific stability pathologies\. Because the router is a small linear map that is usually independent from any parallelism scheme, its internal state, even activations, can be monitored without cross\-device communication\.

A healthy router maintains diversity along both the expert and token axes\. Its weight columns should span distinct directions so that per\-token routing distributions do not collapse\. We study router stability through internal\-state indicators that quantify this diversity directly\.

### 4\.1Router Conditioning and Weight Similarity

The router selects experts through a softmax gate, which is shift\-invariant:softmax⁡\(s\)=softmax⁡\(s−c\)\\operatorname\{softmax\}\(s\)=\\operatorname\{softmax\}\(s\-c\)for any constantcc\. Settingc=w¯⊤​xc=\\bar\{w\}^\{\\top\}x, wherew¯=1n​∑iwi\\bar\{w\}=\\frac\{1\}\{n\}\\sum\_\{i\}w\_\{i\}is the mean of router weights, shows that the routing decision depends only on the centered weights\(wi−w¯\)⊤​x\(w\_\{i\}\-\\bar\{w\}\)^\{\\top\}x\. The ratio between the maximum deviation of router weights and the mean of router weights, i\.e\.,ε:=maxi⁡‖wi−w¯‖‖w¯‖\\varepsilon:=\\frac\{\\max\_\{i\}\\\|w\_\{i\}\-\\bar\{w\}\\\|\}\{\\\|\\bar\{w\}\\\|\}\(defined forw¯≠0\\bar\{w\}\\neq 0, with all router columns nonzero\), is a naturalconditioning ratiofor the router: it measures how large the discriminative deviations\(wi−w¯\)\(w\_\{i\}\-\\bar\{w\}\)are relative to the common modew¯\\bar\{w\}that softmax discards\. Note that softmax removesw¯\\bar\{w\}exactly, so a smallε\\varepsilondoes not corrupt the routing decision itself; rather, it signals an ill\-conditioned, near\-redundant parameterization, in which the large common modew¯\\bar\{w\}dominates the stored weights, leaving the discriminative component\(wi−w¯\)\(w\_\{i\}\-\\bar\{w\}\)with poor relative conditioning\. Similarly,Meituan LongCat Team \([2025](https://arxiv.org/html/2606.28116#bib.bib41)\)mention that they use router weight similaritysim⁡\(WR\)=𝔼i≠j​\[cosine\_similarity​\(wi,wj\)\]\\operatorname\{sim\}\(W\_\{R\}\)=\\mathbb\{E\}\_\{i\\neq j\}\[\\textrm\{cosine\\\_similarity\}\(w\_\{i\},w\_\{j\}\)\]as an indicator during the LLM training, and we can see that the pairwise weight similarity is lower\-bounded by a monotone function of this conditioning ratio, through the following proposition

> Proposition 3 \(Conditioning Ratio Lower\-Bounds Router Weight Similarity\)\. sim⁡\(WR\)≥1−nn−1​ε2\\operatorname\{sim\}\(W\_\{R\}\)\\geq 1\-\\frac\{n\}\{n\-1\}\\,\\varepsilon^\{2\}\(6\)

This shows that the conditioning ratioε\\varepsiloncontrols a lower bound on the router weight similarity: asε→0\\varepsilon\\rightarrow 0the similarity approaches 1, i\.e\. the expert columns collapse onto the common mean and the router becomes redundant and non\-discriminative \(the high\-similarity, low\-stability regime\)\. We defer the proof to Appendix[C](https://arxiv.org/html/2606.28116#A3)\.

We measuresim⁡\(WR\)\\operatorname\{sim\}\(W\_\{R\}\)across open\-source MoE checkpoints; results are summarized in Table[2](https://arxiv.org/html/2606.28116#S4.T2)\. It can be found that different MoE architectures have very different router weight similarity, and the operating point is strongly architecture\-dependent— from near\-orthogonal router columns in the GPT\-OSS family \(sim⁡\(WR\)≈0\\operatorname\{sim\}\(W\_\{R\}\)\\\!\\approx\\\!0\) to highly aligned columns in Qwen3\-35B\-A3B \(sim⁡\(WR\)≈0\.51\\operatorname\{sim\}\(W\_\{R\}\)\\\!\\approx\\\!0\.51\)\. Note that high similaritydoes not necessarily imply a collapsed prediction, but is a risk of low stability due to high redundancies\.

Its computational complexity can be reduced toO​\(n​d\)O\(nd\)by

sim⁡\(WR\)=n​‖R‖22−1n−1,where​R=1n​∑i=1nwi‖wi‖\\operatorname\{sim\}\(W\_\{R\}\)=\\frac\{n\\\|R\\\|\_\{2\}^\{2\}\-1\}\{n\-1\},\\textrm\{ where \}R=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{w\_\{i\}\}\{\\\|w\_\{i\}\\\|\}\(7\)and is therefore a very affordable metric to monitor during training\.

Table 2:Router weight similaritysim⁡\(WR\)=𝔼i≠j​\[cos⁡\(wi,wj\)\]\\operatorname\{sim\}\(W\_\{R\}\)=\\mathbb\{E\}\_\{i\\neq j\}\[\\cos\(w\_\{i\},w\_\{j\}\)\]across open\-source MoE checkpoints, reported as mean±\\pmstd over all MoE layers \(number of expertsn=32,128,256,64,160,256,384n=32,128,256,64,160,256,384, respectively\)\.
### 4\.2The Effective Component of Routers Is Learning\-Rate Sensitive

The similarity analysis above isolates the router directions that distinguish experts\. In matrix form, letCn=In−1n​𝟏𝟏⊤C\_\{n\}=I\_\{n\}\-\\frac\{1\}\{n\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}andWR,c=WR​Cn=\[w1−w¯,…,wn−w¯\]W\_\{R,c\}=W\_\{R\}C\_\{n\}=\[w\_\{1\}\-\\bar\{w\},\\ldots,w\_\{n\}\-\\bar\{w\}\]\. Only this centered component changes the centered logitsδ​\(x\)=WR,c⊤​x\\delta\(x\)=W\_\{R,c\}^\{\\top\}x; the common\-mode component adds the same scalar to every expert score and is removed by softmax\. Token dispatch is therefore controlled by centered marginsmi​j​\(x\)=\(wi−wj\)⊤​xm\_\{ij\}\(x\)=\(w\_\{i\}\-w\_\{j\}\)^\{\\top\}x, rather than by the raw router norm\.

The router can fail at either extreme\. When the routing distribution is nearly uniform \(H​\(p\)→log⁡nH\(p\)\\to\\log n\), many experts have nearly tied scores and the router is uncertain, a failure mode studied byWuet al\.\([2024](https://arxiv.org/html/2606.28116#bib.bib39)\)\. When the distribution is nearly a point mass \(H​\(p\)→0H\(p\)\\to 0\), tokens are routed in a singleton\-like way and the model loses expert diversity\. Modern MoE systems therefore try to keep expert use balanced through auxiliary or loss\-free balancing mechanisms and capacity controls\(Feduset al\.,[2021](https://arxiv.org/html/2606.28116#bib.bib45); Wanget al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib42)\); however, those quantities are realized only after top\-kkassignment or aggregation over a batch\.

Per\-token entropy is a more sensitive readout because it is continuous in the full softmax distribution,

H​\(p​\(x\)\)=−∑ipi​\(x\)​log⁡pi​\(x\)\.H\(p\(x\)\)=\-\\sum\_\{i\}p\_\{i\}\(x\)\\log p\_\{i\}\(x\)\.\(8\)Maximal violation \(MaxVio\)\(Wanget al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib42)\), capacity overflow, and load\-balance counts are downstream discrete readouts: they can stay unchanged while a leading expert’s probability grows inside an already\-fixed top\-kkset\. Locally, this behavioral entropy drop is tied to centered logit energy,

log⁡n−𝔼x​H​\(p\)≈12​n​tr​\(WR,c⊤​Mx​WR,c\),\\log n\-\\mathbb\{E\}\_\{x\}\\,H\(p\)\\;\\approx\\;\\frac\{1\}\{2n\}\\,\\mathrm\{tr\}\\\!\\bigl\(W\_\{R,c\}^\{\\top\}M\_\{x\}\\,W\_\{R,c\}\\bigr\),\(9\)whereMx=𝔼​\[x​x⊤\]M\_\{x\}=\\mathbb\{E\}\[xx^\{\\top\}\]\. This link is used only as a local consistency check; the actual collapse certificate is the behavior\-side entropy on current tokens\.

![Refer to caption](https://arxiv.org/html/2606.28116v1/x1.png)\(a\)Layer\-average entropy vs LRs
![Refer to caption](https://arxiv.org/html/2606.28116v1/x2.png)\(b\)Layer\-average entropy vs GBS

Figure 4:Router per\-token entropy under different learning rates and GBS\.When a stable\-winner feedback loop is present, a large learning rate amplifies it\. For a softmax gate with MoE outputy​\(x\)=∑ipi​\(x\)​Ei​\(x\)y\(x\)=\\sum\_\{i\}p\_\{i\}\(x\)E\_\{i\}\(x\), defineqi​\(x\)=⟨∂ℓ/∂y,Ei​\(x\)⟩q\_\{i\}\(x\)=\\langle\\partial\\ell/\\partial y,E\_\{i\}\(x\)\\rangleandri​\(x\)=∂ℓ/∂si=pi​\(x\)​\(qi​\(x\)−∑jpj​\(x\)​qj​\(x\)\)r\_\{i\}\(x\)=\\partial\\ell/\\partial s\_\{i\}=p\_\{i\}\(x\)\(q\_\{i\}\(x\)\-\\sum\_\{j\}p\_\{j\}\(x\)q\_\{j\}\(x\)\), so𝟏⊤​r=0\\mathbf\{1\}^\{\\top\}r=0and∇WRℓ​\(x\)=x​r​\(x\)⊤\\nabla\_\{W\_\{R\}\}\\ell\(x\)=x\\,r\(x\)^\{\\top\}\(Jacobset al\.,[1991](https://arxiv.org/html/2606.28116#bib.bib37); Shazeeret al\.,[2017](https://arxiv.org/html/2606.28116#bib.bib38)\)\. We analyze this mechanism under the dense\-softmax relaxation of the gate, treatingy​\(x\)=∑ipi​\(x\)​Ei​\(x\)y\(x\)=\\sum\_\{i\}p\_\{i\}\(x\)E\_\{i\}\(x\)as a mixture over all experts and thus setting aside top\-kkhard selection, expert\-capacity limits, and the fact that some implementations route gate gradients only through the selected experts\. The margin\-feedback argument then applies to the soft gate scoressis\_\{i\}that drive selection\. Consider a coherent token regionΩ\\Omegain which expertj⋆j^\{\\star\}is already a slight winner, and lethhbe any competing expert\. Define the time\-indexed winner–competitor margin on a probe tokenx′x^\{\\prime\}as

mj⋆​h​\(x′,t\)=sj⋆​\(x′,t\)−sh​\(x′,t\)=\(wj⋆​\(t\)−wh​\(t\)\)⊤​x′\.m\_\{j^\{\\star\}h\}\(x^\{\\prime\},t\)=s\_\{j^\{\\star\}\}\(x^\{\\prime\},t\)\-s\_\{h\}\(x^\{\\prime\},t\)=\(w\_\{j^\{\\star\}\}\(t\)\-w\_\{h\}\(t\)\)^\{\\top\}x^\{\\prime\}\.For the strongest competitorh⋆h^\{\\star\}in this local window,mj⋆​h⋆m\_\{j^\{\\star\}h^\{\\star\}\}is the top\-two margin\. For the mechanism sketch, write one plain stochastic\-gradient step on tokenxxaswi​\(t\+1\)=wi​\(t\)−η​x​ri​\(x,t\)w\_\{i\}\(t\+1\)=w\_\{i\}\(t\)\-\\eta xr\_\{i\}\(x,t\); adaptive optimizers replace this by their preconditioned effective step, but the same margin\-coherence argument applies\. This update changes the margin onx′x^\{\\prime\}by

Δ​mj⋆​h​\(x′,t\)=−η​\(x′⁣⊤​x\)​\(rj⋆​\(x,t\)−rh​\(x,t\)\)\.\\Delta m\_\{j^\{\\star\}h\}\(x^\{\\prime\},t\)=\-\\eta\(x^\{\\prime\\top\}x\)\\bigl\(r\_\{j^\{\\star\}\}\(x,t\)\-r\_\{h\}\(x,t\)\\bigr\)\.Therefore define the time\-dependent reinforcement coefficient

γh​\(t\)=−𝔼x,x′∈Ω​\[\(x′⁣⊤​x\)​\(rj⋆​\(x,t\)−rh​\(x,t\)\)\]\.\\gamma\_\{h\}\(t\)=\-\\mathbb\{E\}\_\{x,x^\{\\prime\}\\in\\Omega\}\\left\[\(x^\{\\prime\\top\}x\)\(r\_\{j^\{\\star\}\}\(x,t\)\-r\_\{h\}\(x,t\)\)\\right\]\.\(10\)Whenγh​\(t\)\>0\\gamma\_\{h\}\(t\)\>0, the update reinforces the current winner on average over nearby tokens inΩ\\Omegainstead of pulling it back toward its competitors\. Summing the one\-step recurrence gives

𝔼​\[mj⋆​h​\(x′,T\)\]≳mj⋆​h​\(x′,0\)\+η​∑t<Tγh​\(t\)\.\\mathbb\{E\}\[m\_\{j^\{\\star\}h\}\(x^\{\\prime\},T\)\]\\gtrsim m\_\{j^\{\\star\}h\}\(x^\{\\prime\},0\)\+\\eta\\sum\_\{t<T\}\\gamma\_\{h\}\(t\)\.\(11\)Soη\\etascales each reinforcement increment directly\. For the strongest competitor, write the gap asGT=η​∑t<Tγh⋆​\(t\)G\_\{T\}=\\eta\\sum\_\{t<T\}\\gamma\_\{h^\{\\star\}\}\(t\), somj⋆​h⋆​\(x′,T\)≳mj⋆​h⋆​\(x′,0\)\+GTm\_\{j^\{\\star\}h^\{\\star\}\}\(x^\{\\prime\},T\)\\gtrsim m\_\{j^\{\\star\}h^\{\\star\}\}\(x^\{\\prime\},0\)\+G\_\{T\}\. Since

1−p\(1\)​\(x′,T\)≤\(n−1\)​e−mj⋆​h⋆​\(x′,T\)≲\(n−1\)​e−mj⋆​h⋆​\(x′,0\)​e−GT,1\-p\_\{\(1\)\}\(x^\{\\prime\},T\)\\leq\(n\-1\)e^\{\-m\_\{j^\{\\star\}h^\{\\star\}\}\(x^\{\\prime\},T\)\}\\lesssim\(n\-1\)e^\{\-m\_\{j^\{\\star\}h^\{\\star\}\}\(x^\{\\prime\},0\)\}e^\{\-G\_\{T\}\},\(12\)the residual routing mass shrinks exponentially in the accumulated margin gain\. This conditional mechanism is supported by Figure[4](https://arxiv.org/html/2606.28116#S4.F4)\(a\): at fixed GBS, larger learning rates consistently reduce the layer\-average router per\-token entropy, and the reduction is strongest when the auxiliary load\-balancing loss is removed\. Figure[4](https://arxiv.org/html/2606.28116#S4.F4)\(b\) shows a similar pattern for the GBS sweep\. Even at a fixed learning rate, decreasing GBS lowers router entropy, with the orderingGBS=384\>96\>24\\mathrm\{GBS\}=384\>96\>24visible for both learning\-rate settings\. To make the LR–GBS coupling explicit, for global batch sizeBBwrite the empirical reinforcement coefficient as

γ^h,B​\(t\)=γh​\(t\)\+ξh,B​\(t\),𝔼​\[ξh,B​\(t\)\]=0,Var⁡\[ξh,B​\(t\)\]≈σh2​\(t\)B\.\\widehat\{\\gamma\}\_\{h,B\}\(t\)=\\gamma\_\{h\}\(t\)\+\\xi\_\{h,B\}\(t\),\\qquad\\mathbb\{E\}\[\\xi\_\{h,B\}\(t\)\]=0,\\qquad\\operatorname\{Var\}\[\\xi\_\{h,B\}\(t\)\]\\approx\\frac\{\\sigma\_\{h\}^\{2\}\(t\)\}\{B\}\.Then the stochastic component of the one\-step margin update obeys

Var⁡\[Δ​mj⋆​h​\(t\)\]≈η2​Var⁡\[ξh,B​\(t\)\]≈η2​σh2​\(t\)B\.\\operatorname\{Var\}\[\\Delta m\_\{j^\{\\star\}h\}\(t\)\]\\approx\\eta^\{2\}\\operatorname\{Var\}\[\\xi\_\{h,B\}\(t\)\]\\approx\\frac\{\\eta^\{2\}\\sigma\_\{h\}^\{2\}\(t\)\}\{B\}\.Thus a smaller GBS increases the per\-step margin variance at fixed LR\. The local expansion proved in Appendix[D](https://arxiv.org/html/2606.28116#A4)shows why entropy drops: near uniform routing, if the centered\-logit perturbation induced by mini\-batch noise isϵ\\epsilonwith𝔼​\[ϵ\]=0\\mathbb\{E\}\[\\epsilon\]=0, then

𝔼ϵH\(softmax\(δ\+ϵ\)\)=H\(softmax\(δ\)\)−12​n𝔼∥Cnϵ∥22\+O\(∥δ,ϵ∥3\)\.\\mathbb\{E\}\_\{\\epsilon\}H\(\\operatorname\{softmax\}\(\\delta\+\\epsilon\)\)=H\(\\operatorname\{softmax\}\(\\delta\)\)\-\\frac\{1\}\{2n\}\\mathbb\{E\}\\\|C\_\{n\}\\epsilon\\\|\_\{2\}^\{2\}\+O\(\\\|\\delta,\\epsilon\\\|^\{3\}\)\.Smaller GBS therefore lowers average router entropy at fixed LR\. If the trajectory additionally enters the positive\-γh​\(t\)\\gamma\_\{h\}\(t\)stable\-winner regime above, these noise\-induced margin excursions may be reinforced over subsequent steps; the current GBS sweep supports this as a conditional mechanism rather than a standalone proof of collapse\.

![Refer to caption](https://arxiv.org/html/2606.28116v1/signature/p1372b45b_per_token_entropy_first25k.png)\(a\)Router entropy
![Refer to caption](https://arxiv.org/html/2606.28116v1/signature/qkv_full_dw_singular_spectrum_w10_lr.png)\(b\)Singular spectrum ofΔ​W\\Delta Wunder different LRs
![Refer to caption](https://arxiv.org/html/2606.28116v1/signature/qkv_full_dw_singular_spectrum_w10_gbs.png)\(c\)Singular spectrum ofΔ​W\\Delta Wunder different GBSs

Figure 5:Visualization of the fault signatures of the two modules\. \(a\) shows the router per\-token entropy under low\-precision FA, while \(b\) and \(c\) show the singular spectrum ofΔ​W\\Delta Wunder different learning\-rate and GBS settings\. The router indicator is insensitive to the low\-precision attention fault \(a\), whereas theΔ​W\\Delta Wspectrum is insensitive to LR/GBS variation \(b, c\); the two signatures are therefore separable\.

## 5Designing Module\-Specific Monitors from First Principles

The two monitor families developed in Sections[3](https://arxiv.org/html/2606.28116#S3)and[4](https://arxiv.org/html/2606.28116#S4)target different modules with different failure mechanisms\. We advocate that the design principle is to understand the module’s failure mechanism and derive the corresponding monitor, rather than to seek a universal monitoring architecture\. The two case studies below illustrate this principle\.

#### Operator\-level faults \(Flash Attention\)\.

Under the biased low\-precision injection described in Section[3](https://arxiv.org/html/2606.28116#S3), the attention\-side indicators exhibit observable spectrum collapse in a consistent order:Δ2\\Delta\_\{2\}spectra show observable spectrum collapse at∼5,000\{\\sim\}5\{,\}000steps,Δ​W\\Delta Wentropy collapses at∼13,000\{\\sim\}13\{,\}000steps, and loss spikes only at∼22,000\{\\sim\}22\{,\}000steps – a lead time of thousands of steps for the earliest indicator\. Throughout, the router indicator remains in its healthy ranges until the loss diverges as shown in Figure[5\(a\)](https://arxiv.org/html/2606.28116#S4.F5.sf1)\. The fault selectively damages the attention update path without disturbing routing\.

#### Hyperparameter sensitivity \(MoE router\)\.

The stable\-winner feedback loop derived in Section[4](https://arxiv.org/html/2606.28116#S4)predicts that larger learning rates and smaller global batch sizes amplify router entropy collapse\. Figure[4](https://arxiv.org/html/2606.28116#S4.F4)confirms this: at fixed GBS, larger learning rates consistently reduce layer\-average per\-token entropy, and the effect is strongest when the auxiliary load\-balancing loss is removed\. In our observed runs the singular spectrum ofΔ​W\\Delta Wremains in its healthy ranges under hyperparameter\-driven routing changes, as shown in Figure[5\(b\)](https://arxiv.org/html/2606.28116#S4.F5.sf2)and[5\(c\)](https://arxiv.org/html/2606.28116#S4.F5.sf3)\. These signatures are consistent with the two indicator families responding to disjoint failure mechanisms\.

#### From case studies to a design principle\.

These two cases are illustrative, not exhaustive: large\-model training admits many more failure modes – data distribution shift, optimizer state corruption, depth\-scaling instabilities, communication faults – each with its own mechanism\. The transferable lesson is not a fixed monitoring architecture but a*design principle*: each fault class has a physical or algorithmic mechanism, and the mechanism determines which internal observable will fire first\. A practitioner who understands a module’s failure mechanism can derive the corresponding monitor\. This argues for systematic investment in*internal*training metrics grounded in module\-level mechanisms – interpretability and observability of training dynamics – rather than reliance on loss curves and gradient norms alone\.

## 6Limitations

We discuss four open directions\.Attention variants\.The bilinear decomposition in Proposition 2 assumes the explicitWq,WkW\_\{q\},W\_\{k\}parameterization of multi\-head attention\. For MLA, GQA, MQA, and DSA, the effective QK operator is mediated by compression projections, shared heads, or dynamic routing, so theΔ2\\Delta\_\{2\}proxy must be re\-derived for each variant\. Low\-rank update drift is a systematic consequence of biased backward rounding in low\-precision FA, not an artifact of MHA; the spectral monitoring principle transfers, but the concrete algebra and detection thresholds remain variant\-specific\.Precision and fault coverage\.The validation suite covers one fault class per category \(BF16 bit\-shift for operator\-level faults, uniform learning\-rate scaling for hyperparameter\-level faults\)\. Broader coverage of FP8 training, stochastic rounding, and gradient\-clipping interactions is future work\. The forward\-error closure toκ​\(Wk⊤​Wq\)\\kappa\(W\_\{k\}^\{\\top\}W\_\{q\}\)also has a dimensionality mismatch \(operator\-spaceD×DD\\times Dvs\. head\-spacedk×dkd\_\{k\}\\times d\_\{k\}\) that we have not yet resolved\.Two\-stage timing\.The≈\\approx8,000\-step gap betweenΔ2\\Delta\_\{2\}andΔ​W\\Delta Wentropy collapse is empirically robust but lacks a closed\-form prediction\. Weyl amplification accounts for the order of the gap but not its precise magnitude, which likely requires anisotropic noise statistics\. A quantitative detection\-onset analysis via spiked random\-matrix theory is ongoing\.Router indicators\.The algebraic reduction of Equation[9](https://arxiv.org/html/2606.28116#S4.E9)to a purely weight\-only quantity‖WR,c‖F2/\(2​n\)\\\|W\_\{R,c\}\\\|\_\{F\}^\{2\}/\(2n\)requires activation isotropyMx∝IM\_\{x\}\\propto I\. RMSNorm enforces onlytr​\(Mx\)=d\\mathrm\{tr\}\(M\_\{x\}\)=d, and trained Transformer activations are anisotropic\. The resulting divergence between weight\-side and decision\-side router indicators in concentrated\-activation regimes is itself diagnostic, but is not currently used by the monitoring stack\.

## 7Conclusion

We derived internal training monitors for two stability\-critical modules of modern LLMs by asking what each module is supposed to compute and where damage from its known failure mechanism would first appear\.

For FA, the answer is the spectral geometry ofΔ​W\\Delta W\. Biased low\-precision backward errors produce coherent low\-rank drift in accumulated weight updates \(Observation 1\), and the QK\-product decomposition \(Proposition 2\) exposes this drift through the first\-order termΔ2\\Delta\_\{2\}, computable from head\-dimensional cores without materializing fulld×dd\\times dproducts\. In our controlled fault injection,Δ2\\Delta\_\{2\}spectral collapse preceded loss divergence by approximately17,00017\{,\}000steps, andΔ​W\\Delta Wsingular\-spectrum collapse preceded it by approximately9,0009\{,\}000steps\. Router indicators did not respond to this fault\.

For MoE routers, the answer is per\-token entropy and weight similarity\. The conditioning\-ratio bound \(Proposition 3\) and the local weight\-entropy link \(Equation[9](https://arxiv.org/html/2606.28116#S4.E9)\) connect weight\-side redundancy to decision\-side entropy drop\. Learning\-rate and batch\-size sweeps confirmed the predicted sensitivity: larger learning rates and smaller batch sizes amplify entropy collapse, whileΔ​W\\Delta Wspectral indicators remain unchanged\. The two fault signatures do not cross\-contaminate\.

The design principle itself is the transferable part of this work\. Modern LLM architectures continue to introduce modules with their own internal dynamics: persistent memory stores such as Engram\(Chenget al\.,[2026](https://arxiv.org/html/2606.28116#bib.bib52)\), manifold\-constrained residual connections\(Xieet al\.,[2025](https://arxiv.org/html/2606.28116#bib.bib53)\), attention\-based residual gates\(Chenet al\.,[2026](https://arxiv.org/html/2606.28116#bib.bib54)\), and learnable structured\-sparsity mechanisms\(Fanget al\.,[2024](https://arxiv.org/html/2606.28116#bib.bib55)\)each carry failure modes that loss curves and gradient norms cannot attribute to a source\. As these modules enter production training, each will need monitors derived from its own mechanism, not borrowed from attention or routing\. The methodology demonstrated here provides a template for that derivation\.

## Acknowledgments

We thank Xuemin Hong, Jun Li, and Honghui Ge for helpful discussions, constructive feedback on our work, and broader support that helped make this project possible\.

## References

- Singular value decomposition for genome\-wide expression data processing and modeling\.Proceedings of the National Academy of Sciences97\(18\),pp\. 10101–10106\.Cited by:[§3\.2](https://arxiv.org/html/2606.28116#S3.SS2.p1.8)\.
- H\. Bao, R\. Hataya, and R\. Karakida \(2024\)Self\-attention networks localize when qk\-eigenspectrum concentrates\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,External Links:[Link](https://openreview.net/forum?id=aRZjRj41WQ)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.
- S\. Bhojanapalli, C\. Yun, A\. S\. Rawat, S\. J\. Reddi, and S\. Kumar \(2020\)Low\-rank bottleneck in multi\-head attention models\.InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13\-18 July 2020, Virtual Event,Proceedings of Machine Learning Research, Vol\.119,pp\. 864–873\.External Links:[Link](http://proceedings.mlr.press/v119/bhojanapalli20a.html)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.
- G\. Chen, Y\. Zhang, J\. Su, W\. Xu, S\. Pan, Y\. Wang, Y\. Wang, G\. Chen, B\. Yin, Y\. Chen, J\. Yan, and M\. Wei \(2026\)Attention residuals\.ArXiv preprintabs/2603\.15031\.External Links:[Link](https://arxiv.org/abs/2603.15031)Cited by:[§7](https://arxiv.org/html/2606.28116#S7.p4.1)\.
- X\. Cheng, W\. Zeng, D\. Dai, Q\. Chen, B\. Wang, Z\. Xie, K\. Huang, X\. Yu, Z\. Hao, Y\. Li, H\. Zhang, H\. Zhang, D\. Zhao, and W\. Liang \(2026\)Conditional memory via scalable lookup: a new axis of sparsity for large language models\.ArXiv preprintabs/2601\.07372\.External Links:[Link](https://arxiv.org/abs/2601.07372)Cited by:[§7](https://arxiv.org/html/2606.28116#S7.p4.1)\.
- L\. Chizat, E\. Oyallon, and F\. R\. Bach \(2019\)On lazy training in differentiable programming\.InAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\-14, 2019, Vancouver, BC, Canada,H\. M\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d’Alché\-Buc, E\. B\. Fox, and R\. Garnett \(Eds\.\),pp\. 2933–2943\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html)Cited by:[§3](https://arxiv.org/html/2606.28116#S3.p3.4)\.
- A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.\(2022\)PaLM: scaling language modeling with pathways\.External Links:2204\.02311Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- J\. M\. Cohen, S\. Kaur, Y\. Li, J\. Z\. Kolter, and A\. Talwalkar \(2021\)Gradient descent on neural networks typically occurs at the edge of stability\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=jh-rTtvkGeM)Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p3.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-V3 technical report\.External Links:2412\.19437Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-V4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- M\. Dehghani, J\. Djolonga, B\. Mustafa, P\. Padlewski, J\. Heek, J\. Gilmer, A\. P\. Steiner, M\. Caron, R\. Geirhos, I\. Alabdulmohsin, R\. Jenatton, L\. Beyer, M\. Tschannen, A\. Arnab, X\. Wang, C\. R\. Ruiz, M\. Minderer, J\. Puigcerver, U\. Evci, M\. Kumar, S\. van Steenkiste, G\. F\. Elsayed, A\. Mahendran, F\. Yu, A\. Oliver, F\. Huot, J\. Bastings, M\. Collier, A\. A\. Gritsenko, V\. Birodkar, C\. N\. Vasconcelos, Y\. Tay, T\. Mensink, A\. Kolesnikov, F\. Pavetic, D\. Tran, T\. Kipf, M\. Lucic, X\. Zhai, D\. Keysers, J\. J\. Harmsen, and N\. Houlsby \(2023\)Scaling vision transformers to 22 billion parameters\.InInternational Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 7480–7512\.External Links:[Link](https://proceedings.mlr.press/v202/dehghani23a.html)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Dong, J\. Cordonnier, and A\. Loukas \(2021\)Attention is not all you need: pure attention loses rank doubly exponentially with depth\.InProceedings of the 38th International Conference on Machine Learning, ICML 2021, 18\-24 July 2021, Virtual Event,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 2793–2803\.External Links:[Link](http://proceedings.mlr.press/v139/dong21a.html)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.
- G\. Fang, H\. Yin, S\. Muralidharan, G\. Heinrich, J\. Pool, J\. Kautz, P\. Molchanov, and X\. Wang \(2024\)MaskLLM: learnable semi\-structured sparsity for large language models\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/0e9a05f5ce62284c91e4a33498899124-Abstract-Conference.html)Cited by:[§7](https://arxiv.org/html/2606.28116#S7.p4.1)\.
- W\. Fang, H\. Zhang, Z\. Gong, L\. Zeng, X\. Lu, B\. Liu, X\. Wu, Y\. Zheng, Z\. Hu, and X\. Zhang \(2023\)A survey of metrics to enhance training dependability in large language models\.In2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops \(ISSREW\),pp\. 180–185\.Cited by:[§3](https://arxiv.org/html/2606.28116#S3.p2.3)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2021\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.External Links:2101\.03961Cited by:[§4\.2](https://arxiv.org/html/2606.28116#S4.SS2.p2.3)\.
- GLM\-4\.5 Team \(2025\)GLM\-4\.5: agentic, reasoning, and coding \(ARC\) foundation models\.External Links:2508\.06471Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- A\. Golden, S\. Hsia, F\. Sun, B\. Acun, B\. Hosmer, Y\. Lee, Z\. DeVito, J\. Johnson, G\. Wei, D\. Brooks, and C\. Wu \(2024\)Is flash attention stable?\.External Links:2405\.02803Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p2.1),[§1](https://arxiv.org/html/2606.28116#S1.p3.1),[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.28116#S3.p1.2)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.
- I\. C\.F\. Ipsen and A\. K\. Saibaba \(2024\)Stable rank and intrinsic dimension of real and complex matrices\.External Links:2407\.21594Cited by:[§3\.2](https://arxiv.org/html/2606.28116#S3.SS2.p1.8)\.
- R\. A\. Jacobs, M\. I\. Jordan, S\. J\. Nowlan, and G\. E\. Hinton \(1991\)Adaptive mixtures of local experts\.Neural Computation3\(1\),pp\. 79–87\.Cited by:[§4\.2](https://arxiv.org/html/2606.28116#S4.SS2.p4.12),[§4](https://arxiv.org/html/2606.28116#S4.p1.10)\.
- A\. Jacot, C\. Hongler, and F\. Gabriel \(2018\)Neural tangent kernel: convergence and generalization in neural networks\.InAdvances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3\-8, 2018, Montréal, Canada,S\. Bengio, H\. M\. Wallach, H\. Larochelle, K\. Grauman, N\. Cesa\-Bianchi, and R\. Garnett \(Eds\.\),pp\. 8580–8589\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/5a4be1fa34e62bb8a6ec6b91d2462f5a-Abstract.html)Cited by:[§3](https://arxiv.org/html/2606.28116#S3.p3.4)\.
- D\. S\. Kalra, J\. Gagnon\-Audet, A\. Gromov, I\. Mediratta, K\. Niu, A\. H\. Miller, and M\. Shvartsman \(2026\)A scalable measure of loss landscape curvature for analyzing the training dynamics of llms\.ArXiv preprintabs/2601\.16979\.External Links:[Link](https://arxiv.org/abs/2601.16979)Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p3.1)\.
- Kimi Team \(2025\)Kimi k2: open agentic intelligence\.External Links:2507\.20534Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1),[§1](https://arxiv.org/html/2606.28116#S1.p3.1),[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.28116#S3.p1.2),[§4](https://arxiv.org/html/2606.28116#S4.p1.10)\.
- J\. Lee, L\. Xiao, S\. S\. Schoenholz, Y\. Bahri, R\. Novak, J\. Sohl\-Dickstein, and J\. Pennington \(2019\)Wide neural networks of any depth evolve as linear models under gradient descent\.InAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\-14, 2019, Vancouver, BC, Canada,H\. M\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d’Alché\-Buc, E\. B\. Fox, and R\. Garnett \(Eds\.\),pp\. 8570–8581\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/0d1a9651497a38d8b1c3871c84528bd4-Abstract.html)Cited by:[§3](https://arxiv.org/html/2606.28116#S3.p3.4)\.
- J\. Liu, J\. Su, X\. Yao, Z\. Jiang,et al\.\(2025\)Muon is scalable for LLM training\.External Links:2502\.16982Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.
- Meituan LongCat Team \(2025\)LongCat\-Flash technical report\.External Links:2509\.01322Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1),[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.28116#S4.SS1.p1.14),[§4](https://arxiv.org/html/2606.28116#S4.p1.10)\.
- X\. Pan, A\. Philip, Z\. Xie, and O\. Schwartz \(2024\)Dissecting query\-key interaction in vision transformers\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/6216515a5e0b3257c49dcb1647e497d1-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.
- H\. Qiu and Q\. Yao \(2026\)Why low\-precision transformer training fails: an analysis on flash attention\.Note:ICLR 2026External Links:2510\.04212Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p2.1),[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4),[§3\.1](https://arxiv.org/html/2606.28116#S3.SS1.p1.16),[§3\.1](https://arxiv.org/html/2606.28116#S3.SS1.p2.9.9),[§3\.3](https://arxiv.org/html/2606.28116#S3.SS3.p1.5),[§3](https://arxiv.org/html/2606.28116#S3.p1.2),[§3](https://arxiv.org/html/2606.28116#S3.p2.3)\.
- Qwen Team \(2026\)Qwen3\.5\-Omni technical report\.External Links:2604\.15804Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- O\. Roy and M\. Vetterli \(2007\)The effective rank: a measure of effective dimensionality\.InEuropean Signal Processing Conference \(EUSIPCO\),pp\. 606–610\.Cited by:[§3\.2](https://arxiv.org/html/2606.28116#S3.SS2.p1.8)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. V\. Le, G\. E\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24\-26, 2017, Conference Track Proceedings,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§4\.2](https://arxiv.org/html/2606.28116#S4.SS2.p4.12),[§4](https://arxiv.org/html/2606.28116#S4.p1.10)\.
- S\. Smith, M\. Patwary, B\. Norick, P\. LeGresley, S\. Rajbhandari, J\. Casper, Z\. Liu, S\. Prabhumoye, G\. Zerveas, V\. Korthikanti, E\. Zhang, R\. Child, R\. Y\. Aminabadi, J\. Bernauer, X\. Song, M\. Shoeybi, Y\. He, M\. Houston, S\. Tiwary, and B\. Catanzaro \(2022\)Using DeepSpeed and Megatron to train Megatron\-Turing NLG 530b, a large\-scale generative language model\.External Links:2201\.11990Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- S\. Takase, S\. Kiyono, S\. Kobayashi, and J\. Suzuki \(2025\)Spike no more: stabilizing the pre\-training of large language models\.Note:COLM 2025External Links:2312\.16903Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p3.1),[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Wang, H\. Gao, C\. Zhao, X\. Sun, and D\. Dai \(2024\)Auxiliary\-loss\-free load balancing strategy for mixture\-of\-experts\.External Links:2408\.15664Cited by:[§4\.2](https://arxiv.org/html/2606.28116#S4.SS2.p2.3),[§4\.2](https://arxiv.org/html/2606.28116#S4.SS2.p3.1),[§4](https://arxiv.org/html/2606.28116#S4.p1.10)\.
- M\. Wortsman, P\. J\. Liu, L\. Xiao, K\. E\. Everett, A\. A\. Alemi, B\. Adlam, J\. D\. Co\-Reyes, I\. Gur, A\. Kumar, R\. Novak, J\. Pennington, J\. Sohl\-Dickstein, K\. Xu, J\. Lee, J\. Gilmer, and S\. Kornblith \(2024\)Small\-scale proxies for large\-scale transformer training instabilities\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=d8w0pmvXbZ)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Wu, Z\. Qiu, Z\. Wang, H\. Zhao, and J\. Fu \(2024\)GW\-MoE: resolving uncertainty in MoE router with global workspace theory\.External Links:2406\.12375Cited by:[§4\.2](https://arxiv.org/html/2606.28116#S4.SS2.p2.3)\.
- Z\. Xie, Y\. Wei, H\. Cao, C\. Zhao, C\. Deng, J\. Li, D\. Dai, H\. Gao, J\. Chang, L\. Zhao, S\. Zhou, Z\. Xu, Z\. Zhang, W\. Zeng, S\. Hu, Y\. Wang, J\. Yuan, L\. Wang, and W\. Liang \(2025\)mHC: manifold\-constrained hyper\-connections\.ArXiv preprintabs/2512\.24880\.External Links:[Link](https://arxiv.org/abs/2512.24880)Cited by:[§7](https://arxiv.org/html/2606.28116#S7.p4.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui,et al\.\(2025\)Qwen3 technical report\.External Links:2505\.09388Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- Z\. Yao, A\. Gholami, K\. Keutzer, and M\. W\. Mahoney \(2020\)PyHessian: neural networks through the lens of the hessian\.In2020 IEEE International Conference on Big Data \(Big Data\),pp\. 581–590\.External Links:[Document](https://dx.doi.org/10.1109/BigData50022.2020.9378171)Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p3.1)\.
- D\. Yunis, K\. K\. Patel, S\. Wheeler, P\. Savarese, G\. Vardi, K\. Livescu, M\. Maire, and M\. R\. Walter \(2024\)Approaching deep learning through the spectral dynamics of weights\.External Links:2408\.11804Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.
- A\. Zeng, X\. Liu, Z\. Du, Z\. Wang, H\. Lai, M\. Ding, Z\. Yang, Y\. Xu, W\. Zheng, X\. Xia, W\. L\. Tam, Z\. Ma, Y\. Xue, J\. Zhai, W\. Chen, Z\. Liu, P\. Zhang, Y\. Dong, and J\. Tang \(2023\)GLM\-130B: an open bilingual pre\-trained model\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/pdf?id=-Aw0rrrPUF)Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p1.1)\.
- S\. Zhai, T\. Likhomanenko, E\. Littwin, D\. Busbridge, J\. Ramapuram, Y\. Zhang, J\. Gu, and J\. M\. Susskind \(2023\)Stabilizing transformer training by preventing attention entropy collapse\.InInternational Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 40770–40803\.External Links:[Link](https://proceedings.mlr.press/v202/zhai23a.html)Cited by:[§1](https://arxiv.org/html/2606.28116#S1.p3.1),[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian \(2024\)GaLore: memory\-efficient LLM training by gradient low\-rank projection\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,External Links:[Link](https://openreview.net/forum?id=hYHsrKDiX7)Cited by:[§2](https://arxiv.org/html/2606.28116#S2.SS0.SSS0.Px2.p1.4)\.

## Appendix

## Appendix AInitialization Dominance and Raw\-Weight Monitoring

We justify the initialization\-dominance claim used in Section[3](https://arxiv.org/html/2606.28116#S3)\. LetWt=W0\+EtW\_\{t\}=W\_\{0\}\+E\_\{t\}, whereEtE\_\{t\}is the total displacement from initialization\. Weyl’s singular\-value perturbation inequality gives, for every singular indexii,

\|σi​\(Wt\)−σi​\(W0\)\|≤‖Et‖2≤‖Et‖F\.\|\\sigma\_\{i\}\(W\_\{t\}\)\-\\sigma\_\{i\}\(W\_\{0\}\)\|\\leq\\\|E\_\{t\}\\\|\_\{2\}\\leq\\\|E\_\{t\}\\\|\_\{F\}\.\(13\)Thus, when‖Et‖F≤ε​‖W0‖F\\\|E\_\{t\}\\\|\_\{F\}\\leq\\varepsilon\\\|W\_\{0\}\\\|\_\{F\}withε<1\\varepsilon<1, every singular\-value statistic of the raw weight is observed through an initialization\-dominated background\. This does not say that the network is not learning: in lazy or NTK\-like regimes, function values can change while parameter displacement remains small\. It only says that a raw\-weight monitor has poor signal\-to\-noise for faults that first alter the update geometry\. The update incrementΔ​Wt,δ=Wt−Wt−δ\\Delta W\_\{t,\\delta\}=W\_\{t\}\-W\_\{t\-\\delta\}removesW0W\_\{0\}exactly, so spectral concentration in the update is not diluted by initialization energy\.

## Appendix BAccumulation Consequence for Flash Attention

This appendix proves the concentration form of the accumulation consequence \(Observation 1\) used for the monitoring window\. The source model is their per\-step mechanism: low\-precision FA supplies biased scalar coefficientseje\_\{j\}multiplying coherent rank\-one update atomsRj=Xj⊤​\(P​K\)jR\_\{j\}=X\_\{j\}^\{\\top\}\(PK\)\_\{j\}\. This appendix does not introduce a new FA failure mechanism; it only records the accumulation consequence used by theΔ​W\\Delta Wmonitor\.

Index token\-step samples byjjand setRj=Xj⊤​\(P​K\)j∈ℝd×dkR\_\{j\}=X\_\{j\}^\{\\top\}\(PK\)\_\{j\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\}\. AssumeYj=ej​Rj−MY\_\{j\}=e\_\{j\}R\_\{j\}\-Mare independent or martingale\-difference fluctuations withM=𝔼​\[ej​Rj\]M=\\mathbb\{E\}\[e\_\{j\}R\_\{j\}\]and𝔼​‖Yj‖F2≤ν2\\mathbb\{E\}\\\|Y\_\{j\}\\\|\_\{F\}^\{2\}\\leq\\nu^\{2\}\. Then

An:=∑j=1nej​Rj=n​M\+Zn,Zn:=∑j=1nYj\.A\_\{n\}:=\\sum\_\{j=1\}^\{n\}e\_\{j\}R\_\{j\}=nM\+Z\_\{n\},\\qquad Z\_\{n\}:=\\sum\_\{j=1\}^\{n\}Y\_\{j\}\.\(14\)By orthogonality of the centered increments,

𝔼​‖Zn‖F2≤n​ν2\.\\mathbb\{E\}\\\|Z\_\{n\}\\\|\_\{F\}^\{2\}\\leq n\\nu^\{2\}\.\(15\)Markov’s inequality implies‖Zn‖F=Op​\(n\)\\\|Z\_\{n\}\\\|\_\{F\}=O\_\{p\}\(\\sqrt\{n\}\), hence‖Zn‖2=Op​\(n\)\\\|Z\_\{n\}\\\|\_\{2\}=O\_\{p\}\(\\sqrt\{n\}\)\. This proves Equation \([1](https://arxiv.org/html/2606.28116#S3.E1)\)\. The biased meann​MnMgrows linearly, while the zero\-mean residual grows sublinearly\.

Finally, supposeMMhas rankrrand singular gapσr​\(M\)\>0\\sigma\_\{r\}\(M\)\>0\. Weyl’s inequality gives

\|σi​\(An\)−n​σi​\(M\)\|≤‖Zn‖2\.\|\\sigma\_\{i\}\(A\_\{n\}\)\-n\\sigma\_\{i\}\(M\)\|\\leq\\\|Z\_\{n\}\\\|\_\{2\}\.\(16\)Whenn​σr​\(M\)≫‖Zn‖2n\\sigma\_\{r\}\(M\)\\gg\\\|Z\_\{n\}\\\|\_\{2\}, the toprrsingular values ofAnA\_\{n\}are controlled byMMand the remaining singular values are residual\-scale\. Consequently, stable rank and singular\-spectrum entropy of the accumulated update converge toward those of the low\-rank meanMM\. This is the formal sense in which biased low\-precision arithmetic becomes a low\-rankΔ​W\\Delta Wfault only under coherence of the rank\-one atoms\.

For the biased rounding\-error injection used in our experiments, each selected BF16 entry is reinterpreted as its unsigned 16\-bit storage worduu\. The implementation computes

u~=\(u≫n\)≪n,\\tilde\{u\}=\(u\\gg n\)\\ll n,\(17\)and then reinterpretsu~\\tilde\{u\}as a BF16 value before the backward expression consumes it\. This low\-bit masking operation removes the lowestnnstorage bits, thereby discarding low\-order significand information and inducing a deterministic biased rounding error rather than additive real\-valued noise\. For example, withn=3n=3,

original value​1\.1015625:0\|01111111\|0001101BF16=0x3f8d↓\(uint16 \>\> 3\) << 3attacked value​1\.0625:0\|01111111\|0001000BF16=0x3f88\.\\begin\{array\}\[\]\{rcl\}\\text\{original value \}1\.1015625&:&\\text\{\{0\|01111111\|0001101\}\}\_\{\\mathrm\{BF16\}\}=\\text\{\{0x3f8d\}\}\\\\\[2\.0pt\] &&\\downarrow\\ \\text\{\{\(uint16 \>\> 3\) << 3\}\}\\\\\[2\.0pt\] \\text\{attacked value \}1\.0625&:&\\text\{\{0\|01111111\|0001000\}\}\_\{\\mathrm\{BF16\}\}=\\text\{\{0x3f88\}\}\.\\end\{array\}\(18\)The exact tensor and mask width are experimental knobs, but the resulting tensor\-level error has the same algebraic role\. If the backward path uses attacked tensorsO^=O\+EO\\widehat\{O\}=O\+E\_\{O\}andd​O^=d​O\+Ed​O\\widehat\{dO\}=dO\+E\_\{dO\}, the scalar source can include perturbations to bothOOandd​OdO:

δ^−δ=rowsum⁡\(d​O⊙EO\+Ed​O⊙O\+Ed​O⊙EO\)\.\\widehat\{\\delta\}\-\\delta=\\operatorname\{rowsum\}\(dO\\odot E\_\{O\}\+E\_\{dO\}\\odot O\+E\_\{dO\}\\odot E\_\{O\}\)\.\(19\)This implementation\-specific expansion only changes what contributes to the scalareje\_\{j\}\. Perturbingd​OdOalso induces a direct perturbation ofd​P=d​O​V⊤dP=dOV^\{\\top\}, so the main text uses only the abstract source formej​Rje\_\{j\}R\_\{j\}rather than treating theO,d​OO,dOexpansion as a separate mechanism\.

## Appendix CRouter Similarity Bound

#### Proof of the conditioning\-ratio similarity bound\.

Let

ε=maxi⁡‖wi−w¯‖‖w¯‖,ui=wi‖wi‖,e=w¯‖w¯‖\.\\varepsilon=\\frac\{\\max\_\{i\}\\\|w\_\{i\}\-\\bar\{w\}\\\|\}\{\\\|\\bar\{w\}\\\|\},\\qquad u\_\{i\}=\\frac\{w\_\{i\}\}\{\\\|w\_\{i\}\\\|\},\\qquad e=\\frac\{\\bar\{w\}\}\{\\\|\\bar\{w\}\\\|\}\.The statement is meaningful whenw¯≠0\\bar\{w\}\\neq 0and all router columns are nonzero, which we assume below\. Writing the pairwise similarity as the average over ordered distinct pairs,

sim⁡\(WR\)=1n​\(n−1\)​∑i≠jui⊤​uj=‖∑iui‖2−nn​\(n−1\)\.\\operatorname\{sim\}\(W\_\{R\}\)=\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}u\_\{i\}^\{\\top\}u\_\{j\}=\\frac\{\\left\\\|\\sum\_\{i\}u\_\{i\}\\right\\\|^\{2\}\-n\}\{n\(n\-1\)\}\.\(20\)Ifε≥1\\varepsilon\\geq 1, then Equation \([20](https://arxiv.org/html/2606.28116#A3.E20)\) givessim⁡\(WR\)≥−1/\(n−1\)\\operatorname\{sim\}\(W\_\{R\}\)\\geq\-1/\(n\-1\), and since1−nn−1​ε2≤1−nn−1=−1/\(n−1\)1\-\\tfrac\{n\}\{n\-1\}\\varepsilon^\{2\}\\leq 1\-\\tfrac\{n\}\{n\-1\}=\-1/\(n\-1\)\(usingε2≥1\\varepsilon^\{2\}\\geq 1\), the desired bound follows\. It remains to consider0≤ε<10\\leq\\varepsilon<1\. Letθi\\theta\_\{i\}be the angle betweenwiw\_\{i\}andw¯\\bar\{w\}\. Because‖wi−w¯‖≤ε​‖w¯‖\\\|w\_\{i\}\-\\bar\{w\}\\\|\\leq\\varepsilon\\\|\\bar\{w\}\\\|, the pointwiw\_\{i\}lies in the ball of radiusε​‖w¯‖\\varepsilon\\\|\\bar\{w\}\\\|aroundw¯\\bar\{w\}\. This ball does not contain the origin, soθi<π/2\\theta\_\{i\}<\\pi/2\. For an acute ray making angleθi\\theta\_\{i\}withw¯\\bar\{w\}, the closest point on that ray tow¯\\bar\{w\}has distance‖w¯‖​sin⁡θi\\\|\\bar\{w\}\\\|\\sin\\theta\_\{i\}; hencesin⁡θi≤ε\\sin\\theta\_\{i\}\\leq\\varepsilonand

e⊤​ui=cos⁡θi≥1−ε2\.e^\{\\top\}u\_\{i\}=\\cos\\theta\_\{i\}\\geq\\sqrt\{1\-\\varepsilon^\{2\}\}\.Therefore

‖∑iui‖≥e⊤​∑iui≥n​1−ε2\.\\left\\\|\\sum\_\{i\}u\_\{i\}\\right\\\|\\geq e^\{\\top\}\\sum\_\{i\}u\_\{i\}\\geq n\\sqrt\{1\-\\varepsilon^\{2\}\}\.Substituting this into Equation \([20](https://arxiv.org/html/2606.28116#A3.E20)\) yields

sim⁡\(WR\)≥n2​\(1−ε2\)−nn​\(n−1\)=1−nn−1​ε2,\\operatorname\{sim\}\(W\_\{R\}\)\\geq\\frac\{n^\{2\}\(1\-\\varepsilon^\{2\}\)\-n\}\{n\(n\-1\)\}=1\-\\frac\{n\}\{n\-1\}\\varepsilon^\{2\},which proves the proposition\.□\\square

## Appendix DThe Router Weight–Entropy Link

We derive Equation \([9](https://arxiv.org/html/2606.28116#S4.E9)\)\. Letz=WR⊤​x∈ℝnz=W\_\{R\}^\{\\top\}x\\in\\mathbb\{R\}^\{n\}be the expert logits andδ=\(In−1n​𝟏𝟏⊤\)​z=WR,c⊤​x\\delta=\(I\_\{n\}\-\\tfrac\{1\}\{n\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\)z=W\_\{R,c\}^\{\\top\}xthe centered logits \(𝟏⊤​δ=0\\mathbf\{1\}^\{\\top\}\\delta=0\); softmax is invariant under the centering shift, sop=softmax​\(δ\)p=\\mathrm\{softmax\}\(\\delta\)\.

Lemma \(local expansion\)\.For centeredδ\\delta,

ℋ​\(softmax​\(δ\)\)=log⁡n−‖δ‖222​n\+O​\(‖δ‖3\)\.\\mathcal\{H\}\\bigl\(\\mathrm\{softmax\}\(\\delta\)\\bigr\)=\\log n\-\\frac\{\\\|\\delta\\\|\_\{2\}^\{2\}\}\{2n\}\+O\(\\\|\\delta\\\|^\{3\}\)\.\(21\)
Proof\.WithZ=∑jeδjZ=\\sum\_\{j\}e^\{\\delta\_\{j\}\}and∑jδj=0\\sum\_\{j\}\\delta\_\{j\}=0, expanding to second order givesZ=n\+12​‖δ‖22\+O​\(‖δ‖3\)Z=n\+\\tfrac\{1\}\{2\}\\\|\\delta\\\|\_\{2\}^\{2\}\+O\(\\\|\\delta\\\|^\{3\}\), hencelog⁡Z=log⁡n\+‖δ‖22/\(2​n\)\+O​\(‖δ‖3\)\\log Z=\\log n\+\\\|\\delta\\\|\_\{2\}^\{2\}/\(2n\)\+O\(\\\|\\delta\\\|^\{3\}\)\. Writingℋ=log⁡Z−∑ipi​δi\\mathcal\{H\}=\\log Z\-\\sum\_\{i\}p\_\{i\}\\delta\_\{i\}and expandingpi=eδi/Zp\_\{i\}=e^\{\\delta\_\{i\}\}/Zto the same order gives∑ipi​δi=‖δ‖22/n\+O​\(‖δ‖3\)\\sum\_\{i\}p\_\{i\}\\delta\_\{i\}=\\\|\\delta\\\|\_\{2\}^\{2\}/n\+O\(\\\|\\delta\\\|^\{3\}\)\. Subtracting yields Equation \([21](https://arxiv.org/html/2606.28116#A4.E21)\)\. Equivalently, the Hessian of−ℋ\-\\mathcal\{H\}at the uniform point is1n​\(In−1n​𝟏𝟏⊤\)\\tfrac\{1\}\{n\}\(I\_\{n\}\-\\tfrac\{1\}\{n\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\), i\.e\.,I/nI/nrestricted to the centered subspace\.□\\square

Corollary \(weight–entropy link\)\.Substitutingδ=WR,c⊤​x\\delta=W\_\{R,c\}^\{\\top\}xand taking expectations over the token distribution,

log⁡n−𝔼x​ℋ​\(p\)=12​n​tr​\(WR,c⊤​Mx​WR,c\)\+O​\(𝔼​‖δ‖3\),\\log n\-\\mathbb\{E\}\_\{x\}\\,\\mathcal\{H\}\(p\)=\\frac\{1\}\{2n\}\\,\\mathrm\{tr\}\\\!\\bigl\(W\_\{R,c\}^\{\\top\}M\_\{x\}\\,W\_\{R,c\}\\bigr\)\+O\\bigl\(\\mathbb\{E\}\\\|\\delta\\\|^\{3\}\\bigr\),\(22\)withMx=𝔼​\[x​x⊤\]M\_\{x\}=\\mathbb\{E\}\[xx^\{\\top\}\], since𝔼​‖WR,c⊤​x‖22=tr​\(WR,c⊤​Mx​WR,c\)\\mathbb\{E\}\\\|W\_\{R,c\}^\{\\top\}x\\\|\_\{2\}^\{2\}=\\mathrm\{tr\}\(W\_\{R,c\}^\{\\top\}M\_\{x\}W\_\{R,c\}\)\. Under second\-moment isotropyMx≈IdM\_\{x\}\\approx I\_\{d\}this reduces to the weight\-only quantity‖WR,c‖F2/\(2​n\)\\\|W\_\{R,c\}\\\|\_\{F\}^\{2\}/\(2n\)\.

Corollary \(mean\-zero logit perturbations\)\.LetF​\(z\)=ℋ​\(softmax​\(z\)\)F\(z\)=\\mathcal\{H\}\(\\mathrm\{softmax\}\(z\)\)andCn=In−1n​𝟏𝟏⊤C\_\{n\}=I\_\{n\}\-\\tfrac\{1\}\{n\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\. For centered logitsδ\\deltanear zero and a mean\-zero perturbationϵ\\epsilonwith𝔼ϵ​\[ϵ\]=0\\mathbb\{E\}\_\{\\epsilon\}\[\\epsilon\]=0,

𝔼ϵF\(δ\+ϵ\)=F\(δ\)−12​n𝔼ϵ∥Cnϵ∥22\+O\(∥δ,ϵ∥3\),\\mathbb\{E\}\_\{\\epsilon\}F\(\\delta\+\\epsilon\)=F\(\\delta\)\-\\frac\{1\}\{2n\}\\mathbb\{E\}\_\{\\epsilon\}\\\|C\_\{n\}\\epsilon\\\|\_\{2\}^\{2\}\+O\(\\\|\\delta,\\epsilon\\\|^\{3\}\),\(23\)where the remainder is local and third order in‖δ‖\+‖ϵ‖\\\|\\delta\\\|\+\\\|\\epsilon\\\|\.

Proof\.Softmax shift\-invariance givesF​\(z\)=F​\(Cn​z\)F\(z\)=F\(C\_\{n\}z\)\. Applying Equation \([21](https://arxiv.org/html/2606.28116#A4.E21)\) toδ\+ϵ\\delta\+\\epsilonand toδ\\deltagives

F\(δ\+ϵ\)−F\(δ\)=−12​n\(∥Cn\(δ\+ϵ\)∥22−∥Cnδ∥22\)\+O\(∥δ,ϵ∥3\)\.F\(\\delta\+\\epsilon\)\-F\(\\delta\)=\-\\frac\{1\}\{2n\}\\left\(\\\|C\_\{n\}\(\\delta\+\\epsilon\)\\\|\_\{2\}^\{2\}\-\\\|C\_\{n\}\\delta\\\|\_\{2\}^\{2\}\\right\)\+O\(\\\|\\delta,\\epsilon\\\|^\{3\}\)\.Expanding the quadratic term,

‖Cn​\(δ\+ϵ\)‖22−‖Cn​δ‖22=2​\(Cn​δ\)⊤​Cn​ϵ\+‖Cn​ϵ‖22\.\\\|C\_\{n\}\(\\delta\+\\epsilon\)\\\|\_\{2\}^\{2\}\-\\\|C\_\{n\}\\delta\\\|\_\{2\}^\{2\}=2\(C\_\{n\}\\delta\)^\{\\top\}C\_\{n\}\\epsilon\+\\\|C\_\{n\}\\epsilon\\\|\_\{2\}^\{2\}\.Taking expectation overϵ\\epsilon, the cross term vanishes because𝔼ϵ​\[Cn​ϵ\]=Cn​𝔼ϵ​\[ϵ\]=0\\mathbb\{E\}\_\{\\epsilon\}\[C\_\{n\}\\epsilon\]=C\_\{n\}\\mathbb\{E\}\_\{\\epsilon\}\[\\epsilon\]=0\. This yields Equation \([23](https://arxiv.org/html/2606.28116#A4.E23)\)\.□\\square

Scope\.Three caveats bound the use of this link\. First, RMSNorm fixes only the tracetr​\(Mx\)=d\\mathrm\{tr\}\(M\_\{x\}\)=d, not isotropy, so the weight\-only reduction is an extra assumption – this is the fourth limitation in the main text\. Second, the expansion is local: it is quantitative near uniform routing \(all logitsO​\(1\)O\(1\)\) and degrades to a qualitative, direction\-wise monotone statement in the collapsed regimeℋ→0\\mathcal\{H\}\\to 0, where the quadratic form underestimates the true entropy drop\. Third, Equation \([23](https://arxiv.org/html/2606.28116#A4.E23)\) is a second\-order perturbative explanation of why zero\-mean logit noise can lower expected entropy; it is not, by itself, a global proof of router collapse\. The link is therefore a consistency check between the weight\-side and decision\-side indicator families, not a substitute for either\.