The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path

# The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path
Source: [https://arxiv.org/html/2607.20484](https://arxiv.org/html/2607.20484)
###### Abstract

Large Language Models \(LLMs\) are fundamentally limited by representation collapse, a bottleneck that severely degrades long\-context performance\. We identify that existing approaches risk drifting into one of two pathological extremes: homogenization collapse \(e\.g\., attention sinks causing rank deficiency\) and isolation collapse \(e\.g\., local attention causing context disconnection\)\. Through spectral analysis of attention dynamics, we derive an intrinsic trade\-off between mixing efficiency \(spectral gap\) and information capacity \(effective rank\) that standard mechanisms struggle to balance\. To resolve this dilemma, we propose the Topologically Regularized Side\-Path \(TRSP\), a non\-invasive architectural intervention that achieves spectral balance\. TRSP employs a parameter\-free Triangular Box mechanism, scaled by a lightweight, length\-aware gate, to regularize the token interaction topology\. By integrating proximal coupling to preserve effective rank and distal propagation to support non\-degenerate mixing, TRSP promotes a geometrically healthier transition operator without altering core attention\. Experiments show significant improvements across general capabilities and long\-context benchmarks\. Notably, on NoLiMa at8×8\\timesthe training length, TRSP retains83%83\\%accuracy and surpasses the Differential Transformer and Gated Attention by approximately 30 and 50 percentage points, respectively\. Code available at:[https://github\.com/Eziotao\-tyd/TRSP](https://github.com/Eziotao-tyd/TRSP)\.

Machine Learning, ICML

![Refer to caption](https://arxiv.org/html/2607.20484v1/x1.png)Figure 1:Analysis of representation collapse\.\(a\) Token Similarity𝒮\\mathcal\{S\}\. Attention may degenerate into homogenization collapse \(indistinguishable tokens\) or isolation collapse \(context failure\)\. \(b\) Transition Operatorℳ\\mathcal\{M\}\. Homogenization may stem from over\-mixing or an attention sink, while isolation arises from local attention\. \(c\) Topological Connectivity\. These pathologies originate from fully connected/centralized star graphs versus disconnected graphs, respectively\. By applying spectrally guided topology regularization, our approach constructs a non\-degenerate transition operatorℳ\\mathcal\{M\}that effectively mitigates representation collapse\.![Refer to caption](https://arxiv.org/html/2607.20484v1/x2.png)Figure 2:Achieving Spectral Balance via Topological Regularization\.\(a\) Spectral Trade\-off\. Analysis ofℳ\\mathcal\{M\}reveals an intrinsic trade\-off between Information Capacity \(measured by effective rank,ReffR\_\{\\text\{eff\}\}\) and Mixing Efficiency \(measured by spectral gap,γ\\gamma\)\. The patterns in Fig\.[1](https://arxiv.org/html/2607.20484#S0.F1)collapse into either homogenization \(highγ\\gamma, lowReffR\_\{\\text\{eff\}\}\) or isolation \(lowγ\\gamma, highReffR\_\{\\text\{eff\}\}\)\. We require spectral balance to resolve this dilemma\. \(b\) We introduce the Topologically Regularized Side\-Path \(TRSP\) as a non\-invasive regularizer\. It induces a topology in which proximal interactions in shallow layers preserve feature distinctiveness \(Rank\-Preserving\), while distal propagation in deep layers supports global mixing \(Gap\-Preserving\)\.## 1Introduction

Large Language Models \(LLMs\) achieve strong natural language understanding and generation, yet they suffer from a fundamental pathology of representation collapse, identified in studies of over\-mixing and rank collapse\(Donget al\.,[2021](https://arxiv.org/html/2607.20484#bib.bib9); Nociet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib14); Wuet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib10); Barberoet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib51); Saadaet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib11)\)by analyzing how information propagates through transformers: as the context length increases \(inwidth\), repeated mixing by the attention mechanism \(indepth\) drives token representations toward a low\-dimensional, uninformative space\.

In parallel with these theoretical findings, empirical research has explored countermeasures\. First, the ubiquitous attention sink phenomenon\(Xiaoet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib1); Guet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib52)\)—in which massive attention weight is allocated to initial or special tokens—has been interpreted as a natural mechanism to arrest the aforementioned over\-mixing\(Barberoet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib53)\)\. Second, architectural designs based on local or sliding\-window attention\(Jianget al\.,[2023](https://arxiv.org/html/2607.20484#bib.bib76); Teamet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib77); Beltagyet al\.,[2020](https://arxiv.org/html/2607.20484#bib.bib25); Child,[2019](https://arxiv.org/html/2607.20484#bib.bib27)\)restrict the per\-layer receptive field to reduce the cost of long\-context attention, thereby limiting the immediate scope of token mixing\. We argue that both countermeasures avoid one failure mode at the cost of another\. Under a unified spectral lens, we categorize these phenomena as two pathological extremes of representation collapse: \(1\) homogenization collapse \(e\.g\., over\-mixing or the attention sink\), characterized by critically low effective rank\(Wuet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib10)\); and \(2\) isolation collapse, characterized by critically low contextual coherence\(Ethayarajh,[2019](https://arxiv.org/html/2607.20484#bib.bib16)\), in which mixing fails to bridge distant tokens \(e\.g\., truncated local attention\)\.

We visualize these pathologies in Figure[1](https://arxiv.org/html/2607.20484#S0.F1)\. First, the token similarity matrices in \(a\) show that tokens either degenerate into indistinguishable noise \(homogenization\) or fail to capture context \(isolation\)\. Second, we trace this to the transition operatorℳ\\mathcal\{M\}in \(b\), where𝐗out=ℳ⋅𝐗in\\mathbf\{X\}\_\{\\text\{out\}\}=\\mathcal\{M\}\\cdot\\mathbf\{X\}\_\{\\text\{in\}\}\. The homogenization pattern appears as a dense or sink\-dominated operator, whereas isolation corresponds to a banded operator\. Both patterns fail to support effective information flow\. Third, inter\-layer topology analysis in \(c\) links these behaviors to token connectivity: homogenization stems from fully connected or centralized star graphs, whereas isolation stems from disconnected graphs\. This raises a more fundamental question: what spectral conditions characterize a non\-degenerate operatorℳ\\mathcal\{M\}, and why do standard mechanisms fail to satisfy them simultaneously?

To address this, we analyze the spectral properties of the row\-normalized operatorℳ\\mathcal\{M\}via singular value decomposition, as shown in Figure[2](https://arxiv.org/html/2607.20484#S0.F2)\(a\)\. We use two metrics whose maximization is desirable: effective rank \(Reff=∑σi2R\_\{\\text\{eff\}\}=\\sum\\sigma\_\{i\}^\{2\}\)\(Rudelson and Vershynin,[2007](https://arxiv.org/html/2607.20484#bib.bib70); Roy and Vetterli,[2007](https://arxiv.org/html/2607.20484#bib.bib17)\), which measures information capacity via the heaviness of the singular value distribution’s tail, and spectral gap \(γ=1−σ2\\gamma=1\-\\sigma\_\{2\}\), which measures mixing efficiency via the separation of the leading singular values\.111For notational simplicity, throughout this discussionσi\\sigma\_\{i\}denotes the singular values normalized by the largest singular value ofℳ\\mathcal\{M\}; henceσ1=1\\sigma\_\{1\}=1\.We derive an intrinsic relationship between them:Reff=1\+\(1−γ\)2\+∑i=3Nσi2\.R\_\{\\text\{eff\}\}=1\+\(1\-\\gamma\)^\{2\}\+\\sum\_\{i=3\}^\{N\}\\sigma\_\{i\}^\{2\}\.This reveals a fundamental spectral trade\-off: maximizingγ\\gammainevitably suppressesReffR\_\{\\text\{eff\}\}unless the tail \(∑i=3Nσi2\\sum\_\{i=3\}^\{N\}\\sigma\_\{i\}^\{2\}\) is explicitly preserved\. Empirically, models are often trapped in this dilemma: homogenization \(highγ\\gamma, lowReffR\_\{\\text\{eff\}\}\), where strong mixing collapses the manifold, or isolation \(lowγ\\gamma, highReffR\_\{\\text\{eff\}\}\), where weak mixing preserves dimensionality but halts propagation\.

To resolve this trade\-off and target spectral balance \(i\.e\., jointly improvingReffR\_\{\\text\{eff\}\}andγ\\gammawithout collapsing to either extreme\), we propose the Topologically Regularized Side\-Path \(TRSP\)\. We retain the standard attention computation and augment the transition operator with an additive side\-path\. This design injects topological regularization without disrupting the core attention mechanism\. TRSP enforces a hierarchy that combines proximal coupling \(to preserve the tail andReffR\_\{\\text\{eff\}\}\) with distal shortcuts \(to support highγ\\gamma\)\. Specifically, TRSP introduces two key components: \(1\) the Triangular Box \(“triBox”\) mechanism, a parameter\-free operator implemented via cascaded causal box filters with exponential bandwidth expansion\. It creates a seamless scale transition from proximal interactions to distal connections across layers\. \(2\) A long\-context gate, a lightweight gain controller\. With only≈50\\approx 50learnable parameters, the gate regulates side\-path strength from context length and layer depth via the coverage ratio, calibrating triBox injection across scales\.

We conducted extensive experiments evaluating general capabilities in post\-training scenarios \(MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.20484#bib.bib45)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2607.20484#bib.bib46)\)\) and long\-context extrapolation in both pre\- and post\-training settings \(RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib41)\), NoLiMa\([Modarressiet al\.,](https://arxiv.org/html/2607.20484#bib.bib49)\)\)\. Our results confirm that TRSP effectively corrects spectral anomalies and consistently outperforms strong baselines\. Notably, on NoLiMa, TRSP retains 83% accuracy at 8×\\timesthe training length, surpassing the Differential Transformer\(Yeet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib69)\)and Gated Attention\(Qiuet al\.,[2026](https://arxiv.org/html/2607.20484#bib.bib47)\)by approximately 30 and 50 percentage points, respectively\.

## 2Related Work

### 2\.1Representation Collapse and Spectral Analysis

Prior work has analyzed Transformer representation collapse along two complementary axes: layer depth and context length\.Along the depth axis,Donget al\.\([2021](https://arxiv.org/html/2607.20484#bib.bib9)\)show that pure attention loses rank doubly exponentially with depth, driving representations toward a rank\-1 subspace\. This pathology has been linked to vanishing gradients, signal\-propagation failure, and representation degeneration\(Nociet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib14); Heet al\.,[2023](https://arxiv.org/html/2607.20484#bib.bib59); Wuet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib10); Geshkovskiet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib58); Saadaet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib11); Gaoet al\.,[2019](https://arxiv.org/html/2607.20484#bib.bib15)\)\. This phenomenon parallels over\-smoothing in Graph Neural Networks\(Keriven,[2022](https://arxiv.org/html/2607.20484#bib.bib62); Wuet al\.,[2023](https://arxiv.org/html/2607.20484#bib.bib61)\)and has been explicitly formalized within Transformers, where self\-attention is shown to behave as a low\-pass filter that homogenizes features across layers\(Shiet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib60); Wanget al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib71); Nguyenet al\.,[2023](https://arxiv.org/html/2607.20484#bib.bib72)\)\. A complementary perspective shows that whether such smoothing is unavoidable depends on the eigenspectrum of the value and projection weights, leaving room for structural intervention\(Dovononet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib73)\)\.Along the length axis,Veličkovićet al\.\([2025](https://arxiv.org/html/2607.20484#bib.bib63)\)show that, under bounded\-logit assumptions, increasing context length can drive softmax attention toward uniform mixing, whileBarberoet al\.\([2024](https://arxiv.org/html/2607.20484#bib.bib51)\)identify an over\-squashing pathology in causal architectures, in which the unidirectional information flow renders the final\-token representations of distinct sequences arbitrarily close\. These studies primarily diagnose collapse via spectral and geometric tools\(Roy and Vetterli,[2007](https://arxiv.org/html/2607.20484#bib.bib17); Penningtonet al\.,[2017](https://arxiv.org/html/2607.20484#bib.bib20); Ethayarajh,[2019](https://arxiv.org/html/2607.20484#bib.bib16)\), or attempt to mitigate it through targeted modifications of attention masks, normalization, or weights\. We instead \(i\) unify these homogenization phenomena \(both depth\- and length\-wise\) with the opposite extreme of isolation collapse as two ends of a single trade\-off between effective rank and spectral gap, and \(ii\) propose an explicit topological regularizer to balance both simultaneously\.

### 2\.2Long\-Context Modeling and Attention Sink

Long\-context modeling remains a central concern for LLMs, as models often exhibit position\-dependent context utilization, performing worse on information located in the middle of long sequences\(Liuet al\.,[2024b](https://arxiv.org/html/2607.20484#bib.bib50)\)\.

Architectural and computational approaches\.Early work explored recurrence\(Daiet al\.,[2019](https://arxiv.org/html/2607.20484#bib.bib24)\)or sparse attention patterns, including the Sparse Transformer\(Child,[2019](https://arxiv.org/html/2607.20484#bib.bib27)\), Longformer\(Beltagyet al\.,[2020](https://arxiv.org/html/2607.20484#bib.bib25)\), and BigBird\(Zaheeret al\.,[2020](https://arxiv.org/html/2607.20484#bib.bib26)\)\. Recent advances focus on efficient computation, including FlashAttention and its successors\(Daoet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib31); Shahet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib64)\)and Ring Attention\(Liuet al\.,[2024a](https://arxiv.org/html/2607.20484#bib.bib33)\), alongside KV cache compression techniques\(Zhanget al\.,[2023](https://arxiv.org/html/2607.20484#bib.bib32); Geet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib6); Wu and Tu,[2024](https://arxiv.org/html/2607.20484#bib.bib67)\)\. Positional encodings and extension strategies such as ALiBi\(Presset al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib29)\), RoPE\(Suet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib28)\), and YaRN\(Penget al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib30)\)facilitate length extrapolation but are largely orthogonal to the spectral degradation we focus on\. Beyond softmax\-attention Transformers, alternative architectures such as state\-space models\(Gu and Dao,[2024](https://arxiv.org/html/2607.20484#bib.bib74)\)and recurrence\-based linear models\(Penget al\.,[2023](https://arxiv.org/html/2607.20484#bib.bib75)\)replace attention with subquadratic primitives; in this work, we focus on improving the spectral behavior of the standard softmax\-attention Transformer rather than replacing it\.

Attention sink and attention modifications\.The attention sink phenomenon\(Xiaoet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib1)\), in which massive weight is allocated to the initial token, has prompted competing interpretations\. One line of work views sinks as pathological artifacts tied to massive activations\(Sunet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib3)\), activation outliers\(Kaulet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib8)\), or softmax\-induced first\-token bias\(Guet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib52)\), and proposes mitigations via softmax reformulations such as sigmoid attention\(Ramapuramet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib65)\)and softmax\-1\(Kaulet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib8)\), or via gating mechanisms\(Bondarenkoet al\.,[2023](https://arxiv.org/html/2607.20484#bib.bib66)\)\. A second line actively leverages sinks for streaming inference\(Hanet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib57); Xiaoet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib1)\)\.Barberoet al\.\([2025](https://arxiv.org/html/2607.20484#bib.bib53)\)reinterpret sinks as a learned mechanism by which deep Transformers arrest over\-mixing\. We argue that sinks replace one homogenization mode \(uniform over\-mixing\) with another \(first\-token concentration\), both characterized by low effective rank\. Closely related to our work, recent attention modifications—Differential Transformer\(Yeet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib69)\), which cancels attention noise via the difference of two softmax maps, and Gated Attention\(Qiuet al\.,[2026](https://arxiv.org/html/2607.20484#bib.bib47)\), which applies a query\-dependent sigmoid gate after the SDPA output to eliminate sinks—reshape the attention computation itself\. In contrast, TRSP introduces a non\-invasive side\-path that targets the spectral structure of the transition operatorℳ\\mathcal\{M\}without altering standard attention, making it complementary to these methods and readily composable with existing architectures\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2607.20484v1/x3.png)Figure 3:Overview of TRSP\.Left: The overall architecture showing TRSP as a parallel branch\. Middle: Detailed implementations of the Long\-Context Gate \(top\) and the triBox mechanism \(bottom\)\. Right: The hierarchical connectivity pattern showing how bandwidths expand exponentially \(2ℓ2^\{\\ell\}\) across layers\.### 3\.1Overview

As analyzed in §[1](https://arxiv.org/html/2607.20484#S1), we introduce the Topologically Regularized Side\-Path \(TRSP\), a lightweight branch added in parallel to the standard attention layer\. As illustrated in Figure[3](https://arxiv.org/html/2607.20484#S3.F3), TRSP applies a causal triangular filter with a layer\-dependent bandwidth to the hidden states and adds the gated output back to the residual stream, leaving the attention computation unchanged\. The triBox branch provides a fixed multi\-scale route for long\-range signal propagation, and the long\-context gate controls its strength based on context coverage\. This side\-path serves as a non\-invasive structural bias toward spectral balance\.

### 3\.2The Triangular Box \(triBox\) Mechanism

We design triBox to regularize the layer\-wise mixing operatorℳ\\mathcal\{M\}towards spectral balance: sufficient mixing to avoid isolation and sufficient capacity to avoid homogenization\. Concretely, we target a non\-vanishing spectral gap while preventing the singular spectrum ofℳ\\mathcal\{M\}from collapsing to rank one\. Below, we relate this goal to the two collapse modes in §[1](https://arxiv.org/html/2607.20484#S1)via two quantities\.

Mixing efficiency \(spectral gap\)\.For a row\-normalized transition operator, the spectral gapγ​\(ℳ\)=1−σ2\\gamma\(\\mathcal\{M\}\)=1\-\\sigma\_\{2\}quantifies how quickly non\-stationary components contract, thereby indicating the strength of global token mixing\(Levin and Peres,[2017](https://arxiv.org/html/2607.20484#bib.bib13)\)\. A vanishing gap indicates weak long\-range mixing and aligns with isolation collapse\. In theory, we studyγ\\gammaonℳ\\mathcal\{M\}; in experiments, we report the Signal Propagation Rate \(SPR\) as the perturbation gain‖δout‖/‖δin‖\\\|\\delta\_\{\\mathrm\{out\}\}\\\|/\\\|\\delta\_\{\\mathrm\{in\}\}\\\|, used as an empirical proxy for propagation strength, following Transformer signal\-propagation studies\(Nociet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib14); Saadaet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib11)\)and sensitivity viewpoints\(Gouket al\.,[2021](https://arxiv.org/html/2607.20484#bib.bib19)\)\.

Information capacity \(effective rank\)\.Even with adequate mixing, representations can still collapse if energy concentrates on a few singular directions—the hallmark of homogenization collapse\. We therefore monitor the spread of the singular spectrum ofℳ\\mathcal\{M\}\. Following\(Roy and Vetterli,[2007](https://arxiv.org/html/2607.20484#bib.bib17)\), we use effective rank as a continuous notion of dimensionality; in this paper we quantify it by the stable rankRstab​\(ℳ\)=‖ℳ‖F2/‖ℳ‖22R\_\{\\text\{stab\}\}\(\\mathcal\{M\}\)=\\\|\\mathcal\{M\}\\\|\_\{F\}^\{2\}/\\\|\\mathcal\{M\}\\\|\_\{2\}^\{2\}\(Cohenet al\.,[2015](https://arxiv.org/html/2607.20484#bib.bib21)\), which coincides withReff=∑iσi2R\_\{\\text\{eff\}\}=\\sum\_\{i\}\\sigma\_\{i\}^\{2\}in §[1](https://arxiv.org/html/2607.20484#S1)under singular value normalizationσ1=1\\sigma\_\{1\}=1\.

#### 3\.2\.1Implementation via Cascaded Box Filters

triBox is a channel\-wise causal triangular convolution implemented by cascading two causal box filters \(moving averages\), rather than via dense matrix multiplication\. Let𝐗∈ℝT×d\\mathbf\{X\}\\in\\mathbb\{R\}^\{T\\times d\}denote the token hidden states and letb∈ℕb\\in\\mathbb\{N\}be the box window length \(i\.e\., bandwidth\)\. For each channel independently, the causal box filter computes a length\-normalized moving average over the pastbbpositions, using only thet\+1t\{\+\}1tokens available at timettwhent<bt<b:

Boxb​\(𝐗\)t=1wt​∑i=0wt−1𝐗t−i,\\mathrm\{Box\}\_\{b\}\(\\mathbf\{X\}\)\_\{t\}=\\frac\{1\}\{w\_\{t\}\}\\sum\_\{i=0\}^\{w\_\{t\}\-1\}\\mathbf\{X\}\_\{t\-i\},\(1\)wherewt=min⁡\{b,t\+1\}w\_\{t\}=\\min\\\{b,\\,t\{\+\}1\\\}is the effective window size\. A naive sliding window costsO​\(T​b\)O\(Tb\); we instead maintain prefix sums𝐒0=𝟎\\mathbf\{S\}\_\{0\}=\\mathbf\{0\}and𝐒t\+1=𝐒t\+𝐗t\\mathbf\{S\}\_\{t\+1\}=\\mathbf\{S\}\_\{t\}\+\\mathbf\{X\}\_\{t\}\. With start indexat=max⁡\{0,t−b\+1\}a\_\{t\}=\\max\\\{0,\\,t\{\-\}b\{\+\}1\\\}, the same filter is evaluated inO​\(1\)O\(1\)per token as

Boxb​\(𝐗\)t=𝐒t\+1−𝐒att−at\+1,\\mathrm\{Box\}\_\{b\}\(\\mathbf\{X\}\)\_\{t\}=\\frac\{\\mathbf\{S\}\_\{t\+1\}\-\\mathbf\{S\}\_\{a\_\{t\}\}\}\{t\-a\_\{t\}\+1\},\(2\)which is algebraically identical to Eq\. \([1](https://arxiv.org/html/2607.20484#S3.E1)\) and costsO​\(T​d\)O\(Td\)for anybb\. triBox applies the box filter twice:

triBoxb​\(𝐗\)=Boxb​\(Boxb​\(𝐗\)\)\.\\mathrm\{triBox\}\_\{b\}\(\\mathbf\{X\}\)=\\mathrm\{Box\}\_\{b\}\\\!\\big\(\\mathrm\{Box\}\_\{b\}\(\\mathbf\{X\}\)\\big\)\.\(3\)Whent≥2​b−2t\\geq 2b\{\-\}2both box windows are full, the cascade is equivalent to a single causal triangular convolution,

triBoxb​\(𝐗\)t=∑r=02​b−2hb​\(r\)​𝐗t−r,\\mathrm\{triBox\}\_\{b\}\(\\mathbf\{X\}\)\_\{t\}=\\sum\_\{r=0\}^\{2b\-2\}h\_\{b\}\(r\)\\,\\mathbf\{X\}\_\{t\-r\},\(4\)with weightshb​\(r\)=\(b−\|r−\(b−1\)\|\)/b2h\_\{b\}\(r\)=\\bigl\(b\-\\lvert r\-\(b\-1\)\\rvert\\bigr\)/b^\{2\}that decay linearly toward the past and sum to one on\{0,…,2​b−2\}\\\{0,\\ldots,2b\{\-\}2\\\}, yielding a smoother frequency response than a single rectangular window\.

#### 3\.2\.2Dynamic Bandwidth Expansion

To cover multiple scales without making each layer dense, the box\-filter window length grows exponentially with depth\. For layerℓ∈\{0,…,D−1\}\\ell\\in\\\{0,\\ldots,D\{\-\}1\\\}, we set

bℓ=min⁡\{2ℓ,T\},b\_\{\\ell\}=\\min\\\{2^\{\\ell\},\\,T\\\},\(5\)whereTTis the current sequence length andDDis the number of layers\. At layerℓ\\ell, triBox is a local triangular mixer with bandwidthbℓb\_\{\\ell\}, which limits homogenization within its causal receptive field\. Across layers, the dyadic schedulebℓ=2ℓb\_\{\\ell\}=2^\{\\ell\}superposes connections at scales\{±2ℓ\}\\\{\\pm 2^\{\\ell\}\\\}, inducing a sparse Cayley graph onℤT\\mathbb\{Z\}\_\{T\}\(Figure[3](https://arxiv.org/html/2607.20484#S3.F3), Right\)\. Shallow layers therefore realize proximal coupling \(preservingReffR\_\{\\text\{eff\}\}\), while the stacked topology provides long\-range shortcuts that support global mixing \(boundingγ\\gamma\), matching the division in Figure[2](https://arxiv.org/html/2607.20484#S0.F2)\.

#### 3\.2\.3Spectral Properties

We now state the rank and gap guarantees that the triBox topology confers on the composite operatorℳ\\mathcal\{M\}; full proofs are deferred to Appendix[C](https://arxiv.org/html/2607.20484#A3), and their link to the model’s inference\-error bound to Appendix[D](https://arxiv.org/html/2607.20484#A4)\.

Rank lower bound\.Standard attention can concentrate energy on a single sink token, approaching a rank\-one matrix\. The triangular kernel instead spreads energy across its window, so the Frobenius energy of the triBox operator grows linearly with length,‖Mtri‖F2=Θ​\(T\)\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{F\}^\{2\}=\\Theta\(T\)\. This scaling lower\-bounds the effective rank ofℳ\\mathcal\{M\}away from one even under sink collapse \(Appendix[C\.4](https://arxiv.org/html/2607.20484#A3.SS4)\), so the side\-path preserves usable dimensionality asTTgrows\.

Non\-degenerate mixing\.Superposing the dyadic bandwidths across layers wires a cyclic Cayley graph onℤT\\mathbb\{Z\}\_\{T\}, whose algebraic connectivity—the unnormalized spectral gapμ2\\mu\_\{2\}, i\.e\., the smallest non\-zero Laplacian eigenvalue—is constant inTT\(Appendix[C\.3](https://arxiv.org/html/2607.20484#A3.SS3)\)\. This secures robust absolute energy flow, while the corresponding normalized gap decays only asΘ​\(1/log⁡T\)\\Theta\(1/\\log T\)\. A dense variant offsets this decay through depth but costsO​\(T​d​log⁡T\)O\(Td\\log T\)per layer, so we keep the sparse design, retaining the constantμ2\\mu\_\{2\}atO​\(T​d\)O\(Td\)complexity\.

### 3\.3The Long\-Context Gate

triBox fixes the side\-path topology and per\-layer mixing geometry \(§[3\.2](https://arxiv.org/html/2607.20484#S3.SS2)\)\. A fixed scalar injection into the residual stream is brittle: shallow layers with smallbℓb\_\{\\ell\}can over\-smooth short contexts, and different\(T,ℓ\)\(T,\\ell\)pairs require different side\-path gains relative to attention \(replacinggℓg\_\{\\ell\}with a static scalar reduces MMLU to 21\.16%; §[5](https://arxiv.org/html/2607.20484#S5)\)\. We introduce the long\-context gate, a global, input\-agnostic gaingℓg\_\{\\ell\}that depends only on the coverage ratiorℓ=bℓ/Tr\_\{\\ell\}=b\_\{\\ell\}/T\.

#### 3\.3\.1Formulation

Theℓ\\ell\-th layer update is

𝐗out=𝐗in\+Attn​\(𝐗in\)\+gℓ​triBoxbℓ​\(𝐗in\),\\mathbf\{X\}\_\{\\mathrm\{out\}\}=\\mathbf\{X\}\_\{\\mathrm\{in\}\}\+\\mathrm\{Attn\}\(\\mathbf\{X\}\_\{\\mathrm\{in\}\}\)\+g\_\{\\ell\}\\,\\mathrm\{triBox\}\_\{b\_\{\\ell\}\}\(\\mathbf\{X\}\_\{\\mathrm\{in\}\}\),\(6\)wheregℓ∈\(0,1\)g\_\{\\ell\}\\in\(0,1\)scales the side\-path branch\. The gate depends on the coverage ratiorℓ=bℓ/Tr\_\{\\ell\}=b\_\{\\ell\}/T, i\.e\., the fraction of the sequence spanned by the local triBox window at layerℓ\\ell\(Figure[3](https://arxiv.org/html/2607.20484#S3.F3), Middle\)\. With learnable decay exponentϕ=softplus​\(β\)\>0\\phi=\\mathrm\{softplus\}\(\\beta\)\>0, we define

gℓ=σ​\(MLP​\(log2⁡rℓ\)\)⋅rℓϕ\.g\_\{\\ell\}=\\sigma\\big\(\\mathrm\{MLP\}\(\\log\_\{2\}r\_\{\\ell\}\)\\big\)\\cdot r\_\{\\ell\}^\{\\phi\}\.\(7\)Hereσ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid andMLP​\(⋅\)\\mathrm\{MLP\}\(\\cdot\)is a small Multi\-Layer Perceptron; we feedlog2⁡rℓ\\log\_\{2\}r\_\{\\ell\}so that dyadic changes in coverage map to approximately linear inputs for the MLP\. The only trainable gate parameters are the MLP weights and the scalarβ\\beta\(shared globally across layers and tokens;≈50\\approx 50parameters in our setups\)\. Unlike input\-dependent gates in Gated Attention\(Qiuet al\.,[2026](https://arxiv.org/html/2607.20484#bib.bib47)\),gℓg\_\{\\ell\}does not depend on hidden states; it calibrates injection from\(T,ℓ\)\(T,\\ell\)alone\.

Table 1:Main Results on General Capabilities\.We report additional trainable parameters, MMLU and HellaSwag accuracy \(%\\%\), HellaSwag perplexity \(PPL\), and empirical spectral diagnostics \(Rank denotes Stable Rank; definitions in §[4\.1](https://arxiv.org/html/2607.20484#S4.SS1)\) averaged across layers and datasets\. LoRA \+ TRSP achieves the highest accuracies with only 50 additional trainable parameters beyond LoRA\.
#### 3\.3\.2Interaction with Spectral Balance

Scaling bygℓg\_\{\\ell\}sets the effective weight of the triBox mixing matrix in the composite layer operatorℳ=I\+Aattn\+gℓ​Mtri\(ℓ\)\\mathcal\{M\}=I\+A\_\{\\mathrm\{attn\}\}\+g\_\{\\ell\}M\_\{\\mathrm\{tri\}\}^\{\(\\ell\)\}\. The triBox branch itself carries the rank and gap guarantees; the gate specifies*how strongly*that structural component enters the composite operator asTTandℓ\\ellvary\.

Input\-agnostic structural gain\.gℓg\_\{\\ell\}depends only on\(T,ℓ\)\(T,\\ell\)viarℓ=bℓ/Tr\_\{\\ell\}=b\_\{\\ell\}/T\. The side\-path injection therefore cannot be suppressed by input\-specific activation patterns, attention sinks on particular tokens, or adversarial perturbations of𝐗\\mathbf\{X\}\. Instead,gℓg\_\{\\ell\}is fixed for a given forward pass once the sequence length and layer index are known; it acts as a*structural*gain schedule rather than a content\-dependent switch\. This isolates a predictable contribution from the topologically regularized branch withinℳ\\mathcal\{M\}\.

Asymptotic stability\.The factorrℓϕr\_\{\\ell\}^\{\\phi\}captures how side\-path gain should scale asT→∞T\\to\\infty: becauserℓ=bℓ/Tr\_\{\\ell\}=b\_\{\\ell\}/Tshrinks with length, a learnedϕ\>0\\phi\>0increases the relative triBox contribution and counters dilution of the fixed topology\. In practice, this provides a simple, length\-aware calibration rule for how stronglyMtri\(ℓ\)M\_\{\\mathrm\{tri\}\}^\{\(\\ell\)\}enters the composite operator at each layer\.

Remark\.It is important to clarify the scope of these claims\. While the proposed topology strictly ensures the spectral gap and effective rank of the TRSP residual branch itself, the spectrum of the final composite operator is subject to interaction with the data\-dependent attention matrix\. Through additive perturbation theory, injecting a full\-rank component establishes rigorous worst\-case spectral guarantees for the entire composite network, as detailed in Appendix[C](https://arxiv.org/html/2607.20484#A3)\.

Table 2:Long\-Context Extrapolation Results\.RULER fine\-tunes Llama\-3\.2\-1B\-Instruct; NoLiMa trains a 109M Transformer from scratch\. Column headers are the ratiokkof evaluation to training context length \(k∈\{2,4,8\}k\\in\\\{2,4,8\\\}\): RULER uses a 4K training window and is evaluated at 8K, 16K, and 32K; NoLiMa uses 1K training and is evaluated at 2K, 4K, and 8K\. All entries are accuracy \(%\)\. TRSP is best among compared methods at8×8\\timesin both blocks\.

## 4Experiments

We organize the evaluation around four questions: \(i\) whether TRSP improves standard performance without disrupting the base model, \(ii\) whether the gains persist when evaluation contexts extend beyond the training window, \(iii\) whether task\-level improvements are accompanied by less\-collapsed spectral diagnostics, and \(iv\) which components are responsible for the observed behavior\. We study two complementary settings: post\-training on Llama\-3\.2\-1B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib42)\)and training a 109M Llama\-style transformer from scratch on NoLiMa\. In the post\-training setting, we compare against the raw model, LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib43)\), and Gated Attention\(Qiuet al\.,[2026](https://arxiv.org/html/2607.20484#bib.bib47)\); in the from\-scratch setting, we also compare against the Differential Transformer\(Yeet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib69)\)\. Detailed configurations are provided in Appendix[A](https://arxiv.org/html/2607.20484#A1)\.

### 4\.1Experimental Setup

Post\-training setting\.For general capability evaluation, hyperparameter sensitivity, and ablations, we fine\-tune Llama\-3\.2\-1B\-Instruct on Alpagasus\-5k\(Chenet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib48)\)and evaluate on MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.20484#bib.bib45)\)and HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2607.20484#bib.bib46)\)\. For long\-context extrapolation, we fine\-tune on RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib41)\)with a 4K context window and evaluate at 8K, 16K, and 32K\. All post\-training variants use the same tuning data and budget where applicable; TRSP is added as a side\-path plugin to the LoRA setting\.

From\-scratch setting\.To test architectural effects without relying on instruction\-tuned priors, we train a 109M\-parameter transformer from scratch on NoLiMa\([Modarressiet al\.,](https://arxiv.org/html/2607.20484#bib.bib49)\)with a 1K context window and evaluate extrapolation up to 8K\. NoLiMa requires latent associative reasoning without literal surface overlap between queries and targets, making it a controlled stress test of long\-range information retention\.

Metrics\.Beyond task accuracy, we monitor four empirical spectral diagnostics, computed per layer on the token representationsHℓ∈ℝT×dH\_\{\\ell\}\\in\\mathbb\{R\}^\{T\\times d\}, with two probing each axis of the trade\-off in §[1](https://arxiv.org/html/2607.20484#S1)\. For mixing efficiency,*\(i\) Signal Propagation Rate \(SPR\)*is the perturbation gain‖δout‖/‖δin‖\\\|\\delta\_\{\\mathrm\{out\}\}\\\|/\\\|\\delta\_\{\\mathrm\{in\}\}\\\|obtained by injecting a small Gaussian perturbation at the input; it gauges how strongly a signal propagates rather than being damped \(higher is better\), following Transformer signal\-propagation\(Nociet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib14); Saadaet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib11)\)and Lipschitz\-sensitivity\(Gouket al\.,[2021](https://arxiv.org/html/2607.20484#bib.bib19)\)analyses\. For information capacity,*\(ii\) Stable Rank*‖Hℓ‖F2/‖Hℓ‖22\\\|H\_\{\\ell\}\\\|\_\{F\}^\{2\}/\\\|H\_\{\\ell\}\\\|\_\{2\}^\{2\}\(Roy and Vetterli,[2007](https://arxiv.org/html/2607.20484#bib.bib17); Cohenet al\.,[2015](https://arxiv.org/html/2607.20484#bib.bib21)\)estimates the effective dimensionality of the representation \(higher is better\); a value approaching11is the hallmark of homogenization collapse\. We complement these with two stability indicators\.*\(iii\) Spectral Flatness*, the ratio of the geometric to the arithmetic mean of the singular values ofHℓH\_\{\\ell\}\(Gray and Markel,[1974](https://arxiv.org/html/2607.20484#bib.bib78)\), equals11for a perfectly flat, well\-conditioned spectrum and tends to0as energy concentrates on a few directions; we read it as a proxy for numerical stability, consistent with dynamical\-isometry views of well\-conditioned learning\(Saxeet al\.,[2014](https://arxiv.org/html/2607.20484#bib.bib18); Penningtonet al\.,[2017](https://arxiv.org/html/2607.20484#bib.bib20)\)\.*\(iv\) Representation Anisotropy*, the average cosine similarity between random token pairs\(Ethayarajh,[2019](https://arxiv.org/html/2607.20484#bib.bib16)\), measures how concentrated the representation cone is \(lower is better\)\. We stress that low anisotropy is beneficial*only*when paired with high stable rank—a regime we term*structured isotropy*—because a near\-collapsed representation can also appear locally isotropic; we therefore always interpret anisotropy jointly with stable rank\.

### 4\.2Main Results: General Capabilities

We first ask whether TRSP improves standard post\-training performance while remaining lightweight\. Table[1](https://arxiv.org/html/2607.20484#S3.T1)reports accuracy, perplexity, parameter overhead, and spectral diagnostics after fine\-tuning on Alpagasus\-5k\.

Performance and Efficiency\.On accuracy metrics, LoRA \+ TRSP performs best among the post\-training methods compared, reaching 37\.76% on MMLU and 29\.26% on HellaSwag\. This improves on standard LoRA by 1\.75 and 1\.29 points, respectively, while adding only 50 trainable parameters beyond the LoRA adapters\. By contrast, the Gated Attention baseline introduces 67\.2M additional parameters in this setup and performs poorly as a post\-hoc plugin, suggesting that its benefits may depend on different training dynamics\. The raw model retains the lowest HellaSwag perplexity, but TRSP achieves higher downstream accuracy and a lower perplexity than standard LoRA \(3\.05 vs\. 3\.58\)\.

Spectral Diagnostics\.The diagnostic metrics align with the proposed spectral interpretation\. Standard LoRA slightly reduces the Stable Rank from 1\.40 to 1\.36, whereas TRSP increases it to 1\.93, indicating a less concentrated representation spectrum\. TRSP also recovers most of the SPR reduction introduced by LoRA \(1\.59 vs\. 1\.51, close to the raw model’s 1\.60\) and achieves the highest Spectral Flatness \(0\.69\) and the lowest Anisotropy \(0\.18\)\. These trends support the view that the side\-path mitigates representation collapse during post\-training\.

### 4\.3Main Results: Long\-Context Extrapolation

We next test whether TRSP improves performance when evaluation contexts exceed the training window\. Table[2](https://arxiv.org/html/2607.20484#S3.T2)reports both benchmarks under a unified view: RULER under*fine\-tuning*and NoLiMa*from scratch*, with columns indexed by the evaluation\-to\-training length ratio \(2×2\\times–8×8\\times\)\.

RULER \(post\-training\)\.We fine\-tune the Llama\-3\.2\-1B\-Instruct model on the RULER benchmark with a context length of 4K, then evaluate its performance on extended contexts of 8K, 16K, and 32K \(2×2\\times,4×4\\times, and8×8\\timesthe training window\)\. As shown in Table[2](https://arxiv.org/html/2607.20484#S3.T2), standard methods struggle to generalize\. Both the raw model and standard LoRA show a clear downward trend as the context length increases\. The Gated Attention baseline drops to 40–45% accuracy, suggesting that simply adding a learnable gate fails to learn a generalization law\. In contrast, LoRA \+ TRSP achieves the highest accuracy at every tested length\. Crucially, while the trained baselines \(LoRA and Gated Attention\) degrade faster as the context grows, TRSP sustains the strongest absolute accuracy, indicating that the spectral balance better preserves signal integrity over long sequences\. Detailed per\-task accuracy for RULER is provided in Appendix[B](https://arxiv.org/html/2607.20484#A2)\.

NoLiMa \(from scratch\)\.To isolate the architectural benefits from pre\-trained priors, we train models from scratch on the NoLiMa dataset with a fixed 1K context window and test up to 8K\. Table[2](https://arxiv.org/html/2607.20484#S3.T2)compares our method with baselines\. The results show that the standard Transformer and Gated Attention variants suffer catastrophic collapse at longer contexts\. While the Differential Transformer offers improved robustness at 4K \(90\.2%\), it still degrades significantly to 53\.9% at 8K\. In comparison, Transformer \+ TRSP maintains near\-perfect performance at 4K and retains a remarkably high accuracy of 83\.2% at 8K\.

Table 3:Hyperparameter Sensitivity \(MMLU Accuracy %\)\.Comparison of models trained with different learning rates \(lr\)\. LoRA \+ TRSP maintains high performance even at high lrs, whereas baselines degrade or collapse, demonstrating the numerical stability provided by spectral regularization\.Table 4:Component ablation \(MMLU accuracy, %\)\.All models fine\-tune Llama\-3\.2\-1B with LoRA on Alpagasus\-5k\. The default TRSP usesgℓg\_\{\\ell\}, a sparse dyadic topology, and the triangular kernel\. Dense \(Full\-Sweep\) trades a higher per\-layer cost for a small accuracy gain\.![Refer to caption](https://arxiv.org/html/2607.20484v1/x4.png)Figure 4:Spectral–performance coupling on NoLiMa\(from scratch, train 1K\)\.\(a–d\)Layer\-averaged diagnostics \(layers 1–13\) vs\. evaluation context at1×1\\times–8×8\\timestraining length\. TRSP maintains higher stable rank, SPR, and spectral flatness and lower anisotropy than the base Transformer and Gated Attention; the Differential Transformer partially mitigates decay but trails TRSP at long context\.\(e\)NoLiMa accuracy \(%\); values at2×2\\times–8×8\\timesfollow Table[2](https://arxiv.org/html/2607.20484#S3.T2), with all models at100%100\\%at1×1\\times\.

## 5Ablation and Analysis

This section addresses the remaining experimental questions in §[4](https://arxiv.org/html/2607.20484#S4), in the order they appear below: \(iv\) which TRSP components matter for post\-training accuracy and optimization, and \(iii\) whether task gains coincide with healthier spectral diagnostics over increasing context\. We use Alpagasus\-5k / MMLU for component ablations and learning\-rate sensitivity, and NoLiMa \(from scratch\) for length\-wise spectral dynamics\.

### 5\.1Hyperparameter Sensitivity: Optimization Stability

Spectral regularization is intended to keep the mixing operator well\-conditioned; we therefore test whether TRSP improves robustness to the learning rate in post\-training\. We fine\-tune LoRA, LoRA \+ Gated Attention, and LoRA \+ TRSP on Alpagasus\-5k and report MMLU accuracy at learning rates6×10−46\\times 10^\{\-4\},8×10−48\\times 10^\{\-4\}, and1×10−31\\times 10^\{\-3\}\(Table[3](https://arxiv.org/html/2607.20484#S4.T3)\)\.

Standard LoRA is sensitive to this hyperparameter: accuracy falls from 36\.01% to 24\.47% as the learning rate increases, consistent with unstable updates when the operator lacks structural constraints\. Gated Attention remains near 23–25% across all three rates, mirroring its poor post\-hoc behavior in §[4\.2](https://arxiv.org/html/2607.20484#S4.SS2)rather than a length\-calibration issue\. LoRA \+ TRSP is substantially more stable at1×10−31\\times 10^\{\-3\}\(33\.60% vs\. 24\.47% for LoRA\), although its best accuracy still occurs at6×10−46\\times 10^\{\-4\}\(37\.76%\)\. Together with the higher spectral flatness in Table[1](https://arxiv.org/html/2607.20484#S3.T1), these results are consistent with improved training stability under aggressive optimization\.

### 5\.2Ablation Studies: Component Analysis

We ablate the long\-context gategℓg\_\{\\ell\}\(§[3\.3](https://arxiv.org/html/2607.20484#S3.SS3)\), the dyadic bandwidth schedulebℓ=min⁡\{2ℓ,T\}b\_\{\\ell\}=\\min\\\{2^\{\\ell\},T\\\}\(Eq\. \([5](https://arxiv.org/html/2607.20484#S3.E5)\)\), the triangular triBox kernel, and the sparse\-vs\-dense topology \(Appendix[C](https://arxiv.org/html/2607.20484#A3)\)\. All variants fine\-tune Llama\-3\.2\-1B with LoRA on Alpagasus\-5k; we report MMLU accuracy in Table[4](https://arxiv.org/html/2607.20484#S4.T4)\.

##### Long\-Context Gate\.

Replacing the coverage\-dependent gate with a single static scalar eliminates all 50 gate parameters but reduces accuracy to 21\.16%\. A fixed gain cannot match the per\-\(T,ℓ\)\(T,\\ell\)calibration provided byrℓ=bℓ/Tr\_\{\\ell\}=b\_\{\\ell\}/Tand Eq\. \([7](https://arxiv.org/html/2607.20484#S3.E7)\): it either over\-injects triBox locally or leaves the side\-path too weak to support global mixing\.

##### Dyadic topology and triangular kernel\.

We compare three structural variants\.Dense \(Full\-Sweep\)\.Each layer realizes all bandwidths20,…,2D−12^\{0\},\\ldots,2^\{D\-1\}in one pass, as in the dense ablation in Appendix[C](https://arxiv.org/html/2607.20484#A3)\. This achieves 38\.71% MMLU \(\+0\.95\+0\.95pt over default\) but costsO​\(T​d​log⁡T\)O\(Td\\log T\)per layer; the default TRSP retains 97\.5% of that accuracy atO​\(T​d\)O\(Td\)\.w/o dyadic bandwidth\.Fixingbℓ=Tb\_\{\\ell\}=Tat every layer removes the multi\-scale schedule and reduces accuracy to 34\.84%\.w/o triangular kernel\.Using a uniform box instead of the cascaded triangular kernel yields 31\.89%, indicating that the triangular smoothing is important for limiting local over\-mixing relative to a rectangular window\.

### 5\.3Spectral Dynamics over Context Length

To connect the NoLiMa extrapolation results in §[4\.3](https://arxiv.org/html/2607.20484#S4.SS3)to the spectral narrative in §[1](https://arxiv.org/html/2607.20484#S1), we track layer\-averaged diagnostics and task accuracy for the from\-scratch models as the evaluation context grows from1×1\\timesto8×8\\timesthe 1K training window \(Figure[4](https://arxiv.org/html/2607.20484#S4.F4)\)\.

Across this sweep, the base Transformer and Gated Attention show a clear homogenization signature: stable rank and SPR decline, while anisotropy approaches one, indicating thatℳ\\mathcal\{M\}loses effective dimensionality and long\-range mixing weakens\. The Differential Transformer partially slows this decay—especially for SPR—but does not sustain the same separation at8×8\\times\. TRSP consistently occupies the more balanced regime targeted in §[1](https://arxiv.org/html/2607.20484#S1): it maintains the highest stable rank and spectral flatness, keeps SPR well above the baselines at long context \(consistent with the side\-path safety\-net view in §[3\.2](https://arxiv.org/html/2607.20484#S3.SS2)\), and remains the least anisotropic as length increases\.

The same ordering appears in task performance \(Figure[4](https://arxiv.org/html/2607.20484#S4.F4)e; Table[2](https://arxiv.org/html/2607.20484#S3.T2)\)\. All models reach near\-perfect accuracy at1×1\\times, but accuracy diverges sharply at8×8\\times, with TRSP retaining83\.2%83\.2\\%compared with23\.8%23\.8\\%for the base model,33\.6%33\.6\\%for Gated Attention, and53\.9%53\.9\\%for the Differential Transformer\. These context\-resolved trajectories show that NoLiMa gains at long evaluation windows co\-occur with limiting spectral collapse ofℳ\\mathcal\{M\}, complementing the improved post\-training diagnostics in Table[1](https://arxiv.org/html/2607.20484#S3.T1)and the trade\-off picture in Figure[2](https://arxiv.org/html/2607.20484#S0.F2)\.

## 6Conclusion

We studied long\-context degradation by examining the spectral behavior of the transition operatorℳ\\mathcal\{M\}\. Empirically and analytically, standard Transformers tend toward two failure modes: homogenization collapse, in which over\-mixing or sink\-dominated dynamics reduce the effective rank, and isolation collapse, in which restricted mixing preserves local structure but weakens long\-range propagation\. Both extremes reduce the usable information dimensionality and degrade performance as the evaluation context grows beyond training\.

We proposed TRSP, a non\-invasive side\-path that regularizes token\-interaction topology\. By combining the parameter\-free triBox operator with a lightweight, length\-aware gate, TRSP targets spectral balance: sufficient mixing to avoid isolation while preserving rank and isotropy as context lengthens\. Across post\-training on Llama\-3\.2\-1B and a 109M from\-scratch model, TRSP improves MMLU and HellaSwag, extrapolates more reliably on RULER, and retains83\.2%83\.2\\%NoLiMa accuracy at8×8\\timesa 1K training window—about 30 and 50 percentage points above the Differential Transformer and Gated Attention, respectively\. Context\-resolved diagnostics further show that these task gains track trajectories with less spectral collapse ofℳ\\mathcal\{M\}\.

The method adds only≈50\\approx 50trainable parameters, suggesting that topology\-level regularization of the mixing operator is a practical and efficient lever for long\-context modeling\. We view this spectral framing as complementary to positional extrapolation and kernel\-efficiency advances, and hope it helps guide architectures that remain stable as sequence length increases\.

## Acknowledgements

This work was supported in part by the New Generation Artificial Intelligence\-National Science and Technology Major Project \(No\. 2025ZD0122702\), the Shenzhen Medical Research Funds in China \(No\. B2302037\), Natural Science Foundation of China \(No\. U24B6012, 62406167, 61972217, 32071459, 62176249, 62006133, 62271465\), AI for Science \(AI4S\)\-Preferred Program, Peking University Shenzhen Graduate School, China, and the Guangdong S&T Program \(2024B0101010003\)\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

## References

- F\. Barbero, A\. Arroyo, X\. Gu, C\. Perivolaropoulos, P\. Veličković, R\. Pascanu, and M\. M\. Bronstein \(2025\)Why do LLMs attend to the first token?\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=tu4dFUsW5z)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- F\. Barbero, A\. Banino, S\. Kapturowski, D\. Kumaran, J\. G\. M\. Araújo, O\. Vitvitskyi, R\. Pascanu, and P\. Velickovic \(2024\)Transformers need glasses\! information over\-squashing in language tasks\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/b1d35561c4a4a0e0b6012b2af531e149-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- I\. Beltagy, M\. E\. Peters, and A\. Cohan \(2020\)Longformer: the long\-document transformer\.ArXiv preprintabs/2004\.05150\.External Links:[Link](https://arxiv.org/abs/2004.05150)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- N\. Biggs \(1993\)Algebraic graph theory\.Cambridge university press\.Cited by:[§C\.2](https://arxiv.org/html/2607.20484#A3.SS2.1.p1.10),[§C\.2](https://arxiv.org/html/2607.20484#A3.SS2.2.p1.6)\.
- Y\. Bondarenko, M\. Nagel, and T\. Blankevoort \(2023\)Quantizable transformers: removing outliers by helping attention heads do nothing\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/edbcb7583fd8921dad78adecfe06a99b-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- J\. Bourgain and L\. Tzafriri \(1987\)Invertibility of ‘large’submatrices with applications to the geometry of banach spaces and harmonic analysis\.Israel journal of mathematics57\(2\),pp\. 137–224\.Cited by:[§D\.2](https://arxiv.org/html/2607.20484#A4.SS2.1.p1.7)\.
- L\. Chen, S\. Li, J\. Yan, H\. Wang, K\. Gunaratna, V\. Yadav, Z\. Tang, V\. Srinivasan, T\. Zhou, H\. Huang, and H\. Jin \(2024\)AlpaGasus: training a better alpaca with fewer data\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=FdVXgSJhvz)Cited by:[§A\.1](https://arxiv.org/html/2607.20484#A1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p1.1)\.
- R\. Child \(2019\)Generating long sequences with sparse transformers\.ArXiv preprintabs/1904\.10509\.External Links:[Link](https://arxiv.org/abs/1904.10509)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- M\. B\. Cohen, J\. Nelson, and D\. P\. Woodruff \(2015\)Optimal approximate matrix product in terms of stable rank\.ArXiv preprintabs/1507\.02268\.External Links:[Link](https://arxiv.org/abs/1507.02268)Cited by:[§3\.2](https://arxiv.org/html/2607.20484#S3.SS2.p3.4),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- Z\. Dai, Z\. Yang, Y\. Yang, J\. Carbonell, Q\. Le, and R\. Salakhutdinov \(2019\)Transformer\-XL: attentive language models beyond a fixed\-length context\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 2978–2988\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1285),[Link](https://aclanthology.org/P19-1285)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with io\-awareness\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- Y\. Dong, J\. Cordonnier, and A\. Loukas \(2021\)Attention is not all you need: pure attention loses rank doubly exponentially with depth\.InProceedings of the 38th International Conference on Machine Learning, ICML 2021, 18\-24 July 2021, Virtual Event,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 2793–2803\.External Links:[Link](http://proceedings.mlr.press/v139/dong21a.html)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- G\. J\. Dovonon, M\. M\. Bronstein, and M\. J\. Kusner \(2024\)Setting the record straight on transformer oversmoothing\.ArXiv preprintabs/2401\.04301\.External Links:[Link](https://arxiv.org/abs/2401.04301)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§A\.2](https://arxiv.org/html/2607.20484#A1.SS2.p1.7),[§4](https://arxiv.org/html/2607.20484#S4.p1.1)\.
- K\. Ethayarajh \(2019\)How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT\-2 embeddings\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 55–65\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1006),[Link](https://aclanthology.org/D19-1006)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- J\. Gao, D\. He, X\. Tan, T\. Qin, L\. Wang, and T\. Liu \(2019\)Representation degeneration problem in training natural language generation models\.In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\-9, 2019,External Links:[Link](https://openreview.net/forum?id=SkEYojRqtm)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. Gao \(2024\)Model tells you what to discard: adaptive KV cache compression for llms\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=uNrFpDPMyo)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- B\. Geshkovski, C\. Letrouit, Y\. Polyanskiy, and P\. Rigollet \(2025\)A mathematical perspective on transformers\.Bulletin of the American Mathematical Society62\(3\),pp\. 427–479\.Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- H\. Gouk, E\. Frank, B\. Pfahringer, and M\. J\. Cree \(2021\)Regularisation of neural networks by enforcing lipschitz continuity\.Machine Learning110\(2\),pp\. 393–416\.Cited by:[§A\.4](https://arxiv.org/html/2607.20484#A1.SS4.p1.5),[§3\.2](https://arxiv.org/html/2607.20484#S3.SS2.p2.4),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- A\. Gray and J\. Markel \(1974\)A spectral\-flatness measure for studying the autocorrelation method of linear prediction of speech analysis\.IEEE Transactions on Acoustics, Speech, and Signal Processing22\(3\),pp\. 207–217\.External Links:[Document](https://dx.doi.org/10.1109/TASSP.1974.1162572)Cited by:[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- A\. Gu and T\. Dao \(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=tEYskw1VY2)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- X\. Gu, T\. Pang, C\. Du, Q\. Liu, F\. Zhang, C\. Du, Y\. Wang, and M\. Lin \(2025\)When attention sink emerges in language models: an empirical view\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=78Nn4QJTEN)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- C\. Han, Q\. Wang, H\. Peng, W\. Xiong, Y\. Chen, H\. Ji, and S\. Wang \(2024\)LM\-infinite: zero\-shot extreme length generalization for large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 3991–4008\.External Links:[Link](https://aclanthology.org/2024.naacl-long.222)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- B\. Hanin and D\. Rolnick \(2018\)How to start training: the effect of initialization and architecture\.InAdvances in Neural Information Processing Systems,S\. Bengio, H\. Wallach, H\. Larochelle, K\. Grauman, N\. Cesa\-Bianchi, and R\. Garnett \(Eds\.\),Vol\.31,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2018/file/d81f9c1be2e08964bf9f24b15f0e4900-Paper.pdf)Cited by:[§D\.5](https://arxiv.org/html/2607.20484#A4.SS5.1.p1.5)\.
- B\. He, J\. Martens, G\. Zhang, A\. Botev, A\. Brock, S\. L\. Smith, and Y\. W\. Teh \(2023\)Deep transformers without shortcuts: modifying self\-attention for faithful signal propagation\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/pdf?id=NPrsUQgMjKK)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§A\.1](https://arxiv.org/html/2607.20484#A1.SS1.p2.1),[§1](https://arxiv.org/html/2607.20484#S1.p6.1),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.ArXiv preprintabs/2404\.06654\.External Links:[Link](https://arxiv.org/abs/2404.06654)Cited by:[§A\.1](https://arxiv.org/html/2607.20484#A1.SS1.p1.1),[§1](https://arxiv.org/html/2607.20484#S1.p6.1),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§A\.2](https://arxiv.org/html/2607.20484#A1.SS2.p1.7),[§4](https://arxiv.org/html/2607.20484#S4.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1)\.
- P\. Kaul, C\. Ma, I\. Elezi, and J\. Deng \(2025\)From attention to activation: unraveling the enigmas of large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34531–34582\.Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- N\. Keriven \(2022\)Not too little, not too much: a theoretical analysis of graph \(over\)smoothing\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/0f956ca6f667c62e0f71511773c86a59-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- D\. A\. Levin and Y\. Peres \(2017\)Markov chains and mixing times\.Vol\.107,American Mathematical Soc\.\.Cited by:[Appendix D](https://arxiv.org/html/2607.20484#A4.p2.16),[§3\.2](https://arxiv.org/html/2607.20484#S3.SS2.p2.4)\.
- H\. Liu, M\. Zaharia, and P\. Abbeel \(2024a\)RingAttention with blockwise transformers for near\-infinite context\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=WsRHpHH4s0)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024b\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638),[Link](https://aclanthology.org/2024.tacl-1.9)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p1.1)\.
- \[35\]A\. Modarressi, H\. Deilamsalehy, F\. Dernoncourt, T\. Bui, R\. A\. Rossi, S\. Yoon, and H\. SchuetzeNoLiMa: long\-context evaluation beyond literal matching\.InForty\-second International Conference on Machine Learning,Cited by:[§A\.1](https://arxiv.org/html/2607.20484#A1.SS1.p1.1),[§1](https://arxiv.org/html/2607.20484#S1.p6.1),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p2.1)\.
- T\. Nguyen, T\. Nguyen, and R\. G\. Baraniuk \(2023\)Mitigating over\-smoothing in transformers via regularized nonlocal functionals\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/fde1a69a5b6e554b2f1f727197d2651d-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- L\. Noci, S\. Anagnostidis, L\. Biggio, A\. Orvieto, S\. P\. Singh, and A\. Lucchi \(2022\)Signal propagation in transformers: theoretical perspectives and the role of rank collapse\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/ae0cba715b60c4052359b3d52a2cff7f-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2607.20484#S3.SS2.p2.4),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- B\. Peng, E\. Alcaide, Q\. G\. Anthony, A\. Albalak, S\. Arcadinho, S\. Biderman, H\. Cao, X\. Cheng, M\. N\. Chung, L\. Derczynski, X\. Du, M\. Grella, K\. K\. GV, X\. He, H\. Hou, P\. Kazienko, J\. Kocon, J\. Kong, B\. Koptyra, H\. Lau, J\. Lin, K\. S\. I\. Mantri, F\. Mom, A\. Saito, G\. Song, X\. Tang, J\. S\. Wind, S\. Woźniak, Z\. Zhang, Q\. Zhou, J\. Zhu, and R\. Zhu \(2023\)RWKV: reinventing RNNs for the transformer era\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://openreview.net/forum?id=7SaXczaBpG)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- B\. Peng, J\. Quesnelle, H\. Fan, and E\. Shippole \(2024\)YaRN: efficient context window extension of large language models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=wHBfxhZu1u)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- J\. Pennington, S\. S\. Schoenholz, and S\. Ganguli \(2017\)Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice\.InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\-9, 2017, Long Beach, CA, USA,I\. Guyon, U\. von Luxburg, S\. Bengio, H\. M\. Wallach, R\. Fergus, S\. V\. N\. Vishwanathan, and R\. Garnett \(Eds\.\),pp\. 4785–4795\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/d9fc0cdb67638d50f411432d0d41d0ba-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- O\. Press, N\. A\. Smith, and M\. Lewis \(2022\)Train short, test long: attention with linear biases enables input length extrapolation\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=R8sQPpGCv0)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- Z\. Qiu, Z\. Wang, B\. Zheng, Z\. Huang, K\. Wen, S\. Yang, R\. Men, L\. Yu, F\. Huang, S\. Huang, D\. Liu, J\. Zhou, and J\. Lin \(2026\)Gated attention for large language models: non\-linearity, sparsity, and attention\-sink\-free\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=1b7whO4SfY)Cited by:[§A\.2](https://arxiv.org/html/2607.20484#A1.SS2.p1.7),[§A\.2](https://arxiv.org/html/2607.20484#A1.SS2.p2.9),[§1](https://arxiv.org/html/2607.20484#S1.p6.1),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1),[§3\.3\.1](https://arxiv.org/html/2607.20484#S3.SS3.SSS1.p1.12),[Table 1](https://arxiv.org/html/2607.20484#S3.T1.10.10.3),[§4](https://arxiv.org/html/2607.20484#S4.p1.1)\.
- J\. Ramapuram, F\. Danieli, E\. G\. Dhekane, F\. Weers, D\. Busbridge, P\. Ablin, T\. Likhomanenko, J\. Digani, Z\. Gu, A\. Shidani, and R\. Webb \(2025\)Theory, analysis, and best practices for sigmoid self\-attention\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Zhdhg6n2OG)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- O\. Roy and M\. Vetterli \(2007\)The effective rank: a measure of effective dimensionality\.In2007 15th European Signal Processing Conference,Vol\.,pp\. 606–610\.External Links:[Document](https://dx.doi.org/)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p4.11),[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2607.20484#S3.SS2.p3.4),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- M\. Rudelson and R\. Vershynin \(2007\)Sampling from large matrices: an approach through geometric functional analysis\.J\. ACM54\(4\),pp\. 21–es\.External Links:[Document](https://dx.doi.org/10.1145/1255443.1255449),ISSN 0004\-5411,[Link](https://doi.org/10.1145/1255443.1255449)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p4.11)\.
- T\. N\. Saada, A\. Naderi, and J\. Tanner \(2025\)Mind the gap: a spectral analysis of rank collapse and signal propagation in attention layers\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=sRKtbGsebH)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2607.20484#S3.SS2.p2.4),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- A\. M\. Saxe, J\. L\. McClelland, and S\. Ganguli \(2014\)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\.In2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14\-16, 2014, Conference Track Proceedings,Y\. Bengio and Y\. LeCun \(Eds\.\),External Links:[Link](http://arxiv.org/abs/1312.6120)Cited by:[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p3.7)\.
- J\. Shah, G\. Bikshandi, Y\. Zhang, V\. Thakkar, P\. Ramani, and T\. Dao \(2024\)FlashAttention\-3: fast and accurate attention with asynchrony and low\-precision\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/7ede97c3e082c6df10a8d6103a2eebd2-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- H\. Shi, J\. Gao, H\. Xu, X\. Liang, Z\. Li, L\. Kong, S\. M\. S\. Lee, and J\. T\. Kwok \(2022\)Revisiting over\-smoothing in BERT from the perspective of graph\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=dUV91uaXm3)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- D\. A\. Spielman and N\. Srivastava \(2012\)An elementary proof of the restricted invertibility theorem\.Israel Journal of Mathematics190\(1\),pp\. 83–91\.Cited by:[§D\.2](https://arxiv.org/html/2607.20484#A4.SS2.1.p1.7)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)Roformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§A\.2](https://arxiv.org/html/2607.20484#A1.SS2.p2.9),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. Liu \(2024\)Massive activations in large language models\.ArXiv preprintabs/2402\.17762\.External Links:[Link](https://arxiv.org/abs/2402.17762)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé, J\. Ferret, P\. Liu, P\. Tafti, A\. Friesen, M\. Casbon, S\. Ramos, R\. Kumar, C\. L\. Lan, S\. Jerome, A\. Tsitsulin, N\. Vieillard, P\. Stanczyk, S\. Girgin, N\. Momchev, M\. Hoffman, S\. Thakoor, J\. Grill, B\. Neyshabur, O\. Bachem, A\. Walton, A\. Severyn, A\. Parrish, A\. Ahmad, A\. Hutchison, A\. Abdagic, A\. Carl, A\. Shen, A\. Brock, A\. Coenen, A\. Laforge, A\. Paterson, B\. Bastian, B\. Piot, B\. Wu, B\. Royal, C\. Chen, C\. Kumar, C\. Perry, C\. Welty, C\. A\. Choquette\-Choo, D\. Sinopalnikov, D\. Weinberger, D\. Vijaykumar, D\. Rogozińska, D\. Herbison, E\. Bandy, E\. Wang, E\. Noland, E\. Moreira, E\. Senter, E\. Eltyshev, F\. Visin, G\. Rasskin, G\. Wei, G\. Cameron, G\. Martins, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Batra, H\. Dhand, I\. Nardini, J\. Mein, J\. Zhou, J\. Svensson, J\. Stanway, J\. Chan, J\. P\. Zhou, J\. Carrasqueira, J\. Iljazi, J\. Becker, J\. Fernandez, J\. van Amersfoort, J\. Gordon, J\. Lipschultz, J\. Newlan, J\. Ji, K\. Mohamed, K\. Badola, K\. Black, K\. Millican, K\. McDonell, K\. Nguyen, K\. Sodhia, K\. Greene, L\. L\. Sjoesund, L\. Usui, L\. Sifre, L\. Heuermann, L\. Lago, L\. McNealus, L\. B\. Soares, L\. Kilpatrick, L\. Dixon, L\. Martins, M\. Reid, M\. Singh, M\. Iverson, M\. Görner, M\. Velloso, M\. Wirth, M\. Davidow, M\. Miller, M\. Rahtz, M\. Watson, M\. Risdal, M\. Kazemi, M\. Moynihan, M\. Zhang, M\. Kahng, M\. Park, M\. Rahman, M\. Khatwani, N\. Dao, N\. Bardoliwalla, N\. Devanathan, N\. Dumai, N\. Chauhan, O\. Wahltinez, P\. Botarda, P\. Barnes, P\. Barham, P\. Michel, P\. Jin, P\. Georgiev, P\. Culliton, P\. Kuppala, R\. Comanescu, R\. Merhej, R\. Jana, R\. A\. Rokni, R\. Agarwal, R\. Mullins, S\. Saadat, S\. M\. Carthy, S\. Cogan, S\. Perrin, S\. M\. R\. Arnold, S\. Krause, S\. Dai, S\. Garg, S\. Sheth, S\. Ronstrom, S\. Chan, T\. Jordan, T\. Yu, T\. Eccles, T\. Hennigan, T\. Kocisky, T\. Doshi, V\. Jain, V\. Yadav, V\. Meshram, V\. Dharmadhikari, W\. Barkley, W\. Wei, W\. Ye, W\. Han, W\. Kwon, X\. Xu, Z\. Shen, Z\. Gong, Z\. Wei, V\. Cotruta, P\. Kirk, A\. Rao, M\. Giang, L\. Peran, T\. Warkentin, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, D\. Sculley, J\. Banks, A\. Dragan, S\. Petrov, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, S\. Borgeaud, N\. Fiedel, A\. Joulin, K\. Kenealy, R\. Dadashi, and A\. Andreev \(2024\)Gemma 2: improving open language models at a practical size\.External Links:2408\.00118,[Link](https://arxiv.org/abs/2408.00118)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1)\.
- P\. Veličković, C\. Perivolaropoulos, F\. Barbero, and R\. Pascanu \(2025\)Softmax is not enough \(for sharp size generalisation\)\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=S4JmmpnSPy)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- P\. Wang, W\. Zheng, T\. Chen, and Z\. Wang \(2022\)Anti\-oversmoothing in deep vision transformers via the fourier domain analysis: from theory to practice\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=O476oWmiNNp)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- H\. Wu and K\. Tu \(2024\)Layer\-condensed KV cache for efficient inference of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11175–11188\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.602),[Link](https://aclanthology.org/2024.acl-long.602/)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- X\. Wu, A\. Ajorlou, Y\. Wang, S\. Jegelka, and A\. Jadbabaie \(2024\)On the role of attention masks and layernorm in transformers\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/1ac3030fc57850b0fb11dfe9d4880ad7-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p1.1),[§1](https://arxiv.org/html/2607.20484#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- X\. Wu, A\. Ajorlou, Z\. Wu, and A\. Jadbabaie \(2023\)Demystifying oversmoothing in attention\-based graph neural networks\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/6e4cdfdd909ea4e34bfc85a12774cba0-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2607.20484#S2.SS1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[§1](https://arxiv.org/html/2607.20484#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1)\.
- T\. Ye, L\. Dong, Y\. Xia, Y\. Sun, Y\. Zhu, G\. Huang, and F\. Wei \(2025\)Differential transformer\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=OvoCm1gGhN)Cited by:[§A\.2](https://arxiv.org/html/2607.20484#A1.SS2.p2.9),[§1](https://arxiv.org/html/2607.20484#S1.p6.1),[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p3.1),[§4](https://arxiv.org/html/2607.20484#S4.p1.1)\.
- M\. Zaheer, G\. Guruganesh, K\. A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontañón, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang, and A\. Ahmed \(2020\)Big bird: transformers for longer sequences\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, and H\. Lin \(Eds\.\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/c8512d142a2d849725f31a9a7a361ab9-Abstract.html)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1472),[Link](https://aclanthology.org/P19-1472)Cited by:[§A\.1](https://arxiv.org/html/2607.20484#A1.SS1.p2.1),[§1](https://arxiv.org/html/2607.20484#S1.p6.1),[§4\.1](https://arxiv.org/html/2607.20484#S4.SS1.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. W\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2607.20484#S2.SS2.p2.1)\.

## Appendix ADetailed Experimental Setup

This appendix provides the complete implementation details, training configurations, and evaluation protocols used in our experiments\.

### A\.1Datasets and Benchmarks

Training Datasets\.We use three datasets, each tailored to a specific setting\.*Alpagasus\-5k*\(Chenet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib48)\), a GPT\-4\-filtered high\-quality subset of Alpaca, supports the general\-capability evaluation \(MMLU, HellaSwag\) and all ablations, simulating a standard instruction\-tuning scenario\.*RULER*\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib41)\)supports post\-training long\-context extrapolation: we fine\-tune on its training split with a fixed 4K context window so the model learns the task format within a standard window before being tested at 8K, 16K, and 32K\.*NoLiMa*\([Modarressiet al\.,](https://arxiv.org/html/2607.20484#bib.bib49)\)supports the from\-scratch experiments\. It minimizes lexical overlap between a question and its needle, so that answering requires latent associative reasoning rather than literal matching, relying on the world knowledge of a pretrained model to bridge the question and the needle\. Since our 109M models are trained from scratch and lack such priors, we adapt the protocol into a self\-contained corpus: for every instance \(100%100\\%of both training and evaluation data\) we explicitly insert the bridging fact into the haystack, turning each query into a multi\-hop chain over distant inserted statements\. This keeps the no\-literal\-matching property while drawing train and test instances from the same template pool, so that long\-context failures reflect mechanistic spectral decay rather than distribution shift\.

Evaluation Benchmarks\.General capabilities are evaluated on MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.20484#bib.bib45)\)and HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2607.20484#bib.bib46)\)with models fine\-tuned on Alpagasus\-5k\. For long\-context extrapolation, RULER spans four task categories—Retrieval \(NIAH\), Multi\-hop Tracing \(Variable Tracking\), Aggregation \(Common/Frequent Words Extraction\), and Question Answering,1313tasks in total—and we evaluate at 8K, 16K, and 32K using models fine\-tuned on RULER\-4K\. NoLiMa is evaluated up to 8K using the from\-scratch models trained at 1K, reporting the exact\-match accuracy of the retrieved associated value\.

### A\.2Model Architectures

Post\-Training Setting \(Llama\-3\.2\)\.We build on Llama\-3\.2\-1B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib42)\)\. The*Raw*baseline is the unmodified model\.*LoRA*\(Huet al\.,[2022](https://arxiv.org/html/2607.20484#bib.bib43)\)adapts the query, key, value, and output projections \(Wq,Wk,Wv,WoW\_\{q\},W\_\{k\},W\_\{v\},W\_\{o\}\) of the attention layers with rankr=8r=8and scalingα=16\\alpha=16\.*LoRA \+ Gated Attention*adds, on top of LoRA, the gated\-attention mechanism ofQiuet al\.\([2026](https://arxiv.org/html/2607.20484#bib.bib47)\): a query\-dependent, head\-specific sigmoid gate applied elementwise to the SDPA output, realized by a dense per\-layer projection that adds≈\\approx67M trainable parameters\.*LoRA \+ TRSP \(ours\)*adds the triBox side\-path and long\-context gate in parallel to attention,𝐗out=𝐗in\+Attn​\(𝐗in\)\+gℓ​triBoxbℓ​\(𝐗in\)\\mathbf\{X\}\_\{\\mathrm\{out\}\}=\\mathbf\{X\}\_\{\\mathrm\{in\}\}\+\\mathrm\{Attn\}\(\\mathbf\{X\}\_\{\\mathrm\{in\}\}\)\+g\_\{\\ell\}\\,\\mathrm\{triBox\}\_\{b\_\{\\ell\}\}\(\\mathbf\{X\}\_\{\\mathrm\{in\}\}\)\(Eq\. \([6](https://arxiv.org/html/2607.20484#S3.E6)\)\); the only extra trainable parameters are the gate MLP and the scalarβ\\betaof Eq\. \([7](https://arxiv.org/html/2607.20484#S3.E7)\) \(≈50\\approx 50in total\)\.

Pre\-Training Setting \(Custom Tiny Transformer\)\.To isolate architectural effects, we train a 109M\-parameter Transformer from scratch whose configuration follows Llama at a smaller scale: hidden sizedmodel=512d\_\{\\mathrm\{model\}\}=512,D=14D=14layers,88attention heads, MLP ratio4\.04\.0, a32,00032\{,\}000\-token vocabulary \(Llama tokenizer\), and Rotary Positional Embeddings\(Suet al\.,[2024](https://arxiv.org/html/2607.20484#bib.bib28)\)\. We compare four variants that share the same backbone and parameter count \(≈\\approx109\.8M\): the standard Transformer, Transformer \+ Gated Attention\(Qiuet al\.,[2026](https://arxiv.org/html/2607.20484#bib.bib47)\), the Differential Transformer\(Yeet al\.,[2025](https://arxiv.org/html/2607.20484#bib.bib69)\), and Transformer \+ TRSP\. Following the official Differential Transformer implementation, we partition the query and key projections into two groups to form two softmax maps whose difference is the attention score,Attn=softmax​\(Q1​K1⊤\)−λ​softmax​\(Q2​K2⊤\)\\mathrm\{Attn\}=\\mathrm\{softmax\}\(Q\_\{1\}K\_\{1\}^\{\\top\}\)\-\\lambda\\,\\mathrm\{softmax\}\(Q\_\{2\}K\_\{2\}^\{\\top\}\), whereλ\\lambdais a per\-layer learnable scalar shared across heads and reparameterized asλ=exp⁡\(λq1⋅λk1\)−exp⁡\(λq2⋅λk2\)\+λinit\\lambda=\\exp\(\\lambda\_\{q\_\{1\}\}\\\!\\cdot\\\!\\lambda\_\{k\_\{1\}\}\)\-\\exp\(\\lambda\_\{q\_\{2\}\}\\\!\\cdot\\\!\\lambda\_\{k\_\{2\}\}\)\+\\lambda\_\{\\mathrm\{init\}\}; we apply per\-head GroupNorm to the head outputs and halve the number of heads so that the parameter count matches the standard Transformer\.

### A\.3Training Configurations

All models use the AdamW optimizer with a cosine schedule and a3%3\\%warmup\.

Post\-training \(Alpagasus & RULER\)\.For MMLU/HellaSwag and the ablations we fine\-tune on Alpagasus\-5k for33epochs with batch size11at a learning rate of6×10−46\\times 10^\{\-4\}; the sensitivity study additionally sweeps8×10−48\\times 10^\{\-4\}and1×10−31\\times 10^\{\-3\}\. For RULER extrapolation we fine\-tune on the RULER training set at a fixed 4K context for33epochs at6×10−46\\times 10^\{\-4\}\.

Pre\-training \(NoLiMa\)\.We train the 109M models from random initialization on the augmented NoLiMa corpus with a fixed 1K context for55epochs at1×10−41\\times 10^\{\-4\}, using standard next\-token prediction \(causal language modeling\)\.

### A\.4Evaluation Protocols

Spectral metrics\.We compute the diagnostics of §[4\.1](https://arxiv.org/html/2607.20484#S4.SS1)on the layer\-wise token representationsHℓ∈ℝT×dH\_\{\\ell\}\\in\\mathbb\{R\}^\{T\\times d\}\(reported in Table[1](https://arxiv.org/html/2607.20484#S3.T1)and Figure[4](https://arxiv.org/html/2607.20484#S4.F4)\)\.*Stable Rank*is‖Hℓ‖F2/‖Hℓ‖22\\\|H\_\{\\ell\}\\\|\_\{F\}^\{2\}/\\\|H\_\{\\ell\}\\\|\_\{2\}^\{2\}, measuring the effective dimensionality\.*Anisotropy*is the average cosine similarity between random pairs of token representations; lower values indicate a more isotropic distribution, which we treat as beneficial only when accompanied by high stable rank \(structured isotropy\)\.*Signal Propagation Rate \(SPR\)*is the perturbation gain‖δout‖/‖δin‖\\\|\\delta\_\{\\mathrm\{out\}\}\\\|/\\\|\\delta\_\{\\mathrm\{in\}\}\\\|\(Gouket al\.,[2021](https://arxiv.org/html/2607.20484#bib.bib19)\), measured by injecting a Gaussian perturbation \(σ=10−3\\sigma=10^\{\-3\}\) at the embeddings and propagating it through the network\.*Spectral Flatness*is the ratio of the geometric to the arithmetic mean of the singular values ofHℓH\_\{\\ell\}\.

Extrapolation testing\.RULER models trained at 4K are evaluated at 8K, 16K, and 32K, reporting the average score over all sub\-tasks; NoLiMa models trained at 1K are evaluated at 1K, 2K, 4K, and 8K, reporting the exact\-match accuracy of the generated answer against the gold reference\.

## Appendix BDetailed Breakdown of RULER Performance

We provide the fine\-grained performance breakdown across all 13 sub\-tasks of the RULER benchmark in Table[5](https://arxiv.org/html/2607.20484#A2.T5)\. This detailed view reveals specific failure modes of baseline methods that are masked in the aggregated scores, particularly at the extreme context length of 32K\.

Analysis of Sub\-Task Performance\.While the LoRA \+ Gated Attention baseline remains competitive at 8K, it exhibits a catastrophic collapse as the context extends to 32K\. Specifically, on NIAH Multikey 2 and NIAH Multikey 3, its accuracy plummets to near zero \(6\.3% and 0\.0% respectively\)\. In contrast, our TRSP maintains high robustness, achieving 93\.7% and 64\.5% on these tasks, indicating that our spectral regularization helps prevent the attention\-sink phenomenon from cutting off long\-range dependencies\.

Regarding the harder tasks like Common Words Extraction \(CWE\) and Variable Tracking \(VT\), which require precise state tracking over long distances, TRSP achieves 8\.1% on CWE \(vs\. 0\.0% for LoRA\) and 18\.1% on VT \(vs\. 9\.2% for LoRA\)\. This roughly doubles VT accuracy \(18\.1% vs\. 9\.2%\), consistent with our account: by maintaining a healthier spectral gap and effective rank, TRSP better preserves the distinctness of token states over very long sequences\. Standard LoRA degrades sharply at 32K, whereas TRSP mitigates this decay\.

For full transparency, we also note the tasks where the raw instruction\-tuned model retains a clear edge: on NIAH Multi\-Value \(NIAH MV\), NIAH Multi\-Query \(NIAH MQ\), and Frequent\-Words Extraction, the raw model scores far higher \(e\.g\., 80\.6%, 84\.7%, and 63\.5% at 32K\) than both LoRA and LoRA \+ TRSP, which regress to roughly 25–33%\. Because this regression is shared almost identically by LoRA and LoRA \+ TRSP, it reflects a format/distribution shift induced by RULER fine\-tuning rather than a side\-effect of the TRSP branch; TRSP’s gains instead concentrate on the retrieval\- and tracking\-heavy tasks \(NIAH MK2/MK3, CWE, VT\) that most directly stress long\-range spectral health\.

Table 5:Detailed RULER Sub\-Task Performance \(8K, 16K, 32K\)\.We report accuracy \(%\) for each task\. Raw: Llama\-3\.2\-1B\-Instruct\. Gate: LoRA \+ Gated Attention\. Ours: LoRA \+ TRSP\. Best results in each group are bolded\.
## Appendix CTheoretical Analysis of TRSP Topology

This appendix analyzes the spectral properties of the TRSP mixing operator, establishing \(i\) a length\-independent lower bound on its algebraic connectivity and \(ii\) a lower bound on its effective rank\. Throughout,Ω​\(⋅\)\\Omega\(\\cdot\)andΘ​\(⋅\)\\Theta\(\\cdot\)denote asymptotics in the sequence lengthTT, all constants are independent ofTTunless stated otherwise, and modeling assumptions are made explicit where they are used\.

### C\.1Setup and Definitions

We study the per\-channel token\-mixing operator induced by one TRSP layer\. Acting on the token axis, a single channel of the hidden state isx∈ℝTx\\in\\mathbb\{R\}^\{T\}, and the \(linear\) layer update is

xout=\(I\+Aattn\+gMtri\)x=:ℳx,x\_\{\\mathrm\{out\}\}=\\big\(I\+A\_\{\\mathrm\{attn\}\}\+g\\,M\_\{\\mathrm\{tri\}\}\\big\)\\,x=:\\mathcal\{M\}\\,x,\(8\)whereAattn∈ℝT×TA\_\{\\mathrm\{attn\}\}\\in\\mathbb\{R\}^\{T\\times T\}is the row\-stochastic attention matrix,Mtri∈ℝT×TM\_\{\\mathrm\{tri\}\}\\in\\mathbb\{R\}^\{T\\times T\}is the triBox operator, andg:=gℓ∈\(0,1\)g:=g\_\{\\ell\}\\in\(0,1\)is the long\-context gate of Eq\. \([7](https://arxiv.org/html/2607.20484#S3.E7)\) \(fixed within a forward pass\)\.

Symmetrization\.The causal triBoxMtriM\_\{\\mathrm\{tri\}\}is lower\-triangular \(directed\)\. Algebraic connectivity is defined for undirected graphs, so we analyze the symmetrized connectionWtri:=12​\(Mtri\+Mtri⊤\)W\_\{\\mathrm\{tri\}\}:=\\tfrac\{1\}\{2\}\(M\_\{\\mathrm\{tri\}\}\+M\_\{\\mathrm\{tri\}\}^\{\\top\}\)\(and likewiseWattnW\_\{\\mathrm\{attn\}\}\); both are symmetric with nonnegative weights\. We write𝐋∙=𝐃∙−W∙\\mathbf\{L\}\_\{\\bullet\}=\\mathbf\{D\}\_\{\\bullet\}\-W\_\{\\bullet\}for the corresponding combinatorial Laplacian, with𝐃∙\\mathbf\{D\}\_\{\\bullet\}the diagonal degree matrix; each such Laplacian is symmetric positive semidefinite \(PSD\) with smallest eigenvalue0\.

Two spectral quantities\.We distinguish \(i\) the*algebraic connectivity*μ2​\(𝐋\)\\mu\_\{2\}\(\\mathbf\{L\}\), the second\-smallest eigenvalue of𝐋\\mathbf\{L\}\(the Fiedler value\), measuring absolute connectivity; and \(ii\) the*normalized gap*γ\\gamma, the second\-smallest eigenvalue of the random\-walk Laplacianℒ=I−𝐃−1​W\\mathcal\{L\}=I\-\\mathbf\{D\}^\{\-1\}W, measuring the per\-step contraction rate of the induced random walk\. For anrr\-regular graph the two satisfyγ=μ2/r\\gamma=\\mu\_\{2\}/r\.

### C\.2Topological Structure: A Cyclic Cayley Graph

We first identify the connectivity that the triBox branch induces across layers\.

###### Theorem C\.1\(triBox skeleton as a Cayley graph\)\.

Consider the symmetrized connection obtained by superposing the dominant dyadic offsets of the triBox branch across layersℓ=0,…,L\\ell=0,\\dots,LwithL=⌊log2⁡T⌋L=\\lfloor\\log\_\{2\}T\\rfloor\. Its connection graph is the Cayley graphCay​\(ℤT,S\)\\mathrm\{Cay\}\(\\mathbb\{Z\}\_\{T\},S\)on the cyclic groupℤT\\mathbb\{Z\}\_\{T\}with generator setS=\{±2ℓ:ℓ=0,…,L\}S=\\\{\\pm 2^\{\\ell\}:\\ell=0,\\dots,L\\\}\.

###### Proof\.

At layerℓ\\ellthe triBox window of bandwidthbℓ=2ℓb\_\{\\ell\}=2^\{\\ell\}couples each token to its causal neighborhood, whose longest offset is2ℓ2^\{\\ell\}\. Retaining these dominant offsets and symmetrizing, the superposition over layers connects everyn∈ℤTn\\in\\mathbb\{Z\}\_\{T\}ton±2ℓn\\pm 2^\{\\ell\}for allℓ\\ell, i\.e\. exactlySS\. The connection depends only on the difference\(n−n′\)modT\(n\-n^\{\\prime\}\)\\bmod T, so the symmetrized adjacencyWtriW\_\{\\mathrm\{tri\}\}is circulant—the defining property of a Cayley graph onℤT\\mathbb\{Z\}\_\{T\}\(Biggs,[1993](https://arxiv.org/html/2607.20484#bib.bib54)\)\. ∎

###### Lemma C\.3\(Spectrum of the skeleton\)\.

ForCay​\(ℤT,S\)\\mathrm\{Cay\}\(\\mathbb\{Z\}\_\{T\},S\)withS=\{±2ℓ\}ℓ=0LS=\\\{\\pm 2^\{\\ell\}\\\}\_\{\\ell=0\}^\{L\}, the combinatorial Laplacian𝐋tri\\mathbf\{L\}\_\{\\mathrm\{tri\}\}is circulant with eigenvalues, indexed by Fourier modesk∈\{0,…,T−1\}k\\in\\\{0,\\dots,T\-1\\\},

μk​\(𝐋tri\)=2​∑ℓ=0L\(1−cos⁡\(2ℓ​θk\)\),θk=2​π​kT\.\\mu\_\{k\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)=2\\sum\_\{\\ell=0\}^\{L\}\\Big\(1\-\\cos\\big\(2^\{\\ell\}\\,\\theta\_\{k\}\\big\)\\Big\),\\qquad\\theta\_\{k\}=\\frac\{2\\pi k\}\{T\}\.\(9\)

###### Proof\.

The eigenvectors of a symmetric circulant matrix are the Fourier modesvk=\(ei​2​π​k​n/T\)n=0T−1v\_\{k\}=\(e^\{\\mathrm\{i\}\\,2\\pi kn/T\}\)\_\{n=0\}^\{T\-1\}\(Biggs,[1993](https://arxiv.org/html/2607.20484#bib.bib54)\)\. Each generator pair\{\+2ℓ,−2ℓ\}\\\{\+2^\{\\ell\},\-2^\{\\ell\}\\\}contributes degree22and off\-diagonal phasese±i2ℓ​θke^\{\\pm\\mathrm\{i\}2^\{\\ell\}\\theta\_\{k\}\}, hence a Laplacian eigenvalue contribution2−\(ei2ℓ​θk\+e−i2ℓ​θk\)=2​\(1−cos⁡\(2ℓ​θk\)\)2\-\\big\(e^\{\\mathrm\{i\}2^\{\\ell\}\\theta\_\{k\}\}\+e^\{\-\\mathrm\{i\}2^\{\\ell\}\\theta\_\{k\}\}\\big\)=2\\big\(1\-\\cos\(2^\{\\ell\}\\theta\_\{k\}\)\\big\)\. Summing overℓ\\ellgives the claim\. ∎

### C\.3Bounded Spectral Gap

#### C\.3\.1Safety Net: Monotonicity under Parallel Composition

###### Lemma C\.4\(Weyl monotonicity\)\.

LetP,QP,Qbe symmetric withQ⪰0Q\\succeq 0\. Thenλk​\(P\+Q\)≥λk​\(P\)\\lambda\_\{k\}\(P\+Q\)\\geq\\lambda\_\{k\}\(P\)for everykk, where eigenvalues are listed in increasing order\.

###### Proof\.

By the Courant–Fischer min–max theorem,λk​\(P\+Q\)=mindimV=k⁡max0≠v∈V⁡v⊤​\(P\+Q\)​vv⊤​v≥mindimV=k⁡max0≠v∈V⁡v⊤​P​vv⊤​v=λk​\(P\)\\lambda\_\{k\}\(P\+Q\)=\\min\_\{\\dim V=k\}\\max\_\{0\\neq v\\in V\}\\tfrac\{v^\{\\top\}\(P\+Q\)v\}\{v^\{\\top\}v\}\\geq\\min\_\{\\dim V=k\}\\max\_\{0\\neq v\\in V\}\\tfrac\{v^\{\\top\}Pv\}\{v^\{\\top\}v\}=\\lambda\_\{k\}\(P\), where the inequality usesv⊤​Q​v≥0v^\{\\top\}Qv\\geq 0\. ∎

###### Proposition C\.5\(Safety net\)\.

The symmetrized composite connection has Laplacian𝐋=𝐋attn\+g​𝐋tri\\mathbf\{L\}=\\mathbf\{L\}\_\{\\mathrm\{attn\}\}\+g\\,\\mathbf\{L\}\_\{\\mathrm\{tri\}\}, and its algebraic connectivity satisfiesμ2​\(𝐋\)≥g​μ2​\(𝐋tri\)\\mu\_\{2\}\(\\mathbf\{L\}\)\\geq g\\,\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)for any attention pattern\.

###### Proof\.

The Laplacian of a weighted union of edge sets is the sum of the individual Laplacians, so𝐋=𝐋attn\+g​𝐋tri\\mathbf\{L\}=\\mathbf\{L\}\_\{\\mathrm\{attn\}\}\+g\\,\\mathbf\{L\}\_\{\\mathrm\{tri\}\}with𝐋attn⪰0\\mathbf\{L\}\_\{\\mathrm\{attn\}\}\\succeq 0\. Applying[LemmaC\.4](https://arxiv.org/html/2607.20484#A3.Thmtheorem4)withP=g​𝐋triP=g\\,\\mathbf\{L\}\_\{\\mathrm\{tri\}\},Q=𝐋attnQ=\\mathbf\{L\}\_\{\\mathrm\{attn\}\}atk=2k=2givesμ2​\(𝐋\)=λ2​\(𝐋\)≥λ2​\(g​𝐋tri\)=g​μ2​\(𝐋tri\)\\mu\_\{2\}\(\\mathbf\{L\}\)=\\lambda\_\{2\}\(\\mathbf\{L\}\)\\geq\\lambda\_\{2\}\(g\\,\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)=g\\,\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)\. ∎

Thus, even if attention disconnects \(μ2​\(𝐋attn\)→0\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{attn\}\}\)\\to 0\), the composite connectivity stays≥g​μ2​\(𝐋tri\)\>0\\geq g\\,\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)\>0: the side\-path is a “safety net”\.

#### C\.3\.2Constant Algebraic Connectivity of the triBox Skeleton

###### Theorem C\.6\(Length\-independent connectivity\)\.

For everyT≥2T\\geq 2, the triBox skeleton satisfiesμ2​\(𝐋tri\)≥2\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)\\geq 2; in particular it is bounded below by a positive constant independent ofTT\.

###### Proof\.

By[LemmaC\.3](https://arxiv.org/html/2607.20484#A3.Thmtheorem3)it suffices to show that for every nonzero modek∈\{1,…,T−1\}k\\in\\\{1,\\dots,T\-1\\\}there is someℓ∈\{0,…,L\}\\ell\\in\\\{0,\\dots,L\\\}with1−cos⁡\(2ℓ​θk\)≥11\-\\cos\(2^\{\\ell\}\\theta\_\{k\}\)\\geq 1, i\.e\.2ℓ​θkmod2​π∈\[π2,3​π2\]2^\{\\ell\}\\theta\_\{k\}\\bmod 2\\pi\\in\[\\tfrac\{\\pi\}\{2\},\\tfrac\{3\\pi\}\{2\}\]; that single term then givesμk≥2\\mu\_\{k\}\\geq 2\.

Writeψ:=k/T∈\(0,1\)\\psi:=k/T\\in\(0,1\)with binary expansionψ=∑j≥1bj​2−j\\psi=\\sum\_\{j\\geq 1\}b\_\{j\}2^\{\-j\},bj∈\{0,1\}b\_\{j\}\\in\\\{0,1\\\}\. Then2ℓ​θkmod2​π=2​π​\{2ℓ​ψ\}2^\{\\ell\}\\theta\_\{k\}\\bmod 2\\pi=2\\pi\\\{2^\{\\ell\}\\psi\\\}and\{2ℓ​ψ\}=∑j≥1bℓ\+j​2−j\\\{2^\{\\ell\}\\psi\\\}=\\sum\_\{j\\geq 1\}b\_\{\\ell\+j\}2^\{\-j\}has leading bits\(bℓ\+1,bℓ\+2\)\(b\_\{\\ell\+1\},b\_\{\\ell\+2\}\)\. A direct check shows\{2ℓ​ψ\}∈\[14,34\)\\\{2^\{\\ell\}\\psi\\\}\\in\[\\tfrac\{1\}\{4\},\\tfrac\{3\}\{4\}\)iffbℓ\+1≠bℓ\+2b\_\{\\ell\+1\}\\neq b\_\{\\ell\+2\}, which gives2​π​\{2ℓ​ψ\}∈\[π2,3​π2\)2\\pi\\\{2^\{\\ell\}\\psi\\\}\\in\[\\tfrac\{\\pi\}\{2\},\\tfrac\{3\\pi\}\{2\}\)and hence1−cos≥11\-\\cos\\geq 1\.

It remains to find a sign changebℓ\+1≠bℓ\+2b\_\{\\ell\+1\}\\neq b\_\{\\ell\+2\}withℓ∈\{0,…,L\}\\ell\\in\\\{0,\\dots,L\\\}, i\.e\. among the firstL\+2L\+2bits ofψ\\psi\. If there were none, thenb1=⋯=bL\+2b\_\{1\}=\\dots=b\_\{L\+2\}: all0forcesψ<2−\(L\+2\)\\psi<2^\{\-\(L\+2\)\}, all11forcesψ≥1−2−\(L\+2\)\\psi\\geq 1\-2^\{\-\(L\+2\)\}\. But1≤k≤T−11\\leq k\\leq T\-1givesψ∈\[1T,1−1T\]\\psi\\in\[\\tfrac\{1\}\{T\},1\-\\tfrac\{1\}\{T\}\], andL=⌊log2⁡T⌋L=\\lfloor\\log\_\{2\}T\\rflooryields2−\(L\+2\)=2−L/4<1/T2^\{\-\(L\+2\)\}=2^\{\-L\}/4<1/T\(since2−L<2/T2^\{\-L\}<2/T\), contradicting both cases\. Hence a sign change exists andμk≥2\\mu\_\{k\}\\geq 2for allk≠0k\\neq 0, soμ2​\(𝐋tri\)=mink≠0⁡μk≥2\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)=\\min\_\{k\\neq 0\}\\mu\_\{k\}\\geq 2\. ∎

#### C\.3\.3Normalized Gap and the Sparse/Dense Trade\-off

We first noteμ2​\(𝐋tri\)=Θ​\(1\)\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)=\\Theta\(1\): the lower bound≥2\\geq 2is[TheoremC\.6](https://arxiv.org/html/2607.20484#A3.Thmtheorem6), and for the matching upper bound the modek=1k=1\(withθ1=2​π/T\\theta\_\{1\}=2\\pi/T\) gives, via1−cos⁡x≤x2/21\-\\cos x\\leq x^\{2\}/2,

μ2≤μ1=2​∑ℓ=0L\(1−cos⁡\(2ℓ​θ1\)\)≤θ12​∑ℓ=0L4ℓ≤\(2​πT\)2​4L\+13≤16​π23,\\mu\_\{2\}\\leq\\mu\_\{1\}=2\\sum\_\{\\ell=0\}^\{L\}\\big\(1\-\\cos\(2^\{\\ell\}\\theta\_\{1\}\)\\big\)\\leq\\theta\_\{1\}^\{2\}\\sum\_\{\\ell=0\}^\{L\}4^\{\\ell\}\\leq\\Big\(\\frac\{2\\pi\}\{T\}\\Big\)^\{2\}\\frac\{4^\{L\+1\}\}\{3\}\\leq\\frac\{16\\pi^\{2\}\}\{3\},\(10\)using4L\+1≤4​T24^\{L\+1\}\\leq 4T^\{2\}from2L≤T2^\{L\}\\leq T\. The skeleton isrr\-regular with degreer=\|S\|=2​\(L\+1\)=Θ​\(log⁡T\)r=\|S\|=2\(L\+1\)=\\Theta\(\\log T\), so byγ=μ2/r\\gamma=\\mu\_\{2\}/r,

γtri=μ2​\(𝐋tri\)r=Θ​\(1log⁡T\)\.\\gamma\_\{\\mathrm\{tri\}\}=\\frac\{\\mu\_\{2\}\(\\mathbf\{L\}\_\{\\mathrm\{tri\}\}\)\}\{r\}=\\Theta\\\!\\Big\(\\frac\{1\}\{\\log T\}\\Big\)\.\(11\)The*unnormalized*connectivity is thus constant while the*normalized*\(per\-step\) gap decays asΘ​\(1/log⁡T\)\\Theta\(1/\\log T\)\. Two implementations trade these off\.*Dense \(ablation\)\.*Each layer realizes all offsets20,…,2D−12^\{0\},\\dots,2^\{D\-1\}, so a single layer attainsγ=Θ​\(1/log⁡T\)\\gamma=\\Theta\(1/\\log T\); if additionally the depth scales asD=Θ​\(log⁡T\)D=\\Theta\(\\log T\), the depth\-composed contraction is\(1−γ\)D=\(1−Θ​\(1/log⁡T\)\)Θ​\(log⁡T\)=Θ​\(1\)\(1\-\\gamma\)^\{D\}=\\big\(1\-\\Theta\(1/\\log T\)\\big\)^\{\\Theta\(\\log T\)\}=\\Theta\(1\)\(a constant global normalized gap\), at the cost ofO​\(T​d​log⁡T\)O\(Td\\log T\)per\-layer work\.*Standard \(TRSP\)\.*The offsets are distributed across layers, so the Cayley skeleton forms only globally by superposition; this retains the constant unnormalized connectivityμ2=Ω​\(1\)\\mu\_\{2\}=\\Omega\(1\)\([TheoremC\.6](https://arxiv.org/html/2607.20484#A3.Thmtheorem6)\) atO​\(T​d\)O\(Td\)cost, while the global normalized gap staysΘ​\(1/log⁡T\)\\Theta\(1/\\log T\)\. We adopt the standard implementation: by[PropositionC\.5](https://arxiv.org/html/2607.20484#A3.Thmtheorem5)the constant unnormalized connectivity already rules out the disconnection \(isolation\) failure mode, and the dense variant serves only as a theoretical upper bound\. The assumptionD=Θ​\(log⁡T\)D=\\Theta\(\\log T\)is used only for the dense variant\.

### C\.4Effective Rank Lower Bound

We use the stable rank as the effective\-rank surrogate \(consistent with §[3\.2](https://arxiv.org/html/2607.20484#S3.SS2)\),Reff​\(ℳ\)=‖ℳ‖F2/‖ℳ‖22R\_\{\\mathrm\{eff\}\}\(\\mathcal\{M\}\)=\\\|\\mathcal\{M\}\\\|\_\{F\}^\{2\}/\\\|\\mathcal\{M\}\\\|\_\{2\}^\{2\}\.

###### Proposition C\.7\(Frobenius energy\)\.

Let the triBox at the layer of interest have bandwidthbb\. Then

‖ℳ‖F2≥T\+g2​‖Mtri‖F2=T​\(1\+g2​Eb\),\\\|\\mathcal\{M\}\\\|\_\{F\}^\{2\}\\ \\geq\\ T\+g^\{2\}\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{F\}^\{2\}\\ =\\ T\\,\(1\+g^\{2\}E\_\{b\}\),\(12\)whereEb:=‖Mtri‖F2/TE\_\{b\}:=\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{F\}^\{2\}/Tis the per\-row energy of the normalized triangular kernel𝐡\\mathbf\{h\}, satisfyingEb=‖𝐡‖22=2​b2\+13​b3=Θ​\(1/b\)E\_\{b\}=\\\|\\mathbf\{h\}\\\|\_\{2\}^\{2\}=\\tfrac\{2b^\{2\}\+1\}\{3b^\{3\}\}=\\Theta\(1/b\)\. In particularEb=Θ​\(1\)E\_\{b\}=\\Theta\(1\)for proximal layers with bounded bandwidthb=O​\(1\)b=O\(1\)\.

###### Proof\.

Writeℳ=I\+B\\mathcal\{M\}=I\+BwithB=Aattn\+g​MtriB=A\_\{\\mathrm\{attn\}\}\+g\\,M\_\{\\mathrm\{tri\}\}\. Then‖ℳ‖F2=‖I‖F2\+2​⟨I,B⟩\+‖B‖F2=T\+2​tr​\(B\)\+‖B‖F2\\\|\\mathcal\{M\}\\\|\_\{F\}^\{2\}=\\\|I\\\|\_\{F\}^\{2\}\+2\\langle I,B\\rangle\+\\\|B\\\|\_\{F\}^\{2\}=T\+2\\,\\mathrm\{tr\}\(B\)\+\\\|B\\\|\_\{F\}^\{2\}\. BothAattnA\_\{\\mathrm\{attn\}\}andMtriM\_\{\\mathrm\{tri\}\}are entrywise nonnegative, sotr​\(B\)≥0\\mathrm\{tr\}\(B\)\\geq 0; expanding‖B‖F2=‖Aattn‖F2\+2​g​⟨Aattn,Mtri⟩\+g2​‖Mtri‖F2\\\|B\\\|\_\{F\}^\{2\}=\\\|A\_\{\\mathrm\{attn\}\}\\\|\_\{F\}^\{2\}\+2g\\langle A\_\{\\mathrm\{attn\}\},M\_\{\\mathrm\{tri\}\}\\rangle\+g^\{2\}\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{F\}^\{2\}with the entrywise cross term⟨Aattn,Mtri⟩≥0\\langle A\_\{\\mathrm\{attn\}\},M\_\{\\mathrm\{tri\}\}\\rangle\\geq 0gives‖B‖F2≥g2​‖Mtri‖F2\\\|B\\\|\_\{F\}^\{2\}\\geq g^\{2\}\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{F\}^\{2\}\. Hence‖ℳ‖F2≥T\+g2​‖Mtri‖F2\\\|\\mathcal\{M\}\\\|\_\{F\}^\{2\}\\geq T\+g^\{2\}\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{F\}^\{2\}\. For the kernel energy, each \(interior\) row ofMtriM\_\{\\mathrm\{tri\}\}is the normalized triangular kernel𝐡\\mathbf\{h\}withh​\(r\)=\(b−\|r−\(b−1\)\|\)/b2h\(r\)=\\big\(b\-\|r\-\(b\-1\)\|\\big\)/b^\{2\}forr=0,…,2​b−2r=0,\\dots,2b\-2; a direct computation gives

‖𝐡‖22=1b4​∑m=−\(b−1\)b−1\(b−\|m\|\)2=1b4⋅b​\(2​b2\+1\)3=2​b2\+13​b3=Θ​\(1/b\)\.\\\|\\mathbf\{h\}\\\|\_\{2\}^\{2\}=\\frac\{1\}\{b^\{4\}\}\\sum\_\{m=\-\(b\-1\)\}^\{b\-1\}\\big\(b\-\|m\|\\big\)^\{2\}=\\frac\{1\}\{b^\{4\}\}\\cdot\\frac\{b\(2b^\{2\}\+1\)\}\{3\}=\\frac\{2b^\{2\}\+1\}\{3b^\{3\}\}=\\Theta\(1/b\)\.\(13\)Summing theTTrows,‖Mtri‖F2=T⋅Θ​\(1/b\)\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{F\}^\{2\}=T\\cdot\\Theta\(1/b\), soEb=Θ​\(1/b\)E\_\{b\}=\\Theta\(1/b\)\. ∎

###### Theorem C\.8\(Effective rank does not collapse at proximal layers\)\.

Consider a proximal layer with bounded bandwidthb=O​\(1\)b=O\(1\)inTT, and suppose attention degenerates to a rank\-one sink with‖Aattn‖2=Θ​\(T\)\\\|A\_\{\\mathrm\{attn\}\}\\\|\_\{2\}=\\Theta\(\\sqrt\{T\}\)\(e\.g\.Aattn=𝟏​e1⊤A\_\{\\mathrm\{attn\}\}=\\mathbf\{1\}e\_\{1\}^\{\\top\}\)\. Then

lim infT→∞Reff​\(ℳ\)≥1\+g2​Eb\>1\.\\liminf\_\{T\\to\\infty\}R\_\{\\mathrm\{eff\}\}\(\\mathcal\{M\}\)\\ \\geq\\ 1\+g^\{2\}E\_\{b\}\\ \>\\ 1\.\(14\)

###### Proof\.

The banded row\-stochasticMtriM\_\{\\mathrm\{tri\}\}has‖Mtri‖∞=1\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{\\infty\}=1\(row sums\) and‖Mtri‖1=O​\(log⁡b\)\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{1\}=O\(\\log b\)\(column sums, the harmonic factor arising only at the firstO​\(b\)O\(b\)boundary columns\), so‖Mtri‖2≤‖Mtri‖1​‖Mtri‖∞=O​\(log⁡b\)=o​\(T\)\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{2\}\\leq\\sqrt\{\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{1\}\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{\\infty\}\}=O\(\\sqrt\{\\log b\}\)=o\(\\sqrt\{T\}\)\. By the triangle inequality‖ℳ‖2≤‖I‖2\+‖Aattn‖2\+g​‖Mtri‖2=T\+o​\(T\)\\\|\\mathcal\{M\}\\\|\_\{2\}\\leq\\\|I\\\|\_\{2\}\+\\\|A\_\{\\mathrm\{attn\}\}\\\|\_\{2\}\+g\\\|M\_\{\\mathrm\{tri\}\}\\\|\_\{2\}=\\sqrt\{T\}\+o\(\\sqrt\{T\}\), so‖ℳ‖22≤T\+o​\(T\)\\\|\\mathcal\{M\}\\\|\_\{2\}^\{2\}\\leq T\+o\(T\)\. Combining with[PropositionC\.7](https://arxiv.org/html/2607.20484#A3.Thmtheorem7),

Reff​\(ℳ\)=‖ℳ‖F2‖ℳ‖22≥T​\(1\+g2​Eb\)T\+o​\(T\)=1\+g2​Eb1\+o​\(1\)→T→∞1\+g2​Eb\.R\_\{\\mathrm\{eff\}\}\(\\mathcal\{M\}\)=\\frac\{\\\|\\mathcal\{M\}\\\|\_\{F\}^\{2\}\}\{\\\|\\mathcal\{M\}\\\|\_\{2\}^\{2\}\}\\ \\geq\\ \\frac\{T\(1\+g^\{2\}E\_\{b\}\)\}\{T\+o\(T\)\}=\\frac\{1\+g^\{2\}E\_\{b\}\}\{1\+o\(1\)\}\\ \\xrightarrow\[T\\to\\infty\]\{\}\\ 1\+g^\{2\}E\_\{b\}\.\(15\)Sinceg\>0g\>0andEb=Θ​\(1\)\>0E\_\{b\}=\\Theta\(1\)\>0for boundedbb, the limit exceeds11: at proximal layers the side\-path keeps the effective rank bounded away from the rank\-one value to which a pure sink collapses\. For wide \(distal\) layersb=Θ​\(T\)b=\\Theta\(T\)one hasEb=Θ​\(1/T\)E\_\{b\}=\\Theta\(1/T\), so the side\-path there contributes to global mixing \([SectionC\.3](https://arxiv.org/html/2607.20484#A3.SS3)\) rather than to rank preservation—precisely the proximal/distal division of §[3\.2](https://arxiv.org/html/2607.20484#S3.SS2)\. ∎

## Appendix DTheoretical Analysis of Spectral Properties

This appendix derives a*sufficiency*result: under explicit spectral conditions, the inference error of a class of global reasoning tasks admits an upper bound that decreases as the spectral gap and the effective rank increase\. The argument is a worst\-case bound rather than an exact characterization, and we state every assumption where it is used\.

Operator under analysis\.Consistent with §[1](https://arxiv.org/html/2607.20484#S1), the object here is the*row\-normalized transition operator*MMobtained by row\-stochastic normalization of the residual operatorℳ\\mathcal\{M\}of[AppendixC](https://arxiv.org/html/2607.20484#A3); it admits a stationary distributionπ\\piand acts as a Markov mixing operator, the standard setting for contraction analysis\(Levin and Peres,[2017](https://arxiv.org/html/2607.20484#bib.bib13)\)\. For the spectral\-gap step we assumeMMis*reversible*\(equivalently, we analyze itsπ\\pi\-reversibilization12​\(M\+M∗π\)\\tfrac\{1\}\{2\}\(M\+M^\{\*\_\{\\pi\}\}\), withM∗πM^\{\*\_\{\\pi\}\}the adjoint in⟨⋅,⋅⟩π\\langle\\cdot,\\cdot\\rangle\_\{\\pi\}\), so thatMMis self\-adjoint in⟨⋅,⋅⟩π\\langle\\cdot,\\cdot\\rangle\_\{\\pi\}with real spectrum in\[−1,1\]\[\-1,1\]and leading eigenvector𝟏\\mathbf\{1\}\. SinceMMandℳ\\mathcal\{M\}share the same connectivity, the gap and rank guarantees of[AppendixC](https://arxiv.org/html/2607.20484#A3)carry over; we work in theπ\\pi\-weighted norm‖u‖2,π2:=∑iπi​ui2\\\|u\\\|\_\{2,\\pi\}^\{2\}:=\\sum\_\{i\}\\pi\_\{i\}u\_\{i\}^\{2\}\.

### D\.1Assumptions and Setup

Letddbe the feature dimension andLLthe network depth\. We writeXℓ∈ℝT×dX\_\{\\ell\}\\in\\mathbb\{R\}^\{T\\times d\}for the state at layerℓ\\ell\. We model the prediction asy^=𝒩​\(X\)=G​\(ML​X\(0\)\)\\hat\{y\}=\\mathcal\{N\}\(X\)=G\\\!\\big\(M^\{L\}X^\{\(0\)\}\\big\), whereX\(0\)X^\{\(0\)\}is the injected input,MLM^\{L\}is the depth\-LLmixing, andG​\(z\)=f​\(𝒜−1​z\)G\(z\)=f\(\\mathcal\{A\}^\{\-1\}z\)is the readout with linear part𝒜:ℝd→ℝd\\mathcal\{A\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\}; the target isy=f​\(X\)y=f\(X\)withLf:=Lip​\(f\)L\_\{f\}:=\\mathrm\{Lip\}\(f\), and the inference error isℰ:=‖y^−y‖\\mathcal\{E\}:=\\\|\\hat\{y\}\-y\\\|\.

- •\(A1\) Reversible spectral gap\.MMis reversible with stationaryπ\\pi\(M​𝟏=𝟏M\\mathbf\{1\}=\\mathbf\{1\}, self\-adjoint in⟨⋅,⋅⟩π\\langle\\cdot,\\cdot\\rangle\_\{\\pi\}\) and contracts on the zero\-mean subspaceℋ0​\(π\)=\{v:∑iπi​vi=0\}\\mathcal\{H\}\_\{0\}\(\\pi\)=\\\{v:\\sum\_\{i\}\\pi\_\{i\}v\_\{i\}=0\\\}:‖M​v‖2,π≤\(1−γ\)​‖v‖2,π\\\|Mv\\\|\_\{2,\\pi\}\\leq\(1\-\\gamma\)\\\|v\\\|\_\{2,\\pi\}for allv∈ℋ0​\(π\)v\\in\\mathcal\{H\}\_\{0\}\(\\pi\), withγ∈\(0,1\)\\gamma\\in\(0,1\)\.
- •\(A2\) Non\-degenerate stationary distribution\.πmin≥cπ/T\\pi\_\{\\min\}\\geq c\_\{\\pi\}/Tfor a constantcπ\>0c\_\{\\pi\}\>0\.
- •\(A3\) High effective rank\.The stable rank satisfiessr​\(Xℓ\)=‖Xℓ‖F2/‖Xℓ‖22≥rmin\\mathrm\{sr\}\(X\_\{\\ell\}\)=\\\|X\_\{\\ell\}\\\|\_\{F\}^\{2\}/\\\|X\_\{\\ell\}\\\|\_\{2\}^\{2\}\\geq r\_\{\\min\}\.
- •\(A4\) Well\-conditioned readout\.𝒜\\mathcal\{A\}is invertible withκ​\(𝒜\)≤κ¯\\kappa\(\\mathcal\{A\}\)\\leq\\bar\{\\kappa\}, and the residual structure keepsσmax​\(𝒜\)≥c0\>0\\sigma\_\{\\max\}\(\\mathcal\{A\}\)\\geq c\_\{0\}\>0\.
- •\(A5\) Distributed, isotropic readout\.The target is recovered by aggregating evidence across thek=Θ​\(sr​\(Xℓ\)\)k=\\Theta\(\\mathrm\{sr\}\(X\_\{\\ell\}\)\)well\-conditioned feature directions of[LemmaD\.1](https://arxiv.org/html/2607.20484#A4.Thmtheorem1), with task\-irrelevant components that are uncorrelated across these directions and have per\-direction variance at mostσ⟂2\\sigma\_\{\\perp\}^\{2\}\.

The proof combines four lemmas:[LemmaD\.1](https://arxiv.org/html/2607.20484#A4.Thmtheorem1)\(feature injection, from A3/A5\),[LemmaD\.3](https://arxiv.org/html/2607.20484#A4.Thmtheorem3)\(exponential mixing, from A1\),[LemmaD\.4](https://arxiv.org/html/2607.20484#A4.Thmtheorem4)\(pointwise alignment, from A2\), and[LemmaD\.5](https://arxiv.org/html/2607.20484#A4.Thmtheorem5)\(readout stability, from A4\), assembled in[SectionD\.6](https://arxiv.org/html/2607.20484#A4.SS6)\.

### D\.2Lemma 1: Non\-degenerate, Rank\-rich Feature Injection

###### Lemma D\.1\(Restricted invertibility of the state\)\.

Under \(A3\), for anyε∈\(0,1\)\\varepsilon\\in\(0,1\)there is a feature subspaceSS\(a subset of theddfeature coordinates\) of dimensionk≥\(1−ε\)2​sr​\(Xℓ\)≥\(1−ε\)2​rmink\\geq\(1\-\\varepsilon\)^\{2\}\\,\\mathrm\{sr\}\(X\_\{\\ell\}\)\\geq\(1\-\\varepsilon\)^\{2\}r\_\{\\min\}on which the state map is uniformly well\-conditioned:

σmin\(Xℓ\|S\)≥ε‖Xℓ‖2d=:λlow\>0\.\\sigma\_\{\\min\}\\\!\\big\(X\_\{\\ell\}\|\_\{S\}\\big\)\\ \\geq\\ \\varepsilon\\,\\frac\{\\\|X\_\{\\ell\}\\\|\_\{2\}\}\{\\sqrt\{d\}\}\\ =:\\ \\lambda\_\{\\mathrm\{low\}\}\\ \>\\ 0\.\(16\)Consequently the restricted map is left\-invertible with∥\(Xℓ\|S\)†∥2≤1/λlow\\\|\(X\_\{\\ell\}\|\_\{S\}\)^\{\\dagger\}\\\|\_\{2\}\\leq 1/\\lambda\_\{\\mathrm\{low\}\}\.

###### Proof\.

This is the Spielman–Srivastava form of the Bourgain–Tzafriri restricted invertibility theorem\(Bourgain and Tzafriri,[1987](https://arxiv.org/html/2607.20484#bib.bib79); Spielman and Srivastava,[2012](https://arxiv.org/html/2607.20484#bib.bib80)\), applied toXℓ∈ℝT×dX\_\{\\ell\}\\in\\mathbb\{R\}^\{T\\times d\}with itsddcolumns as feature directions: for anyε∈\(0,1\)\\varepsilon\\in\(0,1\)there is a column subsetSSwith\|S\|≥\(1−ε\)2​‖Xℓ‖F2/‖Xℓ‖22=\(1−ε\)2​sr​\(Xℓ\)\|S\|\\geq\(1\-\\varepsilon\)^\{2\}\\\|X\_\{\\ell\}\\\|\_\{F\}^\{2\}/\\\|X\_\{\\ell\}\\\|\_\{2\}^\{2\}=\(1\-\\varepsilon\)^\{2\}\\,\\mathrm\{sr\}\(X\_\{\\ell\}\)andσmin​\(Xℓ\|S\)≥ε​‖Xℓ‖2/d\\sigma\_\{\\min\}\(X\_\{\\ell\}\|\_\{S\}\)\\geq\\varepsilon\\,\\\|X\_\{\\ell\}\\\|\_\{2\}/\\sqrt\{d\}\. Assumption \(A3\) lower\-bounds the size as\|S\|≥\(1−ε\)2​rmin\|S\|\\geq\(1\-\\varepsilon\)^\{2\}r\_\{\\min\}, and the singular\-value bound gives the stated pseudo\-inverse norm\. ∎

The lemma plays two roles\. Its singular\-value floorλlow\>0\\lambda\_\{\\mathrm\{low\}\}\>0certifies that the feature injection is*non\-degenerate*\(the readout can invert it stably\); its dimension count shows that the number of well\-conditioned, independently usable directions grows linearly with the effective rank\. The latter is what the effective rank buys, and it drives the following error reduction\.

###### Corollary D\.2\(Rank\-driven readout averaging\)\.

Under \(A3\) and \(A5\), the readout error contributed by the task\-irrelevant component is at mostσ⟂/k=O​\(σ⟂/sr​\(Xℓ\)\)\\sigma\_\{\\perp\}/\\sqrt\{k\}=O\\\!\\big\(\\sigma\_\{\\perp\}/\\sqrt\{\\mathrm\{sr\}\(X\_\{\\ell\}\)\}\\big\)\.

###### Proof\.

By[LemmaD\.1](https://arxiv.org/html/2607.20484#A4.Thmtheorem1)the target is recovered fromk≥\(1−ε\)2​sr​\(Xℓ\)k\\geq\(1\-\\varepsilon\)^\{2\}\\,\\mathrm\{sr\}\(X\_\{\\ell\}\)well\-conditioned directions\. Aggregatingkkuncorrelated, zero\-mean components of per\-direction variance at mostσ⟂2\\sigma\_\{\\perp\}^\{2\}\(A5\) yields an estimator whose irrelevant\-component variance is at mostσ⟂2/k\\sigma\_\{\\perp\}^\{2\}/k; taking square roots gives error≤σ⟂/k=O​\(σ⟂/sr​\(Xℓ\)\)\\leq\\sigma\_\{\\perp\}/\\sqrt\{k\}=O\(\\sigma\_\{\\perp\}/\\sqrt\{\\mathrm\{sr\}\(X\_\{\\ell\}\)\}\)\. ∎

### D\.3Lemma 2: Exponential Mixing

###### Lemma D\.3\.

Under \(A1\), the depth\-LLoperator satisfies‖ML‖ℋ0→ℋ0≤\(1−γ\)L\\\|M^\{L\}\\\|\_\{\\mathcal\{H\}\_\{0\}\\to\\mathcal\{H\}\_\{0\}\}\\leq\(1\-\\gamma\)^\{L\}, where∥⋅∥ℋ0→ℋ0\\\|\\cdot\\\|\_\{\\mathcal\{H\}\_\{0\}\\to\\mathcal\{H\}\_\{0\}\}is the operator norm induced by∥⋅∥2,π\\\|\\cdot\\\|\_\{2,\\pi\}on the zero\-mean subspace\.

###### Proof\.

By \(A1\),MMis self\-adjoint in⟨⋅,⋅⟩π\\langle\\cdot,\\cdot\\rangle\_\{\\pi\}withM​𝟏=𝟏M\\mathbf\{1\}=\\mathbf\{1\}, so its orthogonal complementℋ0​\(π\)=𝟏⟂π\\mathcal\{H\}\_\{0\}\(\\pi\)=\\mathbf\{1\}^\{\\perp\_\{\\pi\}\}isMM\-invariant; \(A1\) then gives‖M‖ℋ0→ℋ0=sup0≠v∈ℋ0‖M​v‖2,π/‖v‖2,π≤1−γ\\\|M\\\|\_\{\\mathcal\{H\}\_\{0\}\\to\\mathcal\{H\}\_\{0\}\}=\\sup\_\{0\\neq v\\in\\mathcal\{H\}\_\{0\}\}\\\|Mv\\\|\_\{2,\\pi\}/\\\|v\\\|\_\{2,\\pi\}\\leq 1\-\\gamma\. By submultiplicativity of the operator norm over theLLlayers,

‖ML‖ℋ0→ℋ0≤∏ℓ=1L‖M‖ℋ0→ℋ0≤\(1−γ\)L\.\\\|M^\{L\}\\\|\_\{\\mathcal\{H\}\_\{0\}\\to\\mathcal\{H\}\_\{0\}\}\\ \\leq\\ \\prod\_\{\\ell=1\}^\{L\}\\\|M\\\|\_\{\\mathcal\{H\}\_\{0\}\\to\\mathcal\{H\}\_\{0\}\}\\ \\leq\\ \(1\-\\gamma\)^\{L\}\.\(17\)∎

### D\.4Lemma 3: Pointwise Error Alignment

###### Lemma D\.4\.

Under \(A2\), any error vectoreesatisfies‖e‖∞≤T/cπ​‖e‖2,π\\\|e\\\|\_\{\\infty\}\\leq\\sqrt\{T/c\_\{\\pi\}\}\\,\\\|e\\\|\_\{2,\\pi\}\.

###### Proof\.

Leti⋆=arg⁡maxi⁡\|ei\|i^\{\\star\}=\\arg\\max\_\{i\}\|e\_\{i\}\|, so\|ei⋆\|=‖e‖∞\|e\_\{i^\{\\star\}\}\|=\\\|e\\\|\_\{\\infty\}\. Then

‖e‖2,π2=∑i=1Tπi​ei2≥πi⋆​ei⋆2≥πmin​‖e‖∞2≥cπT​‖e‖∞2,\\\|e\\\|\_\{2,\\pi\}^\{2\}=\\sum\_\{i=1\}^\{T\}\\pi\_\{i\}e\_\{i\}^\{2\}\\ \\geq\\ \\pi\_\{i^\{\\star\}\}e\_\{i^\{\\star\}\}^\{2\}\\ \\geq\\ \\pi\_\{\\min\}\\,\\\|e\\\|\_\{\\infty\}^\{2\}\\ \\geq\\ \\frac\{c\_\{\\pi\}\}\{T\}\\,\\\|e\\\|\_\{\\infty\}^\{2\},\(18\)using \(A2\) in the last step\. Rearranging gives‖e‖∞≤T/cπ​‖e‖2,π\\\|e\\\|\_\{\\infty\}\\leq\\sqrt\{T/c\_\{\\pi\}\}\\,\\\|e\\\|\_\{2,\\pi\}\. ∎

### D\.5Lemma 4: Readout Stability

###### Lemma D\.5\.

Under \(A4\), the readoutG​\(z\)=f​\(𝒜−1​z\)G\(z\)=f\(\\mathcal\{A\}^\{\-1\}z\)satisfiesLip​\(G\)≤Lf​κ¯/c0\\mathrm\{Lip\}\(G\)\\leq L\_\{f\}\\,\\bar\{\\kappa\}/c\_\{0\}\.

###### Proof\.

By the chain rule,Lip​\(G\)≤Lip​\(f\)​‖𝒜−1‖2=Lf/σmin​\(𝒜\)=Lf​κ​\(𝒜\)/σmax​\(𝒜\)\\mathrm\{Lip\}\(G\)\\leq\\mathrm\{Lip\}\(f\)\\,\\\|\\mathcal\{A\}^\{\-1\}\\\|\_\{2\}=L\_\{f\}/\\sigma\_\{\\min\}\(\\mathcal\{A\}\)=L\_\{f\}\\,\\kappa\(\\mathcal\{A\}\)/\\sigma\_\{\\max\}\(\\mathcal\{A\}\)\. Usingκ​\(𝒜\)≤κ¯\\kappa\(\\mathcal\{A\}\)\\leq\\bar\{\\kappa\}andσmax​\(𝒜\)≥c0\\sigma\_\{\\max\}\(\\mathcal\{A\}\)\\geq c\_\{0\}from \(A4\) givesLip​\(G\)≤Lf​κ¯/c0\\mathrm\{Lip\}\(G\)\\leq L\_\{f\}\\bar\{\\kappa\}/c\_\{0\}\. The lower boundσmax​\(𝒜\)≥c0\>0\\sigma\_\{\\max\}\(\\mathcal\{A\}\)\\geq c\_\{0\}\>0reflects that the residual \(identity\-skip\) structure keeps the mean squared activation length from vanishing, so the forward map does not contract to zero\(Hanin and Rolnick,[2018](https://arxiv.org/html/2607.20484#bib.bib81)\)\. ∎

### D\.6Sufficiency: the Error Bound

We assemble the lemmas into a bound onℰ=‖y^−y‖\\mathcal\{E\}=\\\|\\hat\{y\}\-y\\\|\. The argument is a worst\-case sketch: we bound the error by the product of how sensitively the readout reacts to its input and how far the mixed state is from the fully mixed \(stationary\) component,

ℰ≤\[Lip​\(G\)⋅ρavg\]⏟readout sensitivity⋅‖ML​X\(0\)−X¯‖⏟mixing residual,\\mathcal\{E\}\\ \\leq\\ \\underbrace\{\\big\[\\,\\mathrm\{Lip\}\(G\)\\cdot\\rho\_\{\\mathrm\{avg\}\}\\,\\big\]\}\_\{\\text\{readout sensitivity\}\}\\ \\cdot\\ \\underbrace\{\\big\\\|M^\{L\}X^\{\(0\)\}\-\\bar\{X\}\\big\\\|\}\_\{\\text\{mixing residual\}\},\(19\)whereX¯\\bar\{X\}is theπ\\pi\-stationary \(fully mixed\) component andρavg=O​\(1/sr​\(Xℓ\)\)\\rho\_\{\\mathrm\{avg\}\}=O\(1/\\sqrt\{\\mathrm\{sr\}\(X\_\{\\ell\}\)\}\)is the readout\-averaging gain of[CorollaryD\.2](https://arxiv.org/html/2607.20484#A4.Thmtheorem2)\.

*Readout sensitivity\.*By[LemmaD\.5](https://arxiv.org/html/2607.20484#A4.Thmtheorem5),Lip​\(G\)≤Lf​κ¯/c0\\mathrm\{Lip\}\(G\)\\leq L\_\{f\}\\bar\{\\kappa\}/c\_\{0\}; by[CorollaryD\.2](https://arxiv.org/html/2607.20484#A4.Thmtheorem2), aggregating over thek=Θ​\(sr​\(Xℓ\)\)k=\\Theta\(\\mathrm\{sr\}\(X\_\{\\ell\}\)\)well\-conditioned directions of[LemmaD\.1](https://arxiv.org/html/2607.20484#A4.Thmtheorem1)contributesρavg=O​\(1/sr​\(Xℓ\)\)\\rho\_\{\\mathrm\{avg\}\}=O\(1/\\sqrt\{\\mathrm\{sr\}\(X\_\{\\ell\}\)\}\)\. Hence the readout sensitivity isO​\(κ​\(𝒜\)/sr​\(Xℓ\)\)O\\\!\\big\(\\kappa\(\\mathcal\{A\}\)/\\sqrt\{\\mathrm\{sr\}\(X\_\{\\ell\}\)\}\\big\)\.

*Mixing residual\.*By[LemmaD\.3](https://arxiv.org/html/2607.20484#A4.Thmtheorem3),‖ML​X\(0\)−X¯‖2,π≤\(1−γ\)L​‖X\(0\)−X¯‖2,π\\\|M^\{L\}X^\{\(0\)\}\-\\bar\{X\}\\\|\_\{2,\\pi\}\\leq\(1\-\\gamma\)^\{L\}\\\|X^\{\(0\)\}\-\\bar\{X\}\\\|\_\{2,\\pi\}; by[LemmaD\.4](https://arxiv.org/html/2607.20484#A4.Thmtheorem4)and \(A2\), the pointwise residual obeys‖ML​X\(0\)−X¯‖∞≤T/cπ​\(1−γ\)L​‖X\(0\)−X¯‖2,π\\\|M^\{L\}X^\{\(0\)\}\-\\bar\{X\}\\\|\_\{\\infty\}\\leq\\sqrt\{T/c\_\{\\pi\}\}\\,\(1\-\\gamma\)^\{L\}\\\|X^\{\(0\)\}\-\\bar\{X\}\\\|\_\{2,\\pi\}\.

Combining the two factors, for constantsC1,C2\>0C\_\{1\},C\_\{2\}\>0\(absorbingLf,κ¯,c0,σ⟂L\_\{f\},\\bar\{\\kappa\},c\_\{0\},\\sigma\_\{\\perp\}and‖X\(0\)−X¯‖2,π\\\|X^\{\(0\)\}\-\\bar\{X\}\\\|\_\{2,\\pi\}\),

ℰ≤C1​κ​\(𝒜\)sr​\(Xℓ\)⏟stability \(A3–A5\)⋅C2​Tcπ​\(1−γ\)L⏟mixing \(A1–A2\)\.\\boxed\{\\ \\mathcal\{E\}\\ \\leq\\ \\underbrace\{\\frac\{C\_\{1\}\\,\\kappa\(\\mathcal\{A\}\)\}\{\\sqrt\{\\mathrm\{sr\}\(X\_\{\\ell\}\)\}\}\}\_\{\\text\{stability \(A3\-\-A5\)\}\}\\cdot\\underbrace\{C\_\{2\}\\sqrt\{\\tfrac\{T\}\{c\_\{\\pi\}\}\}\\,\(1\-\\gamma\)^\{L\}\}\_\{\\text\{mixing \(A1\-\-A2\)\}\}\\ \.\}\(20\)The bound decreases as the effective ranksr​\(Xℓ\)\\mathrm\{sr\}\(X\_\{\\ell\}\)and the spectral gapγ\\gammaincrease and as the condition numberκ​\(𝒜\)\\kappa\(\\mathcal\{A\}\)decreases—precisely the spectral quantities that TRSP regularizes\. The factorT/cπ\\sqrt\{T/c\_\{\\pi\}\}is the worst\-case amplification from theπ\\pi\-weightedL2L\_\{2\}norm to the pointwise norm \([LemmaD\.4](https://arxiv.org/html/2607.20484#A4.Thmtheorem4)\); when \(A2\) is tight,πmin=Θ​\(1/T\)\\pi\_\{\\min\}=\\Theta\(1/T\)and this factor is of order1/πmin1/\\sqrt\{\\pi\_\{\\min\}\}\. Consequently the bound is small only when the depth\-driven contraction\(1−γ\)L\(1\-\\gamma\)^\{L\}dominates theT\\sqrt\{T\}amplification, i\.e\. when the gap is preserved as length grows\. We emphasize that this is a worst\-case sufficiency bound under \(A1\)–\(A5\); in particular the1/sr​\(Xℓ\)1/\\sqrt\{\\mathrm\{sr\}\(X\_\{\\ell\}\)\}improvement relies on the distributed\-readout assumption \(A5\), whereas restricted invertibility \([LemmaD\.1](https://arxiv.org/html/2607.20484#A4.Thmtheorem1)\) alone guarantees only the non\-degeneracy floorλlow\>0\\lambda\_\{\\mathrm\{low\}\}\>0\.