Attention-Aware Routing: Coupling Routing and Attention in MoEs

# Attention-Aware Routing: Coupling Routing and Attention in MoEs
Source: [https://arxiv.org/html/2609.20974](https://arxiv.org/html/2609.20974)
Despoina KosmopoulouAnastasios TsetsilasAffiliation:National Technical University of AthensEfthymios GeorgiouAffiliation:University of BernGiannis KaramanolakisAffiliation:Amazon AGISwastik RoyAffiliation:Amazon AGIAlexandros PotamianosAffiliation:National Technical University of AthensAffiliation:Archimedes RU, Athena RC

###### Abstract

In Mixture\-of\-Experts language models, the router typically selects and weights experts based on the token’s hidden state, utilizing limited contextual information\. We proposeAttention\-Aware Routing \(AAR\), which augments the router with temporal and spectral features extracted from a sliding window of attention weights that represent a summary of the model’scontextualstate, disentangled from the hidden state\. Keeping the base transformer entirely frozen, we train only the routing parameters, isolating routing as the sole variable\. AAR improves GSM8K by\+3\.37\+3\.37pp over a routing\-only SFT baseline on OLMoE\. Beyond performance, we show that routing and attention form a coupled circuit: routing changes at layerllpropagate through the residual stream to amplify attention sinks at layerl\+1l\+1, reshaping attention without any direct update to the attention mechanism itself\. Further, AAR reduces long diverging generation, with incorrect answers getting shorter, while correct answers remain unchanged in length\. Finally, AAR is strongly depth\-sensitive: applying it indiscriminately across layers can degrade factual retrieval, whereas mathematical reasoning gains persist when it is introduced deeper in the network\. This sensitivity exposes a retrieval–reasoning tension across depth and makes layer\-selective AAR a controlled probe of the routing\-relevant information carried by attention at different layers\.

††∗Work done while at NTUA\.†††This work is independent of and outside of the work at Amazon\.## 1Introduction

Mixture\-of\-Experts \(MoE\) architectures\([Shazeer et al\., 2017](https://arxiv.org/html/2609.20974#bib.bib1);[Fedus et al\., 2022](https://arxiv.org/html/2609.20974#bib.bib2);[Lepikhin et al\., 2021](https://arxiv.org/html/2609.20974#bib.bib20)\)scale language model capacity while keeping inference cost manageable by activating only a subset of experts per token\. The router, the mechanism that decides which experts process each token, is therefore a crucial component\([Dikkala et al\., 2023](https://arxiv.org/html/2609.20974#bib.bib21)\): it not only ensures the efficient allocation of computation, but determines which computational pathways shape each token’s representation at every layer\.

Standard routers make this decision typically utilizing the hidden state as their sole input\. Given a token’s hidden state𝐡il\\mathbf\{h\}\_\{i\}^\{l\}, the router computes a linear projection and selects the top\-kkexperts to process it\([Shazeer et al\., 2017](https://arxiv.org/html/2609.20974#bib.bib1)\)\. The hidden state carries contextual information, entangled with other content information within the hidden state\. In this work, we investigate how a model’s performance and behavior can change, if disentanglement of content and contextual relational information is enforced as an inductive routing bias\.

To achieve a simple form of contextual disentanglement, the router can be augmented with a window of recent attention weights\. While hidden states summarize what each token represents, attention weights encode compressed relational intra\-sequence structure: which positions the model attends to, which it ignores, and where focus concentrates\.

We proposeAttention\-Aware Routing \(AAR\), which augments the standard router with a sliding window of attention weights\. The attention weights provide a short summary of how attention was distributed in the latest generation steps\. Training only the routing parameters while keeping the base transformer frozen, AAR isolates routing as the sole variable\. Our main contributions are:

1. 1\.AAR improves mathematical reasoning\.We show that endowing the router with contextual information directly improves downstream performance\. AAR improves GSM8K by\+3\.37\+3\.37pp over routing\-only SFT on OLMoE\([Muennighoff et al\., 2025](https://arxiv.org/html/2609.20974#bib.bib8)\), with consistent gains on mathematical reasoning across benchmarks,achieved by training the routers alone\.
2. 2\.AAR enables and exploits a routing\-attention coupling\.AAR modifies generation from routing alone\. Internally, routing changes at layerllpropagate through the residual stream to amplify attention sinks at layerl\+1l\+1, showing that routing and attention form a coupled circuit\. Externally, AAR reduces diverging generation length: incorrect answers get shorter while correct answers are unaffected\.
3. 3\.AAR reveals a depth\-dependent retrieval–reasoning trade\-off\.AAR is not equally beneficial across layers: early\-layer interventions can disrupt factual retrieval, while mathematical reasoning gains persist when AAR is applied deeper in the network\. Across models, middle\-to\-deep layers provide a useful starting point for effective AAR placement\. This depth sensitivity also turns layer\-selective AAR into a controlled probe of where routing\-relevant information is encoded in attention across the network\.

## 2Related Work

#### MoE routing\.

Standard token\-choice routing computes a softmax over expert logits derived from the token representation and selects the top\-kkexperts\([Shazeer et al\., 2017](https://arxiv.org/html/2609.20974#bib.bib1)\)\. Load\-balancing auxiliary losses encourage uniform expert utilization\([Fedus et al\., 2022](https://arxiv.org/html/2609.20974#bib.bib2)\)\. Expert\-choice routing\([Zhou et al\., 2022](https://arxiv.org/html/2609.20974#bib.bib3)\)inverts the assignment\. Our work keeps token\-choice routing intact and augments it with an attention\-conditioned auxiliary signal\.

#### Various Routers\.

[Qiu et al\. \(2025\)](https://arxiv.org/html/2609.20974#bib.bib16)implement MoE routing with a recurrent unit, shared across layers\. The input to the router at each layer comes from the hidden state at the current layer, as well as previous routing decisions \(the recurrent unit’s hidden state\)\. In[Do et al\. \(2023\)](https://arxiv.org/html/2609.20974#bib.bib18)a fixed randomly initialized hypernetwork generates the router’s parameters, conditioned on a trainable router embedding\. The HyperRouter alleviates representation collapse, achieving low\-entropy routing, and limits the number of experts used during inference\. Similar to these works, our routing method does not rely on the hidden state alone to generate each layer’s routing logits, but incorporates attention weights as an additional signal\.

#### Attention sinks\.

[Xiao et al\. \(2024\)](https://arxiv.org/html/2609.20974#bib.bib13)observe that transformers concentrate disproportionate attention mass on the first token \(the “attention sink”\), which stabilizes representations under streaming contexts\.[Barbero et al\. \(2025\)](https://arxiv.org/html/2609.20974#bib.bib22)show that sink presence slows information mixing and increases robustness to prompt perturbations\. We find that AAR systematically strengthens attention sinks in layers following the routing modification, linking this to improved reasoning\.

#### Layer\-wise functional specialization\.

[Yang et al\. \(2026\)](https://arxiv.org/html/2609.20974#bib.bib24),[Song et al\. \(2026\)](https://arxiv.org/html/2609.20974#bib.bib4), show that shallow LLM layers dominate likelihood and retrieval tasks while mid\-to\-deep layers are essential for reasoning and generation\.[Yang et al\. \(2025\)](https://arxiv.org/html/2609.20974#bib.bib5)distinguish latent reasoning from factual shortcuts across depth\.

#### Attention\-aware interventions for reasoning\.

[Nguyen et al\. \(2026\)](https://arxiv.org/html/2609.20974#bib.bib6)improve chain\-of\-thought reasoning through attention\-aware interventions at inference time, based on empirical hard\-written rules\. Our work differs in that the attention signal is incorporated into the routing mechanism during training, making the effect persistent, learnable and architecture\-native\.

#### Overthinking in chain\-of\-thought\.

[Li et al\. \(2026\)](https://arxiv.org/html/2609.20974#bib.bib7),[Su et al\. \(2025\)](https://arxiv.org/html/2609.20974#bib.bib19)show that longer reasoning chains correlate with higher error rates\. Our finding that AAR reduces output length specifically on incorrect answers connects routing\-induced changes to this line of research\.

## 3Method

### 3\.1Standard MoE Routing

In a typical MoE layer of a transformer LLM at depthll, given a token representation𝐡il\\mathbf\{h\}\_\{i\}^\{l\}after the attention sub\-layer, the router computes expert selection logits as𝐠i=Wr​𝐡il\\mathbf\{g\}\_\{i\}=W\_\{r\}\\mathbf\{h\}\_\{i\}^\{l\}, whereWr∈ℝE×dW\_\{r\}\\in\\mathbb\{R\}^\{E\\times d\}andEEis the number of experts\. Top\-kkexperts are selected after applying a softmax over𝐠i\\mathbf\{g\}\_\{i\}, and the token is processed by a weighted combination of the selected experts’ outputs\([Shazeer et al\., 2017](https://arxiv.org/html/2609.20974#bib.bib1)\)\.

### 3\.2Attention\-Aware Routing

Attention\-Aware Routing operates in parallel to the standard router\. In each selected MoE layerll, an untrained routing module is introduced,AlA\_\{l\}\. Instead of the hidden state, this module routes based on a short window of the token’s attention weights to previous tokens of the sequence\.

![Refer to caption](https://arxiv.org/html/2609.20974v1/plots/arr-final-image.png)Figure 1:Attention\-Aware Routing\.For the current tokenii, attention weights to the previousMMtokens are averaged across heads and assembled into a sliding window𝐰\\mathbf\{w\}, then transformed into the augmented window𝐰~=ϕ⁡\(𝐰\)\\tilde\{\\mathbf\{w\}\}=\\phi\(\\mathbf\{w\}\)\. The attention routerAlA\_\{l\}maps𝐰~\\tilde\{\\mathbf\{w\}\}to expert logits𝐠attn\\mathbf\{g\}^\{\\text\{attn\}\}, which are merged and gated with the standard router’s logits𝐠std=Wr​𝐡\\mathbf\{g\}^\{\\text\{std\}\}=W\_\{r\}\\mathbf\{h\}to produce the final routing decision over expertsE1,…,ENE\_\{1\},\\ldots,E\_\{N\}\.#### Attention window\.

The attention window is a sliding window of sizeMM, including only attention weights to the lastMMtokens in the token’s context\. For the cases there are less thanMMtokens early in the sequence, the window is zero\-padded from the left, keeping its length constant\. The attention weights to the tokens in the lastMMpositions are averaged across the attention heads, assembling into a 1\-dimensional window consisting of one attention weight per previous token\. We denote the resulting raw window by𝐰il∈ℝM\\mathbf\{w\}\_\{i\}^\{l\}\\in\\mathbb\{R\}^\{M\}\.

#### Augmented attention window\.

The attention weights can be further transformed post assembly, to allow learning of more features by the linear routing layer\. Specifically, the augmented attention window𝐰~il\\tilde\{\\mathbf\{w\}\}\_\{i\}^\{l\}is obtained by applying a fixed, non\-learned feature mapϕ\\phito the raw window,

𝐰~il=ϕ⁡\(𝐰il\)∈ℝM′\.\\tilde\{\\mathbf\{w\}\}\_\{i\}^\{l\}=\\phi\(\\mathbf\{w\}\_\{i\}^\{l\}\)\\in\\mathbb\{R\}^\{M^\{\\prime\}\}\.\(1\)The mapϕ\\phicollects one or more views of the window\. In the time domain,ϕ\\phiretains the raw window as\-is \(AART\{\}^\{\\text\{T\}\}\)\. We additionally consider a frequency\-domain view: the DFT magnitude of the window, which is shift\-invariant and exposes spectral structure that a single linear layer could not otherwise recover from the raw window\. Concatenating both views,

ϕ\(𝐰il\)=\[𝐰il∥log\(1\+\|ℱ\(𝐰il\)\|\)\],\\phi\(\\mathbf\{w\}\_\{i\}^\{l\}\)=\\big\[\\,\\mathbf\{w\}\_\{i\}^\{l\}\\;\\\|\\;\\log\\\!\\big\(1\+\|\\mathcal\{F\}\(\\mathbf\{w\}\_\{i\}^\{l\}\)\|\\big\)\\,\\big\],\(2\)whereℱ\\mathcal\{F\}denotes the DFT, we retain the firstK=⌊M/2⌋\+1K=\\lfloor M/2\\rfloor\+1magnitude bins\. Thelog\(1\+⋅\)\\log\(1\+\\cdot\)compresses the dynamic range of the magnitudes before they enter the router\. This givesM′=M\+KM^\{\\prime\}=M\+K; the time\-only variant is recovered by settingϕ⁡\(𝐰il\)=𝐰il\\phi\(\\mathbf\{w\}\_\{i\}^\{l\}\)=\\mathbf\{w\}\_\{i\}^\{l\}, for whichM′=MM^\{\\prime\}=M\.

#### Attention router\.

The augmented window is mapped to expert logits by a single linear router,

𝐠iattn=Al⊤​𝐰~il,Al∈ℝM′×E\.\\mathbf\{g\}^\{\\text\{attn\}\}\_\{i\}=A\_\{l\}^\{\\top\}\\tilde\{\\mathbf\{w\}\}\_\{i\}^\{l\},\\qquad A\_\{l\}\\in\\mathbb\{R\}^\{M^\{\\prime\}\\times E\}\.\(3\)We takeAlA\_\{l\}to be block\-diagonal across the views assembled byϕ\\phi, with one block per view, so that each view is routed by its own block and no cross\-view weights are introduced\.

#### Router interpolation\.

Each view is gated by a confidence weight conditioned on the hidden state and that view’s window\. With𝐠iattn∈ℝ2​E\\mathbf\{g\}^\{\\text\{attn\}\}\_\{i\}\\in\\mathbb\{R\}^\{2E\}the per\-view attention logits andGlG\_\{l\}block\-diagonal,

𝜶i=σ\(Gl\[𝐡il∥𝐰~il\]\)∈ℝ2​E,\\boldsymbol\{\\alpha\}\_\{i\}=\\sigma\\\!\\big\(G\_\{l\}\\,\[\\,\\mathbf\{h\}\_\{i\}^\{l\}\\;\\\|\\;\\tilde\{\\mathbf\{w\}\}\_\{i\}^\{l\}\\,\]\\big\)\\in\\mathbb\{R\}^\{2E\},\(4\)so each gate sees only the hidden state and its own window\. SettingB=12\[IE∥IE\]B=\\tfrac\{1\}\{2\}\[\\,I\_\{E\}\\;\\\|\\;I\_\{E\}\\,\]and𝜷0=𝟏−B​𝜶i\\boldsymbol\{\\beta\}\_\{0\}=\\mathbf\{1\}\-B\\boldsymbol\{\\alpha\}\_\{i\}, the standard router keeps the remaining per\-expert weight and the final logits are

𝐠i=𝜷0⊙𝐠istd\+B⁡\(𝜶i⊙𝐠iattn\)\.\\mathbf\{g\}\_\{i\}=\\boldsymbol\{\\beta\}\_\{0\}\\odot\\mathbf\{g\}^\{\\text\{std\}\}\_\{i\}\+B\\,\\big\(\\boldsymbol\{\\alpha\}\_\{i\}\\odot\\mathbf\{g\}^\{\\text\{attn\}\}\_\{i\}\\big\)\.\(5\)Standard routing is recovered when𝜶i≡𝟎\\boldsymbol\{\\alpha\}\_\{i\}\\equiv\\mathbf\{0\}\.

## 4Experiments

ConfigurationGSM8KBBHMMLUMMLU\-MathHumanEvalIFEval8\-shot CoT acc\.0\-shot acc\.5\-shot acc\.5\-shot acc\.pass@10loose acc\.OLMoE\-0125\-SFT51\.3336\.8053\.9631\.0141\.4656\.19\+ routing\-only SFT51\.99±0\.76\{\\scriptstyle\\pm 0\.76\}36\.17±0\.45\{\\scriptstyle\\pm 0\.45\}53\.91±0\.01\{\\scriptstyle\\pm 0\.01\}30\.45±0\.56\{\\scriptstyle\\pm 0\.56\}41\.46±0\.80\{\\scriptstyle\\pm 0\.80\}56\.56±1\.09\{\\scriptstyle\\pm 1\.09\}\+ AARS,M=40M=4055\.36±0\.74\{\\scriptstyle\\pm 0\.74\}36\.71±0\.46\{\\scriptstyle\\pm 0\.46\}53\.96±0\.20\{\\scriptstyle\\pm 0\.20\}32\.91±0\.54\{\\scriptstyle\\pm 0\.54\}40\.00±1\.26\{\\scriptstyle\\pm 1\.26\}55\.64±0\.41\{\\scriptstyle\\pm 0\.41\}\+ AARS,M=20M=2054\.89±1\.02\{\\scriptstyle\\pm 1\.02\}36\.30±0\.42\{\\scriptstyle\\pm 0\.42\}53\.93±0\.10\{\\scriptstyle\\pm 0\.10\}33\.48±0\.49\{\\scriptstyle\\pm 0\.49\}40\.98±1\.80\{\\scriptstyle\\pm 1\.80\}56\.67±1\.36\{\\scriptstyle\\pm 1\.36\}\+ AARST\{\}^\{\\text\{T\}\}\_\{S\},M=20M=2054\.57±1\.10\{\\scriptstyle\\pm 1\.10\}36\.33±0\.61\{\\scriptstyle\\pm 0\.61\}53\.78±0\.14\{\\scriptstyle\\pm 0\.14\}32\.29±0\.64\{\\scriptstyle\\pm 0\.64\}40\.85±1\.14\{\\scriptstyle\\pm 1\.14\}56\.34±1\.88\{\\scriptstyle\\pm 1\.88\}Qwen1\.5\-MoE\-A2\.7B\-Chat\+ routing\-only SFT53\.48±1\.45\{\\scriptstyle\\pm 1\.45\}40\.96±0\.43\{\\scriptstyle\\pm 0\.43\}59\.22±0\.04\{\\scriptstyle\\pm 0\.04\}42\.06±0\.43\{\\scriptstyle\\pm 0\.43\}39\.26±0\.83\{\\scriptstyle\\pm 0\.83\}36\.45±1\.26\{\\scriptstyle\\pm 1\.26\}\+ AARSQ\{\}\_\{S\_\{Q\}\},M=40M=4055\.36±1\.41\{\\scriptstyle\\pm 1\.41\}40\.51±0\.36\{\\scriptstyle\\pm 0\.36\}59\.02±0\.13\{\\scriptstyle\\pm 0\.13\}41\.40±0\.41\{\\scriptstyle\\pm 0\.41\}40\.48±2\.27\{\\scriptstyle\\pm 2\.27\}35\.82±0\.97\{\\scriptstyle\\pm 0\.97\}

Table 1:AAR compared against trained baselines\. GSM8K consistently improves, with full AAR achieving the best results\.S=\{9,…,15\}S=\\\{9,\\ldots,15\\\}for OLMoE\.SQ=\{14,…,17\}S\_\{Q\}=\\\{14,\\ldots,17\\\}for Qwen1\.5\. Best in bold\.### 4\.1Setup

#### Model\.

For the main experiments, we use OLMoE\-1B\-7B\-SFT\([Muennighoff et al\., 2025](https://arxiv.org/html/2609.20974#bib.bib8)\)111[allenai/OLMoE\-1B\-7B\-0125\-SFT](https://allenai/OLMoE-1B-7B-0125-SFT), a 7B\-parameter MoE model with 1B active parameters per token, 16 MoE layers, and 64 experts per layer \(top\-kkrouting withk=8k=8\)\.

#### Training\.

We fine\-tune only the routing parameters: the original router weightsWrlW\_\{r\}^\{l\}and the auxiliary parameters\(Al,Gl\)\(A\_\{l\},G\_\{l\}\), while keeping all other model parameters frozen\. We use the AdamW optimizer\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.20974#bib.bib26)\)with a batch size of10241024, sequence length of512512, a linear learning rate schedule with LR=1×10−4=1\\times 10^\{\-4\}, and warmup ratio0\.030\.03\. Each configuration is trained with 5 random seeds\. Training is performed on the Leonardo supercomputer at CINECA, using 4×\\timesA100 64GB GPUs; a single OLMoE run takes approximately 4 GPU hours\.

#### Data\.

We fine\-tune on a 25% sample \(∼\\sim220K examples\) of Tulu3\([Lambert et al\., 2025](https://arxiv.org/html/2609.20974#bib.bib15)\), the SFT dataset used for OLMoE, including high\-quality math, code, and instruction\-following samples\.

#### Baselines\.

We compare AAR against the original OLMoE\-0125\-SFT model and identical routing\-only SFT runs without the auxiliary routers, isolating the effect of the additional attention signal from that of continued training\. We verify our results with the same comparisons on Qwen1\.5\([Team, 2024](https://arxiv.org/html/2609.20974#bib.bib23)\)222[Qwen/Qwen1\.5\-MoE\-A2\.7B\-Chat](https://qwen/Qwen1.5-MoE-A2.7B-Chat)\(24 layers, 2\.7B active parameters, 14\.3B in total\), and mathematics\-focused evaluation for Qwen3\.6\([Qwen Team, 2026](https://arxiv.org/html/2609.20974#bib.bib27)\)\(40 layers, 3B active parameters, 35B in total\) with their corresponding chat templates\.

#### Evaluation\.

We evaluate on GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2609.20974#bib.bib9)\), BBH\([Suzgun et al\., 2023](https://arxiv.org/html/2609.20974#bib.bib10)\), MMLU\([Hendrycks et al\., 2021](https://arxiv.org/html/2609.20974#bib.bib17)\), MATH\-500\([Lightman et al\., 2023](https://arxiv.org/html/2609.20974#bib.bib28)\), HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2609.20974#bib.bib11)\), and IFEval\([Zhou et al\., 2023](https://arxiv.org/html/2609.20974#bib.bib12)\)\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2609.20974#S4.T1)consolidates results across reasoning and factual benchmarks\. AAR, applied to layersS=\{9,…,15\}S=\\\{9,\\ldots,15\\\}withM=40M=40, achieves the best GSM8K performance at55\.36±0\.7455\.36\\pm 0\.74, a gain of\+3\.37\+3\.37pp over routing\-only SFT\. AART\{\}^\{\\text\{T\}\}\(AAR utilizing only time\-domain weights\) atM=20M=20already improves GSM8K from51\.9951\.99to54\.5754\.57\(\+2\.58\+2\.58pp\), with the full AAR providing further gain\. Performance on BBH, HumanEval, and IFEval is maintained within noise for both\. GSM8K gains replicate on Qwen1\.5 and are reported in Table[1](https://arxiv.org/html/2609.20974#S4.T1)\.

#### Generalization across benchmarks and scale\.

We next test how AAR’s mathematical reasoning gains extend beyond GSM8K and persist at a larger model scale\. On MATH\-500, AAR improves OLMoE by\+2\.64\+2\.64pp over routing\-only SFT \(Table[2](https://arxiv.org/html/2609.20974#S4.T2)\), showing that the gains extend to a distinct mathematical reasoning benchmark\.

We further evaluate AAR on Qwen3\.6\-MoE, a larger 3B\-active/35B\-total model with 40 layers\. We apply AAR \(M=40M=40\) to full\-attention layers\{23,27,31\}\\\{23,27,31\\\}, located in the middle\-to\-deep portion of the network, and train on 12\.8K samples without architecture\-specific tuning\. AAR improves MATH\-500 by\+1\.13\+1\.13pp over three seeds \(Table[2](https://arxiv.org/html/2609.20974#S4.T2)\), despite the substantially stronger baseline and reduced training budget\. Together, these results show that AAR’s mathematical reasoning gains are not specific to GSM8K and persist on a larger model\.

MethodMATH\-500OLMoERouting\-only SFT14\.00±\\pm1\.83AAR \(L​9L9–1515\)16\.64±\\pm0\.91Qwen3\.6\-MoERouting\-only SFT78\.47±\\pm1\.30AAR \(L​23,27,31L23,27,31\)79\.60±\\pm1\.51Table 2:MATH\-500 accuracy for AARTF\{\}^\{\\text\{TF\}\}and routing\-only SFT\. OLMoE results are averaged over 5 seeds and Qwen3\.6\-MoE results over 3 seeds\.

### 4\.3Window Size and Signal Structure

#### Window size\.

We find that medium\-sized windows,M=20M=20–4040, work best for Attention\-Aware Routing\. Table[3](https://arxiv.org/html/2609.20974#S4.T3)reports GSM8K accuracy as a function ofMMfor AART\{\}^\{\\text\{T\}\}, withS=\{9,…,15\}S=\\\{9,\\ldots,15\\\}: AART\{\}^\{\\text\{T\}\}peaks atM=20M=20, and degrades for large windows\. AAR reaches its best performance atM=40M=40\(Table[1](https://arxiv.org/html/2609.20974#S4.T1)\)\.

#### Intrinsic correlation length of the attention signal\.

The optimal window size for AART\{\}^\{\\text\{T\}\}is explained by the temporal structure of the attention signal itself\. We analyze𝐰il\\mathbf\{w\}\_\{i\}^\{l\}via its normalized autocorrelation function \(ACF\), computed for GSM8K prompts and averaged over query tokens\. In Figure[2](https://arxiv.org/html/2609.20974#S4.F2)we plot the ACF and mark the lag at which it first decays below0\.10\.1, across layers and window sizesMM\. Even atM=128M=128, theeffective memory lengthsaturates at roughly 20 tokens across layers: beyond this lag, successive attention weights are uncorrelated\. This reveals an intrinsic correlation length of∼\\sim20 tokens in the attention signal, so for AART\{\}^\{\\text\{T\}\}a window ofM=20M=20is sufficient to capture the linearly structured information it carries\.

![Refer to caption](https://arxiv.org/html/2609.20974v1/plots/acf_curves.png)Figure 2:Normalized autocorrelation function \(ACF\) of the attention window signalwil​\(t\)w^\{l\}\_\{i\}\(t\)at layers 4, 9, and 12, for example window sizes∈\{20,64,128\}\\in\\\{20,64,128\\\}\. The ACF is averaged over query tokens and prompts from GSM8K \(5\-shot\)\. The dashed vertical line marks lag=20=20\. Across all layers and window sizes, the ACF decays below0\.10\.1by lag∼20\\sim\\\!20, revealing an intrinsic correlation length of2020tokens in the attention signal\. A window ofM=20M=20is thus sufficient to capture most structured information\.ConfigurationMMGSM8Krouting\-only SFT–51\.99±0\.76\{\\scriptstyle\\pm 0\.76\}AART1051\.99±1\.13\{\\scriptstyle\\pm 1\.13\}2054\.10±0\.52\{\\scriptstyle\\pm 0\.52\}12853\.74±0\.91\{\\scriptstyle\\pm 0\.91\}51252\.96±0\.76\{\\scriptstyle\\pm 0\.76\}Table 3:Window size ablation for AART, applied in all layers\. Best in bold\.

## 5From Routing to Generation: Internal and External Effects of AAR

We analyze the effects of AAR on model behavior at two levels: internally, by examining how routing changes propagate to the attention mechanism, and externally, by examining how they manifest in the model’s generation\. For clarity, both analyses use AART\{\}^\{\\text\{T\}\}, which isolates the effect of the time\-domain attention signal alone\.

### 5\.1Internal: AAR Reshapes Attention

To understand the low\-level changes underlying the reasoning improvements, we examine whether and how the attention weights themselves differ between AART\{\}^\{\\text\{T\}\}and baseline models\. We collect attention weights generated when processing GSM8K prompts, using the AART\{\}^\{\\text\{T\}\}models of Table[1](https://arxiv.org/html/2609.20974#S4.T1)withM=20M=20and their corresponding baselines\.

We find that AART\{\}^\{\\text\{T\}\}amplifies attention sinks,i\.e\.the concentration of attention mass on the first token of the sequence\([Xiao et al\., 2024](https://arxiv.org/html/2609.20974#bib.bib13)\), and verify that this effect is associated with AAR via the layer\-selective variant: when AAR is applied from layerllonwards, attention sinks are amplified starting precisely in layerl\+1l\+1\(Figure[3](https://arxiv.org/html/2609.20974#S5.F3)\)\. Routing changes at layerllalter expert selection and weighting, which in turn reshapes the representations passed to layerl\+1l\+1’s attention submodule,without any direct update to the attention mechanism itself\.

![Refer to caption](https://arxiv.org/html/2609.20974v1/plots/attention_sink_jump.png)Figure 3:Top: Absolute weight to the attention sink, per layer \(averaged across prompt’s tokens\)\. Bottom: the absolute difference, isolated\. The different routing decisions of layer 9, because of AAR, lead to different attention weights produced in the attention submodule, at layer 10\.We hypothesize that this mechanism could be one of the ways AAR improves mathematical reasoning\. Prior work has established that attention sink formation stabilizes decoding by anchoring computation to a fixed reference token\([Xiao et al\., 2024](https://arxiv.org/html/2609.20974#bib.bib13);[Liu et al\., 2026](https://arxiv.org/html/2609.20974#bib.bib14)\)\. Under AAR, the model reinforces the pattern of concentrating attention on the first token, through routing alone\. Routing thus acts not only as a selector of expert capabilities, but as an indirect regulator of attention’s processing downstream\.

### 5\.2External: AAR Reduces Long Diverging Generation

We compare answer lengths on GSM8K between matched pairs of AART\{\}^\{\\text\{T\}\}and baseline models in Table[4](https://arxiv.org/html/2609.20974#S5.T4)\. When both models answer incorrectly, AAR\-T produces substantially shorter answers \(mean−11\.8%\-11\.8\\%, P99−14\.9%\-14\.9\\%\)\. When both models are correct, lengths are indistinguishable\. The effect is concentrated in the wrong\-answer regime and consistent across the full output length distribution, suggesting that AAR reduces diverging generation length\([Li et al\., 2026](https://arxiv.org/html/2609.20974#bib.bib7)\): the model commits to an answer sooner when wrong, rather than generating long rambling reasoning chains\. Correct answers are mostly unaffected, indicating the model has not simply learned to be uniformly brief\. Answer lengths also become shorter for the answers that get corrected under AART\{\}^\{\\text\{T\}\}\.

ConditionMetricBaseAAR\-TRel\.Δ\\DeltaBoth CorrectMean68\.267\.6−0\.8%\-0\.8\\%Median54\.054\.00\.0%0\.0\\%P99 \(chars\)251\.0248\.0−1\.2%\-1\.2\\%Both WrongMean163\.7144\.4−11\.8%\-11\.8\\%Median78\.073\.0−6\.4%\-6\.4\\%P99 \(chars\)8513\.07242\.0−14\.9%\-14\.9\\%Ramble Rate5\.63%4\.29%−23\.9%\-23\.9\\%Wrong→\\rightarrowCorrectMean140\.489\.5−36\.25%\-36\.25\\%Median83\.080\.0−3\.6%\-3\.6\\%P99 \(chars\)7306\.01036\.0−85\.8%\-85\.8\\%

Table 4:Length analysis \(tokens\) on GSM8K averaged across seeds and context sizesMM\. Ramble Rate denotes responses exceeding 1,000 characters\. AART\{\}^\{\\text\{T\}\}leaves correct responses nearly unchanged while reducing the length of incorrect reasoning, especially in the tail \(P99\)\. The answer length reduction also persists for answers that get corrected by AART\{\}^\{\\text\{T\}\}\.

## 6AAR and Layer Depth

Applying AAR to every layer is not always beneficial\. Under our router\-only training regime, we find that doing so can substantially degrade performance on retrieval\-heavy tasks, even while improving mathematical performance\. AAR’s effectiveness therefore depends critically on where in the network it is applied\. Across models, our results suggest thattargeting middle\-to\-deep layers provides a robust starting point for improving mathematical reasoning, including when transferring AAR to previously untested architectures\. This depth sensitivity is itself informative\. Since AAR modifies only routing while the rest of the model remains frozen, varying its starting layer provides a controlled intervention on the routing–attention circuit at different depths\. We use this intervention as a depth\-wise probe and uncover a retrieval–reasoning tension: early layers are particularly sensitive for factual retrieval, whereas mathematical reasoning benefits are preserved when AAR is introduced deeper in the network\.

### 6\.1A Retrieval\-Reasoning Tension Across Depth

When AART\{\}^\{\\text\{T\}\}is applied to all layers, MMLU performance degrades by−2\.1\-2\.1pp on average\. This degradation is not uniform across subjects: Table[5](https://arxiv.org/html/2609.20974#S6.T5)shows that performance on reasoning\-oriented subjects \(Math, Computer Science\) is preserved or improved, while retrieval\-heavy subjects \(Biology, History, Philosophy\) suffer consistent deterioration\. This dissociation suggests that AAR’s reasoning\-amplifying behavior interferes with factual retrieval when applied in early layers\.

To localize this effect, we vary the starting layer from which AART\{\}^\{\\text\{T\}\}is applied onwards and evaluate per\-subject MMLU performance \(Figure[4](https://arxiv.org/html/2609.20974#S6.F4)\)\. For retrieval\-heavy subjects, performance is worst when AAR starts early and recovers as the starting layer increases\. For reasoning\-heavy subjects, improvements persist regardless of starting depth\. For subjects combining both, such as Economics or Physics, performance peaks when AAR starts around layers77–1111\. We identify this region as the main locus of the retrieval\-reasoning tension under AAR: early layers are critical for factual lookup and are sensitive to AAR\-driven changes; beyond this region, retrieval\-relevant structure has consolidated and AAR can be applied without disrupting it\.

![Refer to caption](https://arxiv.org/html/2609.20974v1/plots/mmlu_subjects_start_layers.png)Figure 4:AART\{\}^\{\\text\{T\}\}\(M=20M=20\) accuracy across MMLU subjects as a function of starting layer\. Most subjects’ accuracy curves follow a bell shape\.Per\-layer attention weights thus act as a fingerprint of each layer’s information processing: AAR reveals where this tension is concentrated, and that it is readable from the attention signal\. This is consistent with and extends[Song et al\. \(2026\)](https://arxiv.org/html/2609.20974#bib.bib4), who show that shallow layers dominate retrieval tasks while mid\-to\-deep layers are essential for reasoning\. Rather than probing representations post\-hoc, layer\-selective AAR probes the functional role of attention weights by measuring the downstream effect of routing changes directly\.

MMLU SubjectΔ\\DeltaMM=10Δ\\DeltaMM=20Δ\\DeltaMM=64Math\+0\.7\+0\.7\+0\.6\+0\.6\+0\.5\+0\.5Comp\. Sci\.\+0\.5\+0\.5\+0\.4\+0\.4−0\.1\-0\.1Biology−4\.5\-4\.5−3\.4\-3\.4−3\.8\-3\.8History−2\.5\-2\.5−2\.4\-2\.4−2\.2\-2\.2Health−1\.4\-1\.4−1\.8\-1\.8−1\.9\-1\.9Psychology−1\.4\-1\.4−1\.7\-1\.7−1\.2\-1\.2Geography−1\.0\-1\.0\+0\.2\+0\.2\+0\.0\+0\.0Philosophy−3\.2\-3\.2−3\.1\-3\.1−3\.4\-3\.4

Table 5:MMLU subcategoryΔ\\Delta\(absolute % accuracy difference from baseline, 5 seeds\) when AART\{\}^\{\\text\{T\}\}is applied to all layers of OLMoE\. Reasoning\-oriented subjects \(top\) are improved; degradation is concentrated in retrieval\-heavy subjects \(bottom\)\.

## 7Discussion

#### Self\-regulation through routing\.

By providing the router with a fingerprint of the model’s contextual processing, as encoded in the attention weights, we enable a form of self\-regulation: the routers can settle into configurations under which downstream attention patterns shift in ways that co\-occur with improved task performance, making this coupling directly available during optimization\. The effect is reflected internally in the reshaping of attention, and externally in the reduced length of wrong answers\. It is not an explicitly supervised signal; it emerges from training with standard next\-token prediction on the routing parameters alone\.

#### From routing to generation\.

Our results trace a cascade: routing at layerllreshapes the representations passed to layerl\+1l\+1, which alters that layer’s attention weights \(evidenced by the one\-layer\-delayed sink amplification\), and is accompanied by shorter generation on incorrect responses\. That a change confined entirely to routing parameters can produce measurable effects at the level of generation behavior suggests that routing, attention, and generation are tightly coupled, and that routing is a consequential component of how the model processes information, beyond its role as a dispatching mechanism\.

## 8Conclusion

Standard MoE routers select experts based on each token’s hidden state alone\. We illustrate that disentangling the contextual information by augmenting the router with a sliding window of attention weights can meaningfully reshape model behavior through routing alone, improving performance on mathematics\. These routing changes propagate through the network: they amplify attention sinks one layer downstream, and at generation time, they reduce output length on incorrect answers while leaving correct ones unchanged\. The retrieval\-reasoning tension we observe demonstrates that the attention\-routing circuit carries qualitatively different information at different depths, with layer\-selective AAR serving as a natural probe into this structure\. Taken together, our results reinforce the view that routing is not merely a dispatching mechanism, but a consequential degree of freedom: shifting routing from a local, per\-token decision to one informed by the model’s own attention patterns yields better mathematical reasoning performance\.

## Limitations

AAR’s routing mechanism requires explicit attention weight materialization, precluding the use of FlashAttention\([Dao et al\., 2022](https://arxiv.org/html/2609.20974#bib.bib25)\)at AAR\-active layers\. The eager attention fallback introduces overhead linear in sequence length and the number of AAR\-active layers\.

Our experiments focus on post\-SFT adaptation: we train the auxiliary routers jointly with already\-trained routing parameters, a setting that makes the optimal choice of AAR\-active layers dependent on the model’s prior training and may affect cross\-model generalization\. Empirically, the best layer ranges for OLMoE and Qwen both concentrate in middle\-to\-deep layers, but the specific subsets may differ\. We hypothesize that joint training from scratch would mitigate these differences, and leave adaptation during full\-parameter pretraining and SFT to future work\.

## Acknowledgments

We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA \(Italy\) and the LEONARDO consortium through an EuroHPC Development Access call\.

## References

- Barberoet al\.\(2025\)F\. Barbero, A\. Arroyo, X\. Gu, C\. Perivolaropoulos, P\. Veličković, R\. Pascanu, and M\. M\. BronsteinWhy do LLMs attend to the first token?\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=tu4dFUsW5z)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.arXiv preprint\.Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px5.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint\.Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px5.p1.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. ReFlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=H4DqfPSibmx)Cited by:[Limitations](https://arxiv.org/html/2609.20974#Sx1.p1.1)\.
- Dikkalaet al\.\(2023\)N\. Dikkala, N\. Ghosh, R\. Meka, R\. Panigrahy, N\. Vyas, and X\. WangOn the benefits of learning to route in mixture\-of\-experts models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9376–9396\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.583/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.583)Cited by:[§1](https://arxiv.org/html/2609.20974#S1.p1.1)\.
- Doet al\.\(2023\)T\. G\. Do, L\. Khiem, Q\. Pham, T\. Nguyen, T\. Doan, B\. Nguyen, C\. Liu, S\. Ramasamy, X\. Li, and S\. HoiHyperRouter: towards efficient training and inference of sparse mixture of experts\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5754–5765\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.351/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.351)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.JMLR\.Cited by:[§1](https://arxiv.org/html/2609.20974#S1.p1.1),[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px5.p1.1)\.
- Lambertet al\.\(2025\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, X\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. HajishirziTulu 3: pushing frontiers in open language model post\-training\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=i1uGbfHHpH)Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px3.p1.1)\.
- Lepikhinet al\.\(2021\)D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen\{GS\}hard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qrwe7XHTmYb)Cited by:[§1](https://arxiv.org/html/2609.20974#S1.p1.1)\.
- Liet al\.\(2026\)X\. Li, G\. Su, M\. Sachan, and J\. GeipingSample smart, not hard: correctness\-first decoding for better reasoning in LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=pNwCWIBHBC)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px6.p1.1),[§5\.2](https://arxiv.org/html/2609.20974#S5.SS2.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px5.p1.1)\.
- Liuet al\.\(2026\)X\. Liu, G\. Chen, and W\. WangSinkTrack: attention sink based context anchoring for large language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Gg1aPETCL6)Cited by:[§5\.1](https://arxiv.org/html/2609.20974#S5.SS1.p3.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px2.p1.1)\.
- Muennighoffet al\.\(2025\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, E\. P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, N\. A\. Smith, P\. W\. Koh, A\. Singh, and H\. HajishirziOLMoe: open mixture\-of\-experts language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xXTkbTBmqq)Cited by:[item 1](https://arxiv.org/html/2609.20974#S1.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px1.p1.1)\.
- Nguyenet al\.\(2026\)P\. M\. Nguyen, D\. Huu\-Tien, and N\. InoueImproving chain\-of\-thought for logical reasoning via attention\-aware intervention\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 2917–2941\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.152/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.152),ISBN 979\-8\-89176\-386\-9Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px5.p1.1)\.
- Qiuet al\.\(2025\)Z\. Qiu, Z\. Huang, S\. Cheng, Y\. Zhou, Z\. Wang, I\. Titov, and J\. FuLayerwise recurrent router for mixture\-of\-experts\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=eWNEqdH0vk)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px2.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.6\-35B\-A3B: agentic coding power, now open to all\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6-35b-a3b)Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px4.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. V\. Le, G\. E\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InICLR,Cited by:[§1](https://arxiv.org/html/2609.20974#S1.p1.1),[§1](https://arxiv.org/html/2609.20974#S1.p2.1),[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.20974#S3.SS1.p1.1)\.
- Songet al\.\(2026\)X\. Song, K\. Wang, P\. Li, L\. Yin, and S\. LiuDemystifying the roles of llm layers in retrieval, knowledge, and reasoning\.External Links:2510\.02091,[Link](https://arxiv.org/abs/2510.02091)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2609.20974#S6.SS1.p3.1)\.
- Suet al\.\(2025\)J\. Su, J\. Healey, P\. Nakov, and C\. CardieBetween underthinking and overthinking: an empirical study of reasoning length and correctness in llms\.External Links:2505\.00127,[Link](https://arxiv.org/abs/2505.00127)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px6.p1.1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. WeiChallenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13003–13051\.External Links:[Link](https://aclanthology.org/2023.findings-acl.824/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px5.p1.1)\.
- Team \(2024\)Q\. TeamQwen1\.5\-moe: matching 7b model performance with 1/3 activated parameters"\.External Links:[Link](https://qwenlm.github.io/blog/qwen-moe/)Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px4.p1.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2609.20974#S5.SS1.p2.1),[§5\.1](https://arxiv.org/html/2609.20974#S5.SS1.p3.1)\.
- Yanget al\.\(2026\)M\. Yang, J\. Gao, and J\. WuDecoupling knowledge and reasoning in llms: an exploration using cognitive dual\-system theory\.Proceedings of the AAAI Conference on Artificial Intelligence40\(40\),pp\. 34268–34276\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40723),[Document](https://dx.doi.org/10.1609/aaai.v40i40.40723)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px4.p1.1)\.
- Yanget al\.\(2025\)Y\. Yang, H\. Sun, J\. Wang, Q\. Qi, Z\. Zhuang, H\. Wang, P\. Ren, J\. Wang, and J\. LiaoUnveiling internal reasoning modes in LLMs: a deep dive into latent reasoning vs\. factual shortcuts with attribute rate ratio\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 2186–2206\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.111/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.111),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px4.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint\.Cited by:[§4\.1](https://arxiv.org/html/2609.20974#S4.SS1.SSS0.Px5.p1.1)\.
- Zhouet al\.\(2022\)Y\. Zhou, T\. Lei, H\. Liu, N\. Du, Y\. Huang, V\. Y\. Zhao, A\. M\. Dai, Z\. Chen, Q\. V\. Le, and J\. LaudonMixture\-of\-experts with expert choice routing\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=jdJo1HIVinI)Cited by:[§2](https://arxiv.org/html/2609.20974#S2.SS0.SSS0.Px1.p1.1)\.