Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs

# Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs
Source: [https://arxiv.org/html/2606.10722](https://arxiv.org/html/2606.10722)
Ruixuan Huang1Jinyuan Shi2Hantao Huang3,\*Yifan Huang3Ziyi Guan3Hao Zeng2Ian En\-Hsu Yen2,\*Minghui Yu3,\*

###### Abstract

We study dense\-to\-sparse continual training as a way to construct channel\-sparse large language models from dense checkpoints\. Starting from a Qwen2\.5\-8B dense backbone, we continue training at 32K context and introduce a predictor\-gated sparse SwiGLU FFN in the 32K stage\. For each token and layer, we use a low\-rank predictor to produce FFN\-channel routing logits\. We then apply a bank\-wise top\-kkrule to retain 16 channels in every 64\-channel bank, yielding 4x sparsity in the FFN intermediate activation\. Unlike post\-hoc sparse inference methods, the routing module is placed on the main language modeling path and optimized during continual training, enabling the dense model to be upcycled into a hardware\-oriented sparse model\. We report the architecture, training recipe, benchmark performance, and training lessons\. We also identify a layer\-local long\-context failure mode on RULER\-CWE and propose a single\-layer repair algorithm that substantially improves the affected length range\.

\{NoHyper\}††footnotetext:1HKUST\.2Moffett AI\.3Independent Contributor\.\*Corresponding authors\.

## 1Introduction

The cost of large language models \(LLMs\) is increasingly dominated by the parameter size and the amount of computation for every token\. This is especially apparent in decoder\-only Transformers\[[26](https://arxiv.org/html/2606.10722#bib.bib13)\], where each token passes through a dense feed\-forward network \(FFN\) in every layer\. In modern SwiGLU\-based models\[[24](https://arxiv.org/html/2606.10722#bib.bib14)\], the FFN expands the hidden state to a much larger intermediate dimension, applies element\-wise gating, and projects the result back to the model dimension\. During long\-context training and long\-prompt prefill, this computation is applied over large token blocks, even though only a subset of intermediate channels are important for any particular token\[[20](https://arxiv.org/html/2606.10722#bib.bib2)\]\. Reducing this token\-wise FFN cost is therefore a central requirement for making long\-context LLMs more efficient\.

Conditional computation offers a natural route to this goal\. Mixture\-of\-Experts \(MoE\) models reduce computation by routing each token to a small number of expert FFNs, but they introduce separate expert parameter pools, expert\-level routing, load\-balancing concerns, and often all\-to\-all communication\[[17](https://arxiv.org/html/2606.10722#bib.bib15),[7](https://arxiv.org/html/2606.10722#bib.bib16)\]\. A finer\-grained alternative is to view the intermediate channels of a dense FFN as neuron\-level computational units and activate only a subset of them for each token\. It preserves the original dense FFN parameterization while exposing conditional computation inside each layer and can turn a dense FFN into a sparse sub\-FFN whose active components change with the token\.

However, not all channel sparsity leads to usable acceleration\. Contextual sparsity methods demonstrate that LLM computation is strongly input\-dependent\[[20](https://arxiv.org/html/2606.10722#bib.bib2),[1](https://arxiv.org/html/2606.10722#bib.bib3),[35](https://arxiv.org/html/2606.10722#bib.bib4)\]\. Channel\-level methods\[[30](https://arxiv.org/html/2606.10722#bib.bib6)\]show that many FFN channels can be suppressed on a per\-token basis\. However, if the sparse mask is derived from native FFN activations, the routing signal may arrive too late to skip the dominant gate and up projections\. For channel sparsity to become a hardware\-facing computation path, the sparse mask must be available before the main FFN channel computation is materialized\.

This paper studies the dense\-to\-sparse continual\-training process, seeking to convert a dense LLM into a channel\-wise sparse model\. We introduce a lightweight low\-rank predictor that maps each token hidden state to logits over FFN intermediate channels\. The predictor produces the routing signal before the FFN computations\. We then impose a fixed bank\-wise top\-kkrule, which keeps 16 active channels in every 64\-channel bank, giving a 4×\\timesreduction in active FFN intermediate width\. The resulting sparse path is trained with hard sparse execution in the forward pass and a soft surrogate gradient through the top\-kkboundary\. We instantiate this idea in a Qwen2\.5\-8B model backbone\[[31](https://arxiv.org/html/2606.10722#bib.bib12)\]\. The model is first trained from scratch at 8K context length and then continued at 32K context length\. Sparsity is introduced late in the 32K stage, reflecting a practical upcycling scenario\[[16](https://arxiv.org/html/2606.10722#bib.bib9)\]\.

Our experiments compare the resulting sparse model with \(1\) a dense companion run and \(2\) a naive 4×\\timessparse baseline based on dense activation magnitude \(Dense\-4x\)\. Our sparse model is consistently closer to the dense baseline than the naive Dense\-4x baseline across a broad set of evaluation benchmarks, indicating that channel sparsity can be trained as part of the language\-modeling path rather than applied only as a post\-training compression heuristic\.

This paper reveals that sparse upcycling is not lossless\. We highlight two instructive failure modes during our training process\. First, channel\-level routing does not inherit MoE\-style balancing in a straightforward way\. Second, long\-context evaluation exposes failures that are nearly invisible in broad short\-context benchmarks\. On RULER\-CWE\[[12](https://arxiv.org/html/2606.10722#bib.bib11)\], the sparse model remains healthy at shorter lengths but develops a sharp 12K–16K cliff\. We propose a dense fallback intervention that localizes much of this failure to a single layer and the final question\-answer suffix\.

#### Contributions\.

In summary, this paper makes the following contributions:

- •Predictor\-gated channel sparsity pattern\.We introduce a predictor\-gated sparse pattern in which a low\-rank token\-wise predictor produces channel routing logits before FFN channel computation, and a hard bank\-wise top\-kkrule retains 16 channels in every 64\-channel bank\. It can skip inactive gate, up, and down channels when supported by specialized kernels or sparse hardware\.
- •Dense\-to\-sparse continual training recipe\.We present a dense\-to\-sparse continual training recipe that converts a Qwen2\.5\-8B dense model trained at 8K and continued at 32K into a predictor\-gated 4×\\timessparse FFN model, with limited performance degradation\.
- •Training lessons\.We report two training lessons\. First, MoE\-style balance bias does not transfer cleanly to bank\-wise channel routing\. Second, RULER\-CWE reveals a layer\-local long\-context failure that can be partially repaired through a single\-layer intervention\.

## 2Related work

#### Contextual sparsity\.

Prior work shows that LLM computation is input\-dependent and can often be sparsified\. Dejavu predicts token\-dependent sparse sets of attention heads and MLP neurons for efficient inference\[[20](https://arxiv.org/html/2606.10722#bib.bib2)\], while ShadowLLM improves predictor\-based contextual sparsity with stronger importance criteria and deployment\-oriented predictors\[[1](https://arxiv.org/html/2606.10722#bib.bib3)\]\. Sirius highlights that training\-free contextual sparsity can be fragile on complex generation tasks and introduces correction mechanisms to recover quality\[[35](https://arxiv.org/html/2606.10722#bib.bib4)\]\. This paper is more training\-centric, where the sparse gate is inserted into the main SwiGLU FFN computation and optimized during continual training\.

#### Channel\-sparse FFNs\.

Mixture\-of\-Channels \(MoC\) introduces persistent channel sparsity inside SwiGLU FFNs by selecting the top\-kkchannels from the native gate projection for each token\[[30](https://arxiv.org/html/2606.10722#bib.bib6)\]\. However, the sparse decision is still derived from the native gate output, so the gate projection must be evaluated before the active set is known\. SparkTransformer proposes a new sparse architecture for both FFN and attention, using top\-kkactivation masks and low\-cost predictors formed by reallocating existing dimensions\[[32](https://arxiv.org/html/2606.10722#bib.bib7)\]\. This predictor is an internal consequence of Spark’s redesigned parameterization\. The resulting sparse design is not directly compatible with an existing SwiGLU architecture\.

#### MoE and dense\-to\-sparse conversion\.

Expert\-level conditional computation routes tokens to a small number of full FFN experts\. Related dense\-to\-sparse approaches include MoEfication, which partitions FFN parameters into functional experts and builds expert routers\[[33](https://arxiv.org/html/2606.10722#bib.bib8)\], and Aran et al\., which initializes sparse MoE models from dense checkpoints\[[16](https://arxiv.org/html/2606.10722#bib.bib9)\]\. The channel\-level view we use here is MoE\-like in spirit because FFN intermediate channels behave as fine\-grained conditional units, but it does not introduce independent expert parameter pools or expert\-level all\-to\-all dispatch\. It preserves the dense SwiGLU parameterization and performs token\-wise channel selection\.

![Refer to caption](https://arxiv.org/html/2606.10722v1/x1.png)Figure 1:Our proposed predictor\-gated sparse SwiGLU FFN\. A low\-rank predictor maps the token hidden state to FFN\-channel routing logits\.

## 3Methodology

This section describes the predictor\-gated sparse FFN that we use in the dense\-to\-sparse continual\-training process\. We start from the dense SwiGLU computation, then introduce the proposed low\-rank predictor, the bank\-wise top\-kkrouting, and technical details that keep the sparse path trainable\.

### 3\.1Dense SwiGLU FFN

For layerlland token positiontt, let the FFN input bexl,t∈ℝdx\_\{l,t\}\\in\\mathbb\{R\}^\{d\}, whereddis the hidden dimension\. We usedid\_\{i\}for the dense FFN intermediate dimension\. A standard SwiGLU FFN computes

ul,t\\displaystyle u\_\{l,t\}=Wlg​xl,t,\\displaystyle=W^\{g\}\_\{l\}x\_\{l,t\},vl,t\\displaystyle v\_\{l,t\}=Wlu​xl,t,\\displaystyle=W^\{u\}\_\{l\}x\_\{l,t\},\(1\)ml,tdense\\displaystyle m^\{\\mathrm\{dense\}\}\_\{l,t\}=SiLU​\(ul,t\)⊙vl,t=ul,t⊙σ​\(ul,t\)⊙vl,t,\\displaystyle=\\mathrm\{SiLU\}\(u\_\{l,t\}\)\\odot v\_\{l,t\}=u\_\{l,t\}\\odot\\sigma\(u\_\{l,t\}\)\\odot v\_\{l,t\},\(2\)yl,tdense\\displaystyle y^\{\\mathrm\{dense\}\}\_\{l,t\}=Wlo​ml,tdense\.\\displaystyle=W^\{o\}\_\{l\}m^\{\\mathrm\{dense\}\}\_\{l,t\}\.\(3\)whereul,t,vl,t,ml,tdense∈ℝdiu\_\{l,t\},v\_\{l,t\},m^\{\\mathrm\{dense\}\}\_\{l,t\}\\in\\mathbb\{R\}^\{d\_\{i\}\}\. In our 8B\-scale training configuration,d=4096d=4096anddi=14336d\_\{i\}=14336\.

### 3\.2Predictor\-gated Channel Sparsity

We introduce a per\-layer predictorPlP\_\{l\}that maps the token hidden state to logits over FFN intermediate channels:

pl,t=Pl​\(xl,t\)∈ℝdi\.\\displaystyle p\_\{l,t\}=P\_\{l\}\(x\_\{l,t\}\)\\in\\mathbb\{R\}^\{d\_\{i\}\}\.\(4\)The sparse gategl,tg\_\{l,t\}is built from these logits and replaces the dense SwiGLU gateσ​\(ul,t\)\\sigma\(u\_\{l,t\}\):

ml,t\\displaystyle m\_\{l,t\}=ul,t⊙gl,t⊙vl,t,\\displaystyle=u\_\{l,t\}\\odot g\_\{l,t\}\\odot v\_\{l,t\},\(5\)yl,t\\displaystyle y\_\{l,t\}=Wlo​ml,t\.\\displaystyle=W^\{o\}\_\{l\}m\_\{l,t\}\.\(6\)Thus, the predictor directly modulates the FFN intermediate activation and receives gradients from the language modeling loss\.

The predictor is a small two\-layer low\-rank module

Pl​\(x\)=Wlp,2​Wlp,1​x,\\displaystyle P\_\{l\}\(x\)=W^\{p,2\}\_\{l\}W^\{p,1\}\_\{l\}x,\(7\)with dimensions

Wlp,1\\displaystyle W^\{p,1\}\_\{l\}:ℝd→ℝ448,\\displaystyle:\\mathbb\{R\}^\{d\}\\rightarrow\\mathbb\{R\}^\{448\},Wlp,2\\displaystyle W^\{p,2\}\_\{l\}:ℝ448→ℝdi\.\\displaystyle:\\mathbb\{R\}^\{448\}\\rightarrow\\mathbb\{R\}^\{d\_\{i\}\}\.\(8\)The hidden size 448 isdi/32d\_\{i\}/32\. The predictor output also includes a bank\-level affine calibration with learnable scale and bias parameters shared within each 64\-channel bank\.

### 3\.3Bank\-wise Hard Top\-kkwith Soft Surrogate Gradients

The FFN intermediate dimension is partitioned into fixed banks of sizeCbank=64C\_\{\\mathrm\{bank\}\}=64\. For each token, layer, and bank, we keepk=16k=16channels:

pl,t\\displaystyle p\_\{l,t\}=\[pl,t\(1\),…,pl,t\(G\)\],pl,t\(b\)∈ℝCbank,\\displaystyle=\[p^\{\(1\)\}\_\{l,t\},\\ldots,p^\{\(G\)\}\_\{l,t\}\],\\quad p^\{\(b\)\}\_\{l,t\}\\in\\mathbb\{R\}^\{C\_\{\\mathrm\{bank\}\}\},\(9\)Sl,t\(b\)\\displaystyle S^\{\(b\)\}\_\{l,t\}=topk⁡\(pl,t\(b\),k=16\)\.\\displaystyle=\\operatorname\{topk\}\(p^\{\(b\)\}\_\{l,t\},k=16\)\.\(10\)The gate is

gl,t,i=\{σ​\(pl,t,i\),i∈Sl,t\(b\),0,otherwise\.\\displaystyle g\_\{l,t,i\}=\\begin\{cases\}\\sigma\(p\_\{l,t,i\}\),&i\\in S^\{\(b\)\}\_\{l,t\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(11\)Withdi=14336d\_\{i\}=14336, this givesG=di/Cbank=224G=d\_\{i\}/C\_\{\\mathrm\{bank\}\}=224banks and keepsds=G​k=3584d\_\{s\}=Gk=3584channels per token per layer\. Here,dsd\_\{s\}is the sparse intermediate dimension shown in Figure[1](https://arxiv.org/html/2606.10722#S2.F1)\.

Adding a predictor and a discrete top\-kkoperator changes the FFN routing rule, but the end\-to\-end training path remains a standard language\-modeling computation graph with a custom gradient for the mask\. In the forward pass, each bank first normalizes predictor logits, selects the top\-16 channels, and constructs a binary maskhl,th\_\{l,t\}\. The FFN intermediate activation is then

ml,t=ul,t⊙vl,t⊙σ​\(pl,t\)⊙hl,t,\\displaystyle m\_\{l,t\}=u\_\{l,t\}\\odot v\_\{l,t\}\\odot\\sigma\(p\_\{l,t\}\)\\odot h\_\{l,t\},\(12\)so inactive channels are exactly zero in the model path seen by the loss\.

In the backward pass, gradients throughWloW^\{o\}\_\{l\},WlgW^\{g\}\_\{l\},WluW^\{u\}\_\{l\}, and the selected intermediate activations follow ordinary automatic differentiation under the chosen mask\. The non\-differentiable top\-kkselection is handled by a soft top\-kksurrogate: around each bank’s selection threshold, the hard indicator is assigned a sigmoid\-like local gradient\. Thus the language\-modeling loss can still update the predictor logits and predictor parameters near the routing boundary\.

### 3\.4Hardware Implications

Our training and validation are performed on GPUs as a functional simulation of bank\-wise sparse FFN semantics\. The model is trained with a hard sparse gate in the forward path, but the GPU implementation still materializes the dense gate and up projections,Wg​xW^\{g\}xandWu​xW^\{u\}x, before applying the top\-kkmask\. As a result, the practically accelerated part on GPUs is mainly the downstream projectionWo​mW^\{o\}m, where the zeroed intermediate activation can be gathered and multiplied against only the selected rows/columns\. Our experiments therefore validate sparse semantics and model quality rather than demonstrating measured 4x end\-to\-end GPU wall\-clock speedup\.

The algorithmic principle is stronger than what our GPU implementation exposes\. For any inactive channeliiwithhl,t,i=0h\_\{l,t,i\}=0, the term

ul,t,i​vl,t,i​σ​\(pl,t,i\)​hl,t,i\\displaystyle u\_\{l,t,i\}\\,v\_\{l,t,i\}\\,\\sigma\(p\_\{l,t,i\}\)\\,h\_\{l,t,i\}\(13\)contributes exactly zero to the FFN output\. Dedicated sparse hardware can therefore place the predictor\-generated mask before the FFN projections and avoid computing inactive channels in all three FFN stages: the gate projectionWgW^\{g\}, the up projectionWuW^\{u\}, and the down projectionWoW^\{o\}\. Under the 64:16 bank rule, the ideal sparse execution shape is reduced from the dense intermediate widthdid\_\{i\}to the selected sparse widthds=di/4d\_\{s\}=d\_\{i\}/4for each token and layer, while preserving the same mathematical sparse FFN path\. The intended efficiency target is therefore dedicated sparse hardware or specialized sparse kernels\.

## 4Training recipe

This section reports the training\-side context needed to interpret the sparse model and its comparisons\. We describe the Qwen2\.5\-style model backbone and the training data curriculum\.

### 4\.1Model Backbone

The dense backbone follows the Qwen2\.5 architectural family rather than a post\-hoc modification of an open\-source checkpoint\[[31](https://arxiv.org/html/2606.10722#bib.bib12)\]\. The 8B\-scale configuration uses 32 layers, hidden size 4096, FFN intermediate dimension 14336, 32 query heads, and 4 shared key\-value heads\. The tokenizer vocabulary size is 155136\.

We also follow the Qwen2\.5\-style companion recipe\. We double the number of query heads in grouped\-query attention to increase the parameter count\. The sparse model adds a per\-layer low\-rank predictor that produces token\-wise FFN channel routing logits before bank\-wise top\-kkselection\. The predictor adds approximately 0\.26B trainable parameters\.

### 4\.2Pretraining Data

The pretraining corpus is a multilingual, multi\-domain mixture designed to transfer strong English capability while progressively increasing Chinese coverage\. Table[1](https://arxiv.org/html/2606.10722#S4.T1)summarizes the active 8K and 32K schedules used for the model\. The 8K dense stage uses an 8T\-token schedule\. The 32K continuation uses a separate long\-context mixture with 4T tokens\. During the 32K continuation, the RoPE base is increased to support the longer sequence length\.

Table 1:Data mixture and context\-length curriculum\. Content columns report expected sampling shares within each stage; token counts use the reported training budget\.​

### 4\.3Sparse Continual Training

The sparse branch is introduced late in Stage 2\. The dense model first runs the full 8K stage and then continues at 32K\. Predictor sparsity is introduced only after roughly 3T tokens\. In particular, this sparse branch uses weight decay 0\.06, whereas the non\-sparse reference recipe uses weight decay 0\.1\. In Figure[2](https://arxiv.org/html/2606.10722#S5.F2), the x\-axis is aligned to the 4T\-token 32K stage budget\.

We summarize the whole training process as follows:

1. 1\.Dense, 8K stage: train from scratch with the same Qwen2\.5\-8B architecture\.
2. 2\.Dense, 32K stage: resume from the dense 8K checkpoint, increase the context length to 32K, and continue dense training for around 3T tokens\.
3. 3\.Predictor warm\-up: add the low\-rank predictor and train it as an auxiliary branch that imitates the native gate signal; this branch does not affect the main loss and lasts only a small number of tokens\.
4. 4\.Predictor\-gated dense FFN: route the main FFN gate through the predictor and sigmoid so that the predictor affects the language\-modeling loss, but do not yet apply hard top\-kkchannel selection\.
5. 5\.Predictor\-gated top\-kksparse FFN: add bank\-wise top\-kkselection to impose 4x channel sparsity\.

Appendix[A](https://arxiv.org/html/2606.10722#A1)reports a model diagnostic ablation that motivates this predictor\-before\-top\-kkordering\. This ablation runs on a 200M model and is separate from our main evaluation\.

## 5Evaluation

We evaluate the sparse model against two companion baselines:

- •Dense companion: a fully dense run with comparable data and training budget\.
- •Dense\-4x: a naive 4x sparse baseline based on activation magnitude, applied post\-hoc to the dense companion’s training results\.
- •Ours: our predictor\-gated channel\-wise 4x sparsity\.

Table[2](https://arxiv.org/html/2606.10722#S5.T2)compares the benchmarking performance of the three models\. The Dense\-4x and sparse columns append their deltas relative to the dense companion; green indicates a positive delta and red indicates a negative delta\. We summarize the evaluated benchmarks as six categories:

- •Factual QA: SimpleQA\[[28](https://arxiv.org/html/2606.10722#bib.bib17)\]and Chinese SimpleQA\[[9](https://arxiv.org/html/2606.10722#bib.bib18)\]\.
- •Code: MBPP\[[3](https://arxiv.org/html/2606.10722#bib.bib19)\], MBPP\+\[[19](https://arxiv.org/html/2606.10722#bib.bib20)\], MultiPL\-E\[[4](https://arxiv.org/html/2606.10722#bib.bib22)\], MBXP\[[2](https://arxiv.org/html/2606.10722#bib.bib23)\], and LiveCodeBench\[[14](https://arxiv.org/html/2606.10722#bib.bib24)\]\.
- •Reasoning: KOR\-Bench\[[22](https://arxiv.org/html/2606.10722#bib.bib25)\], ZebraLogic\[[18](https://arxiv.org/html/2606.10722#bib.bib26)\], ProcBench\[[8](https://arxiv.org/html/2606.10722#bib.bib27)\], ARC\-AGI\[[5](https://arxiv.org/html/2606.10722#bib.bib28)\], DROP\[[6](https://arxiv.org/html/2606.10722#bib.bib29)\], and BBH\[[25](https://arxiv.org/html/2606.10722#bib.bib30)\]\.
- •Math: AIME\[[23](https://arxiv.org/html/2606.10722#bib.bib31)\]and MATH\[[11](https://arxiv.org/html/2606.10722#bib.bib32)\]\.
- •Composite: LiveBench\[[29](https://arxiv.org/html/2606.10722#bib.bib33)\], SuperGPQA\[[21](https://arxiv.org/html/2606.10722#bib.bib34)\], AGIEval\[[34](https://arxiv.org/html/2606.10722#bib.bib35)\], MMLU\-Pro\[[27](https://arxiv.org/html/2606.10722#bib.bib36)\], C\-Eval\[[13](https://arxiv.org/html/2606.10722#bib.bib37)\], and MMLU\[[10](https://arxiv.org/html/2606.10722#bib.bib38)\]\.
- •Long\-Context: RULER\[[12](https://arxiv.org/html/2606.10722#bib.bib11)\]\.

Table 2:Benchmark results\. Higher is better\. Deltas are score differences versus the dense companion\.BenchmarkDenseDense\-4xSparseFactual QASimpleQA5\.03\.9\-1\.13\.8\-1\.2Chinese SimpleQA29\.722\.4\-7\.327\.5\-2\.2CodeMBPP72\.666\.2\-6\.472\.6\+0\.0MBPP\+67\.262\.2\-5\.065\.1\-2\.1MultiPL\-E53\.446\.8\-6\.654\.1\+0\.7MBXP67\.262\.2\-5\.066\.8\-0\.4LiveCodeBench34\.423\.9\-10\.526\.9\-7\.5ReasoningKOR\-Bench43\.836\.7\-7\.043\.0\-0\.7ZebraLogic12\.510\.8\-1\.713\.3\+0\.8ProcBench10\.111\.1\+1\.011\.3\+1\.2ARC\-AGI8\.96\.1\-2\.86\.4\-2\.5DROP69\.263\.8\-5\.567\.9\-1\.4BBH71\.970\.8\-1\.177\.8\+6\.0
BenchmarkDenseDense\-4xSparseMathAIME 20252\.02\.3\+0\.30\.7\-1\.3AIME 20246\.33\.0\-3\.31\.3\-5\.0MATH57\.751\.9\-5\.757\.1\-0\.6CompositeLiveBench30\.526\.5\-4\.028\.2\-2\.3SuperGPQA20\.023\.2\+3\.224\.9\+4\.9AGIEval64\.658\.9\-5\.764\.4\-0\.3MMLU\-Pro50\.042\.4\-7\.748\.8\-1\.3C\-Eval83\.480\.6\-2\.880\.8\-2\.6MMLU77\.075\.4\-1\.777\.1\+0\.1Long ContextRULER 4K93\.694\.3\+0\.793\.8\+0\.2RULER 8K92\.692\.1\-0\.592\.5\-0\.1RULER 16K89\.388\.6\-0\.688\.6\-0\.7RULER 32K79\.273\.5\-5\.880\.4\+1\.1

Across the results, our predictor\-gated sparsity is usually closer to the dense companion than Dense\-4x\. The sparse model matches or slightly exceeds dense on MBPP, MultiPL\-E and MMLU, etc\. It also substantially reduces the Dense\-4x degradation on Chinese SimpleQA, MBPP\+, MBXP and KOR\-Bench, etc\. The sparse model performs better than Dense\-4x at retaining dense quality\.

#### Loss Curves\.

Figure[2](https://arxiv.org/html/2606.10722#S5.F2)reports the training\-loss trajectory around the sparse transition in the 32K stage\. The dense model is first trained through the 8K stage and is then continued for a 4T\-token 32K stage; the figure zooms into the late 32K continuation where sparse adaptation is introduced\. The colored bands correspond to the sparse\-transition phases in the training recipe: predictor side\-branch warm\-up, predictor\-gated dense FFN, and predictor\-gated top\-kksparse FFN\.

![Refer to caption](https://arxiv.org/html/2606.10722v1/x2.png)Figure 2:Training loss around the sparse transition in the 32K stage\. Colored bands denote predictor side\-branch warm\-up, predictor\-gated dense FFN, and predictor\-gated top\-kksparse FFN\.

## 6Training Lessons and Failure Modes

### 6\.1Channel Balancing

MoE models often use expert\-balancing mechanisms to avoid routing collapse\. Since our FFN channels behave like fine\-grained experts, it is natural to ask whether an expert\-balance bias can be applied at the channel level\. We evaluate a channel\-level routing bias inspired by MoE expert balancing\. In our setting, this does not improve the final training outcome, so the main recipe does not adopt a channel\-level balancing strategy\.

#### Balance\-bias variants\.

For a tokenttand FFN channelcc, letrt,cr\_\{t,c\}be the predictor logit andst,c=σ​\(rt,c\)s\_\{t,c\}=\\sigma\(r\_\{t,c\}\)be the predictor score used by the hard top\-kkgate\. Channels are partitioned into banksℬj\\mathcal\{B\}\_\{j\}of size 64, and 4x sparsity keepsk=16k=16channels in each bank\. A balance\-bias vectorbbis not optimized by backpropagation; it is an online routing correction added only for selection:

𝒮t,j\\displaystyle\\mathcal\{S\}\_\{t,j\}=topkc∈ℬj⁡\(st,c\+bc,k\),\\displaystyle=\\operatorname\{topk\}\_\{c\\in\\mathcal\{B\}\_\{j\}\}\(s\_\{t,c\}\+b\_\{c\},k\),\(14\)gt,c\\displaystyle g\_\{t,c\}=st,c​1​\[c∈𝒮t,j\]\.\\displaystyle=s\_\{t,c\}\\,\\mathbf\{1\}\[c\\in\\mathcal\{S\}\_\{t,j\}\]\.\(15\)The FFN intermediate activation is then multiplied bygt,cg\_\{t,c\}as in the normal predictor\-sparse path\. After accumulating one training step and all\-reducing token counts across data\-parallel ranks, we compute the selected\-token count

nc=∑t𝟏​\[c∈𝒮t,j\],n¯=1C​∑c=1Cnc,\\displaystyle n\_\{c\}=\\sum\_\{t\}\\mathbf\{1\}\[c\\in\\mathcal\{S\}\_\{t,j\}\],\\qquad\\bar\{n\}=\\frac\{1\}\{C\}\\sum\_\{c=1\}^\{C\}n\_\{c\},\(16\)whereCCis the number of local FFN channels in the tensor\-parallel shard\. We evaluated two update rules\. Average balance updates every channel toward the mean usage,

bc←bc\+α​sign⁡\(n¯−nc\)\.\\displaystyle b\_\{c\}\\leftarrow b\_\{c\}\+\\alpha\\,\\operatorname\{sign\}\(\\bar\{n\}\-n\_\{c\}\)\.\(17\)The 0\.05\-threshold balance updates only severely under\-used channels:

bc←bc\+α​1​\[nc<0\.05​n¯\]​sign⁡\(n¯−nc\)\.\\displaystyle b\_\{c\}\\leftarrow b\_\{c\}\+\\alpha\\,\\mathbf\{1\}\[n\_\{c\}<0\.05\\bar\{n\}\]\\,\\operatorname\{sign\}\(\\bar\{n\}\-n\_\{c\}\)\.\(18\)The no\-balance baseline fixesbc=0b\_\{c\}=0and applies no online update\. In the ablation below,α\\alphais the same small step size across balance variants\.

Figure[3](https://arxiv.org/html/2606.10722#S6.F3)shows the training\-loss comparison that motivates this decision\. The ablation runs in the 32K continual\-training window, where the predictor\-sparse model is initialized from a dense checkpoint\. We compare no channel\-balance bias, average balance, and 0\.05\-threshold balance\.

In this ablation window, average balance is consistently worse, and the 0\.05\-threshold balance does not improve over the no\-balance branch\. Therefore, explicitly forcing channel balance is not beneficial\. This is a useful negative result\. Channel\-level routing differs from expert\-level routing: channels are much finer\-grained, grouped into fixed banks, and coupled through the same FFN matrices\. A balance algorithm that works for complete experts may over\-constrain or destabilize channel\-level selection\. We note that this observation is specific to continual training from a dense checkpoint; it does not rule out balance mechanisms during training from scratch, where the router and FFN channels can co\-adapt from initialization\.

![Refer to caption](https://arxiv.org/html/2606.10722v1/x3.png)Figure 3:Channel\-balance ablations during 32K continual training\.

### 6\.2Layer\-local Predictor Failure on RULER\-CWE

Most short\-context and general benchmarks appear close to the dense companion, but RULER\-CWE exposes a sharper long\-context failure mode\. Figure[4](https://arxiv.org/html/2606.10722#S6.F4)presents a fine\-grained length sweep with 512\-token increments, averaged over 50 independent runs with different seeds\. The sparse model is healthy up to 8K but develops an early cliff around 12K–16K, while the dense model remains much more stable at those lengths\. This failure is not a global collapse of the sparse model\. On short\-context reasoning, the sparse model remains close to dense: MMLU accuracy is 0\.7404 for sparse v\.s\. 0\.7439 for dense, and ARC\-Challenge is 0\.9036 for sparse v\.s\. 0\.9181 for dense\.

![Refer to caption](https://arxiv.org/html/2606.10722v1/x4.png)Figure 4:RULER\-CWE average recall@10 as context length increases\.To diagnose the failure, we use a dense fallback intervention that bypasses the sparse predictor at selected FFN layers and token regions\. For a target layer, letmsparsem^\{\\mathrm\{sparse\}\}be the sparse FFN intermediate activation andmdensem^\{\\mathrm\{dense\}\}be the dense FFN intermediate activation\. We compute a norm calibration factor

rl=𝔼​∥mlsparse∥22𝔼​∥mldense∥22,\\displaystyle r\_\{l\}=\\sqrt\{\\frac\{\\mathbb\{E\}\\lVert m^\{\\mathrm\{sparse\}\}\_\{l\}\\rVert\_\{2\}^\{2\}\}\{\\mathbb\{E\}\\lVert m^\{\\mathrm\{dense\}\}\_\{l\}\\rVert\_\{2\}^\{2\}\}\},\(19\)and replace the selected sparse intermediate activation withrl​mldenser\_\{l\}m^\{\\mathrm\{dense\}\}\_\{l\}\.

The main positive result is localized to the 7\-th layer\. In the sweep, layer\-7 repair improves the 8704–12288 range average from 0\.856 to 0\.947 and the 12800–16384 range average from 0\.583 to 0\.802\. This result shows that the RULER\-CWE cliff can be traced to a narrow layer\-local routing failure instead of a whole\-model collapse\. Appendix[B](https://arxiv.org/html/2606.10722#A2)gives the detailed phase and token\-region localization, as well as the observed trade\-offs with short\-context reasoning benchmarks\.

## 7Conclusion

We present a channel\-level dense\-to\-sparse continual\-training recipe for an 8B\-scale language model\. The central idea is to reinterpret FFN intermediate channels as fine\-grained experts and to use a lightweight predictor as a token\-wise router over those channels\. The recipe imposes 4x bank\-wise channel sparsity, trains the predictor on the main language\-modeling path, and converts a dense 8K\-to\-32K training trajectory into a hardware\-oriented sparse FFN model\. Current benchmark results show that the predictor\-gated sparse model is often much closer to the dense companion than a naive Dense\-4x baseline\. The training analysis also exposes two useful failure modes: channel\-level balance bias does not transfer from expert\-level MoE training, and RULER\-CWE can reveal layer\-local long\-context predictor failures that broad short\-context benchmarks do not expose\. The single\-layer repair analysis shows that part of this long\-context failure is localizable and partially recoverable\.

## References

- \[1\]\(2024\)ShadowLLM: predictor\-based contextual sparsity for large language models\.ArXiv preprintabs/2406\.16635\.External Links:[Link](https://arxiv.org/abs/2406.16635)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p3.1),[§2](https://arxiv.org/html/2606.10722#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]B\. Athiwaratkun, S\. K\. Gouda, Z\. Wang, X\. Li, Y\. Tian, M\. Tan, W\. U\. Ahmad, S\. Wang, Q\. Sun, M\. Shang, S\. K\. Gonugondla, H\. Ding, V\. Kumar, N\. Fulton, A\. Farahani, S\. Jain, R\. Giaquinto, H\. Qian, M\. K\. Ramanathan, and R\. Nallapati\(2023\)Multi\-lingual evaluation of code generation models\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/pdf?id=Bo7eeXm6An8)Cited by:[2nd item](https://arxiv.org/html/2606.10722#S5.I2.i2.p1.1)\.
- \[3\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. V\. Le, and C\. Sutton\(2021\)Program synthesis with large language models\.ArXiv preprintabs/2108\.07732\.External Links:[Link](https://arxiv.org/abs/2108.07732)Cited by:[2nd item](https://arxiv.org/html/2606.10722#S5.I2.i2.p1.1)\.
- \[4\]F\. Cassano, J\. Gouwar, D\. Nguyen, S\. Nguyen, L\. Phipps\-Costin, D\. Pinckney, M\. Yee, Y\. Zi, C\. J\. Anderson, M\. Q\. Feldman, A\. Guha, M\. Greenberg, and A\. Jangda\(2023\)MultiPL\-E: a scalable and extensible approach to benchmarking neural code generation\.IEEE Transactions on Software Engineering\.Cited by:[2nd item](https://arxiv.org/html/2606.10722#S5.I2.i2.p1.1)\.
- \[5\]F\. Chollet\(2019\)On the measure of intelligence\.ArXiv preprintabs/1911\.01547\.External Links:[Link](https://arxiv.org/abs/1911.01547)Cited by:[3rd item](https://arxiv.org/html/2606.10722#S5.I2.i3.p1.1)\.
- \[6\]D\. Dua, Y\. Wang, P\. Dasigi, G\. Stanovsky, S\. Singh, and M\. Gardner\(2019\)DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2368–2378\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1246),[Link](https://aclanthology.org/N19-1246)Cited by:[3rd item](https://arxiv.org/html/2606.10722#S5.I2.i3.p1.1)\.
- \[7\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p2.1)\.
- \[8\]I\. Fujisawa, S\. Nobe, H\. Seto, R\. Onda, Y\. Uchida, H\. Ikoma, P\. Chien, and R\. Kanai\(2024\)ProcBench: benchmark for multi\-step reasoning and following procedure\.ArXiv preprintabs/2410\.03117\.External Links:[Link](https://arxiv.org/abs/2410.03117)Cited by:[3rd item](https://arxiv.org/html/2606.10722#S5.I2.i3.p1.1)\.
- \[9\]Y\. He, S\. Li, J\. Liu, Y\. Tan, W\. Wang, H\. Huang, X\. Bu, H\. Guo, C\. Hu, B\. Zheng, Z\. Lin, X\. Liu, D\. Sun, S\. Lin, Z\. Zheng, X\. Zhu, W\. Su, and B\. Zheng\(2024\)Chinese SimpleQA: a chinese factuality evaluation for large language models\.ArXiv preprintabs/2411\.07140\.External Links:[Link](https://arxiv.org/abs/2411.07140)Cited by:[1st item](https://arxiv.org/html/2606.10722#S5.I2.i1.p1.1)\.
- \[10\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[5th item](https://arxiv.org/html/2606.10722#S5.I2.i5.p1.1)\.
- \[11\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[4th item](https://arxiv.org/html/2606.10722#S5.I2.i4.p1.1)\.
- \[12\]C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. Ginsburg\(2024\)RULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p6.1),[6th item](https://arxiv.org/html/2606.10722#S5.I2.i6.p1.1)\.
- \[13\]Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, J\. Lei, Y\. Fu, M\. Sun, and J\. He\(2023\)C\-eval: A multi\-level multi\-discipline chinese evaluation suite for foundation models\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/c6ec1844bec96d6d32ae95ae694e23d8-Abstract-Datasets%5C_and%5C_Benchmarks.html)Cited by:[5th item](https://arxiv.org/html/2606.10722#S5.I2.i5.p1.1)\.
- \[14\]N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica\(2024\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.ArXiv preprintabs/2403\.07974\.External Links:[Link](https://arxiv.org/abs/2403.07974)Cited by:[2nd item](https://arxiv.org/html/2606.10722#S5.I2.i2.p1.1)\.
- \[15\]G\. Kamradt\(2023\)LLMTest\_NeedleInAHaystack: doing simple retrieval from LLM models at various context lengths to measure accuracy\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)GitHub repositoryCited by:[Appendix B](https://arxiv.org/html/2606.10722#A2.p6.1)\.
- \[16\]A\. Komatsuzaki, J\. Puigcerver, J\. Lee\-Thorp, C\. R\. Ruiz, B\. Mustafa, J\. Ainslie, Y\. Tay, M\. Dehghani, and N\. Houlsby\(2023\)Sparse upcycling: training mixture\-of\-experts from dense checkpoints\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/pdf?id=T5nUQDrM4u)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p4.3),[§2](https://arxiv.org/html/2606.10722#S2.SS0.SSS0.Px3.p1.1)\.
- \[17\]D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen\(2021\)GShard: scaling giant models with conditional computation and automatic sharding\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=qrwe7XHTmYb)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p2.1)\.
- \[18\]B\. Y\. Lin, R\. Le Bras, K\. Richardson, A\. Sabharwal, R\. Poovendran, P\. Clark, and Y\. Choi\(2025\)ZebraLogic: on the scaling limits of LLMs for logical reasoning\.ArXiv preprintabs/2502\.01100\.External Links:[Link](https://arxiv.org/abs/2502.01100)Cited by:[3rd item](https://arxiv.org/html/2606.10722#S5.I2.i3.p1.1)\.
- \[19\]J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang\(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html)Cited by:[2nd item](https://arxiv.org/html/2606.10722#S5.I2.i2.p1.1)\.
- \[20\]Z\. Liu, J\. Wang, T\. Dao, T\. Zhou, B\. Yuan, Z\. Song, A\. Shrivastava, C\. Zhang, Y\. Tian, C\. Ré, and B\. Chen\(2023\)Deja vu: contextual sparsity for efficient llms at inference time\.InInternational Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 22137–22176\.External Links:[Link](https://proceedings.mlr.press/v202/liu23am.html)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p1.1),[§1](https://arxiv.org/html/2606.10722#S1.p3.1),[§2](https://arxiv.org/html/2606.10722#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]M\-A\-P Teamet al\.\(2025\)SuperGPQA: scaling LLM evaluation across 285 graduate disciplines\.ArXiv preprintabs/2502\.14739\.External Links:[Link](https://arxiv.org/abs/2502.14739)Cited by:[5th item](https://arxiv.org/html/2606.10722#S5.I2.i5.p1.1)\.
- \[22\]K\. Ma, X\. Du, Y\. Wang, H\. Zhang, Z\. Wen, X\. Qu, J\. Yang, J\. Liu, M\. Liu, X\. Yue, W\. Huang, and G\. Zhang\(2024\)KOR\-Bench: benchmarking language models on knowledge\-orthogonal reasoning tasks\.ArXiv preprintabs/2410\.06526\.External Links:[Link](https://arxiv.org/abs/2410.06526)Cited by:[3rd item](https://arxiv.org/html/2606.10722#S5.I2.i3.p1.1)\.
- \[23\]Mathematical Association of America\(2025\)American invitational mathematics examination \(AIME\)\.Note:[https://maa\.org/student\-programs/amc/](https://maa.org/student-programs/amc/)Cited by:[4th item](https://arxiv.org/html/2606.10722#S5.I2.i4.p1.1)\.
- \[24\]N\. Shazeer\(2020\)GLU variants improve transformer\.ArXiv preprintabs/2002\.05202\.External Links:[Link](https://arxiv.org/abs/2002.05202)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p1.1)\.
- \[25\]M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. Wei\(2023\)Challenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13003–13051\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824),[Link](https://aclanthology.org/2023.findings-acl.824)Cited by:[3rd item](https://arxiv.org/html/2606.10722#S5.I2.i3.p1.1)\.
- \[26\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\-9, 2017, Long Beach, CA, USA,I\. Guyon, U\. von Luxburg, S\. Bengio, H\. M\. Wallach, R\. Fergus, S\. V\. N\. Vishwanathan, and R\. Garnett \(Eds\.\),pp\. 5998–6008\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p1.1)\.
- \[27\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen\(2024\)MMLU\-pro: A more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets%5C_and%5C_Benchmarks%5C_Track.html)Cited by:[5th item](https://arxiv.org/html/2606.10722#S5.I2.i5.p1.1)\.
- \[28\]J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus\(2024\)Measuring short\-form factuality in large language models\.ArXiv preprintabs/2411\.04368\.External Links:[Link](https://arxiv.org/abs/2411.04368)Cited by:[1st item](https://arxiv.org/html/2606.10722#S5.I2.i1.p1.1)\.
- \[29\]C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Naidu, C\. Hegde, Y\. LeCun, T\. Goldstein, W\. Neiswanger, and M\. Goldblum\(2024\)LiveBench: a challenging, contamination\-free LLM benchmark\.ArXiv preprintabs/2406\.19314\.External Links:[Link](https://arxiv.org/abs/2406.19314)Cited by:[5th item](https://arxiv.org/html/2606.10722#S5.I2.i5.p1.1)\.
- \[30\]T\. Wu, Y\. He, B\. Wang, and K\. Yuan\(2025\)Mixture\-of\-channels: exploiting sparse ffns for efficient llms pre\-training and inference\.ArXiv preprintabs/2511\.09323\.External Links:[Link](https://arxiv.org/abs/2511.09323)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p3.1),[§2](https://arxiv.org/html/2606.10722#S2.SS0.SSS0.Px2.p1.2)\.
- \[31\]A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei,et al\.\(2024\)Qwen2\.5 technical report\.ArXiv preprintabs/2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p4.3),[§4\.1](https://arxiv.org/html/2606.10722#S4.SS1.p1.1)\.
- \[32\]C\. You, K\. Wu, Z\. Jia, L\. Chen, S\. Bhojanapalli, J\. Guo, U\. Evci, J\. Wassenberg, P\. Netrapalli, J\. J\. Willcock, S\. Subramanian, F\. Chern, A\. Andreev, S\. Pathak, F\. Yu, P\. Jain, D\. E\. Culler, H\. M\. Levy, and S\. Kumar\(2025\)Spark transformer: reactivating sparsity in ffn and attention\.ArXiv preprintabs/2506\.06644\.External Links:[Link](https://arxiv.org/abs/2506.06644)Cited by:[§2](https://arxiv.org/html/2606.10722#S2.SS0.SSS0.Px2.p1.2)\.
- \[33\]Z\. Zhang, Y\. Lin, Z\. Liu, P\. Li, M\. Sun, and J\. Zhou\(2022\)MoEfication: transformer feed\-forward layers are mixtures of experts\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 877–890\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.71),[Link](https://aclanthology.org/2022.findings-acl.71)Cited by:[§2](https://arxiv.org/html/2606.10722#S2.SS0.SSS0.Px3.p1.1)\.
- \[34\]W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan\(2024\)AGIEval: a human\-centric benchmark for evaluating foundation models\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2299–2314\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.149)Cited by:[5th item](https://arxiv.org/html/2606.10722#S5.I2.i5.p1.1)\.
- \[35\]Y\. Zhou, Z\. Chen, Z\. Xu, V\. Lin, and B\. Chen\(2024\)Sirius: contextual sparsity with correction for efficient llms\.ArXiv preprintabs/2409\.03856\.External Links:[Link](https://arxiv.org/abs/2409.03856)Cited by:[§1](https://arxiv.org/html/2606.10722#S1.p3.1),[§2](https://arxiv.org/html/2606.10722#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix

## Appendix APredictor Top\-k Ablation

This section records the diagnostic ablation that motivates the predictor\-before\-top\-kksparse\-routing order\. This experiment is run on a 200M\-parameter small diagnostic model\. It is not a final quality claim for the 8B sparse model\. The ablation compares several ways to impose sparse FFN routing after a common continual\-training checkpoint:

- •Direct top\-kk: Selects active channels directly from native FFN channel scores\.
- •Direct top\-kkwith compensation: Keeps the same selected channels but rescales the post\-mask activation\.
- •Predictor\-before\-top\-kk\(adopted recipe\): Places the low\-rank predictor before bank\-wise hard top\-kkselection\.

Figure[5](https://arxiv.org/html/2606.10722#A1.F5)shows the corresponding training loss comparison\. The result supports using a trainable predictor before the hard top\-kkoperator\. Direct top\-kkmasking is simpler, but it gives worse loss performance\.

![Refer to caption](https://arxiv.org/html/2606.10722v1/x5.png)Figure 5:Continual\-training ablation comparing predictor before top\-kksparse routing with direct top\-kkmasking variants\.
## Appendix BRULER\-CWE Repair Analysis

This section records the details behind the layer\-7 repair result in Section[6\.2](https://arxiv.org/html/2606.10722#S6.SS2)\. The repair replaces selected sparse FFN intermediate activations with norm\-calibrated dense activations at one layer and then measures which task behavior is recovered or damaged\.

Prompt Prefill Repair is Sufficient for the CWE Recovery\.We first separate the layer\-7 repair by execution phase\. In the all\-call condition, the calibrated dense fallback is applied whenever the target layer is evaluated\. In the prompt\-processing condition, it is applied only while consuming the input prompt\. In the decoding\-only condition, it is applied only after generation begins\. Table[3](https://arxiv.org/html/2606.10722#A2.T3)reports results averaged over 50 different random seeds, with context length increasing in 512\-token steps\.

Table 3:Phase localization for layer\-7 repair on RULER\-CWE\. We report the recall@10\.The main recovery therefore comes from how the model processes the prompt, not from changing the tokens produced during autoregressive decoding\.

Effective Token Region: Final Question\-and\-Answer Suffix\.We then restrict the same layer\-7 prompt\-processing repair to different final prompt regions\. Table[4](https://arxiv.org/html/2606.10722#A2.T4)shows that the strongest localized repair comes from patching the suffix where the prompt asks the question and begins the answer\. Repairing the final 64 tokens and repairing the question\-and\-answer suffix produce nearly identical CWE recovery, while excluding that suffix removes most of the gain\.

Table 4:Token\-region localization for layer\-7 prompt\-processing repair\.This points to a readout\-query failure\. The sparse model appears to process enough evidence that a localized suffix repair can recover the common\-word answer, but the predictor routing at the final question\-and\-answer suffix is not reliable for this long\-CWE prompt\.

Repair is not a General Long\-Context Fix\.The same intervention does not repair Needle\-In\-A\-Haystack \(NIAH\)\[[15](https://arxiv.org/html/2606.10722#bib.bib39)\], where the model must retrieve and reason over a needle embedded in long context\. Table[5](https://arxiv.org/html/2606.10722#A2.T5)shows that final\-suffix repairs do not close the dense\-sparse gap\.

Table 5:NIAH recall under the same layer\-7 repair interventions\.This negative result suggests that the CWE repair restores a small amount of low\-level readout functionality rather than a general long\-context understanding mechanism\. Tasks such as NIAH appear to require distributed retrieval and integration behavior that cannot be recovered by patching one layer and one final suffix region\.

Repair Trades off with MMLU\.The repair also trades off against short\-context reasoning\. Figure[6](https://arxiv.org/html/2606.10722#A2.F6)varies the number of repaired final prompt tokens\. As the repaired window grows, the average long\-CWE gain increases, while MMLU accuracy decreases relative to the sparse baseline\.

![Refer to caption](https://arxiv.org/html/2606.10722v1/x6.png)Figure 6:Layer\-7 final\-token repair shows a trade\-off between long\-CWE recovery and MMLU\.One plausible explanation is that the same layer\-7 suffix routing subspace that is fragile for long\-CWE readout is also useful for ordinary semantic selection in short benchmarks\. Replacing it with dense activations can recover the narrow CWE behavior while disrupting the sparse model’s learned predictor routing on MMLU\.