Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

# Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs
Source: [https://arxiv.org/html/2607.27591](https://arxiv.org/html/2607.27591)
Jinyi Liu1,2, Wei Chen1,2\\corresponding, Pengyu Chen1,2, Xinyi Yuan1,2, Minghe Bai3, Guoquan Wu1,2, Jun Wei1,2

###### Abstract

Feed\-forward networks \(FFNs\) dominate memory traffic and computation in large language model \(LLM\) inference, making them a primary target for activation sparsification\. However, existing training\-free methods suffer substantial model\-quality degradation at high sparsity due to limitations in their channel\-selection strategies\. We observe that the SwiGLU intermediate state provides a highly effective channel\-selection signal, but obtaining it requires costly dense computation\. To address this, we present*Prox*, a two\-stage training\-free framework for sparse SwiGLU FFNs\. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values\. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections\. Across ten LLMs from six model families, Prox outperforms training\-free baselines at all sparsity levels, achieves up to a1\.99×1\.99\\timesend\-to\-end decoding speedup at 70% FFN sparsity, and is compatible with quantization and sparse attention\.

## 1Introduction

Efficient inference is critical for deploying large language models \(LLMs\), especially in resource\-constrained and latency\-sensitive settings\. In small\-batch autoregressive decoding, parameter transfers from off\-chip high\-bandwidth memory \(HBM\) to on\-chip storage dominate latency\. Modern LLMs commonly adopt SwiGLU feed\-forward networks \(FFNs\), whose three projection matrices account for most model parameters, memory traffic, and computation \(Figure[1](https://arxiv.org/html/2607.27591#S1.F1)\), making them a high\-leverage target for acceleration\.

Existing approaches accelerate LLM inference through weight quantization\(Dettmerset al\.[2022](https://arxiv.org/html/2607.27591#bib.bib13); Frantaret al\.[2022](https://arxiv.org/html/2607.27591#bib.bib14); Linet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib2)\)and model pruning\(Maet al\.[2023](https://arxiv.org/html/2607.27591#bib.bib22); Frantar and Alistarh[2023](https://arxiv.org/html/2607.27591#bib.bib38); Ashkbooset al\.[2024](https://arxiv.org/html/2607.27591#bib.bib36); Menet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib35); Qiaoet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib16); Sunet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib40)\), which lower memory footprint or computational cost by altering model precision or structure\. Orthogonal to these static modifications, activation sparsity\(Liuet al\.[2023](https://arxiv.org/html/2607.27591#bib.bib9); Leeet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib4); Cheon and Kang[2025](https://arxiv.org/html/2607.27591#bib.bib7); Liuet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib41); Chenet al\.[2023](https://arxiv.org/html/2607.27591#bib.bib39); Raihan and Aamodt[2020](https://arxiv.org/html/2607.27591#bib.bib29); Kurtzet al\.[2020](https://arxiv.org/html/2607.27591#bib.bib18); Szatkowskiet al\.[2026](https://arxiv.org/html/2607.27591#bib.bib45)\)mitigates memory and compute bottlenecks by dynamically skipping low\-salience channels and associated MACs\. Prior works exploit input\-dependent activation sparsity via either training\-based predictors or training\-free heuristics\. However, these predictors must be trained separately for each model, while existing training\-free schemes often suffer substantial accuracy degradation as sparsity increases due to limitations in their channel\-selection strategies\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x1.png)Figure 1:FFNs account for the majority of parameters and multiply\-accumulate operations \(MACs\) under 4K context length in Qwen3 Transformer blocks across model scales\.Our analysis reveals two findings: \(1\) The SwiGLU intermediate state provides a highly effective channel\-selection signal but requires costly dense computation\. Our oracle experiment demonstrates that selecting via this exact state incurs less than a 3% relative perplexity increase at 70% sparsity across most models\. \(2\) Sparse execution requires only the channel mask, which depends only on the magnitude ranking of the state entries rather than their exact values\. Our experiment shows that a low\-cost input\-sparse proxy preserves this ranking and closely matches the oracle mask\.

Based on these two key findings, we propose*Prox*, a two\-stage training\-free method for sparse SwiGLU inference\. Stage 1 \(*lightweight proxy construction*\) combines magnitude\-based input sparsity with quantized proxy weights to estimate both SwiGLU branches and construct a shared intermediate\-channel mask\. Stage 2 \(*exact sparse computation*\) computes the selected channels with the original up and gate weights, and applies the same mask to the down projection\. Prox further allocates computation between the two stages under a target sparsity budget and uses custom CUDA and Triton\(Tilletet al\.[2019](https://arxiv.org/html/2607.27591#bib.bib43)\)kernels to reduce weight accesses and computation across all three projections\.

We evaluate Prox on ten representative LLMs from six model families\. The results show that Prox consistently outpaces state\-of\-the\-art training\-free baselines, particularly at 60\-70% FFN sparsity, yielding up to a1\.99×1\.99\\timesend\-to\-end decoding speedup with minimal accuracy loss\. Crucially, Prox is fully orthogonal to quantization and sparse attention, enabling synergistic efficiency gains\.

Our main contributions are summarized as follows:

- •Insight\. We identify the FFN intermediate state as an effective channel\-selection signal and show that a low\-cost proxy preserves the magnitude ranking needed to recover the mask induced by the exact state\. This motivates proxy\-based channel selection followed by exact computation of the retained channels\.
- •Prox Framework\. We propose Prox, a training\-free two\-stage framework that uses quantized proxies for channel selection and computes retained values exactly, sparsifying all three SwiGLU projections while preventing error accumulation\.
- •Comprehensive Evaluation\. Our extensive experiments demonstrate that Prox delivers superior accuracy\-efficiency trade\-offs across various models and sparsity levels, while maintaining full compatibility with quantization and sparse attention\.

## 2Background and Related Work

### 2\.1SwiGLU Feed\-Forward Networks

LLMs are typically built with stacked Transformer layers, each comprising an attention block and an FFN\. While early Transformer FFNs employed ReLU or GELU activation functions, Gated Linear Units \(GLUs\)\(Dauphinet al\.[2017](https://arxiv.org/html/2607.27591#bib.bib8); Shazeer[2020](https://arxiv.org/html/2607.27591#bib.bib34)\)have largely superseded them\. Today, SwiGLU\(Shazeer[2020](https://arxiv.org/html/2607.27591#bib.bib34)\)remains the most widely adopted GLU variant in modern LLMs, with representative models including Qwen3\(Team[2025b](https://arxiv.org/html/2607.27591#bib.bib28)\), Llama\-3\(AI@Meta[2024](https://arxiv.org/html/2607.27591#bib.bib20)\), and Mistral\(Jianget al\.[2023](https://arxiv.org/html/2607.27591#bib.bib17)\)\.

A SwiGLU FFN comprises three projection matrices: the up projectionWup∈𝐑dmodel×dffW\_\{\\mathrm\{up\}\}\\in\\mathbf\{R\}^\{d\_\{\\mathrm\{model\}\}\\times d\_\{\\mathrm\{ff\}\}\}, the gate projectionWgate∈𝐑dmodel×dffW\_\{\\mathrm\{gate\}\}\\in\\mathbf\{R\}^\{d\_\{\\mathrm\{model\}\}\\times d\_\{\\mathrm\{ff\}\}\}, and the down projectionWdown∈𝐑dff×dmodelW\_\{\\mathrm\{down\}\}\\in\\mathbf\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\_\{\\mathrm\{model\}\}\}\. Given an input representation𝐱∈𝐑dmodel\\mathbf\{x\}\\in\\mathbf\{R\}^\{d\_\{\\mathrm\{model\}\}\}, its forward computation is defined as

FFN​\(𝐱\)=\(𝐱​Wup⊙SiLU​\(𝐱​Wgate\)\)​Wdown\.\\mathrm\{FFN\}\(\\mathbf\{x\}\)=\\bigl\(\\mathbf\{x\}W\_\{\\mathrm\{up\}\}\\odot\\mathrm\{SiLU\}\(\\mathbf\{x\}W\_\{\\mathrm\{gate\}\}\)\\bigr\)W\_\{\\mathrm\{down\}\}\.For subsequent discussion, we define three variables:

𝐮=𝐱​Wup,𝐡=SiLU​\(𝐱​Wgate\),𝐬=𝐮⊙𝐡,\\mathbf\{u\}=\\mathbf\{x\}W\_\{\\mathrm\{up\}\},\\quad\\mathbf\{h\}=\\mathrm\{SiLU\}\(\\mathbf\{x\}W\_\{\\mathrm\{gate\}\}\),\\quad\\mathbf\{s\}=\\mathbf\{u\}\\odot\\mathbf\{h\},denoting the up\-branch activation, the gated nonlinear activation, and the intermediate SwiGLU state, respectively\.

### 2\.2Activation Sparsity

Activation sparsity mitigates both HBM\-to\-register memory bandwidth overhead and redundant computations on low\-salience channels\. For a linear projection𝐲=𝐱​W\\mathbf\{y\}=\\mathbf\{x\}Wwith𝐱∈𝐑din\\mathbf\{x\}\\in\\mathbf\{R\}^\{d\_\{\\mathrm\{in\}\}\}andW∈𝐑din×doutW\\in\\mathbf\{R\}^\{d\_\{\\mathrm\{in\}\}\\times d\_\{\\mathrm\{out\}\}\}, sparsity can be enforced along either the input dimension or output dimension\. Letℐ​\(𝐦\)=\{i∣mi=1\}\\mathcal\{I\}\(\\mathbf\{m\}\)=\\\{i\\mid m\_\{i\}=1\\\}denote the set of indices retained by a binary mask𝐦\\mathbf\{m\}along the target dimension\.

Input sparsityapplies a mask𝐦∈\{0,1\}din\\mathbf\{m\}\\in\\\{0,1\\\}^\{d\_\{\\mathrm\{in\}\}\}to the input dimension\. Specifically,mi=0m\_\{i\}=0masks theii\-th input channel, eliminating the multiplication ofxix\_\{i\}with the corresponding row ofWW\.

Πin​\(𝐱,W,𝐦\)=𝐱\[ℐ​\(𝐦\)\]​W\[ℐ​\(𝐦\),:\]\.\\Pi^\{\\mathrm\{in\}\}\(\\mathbf\{x\},W,\\mathbf\{m\}\)=\\mathbf\{x\}\_\{\[\\mathcal\{I\}\(\\mathbf\{m\}\)\]\}W\_\{\[\\mathcal\{I\}\(\\mathbf\{m\}\),:\]\}\.
Output sparsityapplies a mask𝐦∈\{0,1\}dout\\mathbf\{m\}\\in\\\{0,1\\\}^\{d\_\{\\mathrm\{out\}\}\}to the output dimension and computes only the selected output coordinates:

Πout​\(𝐱,W,𝐦\)=𝐱​W\[:,ℐ​\(𝐦\)\],\\Pi^\{\\mathrm\{out\}\}\(\\mathbf\{x\},W,\\mathbf\{m\}\)=\\mathbf\{x\}W\_\{\[:,\\mathcal\{I\}\(\\mathbf\{m\}\)\]\},where the selected columns ofWWyield the retained output channels\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x2.png)Figure 2:Computation patterns of representative sparsification methods for SwiGLU FFNs\.The evolution of activation sparsification closely tracks the architectural transition of Transformer FFNs from ReLU\-based designs to modern SwiGLU variants\.

Early activation sparsification methods primarily target ReLU\-based FFNs, skipping input\-dependent inactive or low\-magnitude neurons to reduce computation and memory traffic\(Liet al\.[2022](https://arxiv.org/html/2607.27591#bib.bib19); Liuet al\.[2023](https://arxiv.org/html/2607.27591#bib.bib9); Alizadehet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib21)\)\. While these approaches excel with ReLU\-induced exact or near\-exact zeros, they cannot directly adapt to SiLU\-based SwiGLU FFNs\.

A second line of work, ReLUfication, converts non\-ReLU FFNs to ReLU\-style variants via activation replacement and post\-training adaptation\(Mirzadehet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib30); Zhanget al\.[2024](https://arxiv.org/html/2607.27591#bib.bib31); Songet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib26),[2024](https://arxiv.org/html/2607.27591#bib.bib44)\)\. However, these approaches typically require explicit architectural modifications or resource\-intensive retraining, such as continued training or knowledge distillation\.

For native SwiGLU FFNs, predictor\-based methods estimate salient intermediate channels before executing the full FFN\. For instance,Cheon and Kang \([2025](https://arxiv.org/html/2607.27591#bib.bib7)\)require training model\-specific, layer\-wise low\-rank predictors on FFN inputs to select channels from the intermediate state𝐬\\mathbf\{s\}\. However, limited prediction accuracy can degrade downstream model quality\.

In contrast, training\-free methods sparsify native SwiGLU FFNs using activation magnitudes\. They differ mainly in two dimensions: which activation signals guide channel selection and how the resulting masks induce sparse execution\. Figure[2](https://arxiv.org/html/2607.27591#S2.F2)compares three representative approaches along these two dimensions\. CATS\(Leeet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib4)\)selects intermediate channels based on the magnitude of the densely computed gate activation𝐡\\mathbf\{h\}, inducing output sparsity in the up projection and input sparsity in the down projection \(Figure[2](https://arxiv.org/html/2607.27591#S2.F2)\(a\)\)\. COUNTDOWN\(Cheon and Kang[2025](https://arxiv.org/html/2607.27591#bib.bib7)\)instead performs channel selection using the magnitude of the densely computed up\-branch activation𝐮\\mathbf\{u\}, yielding output sparsity in the gate projection and input sparsity in the down projection \(Figure[2](https://arxiv.org/html/2607.27591#S2.F2)\(b\)\)\.111Throughout this paper, COUNTDOWN refers to the training\-free M\-COUNTDOWN variant unless stated otherwise\.Unlike these single\-branch methods, TEAL\(Liuet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib41)\)applies input sparsity to𝐱\\mathbf\{x\}in the up and gate projections and to𝐬\\mathbf\{s\}in the down projection \(Figure[2](https://arxiv.org/html/2607.27591#S2.F2)\(c\)\) simultaneously\. R\-Sparse follows a similar input\-sparsity pattern while incorporating low\-rank compensation across linear layers\(Zhanget al\.[2025](https://arxiv.org/html/2607.27591#bib.bib48)\)\.

Across the three representative methods in Figure[2](https://arxiv.org/html/2607.27591#S2.F2), model quality deteriorates substantially as sparsity increases, exposing key limitations in their channel\-selection strategies\. CATS and COUNTDOWN estimate intermediate\-channel salience solely from𝐡\\mathbf\{h\}and𝐮\\mathbf\{u\}, respectively; each relies on a single\-branch signal that captures only partial information of the joint intermediate state𝐬=𝐮⊙𝐡\\mathbf\{s\}=\\mathbf\{u\}\\odot\\mathbf\{h\}, leading to misranked salient channels\. TEAL instead applies input sparsity at two successive points: first to𝐱\\mathbf\{x\}, the shared input to the up and gate projections, and then to the resulting intermediate state𝐬\\mathbf\{s\}, the input to the down projection\. The latter therefore operates on an already approximated intermediate state, allowing errors from the two sparsification steps to compound through the FFN\.

These limitations motivate a more effective channel\-selection mechanism tailored for training\-free SwiGLU sparsification\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x3.png)Figure 3:\(a\) Perplexity increase vs\. sparsity under oracle\-𝐬\\mathbf\{s\}channel selection\. \(b\) Empirical distribution of𝐬\\mathbf\{s\}in Qwen3\-8B Layer 8 and fitted Gaussian and Laplace densities; \(c\) Exact vs\. proxy intermediate\-state magnitudes for one representative token from Qwen3\-8B Layer 3 at 70% effective sparsity; colors indicate whether each channel is retained by both top\-ranking selections, only one selection, or neither\. The two selections achieve 82\.77% overlap\. \(d\) Exact/proxy top\-ranking overlap across 512 tokens; the dashed line marks the mean overlap of 82\.04%\.

## 3Motivation

### 3\.1Channel Selection with Intermediate State𝐬\\mathbf\{s\}

Following the weighted\-sum view of SwiGLU FFNs\(Cheon and Kang[2025](https://arxiv.org/html/2607.27591#bib.bib7)\), the FFN output is a weighted sum of down\-projection rows:

FFN​\(𝐱\)=𝐬​Wdown=∑i=1dffsi​Wdown​\[i,:\]\.\\mathrm\{FFN\}\(\\mathbf\{x\}\)=\\mathbf\{s\}W\_\{\\mathrm\{down\}\}=\\sum\_\{i=1\}^\{d\_\{\\mathrm\{ff\}\}\}s\_\{i\}W\_\{\\mathrm\{down\}\}\[i,:\]\.
Analysis\.The input\-dependent coefficientsis\_\{i\}governs the output of each row, making\|si\|\|s\_\{i\}\|a natural metric for channel selection\. Beyond measuring output contribution, the intermediate state𝐬\\mathbf\{s\}is natively aligned with the shared intermediate dimension of all three FFN projections: each index corresponds to an output channel in both up\- and gate\-projections, as well as the matching input channel in the down\-projection\. Accordingly, if the exact intermediate state𝐬\\mathbf\{s\}, denoted as𝐬exact\\mathbf\{s\}\_\{\\mathrm\{exact\}\}, were available prior to FFN execution, a single𝐬\\mathbf\{s\}\-based mask would induce output sparsity in the up\- and gate\-projections alongside input sparsity in the down\-projection, thereby sparsifying all three projections in a unified and consistent manner\.

Assumption and oracle evaluation\.To assess the performance upper bound of𝐬\\mathbf\{s\}\-based channel selection on modern SwiGLU models, we evaluate an ideal diagnostic baseline,*oracle\-𝐬\\mathbf\{s\}*, under the assumption that𝐬exact\\mathbf\{s\}\_\{\\mathrm\{exact\}\}is available at no cost prior to FFN execution\. Given a target sparsityρ\\rho, oracle\-𝐬\\mathbf\{s\}retains the\(1−ρ\)​dff\(1\-\\rho\)d\_\{\\mathrm\{ff\}\}coordinates of𝐬\\mathbf\{s\}with the largest magnitudes to construct a mask across all intermediate channels\.

Figure[3](https://arxiv.org/html/2607.27591#S2.F3)\(a\) empirically validates the above analysis:*oracle\-𝐬\\mathbf\{s\}*preserves model quality across moderate\-to\-high sparsity levels\. Even at 70% sparsity, most tested models experience less than a 3% relative perplexity increase over their dense counterparts\. Figure[3](https://arxiv.org/html/2607.27591#S2.F3)\(b\) provides a complementary explanation for this robustness\. The distribution of𝐬\\mathbf\{s\}is sharply peaked and Laplacian\-like around zero, indicating that many channels carry negligible magnitudes and can be safely skipped with limited impact on FFN output\.

Consequently, while the channel\-selection signal provided by𝐬exact\\mathbf\{s\}\_\{\\mathrm\{exact\}\}serves as an ideal reference, the key challenge in practice is that the exact intermediate state is unavailable until after executing the dense computations for the up\- and gate\-projections\.

### 3\.2Low\-Cost Input\-Sparse Proxy

In fact, sparse execution does not require the exact values of𝐬\\mathbf\{s\}in advance; it only requires the binary channel mask that𝐬exact\\mathbf\{s\}\_\{\\mathrm\{exact\}\}would induce\. Since this mask is determined by the magnitude ranking of entries in𝐬\\mathbf\{s\}, our key insight is that an effective proxy needs only to preserve relative channel rankings rather than accurately construct the exact values of𝐬\\mathbf\{s\}\. We therefore design a low\-cost proxy that bypasses full projection computations and serves exclusively to derive the channel mask\.

Magnitude\-based input sparsity is naturally suited for preserving this ranking, as it minimizes perturbations to the projected values that determine the channel ranking\. Let𝐱sp\\mathbf\{x\}\_\{\\mathrm\{sp\}\}denote the sparse vector retaining the largest\-magnitude entries of𝐱\\mathbf\{x\}, and let𝐫=𝐱−𝐱sp\\mathbf\{r\}=\\mathbf\{x\}\-\\mathbf\{x\}\_\{\\mathrm\{sp\}\}be the discarded residual\. For a fixed density, this selection minimizes‖𝐫‖2\\\|\\mathbf\{r\}\\\|\_\{2\}across all coordinate\-sparse approximations, thus discarding the minimum amount of input energy\. For any projection matrixWW, this residual directly bounds the resulting output error:

‖𝐱​W−𝐱sp​W‖2=‖𝐫​W‖2≤‖𝐫‖2​‖W‖2\.\\begin\{array\}\[\]\{r@\{\}l\}\\\|\\mathbf\{x\}W\-\\mathbf\{x\}\_\{\\mathrm\{sp\}\}W\\\|\_\{2\}&=\\\|\\mathbf\{r\}W\\\|\_\{2\}\\leq\\\|\\mathbf\{r\}\\\|\_\{2\}\\\|W\\\|\_\{2\}\.\\end\{array\}Input sparsity thus reduces the effective input dimension of the projection, bounds the overall perturbation, and preserves all output channels\.

Applying the same sparse input to the up and gate projections yields proxy activations𝐮~\\tilde\{\\mathbf\{u\}\}and𝐡~\\tilde\{\\mathbf\{h\}\}, whose element\-wise product𝐬~=𝐮~⊙𝐡~\\tilde\{\\mathbf\{s\}\}=\\tilde\{\\mathbf\{u\}\}\\odot\\tilde\{\\mathbf\{h\}\}is used to rank intermediate channels by magnitude\. Crucially, the proxy activations are used exclusively to construct the channel mask and are never propagated downstream through the FFN\. Their approximation errors affect model outputs solely via channel\-selection mismatches, which occur primarily when perturbations shift near\-threshold channels across the selection boundary\.

As Figure[3](https://arxiv.org/html/2607.27591#S2.F3)\(c\) illustrates, while exact and input\-sparse proxy values exhibit numerical differences, they yield largely overlapping channel sets\. Figure[3](https://arxiv.org/html/2607.27591#S2.F3)\(d\) further corroborates strong overlap among top\-ranked channels across tokens\. This ranking stability motivates our two\-stage design: the input\-sparse proxy is used exclusively to construct the channel mask, followed by exact computation of the retained channels\.

Appendix[A\.2](https://arxiv.org/html/2607.27591#A1.SS2)provides additional quantitative analyses of intermediate\-state compressibility and proxy ranking preservation across representative FFN layers\.

## 4Methodology

### 4\.1Two\-Stage Sparse Inference

We propose Prox, a two\-stage training\-free framework for sparse SwiGLU FFN inference\. As illustrated in Figure[4](https://arxiv.org/html/2607.27591#S4.F4), given an FFN input𝐱\\mathbf\{x\}, Prox proceeds in the following two stages\.

Stage 1: lightweight proxy construction\.The proxy activations are used exclusively to construct the channel mask𝐦𝐬\\mathbf\{m\}\_\{\\mathbf\{s\}\}\. Their approximation errors thus affect outputs only by altering channel selection, rather than propagating as activation values\. This error isolation enables more aggressive approximation in the proxy path: Prox combines input sparsity with quantized proxy weights to reduce both the effective input dimension and weight memory overhead\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x4.png)Figure 4:Overview of Prox\. A quantized proxy derives a shared intermediate\-channel mask; then the selected intermediate channels are computed with the original weights\.For FFN layerℓ\\ell, lets1s\_\{1\}ands2s\_\{2\}denote the target sparsities of Stage 1 input entries and Stage 2 intermediate channels, respectively\. To achieve the target input sparsitys1s\_\{1\}, Prox constructs a binary input mask𝐦𝐱∈\{0,1\}dmodel\\mathbf\{m\}\_\{\\mathbf\{x\}\}\\in\\\{0,1\\\}^\{d\_\{\\mathrm\{model\}\}\}by retaining entries with magnitudes exceeding a layer\-wise threshold:

\[𝐦𝐱\]i=𝟏​\[\|xi\|≥τ𝐱,ℓs1\],i=1,…,dmodel,\[\\mathbf\{m\}\_\{\\mathbf\{x\}\}\]\_\{i\}=\\mathbf\{1\}\\\!\\left\[\|x\_\{i\}\|\\geq\\tau\_\{\\mathbf\{x\},\\ell\}^\{s\_\{1\}\}\\right\],\\qquad i=1,\\ldots,d\_\{\\mathrm\{model\}\},where𝟏​\[⋅\]\\mathbf\{1\}\[\\cdot\]denotes the indicator function\. The thresholdτ𝐱,ℓs1\\tau\_\{\\mathbf\{x\},\\ell\}^\{s\_\{1\}\}is calibrated offline on layer\-ℓ\\ellinputs to approximately match the target sparsitys1s\_\{1\}over a calibration dataset\. Given quantized proxy weightsW~up\\widetilde\{W\}\_\{\\mathrm\{up\}\}andW~gate\\widetilde\{W\}\_\{\\mathrm\{gate\}\}, the proxy path leverages this input sparsity to compute only entries retained by𝐦𝐱\\mathbf\{m\}\_\{\\mathbf\{x\}\}:

𝐮~=Πin​\(𝐱,W~up,𝐦𝐱\),𝐡~=SiLU​\(Πin​\(𝐱,W~gate,𝐦𝐱\)\)\.\\tilde\{\\mathbf\{u\}\}=\\Pi^\{\\mathrm\{in\}\}\(\\mathbf\{x\},\\widetilde\{W\}\_\{\\mathrm\{up\}\},\\mathbf\{m\}\_\{\\mathbf\{x\}\}\),\\;\\tilde\{\\mathbf\{h\}\}=\\mathrm\{SiLU\}\\\!\\left\(\\Pi^\{\\mathrm\{in\}\}\(\\mathbf\{x\},\\widetilde\{W\}\_\{\\mathrm\{gate\}\},\\mathbf\{m\}\_\{\\mathbf\{x\}\}\)\\right\)\.It then constructs the proxy intermediate signal𝐬~=𝐮~⊙𝐡~\.\\tilde\{\\mathbf\{s\}\}=\\tilde\{\\mathbf\{u\}\}\\odot\\tilde\{\\mathbf\{h\}\}\.

For the target intermediate\-channel sparsitys2s\_\{2\}, Prox applies a corresponding layer\-wise magnitude threshold to the proxy intermediate state:

\[𝐦𝐬\]j=𝟏​\[\|s~j\|≥τ𝐬,ℓs2\],j=1,…,dff,\[\\mathbf\{m\}\_\{\\mathbf\{s\}\}\]\_\{j\}=\\mathbf\{1\}\\\!\\left\[\|\\tilde\{s\}\_\{j\}\|\\geq\\tau\_\{\\mathbf\{s\},\\ell\}^\{s\_\{2\}\}\\right\],\\qquad j=1,\\ldots,d\_\{\\mathrm\{ff\}\},where𝐦𝐬∈\{0,1\}dff\\mathbf\{m\}\_\{\\mathbf\{s\}\}\\in\\\{0,1\\\}^\{d\_\{\\mathrm\{ff\}\}\}denotes the active channels retained for the exact computation path\. Analogous toτ𝐱,ℓs1\\tau\_\{\\mathbf\{x\},\\ell\}^\{s\_\{1\}\}, the thresholdτ𝐬,ℓs2\\tau\_\{\\mathbf\{s\},\\ell\}^\{s\_\{2\}\}is calibrated offline and remains fixed during inference\.

Stage 2: exact sparse computation\.Using the channel mask𝐦𝐬\\mathbf\{m\}\_\{\\mathbf\{s\}\}from Stage 1, Prox performs exact SwiGLU computations exclusively over the selected channels using the original full\-precision model weights\.

𝐮sp=Πout​\(𝐱,Wup,𝐦𝐬\),𝐡sp=SiLU​\(Πout​\(𝐱,Wgate,𝐦𝐬\)\),𝐬sp=𝐮sp⊙𝐡sp,𝐲=Πin​\(𝐬sp,Wdown,𝐦𝐬\)\.\\begin\{array\}\[\]\{r@\{\}l@\{\\,\}r@\{\}l\}\\mathbf\{u\}\_\{\\mathrm\{sp\}\}\\\!&=\\\!\\Pi^\{\\mathrm\{out\}\}\(\\mathbf\{x\},\\mkern\-4\.0muW\_\{\\mathrm\{up\}\},\\mkern\-4\.0mu\\mathbf\{m\}\_\{\\mathbf\{s\}\}\),&\\mathbf\{h\}\_\{\\mathrm\{sp\}\}\\\!&=\\\!\\mathrm\{SiLU\}\\\!\\left\(\\Pi^\{\\mathrm\{out\}\}\(\\mathbf\{x\},\\mkern\-4\.0muW\_\{\\mathrm\{gate\}\},\\mkern\-4\.0mu\\mathbf\{m\}\_\{\\mathbf\{s\}\}\)\\right\),\\\\ \\mathbf\{s\}\_\{\\mathrm\{sp\}\}\\\!&=\\\!\\mathbf\{u\}\_\{\\mathrm\{sp\}\}\\odot\\mathbf\{h\}\_\{\\mathrm\{sp\}\},&\\mathbf\{y\}\\\!&=\\\!\\Pi^\{\\mathrm\{in\}\}\(\\mathbf\{s\}\_\{\\mathrm\{sp\}\},\\\!W\_\{\\mathrm\{down\}\},\\\!\\mathbf\{m\}\_\{\\mathbf\{s\}\}\)\.\\end\{array\}

### 4\.2Sparsity Allocation between Two Stages

s1s\_\{1\}ands2s\_\{2\}jointly govern the overall computational cost of Prox\. Reducing Stage 1 sparsity retains more input entries to yield a more informative proxy signal, while reducing Stage 2 sparsity preserves more intermediate channels for exact SwiGLU computation\. Since both adjustments increase computation,s1s\_\{1\}ands2s\_\{2\}must be jointly optimized under a target FFN compute budget\.

We model computational overhead with a normalized cost framework\. Denseup\-,gate\-, anddown\-projections each carry a unit cost, yielding a total dense FFN computational costCDense=3C\_\{\\mathrm\{Dense\}\}=3\. Letα\\alphadenote the normalized cost of a quantized proxy projection relative to a full dense projection\. Given sparsitys1s\_\{1\}ands2s\_\{2\}, Stage 1 executes two proxy projections over a\(1−s1\)\(1\-s\_\{1\}\)fraction of the input dimension, while Stage 2 executes three full\-precision projections over a\(1−s2\)\(1\-s\_\{2\}\)fraction of the intermediate channels\. The total normalized cost is:

CProx=2​α​\(1−s1\)\+3​\(1−s2\)\.C\_\{\\mathrm\{Prox\}\}=2\\alpha\(1\-s\_\{1\}\)\+3\(1\-s\_\{2\}\)\.We define the resulting effective sparsityeerelative to the dense FFN baseline as:

e=1−CProxCDense=s2−2​α​\(1−s1\)3\.e=1\-\\frac\{C\_\{\\mathrm\{Prox\}\}\}\{C\_\{\\mathrm\{Dense\}\}\}=s\_\{2\}\-\\frac\{2\\alpha\(1\-s\_\{1\}\)\}\{3\}\.
Our budget allocation policy is inspired by the findings of oracle\-𝐬\\mathbf\{s\}in Figure[3](https://arxiv.org/html/2607.27591#S2.F3)\(a\)\. The results demonstrate that sparsifying approximately 70% of the exact intermediate channels incurs negligible performance degradation across the evaluated models, highlighting substantial channel redundancy in Stage 2 exact computation\. We thus setsref=0\.7s\_\{\\mathrm\{ref\}\}=0\.7as a quality\-preserving anchor for Stage 2\. Given a target effective sparsityee, we initially sets2=srefs\_\{2\}=s\_\{\\mathrm\{ref\}\}and allocate the remaining compute budget to Stage 1 according to the effective\-sparsity relation above\.

However, Stage 1 sparsity cannot be increased indefinitely; removing excessive input coordinates diminishes the fidelity of the proxy signal, thereby compromising the reliability of channel selection\. To mitigate this, we constrains1s\_\{1\}to\[s1,min,s1,max\]=\[0,0\.7\]\[s\_\{1,\\min\},s\_\{1,\\max\}\]=\[0,0\.7\], ensuring at least 30% of the input entries are retained\.

If the Stage 1 sparsity implied bys2=srefs\_\{2\}=s\_\{\\mathrm\{ref\}\}falls outside this interval, we clamps1s\_\{1\}to the nearest bound and dynamically adjusts2s\_\{2\}to satisfy the target effective sparsityee\. Detailed budget allocations across operating points are provided in Appendix[A\.4](https://arxiv.org/html/2607.27591#A1.SS4)\.

### 4\.3Hardware\-Aware Implementation

To realize practical acceleration, we construct the Stage 1 proxy weights via symmetric per\-row INT4 quantization of the original FP16WupW\_\{\\mathrm\{up\}\}andWgateW\_\{\\mathrm\{gate\}\}matrices, keeping the compact proxies resident in GPU memory\. We implement dedicated CUDA and Triton kernels tailored for single\-batch autoregressive decoding\.

Stage 1 employs a fused split\-NNCUDA kernel that evaluates the up\- and gate\-proxy projections together, sharing hidden\-state reads and skipping input coordinates rejected by the calibrated threshold\. INT4 unpacking is performed directly in registers, while per\-thread partial sums are reduced in FP32\. The resulting projection sums are rescaled and combined through SwiGLU to form the proxy intermediate state𝐬~\\tilde\{\\mathbf\{s\}\}, whose magnitude is thresholded to produce the Stage 2 channel mask𝐦𝐬\\mathbf\{m\}\_\{\\mathbf\{s\}\}without storing separate FP16 up and gate proxy vectors\. Launch geometries and reduction strategies are autotuned across GPU architectures, matrix shape, and sparsity regimes\.

In Stage 2, a fused output\-sparse kernel computes the exact up and gate projections together with SwiGLU exclusively over active channels selected by𝐦𝐬\\mathbf\{m\}\_\{\\mathbf\{s\}\}; the down projection then executes an input\-sparse GEMV over these channels\. Mask handling remains inside the main computation loops, restricting weight accesses and MACs to active coordinates and channels\.

## 5Experiments

Qwen3Qwen3\.5MinistralMistralLlama3Gemma3VariantSparsity1\.7B4B8B14B4B9B3\.3B7B8B12BDense061\.471\.176\.177\.969\.370\.969\.971\.170\.177\.9CATS40%35\.344\.959\.667\.959\.161\.857\.763\.765\.976\.4COUNTDOWN40%59\.369\.473\.675\.763\.067\.065\.666\.868\.777\.1TEAL40%59\.471\.076\.177\.367\.370\.069\.368\.069\.875\.2Prox40%61\.570\.875\.777\.669\.670\.468\.769\.369\.777\.6CATS50%28\.533\.349\.953\.051\.957\.036\.446\.744\.162\.8COUNTDOWN50%51\.265\.971\.475\.260\.662\.664\.062\.261\.373\.2TEAL50%58\.367\.973\.477\.066\.769\.667\.267\.668\.370\.6Prox50%59\.568\.874\.676\.666\.869\.568\.167\.770\.177\.5CATS60%27\.326\.327\.631\.936\.343\.527\.626\.726\.732\.9COUNTDOWN60%33\.745\.056\.866\.749\.955\.145\.950\.044\.254\.0TEAL60%55\.966\.271\.475\.664\.668\.263\.464\.666\.866\.8Prox60%59\.366\.473\.376\.165\.068\.467\.767\.067\.277\.5TEAL70%41\.852\.863\.370\.354\.161\.252\.259\.457\.162\.0Prox70%56\.558\.768\.674\.859\.163\.065\.367\.563\.474\.8Table 1:Aggregate downstream task scores across model families and sparsity levels\. Higher is better\.Models and datasets\.We evaluate Prox on SwiGLU\-based model families, including Qwen3\(Team[2025b](https://arxiv.org/html/2607.27591#bib.bib28)\), Qwen3\.5\(Qwen Team[2026](https://arxiv.org/html/2607.27591#bib.bib27)\), Ministral, Mistral\(Jianget al\.[2023](https://arxiv.org/html/2607.27591#bib.bib17)\), and Llama\-3\(AI@Meta[2024](https://arxiv.org/html/2607.27591#bib.bib20)\)\. To assess generalization beyond SwiGLU, we also evaluate Gemma\-3\(Team[2025a](https://arxiv.org/html/2607.27591#bib.bib12)\)that features a GeGLU\-style FFN\. Downstream accuracy is evaluated via the EleutherAI LM Harness\(Gaoet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib10)\)across six tasks: 8\-shot GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2607.27591#bib.bib6)\), 5\-shot MMLU\(Hendryckset al\.[2020](https://arxiv.org/html/2607.27591#bib.bib25)\), 10\-shot HellaSwag\(Zellerset al\.[2019](https://arxiv.org/html/2607.27591#bib.bib47)\), zero\-shot ARC\-Easy and 25\-shot ARC\-Challenge\(Clarket al\.[2018](https://arxiv.org/html/2607.27591#bib.bib1)\), and zero\-shot PIQA\(Bisket al\.[2020](https://arxiv.org/html/2607.27591#bib.bib3)\)\.

Baselines\.We compare Prox against three training\-free activation sparsity baselines: CATS\(Leeet al\.[2024](https://arxiv.org/html/2607.27591#bib.bib4)\), COUNTDOWN\(Cheon and Kang[2025](https://arxiv.org/html/2607.27591#bib.bib7)\), and TEAL\(Liuet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib41)\)\. We use M\-COUNTDOWN as the COUNTDOWN baseline because it achieves higher average downstream scores than D\-COUNTDOWN under practical inference\.

To ensure a fair comparison focused on FFN sparsification, we apply all methods exclusively to FFN projections and report sparsity at the FFN\-module level\. Specifically, for CATS and COUNTDOWN, a projection\-level sparsity ofkkyields an effective sparsity of2​k/32k/3; TEAL’s effective sparsity is averaged across its three FFN projections, while Prox’s effective sparsity follows the cost model in Sec\.[4\.2](https://arxiv.org/html/2607.27591#S4.SS2)\. Although the theoretical INT4\-to\-FP16 bit\-width ratio impliesα=0\.25\\alpha=0\.25, it ignores quantization overhead\. We therefore setα=1/3\\alpha=1/3, a single rounded value within the measured INT4\-to\-FP16 GEMV latency\-ratio range across three GPUs and four Qwen3 scales \(details in Appendix[A\.3](https://arxiv.org/html/2607.27591#A1.SS3)\)\.

These baselines primarily target decoding\-phase sparsification\. Nevertheless, for log\-likelihood tasks, we sparsify the full sequence to ensure hidden states are computed under sparsity; for generation tasks such as GSM8K, we maintain a dense prompt prefill and sparsify decoding exclusively\.

### 5\.1Results

Downstream accuracy\.As shown in Table[1](https://arxiv.org/html/2607.27591#S5.T1), Prox demonstrates its superiority at high sparsity levels while remaining competitive with TEAL at 40% sparsity\. Specifically, Prox surpasses TEAL on 8 out of 10 evaluated models at 50% sparsity and consistently outperforms it across all 10 models at 60% and 70% sparsity\. In contrast, CATS and COUNTDOWN exhibit premature degradation as sparsity increases; restricted to sparsifying only two FFN matrices, they fail to reach the 70% operating point sustained by TEAL and Prox\. Detailed perplexity results are provided in Appendix[A\.8](https://arxiv.org/html/2607.27591#A1.SS8)\.

Role of exact computation\.CATS and COUNTDOWN approximate the SwiGLU intermediate state𝐬\\mathbf\{s\}via a single branch, rendering channel selection unreliable under high sparsity\. TEAL can be viewed as a simplified Prox since it directly reuses a sparse approximation of𝐬\\mathbf\{s\}for FFN computation; while this reduces computation, it propagates proxy errors into the output\. In contrast, Prox uses the proxy only for channel selection and computes the selected values exactly, improving robustness at high sparsity\.

End\-to\-end decoding speedup\.We benchmark Prox and TEAL under single\-batch end\-to\-end decoding on an NVIDIA A6000 GPU\. As Figure[5](https://arxiv.org/html/2607.27591#S5.F5)shows, both methods effectively leverage FFN sparsity to boost decoding throughput across models\. Prox achieves a1\.511\.51–1\.99×1\.99\\timesspeedup at 60–70% sparsity\. At 70% sparsity, Prox differs from TEAL in throughput by no more than 2\.9% across all evaluated models and improves the average downstream task performance by 14\.4%\. A detailed latency breakdown and additional throughput measurements on A100 and RTX 3090 GPUs are provided in Appendix[A\.5](https://arxiv.org/html/2607.27591#A1.SS5)\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x5.png)Figure 5:Single\-batch decoding speedup on an NVIDIA A6000 GPU\. Headers give dense throughput \(tokens/s\), and numeric annotations indicate Prox’s speedup at 70% sparsity\.![Refer to caption](https://arxiv.org/html/2607.27591v1/x6.png)Figure 6:Perplexity of quantized Qwen3\-8B across FFN sparsity levels \(log scale; lower is better\)\.Compatibility with model quantization\.Model quantization and FFN sparsification are complementary techniques that can be combined for efficient inference\. We evaluate Prox and three baselines on Qwen3\-8B across four quantization configurations \(AWQ, FP8, W4A16, W8A8\) by measuring WikiText perplexity\. As Figure[6](https://arxiv.org/html/2607.27591#S5.F6)shows, Prox consistently achieves the lowest perplexity across all configurations and sparsities, with its performance advantage becoming increasingly pronounced in high\-sparsity regimes\. Combining low\-bit weights with activation sparsity can further reduce weight traffic and improve inference speed, but realizing these gains requires specialized kernels that jointly support sparse execution and low\-bit arithmetic, which we leave to future work\.

Decoding Throughput \(tok/s\)AccuracyVariant16K32KLongBench /RULERDense \(Qwen3\-8B\)33\.1 \(1\.00×1\.00\\times\)23\.9 \(1\.00×1\.00\\times\)100\.0 / 100\.0RocketKV40\.7 \(1\.23×1\.23\\times\)41\.0 \(1\.72×1\.72\\times\)100\.0 / 100\.0RocketKV \+ Prox63\.6 \(1\.92×1\.92\\times\)62\.5 \(2\.62×2\.62\\times\)96\.6 / 98\.0HShare34\.6 \(1\.05×1\.05\\times\)39\.0 \(1\.63×1\.63\\times\)93\.5 / 99\.0HShare \+ Prox40\.0 \(1\.21×1\.21\\times\)42\.1 \(1\.76×1\.76\\times\)92\.5 / 100\.0Table 2:Compatibility of Prox \(60% FFN sparsity\) with different sparse\-attention backends\.Compatibility with sparse attention\.Prox operates solely on FFNs and is thus orthogonal to sparse\-attention techniques, enabling combined computational savings across both attention and FFN modules\. Table[2](https://arxiv.org/html/2607.27591#S5.T2)presents results integrated with RocketKV\(Behnamet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib32)\)and HShare\(Wuet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib15)\)at 16K and 32K sequence lengths\. Integrating Prox with either backend further improves decoding throughput while largely preserving long\-context accuracy\.

### 5\.2Ablation Studies

Table 3:Ablation results on aggregate downstream scores\.A1: Effect of proxy\-weight precision\.To evaluate whether higher precision improves channel selection, A1 replaces the INT4 proxy weights with FP16 weights, reallocating the two\-stage compute under the same target budget to accommodate the increased proxy cost\. As Table[3](https://arxiv.org/html/2607.27591#S5.T3)shows, A1 consistently underperforms Prox across all eight settings, with performance deficits reaching 9\.0 and 4\.7 points at 70% sparsity on Qwen3\-8B and Qwen3\-14B, respectively\. This confirms that higher proxy precision does not lead to better overall quality under fixed budgets, justifying our low\-bit proxy design in Stage 1\.

A2: Effectiveness of target\-aware compute allocation\.A2 fixes Stage 1 sparsity ats1=0\.8s\_\{1\}=0\.8across all targets and adjusts Stage 2 to meet the budget, whereas Prox dynamically adapts boths1s\_\{1\}ands2s\_\{2\}to the target sparsity\. We selects1=0\.8s\_\{1\}=0\.8as a representative low\-cost fixed baseline \(a full Qwen3\-8B sweep overs1s\_\{1\}is provided in Appendix[A\.9](https://arxiv.org/html/2607.27591#A1.SS9)\)\. Across all eight model–sparsity settings, target\-aware allocation matches or outperforms this fixed baseline, yielding gains of up to 4\.4 points on Qwen3\-8B and 0\.2–0\.3 points on Qwen3\-14B\. Together with the full sweep, these results demonstrate the necessity of dynamically adapting stage\-wise allocations across operating points\.

A3: Necessity of exact computation\.A3 retains Prox’s complete setup but skips Stage 2 exact computation, directly feeding selected proxy values into the down projection\. This evaluates whether quantized proxies can serve as final activation values rather than just channel selectors\. Prox outperforms A3 across all settings, with performance gaps reaching 24\.3 points on Qwen3\-8B and 18\.1 points on Qwen3\-14B at 70% sparsity\. This demonstrates that while quantized proxies effectively identify salient channels, they cannot replace exact activations, confirming that Stage 2 exact computation is essential\.

### 5\.3Discussion

Prox currently targets single\-batch autoregressive decoding, a typical setting for latency\-sensitive, on\-device LLM applications\. Consequently, its primary limitation is the lack of support for large\-batch serving\. Extending proxy\-guided sparsification and its underlying sparse kernels to efficient batched inference remains a promising avenue for future exploration\. Additionally, retaining Stage 1 INT4 proxy weights in GPU memory introduces an approximate 12% weight\-storage overhead\. Future work could reduce this overhead through a sliding\-window scheme that dynamically loads and evicts proxy weights across Transformer layers\.

## 6Conclusion

We present Prox, a training\-free FFN sparsification framework for SwiGLU\-based LLMs\. Prox employs a low\-cost quantized proxy for input\-dependent channel selection, followed by exact value computation of the selected channels\. Combined with a target\-aware stage\-wise allocation policy and tailored sparse kernels, Prox preserves model accuracy and achieves up to a1\.99×1\.99\\timesend\-to\-end decoding speedup at 70% FFN sparsity\. Comprehensive evaluations across ten LLMs from six model families demonstrate that Prox consistently outperforms existing training\-free baselines across all sparsity levels and maintains seamless compatibility with model quantization and sparse attention\.

## References

- Llama 3 model card\.External Links:[Link](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md)Cited by:[§2\.1](https://arxiv.org/html/2607.27591#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- K\. Alizadeh, S\. I\. Mirzadeh, D\. Belenko, S\. Khatamifard, M\. Cho, C\. C\. Del Mundo, M\. Rastegari, and M\. Farajtabar \(2024\)Llm in a flash: efficient large language model inference with limited memory\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12562–12584\.Cited by:[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p5.1)\.
- S\. Ashkboos, M\. Croci, M\. Gennari do Nascimento, T\. Hoefler, and J\. Hensman \(2024\)Slicegpt: compress large language models by deleting rows and columns\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 11682–11701\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- P\. Behnam, Y\. Fu, R\. Zhao, P\. Tsai, Z\. Yu, and A\. Tumanov \(2025\)RocketKV: accelerating long\-context LLM inference via two\-stage KV cache compression\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=RyOpooIxDF)Cited by:[§5\.1](https://arxiv.org/html/2607.27591#S5.SS1.p5.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- X\. Chen, Z\. Liu, H\. Tang, L\. Yi, H\. Zhao, and S\. Han \(2023\)Sparsevit: revisiting activation sparsity for efficient high\-resolution vision transformer\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 2061–2070\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- J\. Cheon and P\. Kang \(2025\)COUNTDOWN: contextually sparse activation filtering out unnecessary weights in down projection\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 28381–28397\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p7.1),[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p8.4),[§3\.1](https://arxiv.org/html/2607.27591#S3.SS1.p1.1),[§5](https://arxiv.org/html/2607.27591#S5.p2.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- Y\. N\. Dauphin, A\. Fan, M\. Auli, and D\. Grangier \(2017\)Language modeling with gated convolutional networks\.InInternational conference on machine learning,pp\. 933–941\.Cited by:[§2\.1](https://arxiv.org/html/2607.27591#S2.SS1.p1.1)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)Gpt3\. int8 \(\): 8\-bit matrix multiplication for transformers at scale\.Advances in neural information processing systems35,pp\. 30318–30332\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- E\. Frantar and D\. Alistarh \(2023\)Sparsegpt: massive language models can be accurately pruned in one\-shot\.InInternational conference on machine learning,pp\. 10323–10337\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§2\.1](https://arxiv.org/html/2607.27591#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- M\. Kurtz, J\. Kopinsky, R\. Gelashvili, A\. Matveev, J\. Carr, M\. Goin, W\. Leiserson, S\. Moore, N\. Shavit, and D\. Alistarh \(2020\)Inducing and exploiting activation sparsity for fast inference on deep neural networks\.InInternational conference on machine learning,pp\. 5533–5543\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- D\. Lee, J\. Lee, G\. Zhang, M\. Tiwari, and A\. Mirhoseini \(2024\)Cats: contextually\-aware thresholding for sparsity in large language models\.arXiv preprint arXiv:2404\.08763\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p8.4),[§5](https://arxiv.org/html/2607.27591#S5.p2.1)\.
- Z\. Li, C\. You, S\. Bhojanapalli, D\. Li, A\. S\. Rawat, S\. J\. Reddi, K\. Ye, F\. Chern, F\. Yu, R\. Guo,et al\.\(2022\)The lazy neuron phenomenon: on emergence of activation sparsity in transformers\.arXiv preprint arXiv:2210\.06313\.Cited by:[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p5.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)Awq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems6,pp\. 87–100\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- J\. Liu, P\. Ponnusamy, T\. Cai, Y\. Kim, B\. Athiwaratkun,et al\.\(2025\)Training\-free activation sparsity in large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 98302–98322\.Cited by:[§A\.2](https://arxiv.org/html/2607.27591#A1.SS2.p3.6),[§1](https://arxiv.org/html/2607.27591#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p8.4),[§5](https://arxiv.org/html/2607.27591#S5.p2.1)\.
- Z\. Liu, J\. Wang, T\. Dao, T\. Zhou, B\. Yuan, Z\. Song, A\. Shrivastava, C\. Zhang, Y\. Tian, C\. Re,et al\.\(2023\)Deja vu: contextual sparsity for efficient llms at inference time\.InInternational Conference on Machine Learning,pp\. 22137–22176\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p5.1)\.
- X\. Ma, G\. Fang, and X\. Wang \(2023\)Llm\-pruner: on the structural pruning of large language models\.Advances in neural information processing systems36,pp\. 21702–21720\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- X\. Men, M\. Xu, Q\. Zhang, Q\. Yuan, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. Chen \(2025\)Shortgpt: layers in large language models are more redundant than you expect\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 20192–20204\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§A\.8](https://arxiv.org/html/2607.27591#A1.SS8.p1.1)\.
- I\. Mirzadeh, K\. Alizadeh\-Vahid, S\. Mehta, C\. C\. Del Mundo, O\. Tuzel, G\. Samei, M\. Rastegari, and M\. Farajtabar \(2024\)Relu strikes back: exploiting activation sparsity in large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 25378–25400\.Cited by:[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p6.1)\.
- K\. Qiao, S\. Zhang, and Y\. Feng \(2025\)IG\-pruning: input\-guided block pruning for large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 10629–10640\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- M\. A\. Raihan and T\. Aamodt \(2020\)Sparse weight activation training\.Advances in Neural Information Processing Systems33,pp\. 15625–15638\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- N\. Shazeer \(2020\)GLU variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[§2\.1](https://arxiv.org/html/2607.27591#S2.SS1.p1.1)\.
- C\. Song, X\. Han, Z\. Zhang, S\. Hu, X\. Shi, K\. Li, C\. Chen, Z\. Liu, G\. Li, T\. Yang,et al\.\(2025\)Prosparse: introducing and enhancing intrinsic activation sparsity within large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 2626–2644\.Cited by:[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p6.1)\.
- Y\. Song, H\. Xie, Z\. Zhang, B\. Wen, L\. Ma, Z\. Mi, and H\. Chen \(2024\)Turbo sparse: achieving llm sota performance with minimal activated parameters\.arXiv preprint arXiv:2406\.05955\.Cited by:[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p6.1)\.
- M\. Sun, Z\. Liu, A\. Bair, and Z\. Kolter \(2024\)A simple and effective pruning approach for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 4942–4964\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- F\. Szatkowski, P\. Będkowski, A\. Devoto, J\. Dubiński, P\. Minervini, M\. Piórczyński, S\. Scardapane, and B\. Wójcik \(2026\)Universal properties of activation sparsity in modern large language models\.External Links:2509\.00454,[Link](https://arxiv.org/abs/2509.00454)Cited by:[§A\.8](https://arxiv.org/html/2607.27591#A1.SS8.p3.1),[§1](https://arxiv.org/html/2607.27591#S1.p2.1)\.
- G\. Team \(2025a\)Gemma 3\.External Links:[Link](https://goo.gle/Gemma3Report)Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- Q\. Team \(2025b\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§2\.1](https://arxiv.org/html/2607.27591#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- P\. Tillet, H\. Kung, and D\. Cox \(2019\)Triton: an intermediate language and compiler for tiled neural network computations\.InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages,pp\. 10–19\.Cited by:[§1](https://arxiv.org/html/2607.27591#S1.p4.1)\.
- H\. Wu, L\. Li, H\. Huang, Y\. Tu, J\. Zhang, M\. Yu, and J\. Yan \(2025\)HShare: fast llm decoding by hierarchical key\-value sharing\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2607.27591#S5.SS1.p5.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)Hellaswag: can a machine really finish your sentence?\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 4791–4800\.Cited by:[§5](https://arxiv.org/html/2607.27591#S5.p1.1)\.
- Z\. Zhang, Y\. Song, G\. Yu, X\. Han, Y\. Lin, C\. Xiao, C\. Song, Z\. Liu, Z\. Mi, and M\. Sun \(2024\)ReLU2wins: discovering efficient activation functions for sparse llms\.arXiv preprint arXiv:2402\.03804\.Cited by:[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p6.1)\.
- Z\. Zhang, Z\. Liu, Y\. Tian, H\. Khaitan, Z\. Wang, and S\. Li \(2025\)R\-sparse: rank\-aware activation sparsity for efficient llm inference\.arXiv preprint arXiv:2504\.19449\.Cited by:[§2\.2](https://arxiv.org/html/2607.27591#S2.SS2.p8.4)\.

## Appendix AAppendix

### A\.1FFN Parameter and MAC Calculation

We calculate the parameter and MAC shares in Figure[1](https://arxiv.org/html/2607.27591#S1.F1)as follows\. LetSSdenote the sequence length,dmodeld\_\{\\mathrm\{model\}\}the hidden dimension,dffd\_\{\\mathrm\{ff\}\}the FFN intermediate dimension,HqH\_\{q\}andHkvH\_\{\\mathrm\{kv\}\}the numbers of query and key–value heads, anddhd\_\{h\}the head dimension\. The query and key–value projection dimensions are defined as:

dq=Hq​dh,dkv=Hkv​dh\.d\_\{q\}=H\_\{q\}d\_\{h\},\\qquad d\_\{\\mathrm\{kv\}\}=H\_\{\\mathrm\{kv\}\}d\_\{h\}\.The parameter counts of the FFN and one Transformer block are

PFFN=3​dmodel​dff,P\_\{\\mathrm\{FFN\}\}=3d\_\{\\mathrm\{model\}\}d\_\{\\mathrm\{ff\}\},PBlock=3​dmodel​dff\+2​dmodel​\(dq\+dkv\)\+2​dmodel\+2​dh,\\begin\{array\}\[\]\{rcl\}P\_\{\\mathrm\{Block\}\}&=&3d\_\{\\mathrm\{model\}\}d\_\{\\mathrm\{ff\}\}\+2d\_\{\\mathrm\{model\}\}\(d\_\{q\}\+d\_\{\\mathrm\{kv\}\}\)\\\\ &&\{\}\+2d\_\{\\mathrm\{model\}\}\+2d\_\{h\},\\end\{array\}Rparam=PFFNPBlock\.R\_\{\\mathrm\{param\}\}=\\frac\{P\_\{\\mathrm\{FFN\}\}\}\{P\_\{\\mathrm\{Block\}\}\}\.At sequence lengthSS, the corresponding MAC counts are

MFFN=3​S​dmodel​dff,M\_\{\\mathrm\{FFN\}\}=3Sd\_\{\\mathrm\{model\}\}d\_\{\\mathrm\{ff\}\},MBlock=3​S​dmodel​dff\+2​S​dmodel​\(dq\+dkv\)\+2​S2​dq,\\begin\{array\}\[\]\{rcl\}M\_\{\\mathrm\{Block\}\}&=&3Sd\_\{\\mathrm\{model\}\}d\_\{\\mathrm\{ff\}\}\+2Sd\_\{\\mathrm\{model\}\}\(d\_\{q\}\+d\_\{\\mathrm\{kv\}\}\)\\\\ &&\{\}\+2S^\{2\}d\_\{q\},\\end\{array\}RMAC=MFFNMBlock\.R\_\{\\mathrm\{MAC\}\}=\\frac\{M\_\{\\mathrm\{FFN\}\}\}\{M\_\{\\mathrm\{Block\}\}\}\.We setS=4096S=4096and use the model configurations in Table[4](https://arxiv.org/html/2607.27591#A1.T4)\. The calculation includes the Q/K/V/O projections and RMSNorm parameters, but excludes embeddings, the LM head, softmax, RoPE, activation functions, and other element\-wise operations\.

### A\.2Activation Distributions and Ranking Preservation

Figure[7](https://arxiv.org/html/2607.27591#A1.F7)contrasts the activation distributions at the two sides of a SwiGLU FFN\. The FFN input𝐱\\mathbf\{x\}, shared by the up and gate projections, is approximately Gaussian, whereas the intermediate state𝐬\\mathbf\{s\}supplied to the down projection is approximately Laplacian\. These distinct shapes clarify why magnitude sparsification plays different roles in exact\-𝐬\\mathbf\{s\}channel selection, TEAL’s successive input\-sparsity scheme, and the selection\-oriented proxy used by Prox\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x7.png)Figure 7:Activation\-value distributions at representative SwiGLU FFN layers of Qwen3\-8B, collected using the same calibration set as threshold calibration: the first 10 contiguous sequences of 2,048 tokens from the Alpaca training split\. The top row shows the shared input𝐱\\mathbf\{x\}to the up and gate projections, together with fitted Gaussian and Laplace densities\. The bottom row shows the intermediate state𝐬\\mathbf\{s\}supplied to the down projection\. The FFN input is Gaussian\-like, whereas the intermediate state is Laplacian\-like and more sharply concentrated around zero\.Intermediate\-state compressibility\.The Laplacian\-like shape of𝐬\\mathbf\{s\}consists of a sharp central peak and comparatively heavy tails, so many channels have negligible magnitudes while a small fraction carries most of the activation energy\. This concentration can be quantified under an ideal Laplace model\. LetS∼Laplace​\(0,b\)S\\sim\\mathrm\{Laplace\}\(0,b\), and prune the fractionppof entries with the smallest magnitudes\. Withap=−log⁡\(1−p\)a\_\{p\}=\-\\log\(1\-p\), the retained fraction of squared activation energy is

qLap​\(p\)=E​\[S2​𝟏​\(\|S\|\>b​ap\)\]E​\[S2\]=1−p2​\(ap2\+2​ap\+2\)\.q\_\{\\mathrm\{Lap\}\}\(p\)=\\frac\{\\mathrm\{E\}\\\!\\left\[S^\{2\}\\mathbf\{1\}\(\|S\|\>ba\_\{p\}\)\\right\]\}\{\\mathrm\{E\}\[S^\{2\}\]\}=\\frac\{1\-p\}\{2\}\\left\(a\_\{p\}^\{2\}\+2a\_\{p\}\+2\\right\)\.Atp=0\.7p=0\.7, retaining only the largest\-magnitude 30% of entries preserves approximately 87\.9% of the squared activation energy\. Although activation energy alone does not determine the final FFN error because each coefficientsis\_\{i\}scales a different row ofWdownW\_\{\\mathrm\{down\}\}, this calculation provides a distributional explanation consistent with the oracle\-𝐬\\mathbf\{s\}result in Figure[3](https://arxiv.org/html/2607.27591#S2.F3)\(a\): most low\-magnitude intermediate channels can be removed with little quality degradation\.

Table 4:Qwen3 configurations used to calculate the FFN parameter and MAC shares in Figure[1](https://arxiv.org/html/2607.27591#S1.F1)\.Value distortion under input sparsity\.The Gaussian\-like FFN input is less concentrated than the Laplacian\-like intermediate state, so magnitude pruning at the same sparsity discards a larger fraction of its energy\. Following the idealized analysis of magnitude\-based activation sparsity in TEAL\(Liuet al\.[2025](https://arxiv.org/html/2607.27591#bib.bib41)\), letX∼𝒩​\(0,σx2\)X\\sim\\mathcal\{N\}\(0,\\sigma\_\{x\}^\{2\}\), assume independent isotropic Gaussian weights, and prune the fractionppof input entries satisfying\|X\|≤σx​zp\|X\|\\leq\\sigma\_\{x\}z\_\{p\}, wherezp=Φ−1​\(\(1\+p\)/2\)z\_\{p\}=\\Phi^\{\-1\}\(\(1\+p\)/2\), andΦ\\Phiandϕ\\phidenote the standard Gaussian cumulative distribution and density functions\. The retained input\-energy fraction is

qG​\(p\)=1−p\+2​zp​ϕ​\(zp\),q\_\{\\mathrm\{G\}\}\(p\)=1\-p\+2z\_\{p\}\\phi\(z\_\{p\}\),and the corresponding relative root\-mean\-square projection error is

η​\(p\)=1−qG​\(p\)=p−2​zp​ϕ​\(zp\)\.\\eta\(p\)=\\sqrt\{1\-q\_\{\\mathrm\{G\}\}\(p\)\}=\\sqrt\{p\-2z\_\{p\}\\phi\(z\_\{p\}\)\}\.For example, at 70% input sparsity,qG​\(0\.7\)≈0\.783q\_\{\\mathrm\{G\}\}\(0\.7\)\\approx 0\.783andη​\(0\.7\)≈0\.465\\eta\(0\.7\)\\approx 0\.465\. Thus, magnitude sparsification is substantially better than discarding inputs indiscriminately, but its projected values can still differ considerably from their dense counterparts\. In methods such as TEAL with successive input\-sparsity stages, approximation errors accumulate as they propagate through the up and gate branches, their element\-wise product, and the down projection, making aggressive input sparsity increasingly fragile\.

Ranking preservation under input sparsity\.Large value error does not imply that the proxy has lost its ranking information\. For one linear\-projection output channel with weight vector𝐰j\\mathbf\{w\}\_\{j\}, decompose𝐱=𝐱sp\+𝐫\\mathbf\{x\}=\\mathbf\{x\}\_\{\\mathrm\{sp\}\}\+\\mathbf\{r\}into the retained input and discarded residual:

yj=⟨𝐱,𝐰j⟩=⟨𝐱sp,𝐰j⟩⏟y~j\+⟨𝐫,𝐰j⟩⏟ej\.y\_\{j\}=\\langle\\mathbf\{x\},\\mathbf\{w\}\_\{j\}\\rangle=\\underbrace\{\\langle\\mathbf\{x\}\_\{\\mathrm\{sp\}\},\\mathbf\{w\}\_\{j\}\\rangle\}\_\{\\tilde\{y\}\_\{j\}\}\+\\underbrace\{\\langle\\mathbf\{r\},\\mathbf\{w\}\_\{j\}\\rangle\}\_\{e\_\{j\}\}\.Under the same independence and isotropy assumptions,y~j\\tilde\{y\}\_\{j\}andeje\_\{j\}are uncorrelated\. Consequently,

Corr​\(yj,y~j\)=Var​\(y~j\)Var​\(yj\)≈qG​\(p\)\.\\mathrm\{Corr\}\(y\_\{j\},\\tilde\{y\}\_\{j\}\)=\\sqrt\{\\frac\{\\mathrm\{Var\}\(\\tilde\{y\}\_\{j\}\)\}\{\\mathrm\{Var\}\(y\_\{j\}\)\}\}\\approx\\sqrt\{q\_\{\\mathrm\{G\}\}\(p\)\}\.At 70% input sparsity, this idealized correlation is approximately0\.8850\.885, despite the relative projection error of 0\.465\. Applying the same sparse input to both SwiGLU branches therefore preserves a substantial shared signal from which Prox constructs𝐬~\\tilde\{\\mathbf\{s\}\}\. Ifs~i=si\+δi\\tilde\{s\}\_\{i\}=s\_\{i\}\+\\delta\_\{i\}, the magnitude order of channelsiiandjjis guaranteed to remain unchanged whenever

\|\|si\|−\|sj\|\|\>\|δi\|\+\|δj\|\.\\bigl\|\|s\_\{i\}\|\-\|s\_\{j\}\|\\bigr\|\>\|\\delta\_\{i\}\|\+\|\\delta\_\{j\}\|\.The large\-magnitude tail channels of the Laplacian\-like𝐬\\mathbf\{s\}can remain well separated from the selection boundary, while ranking disagreements concentrate among channels near that boundary\. The Gaussian calculation does not model SiLU, the product of the two branches, trained\-weight dependencies, or proxy\-weight quantization, so it should be interpreted as a mechanism rather than a guarantee\. The observed 82\.04% exact–proxy selection overlap in Figure[3](https://arxiv.org/html/2607.27591#S2.F3)\(d\) verifies that this mechanism persists in the complete proxy\. Prox exploits this weaker ranking requirement and computes the retained values exactly, whereas TEAL propagates the approximate values themselves\.

### A\.3Empirical Proxy\-Cost Ratios

The allocation model usesα\\alphato approximate the cost of one INT4 proxy GEMV relative to an FP16 GEMV at the same input sparsity\. We measure this latency ratio across NVIDIA A6000, GeForce RTX 3090, and A100 GPUs for four Qwen3 scales and four input sparsity levels, as reported in Table[5](https://arxiv.org/html/2607.27591#A1.T5)\.

The measured ratios range from 0\.234 to 0\.351, with a pooled mean of 0\.285\. We use the single rounded valueα=1/3\\alpha=1/3, which falls within the observed range and avoids device\- or model\-specific allocation tuning\.

Table 5:Measured proxy\-cost ratioα\\alpha: latency of the INT4 proxy GEMV divided by that of the FP16 GEMV at the same input sparsity\. Values are reported across GPUs, Qwen3 scales, and input sparsities\.
### A\.4Stage\-Wise Compute Configurations

Table[6](https://arxiv.org/html/2607.27591#A1.T6)lists the Stage 1 and Stage 2 sparsities used to realize each target effective sparsityeein our experiments\. Applying the allocation rule withsref=0\.7s\_\{\\mathrm\{ref\}\}=0\.7andα=1/3\\alpha=1/3keepss2s\_\{2\}at 70% for the 50% and 60% operating points, withs1s\_\{1\}absorbing the remaining budget\. At the 40% and 70% operating points, the Stage 1 allocation implied bys2=srefs\_\{2\}=s\_\{\\mathrm\{ref\}\}falls outside the allowed range, sos1s\_\{1\}is clamped to 0% and 70%, respectively;s2s\_\{2\}is then adjusted to 62\.2% and 76\.7% so that each configuration still satisfies its targetee\.

Table 6:Stage\-wise allocations used for each target sparsity\. The configurations follow the proposed allocation rule withsref=0\.7s\_\{\\mathrm\{ref\}\}=0\.7,s1∈\[0,0\.7\]s\_\{1\}\\in\[0,0\.7\], andα=1/3\\alpha=1/3\.
### A\.5End\-to\-End Latency Analysis

Figure[8](https://arxiv.org/html/2607.27591#A1.F8)provides a detailed latency breakdown for Prox on Qwen3\-8B at 70% sparsity\. Figure[8](https://arxiv.org/html/2607.27591#A1.F8)\(a\) attributes 4\.72 ms/token to the fused attention, RMSNorm, and residual path and 5\.18 ms/token to the FFN, versus a 9\.88 ms/token Transformer\-block total\. The small aggregation difference arises because the regions are profiled independently\.

Figure[8](https://arxiv.org/html/2607.27591#A1.F8)\(b\) further decomposes the FFN: Stage 2 gate/up is the largest component at 2\.64 ms/token \(51\.0% of FFN time\), while Stage 1 and Stage 2 down take 1\.03 and 1\.47 ms/token, respectively\. The three kernel timings sum to 5\.13 ms/token, 0\.04 ms/token \(0\.8%\) below the aggregate FFN timing because they are independently profiled\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x8.png)Figure 8:Detailed latency breakdown of Prox on Qwen3\-8B with 70% effective FFN sparsity on an NVIDIA A6000 GPU\. Values are normalized per generated token over 36 Transformer layers\. \(a\) The Transformer\-block profile totals 9\.88 ms/token\. \(b\) The three FFN kernels are independently profiled, causing a 0\.04 ms/token difference from the aggregate FFN timing\. Attention, RMSNorm, and residual are reported jointly because Inductor fuses them into one kernel\.We evaluate end\-to\-end decoding throughput on NVIDIA A100, A6000, and GeForce RTX 3090 GPUs using six representative models, with unmeasured model–device configurations marked by dashes\. Table[7](https://arxiv.org/html/2607.27591#A1.T7)reports the dense baseline and the resulting throughput for Prox and TEAL at each target FFN sparsity\.

Table 7:Steady\-state single\-batch end\-to\-end decoding throughput on NVIDIA A100, A6000, and GeForce RTX 3090 GPUs\. Entries report tokens per second; dashes indicate unmeasured model–device configurations\.Across all three GPUs, Prox and TEAL deliver broadly comparable throughput over the evaluated models and sparsity levels, with neither method consistently dominating\.

### A\.6Threshold\-Based Selection and Token\-Adaptive Computation

Prox converts two continuous magnitude signals into binary masks: the FFN input𝐱\\mathbf\{x\}determines the proxy\-path input mask𝐦𝐱\\mathbf\{m\}\_\{\\mathbf\{x\}\}, while the proxy intermediate state𝐬~\\tilde\{\\mathbf\{s\}\}determines the final intermediate\-channel mask𝐦𝐬\\mathbf\{m\}\_\{\\mathbf\{s\}\}\. We use layer\-wise magnitude thresholds for both signals instead of selecting a fixed number of coordinates independently for each token\.

Per\-token top\-kkselection\.Given a salience signal𝐳\\mathbf\{z\}for one token, top\-kkselection retains the indices of itskklargest\-magnitude entries:

\[𝐦top​\-​k​\(𝐳\)\]i=𝟏​\[i∈TopKIndices​\(\|𝐳\|,k\)\]\.\[\\mathbf\{m\}^\{\\mathrm\{top\}\\mbox\{\-\}k\}\(\\mathbf\{z\}\)\]\_\{i\}=\\mathbf\{1\}\\\!\\left\[i\\in\\mathrm\{TopKIndices\}\(\|\\mathbf\{z\}\|,k\)\\right\]\.This rule strictly controls the number of retained coordinates for every token, but uses only the magnitude ranking within that token\. Consequently, two tokens receive the same compute budget even when their overall activation scales or importance distributions differ substantially\. Top\-kktherefore enforces a per\-token budget rather than a shared absolute importance criterion\.

Layer\-wise threshold selection\.For FFN layerℓ\\ell, a threshold rule instead retains every coordinate whose magnitude exceeds a fixed layer\-specific value:

\[𝐦τ​\(𝐳\)\]i=𝟏​\[\|zi\|≥τℓ\]\.\[\\mathbf\{m\}^\{\\tau\}\(\\mathbf\{z\}\)\]\_\{i\}=\\mathbf\{1\}\\\!\\left\[\|z\_\{i\}\|\\geq\\tau\_\{\\ell\}\\right\]\.Unlike top\-kk, this rule does not force all tokens to retain the same number of coordinates\. Tokens with fewer entries above the threshold use less computation, whereas tokens with more salient entries retain more coordinates\. Thresholding therefore controls the average compute budget while allowing that budget to vary with the token\-specific activation distribution\. It also avoids a runtime per\-token ranking operation\.

Measuring token\-adaptive computation\.For tokenttat layerℓ\\ell, Prox applies this rule separately to the input and proxy intermediate signals:

\[𝐦𝐱,ℓ,t\]i=𝟏​\[\|xℓ,t,i\|≥τ𝐱,ℓs1\],\[\\mathbf\{m\}\_\{\\mathbf\{x\},\\ell,t\}\]\_\{i\}=\\mathbf\{1\}\\\!\\left\[\|x\_\{\\ell,t,i\}\|\\geq\\tau\_\{\\mathbf\{x\},\\ell\}^\{s\_\{1\}\}\\right\],\[𝐦𝐬,ℓ,t\]j=𝟏​\[\|s~ℓ,t,j\|≥τ𝐬,ℓs2\]\.\[\\mathbf\{m\}\_\{\\mathbf\{s\},\\ell,t\}\]\_\{j\}=\\mathbf\{1\}\\\!\\left\[\|\\tilde\{s\}\_\{\\ell,t,j\}\|\\geq\\tau\_\{\\mathbf\{s\},\\ell\}^\{s\_\{2\}\}\\right\]\.The thresholdsτ𝐱,ℓs1\\tau\_\{\\mathbf\{x\},\\ell\}^\{s\_\{1\}\}andτ𝐬,ℓs2\\tau\_\{\\mathbf\{s\},\\ell\}^\{s\_\{2\}\}are calibrated to meet the target Stage 1 and Stage 2 sparsities on average, rather than enforcing those sparsities independently for every token\. We define the token\-wise sparsities induced by the two masks as

rℓ,t𝐱=1−‖𝐦𝐱,ℓ,t‖0dmodel,rℓ,t𝐬~=1−‖𝐦𝐬,ℓ,t‖0dff\.r\_\{\\ell,t\}^\{\\mathbf\{x\}\}=1\-\\frac\{\\\|\\mathbf\{m\}\_\{\\mathbf\{x\},\\ell,t\}\\\|\_\{0\}\}\{d\_\{\\mathrm\{model\}\}\},\\qquad r\_\{\\ell,t\}^\{\\tilde\{\\mathbf\{s\}\}\}=1\-\\frac\{\\\|\\mathbf\{m\}\_\{\\mathbf\{s\},\\ell,t\}\\\|\_\{0\}\}\{d\_\{\\mathrm\{ff\}\}\}\.We measure their variation at two scales\. For the within\-layer view, we compute the standard deviation and the 10th and 90th percentiles across tokens in each FFN layer, then average each statistic across layers\. For the across\-layer view, we first average each token’s sparsity over theLLFFN layers,

r¯t𝐱=1L​∑ℓ=1Lrℓ,t𝐱,r¯t𝐬~=1L​∑ℓ=1Lrℓ,t𝐬~,\\bar\{r\}\_\{t\}^\{\\mathbf\{x\}\}=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}r\_\{\\ell,t\}^\{\\mathbf\{x\}\},\\qquad\\bar\{r\}\_\{t\}^\{\\tilde\{\\mathbf\{s\}\}\}=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}r\_\{\\ell,t\}^\{\\tilde\{\\mathbf\{s\}\}\},and then compute the corresponding statistics across tokens\.

Table 8:Token\-wise sparsity variation under layer\-wise threshold selection\. Within\-layer statistics are computed across tokens per layer and then averaged across layers; across\-layer statistics are computed after averaging each token’s sparsity over all FFN layers\.Table[8](https://arxiv.org/html/2607.27591#A1.T8)shows substantial variation in the compute allocated to different tokens\. Within individual layers, the P10–P90 ranges span 17\.33 percentage points for the input mask and 17\.51 points for the proxy\-intermediate mask\. Even after averaging over all FFN layers, the corresponding ranges remain 12\.07 and 11\.96 points\. In contrast, a fixed per\-token top\-kkrule has zero token\-wise sparsity variation by construction\. These results confirm that threshold selection gives Prox a natural token\-adaptive computation mechanism instead of assigning every token the same retained\-channel count\.

Thresholding versus top\-kk\.Having verified that thresholding enables token\-adaptive computation, we directly compare the two selection rules on Qwen3\-8B and Qwen3\-14B at matched target effective sparsities\. Table[9](https://arxiv.org/html/2607.27591#A1.T9)reports the aggregate downstream scores\.

Table 9:Aggregate downstream scores using layer\-wise threshold selection and per\-token top\-kkat matched target effective sparsity\. Higher is better; bold marks the better selection rule in each setting\.Thresholding achieves a higher score in all eight settings and improves the average score by 1\.79 points over top\-kk\. On Qwen3\-8B, its gains range from 1\.03 to 2\.98 points across the four sparsity levels\. On Qwen3\-14B, thresholding is consistently better by 0\.60–1\.88 points\. Thus, preserving an absolute layer\-wise importance criterion consistently improves downstream quality across the evaluated settings at a matched average compute budget\. Having established the motivation and empirical benefits of thresholding, Appendix[A\.7](https://arxiv.org/html/2607.27591#A1.SS7)describes how the layer\-wise thresholds are calibrated and evaluates their runtime sparsity stability\.

### A\.7Threshold Calibration and Runtime Sparsity

Calibration data and scale\.Prox calibrates its fixed layer\-specific thresholds using generic instruction text from the Alpaca training split\. For each model, we tokenize the calibration corpus with its native tokenizer, partition the resulting tokens into contiguous sequences of length 2,048, and use the first 10 sequences for each target\-sparsity configuration\. Thus, each configuration is calibrated on 20,480 token positions\. We independently calibrate the ten models in our main evaluation—Qwen3\-1\.7B, Qwen3\-4B, Qwen3\-8B, Qwen3\-14B, Qwen3\.5\-4B, Qwen3\.5\-9B, Ministral\-3\.3B, Mistral\-7B, Llama\-3\-8B, and Gemma\-3\-12B—at target sparsities of 40%, 50%, 60%, and 70%, producing 40 threshold configurations in total\.

For each layer and each calibration signal, we maintain a reservoir of at most 200,000 activation samples and obtain the threshold directly from the corresponding empirical quantile\. We additionally save a 4,096\-bin activation histogram for each layer to support analysis and reproducibility\. Threshold values are determined separately for each layer, and no additional layer\-wise sparsity profile is imposed: within a configuration, all layers share the same targets1s\_\{1\}ands2s\_\{2\}, while their numerical thresholds are determined by their own activation distributions\. All evaluations use the saved calibration artifacts; thresholds are neither re\-estimated nor updated during inference\.

Sequential layer\-wise calibration\.A direct approach would calibrate every layer independently using activations collected from the dense model, but this creates a mismatch with sparse inference: sparsification in earlier layers changes the input distribution observed by later layers\. Prox therefore calibrates layers in network\-depth order so that each layer observes the effects of all previously finalized sparse layers\. Algorithm[1](https://arxiv.org/html/2607.27591#alg1)summarizes the procedure\.

Algorithm 1Sequential threshold calibration0:Calibration set

𝒟\\mathcal\{D\}and a model with

LLFFN layers

1:for

ℓ=1,…,L\\ell=1,\\ldots,Ldo

2:Run

𝒟\\mathcal\{D\}through layers

1,…,ℓ−11,\\ldots,\\ell\-1using their finalized sparse rules

3:Collect the current\-layer inputs

𝐱ℓ\\mathbf\{x\}\_\{\\ell\}
4:Choose

τ𝐱,ℓs1\\tau\_\{\\mathbf\{x\},\\ell\}^\{s\_\{1\}\}and construct

𝐦𝐱,ℓ\\mathbf\{m\}\_\{\\mathbf\{x\},\\ell\}
5:Compute

𝐬~ℓ\\tilde\{\\mathbf\{s\}\}\_\{\\ell\}using

𝐦𝐱,ℓ\\mathbf\{m\}\_\{\\mathbf\{x\},\\ell\}and the quantized proxy weights

6:Choose

τ𝐬,ℓs2\\tau\_\{\\mathbf\{s\},\\ell\}^\{s\_\{2\}\}and finalize the sparse rule for layer

ℓ\\ell
7:endfor

Specifically, when calibrating layerℓ\\ell, Prox first collects its FFN input𝐱ℓ\\mathbf\{x\}\_\{\\ell\}and determinesτ𝐱,ℓs1\\tau\_\{\\mathbf\{x\},\\ell\}^\{s\_\{1\}\}\. It then applies the resulting input mask and the model\-specific symmetric\-INT4 proxy weights to obtain

𝐬~ℓ=W~up,ℓ​\(𝐦𝐱,ℓ⊙𝐱ℓ\)⊙SiLU​\(W~gate,ℓ​\(𝐦𝐱,ℓ⊙𝐱ℓ\)\),\\tilde\{\\mathbf\{s\}\}\_\{\\ell\}=\\widetilde\{W\}\_\{\\mathrm\{up\},\\ell\}\(\\mathbf\{m\}\_\{\\mathbf\{x\},\\ell\}\\odot\\mathbf\{x\}\_\{\\ell\}\)\\odot\\mathrm\{SiLU\}\\\!\\left\(\\widetilde\{W\}\_\{\\mathrm\{gate\},\\ell\}\(\\mathbf\{m\}\_\{\\mathbf\{x\},\\ell\}\\odot\\mathbf\{x\}\_\{\\ell\}\)\\right\),from which it determinesτ𝐬,ℓs2\\tau\_\{\\mathbf\{s\},\\ell\}^\{s\_\{2\}\}\. All preceding FFNs use their finalized Stage 1 and Stage 2 thresholds, whereas the current layer remains dense while its input activations are collected\. Consequently, each layer’s calibration distribution incorporates the error propagation and distribution shift induced by earlier sparse layers and more closely matches runtime sparse inference\.

Runtime sparsity\.Prox uses fixed magnitude thresholds rather than per\-token top\-kkselection; consequently, the realized mask sparsities can differ from their calibration targets under a new input distribution\. Lets^1\\widehat\{s\}\_\{1\}ands^2\\widehat\{s\}\_\{2\}denote the Stage 1 and Stage 2 sparsities measured by accumulating runtime mask counts\. Consistent with the cost model in Sec\.[4\.2](https://arxiv.org/html/2607.27591#S4.SS2), the measured runtime FFN sparsity is

e^=s^2−2​α​\(1−s^1\)3,α=13\.\\widehat\{e\}=\\widehat\{s\}\_\{2\}\-\\frac\{2\\alpha\(1\-\\widehat\{s\}\_\{1\}\)\}\{3\},\\qquad\\alpha=\\frac\{1\}\{3\}\.This quantity includes both the sparse exact FFN computation and the additional INT4 proxy cost\.

We measuree^\\widehat\{e\}using 100 examples from each of five task datasets: MMLU, HellaSwag, ARC\-Easy, ARC\-Challenge, and PIQA\. For every model and target configuration, we aggregate the Stage 1 and Stage 2 mask counts across all five datasets before applying the cost model\. Table[10](https://arxiv.org/html/2607.27591#A1.T10)reports the resulting mean over the ten evaluated models\.

Target sparsityeeMean measured sparsitye^\\widehat\{e\}40%38\.71%50%48\.74%60%58\.44%70%68\.33%Table 10:Target and runtime FFN sparsities for Prox\. Runtime values aggregate five task datasets and are averaged over ten models, with 100 examples evaluated per dataset\.Across all four operating points, the mean measured sparsity is slightly below the target, with gaps of 1\.26–1\.67 percentage points and no sharp increase at higher target sparsity\. These results indicate that static thresholds calibrated on generic instruction text preserve the intended compute level reasonably well across the evaluated models and task datasets\.

### A\.8Perplexity Results

For this evaluation, we use the same 128 randomly sampled sequences from the WikiText validation set\(Merityet al\.[2016](https://arxiv.org/html/2607.27591#bib.bib46)\)for all models, with a 2,048\-token context and a 512\-token evaluation window\. We sparsify the full sequence so that the hidden states scored for perplexity are computed under sparsity\. Because perplexity is computed over each model’s native token space, Gemma\-3\-12B’s substantially larger vocabulary and different tokenization granularity make its absolute perplexity values not directly comparable with those of the other model families\.

Qwen3Qwen3\.5MinistralMistralLlama3Gemma3VariantSparsity1\.7B4B8B14B4B9B3\.3B7B8B12BDense014\.711\.38\.27\.28\.07\.48\.54\.97\.17\.4CATS40%29\.815\.19\.17\.810\.48\.814\.86\.88\.634\.0COUNTDOWN40%22\.014\.69\.88\.29\.58\.311\.45\.38\.931\.1TEAL40%14\.311\.78\.37\.38\.37\.58\.95\.07\.3807\.0Prox40%14\.511\.58\.37\.28\.37\.68\.95\.07\.536\.0CATS50%98\.361\.615\.111\.615\.812\.295\.335\.517\.4221\.8COUNTDOWN50%47\.720\.812\.710\.412\.510\.616\.96\.412\.6254\.7TEAL50%14\.412\.48\.77\.58\.67\.79\.45\.27\.8898\.8Prox50%14\.711\.88\.47\.38\.57\.69\.35\.17\.8185\.0CATS60%1\.7×1031\.7\{\\times\}10^\{3\}3\.6×1043\.6\{\\times\}10^\{4\}1\.3×1031\.3\{\\times\}10^\{3\}521\.5121\.756\.79\.7×1049\.7\{\\times\}10^\{4\}7\.5×1047\.5\{\\times\}10^\{4\}2\.3×1052\.3\{\\times\}10^\{5\}1\.6×1041\.6\{\\times\}10^\{4\}COUNTDOWN60%1\.6×1031\.6\{\\times\}10^\{3\}56\.534\.121\.936\.624\.464\.613\.540\.2712\.2TEAL60%17\.614\.39\.68\.09\.68\.211\.05\.98\.91\.2×1031\.2\{\\times\}10^\{3\}Prox60%15\.012\.28\.67\.48\.67\.89\.75\.38\.2946\.4TEAL70%48\.824\.516\.110\.013\.210\.316\.88\.613\.52\.9×1032\.9\{\\times\}10^\{3\}Prox70%15\.713\.99\.47\.89\.48\.211\.65\.89\.51\.2×𝟏𝟎𝟑\\mathbf\{1\.2\{\\times\}10^\{3\}\}Table 11:WikiText perplexity across model families and sparsity levels\. Lower scores are better\.Table[11](https://arxiv.org/html/2607.27591#A1.T11)reports WikiText perplexity across model families and sparsity levels\. At 40% sparsity, Prox remains competitive with TEAL\. From 50% sparsity onward, Prox matches or outperforms TEAL on most model families; at 60% and 70%, it achieves lower perplexity on at least nine of the ten evaluated models, with the gap widening markedly at 70%\. CATS and COUNTDOWN degrade earlier, frequently producing large perplexity increases at 50% and 60% sparsity\.

Despite its large perplexity increase under sparsification, Gemma\-3\-12B retains strong downstream accuracy\. Consistent with our observation,Szatkowskiet al\.\([2026](https://arxiv.org/html/2607.27591#bib.bib45)\)report that Gemma\-3\-12B retains approximately 99% of its dense average downstream accuracy at 78\.77% activation sparsity\.

### A\.9Sensitivity to the Fixed Stage 1 Sparsity

Table[12](https://arxiv.org/html/2607.27591#A1.T12)compares Prox with fixed Stage 1 sparsities from 0\.5 to 0\.9 on Qwen3\-8B to test the sensitivity of A2’ss1=0\.8s\_\{1\}=0\.8setting\. For each fixeds1s\_\{1\}and target effective sparsityee, we adjusts2s\_\{2\}using the cost model in Sec\.[4\.2](https://arxiv.org/html/2607.27591#S4.SS2)to match the same compute budget and report the aggregate downstream score under the main six\-task protocol\.

Table 12:Aggregate downstream scores on Qwen3\-8B for Prox and fixed Stage 1 sparsities\. Higher is better; bold marks the best fixed allocation for each target effective sparsity\.The best fixed allocation changes with the target:s1=0\.6s\_\{1\}=0\.6performs best at 40% effective sparsity,s1=0\.8s\_\{1\}=0\.8at 50%, ands1=0\.7s\_\{1\}=0\.7at 60–70%\. Prox matches or outperforms every fixed setting at all four targets; even the strongest single fixed strategy,s1=0\.7s\_\{1\}=0\.7, has a lower average across the four targets \(72\.1 versus 73\.1\)\. At 70%, overly aggressive Stage 1 sparsity also reduces the aggregate downstream score from 68\.6 ats1=0\.7s\_\{1\}=0\.7to 58\.4 ats1=0\.9s\_\{1\}=0\.9\. Thus, no fixed Stage 1 allocation is uniformly best; target\-aware allocation avoids per\-target downstream sweeps\.

### A\.10Sensitivity to the Stage 1 Sparsity Cap

Table[13](https://arxiv.org/html/2607.27591#A1.T13)sweeps the maximum Stage 1 sparsitys1,maxs\_\{1,\\max\}across Qwen3\-4B, Qwen3\-8B, and Qwen3\-14B\. For every setting, we clamps1s\_\{1\}using the corresponding cap and adjusts2s\_\{2\}according to the cost model in Sec\.[4\.2](https://arxiv.org/html/2607.27591#S4.SS2)to preserve the target effective sparsity\.

The cap affects only the high\-sparsity operating points: at 40% and 50% effective sparsity, the unconstrained allocation is feasible for every tested cap, while at 60% only thes1,max=50%s\_\{1,\\max\}=50\\%setting changes the allocation\. At 70% effective sparsity, all tested caps are active, withs2s\_\{2\}adjusted to satisfy the same effective\-sparsity budget\.

Table 13:Sensitivity to the maximum Stage 1 sparsitys1,maxs\_\{1,\\max\}across Qwen3 models\. Entries are aggregate downstream scores; each setting adjusts Stage 2 sparsitys2s\_\{2\}to match the listed effective sparsity\. Higher is better; bold marks the best 70% setting for each model\.Thes1,max=70%s\_\{1,\\max\}=70\\%cap achieves the highest aggregate downstream score at 70% effective sparsity for all three model scales\. Increasing the cap to 90% reduces the scores of Qwen3\-4B and Qwen3\-8B from 58\.7 to 46\.8 and from 68\.6 to 57\.9, respectively\. We therefore uses1,max=70%s\_\{1,\\max\}=70\\%as a shared cap across models without per\-model downstream tuning\.

### A\.11Detailed Analysis with TEAL

TEAL’s block\-wise greedy calibration assigns sparsity separately to the up, gate, and down projections while preserving the target effective sparsity\. Starting from dense computation, it repeatedly increases the sparsity of the projection whose candidate update incurs the smallest block\-outputℓ2\\ell\_\{2\}error on calibration activations, until the target block\-level budget is reached\. Figure[9](https://arxiv.org/html/2607.27591#A1.F9)shows the resulting layer\-wise allocation for Qwen3\-8B at 60% effective sparsity\.

![Refer to caption](https://arxiv.org/html/2607.27591v1/x9.png)Figure 9:Layer\-wise sparsity assigned by TEAL’s greedy calibration on Qwen3\-8B at 60% effective sparsity\. At every layer, the up\-, gate\-, and down\-projection sparsities average to 60%\.The allocation is both projection\- and depth\-dependent\. In the first four layers, the down\-projection sparsity reaches 75–90%, compared with 50–52\.5% for the up projection and 40–52\.5% for the gate projection\. Beyond these early layers, the three rates become closer, but the down projection generally remains the most sparse and the up projection the least sparse\. Thus, TEAL’s calibrated 60% operating point does not correspond to applying a uniform sparsity rate to every projection and layer; it instead reveals that the down projection tolerates more approximation, whereas the up projection is generally more sensitive\.

This fine\-grained search makes TEAL a stronger reference than a uniformly allocated baseline\. In contrast, Prox uses a simple target\-aware heuristic motivated by the oracle\-𝐬\\mathbf\{s\}results: it treats 70% Stage 2 channel sparsity as a quality\-preserving anchor and assigns the remaining budget to Stage 1, using the same allocation across layers\. Despite TEAL’s greater allocation flexibility, Prox scores 73\.3 versus TEAL’s 71\.4 on Qwen3\-8B at the matched 60% operating point and outperforms TEAL on all ten evaluated models at both 60% and 70% sparsity in Table[1](https://arxiv.org/html/2607.27591#S5.T1)\. Together with the exact\-computation ablation in Table[3](https://arxiv.org/html/2607.27591#S5.T3), this result supports the central two\-stage design: the proxy is used only to select channels, while the retained values are computed with the original weights, preventing proxy\-value errors from propagating directly through the final FFN computation\.

The branch\-wise allocation provides a further insight into proxy construction\. The lower sparsity generally assigned to the up projection suggests that preserving its computation is more important under TEAL’s block\-error objective\. This trend is consistent with the comparison between CATS and COUNTDOWN: CATS densely computes the gate\-branch signal𝐡\\mathbf\{h\}for channel selection, whereas COUNTDOWN uses the densely computed up\-branch signal𝐮\\mathbf\{u\}; across the 30 matched model–sparsity settings at 40–60% in Table[1](https://arxiv.org/html/2607.27591#S5.T1), COUNTDOWN always achieves a higher aggregate score than CATS\. Although projection sensitivity and selection\-signal quality are not identical quantities, these results jointly suggest that𝐮\\mathbf\{u\}provides a more informative approximation to the joint intermediate signal𝐬\\mathbf\{s\}than𝐡\\mathbf\{h\}alone\.

Prox currently uses the same sparsity allocation across all layers\. A promising extension is to follow TEAL’s layer\-wise allocation and use different Stage 1 sparsities when constructing the𝐮\\mathbf\{u\}\- and𝐡\\mathbf\{h\}\-branch proxies, which may improve the accuracy of proxy\-based channel selection\.