Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models

# Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
Source: [https://arxiv.org/html/2607.08991](https://arxiv.org/html/2607.08991)
Youngmin Yi Sogang University ymyi@sogang\.ac\.krHoeseok Yang Santa Clara University hoeseok\.yang@scu\.edu

###### Abstract

Efficient inference in Large Language Models \(LLMs\) requires deciding where computation can be reduced while preserving model quality\. We study this problem through multilayer perceptron \(MLP\) activation sparsification and token\-level conditional routing\. We first propose Sensitivity\-Aware Thresholding for Sparsity \(SATS\), a threshold calibration method to choose layerwise gate thresholds using a local MLP output sensitivity proxy rather than calibrating thresholds directly from activation percentiles\. While SATS retains the existing mechanism of sparsifying MLP activations by thresholding gate activations, it replaces percentile\-based calibration with a sensitivity\-aware selection rule\. We then introduce a lightweight token routing framework that dynamically selects between a base path and a modified path on a per\-token basis, rather than applying the modified computation uniformly to all tokens\. We evaluate both methods on multiple recent open\-weight LLMs\. Our results show that SATS improves over the threshold\-based sparsification baseline at matched actual sparsity and that token routing yields a more favorable quality\-throughput trade\-off than static activation modification baselines\. Overall, our results suggest that improved threshold calibration and token routing can improve the quality\-throughput trade\-off in LLMs\.

## 1Introduction

Large Language Models \(LLMs\) are widely used across various tasks, but their inference cost remains a major bottleneck\. A significant portion of this cost comes from the multilayer perceptron \(MLP\) blocks inside each transformer layer\. Therefore, optimizing MLP blocks can speed up on\-device LLM inference\. Recent works\(Leeet al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib2); Zhanget al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib3)\)have shown that these activations contain significant sparsity that can be exploited by thresholding MLP gate activations, which can speed up LLM inference without modifying the overall transformer architecture\. However, deciding which activations to remove and when to apply the modified computation remains a challenge\.

A common strategy to address this challenge is to calibrate one activation threshold per layer from activation statistics from a calibration dataset\. CATS\(Leeet al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib2)\)uses a percentile\-based approach to choose these layerwise thresholds\. But percentile\-based calibration focuses primarily on how many activations are removed, instead of examining how much damage a given threshold causes to the resultant MLP output\. Two adjacent thresholds in the activation layer can have drastically different amounts of information lost in the final output\. Even more broadly, activation modification is applied uniformly across all tokens for a generation prompt, even though the impact of the modification is not necessarily uniform across tokens\.

In this work, we study sparsification of the MLP layer in LLMs at two levels\. First, we proposeSensitivity\-aware Thresholding for Sparsity \(SATS\), a threshold calibration method that replaces percentile\-based threshold selection with a sensitivity\-aware method based on layerwise MLP output distortion\. SATS chooses thresholds to satisfy a local layer sensitivity budget, and the layerwise final operating points are chosen to satisfy a global target sparsity\. Secondly, we introduce a lightweighttoken routingmethod that decides whether to use the dense MLP path or the sparse path \(that uses the activation thresholding\) for each token\. The routing method only requires the token identity, which allows it to be simplified into a look\-up table with negligible effect on the inference speed\.

We evaluate both of the methods on llama 3\.1 8B\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib4)\)and Qwen 3 8B\(Yanget al\.,[2025](https://arxiv.org/html/2607.08991#bib.bib5)\)\. Across both model families, SATS provides improved quality compared to percentile\-based thresholding at matched realized sparsity\. We then show that token routing further improves the quality\-throughput trade\-off relative to static sparse or dense model execution\. Our main contributions are as follows:

- •We propose SATS, a sensitivity\-aware threshold calibration method for activation sparsification that preserves the existing runtime activation threshold masking method
- •We show that SATS improves over percentile\-based threshold calibration at matched realized sparsity on two recent open\-weight LLM families\.
- •We introduce a token\-identity\-based routing method that selects between the dense and sparse computation paths at inference with minimal routing overhead
- •We show that token routing improves the quality\-throughput trade\-off relative to the static sparse execution while remaining faster than dense inference\.

## 2Background

We focus on the transformer\-based LLMs that use Gated MLP blocks, such as llama\(Touvronet al\.,[2023](https://arxiv.org/html/2607.08991#bib.bib16)\)and Qwen\(Baiet al\.,[2023](https://arxiv.org/html/2607.08991#bib.bib17)\)families\. Given an input vectorxx, the gated MLP block can be represented using the following equation:

Gated​\-​MLP​\(x\):=\(Activation​\(x​Wgate\)∗\(x​Wup\)\)​Wdown\\mathrm\{Gated\\text\{\-\}MLP\}\(x\):=\(\\mathrm\{Activation\}\(xW\_\{\\mathrm\{gate\}\}\)\*\(xW\_\{\\mathrm\{up\}\}\)\)W\_\{\\mathrm\{down\}\}\(1\)
whereWg​a​t​eW\_\{gate\},Wu​pW\_\{up\}andWd​o​w​nW\_\{down\}represent the weights in the MLP layer\. While the activation in the MLP layer can vary, SiLU\(Elfwinget al\.,[2018](https://arxiv.org/html/2607.08991#bib.bib18)\)is a widely adopted activation, which can be represented using the following equation:

SiLU:=x∗sigmoid​\(x\)=x1\+e−x\\mathrm\{SiLU\}:=x\*\\mathrm\{sigmoid\}\(x\)=\\frac\{x\}\{1\+e^\{\-x\}\}\(2\)
CATS\(Leeet al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib2)\)seeks to utilize activation sparsity in the MLP layers by applying specific thresholds to sparsify the values in the activation\. Given a sparsity levelkk\(e\.g\. 50%\) as input, the activations over a random subset of the training data, limited to 500 data points, are calculated\. This is used to compute the cutoff threshold as the kth percentile of the resulting values\.

t:=min⁡\{t′:F​\(t′\)≥k\}t:=\\min\\\{t^\{\\prime\}:F\(t^\{\\prime\}\)\\geq k\\\}\(3\)
After calculating the threshold, CATS applies the following activation function on top of the existing activation function in the MLP layer\.

CATSt​\(xj\):=\{xj,if​\|xj\|≥t0,if​\|xj\|<t\\mathrm\{CATS\}\_\{t\}\(x\_\{j\}\):=\\begin\{cases\}x\_\{j\},&\\mathrm\{if\}\\ \|x\_\{j\}\|\\geq t\\\\ 0,&\\mathrm\{if\}\\ \|x\_\{j\}\|<t\\end\{cases\}\(4\)
For the inputS​i​L​U​\(x​Wg​a​t​e\)SiLU\(xW\_\{gate\}\), this becomes:

CATSt​\(SiLU​\(x​Wg​a​t​e\)\):=\{SiLU​\(x​Wg​a​t​e\),if​\|SiLU​\(x​Wg​a​t​e\)\|≥t0,if​\|SiLU​\(x​Wg​a​t​e\)\|<t\\mathrm\{CATS\}\_\{t\}\(\\mathrm\{SiLU\}\(xW\_\{gate\}\)\):=\\begin\{cases\}\\mathrm\{SiLU\}\(xW\_\{gate\}\),&\\mathrm\{if\}\\ \|\\mathrm\{SiLU\}\(xW\_\{gate\}\)\|\\geq t\\\\ 0,&\\mathrm\{if\}\\ \|\\mathrm\{SiLU\}\(xW\_\{gate\}\)\|<t\\end\{cases\}\(5\)
We extend the CATS framework for more damage\-aware thresholding\. CATS controls the global sparsity level, but does not explicitly account for the damage caused by removing specific neurons\. Our proposed Sensitivity\-Aware Thresholding for Sparsity \(SATS\) framework replaces percentile\-based sparsity calibration with a sensitivity\-aware threshold selection, while preserving the hardware\-efficient CATS runtime masking form\.

## 3Method

### 3\.1SATS: Sensitivity\-Aware Thresholding for Sparsity

To quantify the impact of zeroing out neurons under a specific threshold, we define a local MLP distortion metric inspired by the Cumulative Errors of Tail Truncation \(CETT\) introduced inReLU2\\mathrm\{ReLU\}^\{2\}\(Zhanget al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib3)\)\. Their work demonstrated that a larger MLP truncation error is associated with worse model performance, which motivates our method of using local distortion as a calibration signal\. UnlikeReLU2\\mathrm\{ReLU\}^\{2\}, which studies sparsification by thresholding MLP outputs directly, we use this signal only to calibrate the per\-layer gate thresholds for CATS\-style activation masking\.

![Refer to caption](https://arxiv.org/html/2607.08991v1/x1.png)Figure 1:\(a\) Layerwise distortion under different gate thresholding for llama 3\.2 1B\. For representative MLP layers, we sweep candidate gate thresholds and measure the layer sparsity and local MLP output distortion on the calibration split\. Similar sparsity levels can induce different local distortions across layers \(b\) Average local distortion vs downstream average accuracy for a SATS budget sweep on llama 3\.2 1B\. Each point corresponds to a shared layerwise budgetbb\(with its realized sparsity level annotated with the points\)The CATS method selects thresholds from the empirical distribution of post\-gate activations, which directly models sparsity but does not model the distortion caused by thresholding\. Applying adjacent thresholds on the gate layer can contribute differently to the final MLP output depending on the up\-projection value and the down\-projection weights\. As shown in Figure[1](https://arxiv.org/html/2607.08991#S3.F1)\(a\), similar layer sparsity can introduce substantially different local distortion across layers\. Since the sensitivity of the layers to distortion \(distortion measured using Equation[6](https://arxiv.org/html/2607.08991#S3.E6)\) due to thresholding can vary widely, we replace the percentile\-based threshold calibration with a sensitivity\-aware calibration method while preserving the CATS runtime masking method\.

For a candidate thresholdtt, letyℓ​\(x\)y\_\{\\ell\}\(x\)denote the dense MLP output at layerℓ\\ell, and lety~ℓ,t​\(x\)\\tilde\{y\}\_\{\\ell,t\}\(x\)denote the thresholded MLP output obtained after removing values whose post\-gate activation magnitude falls belowtt\. We define the local thresholding error as:

Errℓ​\(t\)=𝔼x​\[‖yℓ​\(x\)−y~ℓ,t​\(x\)‖2‖yℓ​\(x\)‖2\+ϵ\]\\mathrm\{Err\}\_\{\\ell\}\(t\)=\\mathbb\{E\}\_\{x\}\\left\[\\frac\{\|\|y\_\{\\ell\}\(x\)\-\\tilde\{y\}\_\{\\ell,t\}\(x\)\|\|\_\{2\}\}\{\|\|y\_\{\\ell\}\(x\)\|\|\_\{2\}\+\\epsilon\}\\right\]\(6\)
whereϵ\\epsilonis a small constant for numerical stability\. This score measures how much thresholding at a specificttdistorts the MLP output at layerℓ\\ell\.

As in CATS, we first build a candidate threshold set𝒯\\mathcal\{T\}from a calibration dataset using the post\-gate activation magnitudes\. This maintains the inference\-time gate masking method from CATS, however the threshold selection method is modified\. For each candidate threshold, we estimate its local error scoreErrℓ​\(t\)\\mathrm\{Err\}\_\{\\ell\}\(t\)on a calibration dataset\. We then introduce a layer\-wise error budgetbband for each layerℓ\\ell, we select the value of the largest thresholdtℓt\_\{\\ell\}whose local error does not exceed that budget\.

tℓ⋆​\(b\)=max⁡\{t∈𝒯ℓ:Errℓ​\(t\)≤b\}t\_\{\\ell\}^\{\\star\}\(b\)=\\max\\\{t\\in\\mathcal\{T\}\_\{\\ell\}:\\mathrm\{Err\}\_\{\\ell\}\(t\)\\leq b\\\}\(7\)
For any fixed budgetbb, applying this rule independently across all layers yields a model\-wide threshold assignment\.

T⋆​\(b\)=\[t1⋆​\(b\),…,tL⋆​\(b\)\]T^\{\\star\}\(b\)=\[t\_\{1\}^\{\\star\}\(b\),\\dots,t\_\{L\}^\{\\star\}\(b\)\]\(8\)
This defines one sparse model for each choice ofbb\. Larger budgets permit more aggressive thresholding and generally lead to higher sparsity\. Therefore, the choice ofbbis particularly important here\. Whilebbis selected to model the local MLP distortion, it remains a useful signal in modeling the end\-to\-end LLM performance\. Figure[1](https://arxiv.org/html/2607.08991#S3.F1)\(b\) demonstrates that local MLP distortion tracks end\-to\-end model performance consistently, which shows that it is informative enough to guide threshold calibration\.

We treatbbas a calibration parameter used to reach a desired sparsity level\. Specifically, for a target sparsity levelstars\_\{\\mathrm\{tar\}\}, each candidate budgetbbgenerates a threshold assignmentT⋆​\(b\)T^\{\\star\}\(b\), which yields a realized average sparsity S\(b\), defined as:

S​\(b\)=1L​∑ℓ=1Lspℓ​\(tℓ⋆​\(b\)\)S\(b\)=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}\\mathrm\{sp\}\_\{\\ell\}\(t\_\{\\ell\}^\{\\star\}\(b\)\)\(9\)wherespℓ​\(t\)\\mathrm\{sp\}\_\{\\ell\}\(t\)denotes the expected sparsity at layerℓ\\ellunder a given thresholdtt\. We then select the budget whose realized sparsity is closest to the target:

b⋆=arg⁡minb⁡\|S​\(b\)−star\|b^\{\\star\}=\\arg\\min\_\{b\}\|S\(b\)\-s\_\{\\mathrm\{tar\}\}\|\(10\)This makes the roles ofbbandstars\_\{\\mathrm\{tar\}\}distinct\. Budgetbbcontrols which threshold is admissible under the local error criterion, and target sparsitystars\_\{\\mathrm\{tar\}\}is used only in an outer calibration loop to choose among the resulting sparse models\.

In practice, we implement this calibration with a coarse\-to\-fine search\. We first sweep a coarse set of budget values and identify the region whose realized sparsity is closest to our target sparsity\. We then refine locally around that region by searching nearby threshold candidates more densely for each layer\. Conceptually, this approach recasts threshold calibration as a target\-sparsity search over a sensitivity\-constrained threshold selection\.

### 3\.2Token Routing

![Refer to caption](https://arxiv.org/html/2607.08991v1/x2.png)Figure 2:Overview of the proposed token routing method\. For each token,did\_\{i\}is computed on a calibration corpus, and its average value is stored in a lookup table\. During inference, the router uses this lookup table to pass the token to either the base path or the sparse path, depending onτ\\tau\. The base path and the sparse path use the same model weights, and only the MLP layer activation is adjusted\.The activation thresholding methods discussed so far produce a single modified sparse model, which is then applied uniformly whenever the sparse configuration is used\. While this static design is simple, it also makes the same sparse computation path for every token\. Token routing asks whether this decision can be made more selectively\. Instead of applying the sparse path uniformly, we allow each token to choose between the base model path, which preserves original model performance, and a sparse path, which applies activation thresholding for faster inference\.

Considering practical model deployment scenarios, we only use the current token identity for the routing decision\. This keeps the routing overhead minimal, since deployment requires only a lookup into a precomputed table\. Under this constraint, the main goal is to assign each token a score that reflects how safe it is to use the sparse path on average\.

To construct this score, we use a calibration corpus of tokenized sequences\. For each token occurrenceiiin that corpus, we run both the base path and the sparse path on the same token and record the next\-token cross\-entropy loss for that position\. It is important to note that these losses are not intrinsic properties of the token type\. Rather, they are measured from actual occurrences in the calibration corpus and depend on the surrounding sequence context\. Routing calibration, therefore, builds a statistical estimate of how risky the sparse path would be for a given token\.

Letxix\_\{i\}denote the identity of the current token occurrence, and letℓbase\\ell\_\{\\mathrm\{base\}\}andℓsparse\\ell\_\{\\mathrm\{sparse\}\}denote the per\-token cross\-entropy losses of the base path and sparse path, respectively\. For the main routing target, we use the excess damage of the sparse path relative to the base path:

di=max⁡\(0,ℓsparse−ℓbase\)d\_\{i\}=\\max\(0,\\ell\_\{\\mathrm\{sparse\}\}\-\\ell\_\{\\mathrm\{base\}\}\)\(11\)This value is zero when the sparse path is no worse than the base path at occurrenceii, and positive when the sparse path has higher loss\. Therefore, this value approximates how harmful routing that token through the sparse path would be\.

Since the routing mechanism only observes the current token identityxix\_\{i\}, it cannot predict the exact loss difference for each occurrence\. Instead, it estimates a statistical average of this excess damage based on the calibration dataset:

s​\(x\)≈𝔼​\[di∣xi=x\]s\(x\)\\approx\\mathbb\{E\}\[d\_\{i\}\\mid x\_\{i\}=x\]\(12\)Lower scores indicate tokens that are, on average, safer to route through the sparse path\.

We store these estimates in a lookup table over the vocabulary\. Letℐx=\{i:xi=x\}\\mathcal\{I\}\_\{x\}=\\\{i:x\_\{i\}=x\\\}be the set of positions at which tokenxxappears in the routing corpus\. The token score is estimated as

route​\(xi;τ\)=\{sparse pathif​s^​\(xi\)<τ,base pathotherwise\\mathrm\{route\}\(x\_\{i\};\\tau\)=\\begin\{cases\}\\text\{sparse path\}&\\text\{if \}\\hat\{s\}\(x\_\{i\}\)<\\tau,\\\\ \\text\{base path\}&\\text\{otherwise\}\\end\{cases\}\(13\)For our routing method, a token sent to the sparse path uses the activation\-thresholded MLP computation across transformer blocks, while tokens sent to the base path use the base MLP blocks\. The thresholdτ\\tauis used to control the routed\-token fraction, and in turn, the quality\-throughput trade\-off\.

As in SATS, calibration and operating\-point selection are treated separately\. We first estimate the token score table on a routing dataset, then calibrate the routing threshold on a held\-out split\. For a target routed fractionrtarr\_\{\\mathrm\{tar\}\}, each candidate thresholdτ\\tauinduces a realized routed fractionR​\(τ\)R\(\\tau\)on the calibration split, and we select

τ⋆=arg⁡minτ⁡\|R​\(τ\)−rtar\|\\tau^\{\\star\}=\\arg\\min\_\{\\tau\}\|R\(\\tau\)\-r\_\{\\mathrm\{tar\}\}\|\(14\)This produces different routing policies at different operating points\. In our experiments, we report the performance of 25%, 50%, and 75% target routing settings to validate the effectiveness of our approach\. These routing settings are calibration operating points obtained by sweeping routing thresholds on a held\-out calibration dataset and selecting the threshold whose realized routed fraction is closest to the corresponding target \(e\.g\., 25%, 50%, or 75% target sparsity\)\.

## 4Experiments

Base Models:We use the llama 3\.1 8B\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib4)\)and Qwen 3 8B\(Yanget al\.,[2025](https://arxiv.org/html/2607.08991#bib.bib5)\)models to demonstrate the effectiveness of our approaches\.

Sparsity Levels:We evaluate the performance of CATS at 30% and 50% targeted sparsity levels, since at higher sparsity levels \(e\.g\. 70% or 90%\), too much information is removed, which makes them impractical for deployment\. For proper comparison, we find the equivalent operating point that yields a similar realized sparsity level for SATS and compare the performance with CATS\.

For token routing experiments, we pick SATS matched to CATS 50% for the two base models and observe their performance\. Practically, at roughly 30% sparsity level, both the accuracy loss and speedup gain are far too small to benefit from the quality\-throughput trade\-off offered by token routing, so we only demonstrate the effectiveness of the method at SATS matched to CATS 50% sparsity\.

For the final SATS configuration used in our experiments, we apply SATS to all MLP layers except the MLP in the final layer, where we retain the corresponding CATS threshold\. Empirically, we found that thresholding this MLP layer had an outsized impact on text generation performance, and applying SATS there could disproportionately degrade language\-modeling quality even when earlier layers might benefit from SATS\. We therefore keep the CATS threshold in the final MLP layer as a pragmatic guardrail, while using SATS on the remaining layers to reach an equivalent CATS sparsity and demonstrate the effectiveness of our approach\.

Metrics: We report the average accuracy over seven downstream tasks and perplexity on the Wiki\-Text2\(Merityet al\.,[2016](https://arxiv.org/html/2607.08991#bib.bib6)\)test dataset and a held\-out test set from the RefinedWeb\(Penedoet al\.,[2023](https://arxiv.org/html/2607.08991#bib.bib7)\)dataset\. For throughput, we follow CATS and report the FP32 end\-to\-end throughput\. We further conducted experiments to observe the variation in performance and latency between FP32 \(full precision\) and BF16 \(half\-precision\)\.

Datasets:For downstream task evaluation, we use PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2607.08991#bib.bib8)\), OpenBookQA\(Mihaylovet al\.,[2018](https://arxiv.org/html/2607.08991#bib.bib9)\), ARC\-Easy\(Clarket al\.,[2018](https://arxiv.org/html/2607.08991#bib.bib10)\), Winogrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2607.08991#bib.bib11)\), Hellaswag\(Zellerset al\.,[2019](https://arxiv.org/html/2607.08991#bib.bib12)\), SciQ\(Welblet al\.,[2017](https://arxiv.org/html/2607.08991#bib.bib13)\)and BoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2607.08991#bib.bib19)\)from LM Eval Harness\(Gaoet al\.,[2023](https://arxiv.org/html/2607.08991#bib.bib14)\)\. For fine\-tuning base models, we used a small subset of the RefinedWeb dataset\(Penedoet al\.,[2023](https://arxiv.org/html/2607.08991#bib.bib7)\)\. Language modeling quality is measured with perplexity on a held\-out split of RefinedWeb and Wiki\-Text2\(Merityet al\.,[2016](https://arxiv.org/html/2607.08991#bib.bib6)\)test set\.

Computational Environment: For evaluating the quality and end\-to\-end throughput of the models, we utilized NVIDIA A100 40 GB\. For fine\-tuning base models using LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.08991#bib.bib15)\), we used NVIDIA RTX Pro 6000 96 GB\. During inference, we used the transformers v4\.57\.6 Huggingface library, PyTorch v2\.8, and CUDA v12\.8\. We used Triton for our GPU kernels, while torch\.compile was used for evaluating end\-to\-end inference speedup\.

### 4\.1Comparison of SATS and CATS

Table 1:Downstream task performance of base models, CATS\-based models with 30% and 50% targeted sparsity, and SATS\-based models that match their respective realized sparsities\. The SATS\-based models \(with matched realized sparsity comparable CATS\) perform better in terms of average accuracy, which highlights the effectiveness of the calibration method\.Training\-free Performance:The comparative performance of CATS\(Leeet al\.,[2024](https://arxiv.org/html/2607.08991#bib.bib2)\)and the proposed SATS is provided in Table[1](https://arxiv.org/html/2607.08991#S4.T1)and Table[2](https://arxiv.org/html/2607.08991#S4.T2), which demonstrate the downstream task performance and generation performance, respectively\.

The tables show that at the matched 30% operating point, SATS is broadly competitive with CATS on both models\. On llama 3\.1 8B, SATS has a slight improvement in average downstream accuracy relative to CATS, but with a small perplexity cost\. On Qwen 3 8B, SATS improves average accuracy and Wiki\-Text2 perplexity, while RefinedWeb perplexity remains slightly worse\.

The clearer gains appear at the matched 50% operating point\. On llama 3\.1 8B, SATS improves average downstream accuracy from 75\.44 to 76\.2 at an identical sparsity level, while providing better perplexity scores on both RefinedWeb and Wiki\-Text2\. On Qwen 3 8B, SATS improves average downstream accuracy from 73\.99 to 74\.31 while improving perplexity metrics\. This demonstrates that at similar realized sparsity levels, the calibration method of SATS outperforms CATS in terms of performance\. While we reported our accuracy and perplexity results in BF16, we also conducted experiments on the difference in performance between FP32 and BF16 on llama 3\.1 8B on Table[5](https://arxiv.org/html/2607.08991#S4.T5), which demonstrates that the difference between the performance of the model in these data formats are minimal\.

Table 2:Perplexity scores of the base models \(lower is better\), CATS at 30% and 50% target sparsity, and SATS \(with matched realized sparsity comparable to CATS\)\. At a higher sparsity level, SATS\-based calibration outperforms CATS\-based methods\.Fine\-tuned model performance:We compare the performance of the fine\-tuned CATS and SATS on Table[3](https://arxiv.org/html/2607.08991#S4.T3)and Table[4](https://arxiv.org/html/2607.08991#S4.T4)\. While Table[3](https://arxiv.org/html/2607.08991#S4.T3)indicates that the difference between CATS and SATS is reduced in downstream task average accuracy, Table[4](https://arxiv.org/html/2607.08991#S4.T4)highlights that for the generation task performance, SATS still performs better\.

While llama 3\.1 8B models show little difference in downstream task performance compared to training\-free methods, the effect of fine\-tuning becomes more evident in language modeling tasks, where these models have lower levels of perplexity in general\. It is also interesting to note that the fine\-tuned Qwen 3 models outperformed even the corresponding base model\.

Table 3:Downstream task performance of base models,fine\-tunedCATS\-based models with 50% targeted sparsity, andfine\-tunedSATS\-based models that match its realized sparsities\.Table 4:Perplexity scores of the base models \(lower is better\),fine\-tunedCATS at 50% target sparsity, andfine\-tunedSATS \(with matched realized sparsity comparable to CATS\)\.Table 5:Comparison of performance with full precision and half\-precision on llama 3\.1 8B
### 4\.2Evaluation of Token Routing

We use the SATS matched to CATS\-50% operating point and evaluate the effectiveness of the token routing method on both training\-free and fine\-tuned settings\.

Training\-free Methods:The results in Table[6](https://arxiv.org/html/2607.08991#S4.T6)indicate that token routing improves the quality relative to the static SATS method while preserving most of the throughput gain over dense baseline inference\. On llama 3\.1 8B, the 50% target routing provides the best balance between quality and speed in terms of accuracy and perplexity\.

On Qwen 3 8B, the 50% target routing improves average accuracy from 74\.31 to 75\.11, while improving the perplexity metrics\. Though static SATS provides the best throughput gain, routed SATS remain fast while boosting average accuracy\. While we have measured our latency in FP32 throughout the experiments, we also observed the BF16 performance of the proposed token routing method on Table[8](https://arxiv.org/html/2607.08991#S4.T8)for llama 3\.1 8B, which follows similar pattern as the FP32 latency performance of the model\.

Fine\-tuning Methods:The routing results for the fine\-tuned models are shown in Table[7](https://arxiv.org/html/2607.08991#S4.T7)\. The speedup gain in the fine\-tuned routed models is less pronounced compared to the training\-free setting, since the sparse path needs to perform both the thresholded base MLP computation and the LoRA update\. In our initial experiments, the naive implementation introduced substantial extra overhead from separate low\-rank operations\. To mitigate this cost, we use LoRA\-aware sparse kernels that combine selected LoRA computations with the sparse MLP path and reduce kernel\-launch and small\-matrix overhead through custom Triton kernels\. Though this additional computation narrows the relative gain, the routed fine\-tuned models still retain a clear throughput advantage over dense inference\.

After fine\-tuning the models, the sparse model performance improves notably for Qwen 3\. It is interesting to note that with the improved performance of the fine\-tuned Qwen 3 model, routing the tokens to the SATS path more \(e\.g\., 50% or 75% route target\) improves the accuracy, even surpassing the performance of both static dense and sparse models in terms of average accuracy\.

Table 6:Downstream task accuracy and language modeling performance of base model, token routed SATS, and static SATS \(both use the SATS matched to CATS 50% target sparsity\)\. The target route percentage is measured from a calibration dataset\. The results indicate that the token routing offers higher accuracy while offering faster throughput over the baseline dense model\.Table 7:Downstream task accuracy and language modeling performance of base model, token routed SATSfine\-tunedand static SATSfine\-tuned\(both use the SATS matched to CATS 50% target sparsity\)\. The target route percentage is measured from a calibration dataset\. The results indicate that the token routing offers higher accuracy while offering faster throughput over the baseline dense model\.ModelMethodTokens/Secllama 3\.1 8BBaseline052\.2Routed SATS2550\.25054\.17554\.3Static SATS10057Table 8:BF16 \(half\-precision\) latency of base llama 3\.1 8B model, token routed SATS and static SATS \(both use the SATS matched to CATS 50% target sparsity\)

## 5Discussion and Conclusion

We studied how to utilize sparsity in LLMs through sensitivity\-aware activation thresholding and token\-level routing\. Our results indicate that the proposed SATS improves over percentile\-based calibration at similar sparsity levels\. We also demonstrated that token routing improves downstream accuracy and language\-modeling quality while remaining faster than dense inference\.

These results indicate that both activation\-level and token\-level sparsity can be utilized for efficient LLM inference\. Better threshold calibration improves the static sparse model quality, while token routing allows selectively applying the sparse model for a better quality\-throughput trade\-off\. Overall, the proposed SATS and token routing improve practical LLM inference efficiency\.

## References

- J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§2](https://arxiv.org/html/2607.08991#S2.p1.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)Boolq: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 \(long and short papers\),pp\. 2924–2936\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- S\. Elfwing, E\. Uchibe, and K\. Doya \(2018\)Sigmoid\-weighted linear units for neural network function approximation in reinforcement learning\.Neural networks107,pp\. 3–11\.Cited by:[§2](https://arxiv.org/html/2607.08991#S2.p2.3)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2023\)A framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2607.08991#S1.p4.1),[§4](https://arxiv.org/html/2607.08991#S4.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p7.1)\.
- D\. Lee, J\. Lee, G\. Zhang, M\. Tiwari, and A\. Mirhoseini \(2024\)Cats: contextually\-aware thresholding for sparsity in large language models\.arXiv preprint arXiv:2404\.08763\.Cited by:[§1](https://arxiv.org/html/2607.08991#S1.p1.1),[§1](https://arxiv.org/html/2607.08991#S1.p2.1),[§2](https://arxiv.org/html/2607.08991#S2.p4.1),[§4\.1](https://arxiv.org/html/2607.08991#S4.SS1.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p5.1),[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2381–2391\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- G\. Penedo, Q\. Malartic, D\. Hesslow, R\. Cojocaru, A\. Cappelli, H\. Alobeidli, B\. Pannier, E\. Almazrouei, and J\. Launay \(2023\)The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only\.arXiv preprint arXiv:2306\.01116\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p5.1),[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§2](https://arxiv.org/html/2607.08991#S2.p1.1)\.
- J\. Welbl, N\. F\. Liu, and M\. Gardner \(2017\)Crowdsourcing multiple choice science questions\.InProceedings of the 3rd Workshop on Noisy User\-generated Text,pp\. 94–106\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2607.08991#S1.p4.1),[§4](https://arxiv.org/html/2607.08991#S4.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)Hellaswag: can a machine really finish your sentence?\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 4791–4800\.Cited by:[§4](https://arxiv.org/html/2607.08991#S4.p6.1)\.
- Z\. Zhang, Y\. Song, G\. Yu, X\. Han, Y\. Lin, C\. Xiao, C\. Song, Z\. Liu, Z\. Mi, and M\. Sun \(2024\)ReLU2 wins: discovering efficient activation functions for sparse llms\.arXiv preprint arXiv:2402\.03804\.Cited by:[§1](https://arxiv.org/html/2607.08991#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.08991#S3.SS1.p1.2)\.