LRCC: Generalizing Low-Rank Compression with Conditional Computation

# LRCC: Generalizing Low-Rank Compression with Conditional Computation
Source: [https://arxiv.org/html/2610.08858](https://arxiv.org/html/2610.08858)
Thomas V\. Fontanari\*Maximo Rulli Federico Alvetreti

###### Abstract

Low\-rank compression reduces the cost of pretrained language models by replacing linear transformations with low\-rank factorizations\. However, conventional methods use a fixed rank allocation during inference, assigning the same amount of compute regardless of the input token\. We introduce Low\-Rank Conditional Computation \(LRCC\), which adds token\-dependent computation to pretrained models by training one lightweight router per Transformer block to select among a small set of nested low\-rank paths\. During training, the low\-rank factors remain frozen, and only the routers are optimized\. We evaluateLRCCon Llama and Qwen models for language modeling and zero\-shot downstream tasks\. Within the same average active\-parameter budget,LRCCimproves the predictive performance over static low\-rank compression, including a 7\.6 percentage\-point gain in average downstream accuracy on Llama\-2\-7B over static methods\. At matched batch\-size\-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama\-3\.2\-1B and remains competitive on Llama\-2\-7B, without specialized kernels\. Finally, we assess the usefulness of assigning a token\-wise path by analyzing the routers’ path choices\.

$\*$$\*$footnotetext:Correspondence to thomas\.fontanari@uniroma1\.it## 1Introduction

Foundation models\([Bommasani et al\., 2021](https://arxiv.org/html/2610.08858#bib.bib1)\)achieve strong predictive performance across several tasks\([Devlin et al\., 2019](https://arxiv.org/html/2610.08858#bib.bib36);[Brown et al\., 2020](https://arxiv.org/html/2610.08858#bib.bib37);[Radford et al\., 2021](https://arxiv.org/html/2610.08858#bib.bib38)\)but require substantial computational and memory resources, motivating efficiency techniques\. Low\-rank compression methods\([Chen et al\., 2021](https://arxiv.org/html/2610.08858#bib.bib11)\)approximate the original weight matrices with low\-rank products, typically obtained using SVD or its activation\-aware variants\. Given activations collected on a calibration dataset, these methods seek a rank\-constrained approximation that minimizes the reconstruction error of each layer’s outputs\. Layer\-specific ranks are then selected to satisfy a global budget via fixed heuristics or learned allocations\([Qi et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib4);[Qinsi et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib7)\)\. Conditional computation\([Scardapane et al\., 2024](https://arxiv.org/html/2610.08858#bib.bib22);[Pfeiffer et al\., 2023](https://arxiv.org/html/2610.08858#bib.bib23)\)instead adapts the computational graph to input activations\. Applying it to pretrained models, however, can require additional training and compute modules alongside lightweight routers\.

Previous work has demonstrated that SVD\-like decompositions of linear layers can be used to produce many\-in\-one models, where the smaller ones correspond to a subset of the weights of larger ones\([Yuan et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib2);[Zaccone et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib8)\)\. The nested structure of these approaches provides an inexpensive basis for conditional computation\. Routing over this hierarchy at every linear layer would nevertheless require many routers, each choosing among thousands of possible ranks\. For example, a linear layer in Llama\-2\-7B can have up to40964096rank choices, and each Transformer block contains seven such layers\. The challenge is to expose useful flexibility without making routing itself expensive\.

This work asks two questions:*can token\-dependent rank allocation improve predictive performance within the same active\-parameter budget, and do these gains translate into better quality at matched decoding latency?*We distinguish these questions because activating fewer parameters does not by itself guarantee lower latency\. We address these questions by proposingLow\-RankConditionalComputation \(LRCC\)\. Through low\-rank decomposition and rank allocation strategies, we construct a small set of models with shared weights and different computational costs\. The smaller models correspond to a subset of the ranks used in the larger ones, leading to a nested family of models\. A single router per Transformer block is then trained to select which subset of ranks to activate within the block, while the pretrained model and shared low\-rank factors remain frozen\. By using a small number of submodels and limiting the routers to the block level, we are able to show that token\-dependent computation can be recovered from an existing low\-rank hierarchy by training only lightweight routers, while still producing meaningful quality gains under controlled compute budgets\.

Our main contributions are summarized below:

1. 1\.We introduceLRCC, which combines shared low\-rank factors with block\-level routing to allocate ranks dynamically to each token\.
2. 2\.We show thatLRCCimproves predictive performance over static low\-rank compression within the same active\-parameter budget, with gains of up to7\.67\.6percentage points in average downstream accuracy\. It also achieves competitive quality at matched batch\-size\-1 decoding latency without specialized CUDA kernels\.
3. 3\.We introduce a target\-cost annealing scheme that produces a family of router checkpoints in one progressive training run, and analyze how the number of paths, routing choices, and deployment costs affect the resulting models\.

Figure 1:Overview ofLRCC\. First, \(a\) each linear weight matrix is decomposed into ordered rank\-one components, and a rank\-allocation algorithm constructs nested paths at different target costs\. Then, \(b\) the low\-rank factors are frozen and one router per block is trained to select a path for each token, balancing task and compression losses\.
## 2Method

Conventional low\-rank compression methods\([Yuan et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib2);[Wang et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib5);[Parkina and Rakhuba, 2025](https://arxiv.org/html/2610.08858#bib.bib3);[Qi et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib4);[Qinsi et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib7)\)replace each linear weight matrix with a product of smaller matrices and retain only a fixed number of rank components in each layer\. The number retained in each layer is chosen so that the compressed model meets a predefined inference\-cost budget, typically expressed as a fraction of the original model’s cost\.

In contrast,LRCCdynamically determines the number of ranks activated in each linear layer of a Transformer block for each token\. More precisely, we define multiplepathsthrough each Transformer block, where each path specifies the number of ranks activated in every linear layer of the block\. As in Matryoshka learning\([Kusupati et al\., 2022](https://arxiv.org/html/2610.08858#bib.bib12)\), the paths are ordered and nested: lower\-cost paths retain subsets of the components used by higher\-cost paths\. Consequently, all paths share the same parameter base and can be selected using a single path index\. We train only the block\-level routers that choose among the possible paths for each token, while the remaining parameters are frozen\.[Figure1](https://arxiv.org/html/2610.08858#S1.F1)illustrates the framework\.

### 2\.1Layer Decomposition

For each linear weight matrix𝐖b​ℓ∈ℝdin×dout\\mathbf\{W\}\_\{b\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times d\_\{\\mathrm\{out\}\}\}at layerℓ\\ellin Transformer blockbb, we compute an activation\-aware decomposition using a matrix of input activations𝐗b​ℓ∈ℝn×din\\mathbf\{X\}\_\{b\\ell\}\\in\\mathbb\{R\}^\{n\\times d\_\{\\mathrm\{in\}\}\}, wherennis the number of token activations collected on a calibration dataset\. More formally, we solve for each layer

arg​min𝐖~b​ℓ,r⁡‖𝐗b​ℓ​𝐖b​ℓ−𝐗b​ℓ​𝐖~b​ℓ,r‖Fs\.t\.rank⁡\(𝐖~b​ℓ,r\)≤r\.\\argmin\_\{\\widetilde\{\\mathbf\{W\}\}\_\{b\\ell,r\}\}\\left\\\|\\mathbf\{X\}\_\{b\\ell\}\\mathbf\{W\}\_\{b\\ell\}\-\\mathbf\{X\}\_\{b\\ell\}\\widetilde\{\\mathbf\{W\}\}\_\{b\\ell,r\}\\right\\\|\_\{F\}\\quad\\text\{s\.t\.\}\\quad\\operatorname\{rank\}\(\\widetilde\{\\mathbf\{W\}\}\_\{b\\ell,r\}\)\\leq r\.\(1\)Let𝐕b​ℓ∈ℝdout×dout\\mathbf\{V\}\_\{b\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{out\}\}\}denote the matrix of right singular vectors of𝐗b​ℓ​𝐖b​ℓ\\mathbf\{X\}\_\{b\\ell\}\\mathbf\{W\}\_\{b\\ell\}with columns ordered by decreasing singular value, and let𝐕b​ℓ,:r\\mathbf\{V\}\_\{b\\ell,:r\}denote its firstrrcolumns\. Then,[Equation1](https://arxiv.org/html/2610.08858#S2.E1)has an analytical solution given by

𝐖~b​ℓ,r=𝐖b​ℓ𝐕b​ℓ,:r𝐕b​ℓ,:r⊤\.\\widetilde\{\\mathbf\{W\}\}\_\{b\\ell,r\}=\\mathbf\{W\}\_\{b\\ell\}\\mathbf\{V\}\_\{b\\ell,:r\}\\mathbf\{V\}\_\{b\\ell,:r\}^\{\\top\}\.\(2\)Appendix[B](https://arxiv.org/html/2610.08858#A2)proves optimality and nestedness and Appendix[E\.1](https://arxiv.org/html/2610.08858#A5.SS1)describes how to compute the factors efficiently\.

[Equation2](https://arxiv.org/html/2610.08858#S2.E2)has the important property that, similar to standard SVD, optimal solutions for different target ranksrrare nested\. Hence, we can obtain the best rank\-rractivation\-aware approximation by indexing the firstrrcolumns of the precomputed matrices𝐀b​ℓ=𝐖b​ℓ​𝐕b​ℓ\\mathbf\{A\}\_\{b\\ell\}=\\mathbf\{W\}\_\{b\\ell\}\\mathbf\{V\}\_\{b\\ell\}and𝐁b​ℓ,=𝐕b​ℓ\\mathbf\{B\}\_\{b\\ell,\}=\\mathbf\{V\}\_\{b\\ell\}\. More concretely,

𝐖~b​ℓ,r=\(𝐖b​ℓ𝐕b​ℓ,:r\)𝐕b​ℓ,:r⊤=𝐀b​ℓ,:r𝐁b​ℓ,:r⊤\.\\widetilde\{\\mathbf\{W\}\}\_\{b\\ell,r\}=\(\\mathbf\{W\}\_\{b\\ell\}\\mathbf\{V\}\_\{b\\ell,:r\}\)\\mathbf\{V\}\_\{b\\ell,:r\}^\{\\top\}=\\mathbf\{A\}\_\{b\\ell,:r\}\\mathbf\{B\}\_\{b\\ell,:r\}^\{\\top\}\.\(3\)Naively replacing a weight matrix with the product of its low\-rank factors does not necessarily reduce computation\. A dense linear transformation requires approximately2​din​dout2d\_\{\\mathrm\{in\}\}d\_\{\\mathrm\{out\}\}FLOPs per token, whereas its rank\-rrfactorization requires approximately2​r​\(din\+dout\)2r\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\)FLOPs, yielding a computational benefit only whenr<din​doutdin\+doutr<\\frac\{d\_\{\\mathrm\{in\}\}d\_\{\\mathrm\{out\}\}\}\{d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\}\.

### 2\.2Path Construction

In[Section2\.1](https://arxiv.org/html/2610.08858#S2.SS1), we explained how to produce the decompositions of each linear transform in the model\. Crucially, the decompositions form a nested structure, where retaining more components cannot increase reconstruction error in[Equation1](https://arxiv.org/html/2610.08858#S2.E1), but induces a larger number of FLOPs\. Routing over every possible rank at every layer would introduce many routing decisions and a large choice set for each router\. Our path\-count ablation in[Section4\.1](https://arxiv.org/html/2610.08858#S4.SS1)shows that a small set of choices is sufficient to capture most of the quality gains\.

Instead, we simplify the routing problem by placing routers only at the beginning of Transformer blocks and restricting them to a small set of choices\. We first constructTTsubmodels at different target costs using the rank\-allocation procedure of Swift\-SVD\([Qi et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib4)\)\. Given a target cost, Swift\-SVD assigns a rank to each linear layer according to its estimated importance to model performance\.

More formally, submodelmmwithm=1,…,Tm=1,\.\.\.,Tis defined by the rank allocation

ℛm=\{rb​ℓ​m\}b,ℓ,\\mathcal\{R\}\_\{m\}=\\\{r\_\{b\\ell m\}\\\}\_\{b,\\ell\},whererb​ℓ​mr\_\{b\\ell m\}denotes the rank retained by linear layerℓ\\ellin Transformer blockbbunder submodelmm\. Equivalently, each weight matrix of the submodel is approximated as

𝐖~b​ℓ\(m\)=𝐀b​ℓ,:rb​ℓ​m𝐁b​ℓ,:rb​ℓ​m⊤\.\\widetilde\{\\mathbf\{W\}\}\_\{b\\ell\}^\{\(m\)\}=\\mathbf\{A\}\_\{b\\ell,:r\_\{b\\ell m\}\}\\mathbf\{B\}\_\{b\\ell,:r\_\{b\\ell m\}\}^\{\\top\}\.\(4\)Thus, each submodel specifies a different computational budget by assigning a particular rank to every linear layer\. These whole\-model allocations define the choices available to each block\-level router\. In blockbb, pathpb​mp\_\{bm\}is the restriction of submodelmm’s allocation to that block,

pb​m=\{rb​ℓ​m\}ℓ\.p\_\{bm\}=\\\{r\_\{b\\ell m\}\\\}\_\{\\ell\}\.Selectingpb​mp\_\{bm\}therefore determines simultaneously how many ranks are activated in each linear layer of blockbb\.

We can count the number of parameters in a path based on its set of ranks\. A factorized matrix of rankrractivatesr⁡\(din\+dout\)r\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\)parameters, since it is represented by two factors of sizesdin×rd\_\{\\mathrm\{in\}\}\\times randdout×rd\_\{\\mathrm\{out\}\}\\times r\. Therefore, the parameter count of pathpb​mp\_\{bm\}is

Pb​m=∑ℓrb​ℓ​m​\(din,b​ℓ\+dout,b​ℓ\)\.P\_\{bm\}=\\sum\_\{\\ell\}r\_\{b\\ell m\}\\left\(d\_\{\\mathrm\{in\},b\\ell\}\+d\_\{\\mathrm\{out\},b\\ell\}\\right\)\.\(5\)
LRCCtrains routers to assign each token to a corresponding path\. Specifically, for each token entering a Transformer blockbb, the routergb:ℝdh→\[0,1\]Tg\_\{b\}:\\mathbb\{R\}^\{d\_\{h\}\}\\rightarrow\[0,1\]^\{T\}is a learnable linear transform that maps the input activationsh∈ℝdhh\\in\\mathbb\{R\}^\{d\_\{h\}\}to a Gumbel\-Softmax distribution over theTTavailable paths\. During both training and inference, a hard one\-hot selection determines the executed path and hence the number of activated parameters\.

### 2\.3Router Training

The routers are optimized to minimize the task loss and acompression loss, which we define in this section\. To preserve differentiability and retain the hard one\-hot selection of the routers, we use the straight\-through estimator \(STE\)\([Jang et al\., 2017](https://arxiv.org/html/2610.08858#bib.bib39)\)\. During training, all pretrained model parameters and low\-rank factors remain frozen, and only the routers are updated\.LRCCuses the ratio of activated parameters relative to the original model as the cost of processing a given token\. Letcbc\_\{b\}denote the average normalized cost of blockbb, and letCbC\_\{b\}denote the normalized cost of its original non\-factorized implementation\. Because a low\-rank factorization can activate more parameters than the original block, we softly cap its cost during training using a smooth approximate ofmin⁡\(cb,Cb\)\\min\(c\_\{b\},C\_\{b\}\)given by

c¯b=cb−Cbκ​softplus⁡\(κ⁡\(cb−Cb\)Cb\),\\bar\{c\}\_\{b\}=c\_\{b\}\-\\frac\{C\_\{b\}\}\{\\kappa\}\\operatorname\{softplus\}\\left\(\\frac\{\\kappa\(c\_\{b\}\-C\_\{b\}\)\}\{C\_\{b\}\}\\right\),\(6\)whereκ\\kappais a hyperparameter that determines the smoothness of the approximation\. This reflects that blocks whose factorized implementation is more expensive than the original one should be kept non\-factorized when deployed, while avoiding the zero\-gradient region introduced by a hard cap\.

The global cost and corresponding compression loss are then

cglobal=∑b=1Bc¯b,ℒc=max⁡\(0,cglobal−t\),c\_\{\\mathrm\{global\}\}=\\sum\_\{b=1\}^\{B\}\\bar\{c\}\_\{b\},\\qquad\\mathcal\{L\}\_\{c\}=\\max\\left\(0,c\_\{\\mathrm\{global\}\}\-t\\right\),\(7\)wherettis the user\-defined target cost\. The routers are then trained to jointly minimize the task lossℒt​a​s​k\\mathcal\{L\}\_\{task\}and the computational cost:

ℒ=ℒtask\+λs​ℒc,\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{task\}\}\+\\lambda\_\{s\}\\mathcal\{L\}\_\{\\mathrm\{c\}\},\(8\)whereλs\\lambda\_\{s\}is an adaptive weight updated at training stepssaccording to the current constraint violation, as described in[Section2\.4](https://arxiv.org/html/2610.08858#S2.SS4)\. Further details on the computation ofcbc\_\{b\}and the soft cap are given in Appendix[C](https://arxiv.org/html/2610.08858#A3)\.

### 2\.4Target\-cost Annealing

Because the paths are nested,LRCCnaturally supports producing routers for multiple target costs in a single progressive training run, moving from a high\-cost model to one with fewer activated parameters\. We start with a high target cost,0\.950\.95in our experiments, and train the router until convergence \(or until a maximum number of epochs is reached\)\. Then, we reduce the target cost to a slightly lower value,0\.9250\.925in our case, and continue training the router weights to produce a new checkpoint\. We stop when the final target cost is reached, or when convergence fails for some target, i\.e\., the training process failed to obtain a router inducing a cost within an acceptable distance of the current target\.

As compression becomes more aggressive, the task’s loss increases and can dominate the total cost, making it difficult to reach a checkpoint sufficiently close to the target\. To avoid this, we introduce an adaptive compression\-loss weightλs\\lambda\_\{s\}, which is updated after each optimization step based on the constraint violation:

λs\+1=max⁡\(0,λs\+α⁡\(c𝗀𝗅𝗈𝖻𝖺𝗅−t\)\),\\lambda\_\{s\+1\}=\\max\(0,\\lambda\_\{s\}\+\\alpha\(c\_\{\\mathsf\{global\}\}\-t\)\),\(9\)whereα\\alphais a hyperparameter\. Intuitively, whenc𝗀𝗅𝗈𝖻𝖺𝗅\>tc\_\{\\mathsf\{global\}\}\>t, the compression\-loss weight increases, and whenc𝗀𝗅𝗈𝖻𝖺𝗅<tc\_\{\\mathsf\{global\}\}<t, it decreases, allowing the router to activate more parameters\.

## 3Results

Models\.We evaluate the performance ofLRCCon Llama\-3\.2\-1B\([Meta AI, 2024](https://arxiv.org/html/2610.08858#bib.bib32)\), Llama\-2\-7B\([Touvron et al\., 2023](https://arxiv.org/html/2610.08858#bib.bib33)\)and Qwen3\-4B\([Yang et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib34)\)models, as they allow us to compare with the low\-rank baseline methods\([Qi et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib4);[Parkina and Rakhuba, 2025](https://arxiv.org/html/2610.08858#bib.bib3);[Wang et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib5)\)\.

LRCC setup\.We decompose all linear layers in each Transformer block, including both attention and MLP projections, as described in[Section2\.1](https://arxiv.org/html/2610.08858#S2.SS1), and compute the ranks assigned to each block path using a rank\-allocation algorithm, as described in[Section2\.2](https://arxiv.org/html/2610.08858#S2.SS2)\. In all cases, we follow previous work and use 256 2,048\-token samples from the training set of WikiText\-2\([Merity et al\., 2017](https://arxiv.org/html/2610.08858#bib.bib25)\)to decompose the weights and train the router\. We use the standard causal language\-modeling loss as the task lossℒtask\\mathcal\{L\}\_\{\\mathrm\{task\}\}\. Training starts at a target cost of 0\.975, and we progressively obtain checkpoints down to 0\.3 in steps of 0\.025 using the annealing scheme described in[Section2\.4](https://arxiv.org/html/2610.08858#S2.SS4)\. Finally, during evaluation, we use 64 WikiText\-2 validation samples to estimate the average activated parameter count of each block\. Blocks for which this exceeds the parameter count of the corresponding block with non\-factorized linear layers are deployed in non\-factorized form\.

Datasets\.The resulting models are evaluated on the WikiText\-2 test set for language modeling and on several zero\-shot downstream performance using ARC\-Easy and ARC\-Challenge\([Clark et al\., 2018](https://arxiv.org/html/2610.08858#bib.bib26)\), BoolQ\([Clark et al\., 2019](https://arxiv.org/html/2610.08858#bib.bib27)\), HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.08858#bib.bib28)\), OpenBookQA\([Mihaylov et al\., 2018](https://arxiv.org/html/2610.08858#bib.bib29)\), PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.08858#bib.bib30)\), and WinoGrande\([Sakaguchi et al\., 2020](https://arxiv.org/html/2610.08858#bib.bib31)\)\.

Static baselines\.We compareLRCCwith several state\-of\-the\-art low\-rank compression methods, namely SVD\-LLM\([Wang et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib5)\), COALA\([Parkina and Rakhuba, 2025](https://arxiv.org/html/2610.08858#bib.bib3)\)and Swift\-SVD\([Qi et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib4)\)\. Since COALA and SVD\-LLM do not propose a rank allocation method, we also evaluate variants using ranks obtained with theSwift\-SVDrank allocation strategy\. We omit SVD\-LLM’s optional fine\-tuning stage to isolate the effect of the compression method under a matched adaptation budget, since the other baselines do not use post\-compression fine\-tuning of the weights and an analogous refinement stage could likewise be applied toLRCCand the baselines\. See Appendix[D](https://arxiv.org/html/2610.08858#A4)for details and additional results\.

Table 1:Predictive performance of Llama\-2\-7B within the indicated active\-parameter budgets\. LRCC achieves the highest average accuracy at all compression ratios\. Results for Llama\-2\-7B compressed using 256 samples of sequence length 2048\. Avg\. reports mean accuracy over the seven reasoning tasks; Wiki2 reports perplexity\.### 3\.1Predictive Performances

[Table1](https://arxiv.org/html/2610.08858#S3.T1)shows the predictive performance results across selected ratios for Llama\-2\-7B \(see Appendix[D\.1](https://arxiv.org/html/2610.08858#A4.SS1)for results on Llama\-3\.2\-1B and Qwen3\-4B\)\. SinceLRCCis dynamic, its actual average ratio of activated parameters slightly differs from the specified target costs\. In cases where the actual cost was higher on a task than the target cost, we show the performance obtained using a router with lower target costs, to ensure a fairer comparison\. The actual average costs for each model in each task shown are specified in[Table7](https://arxiv.org/html/2610.08858#A4.T7)\. In[Section4\.3](https://arxiv.org/html/2610.08858#S4.SS3), we show that the actual cost of theLRCCmodels follow closely their target costs across several datasets not seen during training\. At the90%90\\%budget,LRCC\-9 reaches62\.7%62\.7\\%average accuracy, compared with55\.1%55\.1\\%for the best static baseline, a gain of7\.67\.6percentage points\. At the70%70\\%budget,LRCC\-5 improves average accuracy from45\.4%45\.4\\%to49\.2%49\.2\\%\. Both variants also improve WikiText\-2 perplexity at these budgets\. The gains extend across most individual tasks, showcasing the beneficial effect of introducing conditional computation to the low\-rank decompositions\. Several static baselines obtain similar results among themselves, particularly on Llama\-2\-7B\. This is consistent with their shared activation\-aware reconstruction objective: in exact arithmetic and under the same calibration and rank allocation, their decompositions target equivalent optimal output approximations, while their numerical procedures differ\. Among the evaluated combinations, COALA\+Swift generally gives the strongest static results on Llama, so we use it for the trade\-off curves\.

Figure 2:Predictive performance versus actual active\-parameter ratio forLRCC\-5 and COALA\+Swift\. Average accuracy is the mean over the seven downstream tasks\.[Figure2](https://arxiv.org/html/2610.08858#S3.F2)comparesLRCC\-5 with COALA\+Swift over a wider range of measured active\-parameter ratios\. We evaluate checkpoints with target costs from0\.30\.3to0\.950\.95in steps of0\.050\.05and plot their actual average costs on each dataset\. The path\-count ablation in[Section4\.1](https://arxiv.org/html/2610.08858#S4.SS1)motivates using five paths and Appendix[D\.4](https://arxiv.org/html/2610.08858#A4.SS4)gives additional results\.LRCCimproves the quality–cost trade\-off in almost all plotted settings, with particularly large gains on the downstream tasks\. We hypothesize that aggressive static truncation removes subspaces important for particular tasks, leading to sharp performance drops\([Ahmad et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib24)\)\.LRCCcan retain such components for selected tokens while meeting an average budget, providing a possible explanation for the gains\. We also observe a greater language\-modeling benefit on Llama\-3\.2\-1B\. One possible explanation is that larger models are more robust to compression\([Yuan et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib2)\)\.

To test whether the quality gains extend beyond Llama, we compareLRCCwithSwift\-SVDon Qwen3\-4B\. Appendix[D\.1](https://arxiv.org/html/2610.08858#A4.SS1)reports the results and the measured\-cost curves in[Figure7](https://arxiv.org/html/2610.08858#A4.F7)\.LRCCimproves perplexity and average downstream accuracy across the reported budgets\.

### 3\.2Computational Performance

We also measured the latency for models with target costs from0\.30\.3to0\.90\.9on samples from the test sets\. Both LRCC and the baseline were compiled with TorchDynamo and TorchInductor using identical settings\. Following[Lin et al\. \(2024\)](https://arxiv.org/html/2610.08858#bib.bib35), we measured batch\-size\-1 autoregressive decoding, which is a setting relevant to local, resource\-constrained deployment\. Each measurement was repeated 5 times on a DGX Spark \(NVIDIA GB10 Grace Blackwell\) with 128 GB of unified memory\. We teacher\-forced the ground\-truth task tokens through the models to ensure identical inputs for both deployments and measured latency as the time required to decode each complete sequence divided by its number of decoded tokens\. For multiple\-choice tasks, we timed each candidate continuation and treated all continuations associated with the same document as one sample cluster\.[Figure3](https://arxiv.org/html/2610.08858#S3.F3)shows the resulting latency–quality trade\-offs\. Appendix[D\.2](https://arxiv.org/html/2610.08858#A4.SS2)gives the latency tables and measurement protocol; Appendix[D\.4](https://arxiv.org/html/2610.08858#A4.SS4)reports compilation and prefill times\.LRCC\-5 achieves better predictive performance at the same latency on all tasks for Llama\-3\.2\-1B and on downstream tasks for Llama\-2\-7B at higher latencies\. The results are mixed for Llama\-2\-7B on WikiText\-2\.

Figure 3:Pareto curves ofLRCC\-5 and COALA\+Swift across various decoding latencies \(ms/token\) applied to Llama\-2\-7B and Llama\-3\.2\-1B\. The average accuracy is obtained by averaging the results on the commonsense reasoning downstream tasks\.

## 4Analysis

LRCCassigns variable amounts of compute to different input samples by choosing a different path among available options\. We now ask how many paths are needed, whether the routers use them differently across tokens, and whether the learned budgets transfer beyond the router\-training data\.

### 4\.1Effect of the Number of Paths

LRCCrequires determining the number of available paths within each block\. To better understand the influence of this hyperparameter, we trained models withT∈\{2,3,5,9,10,17,33\}T\\in\\\{2,3,5,9,10,17,33\\\}paths per block and measured perplexity on the WikiText\-2 validation set\.[Figure4](https://arxiv.org/html/2610.08858#S4.F4)shows the validation perplexity obtained across several target costs and number of paths\. Perplexity increases sharply when only two paths are available and remains notably higher with three paths\. The effects are particularly pronounced at lower target costs, suggesting that the limited granularity in these settings makes it harder to find a feasible solution\. Increasing the number of paths beyond five provides comparatively little improvement in most settings\. Additional results for Llama\-3\.2\-1B, and Torch compilation, prefill and decoding times across various path cardinalities are shown in Appendix[D\.4](https://arxiv.org/html/2610.08858#A4.SS4)\.

\(a\)![Refer to caption](https://arxiv.org/html/2610.08858v1/n_paths_heatmap_llama-2-7b.png)\(b\)

Figure 4:Ablation on the number of paths for Llama\-2\-7B\. \(a\) shows WikiText\-2 validation perplexity as a function of the number of paths for selected target costs, while \(b\) shows the results for all target costs from0\.30\.3to0\.9750\.975in steps of0\.0250\.025\. Hyperparameter configurations for which no run converged are shown in white\.
### 4\.2Token\-dependent Routing

To test whether the routers collapse to a single path for all tokens, we analyzed their choices for 64 samples from WikiText\-2, each comprising 2,048 tokens\. Specifically, we measured the average computational cost across each block and the effective fraction of paths at different target costs\. For each router, we define the effective fraction of paths as

Feff=exp⁡\(H⁡\(f\)\)T,F\_\{\\mathrm\{eff\}\}=\\frac\{\\exp\(H\(f\)\)\}\{T\},\(10\)whereffdenotes the empirical distribution of path selections across tokens,H⁡\(f\)H\(f\)is its entropy computed with natural logarithms, andTTis the number of available paths\. This quantity equals11when all paths are selected with equal frequency and1/T1/Twhen all tokens are assigned to a single path\.[Figure5](https://arxiv.org/html/2610.08858#S4.F5)\(a\) shows the results for Llama\-2\-7B\.

![Refer to caption](https://arxiv.org/html/2610.08858v1/block_router_stacked_heatmaps_llama-2-7b_wikitext2.png)\(a\)
\(b\)
Figure 5:Router choices for Llama\-2\-7B on WikiText\-2\. \(a\) Average per\-block cost \(top\) and effective fraction of paths \(bottom\) for each target\. The dots indicate blocks that are deployed without factorization\. \(b\) Distributions of the costs assigned to word and non\-word tokens for each target\.The results show a backbone of blocks retained in dense form, alongside blocks that route tokens through different low\-rank paths\. As the target cost decreases, the number of blocks that exhibit conditional routing increases\. Multiple paths remain in use, with some Transformer blocks using almost all available paths\. Together, these observations show that the routed blocks do not collectively collapse to a static rank allocation, even without an explicit path\-balancing loss\.

We also analyze whether certain tokens were more often assigned to low\-cost paths\. As shown in[Figure5](https://arxiv.org/html/2610.08858#S4.F5)\(b\), non\-word tokens \(i\.e\., tokens containing no alphanumeric characters\) are more frequently assigned to lower\-cost paths\. This provides evidence that compute allocation varies with token type\. Appendix[D\.5](https://arxiv.org/html/2610.08858#A4.SS5)reports similar patterns for Llama\-3\.2\-1B\.

### 4\.3Target vs\. Actual Deployment Cost

LRCCrouters are fitted to use a certain amount of compute on average, but this is not strictly enforced during inference\. Hence, we measured the actual ratio of activated parameters ofLRCCacross different datasets used for testing\.[Figure6](https://arxiv.org/html/2610.08858#S4.F6)shows the results for Llama\-2\-7B\.

Figure 6:Target versus actual deployment cost for Llama\-2\-7B withLRCC\-5\.LRCCcosts remain close to the target across different datasets\.Although the routers were trained only on WikiText\-2, actual deployment costs remain close to their targets across the evaluated datasets\. Appendix[D\.3](https://arxiv.org/html/2610.08858#A4.SS3)presents the results for Llama\-3\.2\-1B and Qwen3\-4B\.

## 5Related Work

##### Low\-rank Compression

Several low\-rank compression methods have been proposed in recent years\([Shihab et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib6);[Qi et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib4);[Parkina and Rakhuba, 2025](https://arxiv.org/html/2610.08858#bib.bib3);[Wang et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib5);[Yuan et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib2);[Qinsi et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib7)\)\.[Zaccone et al\. \(2026\)](https://arxiv.org/html/2610.08858#bib.bib8)and[Rauba and van der Schaar \(2026\)](https://arxiv.org/html/2610.08858#bib.bib9)construct nested low\-rank submodels to support multiple compute budgets within a single model\. Concurrent work by[Zhu et al\. \(2026\)](https://arxiv.org/html/2610.08858#bib.bib10)selects prompt\-dependent singular components under fixed per\-layer rank budgets and reuses the selected subspaces across decoding steps\. These approaches offer either selectable global budgets or prompt\-dependent subspaces\.LRCCinstead varies the active rank at each block for each token, under an average cost target\.

##### Conditional Computation

Mixture\-of\-Experts\([Shazeer et al\., 2017](https://arxiv.org/html/2610.08858#bib.bib14)\)has become standard in LLM pretraining, and several works have proposed architectures with dynamic computational graphs \(e\.g\.,\([Raposo et al\., 2024](https://arxiv.org/html/2610.08858#bib.bib15);[Ainslie et al\., 2023](https://arxiv.org/html/2610.08858#bib.bib16);[Devvrit et al\., 2024](https://arxiv.org/html/2610.08858#bib.bib17)\)\)\. However, these approaches generally require training the model from scratch\. LikeLRCC, some approaches adapt pretrained checkpoints\. Moefication and upcycling are arguably among the best known\([Zhang et al\., 2022](https://arxiv.org/html/2610.08858#bib.bib20);[Komatsuzaki et al\., 2023](https://arxiv.org/html/2610.08858#bib.bib18);[Szatkowski et al\., 2024](https://arxiv.org/html/2610.08858#bib.bib19)\), but other forms have been studied \(e\.g\.\([Wójcik et al\., 2025](https://arxiv.org/html/2610.08858#bib.bib13);[Cai et al\., 2024](https://arxiv.org/html/2610.08858#bib.bib21)\)\)\. These techniques are generally either difficult to apply to existing models \(e\.g\., Moefication is tailored to ReLU\-induced activation sparsity\) or require training additional compute modules, whereasLRCCtrains only lightweight routers over fixed factors\.

## 6Conclusion

In this work, we introducedLRCC, a technique for adding conditional computation to pretrained Transformers by training block\-level routers over shared low\-rank factors\.LRCCimproves predictive performance within fixed active\-parameter budgets on Llama models, with gains of up to7\.67\.6percentage points in average downstream accuracy; the Qwen experiments extend the quality gains to another model family\. At matched batch\-size\-1 decoding latency,LRCCimproves perplexity and average downstream accuracy on Llama\-3\.2\-1B and remains competitive on Llama\-2\-7B, without custom CUDA kernels\. Our analysis shows that a small set of paths captures most of the quality gains and that the routers retain token\-dependent path choices without an explicit balancing loss\. Together, these results establish shared low\-rank hierarchies as a practical basis for introducing conditional computation into pretrained models\.

### AI use statement

We used generative AI tools, including coding agents, to discuss, critique, and refine our methodological ideas, improve the exposition of proof sketches, and assist with implementing code, given our specifications of the algorithms and requirements\. We also used generative AI tools for literature search and to revise text passages for clarity and presentation\. AI tools were not used to independently formulate the proposed method, design or run experiments, interpret experimental results, or generate experimental claims\. AI\-assisted code was manually inspected, mathematical arguments were independently verified, and citations were checked against the original sources\. We take responsibility for the final content of the paper and all AI\-assisted work\.

### Ethics statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

### Reproducibility statement

[Section2\.1](https://arxiv.org/html/2610.08858#S2.SS1)describes the activation\-aware low\-rank decomposition,[Section2\.2](https://arxiv.org/html/2610.08858#S2.SS2)the construction of block\-level paths,[Section2\.3](https://arxiv.org/html/2610.08858#S2.SS3)the router training objective, and[Section2\.4](https://arxiv.org/html/2610.08858#S2.SS4)the target\-cost annealing procedure\. Appendix[B](https://arxiv.org/html/2610.08858#A2)proves the low\-rank approximation result\. Appendix[C](https://arxiv.org/html/2610.08858#A3)describes the cost computation and compression objective for training, and Appendix[E](https://arxiv.org/html/2610.08858#A5)reports implementation details, training settings, and baseline hyperparameters\. Appendix[D\.2](https://arxiv.org/html/2610.08858#A4.SS2)describes the latency\-measurement protocol\.

#### Acknowledgments

We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA \(Italy\)\. Donatella Genovese is supported by PNRR MUR project PE0000013\-FAIR\.

## References

- Ahmadet al\.\(2025\)A\. Ahmad, A\. Joshi, and A\. ModiBeyond components: singular vector\-based interpretability of transformer circuits\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=7UbXEQNny7)Cited by:[§3\.1](https://arxiv.org/html/2610.08858#S3.SS1.p2.1)\.
- Ainslieet al\.\(2023\)J\. Ainslie, T\. Lei, M\. de Jong, S\. Ontanon, S\. Brahma, Y\. Zemlyanskiy, D\. Uthus, M\. Guo, J\. Lee\-Thorp, Y\. Tay, Y\. Sung, and S\. SanghaiCoLT5: faster long\-range transformers with conditional computation\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://openreview.net/forum?id=h96N32OkAx)Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.Proceedings of the AAAI Conference on Artificial Intelligence34\(05\),pp\. 7432–7439\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6239)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p3.1)\.
- Bommasaniet al\.\(2021\)R\. Bommasani, D\. A\. Hudson, E\. Adeli, R\. Altman, S\. Arora, S\. von Arx, M\. S\. Bernstein, J\. Bohg, A\. Bosselut, E\. Brunskill,et al\.On the opportunities and risks of foundation models\.arXiv preprint arXiv:2108\.07258\.Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1)\.
- Caiet al\.\(2024\)R\. Cai, S\. Muralidharan, G\. Heinrich, H\. Yin, Z\. Wang, J\. Kautz, and P\. MolchanovFlextron: many\-in\-one flexible large language model\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=9vKRhnflAs)Cited by:[§C\.1](https://arxiv.org/html/2610.08858#A3.SS1.p2.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)P\. Chen, H\. Yu, I\. S\. Dhillon, and C\. HsiehDRONE: data\-aware low\-rank compression for large NLP models\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=sthiz9zeXGG)Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Link](https://aclanthology.org/N19-1300/),[Document](https://dx.doi.org/10.18653/v1/N19-1300)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p3.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p3.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1)\.
- Devvritet al\.\(2024\)F\. Devvrit, S\. Kudugunta, A\. Kusupati, T\. Dettmers, K\. Chen, I\. S\. Dhillon, Y\. Tsvetkov, H\. Hajishirzi, S\. M\. Kakade, A\. Farhadi, and P\. JainMatFormer: nested transformer for elastic inference\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=fYa6ezMxD5)Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Izenman \(1975\)A\. J\. IzenmanReduced\-rank regression for the multivariate linear model\.Journal of Multivariate Analysis5\(2\),pp\. 248–264\.External Links:[Document](https://dx.doi.org/10.1016/0047-259X%2875%2990042-1)Cited by:[Appendix B](https://arxiv.org/html/2610.08858#A2.SS0.SSS0.Px2.p1.1),[Appendix B](https://arxiv.org/html/2610.08858#A2.p1.1)\.
- Janget al\.\(2017\)E\. Jang, S\. Gu, and B\. PooleCategorical reparameterization with gumbel\-softmax\.InInternational Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2610.08858#S2.SS3.p1.1)\.
- Komatsuzakiet al\.\(2023\)A\. Komatsuzaki, J\. Puigcerver, J\. Lee\-Thorp, C\. R\. Ruiz, B\. Mustafa, J\. Ainslie, Y\. Tay, M\. Dehghani, and N\. HoulsbySparse upcycling: training mixture\-of\-experts from dense checkpoints\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=T5nUQDrM4u)Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Kusupatiet al\.\(2022\)A\. Kusupati, G\. Bhatt, A\. Rege, M\. Wallingford, A\. Sinha, V\. Ramanujan, W\. Howard\-Snyder, K\. Chen, S\. M\. Kakade, P\. Jain, and A\. FarhadiMatryoshka representation learning\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=9njZa1fm35)Cited by:[§2](https://arxiv.org/html/2610.08858#S2.p2.1)\.
- Linet al\.\(2024\)J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. HanAwq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems6,pp\. 87–100\.Cited by:[§3\.2](https://arxiv.org/html/2610.08858#S3.SS2.p1.1)\.
- Merityet al\.\(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p2.1)\.
- Meta AI \(2024\)Meta AILlama 3\.2 model card\.Note:[https://github\.com/meta\-llama/llama\-models/blob/main/models/llama3\_2/MODEL\_CARD\.md](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p1.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2381–2391\.External Links:[Link](https://aclanthology.org/D18-1260/),[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p3.1)\.
- Parkina and Rakhuba \(2025\)U\. Parkina and M\. RakhubaCOALA: numerically stable and efficient framework for context\-aware low\-rank approximation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=z1wIUZtBmK)Cited by:[§E\.1](https://arxiv.org/html/2610.08858#A5.SS1.p3.1),[§2](https://arxiv.org/html/2610.08858#S2.p1.1),[§3](https://arxiv.org/html/2610.08858#S3.p1.1),[§3](https://arxiv.org/html/2610.08858#S3.p4.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Pfeifferet al\.\(2023\)J\. Pfeiffer, S\. Ruder, I\. Vulić, and E\. M\. PontiModular deep learning\.arXiv preprint arXiv:2302\.11529\.Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1)\.
- Qiet al\.\(2026\)R\. Qi, Y\. Liu, X\. Wu, X\. Wang, M\. Li, C\. Chen, J\. Chen, Y\. Chen, and Q\. WengSwift\-SVD: theoretical optimality meets practical efficiency in low\-rank LLM compression\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=nAQ4h8FpdM)Cited by:[§E\.1](https://arxiv.org/html/2610.08858#A5.SS1.p1.1),[§E\.1](https://arxiv.org/html/2610.08858#A5.SS1.p4.1),[§1](https://arxiv.org/html/2610.08858#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.08858#S2.SS2.p2.1),[§2](https://arxiv.org/html/2610.08858#S2.p1.1),[§3](https://arxiv.org/html/2610.08858#S3.p1.1),[§3](https://arxiv.org/html/2610.08858#S3.p4.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Qinsiet al\.\(2025\)W\. Qinsi, J\. Ke, M\. Tomizuka, K\. Keutzer, and C\. XuDobi\-SVD: differentiable SVD for LLM compression and some new perspectives\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=kws76i5XB8)Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1),[§2](https://arxiv.org/html/2610.08858#S2.p1.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1)\.
- Raposoet al\.\(2024\)D\. Raposo, S\. Ritter, B\. Richards, T\. Lillicrap, P\. C\. Humphreys, and A\. SantoroMixture\-of\-depths: dynamically allocating compute in transformer\-based language models\.arXiv preprint arXiv:2404\.02258\.Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Rauba and van der Schaar \(2026\)P\. Rauba and M\. van der SchaarDeep hierarchical learning with nested subspace networks for large language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ymUOPsbxLi)Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Sakaguchiet al\.\(2020\)K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.Proceedings of the AAAI Conference on Artificial Intelligence34\(05\),pp\. 8732–8740\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6399)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p3.1)\.
- Scardapaneet al\.\(2024\)S\. Scardapane, A\. Baiocchi, A\. Devoto, V\. Marsocci, P\. Minervini, and J\. PomponiConditional computation in neural networks: principles and research trends\.Intelligenza Artificiale18\(1\),pp\. 175–190\.Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Shihabet al\.\(2026\)I\. F\. Shihab, S\. Akter, and A\. SharmaBeyond variance: knowledge\-aware LLM compression via fisher\-aligned subspace diagnostics\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 47938–47954\.External Links:[Link](https://aclanthology.org/2026.acl-long.2214/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.2214),ISBN 979\-8\-89176\-390\-6Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Szatkowskiet al\.\(2024\)F\. Szatkowski, B\. Wójcik, M\. Piórczyński, and S\. ScardapaneExploiting activation sparsity with dense to dynamic\-k mixture\-of\-experts conversion\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=38UFpdt3Tr)Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, Y\. Zheng, Z\. Wan, and M\. ZhangSVD\-LLM: truncation\-aware singular value decomposition for large language model compression\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=LNYIUouhdt)Cited by:[§2](https://arxiv.org/html/2610.08858#S2.p1.1),[§3](https://arxiv.org/html/2610.08858#S3.p1.1),[§3](https://arxiv.org/html/2610.08858#S3.p4.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Wójciket al\.\(2025\)B\. Wójcik, A\. Devoto, K\. Pustelnik, P\. Minervini, and S\. ScardapaneAdaptive computation modules: granular conditional computation for efficient inference\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 21510–21518\.Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang,et al\.Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p1.1)\.
- Yuanet al\.\(2025\)Z\. Yuan, Y\. Shang, Y\. Song, D\. Yang, Q\. Wu, Y\. Yan, and G\. SunASVD: activation\-aware singular value decomposition for compressing large language models\.External Links:2312\.05821,[Link](https://arxiv.org/abs/2312.05821)Cited by:[§1](https://arxiv.org/html/2610.08858#S1.p2.1),[§2](https://arxiv.org/html/2610.08858#S2.p1.1),[§3\.1](https://arxiv.org/html/2610.08858#S3.SS1.p2.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Zacconeet al\.\(2026\)R\. Zaccone, S\. Laskaridis, M\. Ciccone, and S\. HorváthFlexRank: nested low\-rank knowledge decomposition for adaptive model deployment\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=DK0kvnNelx)Cited by:[Appendix A](https://arxiv.org/html/2610.08858#A1.p1.1),[§E\.1](https://arxiv.org/html/2610.08858#A5.SS1.p2.1),[§1](https://arxiv.org/html/2610.08858#S1.p2.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/),[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[§3](https://arxiv.org/html/2610.08858#S3.p3.1)\.
- Zhanget al\.\(2022\)Z\. Zhang, Y\. Lin, Z\. Liu, P\. Li, M\. Sun, and J\. ZhouMoefication: transformer feed\-forward layers are mixtures of experts\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 877–890\.Cited by:[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2026\)H\. Zhu, Z\. Mi, G\. L\. Zhang, and S\. HuangDifferent prompts, different ranks: prompt\-aware dynamic rank selection for SVD\-based LLM compression\.arXiv preprint arXiv:2605\.08568\.Cited by:[§D\.4](https://arxiv.org/html/2610.08858#A4.SS4.p2.1),[§5](https://arxiv.org/html/2610.08858#S5.SS0.SSS0.Px1.p1.1)\.

## Appendix ALimitations

Our work focuses on demonstrating that conditioning the activated ranks on input activations can improve the quality–compute trade\-off of low\-rank compressed models, but several directions remain open\. First, we have not evaluatedLRCCon nested subspace models that refine the factorized weights through fine\-tuning, such as FlexRank\([Zaccone et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib8)\)\. Combining such refinement with conditional rank selection may yield further gains\. Second, we demonstrated latency improvements only for batch\-size\-1 decoding, and future work should investigate optimized implementations for prefill and larger batch sizes\. Finally, our paths are constructed before router training from a fixed set of nested rank allocations\. Jointly learning the path structure and the routing policy, or exploring more flexible routing granularities, could further improve the allocation of computation across inputs\.

## Appendix BLow\-rank Approximation Results

This section establishes the optimality and nestedness of the activation\-aware rank\-rrapproximation used in[Section2\.1](https://arxiv.org/html/2610.08858#S2.SS1), and its connection to reduced\-rank regression\([Izenman, 1975](https://arxiv.org/html/2610.08858#bib.bib40)\)\. Efficient computation of the factors is discussed in Appendix[E\.1](https://arxiv.org/html/2610.08858#A5.SS1)\.

##### Optimality and nestedness\.

For simplicity, we omit the block and layer subscripts\. Let𝐖∈ℝdin×dout\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times d\_\{\\mathrm\{out\}\}\}be a linear weight matrix and let𝐗∈ℝn×din\\mathbf\{X\}\\in\\mathbb\{R\}^\{n\\times d\_\{\\mathrm\{in\}\}\}contain its calibration inputs\. We consider the rank\-constrained reconstruction problem

min𝐖~r⁡‖𝐗𝐖−𝐗​𝐖~r‖Fs\.t\.rank⁡\(𝐖~r\)≤r\.\\min\_\{\\widetilde\{\\mathbf\{W\}\}\_\{r\}\}\\left\\\|\\mathbf\{X\}\\mathbf\{W\}\-\\mathbf\{X\}\\widetilde\{\\mathbf\{W\}\}\_\{r\}\\right\\\|\_\{F\}\\qquad\\text\{s\.t\.\}\\quad\\operatorname\{rank\}\(\\widetilde\{\\mathbf\{W\}\}\_\{r\}\)\\leq r\.\(11\)Let

𝐘=𝐗𝐖=𝐔​𝚺​𝐕⊤\\mathbf\{Y\}=\\mathbf\{X\}\\mathbf\{W\}=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{V\}^\{\\top\}\(12\)be an SVD of the layer outputs, with singular values ordered in decreasing order, and let𝐕:r\\mathbf\{V\}\_\{:r\}denote the firstrrcolumns of𝐕\\mathbf\{V\}\. We define

𝐖~r=𝐖𝐕:r𝐕:r⊤\.\\widetilde\{\\mathbf\{W\}\}\_\{r\}=\\mathbf\{W\}\\mathbf\{V\}\_\{:r\}\\mathbf\{V\}\_\{:r\}^\{\\top\}\.\(13\)Sincerank\(𝐖𝐕:r𝐕:r⊤\)≤r\\operatorname\{rank\}\(\\mathbf\{W\}\\mathbf\{V\}\_\{:r\}\\mathbf\{V\}\_\{:r\}^\{\\top\}\)\\leq r, this is a feasible solution to Equation[11](https://arxiv.org/html/2610.08858#A2.E11)\. Its output on the calibration inputs is

𝐗​𝐖~r\\displaystyle\\mathbf\{X\}\\widetilde\{\\mathbf\{W\}\}\_\{r\}=𝐗𝐖𝐕:r𝐕:r⊤\\displaystyle=\\mathbf\{X\}\\mathbf\{W\}\\mathbf\{V\}\_\{:r\}\\mathbf\{V\}\_\{:r\}^\{\\top\}\(14\)=𝐘𝐕:r𝐕:r⊤\\displaystyle=\\mathbf\{Y\}\\mathbf\{V\}\_\{:r\}\\mathbf\{V\}\_\{:r\}^\{\\top\}\(15\)=𝐔:r𝚺:r𝐕:r⊤=𝐘r,\\displaystyle=\\mathbf\{U\}\_\{:r\}\\mathbf\{\\Sigma\}\_\{:r\}\\mathbf\{V\}\_\{:r\}^\{\\top\}=\\mathbf\{Y\}\_\{r\},\(16\)where𝐘r\\mathbf\{Y\}\_\{r\}is the truncated rank\-rrSVD of𝐘\\mathbf\{Y\}\. For any other feasible matrix𝐖^r\\widehat\{\\mathbf\{W\}\}\_\{r\},

rank⁡\(𝐗​𝐖^r\)≤rank⁡\(𝐖^r\)≤r\.\\operatorname\{rank\}\(\\mathbf\{X\}\\widehat\{\\mathbf\{W\}\}\_\{r\}\)\\leq\\operatorname\{rank\}\(\\widehat\{\\mathbf\{W\}\}\_\{r\}\)\\leq r\.\(17\)Therefore, by the Eckart–Young–Mirsky theorem,

‖𝐗𝐖−𝐗​𝐖~r‖F=‖𝐘−𝐘r‖F≤‖𝐗𝐖−𝐗​𝐖^r‖F\.\\left\\\|\\mathbf\{X\}\\mathbf\{W\}\-\\mathbf\{X\}\\widetilde\{\\mathbf\{W\}\}\_\{r\}\\right\\\|\_\{F\}=\\left\\\|\\mathbf\{Y\}\-\\mathbf\{Y\}\_\{r\}\\right\\\|\_\{F\}\\leq\\left\\\|\\mathbf\{X\}\\mathbf\{W\}\-\\mathbf\{X\}\\widehat\{\\mathbf\{W\}\}\_\{r\}\\right\\\|\_\{F\}\.\(18\)Hence,𝐖~r\\widetilde\{\\mathbf\{W\}\}\_\{r\}is a global optimum of Equation[11](https://arxiv.org/html/2610.08858#A2.E11)\.

Moreover, define the full factors

𝐀=𝐖𝐕,𝐁=𝐕\.\\mathbf\{A\}=\\mathbf\{W\}\\mathbf\{V\},\\qquad\\mathbf\{B\}=\\mathbf\{V\}\.\(19\)For every target rankrr, the optimal approximation can then be obtained by indexing the firstrrcolumns of these same factors:

𝐖~r=𝐀:r𝐁:r⊤=\(𝐖𝐕:r\)𝐕:r⊤\.\\widetilde\{\\mathbf\{W\}\}\_\{r\}=\\mathbf\{A\}\_\{:r\}\\mathbf\{B\}\_\{:r\}^\{\\top\}=\(\\mathbf\{W\}\\mathbf\{V\}\_\{:r\}\)\\mathbf\{V\}\_\{:r\}^\{\\top\}\.\(20\)Thus, a single pair of ordered factors𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}simultaneously contains a globally optimal solution for every target rank\. In particular, the solutions are nested, since increasing the rank fromrrtor\+1r\+1only adds the next rank\-one component,

𝐖~r\+1=𝐖~r\+\(𝐖𝐯r\+1\)​𝐯r\+1⊤\.\\widetilde\{\\mathbf\{W\}\}\_\{r\+1\}=\\widetilde\{\\mathbf\{W\}\}\_\{r\}\+\(\\mathbf\{W\}\\mathbf\{v\}\_\{r\+1\}\)\\mathbf\{v\}\_\{r\+1\}^\{\\top\}\.\(21\)

##### Connection to reduced\-rank regression\.

Equation[11](https://arxiv.org/html/2610.08858#A2.E11)can also be viewed as a special instance of reduced\-rank regression \(RRR\)\([Izenman, 1975](https://arxiv.org/html/2610.08858#bib.bib40)\)\. Given predictors𝐗\\mathbf\{X\}and multivariate responses𝐘\\mathbf\{Y\}, RRR solves

min𝐂⁡‖𝐘−𝐗𝐂‖Fs\.t\.rank⁡\(𝐂\)≤r\.\\min\_\{\\mathbf\{C\}\}\\\|\\mathbf\{Y\}\-\\mathbf\{X\}\\mathbf\{C\}\\\|\_\{F\}\\qquad\\text\{s\.t\.\}\\quad\\operatorname\{rank\}\(\\mathbf\{C\}\)\\leq r\.\(22\)Let𝐂OLS\\mathbf\{C\}\_\{\\mathrm\{OLS\}\}be an ordinary least\-squares solution and let𝐘^OLS=𝐗𝐂OLS\\widehat\{\\mathbf\{Y\}\}\_\{\\mathrm\{OLS\}\}=\\mathbf\{X\}\\mathbf\{C\}\_\{\\mathrm\{OLS\}\}\. If𝐕:r\\mathbf\{V\}\_\{:r\}contains the leadingrrright singular vectors of𝐘^OLS\\widehat\{\\mathbf\{Y\}\}\_\{\\mathrm\{OLS\}\}, a reduced\-rank solution is

𝐂r=𝐂OLS𝐕:r𝐕:r⊤\.\\mathbf\{C\}\_\{r\}=\\mathbf\{C\}\_\{\\mathrm\{OLS\}\}\\mathbf\{V\}\_\{:r\}\\mathbf\{V\}\_\{:r\}^\{\\top\}\.\(23\)In our setting the responses are exactly the outputs of the known pretrained linear transform,𝐘=𝐗𝐖\\mathbf\{Y\}=\\mathbf\{X\}\\mathbf\{W\}\. Consequently,𝐖\\mathbf\{W\}itself is an ordinary least\-squares solution with zero residual, so that𝐘^OLS=𝐗𝐖\\widehat\{\\mathbf\{Y\}\}\_\{\\mathrm\{OLS\}\}=\\mathbf\{X\}\\mathbf\{W\}\. Substituting𝐂OLS=𝐖\\mathbf\{C\}\_\{\\mathrm\{OLS\}\}=\\mathbf\{W\}into the RRR solution therefore recovers

𝐂r=𝐖𝐕:r𝐕:r⊤=𝐖~r,\\mathbf\{C\}\_\{r\}=\\mathbf\{W\}\\mathbf\{V\}\_\{:r\}\\mathbf\{V\}\_\{:r\}^\{\\top\}=\\widetilde\{\\mathbf\{W\}\}\_\{r\},\(24\)providing an alternative interpretation of Equation[13](https://arxiv.org/html/2610.08858#A2.E13)\.

## Appendix CRouter Training Details

### C\.1Router Training

For each token representation entering blockbb, the router produces a probability distribution over itsTTpaths\. During both training and inference, the forward pass selects a single path using the highest router probability\. The executed path and its parameter count are computed from this hard one\-hot selection, while the straight\-through estimator \(STE\) propagates gradients through the soft probabilities during training\.

To encourage the router to meet a target activated\-parameter ratiot∈\[0,1\]t\\in\[0,1\], we use acompression losssimilar to[Cai et al\. \(2024\)](https://arxiv.org/html/2610.08858#bib.bib21), defined over a batch ofNNsequences of lengthSSas

ℒc=max⁡\(0,c𝗀𝗅𝗈𝖻𝖺𝗅−t\),\\mathcal\{L\}\_\{c\}=\\max\(0,c\_\{\\mathsf\{global\}\}\-t\),\(25\)where

c𝗀𝗅𝗈𝖻𝖺𝗅=∑b=1Bsoftcapb⁡\(cb\)c\_\{\\mathsf\{global\}\}=\\sum\_\{b=1\}^\{B\}\\operatorname\{softcap\}\_\{b\}\(c\_\{b\}\)\(26\)is a differentiable surrogate for the global activated\-parameter ratio\. The blockwise soft cap approximates the cost of retaining a dense \(i\.e\., non\-factorized\) block when its factorized implementation would be more expensive\. We define these terms below\.

Letyi​j​b​m∈\{0,1\}y\_\{ijbm\}\\in\\\{0,1\\\}indicate whether tokenjjin sequenceiiselects pathmmin blockbb\. The normalized average cost of blockbbis

cb=1N​S​P0​∑i=1N∑j=1S∑m=1Tyi​j​b​m​Pb​m,c\_\{b\}=\\frac\{1\}\{NSP\_\{0\}\}\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{S\}\\sum\_\{m=1\}^\{T\}y\_\{ijbm\}P\_\{bm\},\(27\)where

P0=∑b=1BPbdenseP\_\{0\}=\\sum\_\{b=1\}^\{B\}P\_\{b\}^\{\\mathrm\{dense\}\}\(28\)is the total parameter count of the original weight matrices across all Transformer blocks \(the embedding matrix and language\-model output head are not counted, as they are not compressed\)\. Thus,cbc\_\{b\}is the contribution of blockbbto the ratio of activated parameters relative to the original model\. The normalized dense \(non\-factorized\) cost of blockbbis

Cb=PbdenseP0\.C\_\{b\}=\\frac\{P\_\{b\}^\{\\mathrm\{dense\}\}\}\{P\_\{0\}\}\.\(29\)
The blockwise soft cap is defined as

softcapb⁡\(cb\)=cb−Cbκ​softplus⁡\(κ⁡\(cb−Cb\)Cb\),\\operatorname\{softcap\}\_\{b\}\(c\_\{b\}\)=c\_\{b\}\-\\frac\{C\_\{b\}\}\{\\kappa\}\\operatorname\{softplus\}\\\!\\left\(\\frac\{\\kappa\(c\_\{b\}\-C\_\{b\}\)\}\{C\_\{b\}\}\\right\),\(30\)wheresoftplus⁡\(x\)=log⁡\(1\+ex\)\\operatorname\{softplus\}\(x\)=\\log\(1\+e^\{x\}\)andκ\\kappacontrols the slope\. This function smoothly approximatesmin⁡\(cb,Cb\)\\min\(c\_\{b\},C\_\{b\}\)and approachesCbC\_\{b\}as the factorized block’s average cost increases above the dense cost\. Since∑bCb=1\\sum\_\{b\}C\_\{b\}=1, all block costs use the original model’s Transformer weights as their common denominator\. A hard cap would limit the maximum cost toCbC\_\{b\}, reflecting that inefficient blocks \(i\.e\., wherecb≥Cbc\_\{b\}\\geq C\_\{b\}\) are retained in dense form at deployment\. However, it would also create a zero\-gradient region, preventing the compression loss from providing an optimization signal\. The soft cap instead retains gradients above the dense cost, but is slightly lower than the hard\-capped cost\. We therefore report actual deployment costs separately in[Sections4\.3](https://arxiv.org/html/2610.08858#S4.SS3)and[D\.3](https://arxiv.org/html/2610.08858#A4.SS3)\.

During training, we freeze all model parameters except the routers and minimize the sum of the task and compression losses

ℒ=ℒtask\+λs​ℒc,\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{task\}\}\+\\lambda\_\{s\}\\mathcal\{L\}\_\{\\mathrm\{c\}\},\(31\)whereλs\\lambda\_\{s\}is an adaptive compression\-loss weight, updated at each training stepssaccording to the constraint violation as described in[Section2\.4](https://arxiv.org/html/2610.08858#S2.SS4)\. The target\-cost schedule and training hyperparameters are given in Appendix[E](https://arxiv.org/html/2610.08858#A5)\.

## Appendix DAdditional Experiments

### D\.1Additional Predictive Performance Results

Similarly to the results presented in[Section3\.1](https://arxiv.org/html/2610.08858#S3.SS1), we conduct an analysis of the downstream performance ofLRCCcompared to COALA, SVD\-LLM, SVD\-LLM, SVD\-LLM\+Swift, COALA\+Swift, Swift\-SVD \(see[Section3\.1](https://arxiv.org/html/2610.08858#S3.SS1)for more details\)\. We present the complete results for Llama\-2\-7B, Llama\-3\.2\-1B and Qwen3\-4B respectively in[Table2](https://arxiv.org/html/2610.08858#A4.T2),[Table3](https://arxiv.org/html/2610.08858#A4.T3)and[Table4](https://arxiv.org/html/2610.08858#A4.T4)\. As can be observed from the two tables,LRCC\-5 andLRCC\-9 continue to surpass static methods in nearly all evaluations\. This demonstrates that models benefit from dynamically allocating different ranks depending on the tokens’ needed accuracy, highlighting the strong assumptions static approaches do about models’ computations\.

In[Figure7](https://arxiv.org/html/2610.08858#A4.F7)we compareLRCC\-5 withSwift\-SVDon Qwen3\-4B across measured active\-parameter ratios\.LRCCachieves lower perplexity at every ratio, with the largest gap under aggressive compression, whereSwift\-SVDdegrades sharply\. On the downstream tasks,LRCCmatches or exceedsSwift\-SVDthroughout, with the largest gains at intermediate ratios\. These trends mirror the Llama results in[Figure2](https://arxiv.org/html/2610.08858#S3.F2), suggesting that the benefits ofLRCCare not specific to one model family\.

Figure 7:Pareto curves ofLRCC\-5 andSwift\-SVDacross various cost ratios applied to Qwen3\-4B\. The average accuracy is obtained by averaging the results on the commonsense reasoning downstream tasks\.Table 2:Performance of Llama\-2\-7B compressed using 256 samples with sequence length 2048 on a language modeling dataset \(measured using perplexity\) and seven commonsense reasoning datasets \(measured using accuracy\)\.Table 3:Performance of Llama\-3\.2\-1B compressed using 256 samples with sequence length 2048 on a language modeling dataset \(measured using perplexity\) and seven commonsense reasoning datasets \(measured using accuracy\)\.Table 4:Performance of Qwen3\-4B compressed using 256 samples with sequence length 2048 on a language modeling dataset \(measured using perplexity\) and seven commonsense reasoning datasets \(measured using accuracy\)\.
### D\.2Computational Performance Results

This section provides the measurement protocol and per\-task latency results for[Section3\.2](https://arxiv.org/html/2610.08858#S3.SS2)\. We measured the latency ofLRCCcompared with COALA\+Swift, a representative low\-rank compression approach, across several target costs and downstream tasks\. We teacher\-forced the ground\-truth task tokens through the model to ensure identical inputs for both deployments and measured latency as the time required to decode each complete sequence divided by its number of decoded tokens\. For multiple\-choice tasks, we timed each candidate continuation and treated all continuations associated with the same document as one sample cluster\. All measurements used a batch size of 1\.

The latency estimates are based on four randomly selected 2,048\-token chunks from WikiText\-2, 16 documents from WinoGrande, eight from OpenBookQA, and four from each remaining task: ARC\-Challenge, ARC\-Easy, BoolQ, HellaSwag, and PIQA\. These sample counts were selected based on pilot validation runs, with additional samples allocated to datasets exhibiting greater document\-level variability\. Paired document\-cluster bootstrap intervals for the final workloads had relative half\-widths of at most2\.14%2\.14\\%\. All candidate continuations from a sampled document were kept together during resampling to avoid treating correlated choices as independent observations\.

Both methods were compiled using TorchDynamo and TorchInductor with identical settings\. CUDA graphs were disabled because we found their support limited for conditional methods and to ensure a fair comparison\. We performed three 64\-token warm\-up runs before each measurement to exclude compilation and initialization from the timed region \(compilation and prefill times are explored in Appendix[D\.4](https://arxiv.org/html/2610.08858#A4.SS4)\)\. Experiments were conducted on an NVIDIA GB10 Grace Blackwell with 128 GB of unified memory\.

We repeated each complete experiment five times to estimate run\-to\-run variability\. Each table entry reports the mean latency in milliseconds per token across the five executions, and the subscripted±\\pmterm gives the half\-width of a 95% Student\-ttconfidence interval computed from the five execution\-level measurements \(n=5n=5,d​f=4df=4\)\.[Table5](https://arxiv.org/html/2610.08858#A4.T5)and[Table6](https://arxiv.org/html/2610.08858#A4.T6)show the results\.

At the same nominal target cost,LRCCis slower on Llama\-2\-7B, while the ordering varies with the target cost on Llama\-3\.2\-1B\.LRCCintroduces overhead from evaluating the router and dispatching the selected computational path, but its executed cost also depends on the routing decisions\. As discussed in[Section3](https://arxiv.org/html/2610.08858#S3), at matched latencies,LRCCproduces competitive predictive performance and outperforms the baseline in several settings\. Furthermore, our implementation does not use specialized fused CUDA kernels, leaving opportunities for further speedups\. For example, fusing the router with layer normalization could reduce the number of times the residual stream is read from memory\.

Table 5:Decode latency \(ms/token\) forLRCC\-5 and COALA\+Swift \(shown as COALA\+\) at matched target costs for Llama\-2\-7B at batch size 1\. Each value corresponds to the mean latency across five benchmark executions\. We also show the 95% Student\-ttconfidence interval half\-width \(n=5n=5,d​f=4df=4\)\. Both methods use identical tokenized inputs in each execution\.Table 6:Decode latency \(ms/token\) forLRCC\-5 and COALA\+Swift \(shown as COALA\+\) at matched target costs for Llama\-3\.2\-1B at batch size 1\. Each value corresponds to the mean latency across five benchmark executions\. We also show the 95% Student\-ttconfidence interval half\-width \(n=5n=5,d​f=4df=4\)\. Both methods use identical tokenized inputs in each execution\.
### D\.3Additional Results on Target vs\. Actual Deployment Cost

Due to the input\-dependence of theLRCCmethod, the average ratio of activated parameters is not strictly the same as the specified in the target cost\.[Table7](https://arxiv.org/html/2610.08858#A4.T7)shows the actual ratio of activated parameters in the different datasets, corresponding to the same checkpoints whose results were reported in Appendix[D](https://arxiv.org/html/2610.08858#A4)and for Llama\-2\-7B in[Table1](https://arxiv.org/html/2610.08858#S3.T1)\.

Table 7:Actual deployment cost \(activated\-parameter ratio\) for theLRCC\-5 andLRCC\-9 configurations used during evaluation\.Additionally,[Figure8](https://arxiv.org/html/2610.08858#A4.F8)and[Figure9](https://arxiv.org/html/2610.08858#A4.F9)show the actual deployment costs versus the target cost used during training for Llama\-3\.2\-1B and Qwen3\-4B, similarly to[Section4\.3](https://arxiv.org/html/2610.08858#S4.SS3)\.

Figure 8:Target versus actual deployment cost for Llama\-3\.2\-1B withLRCC\-5\. Each panel shows one evaluation task; the dashed line indicates exact agreement between the target and actual deployment cost\.Figure 9:Target versus actual deployment cost for Qwen3\-4B withLRCC\-5\. Each panel shows one evaluation task; the dashed line indicates exact agreement between the target and actual deployment cost\.
### D\.4Ablation on the Number of Paths

As described in[Section4\.1](https://arxiv.org/html/2610.08858#S4.SS1), we trained models withP∈\{2,3,5,9,10,17,33\}P\\in\\\{2,3,5,9,10,17,33\\\}paths per block to understand the influence of this hyperparameter\. Each path corresponds to a rank assignment with a target cost uniformly spaced between0\.10\.1and1\.01\.0\. At each refinement, we retain the previous paths and add paths with target costs midway between them\. For example, withP=2P=2, we use paths constructed by the rank\-allocation algorithm with target costs of0\.10\.1and1\.01\.0\. ForP=3P=3, we add a path with target cost0\.550\.55\. The results for Llama\-2\-7B are shown in[Figure4](https://arxiv.org/html/2610.08858#S4.F4)\.[Figure10](https://arxiv.org/html/2610.08858#A4.F10)shows the perplexities for Llama\-3\.2\-1B\. Perplexity increases sharply with only two paths, but increasing the number of choices beyond five provides comparatively little improvement\. This agrees with the results for Llama\-2\-7B in[Section4\.1](https://arxiv.org/html/2610.08858#S4.SS1)\.

\(a\)![Refer to caption](https://arxiv.org/html/2610.08858v1/n_paths_heatmap_llama-3.2-1B.png)\(b\)

Figure 10:Ablation on the number of paths for Llama\-3\.2\-1B\. \(a\) shows WikiText\-2 validation perplexity as a function of the number of paths for selected target costs, while \(b\) shows the results for all target costs from0\.30\.3to0\.9750\.975in steps of0\.0250\.025\.We also evaluate the effect of the number of paths on compilation, prefill with 1,024 tokens, and decoding for models with a target cost of0\.70\.7\. For this ablation, we decode 64 tokens after prefilling with samples from WikiText\-2, rather than using the complete task sequences timed in[SectionD\.2](https://arxiv.org/html/2610.08858#A4.SS2)\.[Figure11](https://arxiv.org/html/2610.08858#A4.F11)shows that compilation time is the metric most affected by the number of paths\. Compiling the 33\-path models was infeasible because graph capture exceeded TorchDynamo’s recursion depth\. Compilation also became substantially slower as the number of paths increased\. Increasing the number of paths has little effect on the decoding and prefill latencies of Llama\-2\-7B but increases the decoding latency of Llama\-3\.2\-1B\. Overall, the 5\-path models provide a good trade\-off between predictive and computational performance\. Finally, we note that the prefill implementation ofLRCCcomputes the first factor multiplication at full rank, masks the intermediate representation according to the selected path, and then applies the second factor, thus performing computation for ranks that are subsequently masked\. The prefill implementation therefore does not yet exploit all of the computation savings available from routing; we leave its optimization to future work\. Other optimizations could improve the prefill performance of both static low\-rank and LRCC models relative to the dense base model, for example by fusing operations that share the same inputs\([Zhu et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib10)\)\.

\(a\)Llama\-2\-7B\(b\)Llama\-3\.2\-1B
Figure 11:Effect of the number of paths on compilation, prefill, and decoding times\. Increasing the number of paths leads to substantially higher compilation times and slightly higher decoding times\.
### D\.5Router Choices

We repeated the same experiments described in[Section4\.2](https://arxiv.org/html/2610.08858#S4.SS2)using Llama\-3\.2\-1B\. The results are shown in[Figure12](https://arxiv.org/html/2610.08858#A4.F12)\. As in Llama\-2\-7B, a backbone of blocks is retained in dense form, while other Transformer blocks use multiple paths across tokens\.[Figure12](https://arxiv.org/html/2610.08858#A4.F12)\(b\) also suggests thatNon\-Wordtokens are assigned less compute on average\.

![Refer to caption](https://arxiv.org/html/2610.08858v1/block_router_stacked_heatmaps_llama32.png)\(a\)
\(b\)
Figure 12:Router choices for Llama\-3\.2\-1B on WikiText\-2\. \(a\) Average per\-block cost \(top\) and effective fraction of paths \(bottom\) for each target\. The dots indicate blocks that are deployed without factorization\. \(b\) Distributions of the costs assigned to word and non\-word tokens for each target\.

## Appendix EImplementation Details

All training was conducted on A100 GPUs with 64GB of VRAM\. We trained linear block\-level routers on 256 WikiText\-2 sequences of length 2,048 with BF16 parameters using Adam with a constant learning rate of10−310^\{\-3\}and effective batch size of 4\. Training annealed over 28 target costs, from0\.9750\.975to0\.3000\.300in steps of−0\.025\-0\.025with 12 epochs per target by default and up to 30 epochs when no valid checkpoint was found\. A training epoch took, on average, 2 minutes 24 seconds for Llama\-2\-7B and 28 seconds for Llama\-3\.2\-1B\. Both models required approximately 350 epochs to produce all 28 checkpoints, corresponding to approximately 14 hours for Llama\-2\-7B and 2\.3 hours for Llama\-3\.2\-1B, or about 30 and 5 minutes per checkpoint, respectively\.

For the compression loss, we set the soft\-cap slope toκ=20\\kappa=20\. We initialized the compression\-loss weight atλ0=1\\lambda\_\{0\}=1\. We selected the dual learning rateα\\alphaforλ\\lambdaand the Gumbel\-Softmax temperature schedule using a model\-specific grid of four configurations\. For Llama\-3\.2\-1B, we searched dual learning ratesα∈\{0\.4,0\.8\}\\alpha\\in\\\{0\.4,0\.8\\\}and temperature schedules of a fixedτ=0\.5\\tau=0\.5and→0\.52\.0\\\!\\rightarrow\\\!0\.5\. For Llama\-2\-7B, we searchedα∈\{1\.2,1\.6\}\\alpha\\in\\\{1\.2,1\.6\\\}and temperature schedules of a fixedτ=2\.0\\tau=2\.0and→0\.52\.0\\\!\\rightarrow\\\!0\.5\. Among the four runs, we selected the configuration with the lowest WikiText\-2 validation loss at the largest number of the 28 target costs\. We repeated this search independently for each path count in\{2,3,5,9,10,17,33\}\\\{2,3,5,9,10,17,33\\\}\.

Using the authors’ source code, we reimplemented COALA, Swift\-SVD, and SVD\-LLM in our framework to ensure consistent experimental settings and evaluation\. We omit SVD\-LLM’s optional fine\-tuning stage so that no method fine\-tunes the pretrained weights after compression;LRCCadditionally trains its routers\. An analogous weight\-refinement stage could be applied toLRCCand the baselines\. For COALA, we searched accumulation values\{2,4\}\\\{2,4\\\}and regularization valuesμ∈\{0,1\}\\mu\\in\\\{0,1\\\}, at each target cost in\{0\.3,0\.5,0\.7,0\.9\}\\\{0\.3,0\.5,0\.7,0\.9\\\}\. We found that an accumulation value of 2 andμ=0\\mu=0gave the best WikiText\-2 validation performance, although differences were small\. We used these values for all COALA decompositions, including the base factorization of the Llama models inLRCC\. For Swift\-SVD, we used their recommended layer importance weight of0\.60\.6in Llama\-2\-7B\. For Llama\-3\.2\-1B, we observed that the best results were obtained with a layer importance weight of1\.01\.0\. We used the recommended defaults for the remaining hyperparameters\. SVD\-LLM required no hyperparameter tuning\. Finally, COALA\+Swift and SVD\-LLM\+Swift combine the respective decompositions with the rank mappings produced by Swift\-SVD\. We refer to the original papers for definitions of the baseline\-specific hyperparameters\.

### E\.1Efficient Computation

The optimal factors require only the leading right singular vectors of the layer outputs𝐘=𝐗𝐖\\mathbf\{Y\}=\\mathbf\{X\}\\mathbf\{W\}, and different methods compute this subspace in different ways\. Swift\-SVD\([Qi et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib4)\), for example, avoids storing the full output matrix by incrementally accumulating its Gram matrix over calibration batches,

𝐆Y=𝐘⊤​𝐘=∑j𝐘j⊤​𝐘j,𝐘j=𝐗j​𝐖\.\\mathbf\{G\}\_\{Y\}=\\mathbf\{Y\}^\{\\top\}\\mathbf\{Y\}=\\sum\_\{j\}\\mathbf\{Y\}\_\{j\}^\{\\top\}\\mathbf\{Y\}\_\{j\},\\qquad\\mathbf\{Y\}\_\{j\}=\\mathbf\{X\}\_\{j\}\\mathbf\{W\}\.\(32\)Since

𝐆Y=𝐕​𝚺2​𝐕⊤,\\mathbf\{G\}\_\{Y\}=\\mathbf\{V\}\\mathbf\{\\Sigma\}^\{2\}\\mathbf\{V\}^\{\\top\},\(33\)a single eigendecomposition of the accumulated matrix directly provides𝐕\\mathbf\{V\}and the squared singular values\.

FlexRank\([Zaccone et al\., 2026](https://arxiv.org/html/2610.08858#bib.bib8)\)uses a closely related strategy, but instead incrementally accumulates the second moment of the*input*activations,

𝐆X=𝐗⊤​𝐗=∑j𝐗j⊤​𝐗j\.\\mathbf\{G\}\_\{X\}=\\mathbf\{X\}^\{\\top\}\\mathbf\{X\}=\\sum\_\{j\}\\mathbf\{X\}\_\{j\}^\{\\top\}\\mathbf\{X\}\_\{j\}\.\(34\)Writing𝐒=𝐆X1/2\\mathbf\{S\}=\\mathbf\{G\}\_\{X\}^\{1/2\}, the right singular vectors of𝐒𝐖\\mathbf\{S\}\\mathbf\{W\}are the same as those of the complete output matrix because

\(𝐒𝐖\)⊤​\(𝐒𝐖\)=𝐖⊤​𝐗⊤​𝐗𝐖=𝐘⊤​𝐘\.\(\\mathbf\{S\}\\mathbf\{W\}\)^\{\\top\}\(\\mathbf\{S\}\\mathbf\{W\}\)=\\mathbf\{W\}^\{\\top\}\\mathbf\{X\}^\{\\top\}\\mathbf\{X\}\\mathbf\{W\}=\\mathbf\{Y\}^\{\\top\}\\mathbf\{Y\}\.\(35\)This makes the memory required for calibration independent of the number of calibration tokens\.

COALA\([Parkina and Rakhuba, 2025](https://arxiv.org/html/2610.08858#bib.bib3)\)instead proposes avoiding the explicit formation of a Gram matrix\. In our matrix convention, let

𝐗=𝐐𝐑\\mathbf\{X\}=\\mathbf\{Q\}\\mathbf\{R\}\(36\)be a thin QR decomposition of the calibration activations, computed with tall\-skinny QR \(TSQR\) when𝐗\\mathbf\{X\}is too large to materialize at once\. Since𝐐⊤​𝐐=𝐈\\mathbf\{Q\}^\{\\top\}\\mathbf\{Q\}=\\mathbf\{I\},

‖𝐗⁡\(𝐖−𝐖~\)‖F=‖𝐑⁡\(𝐖−𝐖~\)‖F\.\\left\\\|\\mathbf\{X\}\(\\mathbf\{W\}\-\\widetilde\{\\mathbf\{W\}\}\)\\right\\\|\_\{F\}=\\left\\\|\\mathbf\{R\}\(\\mathbf\{W\}\-\\widetilde\{\\mathbf\{W\}\}\)\\right\\\|\_\{F\}\.\(37\)Furthermore,

\(𝐑𝐖\)⊤​\(𝐑𝐖\)=𝐖⊤​𝐑⊤​𝐑𝐖=𝐖⊤​𝐗⊤​𝐗𝐖=𝐘⊤​𝐘\.\(\\mathbf\{R\}\\mathbf\{W\}\)^\{\\top\}\(\\mathbf\{R\}\\mathbf\{W\}\)=\\mathbf\{W\}^\{\\top\}\\mathbf\{R\}^\{\\top\}\\mathbf\{R\}\\mathbf\{W\}=\\mathbf\{W\}^\{\\top\}\\mathbf\{X\}^\{\\top\}\\mathbf\{X\}\\mathbf\{W\}=\\mathbf\{Y\}^\{\\top\}\\mathbf\{Y\}\.\(38\)Therefore, an SVD of the substantially smaller matrix𝐑𝐖\\mathbf\{R\}\\mathbf\{W\}yields the same right singular subspace𝐕\\mathbf\{V\}in exact arithmetic\. Unlike approaches based on the normal equations, this procedure does not explicitly form𝐗⊤​𝐗\\mathbf\{X\}^\{\\top\}\\mathbf\{X\}or require its inversion\. This is numerically advantageous for ill\-conditioned calibration activations, since forming a Gram matrix squares the condition number and can amplify floating\-point errors\.

In our experiments, we used COALA’s QR strategy to compute the decompositions for the Llama models\. For Qwen3\-4B, the COALA decomposition did not converge, so we accumulated the output Gram matrix over calibration batches following[Qi et al\. \(2026\)](https://arxiv.org/html/2610.08858#bib.bib4)\.