Dynamic Parameterization Is Not Dynamic Inference

# Dynamic Parameterization Is Not Dynamic Inference
Source: [https://arxiv.org/html/2607.26192](https://arxiv.org/html/2607.26192)
###### Abstract

Input\-dependent controller coefficients are often treated as evidence of dynamic inference or computational savings\. This interpretation conflates three properties: coefficient variation, dependence of a frozen model on how coefficients are assigned to inputs, and conditional execution\. We focus on the second property and formulate a general principle of frozen\-controller auditing\. We provide one concrete implementation,Frozen\-ControllerAuditing \(FCA\), which caches the complete coefficient tensor along an unperturbed trajectory, disables the controller, and replays the frozen model with cross\-input reassignment, token shuffling, and static profiles estimated from an independent calibration set\. Because the coefficients are cached before any intervention, performance changes under replay measure assignment dependence without feedback from recomputing the controller on perturbed hidden states\. Across seven independently trained 76M FeatureGate Transformers and three 504M models, static layerwise profiles retain 98\.70% and 99\.43% of the Correct\-to\-GlobalMean performance gap, respectively\. Layer identity explains 87% to 96% of the coefficient variance\. FeatureGate nevertheless executes every Transformer block, and its measured inference is 30\.8% slower than Dense\. On the public MUDDPythia\-1\.4B checkpoint, cross\-input reassignment and token shuffling increase NLL by 1\.9067 and 2\.9637, respectively\. These penalties show that the model depends strongly on content\-conditioned cross\-layer assignment\. MUDDPythia also executes every Transformer block\. The results show that dynamic parameterization alone does not establish dynamic inference and that functional dynamics do not establish computational savings\. Claims about dynamic models should separately report coefficient variation, functional dependence of the frozen model, and actual execution\.

## 1Introduction

Adaptive computation seeks to allocate model capacity according to the requirements of each input or token\. Variable\-step models and adaptive\-depth networks change the number of refinement steps applied to a representation\(Graves[2016](https://arxiv.org/html/2607.26192#bib.bib2); Dehghaniet al\.[2019](https://arxiv.org/html/2607.26192#bib.bib3); Baninoet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib12); Houet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib29)\)\. Early\-exit and layer\-skipping systems terminate computation or bypass selected blocks for particular inputs\(Teerapittayanonet al\.[2016](https://arxiv.org/html/2607.26192#bib.bib23); Wuet al\.[2018](https://arxiv.org/html/2607.26192#bib.bib33); Elbayadet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib4); Schusteret al\.[2022](https://arxiv.org/html/2607.26192#bib.bib5)\)\. Sparse expert and token\-routing methods activate only selected experts or tokens under a computation budget\(Shazeeret al\.[2017](https://arxiv.org/html/2607.26192#bib.bib34); Feduset al\.[2022](https://arxiv.org/html/2607.26192#bib.bib36); Raoet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib41); Raposoet al\.[2024](https://arxiv.org/html/2607.26192#bib.bib6)\)\. Although these approaches are often described collectively as dynamic inference, they modify different aspects of inference\. Hard routing changes which layers, tokens, or experts are executed and can therefore reduce FLOPs or latency\. Soft residual gating evaluates every block and only rescales the corresponding updates\. Dynamic cross\-layer mixing can also preserve the full computation graph while changing how representations from different depths contribute to later computation\. All of these mechanisms may produce input\-dependent coefficients, but coefficient variation alone does not determine whether the computed function or the executed computation is genuinely dynamic\.

Existing evaluations often blur this distinction\. Gate heatmaps, coefficient variance, and cross\-input differences establish that the outputs of a controller are nonconstant, but they do not establish that the frozen model requires the original correspondence between coefficients and inputs\. A controller may vary substantially across inputs and tokens even when an input\-independent layerwise profile reproduces almost the same inference function\. Conversely, a dynamic mixing mechanism may depend critically on aligning each coefficient tensor with the content and token positions from which it was generated\. Neither outcome implies computational savings when every block remains active\. Therefore, claims about dynamic models require three separate questions: whether the coefficients vary, whether the frozen inference function depends on their input\-specific assignment, and whether the mechanism changes the computation that is actually executed\.

![Refer to caption](https://arxiv.org/html/2607.26192v1/x1.png)Figure 1:Overview of Frozen\-Controller Auditing \(FCA\)\.FCA distinguishes coefficient variation, functional dependence, and actual execution\. It first caches controller coefficients along the unperturbed trajectory, then disables the controller and replays the frozen model with reassigned, shuffled, or static coefficients\. The audit finds that a static depth profile closely reproduces the inference behavior of FeatureGate, whereas MUDDPythia depends strongly on the alignment of cross\-layer coefficients with their original inputs and token positions\. Both models still execute every Transformer block\. Dynamic coefficients alone therefore do not establish conditional execution or computational savings\.The key insight of this work is that functional dependence can be tested directly through interventions on a frozen model\. The evaluation first records the complete coefficient tensor along the unperturbed computation trajectory\. It then disables the controller and replays the same model with the cached coefficients either preserved or systematically transformed\. Because the coefficients are collected before any intervention and are not recomputed from perturbed hidden states, the resulting performance change isolates dependence on coefficient assignment from feedback through the controller\. This idea defines a general frozen\-controller auditing principle\. We instantiate the principle as Frozen\-Controller Audit, or FCA, using cross\-input reassignment, token shuffling, static profiles estimated from an independent calibration set, and separate measurements of execution\. FCA is one natural implementation of the broader principle rather than its only possible form; other coefficient\-preserving interventions can be designed for different controller structures and architectural claims\.

We apply FCA to two deliberately contrasting model families\. FeatureGate is a soft residual gating mechanism for decoder\-only Transformers\(Vaswaniet al\.[2017](https://arxiv.org/html/2607.26192#bib.bib1)\)\. It produces a coefficient for each layer and token but does not skip any block\. Across seven independently trained 76M models and three independently trained 504M models, the outputs of FeatureGate vary across inputs and tokens, yet a static layerwise depth profile retains 98\.70% and 99\.43% of the Correct\-to\-GlobalMean performance gap, respectively\. Layer identity explains most of the coefficient variance, and interventions that disrupt input or token assignment incur only small penalties\. In contrast, the public MUDDPythia\-1\.4B checkpoint, which uses position\-specific dynamic cross\-layer mixing for the query, key, value, and residual streams\(Xiaoet al\.[2025](https://arxiv.org/html/2607.26192#bib.bib15)\), is highly sensitive to cross\-input reassignment, token shuffling, and static replacement\. The same audit therefore distinguishes a controller whose dynamic variation is largely replaceable from a controller whose dynamic assignment has a substantial functional role\. Nevertheless, both model families execute every Transformer block, and the measured inference of FeatureGate is slower than that of the Dense baseline\.

Our contributions are as follows:

1. 1\.We formulate a three\-way evaluation principle that separates coefficient variation, functional dependence of the frozen model, and actual execution, thereby clarifying the evidence required for claims about dynamic parameterization, dynamic inference, and conditional computation\.
2. 2\.We introduce FCA as a reproducible implementation of frozen\-controller intervention, combining two\-stage caching and replay, coefficient\-preserving reassignment, independently calibrated static profiles, factorial interaction analysis, and explicit execution measurements\.
3. 3\.We provide a multi\-scale and multi\-model empirical study showing that FeatureGate primarily learns a stable depth profile, whereas MUDDPythia depends strongly on content\-conditioned cross\-layer assignment; we further show that neither form of dynamic parameterization establishes block skipping or computational savings\.

## 2Related Work

### 2\.1Dynamic Networks and Input\-Conditioned Parameterization

Dynamic networks adjust parameters or computation paths across samples, spatial locations, or sequence positions\(Hanet al\.[2022](https://arxiv.org/html/2607.26192#bib.bib18)\)\. One branch of this literature directly generates or combines input\-dependent weights\. Dynamic Filter Networks generate convolutional filters from the input, HyperNetworks use one network to produce the parameters of another, and CondConv and Dynamic Convolution combine several kernels with input\-dependent weights\(Jiaet al\.[2016](https://arxiv.org/html/2607.26192#bib.bib19); Haet al\.[2017](https://arxiv.org/html/2607.26192#bib.bib20); Yanget al\.[2019](https://arxiv.org/html/2607.26192#bib.bib21); Chenet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib22)\)\. Input\-conditioned parameters can be useful even when the computation graph remains dense and fixed\. Dynamic parameterization therefore does not establish dynamic execution, and coefficient variation alone does not show that the trained function uses that variation\.

### 2\.2Adaptive Depth and Early Exit

Adaptive Computation Time and the Universal Transformer vary the number of recurrent computation steps, while PonderNet learns a stopping distribution\(Graves[2016](https://arxiv.org/html/2607.26192#bib.bib2); Dehghaniet al\.[2019](https://arxiv.org/html/2607.26192#bib.bib3); Baninoet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib12)\)\. In vision models, BranchyNet, MSDNet, and Shallow\-Deep Networks use intermediate classifiers for early exit\. SkipNet and BlockDrop instead learn input\-dependent block\-skipping policies\(Teerapittayanonet al\.[2016](https://arxiv.org/html/2607.26192#bib.bib23); Huanget al\.[2018](https://arxiv.org/html/2607.26192#bib.bib24); Kayaet al\.[2019](https://arxiv.org/html/2607.26192#bib.bib25); Wanget al\.[2018](https://arxiv.org/html/2607.26192#bib.bib32); Wuet al\.[2018](https://arxiv.org/html/2607.26192#bib.bib33)\)\.

Transformer models adopt similar ideas\. Depth\-Adaptive Transformer, DeeBERT, FastBERT, PABEE, LeeBERT, BERxiT, and CALM reduce the number of executed layers through confidence criteria or learned exit rules\(Elbayadet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib4); Xinet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib26); Liuet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib27); Zhouet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib28); Zhu[2021](https://arxiv.org/html/2607.26192#bib.bib30); Xinet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib31); Schusteret al\.[2022](https://arxiv.org/html/2607.26192#bib.bib5)\)\. DynaBERT exposes subnetworks with adjustable width and depth, LayerDrop trains networks that permit depth pruning, and Mixture\-of\-Depths selects which tokens enter a block under a fixed capacity budget\(Houet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib29); Fanet al\.[2020](https://arxiv.org/html/2607.26192#bib.bib8); Raposoet al\.[2024](https://arxiv.org/html/2607.26192#bib.bib6)\)\. Claims in this family concern execution, so accuracy alone is insufficient\. The evidence must also include executed depth, FLOPs, latency, or throughput\.

### 2\.3Sparse Expert Routing

Sparse MoE models expand parameter capacity while activating only a small subset of experts for each example\(Shazeeret al\.[2017](https://arxiv.org/html/2607.26192#bib.bib34)\)\. GShard combines sparse routing with automatic sharding, Switch Transformer simplifies training with top\-1 routing, and BASE Layers use balanced assignment to control expert load\(Lepikhinet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib35); Feduset al\.[2022](https://arxiv.org/html/2607.26192#bib.bib36); Lewiset al\.[2021](https://arxiv.org/html/2607.26192#bib.bib37)\)\. V\-MoE, GLaM, and Expert Choice extend sparse routing to vision, large language models, and expert\-driven token assignment, respectively\(Riquelmeet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib38); Duet al\.[2022](https://arxiv.org/html/2607.26192#bib.bib39); Zhouet al\.[2022](https://arxiv.org/html/2607.26192#bib.bib40)\)\. These models distinguish total parameters from activated parameters, but routing distributions remain descriptive\. Establishing dependence on an input\-to\-expert assignment requires an intervention that changes the assignment while holding expert weights and routing values fixed\.

### 2\.4Token\-Level Dynamic Computation

Another family of methods changes later computation at the token level\. DynamicViT progressively removes unimportant visual tokens, EViT reorganizes and fuses tokens with low attention, and Dynamic Transformer chooses token resolution and exit depth according to image difficulty\(Raoet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib41); Lianget al\.[2022](https://arxiv.org/html/2607.26192#bib.bib42); Wanget al\.[2021](https://arxiv.org/html/2607.26192#bib.bib43)\)\. A\-ViT extends adaptive halting to spatial tokens, AdaViT jointly selects patches, attention heads, and blocks, Adaptive Token Sampling chooses an input\-dependent number of tokens, and Token Merging reduces sequence length by merging tokens between layers\(Yinet al\.[2022](https://arxiv.org/html/2607.26192#bib.bib44); Menget al\.[2022b](https://arxiv.org/html/2607.26192#bib.bib45); Fayyazet al\.[2022](https://arxiv.org/html/2607.26192#bib.bib46); Bolyaet al\.[2023](https://arxiv.org/html/2607.26192#bib.bib47)\)\. Token\-dependent coefficients constitute conditional computation only when they alter later execution\. Stable absolute\-position structure can also appear as token variation, so a suitable static baseline should preserve the joint structure of layer and position rather than only a layer mean\.

### 2\.5Residual Scaling and Cross\-Layer Information Flow

Stochastic Depth randomly skips residual blocks during training\. Fixup, ReZero, SkipInit, LayerScale, and DeepNorm instead improve deep\-network optimization through initialization or static residual scaling\(Huanget al\.[2016](https://arxiv.org/html/2607.26192#bib.bib7); Zhanget al\.[2019](https://arxiv.org/html/2607.26192#bib.bib48); Bachlechneret al\.[2021](https://arxiv.org/html/2607.26192#bib.bib9); De and Smith[2020](https://arxiv.org/html/2607.26192#bib.bib49); Touvronet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib10); Wanget al\.[2022](https://arxiv.org/html/2607.26192#bib.bib11)\)\. These methods show that layerwise coefficients can be useful without depending on the input\.

Related architectures improve the flow of information across depth\. DenseNet introduces dense connections; Transparent Attention, DLCL, and RealFormer reuse representations or attention scores from earlier layers; DenseFormer aggregates earlier representations through a depth\-weighted average\(Huanget al\.[2017](https://arxiv.org/html/2607.26192#bib.bib50); Bapnaet al\.[2018](https://arxiv.org/html/2607.26192#bib.bib51); Wanget al\.[2019](https://arxiv.org/html/2607.26192#bib.bib52); Heet al\.[2021](https://arxiv.org/html/2607.26192#bib.bib53); Pagliardiniet al\.[2024](https://arxiv.org/html/2607.26192#bib.bib54)\)\. Hyper\-Connections and MUDDFormer add dynamic connections across streams or layers\(Zhuet al\.[2025](https://arxiv.org/html/2607.26192#bib.bib55); Xiaoet al\.[2025](https://arxiv.org/html/2607.26192#bib.bib15)\)\. Our focus is not a new connection structure\. We test whether a dynamic connection remains functionally necessary after training or can be replaced by a static structure\.

### 2\.6Intervention\-Based Functional Analysis

The presence of a component, variation in its activation, or concentrated attention does not make the component functionally necessary\. Attention\-head ablations show that many visible heads can be removed at inference with little effect on performance\(Michelet al\.[2019](https://arxiv.org/html/2607.26192#bib.bib56)\)\. Causal mediation analysis, interchange intervention training, and causal tracing use internal interventions to identify the components and representations on which model behavior depends\(Viget al\.[2020](https://arxiv.org/html/2607.26192#bib.bib57); Geigeret al\.[2022](https://arxiv.org/html/2607.26192#bib.bib58); Menget al\.[2022a](https://arxiv.org/html/2607.26192#bib.bib59)\)\. FCA follows the same principle of testing function through intervention rather than inferring it from correlation\. Dynamic controllers require two additional controls: coefficients must be collected before the trajectory is perturbed, and each intervention should preserve the coefficient multiset or stable structure whenever possible\. These controls separate controller feedback from the effect of coefficient assignment\.

## 3Method: Frozen\-Controller Auditing

### 3\.1Problem Definition

Let a controller map inputxxto a coefficient tensor

A​\(x\)=\{αℓ,t,s,k​\(x\)\},A\(x\)=\\\{\\alpha\_\{\\ell,t,s,k\}\(x\)\\\},\(1\)whereℓ\\ellindexes the target layer,ttthe token position,ssthe computation stream, andkkan optional source\-layer index\. Scalar residual gating is the special case with one stream and no source\-layer mixture\. MUDDFormer uses the full tensor over target layers, tokens, streams, and source layers\.

We distinguish three properties\. Coefficient variation asks whetherA​\(x\)A\(x\)changes with the input or position\. Functional dependence asks whether the output of a fixed model depends on the correspondence between coefficient values and their original input coordinates\. Actual execution asks whether the model skips layers, tokens, or experts and whether this change reduces FLOPs, latency, or throughput cost\. FCA primarily tests functional dependence\. System measurements address actual execution separately\.

### 3\.2Two\-Stage Caching and Replay

The first stage records coefficients along the unmodified model trajectory:

Ai=A​\(xi;Correct trajectory\)\.A\_\{i\}=A\(x\_\{i\};\\ \\text\{Correct trajectory\}\)\.\(2\)The second stage disables the controller and replays the model with coefficients transformed byTT:

ℒT=ℒ​\(xi;Replay​\(T​\(Ai\)\)\)\.\\mathcal\{L\}\_\{T\}=\\mathcal\{L\}\\\!\\left\(x\_\{i\};\\ \\text\{Replay\}\(T\(A\_\{i\}\)\)\\right\)\.\(3\)We define the intervention penalty as

Δ​NLL​\(T\)=ℒT−ℒCorrect\.\\Delta\\mathrm\{NLL\}\(T\)=\\mathcal\{L\}\_\{T\}\-\\mathcal\{L\}\_\{\\text\{Correct\}\}\.\(4\)BecauseAiA\_\{i\}is fixed before the intervention,Δ​NLL\\Delta\\mathrm\{NLL\}excludes feedback in which perturbed hidden states change the controller output again\. Every audit begins with an exact replay of the original coefficients\. The intervention results are valid only if this replay reproduces the original logits and NLL\.

Algorithm 1Frozen\-ControllerAudit \(FCA\)1:Frozen model

ff, controller

cc, evaluation set

𝒟\\mathcal\{D\}, independent calibration set

𝒞\\mathcal\{C\}, intervention family

𝒯\\mathcal\{T\}
2:Run the unmodified model on

𝒞\\mathcal\{C\}and estimate static profiles that preserve layer, position, stream, and source\-layer structure

3:foreach evaluation example

xi∈𝒟x\_\{i\}\\in\\mathcal\{D\}do

4:Run

\(f,c\)\(f,c\)along the Correct trajectory and cache the full coefficient tensor

AiA\_\{i\}
5:Disable

cc, replay

AiA\_\{i\}without modification, and verify exact agreement in logits and NLL

6:foreach transformation

T∈𝒯T\\in\\mathcal\{T\}do

7:Construct a cross\-input reassignment, token shuffle, or static replacement

T​\(Ai\)T\(A\_\{i\}\)
8:Replay

T​\(Ai\)T\(A\_\{i\}\)in the same frozen model and record per\-example NLL and execution statistics

9:endfor

10:endfor

11:Compute paired

Δ​NLL\\Delta\\mathrm\{NLL\}values, factor effects, the interaction effect, static retention, and execution measures

### 3\.3Assignment Interventions and Static Profiles

The Correct condition returns the cached coefficients of each input to their original coordinates\. Cross\-input applies a global derangement across examples\. It preserves coordinate structure and the complete coefficient multiset but changes the input from which each tensor originates\. Token\-shuffle independently permutes token positions within the relevant layers and computation streams\. It preserves coefficient values and support while breaking token alignment\. Cross\+shuffle applies both transformations, with paired sampling of donor mappings and position permutations\.

Layer×\\timesPositionMean averages coefficients over examples in the independent calibration set while preserving target layer, computation stream, absolute position, and source\-layer structure\. It removes content dependence but retains stable position patterns\. StaticProfile also averages over token positions\. For FeatureGate, this condition is the layerwise LayerMean\. For MUDDPythia, it is StaticMixMean over target layer, computation stream, and source layer\.

StructureRemoved provides a model\-specific reference that removes the structure of interest\. FeatureGate uses a single GlobalMean\. MUDDPythia uses SourceUniformMean, which removes source\-layer preference while preserving the signed coefficient sum for each target layer and stream\. These references have different mathematical meanings and support only within\-model interpretation\. They do not provide a causal ranking across the two model families\.

The matching constraints above apply to controller coefficients, coefficient means, or signed sums\. An effective residual update also depends on hidden states and block outputs\. Coefficient matching is therefore not exact matching of residual updates unless the intervention also controls those update vectors\.

### 3\.4Factor Effects and Static Retention

LetC​CCCdenote correct input and token assignments,D​CDCa changed input donor with token order preserved,C​SCSthe correct input with shuffled tokens, andD​SDSa changed input donor with shuffled tokens\. The paired factor effects are

Einput\\displaystyle E\_\{\\text\{input\}\}=12​\[\(D​C−C​C\)\+\(D​S−C​S\)\],\\displaystyle=\\tfrac\{1\}\{2\}\\left\[\(DC\-CC\)\+\(DS\-CS\)\\right\],\(5\)Etoken\\displaystyle E\_\{\\text\{token\}\}=12​\[\(C​S−C​C\)\+\(D​S−D​C\)\],\\displaystyle=\\tfrac\{1\}\{2\}\\left\[\(CS\-CC\)\+\(DS\-DC\)\\right\],\(6\)Einteraction\\displaystyle E\_\{\\text\{interaction\}\}=D​S−D​C−C​S\+C​C\.\\displaystyle=DS\-DC\-CS\+CC\.\(7\)These quantities are factorial contrasts, not an additive decomposition into independent mechanisms\. A negative interaction means that the two interventions damage overlapping functions or otherwise combine nonadditively\.

For a static profileSSand the model\-specific structure\-removal referenceRR, we define

Retention​\(S\)=1−Δ​NLL​\(S\)Δ​NLL​\(R\)\.\\mathrm\{Retention\}\(S\)=1\-\\frac\{\\Delta\\mathrm\{NLL\}\(S\)\}\{\\Delta\\mathrm\{NLL\}\(R\)\}\.\(8\)Retention must be reported with the absoluteΔ​NLL\\Delta\\mathrm\{NLL\}and interpreted only within the same model and evaluation set\.

## 4Experiments and Analysis

The experiments address four questions in sequence: \(a\) whether an input\-conditioned controller outperforms static parameter sharing when gating strength is matched; \(b\) whether the frozen FeatureGate model depends on dynamic coefficient assignment; \(c\) whether the same audit detects strong functional dependence in an external model; and \(d\) whether these dynamic mechanisms reduce the computation actually executed during inference\. We discuss each result immediately after the experiment that addresses the corresponding question and do not place the results in a separate section\.

### 4\.1Models and evidence protocol

The 76M FeatureGate model is a decoder\-only Transformer with 24 layers, a hidden dimension of 512, and 8 attention heads, totaling 75\.9M parameters\. We train seven instances with independent random seeds and evaluate their byte\-level language modeling performance on OWT and WT103\(Gokaslan and Cohen[2019](https://arxiv.org/html/2607.26192#bib.bib14); Merityet al\.[2017](https://arxiv.org/html/2607.26192#bib.bib13)\)\. The residual update is

hℓ\+1,t=hℓ,t\+gℓ,t​\(x\)​Fℓ​\(hℓ\)t\.h\_\{\\ell\+1,t\}=h\_\{\\ell,t\}\+g\_\{\\ell,t\}\(x\)\\,F\_\{\\ell\}\(h\_\{\\ell\}\)\_\{t\}\.Dense fixesg=1g=1; Uniform learns a single global coefficient; LayerStatic learns one coefficient per layer; and FeatureGate generates input\-conditioned coefficients from token features\. The FeatureGate controller uses attention entropy, the KL divergence between the attention distributions of adjacent layers, a normalized log\-partition statistic, and the relative update norm as inputs\. All soft\-gated models execute every network block\. The 504M extension has 40 layers, a hidden dimension of 1024, and 16 attention heads\. We train three instances with independent random seeds\. This experiment tests whether the mechanism changes with scale\. Its training setup is not intended to establish compute optimality at approximately 500M parameters\.

For the external control, we use the publicCaiyun\-AI/MUDDPythia\-1\.4Bcheckpoint at commit3c241c8\. The model has 24 layers and 1\.420B parameters and produces signed dynamic cross\-layer mixing coefficients for the query, key, value, and residual streams\. We estimate the static profiles of FeatureGate and MUDDPythia from 512 WT103 validation windows and evaluate them on 512 nonoverlapping WT103 test windows\. Each randomized condition uses 20 paired reassignments\. For FeatureGate, each training seed is an independent unit of inference\. MUDDPythia has only one public checkpoint, so its intervals describe stability across evaluation windows and randomization error for that checkpoint rather than uncertainty across independently trained checkpoints\.

Before any intervention, each audit verifies that replaying the original coefficients reproduces the unperturbed model outputs\. The primary FeatureGate analysis uses checkpoint\-level bootstrap confidence intervals and exact two\-sided sign\-flip tests\. We apply Holm correction to the three factorial effects\. We specify two criteria before running the audit\. We consider the LayerMean effect practically small if the upper bound of its 95% confidence interval is below 0\.002 NLL\. We consider static retention high if the lower bound of its interval exceeds 95%\.

### 4\.2Dynamic Parameterization and Training Performance

This experiment tests whether an input\-conditioned controller provides better language\-modeling performance than static parameter sharing\. Without a stable advantage, variation in the controller output is not necessary to reach the observed accuracy\.

Table 1:Main training comparison\. Lower Byte\-PPL is better\. OWT uses held\-out validation data, and WT103 uses the complete official test split\. Values are the mean and standard deviation across training seeds\.SettingMethodnnByte\-PPLGate massOWT 76MUniform73\.7405±0\.05153\.7405\\pm 0\.05150\.6100OWT 76MFeatureGate73\.7437±0\.05033\.7437\\pm 0\.05030\.6102WT103 76MLayerStatic73\.0842±0\.00613\.0842\\pm 0\.00610\.5971WT103 76MFeatureGate73\.0862±0\.00643\.0862\\pm 0\.00640\.6006WT103 76MUniform73\.0899±0\.00553\.0899\\pm 0\.00550\.6000WT103 504MFeatureGate32\.6520±0\.00462\.6520\\pm 0\.00460\.5952WT103 504MLayerStatic32\.6547±0\.00312\.6547\\pm 0\.00310\.5953WT103 504MUniform\-Matched32\.6560±0\.00312\.6560\\pm 0\.00310\.5953WT103 504MUniform\-0\.6032\.6574±0\.00312\.6574\\pm 0\.00310\.6000Table[1](https://arxiv.org/html/2607.26192#S4.T1)shows that FeatureGate does not outperform the mass\-matched Uniform baseline on OWT at 76M scale\. The two methods have nearly identical gate mass, so the result cannot be attributed to a smaller residual budget for FeatureGate\. On WT103 at 76M scale, LayerStatic outperforms Uniform for all seven seeds, whereas FeatureGate does not consistently outperform LayerStatic\. WT103 therefore benefits from a nonuniform depth profile, but a single learned coefficient per layer captures this benefit\. Per\-token coefficient generation is not required\.

At 504M scale, FeatureGate has the lowest Byte\-PPL and outperforms mass\-matched Uniform for all three seeds, but the average gap is only 0\.004000 Byte\-PPL\. The advantage over LayerStatic is smaller\. Dynamic parameterization may provide a slight optimization benefit during training, even when a static profile closely approximates the resulting function\. The next experiment tests this distinction by intervening after the controller is frozen\.

### 4\.3Dynamic Assignment Dependence of FeatureGate

This experiment tests whether the trained FeatureGate function depends on dynamic assignment\. The audit fixes coefficient values and then breaks their correspondence with inputs or tokens, so the performance change measures the functional role of that correspondence\.

![Refer to caption](https://arxiv.org/html/2607.26192v1/x2.png)Figure 2:Frozen\-Controller Audit results for 76M\-parameter FeatureGate models on WT103\.Each point shows a mean intervention penalty\. Horizontal error bars give95%95\\%confidence intervals across seven independently trained checkpoints\. Assignment and static\-profile interventions increase NLL by only about10−310^\{\-3\}\. GlobalMean removes the learned depth structure and produces a much larger penalty, so we plot it on a separate linear scale\. All intervention effects are positive for every seed\. The MC SE values report within\-checkpoint randomization error\.Every intervention increases NLL for all seven seeds, so input identity and token alignment are not completely irrelevant\. The effects are nevertheless small\. Cross\-input increases NLL by 0\.000951, and Token\-shuffle increases it by 0\.001905\. Replacing all dynamic gates with one layerwise profile estimated from independent validation data increases NLL by only 0\.001287\. This value satisfies the prespecified threshold for a practically small effect and retains 98\.70% of the performance gap from Correct to GlobalMean\.

The contrast between LayerMean and GlobalMean explains what the static result preserves\. Both conditions remove input and token variation, but only LayerMean retains the depth profile\. Preserving this profile costs about 0\.0013 NLL, whereas flattening it costs nearly 0\.098 NLL\. The model depends strongly on where residual mass is placed across depth and only weakly on the input\-dependent and token\-dependent variation around that profile\. Layer×\\timesPositionMean reduces the penalty further to 0\.000562 NLL\. Some apparent token adaptivity therefore comes from a stable absolute\-position pattern rather than input content\.

![Refer to caption](https://arxiv.org/html/2607.26192v1/x3.png)Figure 3:Balanced functional variance analysis of FeatureGate coefficients\. Layer identity explains 87% to 96% of the variance across corpora and controller variants\. The input, token, and interaction terms are each below 1%\.The variance decomposition provides separate descriptive evidence\. Layer identity explains 87% to 96% of coefficient variance, while the input and token effects are small\. This analysis does not replace the functional interventions, but it agrees with them: the controller primarily learns stable layerwise levels\. The 504M audit follows the same pattern\. Cross\-input, Token\-shuffle, and LayerMean increase NLL by 0\.000891, 0\.001574, and 0\.001177, respectively, whereas the mass\-matched GlobalMean increases NLL by 0\.205301\. The static layerwise profile retains 99\.43% of relative performance\. Scaling increases the cost of destroying the fixed depth profile, not the cost of removing dynamic assignment\.

![Refer to caption](https://arxiv.org/html/2607.26192v1/x4.png)Figure 4:Static profile retention for FeatureGate and MUDDPythia\.FeatureGate LayerMean retains 98\.70% and 99\.43% of the Correct\-to\-GlobalMean performance gap at 76M and 504M, respectively\. Horizontal bars show 95% confidence intervals across independently trained checkpoints\. For MUDDPythia, StaticMixMean retains 73\.33% of the gap relative to SourceUniformMean\. This value is a point estimate from a single checkpoint\. The corresponding absolute penalties are reported with each retention value\. Because the two model families use different references for structure removal, retention should be interpreted within each family rather than used for direct comparison across models\.The paired factor effects areEinput=0\.000466E\_\{\\text\{input\}\}=0\.000466,Etoken=0\.001421E\_\{\\text\{token\}\}=0\.001421, andEinteraction=−0\.000968E\_\{\\text\{interaction\}\}=\-0\.000968\. The negative interaction indicates that cross\-input reassignment and token shuffling disrupt overlapping functions rather than two large, independent mechanisms whose costs add\. The absolute effects, retention ratios, position control, and variance decomposition support a narrow conclusion: FeatureGate is dynamically parameterized, but its trained inference function is nearly static\.

### 4\.4Strong Functional Dependence in MUDDPythia

The small effects for FeatureGate admit another explanation: the audit itself may be insensitive to dynamic dependence\. We test this possibility by applying the same caching, replay, reassignment, and static\-profile protocol to the public MUDDPythia\-1\.4B checkpoint\.

Table 2:FCA results for the public MUDDPythia\-1\.4B checkpoint\. Intervals are descriptive paired\-window bootstrap intervals\. MC SE is estimated from 20 paired randomizations and does not represent checkpoint\-level population uncertainty\.The effect scale differs sharply from FeatureGate\. Cross\-input increases NLL by 1\.9067 while preserving the full coefficient multiset, and Token\-shuffle increases it by 2\.9637\. These penalties are roughly three orders of magnitude larger than the corresponding effects for FeatureGate\. The coefficients determine which earlier\-layer representations enter the query, key, value, and residual streams\. Their values therefore remain useful only when aligned with the content and positions that produce them\.

Static replacement also causes large losses\. StaticMixMean increases NLL by 1\.9798\. Layer×\\timesPositionMean retains absolute\-position structure and reduces the penalty to 1\.0190, which shows that MUDDPythia also contains stable position patterns\. Those patterns do not remove the dependence on content\. SourceUniformMean increases NLL by 7\.4237, so source\-layer preference is itself an important structure\. StaticMixMean and Layer×\\timesPositionMean retain 73\.33% and 86\.27% relative to SourceUniformMean\. These ratios coexist with large absolute NLL penalties and cannot by themselves support the claim that most function is preserved\.

The factor effects areEinput=0\.974430E\_\{\\text\{input\}\}=0\.974430,Etoken=2\.031426E\_\{\\text\{token\}\}=2\.031426, andEinteraction=−1\.864572E\_\{\\text\{interaction\}\}=\-1\.864572\. The negative interaction again indicates nonadditive overlap\. More importantly, replaying the original cached coefficients exactly matches both NLL and the logits of the first batch under the normal forward pass\. The large effects therefore do not arise from an inaccurate replay implementation\. The same protocol distinguishes the two model families: dynamic variation in FeatureGate is largely replaceable by a static profile, whereas dynamic cross\-layer assignment in MUDDPythia has a substantial functional role\.

![Refer to caption](https://arxiv.org/html/2607.26192v1/x5.png)Figure 5:Absolute intervention penalties relative to Correct on a logarithmic axis\. Assignment and static\-profile penalties remain near10−310^\{\-3\}NLL for FeatureGate, while the penalties for MUDDPythia are larger by one to three orders of magnitude\.
### 4\.5Dynamic Function and Actual Execution

Functional dynamics do not imply conditional execution, so the final experiment directly measures executed blocks, latency, and throughput\. Every 76M soft\-gated model executes all 24 Transformer blocks\. With batch size 8 in the current dense PyTorch implementation, Dense takes 19\.36 ms per batch\. Uniform, LayerStatic, and FeatureGate take 23\.00, 24\.10, and 25\.31 ms, respectively\. FeatureGate is 30\.8% slower than Dense and has 23\.5% lower throughput because its controller adds computation without removing any block\.

The FixedDepth15 baseline executes 15 blocks and reduces latency by 36\.7%, but it also loses accuracy\. This comparison shows that actual skipping must appear in both the execution graph and system measurements\. Average gate values cannot establish it\. Post\-training thresholding of FeatureGate produces meaningful block skipping only when PPL degrades severely, which further shows that gate mass is a residual\-coefficient statistic rather than a compute measure\.

MUDDPythia also executes every Transformer block\. Its large FCA penalties show that dynamic cross\-layer mixing has a functional role, but they do not provide evidence of block skipping\. FeatureGate therefore exhibits coefficient variation but little functional dependence and no conditional execution\. MUDDPythia exhibits coefficient variation and strong functional dependence, but still no conditional execution\.

## 5Discussion

The two model families show why dynamic has no single inference meaning\. The controller of FeatureGate varies across inputs and tokens, and dynamic parameterization may offer a small optimization benefit at the larger scale\. Yet a static depth profile estimated from independent data nearly reproduces the trained function\. MUDDPythia, by contrast, depends strongly on the correspondence between coefficients, content, and position\. A small static\-profile penalty indicates that controller variation is largely replaceable in the frozen inference function, rather than that the controller is constant\. A large penalty establishes a functional role for the dynamic connection, but it provides no evidence of computation savings\. Studies of dynamic architectures should therefore report controller variation, frozen\-model interventions, and actual execution statistics, with separate evidence for each claim\.

## 6Limitations

Population\-level inference across seven training seeds covers only the FeatureGate family in one byte\-level decoder\-only Transformer\. The 504M study uses three seeds and a training schedule that is not compute optimal, so it supports only the direction of the scaling result\. OpenWebText has no official untouched test split, and the strongest repeated\-randomization inference therefore uses WikiText\-103\. External validation covers one MUDDPythia checkpoint and evaluates it on WikiText\-103 rather than its pretraining distribution, so the result does not generalize to all MUDDFormer training runs\. The MUDDPythia experiment evaluates dynamic cross\-layer mixing rather than block skipping and therefore validates only the sensitivity of FCA to functional dynamics\. Static profiles match coefficient structure, means, or signed sums, but they do not exactly match effective residual updates, which also depend on hidden states and block outputs\. Finally, the StructureRemoved conditions have different mathematical definitions for the two model families and do not support cross\-model causal ranking\.

## 7Conclusion

Variation in controller outputs has no unique functional interpretation\. For FeatureGate, a static layerwise profile nearly reproduces the trained inference function, assignment interventions cause only about10−310^\{\-3\}NLL loss, and soft gating does not reduce actual execution\. For the public MUDDPythia\-1\.4B checkpoint, the same audit finds large losses under cross\-input reassignment, token shuffling, and static mixing, which shows that the model materially depends on content\-conditioned cross\-layer connections\. FCA provides a diagnostic rather than a verdict on dynamic controllers\. It is a reproducible implementation of the frozen\-controller auditing principle\. Reliable claims about dynamic models require separate measurements of coefficient variation, functional dependence, and actual execution\.

## References

- T\. Bachlechner, B\. P\. Majumder, H\. Mao, G\. Cottrell, and J\. McAuley \(2021\)ReZero is all you need: fast convergence at large depth\.InProceedings of the 37th Conference on Uncertainty in Artificial Intelligence,pp\. 1352–1361\.External Links:[Link](https://proceedings.mlr.press/v161/bachlechner21a.html)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p1.1)\.
- A\. Banino, J\. Balaguer, and C\. Blundell \(2021\)PonderNet: learning to ponder\.arXiv preprint arXiv:2107\.05407\.External Links:[Link](https://arxiv.org/abs/2107.05407)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- A\. Bapna, M\. Chen, O\. Firat, Y\. Cao, and Y\. Wu \(2018\)Training deeper neural machine translation models with transparent attention\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 3028–3033\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1338)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p2.1)\.
- D\. Bolya, C\. Fu, X\. Dai, P\. Zhang, C\. Feichtenhofer, and J\. Hoffman \(2023\)Token merging: your vit but faster\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=JroZRaRw7Eu)Cited by:[§2\.4](https://arxiv.org/html/2607.26192#S2.SS4.p1.1)\.
- Y\. Chen, X\. Dai, M\. Liu, D\. Chen, L\. Yuan, and Z\. Liu \(2020\)Dynamic convolution: attention over convolution kernels\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11030–11039\.External Links:[Link](https://openaccess.thecvf.com/content_CVPR_2020/html/Chen_Dynamic_Convolution_Attention_Over_Convolution_Kernels_CVPR_2020_paper.html)Cited by:[§2\.1](https://arxiv.org/html/2607.26192#S2.SS1.p1.1)\.
- S\. De and S\. Smith \(2020\)Batch normalization biases residual blocks towards the identity function in deep networks\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/e6b738eca0e6792ba8a9cbcba6c1881d-Abstract.html)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p1.1)\.
- M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and L\. Kaiser \(2019\)Universal transformers\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1807.03819)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- N\. Du, Y\. Huang, A\. M\. Dai, S\. Tong, D\. Lepikhin, Y\. Xu, M\. Krikun, Y\. Zhou, A\. W\. Yu, O\. Firat, B\. Zoph, L\. Fedus, M\. P\. Bosma, Z\. Zhou, T\. Wang, E\. Wang, K\. Webster, M\. Pellat, K\. Robinson, K\. Meier\-Hellstern, T\. Duke, L\. Dixon, K\. Zhang, Q\. V\. Le, Y\. Wu, Z\. Chen, and C\. Cui \(2022\)GLaM: efficient scaling of language models with mixture\-of\-experts\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 5547–5569\.External Links:[Link](https://proceedings.mlr.press/v162/du22c.html)Cited by:[§2\.3](https://arxiv.org/html/2607.26192#S2.SS3.p1.1)\.
- M\. Elbayad, J\. Gu, E\. Grave, and M\. Auli \(2020\)Depth\-adaptive transformer\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1910.10073)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- A\. Fan, E\. Grave, and A\. Joulin \(2020\)Reducing transformer depth on demand with structured dropout\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SylO2yStDr)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- M\. Fayyaz, S\. A\. Koohpayegani, F\. R\. Jafari, S\. Sengupta, H\. R\. V\. Joze, E\. Sommerlade, H\. Pirsiavash, and J\. Gall \(2022\)Adaptive token sampling for efficient vision transformers\.InEuropean Conference on Computer Vision,External Links:[Link](https://www.ecva.net/papers/eccv_2022/papers_ECCV/html/4901_ECCV_2022_paper.php)Cited by:[§2\.4](https://arxiv.org/html/2607.26192#S2.SS4.p1.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.External Links:[Link](https://www.jmlr.org/beta/papers/v23/21-0998.html)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.26192#S2.SS3.p1.1)\.
- A\. Geiger, Z\. Wu, H\. Lu, J\. Rozner, E\. Kreiss, T\. Icard, N\. Goodman, and C\. Potts \(2022\)Inducing causal structure for interpretable neural networks\.InProceedings of the 39th International Conference on Machine Learning,pp\. 7324–7338\.External Links:[Link](https://proceedings.mlr.press/v162/geiger22a.html)Cited by:[§2\.6](https://arxiv.org/html/2607.26192#S2.SS6.p1.1)\.
- A\. Gokaslan and V\. Cohen \(2019\)OpenWebText corpus\.Note:Open\-source reproduction of the WebText corpusExternal Links:[Link](https://skylion007.github.io/OpenWebTextCorpus/)Cited by:[§4\.1](https://arxiv.org/html/2607.26192#S4.SS1.p1.2)\.
- A\. Graves \(2016\)Adaptive computation time for recurrent neural networks\.arXiv preprint arXiv:1603\.08983\.External Links:[Link](https://arxiv.org/abs/1603.08983)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- D\. Ha, A\. M\. Dai, and Q\. V\. Le \(2017\)HyperNetworks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rkpACe1lx)Cited by:[§2\.1](https://arxiv.org/html/2607.26192#S2.SS1.p1.1)\.
- Y\. Han, G\. Huang, S\. Song, L\. Yang, H\. Wang, and Y\. Wang \(2022\)Dynamic neural networks: a survey\.IEEE Transactions on Pattern Analysis and Machine Intelligence44\(11\),pp\. 7436–7456\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2021.3117837)Cited by:[§2\.1](https://arxiv.org/html/2607.26192#S2.SS1.p1.1)\.
- R\. He, A\. Ravula, B\. Kanagal, and J\. Ainslie \(2021\)RealFormer: transformer likes residual attention\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 929–943\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.81)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p2.1)\.
- L\. Hou, Z\. Huang, L\. Shang, X\. Jiang, X\. Chen, and Q\. Liu \(2020\)DynaBERT: dynamic bert with adaptive width and depth\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/hash/6f5216f8d89b086c18298e043bfe48ed-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- G\. Huang, D\. Chen, T\. Li, F\. Wu, L\. van der Maaten, and K\. Q\. Weinberger \(2018\)Multi\-scale dense networks for resource efficient image classification\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Hk2aImxAb)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- G\. Huang, Z\. Liu, L\. van der Maaten, and K\. Q\. Weinberger \(2017\)Densely connected convolutional networks\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 4700–4708\.External Links:[Link](https://openaccess.thecvf.com/content_cvpr_2017/html/Huang_Densely_Connected_Convolutional_CVPR_2017_paper.html)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p2.1)\.
- G\. Huang, Y\. Sun, Z\. Liu, D\. Sedra, and K\. Q\. Weinberger \(2016\)Deep networks with stochastic depth\.InEuropean Conference on Computer Vision,External Links:[Document](https://dx.doi.org/10.1007/978-3-319-46493-0%5F39)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p1.1)\.
- X\. Jia, B\. D\. Brabandere, T\. Tuytelaars, and L\. V\. Gool \(2016\)Dynamic filter networks\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper/2016/hash/8bf1211fd4b7b94528899de0a43b9fb3-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2607.26192#S2.SS1.p1.1)\.
- Y\. Kaya, S\. Hong, and T\. Dumitras \(2019\)Shallow\-deep networks: understanding and mitigating network overthinking\.InProceedings of the 36th International Conference on Machine Learning,pp\. 3301–3310\.External Links:[Link](https://proceedings.mlr.press/v97/kaya19a.html)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen \(2021\)GShard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations,External Links:[Link](https://iclr.cc/virtual/2021/poster/3196)Cited by:[§2\.3](https://arxiv.org/html/2607.26192#S2.SS3.p1.1)\.
- M\. Lewis, S\. Bhosale, T\. Dettmers, N\. Goyal, and L\. Zettlemoyer \(2021\)BASE layers: simplifying training of large, sparse models\.InProceedings of the 38th International Conference on Machine Learning,pp\. 6265–6274\.External Links:[Link](https://proceedings.mlr.press/v139/lewis21a.html)Cited by:[§2\.3](https://arxiv.org/html/2607.26192#S2.SS3.p1.1)\.
- Y\. Liang, C\. Ge, Z\. Tong, Y\. Song, J\. Wang, and P\. Xie \(2022\)Not all patches are what you need: expediting vision transformers via token reorganizations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/pdf/feb0c5a2e1c1fc63509c2e528ca07aa95aea2d5e.pdf)Cited by:[§2\.4](https://arxiv.org/html/2607.26192#S2.SS4.p1.1)\.
- W\. Liu, P\. Zhou, Z\. Wang, Z\. Zhao, H\. Deng, and Q\. Ju \(2020\)FastBERT: a self\-distilling bert with adaptive inference time\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 6035–6044\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.537)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022a\)Locating and editing factual associations in gpt\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html)Cited by:[§2\.6](https://arxiv.org/html/2607.26192#S2.SS6.p1.1)\.
- L\. Meng, H\. Li, B\. Chen, S\. Lan, Z\. Wu, Y\. Jiang, and S\. Lim \(2022b\)AdaViT: adaptive vision transformers for efficient image recognition\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12309–12318\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2022/papers/Meng_AdaViT_Adaptive_Vision_Transformers_for_Efficient_Image_Recognition_CVPR_2022_paper.pdf)Cited by:[§2\.4](https://arxiv.org/html/2607.26192#S2.SS4.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1609.07843)Cited by:[§4\.1](https://arxiv.org/html/2607.26192#S4.SS1.p1.2)\.
- P\. Michel, O\. Levy, and G\. Neubig \(2019\)Are sixteen heads really better than one?\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/hash/2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html)Cited by:[§2\.6](https://arxiv.org/html/2607.26192#S2.SS6.p1.1)\.
- M\. Pagliardini, A\. Mohtashami, F\. Fleuret, and M\. Jaggi \(2024\)DenseFormer: enhancing information flow in transformers via depth weighted averaging\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/f67449c7ab72f441d3a713b046c6818c-Abstract-Conference.html)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p2.1)\.
- Y\. Rao, W\. Zhao, B\. Liu, J\. Lu, J\. Zhou, and C\. Hsieh \(2021\)DynamicViT: efficient vision transformers with dynamic token sparsification\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/hash/747d3443e319a22747fbb873e8b2f9f2-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.4](https://arxiv.org/html/2607.26192#S2.SS4.p1.1)\.
- D\. Raposo, S\. Ritter, B\. Richards, T\. Lillicrap, P\. C\. Humphreys, and A\. Santoro \(2024\)Mixture\-of\-depths: dynamically allocating compute in transformer\-based language models\.arXiv preprint arXiv:2404\.02258\.External Links:[Link](https://arxiv.org/abs/2404.02258)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- C\. Riquelme, J\. Puigcerver, B\. Mustafa, M\. Neumann, R\. Jenatton, A\. S\. Pinto, D\. Keysers, and N\. Houlsby \(2021\)Scaling vision with sparse mixture of experts\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/hash/48237d9f2dea8c74c2a72126cf63d933-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2607.26192#S2.SS3.p1.1)\.
- T\. Schuster, A\. Fisch, J\. Gupta, M\. Dehghani, D\. Bahri, V\. Q\. Tran, Y\. Tay, and D\. Metzler \(2022\)Confident adaptive language modeling\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/6fac9e316a4ae75ea244ddcef1982c71-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. V\. Le, G\. E\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.26192#S2.SS3.p1.1)\.
- S\. Teerapittayanon, B\. McDanel, and H\. T\. Kung \(2016\)BranchyNet: fast inference via early exiting from deep neural networks\.In23rd International Conference on Pattern Recognition,pp\. 2464–2469\.External Links:[Document](https://dx.doi.org/10.1109/ICPR.2016.7900006)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- H\. Touvron, M\. Cord, A\. Sablayrolles, G\. Synnaeve, and H\. Jégou \(2021\)Going deeper with image transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 32–42\.External Links:[Link](https://openaccess.thecvf.com/content/ICCV2021/html/Touvron_Going_Deeper_With_Image_Transformers_ICCV_2021_paper.html)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p4.1)\.
- J\. Vig, S\. Gehrmann, Y\. Belinkov, S\. Qian, D\. Nevo, S\. Sakenis, J\. Huang, Y\. Singer, and S\. Shieber \(2020\)Causal mediation analysis for interpreting neural nlp: the case of gender bias\.arXiv preprint arXiv:2004\.12265\.External Links:[Link](https://arxiv.org/abs/2004.12265)Cited by:[§2\.6](https://arxiv.org/html/2607.26192#S2.SS6.p1.1)\.
- H\. Wang, S\. Ma, L\. Dong, S\. Huang, D\. Zhang, and F\. Wei \(2022\)DeepNet: scaling transformers to 1,000 layers\.arXiv preprint arXiv:2203\.00555\.External Links:[Link](https://arxiv.org/abs/2203.00555)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p1.1)\.
- Q\. Wang, B\. Li, T\. Xiao, J\. Zhu, C\. Li, D\. F\. Wong, and L\. S\. Chao \(2019\)Learning deep transformer models for machine translation\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 1810–1822\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1176)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p2.1)\.
- X\. Wang, F\. Yu, Z\. Dou, T\. Darrell, and J\. E\. Gonzalez \(2018\)SkipNet: learning dynamic routing in convolutional networks\.InProceedings of the European Conference on Computer Vision,pp\. 409–424\.External Links:[Link](https://openaccess.thecvf.com/content_ECCV_2018/html/Xin_Wang_SkipNet_Learning_Dynamic_ECCV_2018_paper.html)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- Y\. Wang, R\. Huang, S\. Song, Z\. Huang, and G\. Huang \(2021\)Not all images are worth 16x16 words: dynamic transformers for efficient image recognition\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/64517d8435994992e682b3e4aa0a0661-Abstract.html)Cited by:[§2\.4](https://arxiv.org/html/2607.26192#S2.SS4.p1.1)\.
- Z\. Wu, T\. Nagarajan, A\. Kumar, S\. Rennie, L\. S\. Davis, K\. Grauman, and R\. Feris \(2018\)BlockDrop: dynamic inference paths in residual networks\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 8817–8826\.External Links:[Link](https://openaccess.thecvf.com/content_cvpr_2018/html/Wu_BlockDrop_Dynamic_Inference_CVPR_2018_paper.html)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p1.1)\.
- D\. Xiao, Q\. Meng, S\. Li, and X\. Yuan \(2025\)MUDDFormer: breaking residual bottlenecks in transformers via multiway dynamic dense connections\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 68440–68458\.External Links:[Link](https://proceedings.mlr.press/v267/xiao25d.html)Cited by:[§1](https://arxiv.org/html/2607.26192#S1.p4.1),[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p2.1)\.
- J\. Xin, R\. Tang, J\. Lee, Y\. Yu, and J\. Lin \(2020\)DeeBERT: dynamic early exiting for accelerating bert inference\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 2246–2251\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.204)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- J\. Xin, R\. Tang, Y\. Yu, and J\. Lin \(2021\)BERxiT: early exiting for bert with better fine\-tuning and extension to regression\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 91–104\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.8)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- B\. Yang, G\. Bender, Q\. V\. Le, and J\. Ngiam \(2019\)CondConv: conditionally parameterized convolutions for efficient inference\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/f2201f5191c4e92cc5af043eebfd0946-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2607.26192#S2.SS1.p1.1)\.
- H\. Yin, A\. Vahdat, J\. M\. Alvarez, A\. Mallya, J\. Kautz, and P\. Molchanov \(2022\)A\-vit: adaptive tokens for efficient vision transformer\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10809–10818\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2022/html/Yin_A-ViT_Adaptive_Tokens_for_Efficient_Vision_Transformer_CVPR_2022_paper.html)Cited by:[§2\.4](https://arxiv.org/html/2607.26192#S2.SS4.p1.1)\.
- H\. Zhang, Y\. N\. Dauphin, and T\. Ma \(2019\)Fixup initialization: residual learning without normalization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=H1gsz30cKX)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p1.1)\.
- W\. Zhou, C\. Xu, T\. Ge, J\. McAuley, K\. Xu, and F\. Wei \(2020\)BERT loses patience: fast and robust inference with early exit\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/hash/d4dd111a4fd973394238aca5c05bebe3-Abstract.html)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.
- Y\. Zhou, T\. Lei, H\. Liu, N\. Du, Y\. Huang, V\. Y\. Zhao, A\. M\. Dai, Z\. Chen, Q\. V\. Le, and J\. Laudon \(2022\)Mixture\-of\-experts with expert choice routing\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abstract-Conference.html)Cited by:[§2\.3](https://arxiv.org/html/2607.26192#S2.SS3.p1.1)\.
- D\. Zhu, H\. Huang, Z\. Huang, Y\. Zeng, Y\. Mao, B\. Wu, Q\. Min, and X\. Zhou \(2025\)Hyper\-connections\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=9FqARW7dwB)Cited by:[§2\.5](https://arxiv.org/html/2607.26192#S2.SS5.p2.1)\.
- W\. Zhu \(2021\)LeeBERT: learned early exit for bert with cross\-level optimization\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,pp\. 2968–2980\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.231)Cited by:[§2\.2](https://arxiv.org/html/2607.26192#S2.SS2.p2.1)\.