Feature Starvation as Geometric Instability in Sparse Autoencoders

# Feature Starvation as Geometric Instability in Sparse Autoencoders
Source: [https://arxiv.org/html/2605.05341](https://arxiv.org/html/2605.05341)
Faris Chaudhry Imperial College London South Kensington Campus, London SW7 2AZ, UK faris\.chaudhry22@imperial\.ac\.uk &Keisuke Yano Institute for Statistical Mathematics 10\-3 Midori\-cho, Tachikawa Tokyo 190\-8562, Japan yano@ism\.ac\.jp Anthea Monod Imperial College London South Kensington Campus, London SW7 2AZ, UK a\.monod@imperial\.ac\.uk

###### Abstract

Sparse autoencoders \(SAEs\) are used to disentangle the dense, polysemantic internal representations of large language models \(LLMs\) into interpretable, monosemantic concepts\. However, standardℓ1\\ell\_\{1\}\-regularized SAEs suffer from feature starvation \(dead neurons\) and shrinkage bias, often requiring computationally expensive heuristic resampling and nondifferentiable hard\-masking methods to bypass these challenges\. We argue that feature starvation is not merely an empirical artifact of poor data diversity, but a fundamental optimization\-geometric pathology of overcomplete dictionaries: theℓ1\\ell\_\{1\}\-induced sparse coding map is unstable and fundamentally misaligned with shallow, amortized encoders\. To address this structural instability, we introduce adaptive elastic net SAEs \(AEN\-SAEs\), a fully differentiable architecture grounded in classical sparse regression\. AEN\-SAEs combine anℓ2\\ell\_\{2\}structural term that enforces strong convexity and Lipschitz stability with adaptiveℓ1\\ell\_\{1\}reweighting that eliminates shrinkage bias and suppresses spurious features, thereby jointly controlling the curvature and interaction structure of the induced polyhedral geometry\. Theoretically, we show that AEN\-SAEs yield a Lipschitz\-continuous sparse coding map and recover the global feature support under mild assumptions\. Empirically, across synthetic settings and LLMs \(Pythia 70M, Llama 3\.1 8B\), AEN\-SAEs mitigate feature starvation without auxiliary heuristics while maintaining competitive reconstruction abilities\.

## 1Introduction

Large language models \(LLMs\) encode complex, polysemantic concepts within their dense residual streams, rendering their internal decision\-making processes opaque\. Recently, sparse autoencoders \(SAEs\) have emerged as a primary technique in mechanistic interpretability to linearly disentangle these representations into interpretable, monosemantic features\[[10](https://arxiv.org/html/2605.05341#bib.bib3),[4](https://arxiv.org/html/2605.05341#bib.bib1),[9](https://arxiv.org/html/2605.05341#bib.bib2)\]\. By training an overcomplete dictionary with anℓ1\\ell\_\{1\}sparsity penalty, SAEs project dense activations into a higher\-dimensional space where individual basis directions correspond to human\-understandable concepts\.

The specific setup of an SAE intervention is illustrated in Figure[1](https://arxiv.org/html/2605.05341#S1.F1), building upon classical autoencoder frameworks for representation learning\[[18](https://arxiv.org/html/2605.05341#bib.bib4)\]\. Given a pretrained, frozen LLM withFFlayers, input tokensX0X\_\{0\}are processed up to a chosen intermediate layerMMto extract a dense representationXMX\_\{M\}\. Middle layers are specifically targeted, as empirical evidence suggests this is where the most complex, polysemantic reasoning occurs, while earlier and later layers are heavily biased toward raw lexical processing and logit unembedding, respectively\[[15](https://arxiv.org/html/2605.05341#bib.bib5)\]\. The SAE encoder projectsXMX\_\{M\}into a sparse, high\-dimensional latent spaceh​\(XM\)h\(X\_\{M\}\)\. The decoder then maps this projection back to the original dimension to produce the reconstructed activationX^M\\hat\{X\}\_\{M\}\. By substituting the original activations with these reconstructionsX^M\\hat\{X\}\_\{M\}and measuring downstream recovery, individual basis directions corresponding to human\-understandable concepts can be isolated and interpreted\.

![Refer to caption](https://arxiv.org/html/2605.05341v1/figures/sae_diagram.png)Figure 1:Sparse autoencoder setup\.The SAE compresses intermediate LLM activationsXMX\_\{M\}into a sparse representationh​\(XM\)h\(X\_\{M\}\)and reconstructs them asX^M\\hat\{X\}\_\{M\}\. To evaluate the dictionary’s fidelity, the true activations are replaced withX^M\\hat\{X\}\_\{M\}, and the forward pass is resumed through the frozen LLM to yield the reconstructed final logitsX^F\\hat\{X\}\_\{F\}\. Performance is measured by the divergence between the original outputsXFX\_\{F\}andX^F\\hat\{X\}\_\{F\}\.Despite their empirical success, standardℓ1\\ell\_\{1\}\-regularized SAEs suffer from severe optimization pathologies—most notably,feature starvation\(dead neurons\) andmagnitude shrinkage\(systematic underestimation of feature activations\)\. While various architectural patches have been proposed to circumvent this \(ranging from computationally expensive resampling heuristics to nondifferentiable hard\-masking\), these techniques treat the symptoms rather than addressing the underlying geometric root cause\. In practice, such approaches rely heavily on auxiliary mechanisms—such as proxy gradients, resampling strategies, or additional loss terms—to prevent feature collapse\. These interventions are often computationally expensive, introduce additional complexity, and can substantially increase the training cost required to reach competitive performance\.

In this work, we show that feature starvation arises from the polyhedral geometry of the underlying sparse coding problem\. Training an SAE is an attempt to approximate the computationally heavy least absolute shrinkage and selection operator \(LASSO\) objective\[[37](https://arxiv.org/html/2605.05341#bib.bib17)\]in a single, amortized forward pass\. However, standardℓ1\\ell\_\{1\}formulations induce an unstable and discontinuous sparse coding map, which is fundamentally misaligned with shallow, amortized encoders and leads to systematic training failure\. To address these challenges, we introduce*adaptive elastic net sparse autoencoders*\(AEN\-SAEs\), a fully differentiable architecture that reduces the need for auxiliary heuristics while improving computational efficiency\. Drawing on classical sparse regression, AEN\-SAEs apply an adaptiveℓ1\\ell\_\{1\}penalty that vanishes for true signals to eliminate shrinkage bias\. Additionally, we couple this with a constantℓ2\\ell\_\{2\}structural anchor\. Thisℓ2\\ell\_\{2\}penalty ensures strong convexity and explicitly bounds the stability constant of the sparse coding map\. Consequently, the AEN\-SAE can be interpreted as a form of diagonal reweighting analogous to preconditioning, yielding a stable and fully differentiable architecture that naturally avoids dead features\. Our main contributions are as follows:

- •Theoretical grounding:We formalize the feature starvation pathology of standard SAEs through the lens of sparse recovery\. We show that our AEN\-SAE formulation satisfies oracle\-like selection consistency while explicitly bounding the stability constant to guarantee polyhedral stability in the activation space\.
- •Architectural efficiency:We introduce a scalable, streaming mechanism for computing adaptive LASSO weights without requiring computationally expensive proxy gradients\.
- •Empirical scaling:Across controlled synthetic environments and real\-world models \(Pythia 70M\[[3](https://arxiv.org/html/2605.05341#bib.bib10)\], Llama 3\.1 8B\[[16](https://arxiv.org/html/2605.05341#bib.bib9)\]\), we demonstrate that AEN\-SAEs mitigate dead features and achieve competitive reconstruction curves compared to baselines\.

## 2Background and Related Work

We review the relevant background on SAEs and classical sparse regression and introduce the geometric perspective that underpins our approach\. In particular, we connect SAE training to amortized sparse recovery and highlight the stability limitations ofℓ1\\ell\_\{1\}\-based formulations that motivate our method\.

### 2\.1Mechanistic Interpretability and SAEs

The superposition hypothesis posits that neural networks encode more features than their ambient dimensionality by packing them into approximately orthogonal directions in activation space\[[10](https://arxiv.org/html/2605.05341#bib.bib3)\]\. SAEs aim to decode this structure by learning an overcomplete dictionaryD∈ℝdmodel×ddictD\\in\\mathbb\{R\}^\{d\_\{\\textit\{model\}\}\\times d\_\{\\textit\{dict\}\}\}\(withddict≫dmodeld\_\{\\textit\{dict\}\}\\gg d\_\{\\textit\{model\}\}\) that reconstructs the residual streamx∈ℝdmodelx\\in\\mathbb\{R\}^\{d\_\{\\textit\{model\}\}\}as a sparse linear combination of feature directions\.

The standard SAE objective takes the form of anℓ1\\ell\_\{1\}\-regularized reconstruction loss:

ℒSAE=𝔼x​\[12​‖x−\(D​h​\(x\)\+bdec\)‖22\+λ1​‖h​\(x\)‖1\],\\mathcal\{L\}\_\{\\text\{SAE\}\}=\\mathbb\{E\}\_\{x\}\\left\[\\frac\{1\}\{2\}\\\|x\-\(Dh\(x\)\+b\_\{\\text\{dec\}\}\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{1\}\\\|h\(x\)\\\|\_\{1\}\\right\],whereh​\(x\)=ReLU​\(Wenc​x\+benc\)h\(x\)=\\text\{ReLU\}\(W\_\{\\text\{enc\}\}x\+b\_\{\\text\{enc\}\}\)denotes the encoder’s predicted sparse activation\. In practice, optimizing this objective via gradient descent leads to two characteristic pathologies:shrinkage biasandfeature starvation\. Theℓ1\\ell\_\{1\}penalty systematically suppresses the magnitude of active features and prevents them from reaching their true magnitudes\. Simultaneously, the encoder may fail to activate certain features altogether, trapping their preactivations in the negative regime\. Since the derivative of the ReLU vanishes on negative inputs, these features receive no gradient signal and remain permanently inactive, effectively becoming*dead neurons*\. This phenomenon is known as*feature starvation*and is the central failure mode we address in this work\.

Early attempts to mitigate feature starvation relied on heuristic resampling of dead neurons\[[4](https://arxiv.org/html/2605.05341#bib.bib1)\]\. To avoid full dictionary resets, subsequent work introducedGhost Gradients—an empirical heuristic that provides a proxy gradient signal to inactive features\. Variants of this approach have become standard in large\-scale SAE training, where auxiliary losses are used to artificially revive dead neurons\[[14](https://arxiv.org/html/2605.05341#bib.bib6)\]\. While effective in practice, these methods are fundamentally heuristic: they introduce additional optimization pathways that are not derived from the underlying objective, require maintaining auxiliary gradient buffers, and artificially route loss, thereby increasing computational overhead\.

Architectural innovations have sought to decouple the two failure modes ofℓ1\\ell\_\{1\}\. Gated SAEs separate feature selection from magnitude estimation, reducing shrinkage bias\[[32](https://arxiv.org/html/2605.05341#bib.bib7)\]\. More recently, hard\-masking approaches such as JumpReLU\[[33](https://arxiv.org/html/2605.05341#bib.bib43),[24](https://arxiv.org/html/2605.05341#bib.bib42)\], TopK\[[14](https://arxiv.org/html/2605.05341#bib.bib6)\], and BatchTopK\[[5](https://arxiv.org/html/2605.05341#bib.bib13)\]enforce a strictlyKK\-sparse bottleneck and have emerged as state\-of\-the\-art in efficiency\. However, hard truncation exacerbates feature starvation, and these methods still rely heavily on auxiliary losses to prevent feature collapse\[[14](https://arxiv.org/html/2605.05341#bib.bib6)\]\. Related approaches, such as feature choice SAEs\[[1](https://arxiv.org/html/2605.05341#bib.bib14)\], introduce additional routing mechanisms to achieve similar effects\.

While empirically effective, hard\-masking approaches introduce the fundamental theoretical limitation of nondifferentiability\. Enforcing exact sparsity corresponds to solving anℓ0\\ell\_\{0\}\-constrained problem, which is combinatorial and NP\-hard\[[12](https://arxiv.org/html/2605.05341#bib.bib11)\]\. Recent work\[[22](https://arxiv.org/html/2605.05341#bib.bib15)\]has begun to incorporate ideas from sparse recovery theory into SAE design, showing that standard training can cause features to absorb or wedge together\[[7](https://arxiv.org/html/2605.05341#bib.bib16)\]\. These methods mitigate such effects by explicitly regularizing the dictionary to maintain quasi\-orthogonality \(low mutual coherence\)\. Similarly, contemporary architectural alternatives have sought to resolve feature absorption through hierarchical constraints\. Matryoshka SAEs\[[6](https://arxiv.org/html/2605.05341#bib.bib44)\]force smaller nested dictionaries to learn independent, high\-level concepts without relying on larger dictionaries, while subsequent extensions use attribution\-guided distillation to freeze core features and reduce redundancy\[[27](https://arxiv.org/html/2605.05341#bib.bib45)\]\. However, like previous methods, these approaches still rely heavily on nondifferentiable hard\-masking mechanisms\.

### 2\.2Classical Polyhedral Sparse Regression

The behavior ofℓ1\\ell\_\{1\}\-regularized SAEs is closely related to classical sparse recovery\[[12](https://arxiv.org/html/2605.05341#bib.bib11),[38](https://arxiv.org/html/2605.05341#bib.bib12)\]\. For a fixed dictionaryDD, the ideal sparse representationh∗​\(x\)h^\{\*\}\(x\)is given by the*LASSO problem*\[[37](https://arxiv.org/html/2605.05341#bib.bib17)\]:

h∗​\(x\)=argminh​\(x\)∈ℝddict⁡12​‖x−D​h​\(x\)‖22\+λ1​‖h​\(x\)‖1\.h^\{\*\}\(x\)=\\operatorname\{argmin\}\_\{h\(x\)\\in\\mathbb\{R\}^\{d\_\{\\text\{dict\}\}\}\}\\frac\{1\}\{2\}\\\|x\-Dh\(x\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{1\}\\\|h\(x\)\\\|\_\{1\}\.\(1\)The goal is to recover the*active set*𝒜=\{i:hi∗≠0\}\\mathcal\{A\}=\\\{i:h^\{\*\}\_\{i\}\\neq 0\\\}, whose stability is governed by geometric properties of the dictionary—in particular mutual incoherence and the conditioning of the active Gram matrix\.

Classical recovery guarantees for LASSO rely on the*irrepresentability condition*\(IRC\)\[[38](https://arxiv.org/html/2605.05341#bib.bib12),[40](https://arxiv.org/html/2605.05341#bib.bib18)\], which requires that inactive features are not too correlated with the active set\. In overcomplete LLM dictionaries, this condition is necessarily violated, leading theℓ1\\ell\_\{1\}penalty to suppress true feature magnitudes in order to avoid selecting correlated noise\.*Adaptive LASSO*\[[43](https://arxiv.org/html/2605.05341#bib.bib20)\]addresses this suppression by reweighting the penalty,wi∝\|h^i\|−γw\_\{i\}\\propto\|\\hat\{h\}\_\{i\}\|^\{\-\\gamma\}with an initial estimateh^\\hat\{h\}, achieving oracle support recovery where the active set is perfectly isolated and shrinkage bias is reduced\. However, it requires a two\-stage procedure—an initial estimate followed by reweighted optimization—which is incompatible with efficient amortized inference in LLMs\. While online variants exist\[[26](https://arxiv.org/html/2605.05341#bib.bib21)\], they do not directly translate to the highly parallel, low\-overhead setting required for LLMs\.

A second factor related to stability is the curvature of the active solution, determined by the conditioning of the active Gram matrixD𝒜⊤​D𝒜D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\. When active features are highly collinear, this matrix becomes ill\-conditioned, flattening the loss landscape and leading to unstable solutions\. Elastic net regularization\[[41](https://arxiv.org/html/2605.05341#bib.bib19)\]addresses this problem by introducing anℓ2\\ell\_\{2\}penalty \(λ2​‖h‖22\\lambda\_\{2\}\\\|h\\\|\_\{2\}^\{2\}\), which enforces strong convexity\. However, it comes at the cost of increased shrinkage and feature grouping, which can degrade interpretability\.

### 2\.3Amortized LASSO and Polyhedral Geometry

We interpret SAE training as learning an amortized approximation to the LASSO solution map: For a fixed dictionaryDD, the encoderh​\(x\)h\(x\)aims to predict the optimal sparse codeh∗​\(x\)h^\{\*\}\(x\)for each inputxxin a single forward pass\.

Standard approaches to amortized sparse recovery unroll iterative solvers into deep architectures such as LISTA\[[17](https://arxiv.org/html/2605.05341#bib.bib25),[8](https://arxiv.org/html/2605.05341#bib.bib26)\]\. However, these deep architectures are computationally intractable at the scale of large language models\. In contrast, SAEs rely on a strictly shallow architecture, typically a single linear projection followed by a ReLU—placing strong constraints on the complexity of the mappings they can represent\. Our work addresses the resulting challenge of ensuring that theℓ1\\ell\_\{1\}\-induced solution map is sufficiently stable to be learned by such shallow encoders\.

A key distinction between classical sparse recovery and the SAE setting has to do with the role of sparsity\. In classical compressed sensing, the sparsity level is an unknown, fixed property of the data\-generating process\. In contrast, in mechanistic interpretability, sparsity is typically a design choice, selected to balance interpretability and reconstruction\.

The ability to learn a shallow amortized map depends critically on the stability of the target solution\. Under anℓ1\\ell\_\{1\}penalty, the solution space is polyhedral, and the sensitivity of the optimal sparse codesh∗​\(x\)h^\{\*\}\(x\)to perturbations inxxis governed by a local error bound characterized by theHoffman constantHH\[[20](https://arxiv.org/html/2605.05341#bib.bib23),[34](https://arxiv.org/html/2605.05341#bib.bib24)\]\. This constant depends on two geometric properties of the dictionary: the conditioning of the active Gram matrix, viaλmin​\(D𝒜⊤​D𝒜\)\\lambda\_\{\\min\}\(D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\), and the interaction between active and inactive features, measured by‖D𝒜c⊤​D𝒜‖2\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\\\|\_\{2\}\. For a given active set, it scales as

H𝒜∼𝒪​\(‖D𝒜c⊤​D𝒜‖2⋅1λmin​\(D𝒜⊤​D𝒜\)\)\.H\_\{\\mathcal\{A\}\}\\sim\\mathcal\{O\}\\left\(\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\\\|\_\{2\}\\cdot\\frac\{1\}\{\\lambda\_\{\\min\}\(D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\)\}\\right\)\.As the dictionary becomes more coherent,HHdiverges, and the solution map becomes increasingly nonsmooth\. This instability makes the mapping difficult for shallow encoders to approximate\.

## 3Adaptive Elastic Net SAEs \(AEN\-SAEs\)

To address the geometric instability ofℓ1\\ell\_\{1\}\-based sparse coding, we introduce*adaptive elastic net sparse autoencoders*\(AEN\-SAEs\), which combine adaptive sparsity with explicit curvature control\. The key idea is to stabilize the amortized solution map by simultaneously suppressing spurious feature interactions and enforcing strong convexity\.

Leth​\(x\)∈ℝddicth\(x\)\\in\\mathbb\{R\}^\{d\_\{\\text\{dict\}\}\}denote the sparse latent representation predicted by the encoder for an inputx∈ℝdmodelx\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}\. We define the AEN\-SAE objective as:

ℒA​E​N=𝔼x​\[12​‖x−\(D​h​\(x\)\+bdec\)‖22\+λ1​∑i=1ddictwi​\|hi​\(x\)\|\+λ2​‖h​\(x\)‖22\]\.\\mathcal\{L\}\_\{AEN\}=\\mathbb\{E\}\_\{x\}\\left\[\\frac\{1\}\{2\}\\\|x\-\(Dh\(x\)\+b\_\{\\text\{dec\}\}\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{1\}\\sum\_\{i=1\}^\{d\_\{\\text\{dict\}\}\}w\_\{i\}\|h\_\{i\}\(x\)\|\+\\lambda\_\{2\}\\\|h\(x\)\\\|\_\{2\}^\{2\}\\right\]\.\(2\)Here,wiw\_\{i\}denotes a feature\-specific penalty weight for theii\-th feature, andλ1,λ2\>0\\lambda\_\{1\},\\lambda\_\{2\}\>0control the sparsity and structural regularization, respectively\. Compared to the standard SAE objective, this replaces the uniformℓ1\\ell\_\{1\}penalty with an adaptive weightedℓ1\\ell\_\{1\}penalty to isolate the active set, and introduces a fixedℓ2\\ell\_\{2\}structural anchor to bound active curvature and ensure stability through strong convexity\.

A direct implementation of adaptive LASSO requires a two\-pass algorithm, which is computationally infeasible for streaming LLM tokens\. Instead, we implement a lightweight online approximation based on an exponential moving average \(EMA\) of feature activations\. Leth\(t\)h^\{\(t\)\}denote the post\-ReLU sparse activations at training stepttacross a batch of sizeBB\. We update the EMA synchronously:

h¯\(t\)=β​h¯\(t−1\)\+\(1−β\)​1B​∑b=1B\|hb\(t\)\|,\\bar\{h\}^\{\(t\)\}=\\beta\\bar\{h\}^\{\(t\-1\)\}\+\(1\-\\beta\)\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\|h\_\{b\}^\{\(t\)\}\|,whereβ\\betais the momentum factor\.

To ensure robustness across layers and models, we employ a scale\-invariant adaptive weighting mechanism\. Letp∈\(0,1\]p\\in\(0,1\]denote a top\-percentile hyperparameter\. At steptt, let𝒮p\(t\)\\mathcal\{S\}\_\{p\}^\{\(t\)\}be the index set of the topp⋅dd​i​c​tp\\cdot d\_\{dict\}features with the largest EMA activations\. We define a reference activation as the mean activity over this set\. The raw adaptive weight for featureiiis then calculated asynchronously relative to this reference:

h¯ref\(t\)=1\|𝒮p\(t\)\|​∑j∈𝒮p\(t\)h¯j\(t\)andwi\(t\)=clip​\(\(h¯ref\(t\)h¯i\(t\)\+ϵ\)γ,wmin,wmax\)\.\\bar\{h\}\_\{\\text\{ref\}\}^\{\(t\)\}=\\frac\{1\}\{\|\\mathcal\{S\}\_\{p\}^\{\(t\)\}\|\}\\sum\_\{j\\in\\mathcal\{S\}\_\{p\}^\{\(t\)\}\}\\bar\{h\}\_\{j\}^\{\(t\)\}\\quad\\text\{and\}\\quad w\_\{i\}^\{\(t\)\}=\\mathrm\{clip\}\\\!\\left\(\\left\(\\frac\{\\bar\{h\}\_\{\\mathrm\{ref\}\}^\{\(t\)\}\}\{\\bar\{h\}\_\{i\}^\{\(t\)\}\+\\epsilon\}\\right\)^\{\\gamma\},\\,w\_\{\\min\},\\,w\_\{\\max\}\\right\)\.whereϵ=10−5\\epsilon=10^\{\-5\}ensures numerical stability, and the bounds\[wmin,wmax\]\[w\_\{\\min\},w\_\{\\max\}\]prevent extreme gradient scaling\. This normalizes feature importance relative to a high\-activity reference set and ensures consistent scaling across layers\.

Adaptive weighting introduces a concentration dynamic: if a feature remains inactive early in training, its EMA decays toward zero, causing its penalty weight to saturate atwmaxw\_\{\\max\}and effectively eliminating it from the model\. To prevent premature feature death during this initialization phase, we introduce a delayed linear warmup scheduleρ​\(t\)∈\[0,1\]\\rho\(t\)\\in\[0,1\]:

ρ​\(t\)=\{0,t<Twarmup,min⁡\(1,t−TwarmupTramp\),t≥Twarmup\.\\rho\(t\)=\\begin\{cases\}0,&t<T\_\{\\mathrm\{warmup\}\},\\\\\[3\.0pt\] \\min\\\!\\left\(1,\\dfrac\{t\-T\_\{\\mathrm\{warmup\}\}\}\{T\_\{\\mathrm\{ramp\}\}\}\\right\),&t\\geq T\_\{\\mathrm\{warmup\}\}\.\\end\{cases\}The effective penalty weight is interpolated aswi,eff\(t\)=1\.0\+ρ​\(t\)​\(wi\(t\)−1\.0\)w\_\{i,\\mathrm\{eff\}\}^\{\(t\)\}=1\.0\+\\rho\(t\)\(w\_\{i\}^\{\(t\)\}\-1\.0\), so that training begins with a uniformℓ1\\ell\_\{1\}penalty and transitions to the adaptive regime as features become established\.

Theℓ2\\ell\_\{2\}term controls the conditioning of the active Gram matrix, while the adaptive weighting mechanism suppresses interaction leakage between active and inactive features\. Inactive features receive large penalties \(wi→wmaxw\_\{i\}\\to w\_\{\\max\}\), driving their activations toward zero and reducing the interaction term‖D𝒜c⊤​D𝒜‖2\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\\\|\_\{2\}\. Conversely, features that are consistently active are assigned small weights \(wi→wminw\_\{i\}\\to w\_\{\\min\}\), effectively removing theℓ1\\ell\_\{1\}penalty on the active set and avoiding shrinkage bias\.

Theoretical results for the AEN\-SAE architecture are provided in Appendix[A](https://arxiv.org/html/2605.05341#A1)\. In particular, under a timescale separation assumption, we show that the proposed weighting mechanism achieves oracle\-like support recovery in the idealized setting, identifying the active set and reducing shrinkage bias\.

## 4Experiments

We evaluate AEN\-SAEs on synthetic and real\-world settings to test their robustness under dictionary coherence\. We begin with a controlled spiked\-model experiment to isolate the effect of coherence, then evaluate on Pythia 70M and Llama 3\.1 8B to assess performance on real LLM activations and hyperparameter transfer across model sizes\. Full experimental and implementation details are provided in Appendix[B](https://arxiv.org/html/2605.05341#A2)\.

For LLM\-based experiments, we stream text from a deduplicated version of the Pile\[[13](https://arxiv.org/html/2605.05341#bib.bib37)\], extract residual\-stream activations from a frozen model at a chosen layer, and form token\-level batches\. Activations areℓ2\\ell\_\{2\}\-normalized per token to ensure consistent scaling across models\.

SAEs are trained with Adam at a fixed learning rate using reconstruction loss with regularization\. Gradients are clipped for stability, and decoder columns are renormalized after each step\. Adaptive\-weight models update EMA statistics online and apply reweighting after a warmup period\. No auxiliary resampling or proxy\-gradient methods are used for any model\.

We evaluate reconstruction \(MSE, explained variance\), sparsity \(ℓ0\\ell\_\{0\}\), and feature starvation \(dead neuron rate\)\. To probe geometric structure, we also measure dictionary coherence, condition numbers, and feature\-utilization statistics \(e\.g\., entropy and Gini coefficient\)\. For LLMs, we additionally assess downstream fidelity via reconstruction patching\. Full metric definitions are provided in Appendix[C](https://arxiv.org/html/2605.05341#A3)\.

### 4\.1Spiked Model: Coherence\-Induced Instability

To isolate the effect of dictionary coherence, we consider a synthetic spiked model where each atom is constructed as a mixture of an independent direction and a shared component\[[38](https://arxiv.org/html/2605.05341#bib.bib12)\]:

dj=1−ρ​uj\+ρ​v‖1−ρ​uj\+ρ​v‖2,d\_\{j\}=\\frac\{\\sqrt\{1\-\\rho\}u\_\{j\}\+\\sqrt\{\\rho\}v\}\{\\\|\\sqrt\{1\-\\rho\}u\_\{j\}\+\\sqrt\{\\rho\}v\\\|\_\{2\}\},whereρ∈\[0,1\]\\rho\\in\[0,1\]controls the expected pairwise coherence\. This allows for the variation of coherence while keeping sparsity and noise fixed\. A detailed description of the spiked model construction and additional experimental results are provided in Appendix[D](https://arxiv.org/html/2605.05341#A4)\.

We generate data by samplingkk\-sparse codes \(k=16k=16\) and formingx∗=D∗​h∗x^\{\*\}=D^\{\*\}h^\{\*\}using a fixed teacher dictionary \(dmodel=256d\_\{\\text\{model\}\}=256,ddict=1024d\_\{\\text\{dict\}\}=1024\)\. Full experimental details are provided in Appendix[B](https://arxiv.org/html/2605.05341#A2)\.

Low coherence \(ρ=0\\rho=0\)\.When dictionary atoms are independent, all methods perform well \(Figure[2](https://arxiv.org/html/2605.05341#S4.F2)\)\. In this regime, feature selection is trivial and even hard\-masking methods exhibit low failure rates\. AEN\-SAE maintains full dictionary utilization with no dead features, at the cost of a small reduction in explained variance due to theℓ2\\ell\_\{2\}regularization\.

![Refer to caption](https://arxiv.org/html/2605.05341v1/figures/spiked_pareto.png)Figure 2:Pareto curves for spiked dictionary \(ρ=0\\rho=0\)\.When dictionary atoms are uncorrelated, all architectures work reasonably well, with similar explained variance curves \(right\)\. However, even in this simple case, dead neurons in TopK exist while AEN\-SAE solves this\. Note that adaptive LASSO concentrates around the true sparsity level and thus there is no Pareto curve; it is less flexible for mechanistic interpretability since the sparsity is an interpretability choice rather than ground truth\.High coherence \(ρ=0\.9\\rho=0\.9\)\.In contrast, high coherence induces severe instability across standard SAE architectures\. Continuous relaxations become highly sensitive to hyperparameters, while adaptive LASSO frequently collapses to degenerate solutions\. Hard\-masking methods such as TopK suffer from severe feature starvation, eliminating over 95% of the dictionary at typical sparsity levels \(Table[1](https://arxiv.org/html/2605.05341#S4.T1)\)\.

AEN\-SAE mitigates this failure mode by stabilizing both curvature and feature interactions\. While it does not completely eliminate feature starvation, it reduces dead features to≈41%\\approx 41\\%and maintains substantially higher reconstruction quality\. Moreover, AEN yields significantly improved geometric structure, reducing the active condition number \(κ≈21\\kappa\\approx 21vs\.769769for elastic net\) and lowering interaction leakage\.

These results demonstrate that coherence fundamentally limits the stability ofℓ1\\ell\_\{1\}\-based sparse coding, and that AEN provides a principled mechanism for mitigating this instability\.

Table 1:Synthetic spiked model comparison \(ρ=0\.9\\rho=0\.9\)\.Head\-to\-head performance of TopK vs\. AEN\-SAE in a highly collinear regime at matched empirical sparsity levels\.
### 4\.2Pythia 70M

We evaluate AEN\-SAEs on activations from a frozen Pythia 70M model\[[3](https://arxiv.org/html/2605.05341#bib.bib10)\], training an8×8\\timesovercomplete dictionary \(ddict=4096d\_\{\\text\{dict\}\}=4096\) on layer\-3 residuals\. Full experimental details and extended analysis for Pythia 70M are provided in Appendix[E](https://arxiv.org/html/2605.05341#A5)\.

Hyperparameters and efficiency\.Although AEN\-SAEs introduce additional hyperparameters, we find them stable across runs and consistent with classical adaptive elastic net guidance\. In practice, only the base sparsity penaltyλ1\\lambda\_\{1\}needs to be tuned to reach a desired sparsity level\. The method introduces negligible overhead \(<0\.1%<0\.1\\%FLOPs increase compared to TopK\) and requires no auxiliary resampling or proxy\-gradient mechanisms\. A detailed hyperparameter sweep and robustness analysis are provided in Appendix[E\.3](https://arxiv.org/html/2605.05341#A5.SS3)\.

Dictionary coherence as a failure modality\.Unlike synthetic settings, LLM activations exhibit strong feature correlations, inducing a highly coherent dictionary\. In this regime, standard SAE variants fail in distinct ways\. Vanillaℓ1\\ell\_\{1\}requires large penalties to suppress correlated features, leading to severe shrinkage \(up to≈25%\\approx 25\\%magnitude loss atℓ0≈22\\ell\_\{0\}\\approx 22\) and high sensitivity to hyperparameters\. Elastic net stabilizes training but produces dense, poorly conditioned representations \(medianκ\>1013\\kappa\>10^\{13\}\)\. Adaptive LASSO lacks a structuralℓ2\\ell\_\{2\}anchor and becomes numerically unstable under collinearity and frequently collapses, which is consistent with known limitations ofℓ1\\ell\_\{1\}methods in correlated settings\.

Reconstruction vs\. feature utilization\.Figure[3](https://arxiv.org/html/2605.05341#S4.F3)and Table[2](https://arxiv.org/html/2605.05341#S4.T2)illustrate a clear trade\-off between raw reconstruction and feature utilization\. Atℓ0≈32\\ell\_\{0\}\\approx 32, TopK achieves higher explained variance \(0\.760\.76\), but does so by concentrating capacity into highly redundant feature hubs \(max coherence0\.990\.99\)\. Despite this high extreme, TopK exhibits a low p90 coherence \(0\.327\), suggesting that it learns a sparse set of nearly identical high\-frequency features while starving the surrounding correlated semantic space \(nearly 14% dead features\)\.

In contrast, AEN distributes activation mass across correlated feature groups, which substantially reduces feature starvation \(down to≈1%\\approx 1\\%atℓ0≈32\\ell\_\{0\}\\approx 32\) and lowers extreme redundancy \(max coherence0\.960\.96, p90 coherence0\.5160\.516\)\. This results in a more balanced and geometrically structured dictionary, though with a modest reduction in explained variance\. AEN operates at a different point in the trade\-off space, prioritizing feature utilization and geometric stability over maximal reconstruction\.

![Refer to caption](https://arxiv.org/html/2605.05341v1/figures/pythia70m_pareto.png)Figure 3:Pareto frontiers on the Pythia 70M residual stream across three random seeds for all tested configurations\.The left panel illustrates feature starvation\. The right panel displays raw reconstruction fidelity\. Error bars denote\[min,max\]\[\\min,\\max\]bounds\.Table 2:Head\-to\-head performance on Pythia 70M\.Comparing AEN to the TopK baseline at targeted sparsity bottlenecks across three random seeds\. Results are reported as mean \[min, max\]\.Feature utilization at scale\.At higher sparsity levels \(ℓ0≈128\\ell\_\{0\}\\approx 128\), the difference becomes more pronounced \(Table[3](https://arxiv.org/html/2605.05341#S4.T3)\)\. TopK concentrates86%86\\%of activation mass into the top10%10\\%of features \(Gini0\.9030\.903\), indicating severe underutilization of capacity\. AEN mitigates this effect, reducing the Gini coefficient to0\.6700\.670and increasing entropy, while lowering dead features to≈3%\\approx 3\\%\. This behavior is driven by the adaptive reweighting mechanism: highly active features are penalized more strongly, which prevents dominance of redundant hubs and encourages recruitment of underutilized features\.

Table 3:Feature utilization and starvation at relaxed sparsities \(ℓ0≈128\\ell\_\{0\}\\approx 128\)\.Results are reported as mean \[min, max\] across random seeds\. While TopK maintains higher explained variance, it does so by permanently starving≈12%\\approx 12\\%of the dictionary and concentrating over86%86\\%of its activation mass into a narrow subset of features\. AEN resolution of this concentration is statistically robust across seeds, yielding higher entropy, a vastly lower Gini coefficient, and a fully utilized dictionary\.Overall, these results demonstrate that while hard\-masking methods maximize reconstruction, they do so by collapsing onto a highly redundant subset of features\. AEN instead provides a principled mechanism for improving feature utilization and geometric stability under coherence, without relying on auxiliary heuristics\.

### 4\.3Llama 3\.1 8B

To evaluate AEN\-SAEs at scale, we extract dense representations from the middle layer \(layer 16\) of a frozen Llama 3\.1 8B\[[16](https://arxiv.org/html/2605.05341#bib.bib9)\]\(dmodel=4096d\_\{\\text\{model\}\}=4096\) and train a32×32\\timesovercomplete dictionary \(ddict=131,072d\_\{\\text\{dict\}\}=131,072\)\. We stream the deduplicated Pile dataset with a sequence length of 128, applying per\-tokenℓ2\\ell\_\{2\}normalization\. Optimization proceeds via Adam \(batch size 4,096, learning rate5×10−45\\times 10^\{\-4\}\) for 100,000 steps \(≈400\\approx 400M tokens\), utilizing a 4,000\-step linear warmup for the AEN\-SAE adaptive weights\. The SAE is trained in full precision \(FP32\) against BF16 LLM representations, and final downstream validation is performed on a held\-out shard of 20,000 documents\.

Hyperparameter transfer and scale invariance\.Exhaustive hyperparameter tuning at this scale is computationally prohibitive\. However, AEN\-SAEs exhibit strong scale invariance; specifically, structural hyperparameters transfer directly from smaller models\. This approach is philosophically aligned with maximal update parameterization \(μ​P\\mu P\)\[[39](https://arxiv.org/html/2605.05341#bib.bib40)\], which establishes scaling laws that keep optimal hyperparameters stable as model width increases\. We thus fix the structural hyperparameters\(λ2=10−4,γ=0\.5,β=0\.9999,p=0\.05,wmin=0\.01,wmax=10\.0\)\(\\lambda\_\{2\}=10^\{\-4\},\\gamma=0\.5,\\beta=0\.9999,p=0\.05,w\_\{\\min\}=0\.01,w\_\{\\max\}=10\.0\)based on Pythia 70M and tune onlyλ1\\lambda\_\{1\}to target sparsity and hit targetℓ0\\ell\_\{0\}bottlenecks\. Empirically, this transfer is successful and enables reliable deployment without large\-scale hyperparameter sweeps\.

Failure of hard\-masking at scale\.Table[4](https://arxiv.org/html/2605.05341#S4.T4)reveals the geometric cost of hard\-masking for large\-scale models\. Across all sparsity levels, TopK exhibits extreme feature starvation, with≈80%\\approx 80\\%of the dictionary remaining inactive\. At the same time, it achieves high reconstruction by collapsing onto a highly redundant subset of features, evidenced by maximum coherence reaching1\.0001\.000immediately atℓ0≈32\\ell\_\{0\}\\approx 32and maintaining a heavily skewed p90 coherence across all capacities\. This behavior reflects a degenerate solution: rather than exploring the available 131,072\-dimensional capacity, the model concentrates activation mass into a small set of nearly identical features, leaving most of the dictionary unused\.

AEN: utilization under coherence\.In contrast, AEN\-SAEs explicitly suppress the redundancy exhibited by TopK\. By penalizing frequently co\-activated features, AEN redistributes activation mass across correlated groups, reducing both feature starvation and extreme coherence\. For example, atℓ0≈128\\ell\_\{0\}\\approx 128, the p90 coherence drops from0\.4380\.438\(TopK\) to0\.2180\.218, while dead features are reduced from≈80%\\approx 80\\%to≈39%\\approx 39\\%\. Notably, even AEN retains a nontrivial fraction of inactive features, suggesting that complete elimination of feature starvation may be fundamentally constrained and not purely an optimization issue, but also a consequence of sampling limitations at this scale\.

While AEN incurs a reduction in explained variance, this reflects a different operating point in the trade\-off space: AEN prioritizes feature utilization and geometric stability over maximal reconstruction\.

Scaling behavior\.Our results confirm that the benefits of AEN persist at large scale\. The same geometric mechanism observed in synthetic and mid\-scale experiments\-coherence\-induced collapse and its mitigation via adaptive reweighting—remains the dominant factor governing performance\. Crucially, this is achieved without auxiliary heuristics and with hyperparameters transferred directly from smaller models\. Overall, AEN\-SAEs provide a scalable approach to sparse representation learning that prioritizes feature utilization and geometric stability in regimes where standard methods degenerate\. Notably, this performance is achieved with negligible computational overhead relative to TopK and without auxiliary resampling or proxy\-gradient mechanisms\.

Table 4:Head\-to\-head performance on Llama 3\.1 8B\.Comparing AEN to TopK baseline at targeted sparsity bottlenecks\.

## 5Discussion, Limitations, and Future Research

In this work, we reframed feature starvation and shrinkage bias in sparse autoencoders as geometric and optimization pathologies arising from overcomplete, highly coherent dictionaries\. Building on classical results from high\-dimensional statistics, we introduced AEN\-SAEs: a fully differentiable architecture that combines online adaptiveℓ1\\ell\_\{1\}reweighting with anℓ2\\ell\_\{2\}structural term to stabilize the sparse coding map\.

Our analysis shows that coherence is a primary driver of failure in standard SAE formulations\. In such regimes,ℓ1\\ell\_\{1\}\-based methods become unstable, while hard\-masking approaches achieve strong reconstruction by collapsing onto a small, highly redundant subset of features\. AEN\-SAEs provide a principled alternative: by directly controlling curvature and feature interactions, they improve feature utilization and geometric stability without relying on auxiliary heuristics such as resampling or proxy gradients\.

Beyond architectural design, our results highlight the importance of evaluating sparse representations through geometric diagnostics, including dictionary coherence, interaction between active and inactive sets, and the conditioning of the active manifold\. These metrics provide a more faithful view of representation structure than reconstruction alone, and are particularly relevant for interpretability, where underutilized dictionaries may fail to capture rare but semantically important features\.

Limitations and future work\.While AEN\-SAEs improve stability and utilization, several limitations remain\. First, as with allℓ1\\ell\_\{1\}\-based methods, achieving a target sparsity level requires tuningλ1\\lambda\_\{1\}, which is less direct than specifying a fixedkkin hard\-masking approaches\. Developing adaptive or self\-tuning sparsity mechanisms remains an important direction\.

Second, although AEN significantly reduces feature starvation, a nontrivial fraction of neurons remain inactive at scale \(e\.g\.,4040–50%50\\%at the 8B model\)\. This suggests that feature inactivity is not purely an optimization failure but also reflects sampling limitations at scale: finite training horizons may be insufficient to activate highly specialized features in very large dictionaries\.

Third, our evaluation focuses on geometric and statistical properties of the learned representations\. A comprehensive assessment of human\-centered interpretability\[[25](https://arxiv.org/html/2605.05341#bib.bib39)\]remains for future work, and will likely require both larger\-scale training and dedicated evaluation protocols\.

Finally, our results suggest a promising hybrid direction: combining geometry\-aware methods such as AEN with structure\-informed resampling strategies based on coherence or feature utilization, or integrating adaptiveℓ2\\ell\_\{2\}anchors into hierarchical architectures such as Matryoshka SAEs\[[6](https://arxiv.org/html/2605.05341#bib.bib44)\]to enforce both geometric stability and multi\-level feature abstraction\.

## Acknowledgments and Disclosure of Funding

A\.M\. is supported by the EPSRC AI Hub on Mathematical Foundations of Intelligence: An “Erlangen Programme” for AI \[EP/Y028872/1\]\. K\.Y\. is supported by JSPS KAKENHI \(24K15120, 24H00247, 26K02871\)\.

## References

- \[1\]\(2024\)Adaptive sparse allocation with mutual choice & feature choice sparse autoencoders\.External Links:2411\.02124,[Link](https://doi.org/10.48550/arXiv.2411.02124)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p4.2)\.
- \[2\]P\. Bartlett, D\. J\. Foster, and M\. Telgarsky\(2017\)Spectrally\-normalized margin bounds for neural networks\.External Links:1706\.08498,[Link](https://doi.org/10.48550/arXiv.1706.08498)Cited by:[§A\.1](https://arxiv.org/html/2605.05341#A1.SS1.p2.2)\.
- \[3\]S\. Biderman, H\. Schoelkopf, Q\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. van der Wal\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.External Links:2304\.01373,[Link](https://doi.org/10.48550/arXiv.2304.01373)Cited by:[§E\.1](https://arxiv.org/html/2605.05341#A5.SS1.p1.3),[3rd item](https://arxiv.org/html/2605.05341#S1.I1.i3.p1.1),[§4\.2](https://arxiv.org/html/2605.05341#S4.SS2.p1.2)\.
- \[4\]T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. L\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah\(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.External Links:[Link](https://transformer-circuits.pub/2023/monosemantic-features)Cited by:[§1](https://arxiv.org/html/2605.05341#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p3.1)\.
- \[5\]B\. Bussmann, P\. Leask, and N\. Nanda\(2024\)BatchTopK sparse autoencoders\.External Links:2412\.06410,[Link](https://doi.org/10.48550/arXiv.2412.06410)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p4.2)\.
- \[6\]B\. Bussmann, N\. Nabeshima, A\. Karvonen, and N\. Nanda\(2025\)Learning multi\-level features with matryoshka sparse autoencoders\.External Links:2503\.17547,[Link](https://doi.org/10.48550/arXiv.2503.17547)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p5.1),[§5](https://arxiv.org/html/2605.05341#S5.p7.1)\.
- \[7\]D\. Chanin, J\. Wilken\-Smith, T\. Dulka, H\. Bhatnagar, S\. Golechha, and J\. Bloom\(2025\)A is for absorption: studying feature splitting and absorption in sparse autoencoders\.External Links:2409\.14507,[Link](https://doi.org/10.48550/arXiv.2409.14507)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p5.1)\.
- \[8\]X\. Chen, J\. Liu, Z\. Wang, and W\. Yin\(2021\)Hyperparameter tuning is all you need for LISTA\.External Links:2110\.15900,[Link](https://doi.org/10.48550/arXiv.2110.15900)Cited by:[§2\.3](https://arxiv.org/html/2605.05341#S2.SS3.p2.1)\.
- \[9\]H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey\(2023\)Sparse autoencoders find highly interpretable features in language models\.External Links:2309\.08600,[Link](https://doi.org/10.48550/arXiv.2309.08600)Cited by:[§1](https://arxiv.org/html/2605.05341#S1.p1.1)\.
- \[10\]N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen, R\. Grosse, S\. McCandlish, J\. Kaplan, D\. Amodei, M\. Wattenberg, and C\. Olah\(2022\)Toy models of superposition\.External Links:2209\.10652,[Link](https://doi.org/10.48550/arXiv.2209.10652)Cited by:[§1](https://arxiv.org/html/2605.05341#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p1.3)\.
- \[11\]J\. Fan and J\. Lv\(2008\)Sure independence screening for ultra\-high dimensional feature space\.External Links:math/0612857,[Link](https://doi.org/10.48550/arXiv.math/0612857)Cited by:[§E\.4](https://arxiv.org/html/2605.05341#A5.SS4.p1.1)\.
- \[12\]S\. Foucart and H\. Rauhut\(2013\)A mathematical introduction to compressive sensing\.Applied and Numerical Harmonic Analysis,Birkhäuser,New York\.External Links:[Link](https://doi.org/10.1007/978-0-8176-4948-7)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p5.1),[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p1.3)\.
- \[13\]L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. Leahy\(2020\)The Pile: an 800gb dataset of diverse text for language modeling\.External Links:2101\.00027,[Link](https://doi.org/10.48550/arXiv.2101.00027)Cited by:[Appendix B](https://arxiv.org/html/2605.05341#A2.SS0.SSS0.Px1.p1.1),[§E\.1](https://arxiv.org/html/2605.05341#A5.SS1.p2.3),[§4](https://arxiv.org/html/2605.05341#S4.p2.1)\.
- \[14\]L\. Gao, T\. D\. la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. Wu\(2024\)Scaling and evaluating sparse autoencoders\.External Links:2406\.04093,[Link](https://doi.org/10.48550/arXiv.2406.04093)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p3.1),[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p4.2)\.
- \[15\]M\. Geva, A\. Caciularu, K\. R\. Wang, and Y\. Goldberg\(2022\)Transformer feed\-forward layers build predictions by promoting concepts in the vocabulary space\.External Links:2203\.14680,[Link](https://doi.org/10.48550/arXiv.2203.14680)Cited by:[§1](https://arxiv.org/html/2605.05341#S1.p2.8)\.
- \[16\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma\(2024\)The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://doi.org/10.48550/arXiv.2407.21783)Cited by:[3rd item](https://arxiv.org/html/2605.05341#S1.I1.i3.p1.1),[§4\.3](https://arxiv.org/html/2605.05341#S4.SS3.p1.6)\.
- \[17\]K\. Gregor and Y\. LeCun\(2010\)Learning fast approximations of sparse coding\.InProceedings of the 27th International Conference on Machine Learning,ICML\.External Links:[Link](https://icml.cc/Conferences/2010/papers/449.pdf)Cited by:[§2\.3](https://arxiv.org/html/2605.05341#S2.SS3.p2.1)\.
- \[18\]G\. E\. Hinton and R\. R\. Salakhutdinov\(2006\)Reducing the dimensionality of data with neural networks\.Science313\(5786\),pp\. 504–507\.External Links:[Link](https://doi.org/10.1126/science.1127647)Cited by:[§1](https://arxiv.org/html/2605.05341#S1.p2.8)\.
- \[19\]S\. Hochreiter and J\. Schmidhuber\(1997\)Flat minima\.Neural Computation9\(1\),pp\. 1–42\.External Links:[Link](https://doi.org/10.1162/neco.1997.9.1.1)Cited by:[§A\.1](https://arxiv.org/html/2605.05341#A1.SS1.p2.2)\.
- \[20\]A\. J\. Hoffman\(1952\)On approximate solutions of systems of linear inequalities\.Journal of Research of the National Bureau of Standards49\(4\),pp\. 263–265\.External Links:[Link](https://nvlpubs.nist.gov/nistpubs/jres/049/4/v49.n04.a05.pdf)Cited by:[§2\.3](https://arxiv.org/html/2605.05341#S2.SS3.p4.6)\.
- \[21\]N\. S\. Keskar, D\. Mudigere, J\. Nocedal, M\. Smelyanskiy, and P\. T\. P\. Tang\(2017\)On large\-batch training for deep learning: generalization gap and sharp minima\.External Links:1609\.04836,[Link](https://doi.org/10.48550/arXiv.1609.04836)Cited by:[§A\.1](https://arxiv.org/html/2605.05341#A1.SS1.p2.2)\.
- \[22\]S\. Lee, A\. Davies, M\. E\. Canby, and J\. Hockenmaier\(2025\)Evaluating and designing sparse autoencoders by approximating quasi\-orthogonality\.External Links:2503\.24277,[Link](https://doi.org/10.48550/arXiv.2503.24277)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p5.1)\.
- \[23\]M\. Li and L\. Janson\(2024\)Optimal ablation for interpretability\.External Links:2409\.09951,[Link](https://doi.org/10.48550/arXiv.2409.09951)Cited by:[§C\.6](https://arxiv.org/html/2605.05341#A3.SS6.p3.1)\.
- \[24\]T\. Lieberum, S\. Rajamanoharan, A\. Conmy, L\. Smith, N\. Sonnerat, V\. Varma, J\. Kramár, A\. Dragan, R\. Shah, and N\. Nanda\(2024\)Gemma scope: open sparse autoencoders everywhere all at once on gemma 2\.External Links:2408\.05147,[Link](https://doi.org/10.48550/arXiv.2408.05147)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p4.2)\.
- \[25\]J\. Liu, T\. Blanton, Y\. Elazar, S\. Min, Y\. Chen, A\. Chheda\-Kothary, H\. Tran, B\. Bischoff, E\. Marsh, M\. Schmitz, C\. Trier, A\. Sarnat, J\. James, J\. Borchardt, B\. Kuehl, E\. Cheng, K\. Farley, S\. Sreeram, T\. Anderson, D\. Albright, C\. Schoenick, L\. Soldaini, D\. Groeneveld, R\. Y\. Pang, P\. W\. Koh, N\. A\. Smith, S\. Lebrecht, Y\. Choi, H\. Hajishirzi, A\. Farhadi, and J\. Dodge\(2025\)OLMoTrace: tracing language model outputs back to trillions of training tokens\.External Links:2504\.07096,[Link](https://doi.org/10.48550/arXiv.2504.07096)Cited by:[§5](https://arxiv.org/html/2605.05341#S5.p6.1)\.
- \[26\]J\. Mairal, F\. Bach, J\. Ponce, and G\. Sapiro\(2010\)Online learning for matrix factorization and sparse coding\.Journal of Machine Learning Research11\(1\)\.External Links:[Link](http://jmlr.org/papers/v11/mairal10a.html)Cited by:[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p2.3)\.
- \[27\]C\. P\. Martin\-Linares and J\. P\. Ling\(2025\)Attribution\-guided distillation of matryoshka sparse autoencoders\.External Links:2512\.24975,[Link](https://doi.org/10.48550/arXiv.2512.24975)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p5.1)\.
- \[28\]N\. Nanda\(2022\)A comprehensive mechanistic interpretability explainer & glossary\.External Links:[Link](https://www.neelnanda.io/mechanistic-interpretability/glossary)Cited by:[§C\.6](https://arxiv.org/html/2605.05341#A3.SS6.p3.1)\.
- \[29\]B\. Neyshabur, S\. Bhojanapalli, D\. McAllester, and N\. Srebro\(2017\)Exploring generalization in deep learning\.External Links:1706\.08947,[Link](https://doi.org/10.48550/arXiv.1706.08947)Cited by:[§A\.1](https://arxiv.org/html/2605.05341#A1.SS1.p2.2)\.
- \[30\]S\. T\. Piantadosi\(2014\)Zipf’s word frequency law in natural language: a critical review and future directions\.Psychonomic Bulletin & Review21\(5\),pp\. 1112–1130\.External Links:[Link](https://doi.org/10.3758/s13423-014-0585-6)Cited by:[§A\.2](https://arxiv.org/html/2605.05341#A1.SS2.p5.10)\.
- \[31\]N\. Pochinkov, B\. Pasero, and S\. Shibayama\(2024\)Investigating neuron ablation in attention heads: the case for peak activation centering\.External Links:2408\.17322,[Link](https://doi.org/10.48550/arXiv.2408.17322)Cited by:[§C\.6](https://arxiv.org/html/2605.05341#A3.SS6.p3.1)\.
- \[32\]S\. Rajamanoharan, A\. Conmy, L\. Smith, T\. Lieberum, V\. Varma, J\. Kramár, R\. Shah, and N\. Nanda\(2024\)Improving dictionary learning with gated sparse autoencoders\.External Links:2404\.16014,[Link](https://doi.org/10.48550/arXiv.2404.16014)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p4.2)\.
- \[33\]S\. Rajamanoharan, T\. Lieberum, N\. Sonnerat, A\. Conmy, V\. Varma, J\. Kramár, and N\. Nanda\(2024\)Jumping ahead: improving reconstruction fidelity with jumprelu sparse autoencoders\.External Links:2407\.14435,[Link](https://doi.org/10.48550/arXiv.2407.14435)Cited by:[§2\.1](https://arxiv.org/html/2605.05341#S2.SS1.p4.2)\.
- \[34\]S\. M\. Robinson\(1981\)Some continuity properties of polyhedral multifunctions\.Mathematical Programming at Oberwolfach14,pp\. 206–214\.External Links:[Link](https://doi.org/10.1007/BFb0120929)Cited by:[§2\.3](https://arxiv.org/html/2605.05341#S2.SS3.p4.6)\.
- \[35\]T\. Rockafellar and R\. Wets\(1998\)Variational analysis\.Springer\.External Links:[Link](https://doi.org/10.1007/978-3-642-02431-3)Cited by:[§A\.1](https://arxiv.org/html/2605.05341#A1.SS1.6.p1.11),[§A\.1](https://arxiv.org/html/2605.05341#A1.SS1.6.p1.15)\.
- \[36\]J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu\(2023\)RoFormer: enhanced transformer with rotary position embedding\.External Links:2104\.09864,[Link](https://doi.org/10.48550/arXiv.2104.09864)Cited by:[§C\.6](https://arxiv.org/html/2605.05341#A3.SS6.p3.1)\.
- \[37\]R\. Tibshirani\(1996\)Regression shrinkage and selection via the Lasso\.Journal of the Royal Statistical Society: Series B \(Methodological\)58\(1\),pp\. 267–288\.External Links:[Link](https://doi.org/10.1111/j.2517-6161.1996.tb02080.x)Cited by:[§1](https://arxiv.org/html/2605.05341#S1.p4.4),[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p1.3)\.
- \[38\]M\. J\. Wainwright\(2019\)High\-dimensional statistics: a non\-asymptotic viewpoint\.Cambridge Series in Statistical and Probabilistic Mathematics,Cambridge University Press\.External Links:[Link](https://doi.org/10.1017/9781108627771)Cited by:[§A\.1](https://arxiv.org/html/2605.05341#A1.SS1.p3.4),[§A\.2](https://arxiv.org/html/2605.05341#A1.SS2.p1.3),[§C\.5](https://arxiv.org/html/2605.05341#A3.SS5.p2.3),[§D\.1](https://arxiv.org/html/2605.05341#A4.SS1.p1.3),[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p1.3),[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p2.3),[§4\.1](https://arxiv.org/html/2605.05341#S4.SS1.p1.2)\.
- \[39\]G\. Yang, E\. J\. Hu, I\. Babuschkin, S\. Sidor, X\. Liu, D\. Farhi, N\. Ryder, J\. Pachocki, W\. Chen, and J\. Gao\(2022\)Tensor programs v: tuning large neural networks via zero\-shot hyperparameter transfer\.External Links:2203\.03466,[Link](https://doi.org/10.48550/arXiv.2203.03466)Cited by:[§4\.3](https://arxiv.org/html/2605.05341#S4.SS3.p2.4)\.
- \[40\]P\. Zhao and B\. Yu\(2006\)On model selection consistency of Lasso\.Journal of Machine Learning Research7\(90\),pp\. 2541–2563\.External Links:[Link](http://jmlr.org/papers/v7/zhao06a.html)Cited by:[§A\.2](https://arxiv.org/html/2605.05341#A1.SS2.p1.3),[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p2.3)\.
- \[41\]H\. Zou and T\. Hastie\(2005\)Regularization and variable selection via the elastic net\.Journal of the Royal Statistical Society: Series B \(Statistical Methodology\)67\(2\),pp\. 301–320\.External Links:[Link](https://doi.org/10.1111/j.1467-9868.2005.00503.x)Cited by:[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p3.3)\.
- \[42\]H\. Zou and H\. H\. Zhang\(2009\)On the adaptive elastic\-net with a diverging number of parameters\.The Annals of Statistics37\(4\),pp\. 1733–1751\.External Links:[Link](https://doi.org/10.1214/08-AOS625)Cited by:[§A\.2](https://arxiv.org/html/2605.05341#A1.SS2.p3.3),[§E\.4](https://arxiv.org/html/2605.05341#A5.SS4.SSS0.Px3.p1.1)\.
- \[43\]H\. Zou\(2006\)The adaptive Lasso and its oracle properties\.Journal of the American Statistical Association101\(476\),pp\. 1418–1429\.External Links:[Link](https://doi.org/10.1198/016214506000000735)Cited by:[§A\.2](https://arxiv.org/html/2605.05341#A1.SS2.p3.3),[§A\.2](https://arxiv.org/html/2605.05341#A1.SS2.p6.10),[§E\.4](https://arxiv.org/html/2605.05341#A5.SS4.p1.1),[§2\.2](https://arxiv.org/html/2605.05341#S2.SS2.p2.3)\.

## Appendix ATheoretical Guarantees: Polyhedral Stability and Oracle Consistency

In this section, we formalize the statistical and stability guarantees of the AEN\-SAE\. For notational simplicity during the subsequent matrix calculus derivations, we assume the input residual stream vectorxxhas been precentered by the explicit decoder bias \(x←x−bdecx\\leftarrow x\-b\_\{\\text\{dec\}\}\)\. This allows the reconstruction error to be written more cleanly as‖x−D​h‖22/2\\\|x\-Dh\\\|\_\{2\}^\{2\}/2without loss of generality\.

### A\.1Polyhedral and Lipschitz Stability

###### Theorem 1\(Active Curvature Lower Bound\)\.

Fix a latent representationh∈ℝ\+ddicth\\in\\mathbb\{R\}\_\{\+\}^\{d\_\{\\text\{dict\}\}\}and define its active features by𝒜:=\{i:hi≠0\}\\mathcal\{A\}:=\\\{i:h\_\{i\}\\neq 0\\\}\. Treating the adaptive weights as fixed, the Hessian with respect to the active latent representations∇h𝒜2ℒA​E​N\\nabla^\{2\}\_\{h\_\{\\mathcal\{A\}\}\}\\mathcal\{L\}\_\{AEN\}satisfies:

λmin​\(∇h𝒜2ℒA​E​N\)=λmin​\(D𝒜⊤​D𝒜\)\+2​λ2≥2​λ2\>0\.\\lambda\_\{\\min\}\(\\nabla^\{2\}\_\{h\_\{\\mathcal\{A\}\}\}\\mathcal\{L\}\_\{AEN\}\)=\\lambda\_\{\\min\}\(D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\)\+2\\lambda\_\{2\}\\geq 2\\lambda\_\{2\}\>0\.

###### Proof\.

For a fixed active set𝒜\\mathcal\{A\}, the AEN\-SAE objective evaluates to:

ℒ𝒜​\(h𝒜\)=12​‖x−D𝒜​h𝒜‖22\+λ1​∑i∈𝒜wi​\|hi\|\+λ2​‖h𝒜‖22\.\\mathcal\{L\}\_\{\\mathcal\{A\}\}\(h\_\{\\mathcal\{A\}\}\)=\\frac\{1\}\{2\}\\\|x\-D\_\{\\mathcal\{A\}\}h\_\{\\mathcal\{A\}\}\\\|\_\{2\}^\{2\}\+\\lambda\_\{1\}\\sum\_\{i\\in\\mathcal\{A\}\}w\_\{i\}\|h\_\{i\}\|\+\\lambda\_\{2\}\\\|h\_\{\\mathcal\{A\}\}\\\|\_\{2\}^\{2\}\.
To compute the Hessian, we first derive the gradient with respect to the active latent vectorh𝒜h\_\{\\mathcal\{A\}\}\. Since the absolute value function is locally linear, and its derivative is strictly defined by the sign function, the standard matrix calculus gives the gradient given by:

∇h𝒜ℒ𝒜=−D𝒜⊤​\(x−D𝒜​h𝒜\)⏟Reconstruction gradient\+λ1​W𝒜​sign​\(h𝒜\)⏟ℓ1​gradient\+2​λ2​h𝒜⏟ℓ2​gradient,\\nabla\_\{h\_\{\\mathcal\{A\}\}\}\\mathcal\{L\}\_\{\\mathcal\{A\}\}=\\underbrace\{\-D\_\{\\mathcal\{A\}\}^\{\\top\}\(x\-D\_\{\\mathcal\{A\}\}h\_\{\\mathcal\{A\}\}\)\}\_\{\\text\{Reconstruction gradient\}\}\+\\underbrace\{\\lambda\_\{1\}W\_\{\\mathcal\{A\}\}\\text\{sign\}\(h\_\{\\mathcal\{A\}\}\)\}\_\{\\ell\_\{1\}\\text\{ gradient\}\}\+\\underbrace\{2\\lambda\_\{2\}h\_\{\\mathcal\{A\}\}\}\_\{\\ell\_\{2\}\\text\{ gradient\}\},whereW𝒜W\_\{\\mathcal\{A\}\}is a diagonal matrix of the active adaptive weights\.

Taking the derivative of this gradient with respect toh𝒜h\_\{\\mathcal\{A\}\}yields the Hessian\. Distributing the transpose in the reconstruction term yields a constantD𝒜⊤​D𝒜D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}component\. Because theℓ1\\ell\_\{1\}gradient term \(λ1​W𝒜⋅sign​\(h𝒜\)\\lambda\_\{1\}W\_\{\\mathcal\{A\}\}\\cdot\\text\{sign\}\(h\_\{\\mathcal\{A\}\}\)\) is constant within this active orthant, its second derivative vanishes\. The Hessian is therefore derived from the remaining quadratic andℓ2\\ell\_\{2\}terms:

∇h𝒜2ℒ𝒜=D𝒜⊤​D𝒜\+2​λ2​I\|𝒜\|\.\\nabla^\{2\}\_\{h\_\{\\mathcal\{A\}\}\}\\mathcal\{L\}\_\{\\mathcal\{A\}\}=D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\+2\\lambda\_\{2\}I\_\{\|\\mathcal\{A\}\|\}\.
Applying the Rayleigh quotient formula, the minimum eigenvalue is:

λmin​\(D𝒜⊤​D𝒜\+2​λ2​I\|𝒜\|\)=λmin​\(D𝒜⊤​D𝒜\)\+2​λ2\.\\lambda\_\{\\min\}\(D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\+2\\lambda\_\{2\}I\_\{\|\\mathcal\{A\}\|\}\)=\\lambda\_\{\\min\}\(D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\)\+2\\lambda\_\{2\}\.
Since the Gram matrixD𝒜⊤​D𝒜D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}is positive semi\-definite by definition \(λmin​\(D𝒜⊤​D𝒜\)≥0\\lambda\_\{\\min\}\(D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\)\\geq 0\), the Hessian’s eigenvalues are strictly bounded below by2​λ22\\lambda\_\{2\}, which completes the proof\. ∎

As established in the main text, the Hoffman constant for the active polyhedra scales inversely with the minimum eigenvalue of the active Gram matrix\. Theorem[1](https://arxiv.org/html/2605.05341#Thmtheorem1)explicitly guarantees that the Hoffman bound is capped by𝒪​\(1/2​λ2\)\\mathcal\{O\}\(1/2\\lambda\_\{2\}\)\. This prevents the activation polyhedra from collapsing into degenerate valleys, avoiding a key geometric source of instability associated with feature starvation\.

Furthermore, thisℓ2\\ell\_\{2\}curvature has a secondary desirable effect in deep learning generalization\. By avoiding discontinuous ravines, theℓ2\\ell\_\{2\}penalty forces the optimization to converge toward broader, flatter local minima in the representation space\[[19](https://arxiv.org/html/2605.05341#bib.bib31),[21](https://arxiv.org/html/2605.05341#bib.bib30)\]\. As we will formalize in Theorem[2](https://arxiv.org/html/2605.05341#Thmtheorem2), this flat geometry explicitly bounds the Lipschitz constant of the target mapping\. Classical statistical learning theory dictates that such Lipschitz regularization is strictly necessary to close the generalization gap, ensuring the learned dictionary maintains high representational fidelity on unseen text\[[2](https://arxiv.org/html/2605.05341#bib.bib28),[29](https://arxiv.org/html/2605.05341#bib.bib29)\]\.

###### Theorem 2\(Lipschitz Stability of the Latent Encoder Mapping\)\.

Leth†​\(x\)h^\{\\dagger\}\(x\)be the optimal sparse code for an inputxxunder the AEN\-SAE objective:

h†​\(x\)=argminh​\(x\)∈ℝ\+ddict⁡\[12​‖x−D​h​\(x\)‖22\+λ1​∑i=1ddictwi​\|hi​\(x\)\|\+λ2​‖h​\(x\)‖22\],h^\{\\dagger\}\(x\)=\\operatorname\{argmin\}\_\{h\(x\)\\in\\mathbb\{R\}^\{d\_\{\\text\{dict\}\}\}\_\{\+\}\}\\left\[\\frac\{1\}\{2\}\\\|x\-Dh\(x\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{1\}\\sum\_\{i=1\}^\{d\_\{\\text\{dict\}\}\}w\_\{i\}\|h\_\{i\}\(x\)\|\+\\lambda\_\{2\}\\\|h\(x\)\\\|\_\{2\}^\{2\}\\right\],where the adaptive weights are treated as fixed\. Then the target mapping from the residual streamxxto the latent spacehhis Lipschitz continuous with constantL≤1/\(2​2​λ2\)L\\leq 1/\(2\\sqrt\{2\\lambda\_\{2\}\}\):

‖h†​\(x1\)−h†​\(x2\)‖2≤12​2​λ2​‖x1−x2‖2\.\\\|h^\{\\dagger\}\(x\_\{1\}\)\-h^\{\\dagger\}\(x\_\{2\}\)\\\|\_\{2\}\\leq\\frac\{1\}\{2\\sqrt\{2\\lambda\_\{2\}\}\}\\\|x\_\{1\}\-x\_\{2\}\\\|\_\{2\}\.

###### Proof\.

Fix anyx1≠x2x\_\{1\}\\neq x\_\{2\}\. Letδh≥0\\delta\_\{h\\geq 0\}denote the indicator function such thatδh≥0=0\\delta\_\{h\\geq 0\}=0forh≥0h\\geq 0andδh≥0=∞\\delta\_\{h\\geq 0\}=\\inftyotherwise\. The first order optimality condition \(e\.g\.,\[[35](https://arxiv.org/html/2605.05341#bib.bib38), Theorem 8\.15\]\) implies that there exist subgradientss1s\_\{1\}ands2s\_\{2\}of∑i=1dd​i​c​twi​\|hi\|\+δh≥0\\sum\_\{i=1\}^\{d\_\{dict\}\}w\_\{i\}\|h\_\{i\}\|\+\\delta\_\{h\\geq 0\}\(with respect toh=\(h1,…,hdd​i​c​t\)⊤h=\(h\_\{1\},\\ldots,h\_\{d\_\{dict\}\}\)^\{\\top\}\) ath†​\(x1\)h^\{\\dagger\}\(x\_\{1\}\)andh†​\(x2\)h^\{\\dagger\}\(x\_\{2\}\)for which

D⊤​\{D​h†​\(x1\)−x1\}\+2​λ2​h†​\(x1\)\+λ1​s1\\displaystyle D^\{\\top\}\\\{Dh^\{\\dagger\}\(x\_\{1\}\)\-x\_\{1\}\\\}\+2\\lambda\_\{2\}h^\{\\dagger\}\(x\_\{1\}\)\+\\lambda\_\{1\}s\_\{1\}=0,\\displaystyle=0,D⊤​\{D​h†​\(x2\)−x2\}\+2​λ2​h†​\(x2\)\+λ1​s2\\displaystyle D^\{\\top\}\\\{Dh^\{\\dagger\}\(x\_\{2\}\)\-x\_\{2\}\\\}\+2\\lambda\_\{2\}h^\{\\dagger\}\(x\_\{2\}\)\+\\lambda\_\{1\}s\_\{2\}=0\.\\displaystyle=0\.These give

\(D⊤​D\+2​λ2​I\)​\{h†​\(x1\)−h†​\(x2\)\}−D⊤​\(x1−x2\)\+λ1​\(s1−s2\)=0\.\(D^\{\\top\}D\+2\\lambda\_\{2\}I\)\\\{h^\{\\dagger\}\(x\_\{1\}\)\-h^\{\\dagger\}\(x\_\{2\}\)\\\}\-D^\{\\top\}\(x\_\{1\}\-x\_\{2\}\)\+\\lambda\_\{1\}\(s\_\{1\}\-s\_\{2\}\)=0\.Taking the inner product with respect toΔ​h:=h†​\(x1\)−h†​\(x2\)\\Delta h:=h^\{\\dagger\}\(x\_\{1\}\)\-h^\{\\dagger\}\(x\_\{2\}\)gives

\(Δh\)⊤\(D⊤D\+2λ2I\)Δh−⟨D⊤\(x1−x2\),Δh\)⟩\+λ1⟨s1−s2,Δh\)⟩=0\.\(\\Delta h\)^\{\\top\}\(D^\{\\top\}D\+2\\lambda\_\{2\}I\)\\Delta h\-\\langle D^\{\\top\}\(x\_\{1\}\-x\_\{2\}\),\\Delta h\)\\rangle\+\\lambda\_\{1\}\\langle s\_\{1\}\-s\_\{2\},\\Delta h\)\\rangle=0\.Since the convexity of∑i=1dd​i​c​twi​\|hi\|\+δh≥0\\sum\_\{i=1\}^\{d\_\{dict\}\}w\_\{i\}\|h\_\{i\}\|\+\\delta\_\{h\\geq 0\}with respect tohhimplies the monotonicity of the subdifferential⟨s1−s2,Δh\)⟩=⟨s1−s2,h†\(x1\)−h†\(x2\)⟩≥0\\langle s\_\{1\}\-s\_\{2\},\\Delta h\)\\rangle=\\langle s\_\{1\}\-s\_\{2\},h^\{\\dagger\}\(x\_\{1\}\)\-h^\{\\dagger\}\(x\_\{2\}\)\\rangle\\geq 0\(e\.g\.,\[[35](https://arxiv.org/html/2605.05341#bib.bib38), Theorem 12\.17\]\), this is further reduced to

\(Δh\)⊤\(D⊤D\+2λ2I\)Δh−⟨D⊤\(x1−x2\),Δh\)⟩≤0,\(\\Delta h\)^\{\\top\}\(D^\{\\top\}D\+2\\lambda\_\{2\}I\)\\Delta h\-\\langle D^\{\\top\}\(x\_\{1\}\-x\_\{2\}\),\\Delta h\)\\rangle\\leq 0,and, together with the Cauchy–Schwarz inequality, we have

‖D​Δ​h‖22\+2​λ2​‖Δ​h‖22−‖x1−x2‖2​‖D​Δ​h‖2≤0\.\\\|D\\Delta h\\\|^\{2\}\_\{2\}\+2\\lambda\_\{2\}\\\|\\Delta h\\\|\_\{2\}^\{2\}\-\\\|x\_\{1\}\-x\_\{2\}\\\|\_\{2\}\\\|D\\Delta h\\\|\_\{2\}\\leq 0\.Completing the square gives

\{‖D​Δ​h‖2−‖x1−x2‖2/2\}2−‖x1−x2‖22/4\+2​λ2​‖Δ​h‖22≤0,\\\{\\\|D\\Delta h\\\|\_\{2\}\-\\\|x\_\{1\}\-x\_\{2\}\\\|\_\{2\}/2\\\}^\{2\}\-\\\|x\_\{1\}\-x\_\{2\}\\\|\_\{2\}^\{2\}/4\+2\\lambda\_\{2\}\\\|\\Delta h\\\|\_\{2\}^\{2\}\\leq 0,implying that‖h†​\(x1\)−h†​\(x2\)‖22≤‖x1−x2‖22/\(8​λ2\)\\\|h^\{\\dagger\}\(x\_\{1\}\)\-h^\{\\dagger\}\(x\_\{2\}\)\\\|\_\{2\}^\{2\}\\leq\\\|x\_\{1\}\-x\_\{2\}\\\|\_\{2\}^\{2\}/\(8\\lambda\_\{2\}\)and thus completing the proof\. ∎

The bound established in Theorem[2](https://arxiv.org/html/2605.05341#Thmtheorem2)exposes the mechanical failure of standard SAEs\. In a standardℓ1\\ell\_\{1\}framework \(λ2=0\\lambda\_\{2\}=0\), the Lipshitz stability is governed by the inverse of the minimum singular value of the active dictionaryσmin​\(D𝒜\)\\sigma\_\{\\min\}\(D\_\{\\mathcal\{A\}\}\)\. As mutual coherence increases \(as in the spiked model\[[38](https://arxiv.org/html/2605.05341#bib.bib12)\]\),σmin→0\\sigma\_\{\\min\}\\to 0, causing the Hoffman constant of the solution map to diverge to infinity\.

Simultaneously, the expressive capacity of a shallow linear encoderh​\(x\)=ReLU​\(Wenc​x\+benc\)h\(x\)=\\text\{ReLU\}\(W\_\{\\text\{enc\}\}x\+b\_\{\\text\{enc\}\}\)is strictly limited by the spectral norm of its weight matrix,‖Wenc‖2\\\|W\_\{\\text\{enc\}\}\\\|\_\{2\}\. To successfully approximate the discontinuousℓ1\\ell\_\{1\}target manifold, the optimization would require‖Wenc‖2→∞\\\|W\_\{\\text\{enc\}\}\\\|\_\{2\}\\to\\infty\. Because gradient descent naturally constrains weight norms, the encoder structurally lacks the spectral capacity to track these discontinuities\. Unable to map the target, the network effectively abandons the unstable features by pushing their pre\-activations below the ReLU threshold, causing irreversible feature starvation\. By bounding the target Lipschitz constant to1/\(2​2​λ2\)1/\(2\\sqrt\{2\\lambda\_\{2\}\}\), the AEN\-SAE ensures the target manifold remains comfortably within the spectral capacity ofWencW\_\{\\text\{enc\}\}\.

Finally, the result of Theorem[2](https://arxiv.org/html/2605.05341#Thmtheorem2)would suggest that the map becomes more stable and easier to learn asλ2\\lambda\_\{2\}is increased\. While this is true, it should be noted that, whenλ2\\lambda\_\{2\}is too high relative toλ1\\lambda\_\{1\}, the loss will function closer to Ridge regression with a grouping effect rather than LASSO with feature selection\. That is, representations will remain dense and not allow for any monosemantic interpretation\.

### A\.2Assumptions for Oracle Selection Consistency

In classical high\-dimensional statistics, standardℓ1\\ell\_\{1\}recovery relies on the strict IRC\[[38](https://arxiv.org/html/2605.05341#bib.bib12),[40](https://arxiv.org/html/2605.05341#bib.bib18)\], which assumes that noise features are largely orthogonal to true signal features\. In overcomplete LLM dictionaries \(ddict≫dmodeld\_\{\\text\{dict\}\}\\gg d\_\{\\text\{model\}\}\), features are naturally highly collinear to capture polysemantic nuance\. This inevitably violates the IRC, causing standardℓ1\\ell\_\{1\}SAEs to collapse and misidentify features\.

Further, the true active support is assumed to be static across the dataset in classical sparse regression\. In contrast, LLM residual streams exhibit a dynamiclocalsparsity, where for an encoderhh, the active features𝒜​\(x\):=\{i:\|hi​\(x\)\|\>0\}\\mathcal\{A\}\(x\):=\\\{i:\|h\_\{i\}\(x\)\|\>0\\\}change for every input tokenxx\. To evaluate the structural health of the dictionary, we define theglobal active support𝒜global=\{i:𝔼x​\[\|hi​\(x\)\|\]\>0\}\\mathcal\{A\}\_\{\\text\{global\}\}=\\\{i:\\mathbb\{E\}\_\{x\}\[\|h\_\{i\}\(x\)\|\]\>0\\\}as the set of all statistically valid semantic concepts across the data distribution\. Noise artifacts and dead features inherently belong to the inactive set \(where𝔼x​\[\|hi​\(x\)\|\]=0\\mathbb\{E\}\_\{x\}\[\|h\_\{i\}\(x\)\|\]=0\)\.

A primary theoretical advantage of the AEN\-SAE is that it achieves oracle selection consistency for the global active support without requiring the IRC\[[43](https://arxiv.org/html/2605.05341#bib.bib20),[42](https://arxiv.org/html/2605.05341#bib.bib22)\]\. By conditioning the problem via theℓ2\\ell\_\{2\}anchor and dynamically penalizing noise via adaptive weights, we reduce the theoretical requirements to two assumptions\. Let𝒜global∗\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}be the true global active support\. Leth∗:ℝdmodel→ℝddicth^\{\*\}:\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}\\to\\mathbb\{R\}^\{d\_\{\\text\{dict\}\}\}be the oracle sparse encoder defined by

h∗​\(x\)=argminh​\(x\)∈ℝ\+ddict:hi​\(x\)=0,i∉𝒜global∗⁡\[12​‖x−D​h​\(x\)‖22\+λ2​‖h​\(x\)‖22\]h^\{\*\}\(x\)=\\operatorname\{argmin\}\_\{h\(x\)\\in\\mathbb\{R\}\_\{\+\}^\{d\_\{\\text\{dict\}\}\}:h\_\{i\}\(x\)=0,i\\notin\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\}\\left\[\\frac\{1\}\{2\}\\\|x\-Dh\(x\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{2\}\\\|h\(x\)\\\|\_\{2\}^\{2\}\\right\]
###### Assumption A1\(Signal Lower Bound\)\.

The true active features have expected magnitudes strictly bounded away from zero:mini∈𝒜global∗⁡𝔼x​\[\|hi∗​\(x\)\|\]≥c\>0\\min\_\{i\\in\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\}\\mathbb\{E\}\_\{x\}\[\|h^\{\*\}\_\{i\}\(x\)\|\]\\geq c\>0\.

Assumption[A1](https://arxiv.org/html/2605.05341#Thmassumption1)is quite mild, and in fact required for model identifiability to separate the active and inactive features in expectation\. In mechanistic interpretability, a ‘true’ feature represents a semantic concept that exerts a causal impact on downstream logit generation\. Because transformer residual streams are high\-dimensional and subject to LayerNorm, any feature with an infinitesimally small activation \(c≈0c\\approx 0\) would be washed out by the baseline variance of the stream\. Therefore, impactful concepts possess a macroscopic activation threshold bounded strictly away from zero\.

###### Assumption A2\(Timescale Separation\)\.

The dictionary updates occur on a sufficiently slow timescale relative to the EMA tracking such that, ast→∞t\\to\\infty, the EMA buffer converges to the true marginal expected activations:h¯i\(t\)→𝔼x​\[\|hi∗​\(x\)\|\]\\bar\{h\}\_\{i\}^\{\(t\)\}\\to\\mathbb\{E\}\_\{x\}\[\|h\_\{i\}^\{\*\}\(x\)\|\]\.

Assumption[A2](https://arxiv.org/html/2605.05341#Thmassumption2)addresses the fundamental challenge of applying online, adaptive estimators to natural language, which follows a heavily skewed Zipfian \(power\-law\) distribution\[[30](https://arxiv.org/html/2605.05341#bib.bib27)\]\. While raw syntactic features fire constantly, highly specific semantic features \(e\.g\., proper nouns or niche factual knowledge\) may only appear once every several million tokens\. We satisfy the timescale separation requirement through two explicit mechanisms designed for this long\-tail distribution\. First, we hold the warmup schedule atρ​\(t\)=0\\rho\(t\)=0for millions of tokens to allow the buffer to safely initialize\. Second, we calibrate the EMA momentumβ\\betasuch that the buffer update rate, defined asηema:=\(1−β\)\\eta\_\{\\text\{ema\}\}:=\(1\-\\beta\), is several orders of magnitude larger than the dictionary learning rateηD\\eta\_\{D\}\. In our implementation, we observe empirical dictionary updates of𝒪​\(10−7\)\\mathcal\{O\}\(10^\{\-7\}\)per step, while our choice ofβ=0\.999\\beta=0\.999yields a buffer rate of10−310^\{\-3\}\. Due to this, the dictionary remains quasistatic relative to the effective memory horizon,ℋ:=11−β\\mathcal\{H\}:=\\frac\{1\}\{1\-\\beta\}\. Because the horizonℋ×B\\mathcal\{H\}\\times B\(whereBBis the batch size\) spans tens of millions of tokens, the EMA converges to true marginals without artificially decaying healthy, rare semantic concepts to zero\.

###### Assumption A3\(Mild Decay Condition of the Penalties\)\.

For theℓ1\\ell\_\{1\}penaltyλ1\\lambda\_\{1\}and the dynamic adaptive penalty weightswi=wi\(t\)w\_\{i\}=w\_\{i\}^\{\(t\)\}, the factormaxi∈𝒜global∗⁡λ1​wi\(t\)\\max\_\{i\\in\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\}\\lambda\_\{1\}w\_\{i\}^\{\(t\)\}becomes negligible ast→∞t\\to\\infty, while the factormini∉𝒜global∗⁡λ1​wi\(t\)\\min\_\{i\\notin\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\}\\lambda\_\{1\}w\_\{i\}^\{\(t\)\}diverges ast→∞t\\to\\infty

Assumption[A3](https://arxiv.org/html/2605.05341#Thmassumption3)is also quite mild as we discuss in Sections[3](https://arxiv.org/html/2605.05341#S3)and[A\.4](https://arxiv.org/html/2605.05341#A1.SS4)\. In our practical online formulation, this corresponds to an asymptotic decay rate condition onλ1\\lambda\_\{1\}in the adaptive LASSO literature\[[43](https://arxiv.org/html/2605.05341#bib.bib20)\]\. As shown in the proof of Theorem[3](https://arxiv.org/html/2605.05341#Thmtheorem3), under Assumptions[A1](https://arxiv.org/html/2605.05341#Thmassumption1)–[A2](https://arxiv.org/html/2605.05341#Thmassumption2), the adaptive penalty weights satisfy the following behavior: fori∈𝒜global∗i\\in\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\},wi\(t\)→𝔼x​\[\|hi∗​\(x\)\|\]−γ<c−γw^\{\(t\)\}\_\{i\}\\to\\mathbb\{E\}\_\{x\}\[\|h^\{\*\}\_\{i\}\(x\)\|\]^\{\-\\gamma\}<c^\{\-\\gamma\}, whereas fori∉𝒜global∗i\\notin\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\},wi\(t\)w^\{\(t\)\}\_\{i\}diverges\. If the weights are clipped to the interval\(wmin,wmax\)\(w\_\{\\min\},w\_\{\\max\}\), thenwi\(t\)w^\{\(t\)\}\_\{i\}reacheswmaxw\_\{\\max\}for inactive coordinates\. Thus, in practice, the assumption is satisfied whenλ1​c−γ≪1\\lambda\_\{1\}c^\{\-\\gamma\}\\ll 1andλ1​wmax≫1\\lambda\_\{1\}w\_\{\\max\}\\gg 1\.

### A\.3Oracle Selection Consistency

We now show that the idealized AEN\-SAE targeted objective asymptotically identifies this global support and eliminates magnitude shrinkage for valid features, regardless of dictionary coherence\.

###### Theorem 3\(Global Oracle Selection and Shrinkage Elimination\)\.

Assume[A1](https://arxiv.org/html/2605.05341#Thmassumption1)\-[A3](https://arxiv.org/html/2605.05341#Thmassumption3)\. Let the idealized adaptive penalty weights be constructed asw~i\(t\)=\(h¯i\(t\)\)−γ\\tilde\{w\}^\{\(t\)\}\_\{i\}=\(\\bar\{h\}^\{\(t\)\}\_\{i\}\)^\{\-\\gamma\}\. Then, the AEN\-SAE update

h^\(t\)=argminh​\(x\)∈ℝ\+ddict⁡\[12​‖x−D​h​\(x\)‖22\+λ1​∑i=1ddictw~i\(t\)​hi​\(x\)\+λ2​‖h​\(x\)‖22\]\\hat\{h\}^\{\(t\)\}=\\operatorname\{argmin\}\_\{h\(x\)\\in\\mathbb\{R\}\_\{\+\}^\{d\_\{\\text\{dict\}\}\}\}\\left\[\\frac\{1\}\{2\}\\\|x\-Dh\(x\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{1\}\\sum\_\{i=1\}^\{d\_\{\\text\{dict\}\}\}\\tilde\{w\}^\{\(t\)\}\_\{i\}h\_\{i\}\(x\)\+\\lambda\_\{2\}\\\|h\(x\)\\\|\_\{2\}^\{2\}\\right\]identifies the true global support with probability approaching 1\. That is: under the assumption𝔼x​\[‖x‖22\]<∞\\mathbb\{E\}\_\{x\}\[\\\|x\\\|\_\{2\}^\{2\}\]<\\infty, we have, for any thresholdτ∈\(0,c\)\\tau\\in\(0,c\),

limt→∞P​\(\{i:𝔼x​\[h^i\(t\)​\(x\)∣w~\(t\)\]\>τ\}=𝒜global∗\)=1\.\\lim\_\{t\\to\\infty\}P\\left\(\\\{i:\\mathbb\{E\}\_\{x\}\[\\hat\{h\}^\{\(t\)\}\_\{i\}\(x\)\\mid\\tilde\{w\}^\{\(t\)\}\]\>\\tau\\\}=\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\\right\)=1\.\(3\)Also,h^\(t\)\\hat\{h\}^\{\(t\)\}eliminates theℓ1\\ell\_\{1\}\-induced shrinkage:h^\(t\)​\(x\)→h∗​\(x\)\\hat\{h\}^\{\(t\)\}\(x\)\\to h^\{\*\}\(x\)\.

###### Proof\.

To establish selection consistency without the IRC, we follow several steps\. Letμi∗:=𝔼x​\[\|hi∗​\(x\)\|\]\\mu^\{\*\}\_\{i\}:=\\mathbb\{E\}\_\{x\}\[\|h^\{\*\}\_\{i\}\(x\)\|\]\.

Step 1: Weight separation\. Fixi∈𝒜global∗i\\in\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\. From the signal lower bound assumption \(Assumption[A1](https://arxiv.org/html/2605.05341#Thmassumption1)\), we haveμi∗\>0\\mu^\{\*\}\_\{i\}\>0\. The continuous mapping theorem with the timescale separation assumption \(Assumption[A2](https://arxiv.org/html/2605.05341#Thmassumption2)\) yieldsw~i\(t\)=\(h¯i\(t\)\)−γ→\(μi∗\)−γ≤c−γ<∞\\tilde\{w\}^\{\(t\)\}\_\{i\}=\(\\bar\{h\}^\{\(t\)\}\_\{i\}\)^\{\-\\gamma\}\\to\(\\mu^\{\*\}\_\{i\}\)^\{\-\\gamma\}\\leq c^\{\-\\gamma\}<\\inftyast→∞t\\to\\infty\. Next, fixi∉𝒜global∗i\\notin\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\. By definition,μi∗=0\\mu\_\{i\}^\{\*\}=0and thusw~i\(t\)→∞\\tilde\{w\}\_\{i\}^\{\(t\)\}\\to\\infty\. Note that these separations do not use Assumption[A3](https://arxiv.org/html/2605.05341#Thmassumption3)\.

Step 2: Convergence of the AEN\-SAE objective and shrinkage elimination\.The weight separations together with Assumption[A3](https://arxiv.org/html/2605.05341#Thmassumption3)give the pointwise convergence of the AEN\-SAE objective:

12​‖x−D​h​\(x\)‖22\+λ1​∑i=1ddictw~i\(t\)​hi​\(x\)\+λ2​‖h​\(x\)‖22→12​‖x−D​h​\(x\)‖22\+λ2​‖h​\(x\)‖22\+1hi​\(x\)=0,i∉𝒜global∗\.\\displaystyle\\begin\{split\}&\\frac\{1\}\{2\}\\\|x\-Dh\(x\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{1\}\\sum\_\{i=1\}^\{d\_\{\\text\{dict\}\}\}\\tilde\{w\}^\{\(t\)\}\_\{i\}h\_\{i\}\(x\)\+\\lambda\_\{2\}\\\|h\(x\)\\\|\_\{2\}^\{2\}\\\\ &\\to\\frac\{1\}\{2\}\\\|x\-Dh\(x\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{2\}\\\|h\(x\)\\\|\_\{2\}^\{2\}\+1\_\{h\_\{i\}\(x\)=0,i\\notin\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\}\.\\end\{split\}\(4\)The first term of the AEN\-SAE objective gives a uniform bound on the minimizers, so the minimization can be restricted to a compact subset of the feasible region\. Since the limiting objective that definesh∗h^\{\*\}is strongly convex and therefore has a unique minimizer, the minimizers satisfyh^\(t\)​\(x\)→h∗​\(x\)\\hat\{h\}^\{\(t\)\}\(x\)\\to h^\{\*\}\(x\), which shows the shrinkage elimination\. As𝔼x​\[‖x‖22\]<∞\\mathbb\{E\}\_\{x\}\[\\\|x\\\|\_\{2\}^\{2\}\]<\\infty, we also have𝔼x​\[h^\(t\)​\(x\)∣w~\(t\)\]→μ∗\\mathbb\{E\}\_\{x\}\[\\hat\{h\}^\{\(t\)\}\(x\)\\mid\\tilde\{w\}^\{\(t\)\}\]\\to\\mu^\{\*\}\.

Step 3: Global oracle selection\.Fixi∈𝒜global∗i\\in\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\. The convergence𝔼x​\[h^i\(t\)​\(x\)∣w~\(t\)\]→μi∗\\mathbb\{E\}\_\{x\}\[\\hat\{h\}^\{\(t\)\}\_\{i\}\(x\)\\mid\\tilde\{w\}^\{\(t\)\}\]\\to\\mu\_\{i\}^\{\*\}, together with the signal lower bound \(Assumption[A1](https://arxiv.org/html/2605.05341#Thmassumption1)\), implies that taking a sufficiently largettgives𝔼x​\[h^i\(t\)​\(x\)∣w~\(t\)\]\>τ\\mathbb\{E\}\_\{x\}\[\\hat\{h\}^\{\(t\)\}\_\{i\}\(x\)\\mid\\tilde\{w\}^\{\(t\)\}\]\>\\tau\. Meanwhile, fixi∉𝒜global∗i\\notin\\mathcal\{A\}^\{\*\}\_\{\\text\{global\}\}\. The convergence𝔼x​\[h^i\(t\)​\(x\)∣w~\(t\)\]→0\\mathbb\{E\}\_\{x\}\[\\hat\{h\}^\{\(t\)\}\_\{i\}\(x\)\\mid\\tilde\{w\}^\{\(t\)\}\]\\to 0implies that taking a sufficiently largettgives𝔼x​\[h^i\(t\)​\(x\)∣w~\(t\)\]<τ\\mathbb\{E\}\_\{x\}\[\\hat\{h\}^\{\(t\)\}\_\{i\}\(x\)\\mid\\tilde\{w\}^\{\(t\)\}\]<\\tau\. Therefore, we get \([3](https://arxiv.org/html/2605.05341#A1.E3)\), which completes the proof\. ∎

The oracle property has implications beyond feature selection; it also improves the geometric stability of the target manifold\. By allowing noise features to receive an infinitely diverging penalty, the adaptive weights effectively decouple the active signal from ambient dictionary noise, naturally suppressing the interaction leakage term of the Hoffman bound\.

###### Corollary 1\(Suppression of Interaction Leakage\)\.

Leth^\\hat\{h\}be the AEN\-SAE code given the weightswiw\_\{i\}s\. For active features𝒜=\{i:h^i≠0\}\\mathcal\{A\}=\\\{i:\\hat\{h\}\_\{i\}\\neq 0\\\}, letΔ​h𝒜\\Delta h\_\{\\mathcal\{A\}\}be a local perturbation along the active manifold\. If the minimum weight for inactive features satisfiesλ1​mini∈𝒜c⁡wi\>‖D𝒜c⊤​\(D𝒜​Δ​h𝒜−x\)‖∞\\lambda\_\{1\}\\min\_\{i\\in\\mathcal\{A\}^\{c\}\}w\_\{i\}\>\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}\(D\_\{\\mathcal\{A\}\}\\Delta h\_\{\\mathcal\{A\}\}\-x\)\\\|\_\{\\infty\}, then the strict subdifferential inclusion for the inactive orthant is preserved\. Consequently, the local Hoffman boundHHis independent of the interaction term‖D𝒜c⊤​D𝒜‖2\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\\\|\_\{2\}\.

###### Proof\.

We begin with the first order optimality condition to the AEN\-SAE objective:

−Di⊤​\(x−D​h^\)\+2​λ2​h^i\+λ1​wi⋅sign​\(h^i\)=0\\displaystyle\-D\_\{i\}^\{\\top\}\(x\-D\\hat\{h\}\)\+2\\lambda\_\{2\}\\hat\{h\}\_\{i\}\+\\lambda\_\{1\}w\_\{i\}\\cdot\\text\{sign\}\(\\hat\{h\}\_\{i\}\)=0\\quadfor​h^i≠0,\\displaystyle\\text\{for \}\\hat\{h\}\_\{i\}\\neq 0,\(5\)\|Di⊤​\(D​h^−x\)\|≤λ1​wi\\displaystyle\|D\_\{i\}^\{\\top\}\(D\\hat\{h\}\-x\)\|\\leq\\lambda\_\{1\}w\_\{i\}\\quadfor​h^i=0\.\\displaystyle\\text\{for \}\\hat\{h\}\_\{i\}=0\.\(6\)
A local perturbationΔ​h𝒜\\Delta h\_\{\\mathcal\{A\}\}in the active set shifts the gradient residual for the inactive set byD𝒜c⊤​\(D𝒜​Δ​h𝒜−x\)D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}\(D\_\{\\mathcal\{A\}\}\\Delta h\_\{\\mathcal\{A\}\}\-x\)\. For the polyhedral topology to remain stable \(i\.e\., no inactive feature improperly enters the support\), the first order optimality condition for the inactive orthant \([6](https://arxiv.org/html/2605.05341#A1.E6)\) must continue to hold under this perturbation\. This requires the subgradient barrier to absorb the induced gradient leakage:

‖D𝒜c⊤​\(D𝒜​Δ​h𝒜−x\)‖∞≤λ1​wi∈𝒜c\.\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}\(D\_\{\\mathcal\{A\}\}\\Delta h\_\{\\mathcal\{A\}\}\-x\)\\\|\_\{\\infty\}\\leq\\lambda\_\{1\}w\_\{i\\in\\mathcal\{A\}^\{c\}\}\.Under this condition, the inactive variablesh𝒜ch\_\{\\mathcal\{A\}^\{c\}\}are thus clamped to zero\. Therefore the local polyhedral stability is decoupled from the interaction leakage‖D𝒜c⊤​D𝒜‖2\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\\\|\_\{2\}, and is instead controlled by the active curvature established in Theorem[1](https://arxiv.org/html/2605.05341#Thmtheorem1), which completes the proof\. ∎

Note that in a standardℓ1\\ell\_\{1\}SAE, the subgradient barrier for inactive features is uniformlyλ1\\lambda\_\{1\}, meaning any cross\-correlation leakage exceedingλ1\\lambda\_\{1\}causes a discontinuous boundary shift, drivingH→∞H\\to\\infty\. In the AEN\-SAE, however, the clipped adaptive penaltyclip​\(\(h¯i\(t\)\)−γ,wmin,wmax\)\\text\{clip\}\(\(\\bar\{h\}^\{\(t\)\}\_\{i\}\)^\{\-\\gamma\},w\_\{\\min\},w\_\{\\max\}\)for inactive features diverges towardwmaxw\_\{\\max\}; see the proof of Theorem[3](https://arxiv.org/html/2605.05341#Thmtheorem3)\. So choosingλ1​wmax\\lambda\_\{1\}w\_\{\\max\}sufficiently large guarantees the stated condition\.

Corollary[1](https://arxiv.org/html/2605.05341#Thmcorollary1)completes our analysis of the Hoffman bound in Section[2\.3](https://arxiv.org/html/2605.05341#S2.SS3)\. While the overcomplete dictionaryDDis inherently highly collinear \(which would normally cause the numerator‖D𝒜c⊤​D𝒜‖2\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\\\|\_\{2\}to diverge\), the AEN\-SAE’s adaptive weights neutralize its impact\. Combined with Theorem[1](https://arxiv.org/html/2605.05341#Thmtheorem1), which bounds the denominator viaλ2\\lambda\_\{2\}, the AEN\-SAE aims to solve the root causes of polyhedral optimization pathologies\.

### A\.4Numerical Stability and Implementation Details

While Theorems[2](https://arxiv.org/html/2605.05341#Thmtheorem2)and[3](https://arxiv.org/html/2605.05341#Thmtheorem3)establish the stability and asymptotic consistency of the idealized AEN\-SAE, deploying this algorithm at the scale of LLMs \(ddict\>105d\_\{\\text\{dict\}\}\>10^\{5\}\) introduces several numerical and optimization constraints\. To translate the idealized mathematical oracle into the robust, highly scalable implementation used in our experiments, we introduce three necessary engineering modifications:

#### 1\. Scale invariance from reference features\.

The idealized weightw~i=\(h¯i\)−γ\\tilde\{w\}\_\{i\}=\(\\bar\{h\}\_\{i\}\)^\{\-\\gamma\}is highly sensitive to the raw magnitude of the LLM residual stream, which varies drastically across different models and intermediate layers\. To make the architecture transferable without exhaustive hyperparameter retuning, we introduce a scale\-invariant referenceh¯ref\(t\)\\bar\{h\}\_\{\\text\{ref\}\}^\{\(t\)\}, defined as the mean activity of the topK%K\\%features\. The penalty is calculated relative to this active cohort:wi\(t\)∝\(h¯ref\(t\)/h¯i\(t\)\)γw\_\{i\}^\{\(t\)\}\\propto\(\\bar\{h\}\_\{\\text\{ref\}\}^\{\(t\)\}/\\bar\{h\}\_\{i\}^\{\(t\)\}\)^\{\\gamma\}\. This ensures the relative penalty distribution remains identical regardless of whether the layer’s activations are bounded by10010^\{0\}or10210^\{2\}\.

#### 2\. Numerical bounds \(wminw\_\{\\min\}andwmaxw\_\{\\max\}\)\.

In theory, we rely on the idealized limitw~i→∞\\tilde\{w\}\_\{i\}\\to\\infty\. In 16\-bit \(BF16\) or 32\-bit floating\-point hardware, this unbounded growth causes numerical instability and gradient explosion\. We therefore clamp the maximum penalty towmaxw\_\{\\max\}\. As long asλ1​wmax\\lambda\_\{1\}w\_\{\\max\}is sufficiently larger than the local gradient residual, the KKT inequality holds, and the noise feature remains safely dead\. Conversely, we clamp the minimum penalty towminw\_\{\\min\}\(e\.g\.,0\.10\.1\) to prevent underflow, recognizing that in practice, shrinkage is not strictly zero, but safely bounded to a negligibleλ1​wmin\\lambda\_\{1\}w\_\{\\min\}\.

#### 3\. The cold\-start warmup schedule\.

The asymptotic proof naturally assumes the system has already reached the steady\-state expectation𝔼​\[\|hi∗\|\]\\mathbb\{E\}\[\|h\_\{i\}^\{\*\}\|\]\. However, during the initial steps of neural network training, feature activations are driven by random initialization noise\. If an ultimately useful feature initializes poorly, its EMA drops, its weight spikes towmaxw\_\{\\max\}, and it is permanently suppressed before the encoder has the opportunity to route data to it\. We mitigate this problem by interpolating the adaptive weights with a delayed linear warmup scheduleρ​\(t\)∈\[0,1\]\\rho\(t\)\\in\[0,1\]\. This effectively enforces a uniformℓ1\\ell\_\{1\}penalty until the dictionary has sufficiently aligned with the data manifold, satisfying the timescale separation requirement \(Assumption[A2](https://arxiv.org/html/2605.05341#Thmassumption2)\) before the adaptive penalties take effect\.

## Appendix BExperimental Setup and Full Training Details

#### Data pipeline and activation extraction\.

For LLM\-based experiments, we stream text from a deduplicated version of the Pile\[[13](https://arxiv.org/html/2605.05341#bib.bib37)\]\. Tokens are packed contiguously into fixed\-length sequences without padding to maximize throughput\. Residual\-stream activations are extracted from a frozen LLM via a forward hook at a chosen layer, and flattened into token\-level batches\.

To ensure consistent scaling across models, each activation vectorx∈ℝdmodelx\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}isℓ2\\ell\_\{2\}\-normalized per token by rescaling to‖x‖2=dmodel\\\|x\\\|\_\{2\}=\\sqrt\{d\_\{\\text\{model\}\}\}, yielding approximately unit variance per coordinate\.

A circular buffer is used to accumulate activations into fixed\-size SAE minibatches, improving hardware utilization and throughput\.

#### Training Procedure\.

At each training step, the SAE reconstructs activations and optimizes the reconstruction loss with regularization using Adam at a fixed learning rate\. Gradients are clipped to prevent instability and decoder columns are renormalized to unitℓ2\\ell\_\{2\}norm after each update\.

Encoder and decoder weights are initialized using Kaiming uniform initialization\. The decoder bias is initialized to zero and the decoder dictionary is explicitly normalized both at initialization and after each optimization step\.

For adaptive\-weight models, EMA statistics are updated at every step, and the adaptive reweighting mechanism is activated only after a warmup period\. This ensures stable initialization before adaptive penalties are applied\.

#### Baselines and compute normalization\.

We include a TopK baseline without auxiliary resampling or proxy\-gradient mechanisms\. This serves two purposes: \(i\) to normalize computational cost across methods, and \(ii\) to isolate whether AEN\-SAEs resolve feature starvation without relying on heuristic interventions\. No auxiliary resampling or proxy\-gradient methods are used for any model, including AEN\-SAEs\.

#### Metrics and evaluation\.

We log reconstruction and sparsity metrics, including mean squared error \(MSE\), explained variance,ℓ0\\ell\_\{0\}sparsity, and theℓ1/ℓ0\\ell\_\{1\}/\\ell\_\{0\}ratio\. Feature starvation is quantified via dead neuron statistics, including windowed dead\-feature percentage and recovery rates\. We additionally track shrinkage ratios and feature\-utilization summaries\.

To evaluate geometric structure, we compute dictionary diagnostics such as mutual coherence, condition numbers of active Gram matrices, and interaction leakage measures\. Feature\-utilization metrics include nromalized entropy, Gini coefficient, and mass concentration across feature subsets\.

For LLM\-based experiments, we assess downstream fidelity by patching SAE reconstructions into the frozen model and measuring cross\-entropy \(CE\) degradation, CE recovery relative to a batch\-mean ablation baseline, and KL divergence between output distributions\.

#### Validation protocol\.

Validation is performed in two regimes\. First, an online validation stream loops over a held\-out dataset shard and evaluates a subset of documents at each logging step to track training dynamics\. Second, a final validation pass is conducted over a non\-looping held\-out shard, which is exhausted once to compute aggregate metrics reported in tables\.

## Appendix CExtended Discussion of Metrics

In this appendix, we describe each metrics logged in detail\. Some of these are inspired by classical high\-dimensional statistics and may offer novel ways to measure SAE health\. Throughout, letx∈ℝdx\\in\\mathbb\{R\}^\{d\}denote an input activation,x^\\hat\{x\}its SAE reconstruction,h∈ℝddicth\\in\\mathbb\{R\}^\{d\_\{\\text\{dict\}\}\}the latent code, andD∈ℝdmodel×ddictD\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{dict\}\}\}the decoder dictionary with columnsdjd\_\{j\}\. Let𝒜\\mathcal\{A\}denote an active feature set,𝒜c\\mathcal\{A\}^\{c\}its complement, andϵ\>0\\epsilon\>0a small numerical constant\.

For all scalar metrics, we report the mean, standard deviation, and the 10th, 50th, and 90th percentiles\. Where relevant, we additionally track the minimum and maximum values\.

### C\.1Reconstruction and Sparsity

The most natural ways of measuring reconstruction performance are the mean\-squared error and explained variance

MSE=𝔼​\[‖x−x^‖22\],\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\\|x\-\\hat\{x\}\\\|\_\{2\}^\{2\}\\right\],Explained variance=1−∑j=1dmodelVar​\(xj−x^j\)∑j=1dmodelVar​\(xj\)\.\\displaystyle=1\-\\frac\{\\sum\_\{j=1\}^\{d\_\{\\text\{model\}\}\}\\mathrm\{Var\}\(x\_\{j\}\-\\hat\{x\}\_\{j\}\)\}\{\\sum\_\{j=1\}^\{d\_\{\\text\{model\}\}\}\\mathrm\{Var\}\(x\_\{j\}\)\}\.
For an SAE, the goal is for the internal representation to be sparse\. Hence, we measure the average sparsity by the number of active features:

ℓ0active features=𝔼​\[∑i=1ddict𝟏​\{\|hi\|\>ϵ\}\]\.\\text\{$\\ell\_\{0\}$ active features\}=\\mathbb\{E\}\\\!\\left\[\\sum\_\{i=1\}^\{d\_\{\\text\{dict\}\}\}\\mathbf\{1\}\\\{\|h\_\{i\}\|\>\\epsilon\\\}\\right\]\.The effect of shrinkage bias can be measured with the ratio of reconstructed to originalℓ1\\ell\_\{1\}magnitude, or theℓ1\\ell\_\{1\}magnitude per active feature\. Respectively:

Feature shrinkage ratio=𝔼​\[‖x^‖1\]𝔼​\[‖x‖1\],\\displaystyle=\\frac\{\\mathbb\{E\}\\\!\\left\[\\\|\\hat\{x\}\\\|\_\{1\}\\right\]\}\{\\mathbb\{E\}\\\!\\left\[\\\|x\\\|\_\{1\}\\right\]\},ℓ1/ℓ0\\ell\_\{1\}/\\ell\_\{0\}ratio=𝔼​\[‖h‖1max⁡\(‖h‖0,1\)\],\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\frac\{\\\|h\\\|\_\{1\}\}\{\\max\(\\\|h\\\|\_\{0\},1\)\}\\right\],where a feature shrinkage ratio of 1 means the magnitude of the original signal has been preserved\.

### C\.2Dead Neuron Metrics

A central focus of our study is on the existence and recovery of dead neurons\. Letmim\_\{i\}be the maximum absolute activation of featureiiover a fixed window of training steps\. We define the dead neuron percentage as:

Dead neurons pct=100⋅1ddict​∑i=1ddict𝟏​\{mi≤ϵ\}\.\\text\{Dead neurons pct\}=100\\cdot\\frac\{1\}\{d\_\{\\text\{dict\}\}\}\\sum\_\{i=1\}^\{d\_\{\\text\{dict\}\}\}\\mathbf\{1\}\\\{m\_\{i\}\\leq\\epsilon\\\}\.Given two consecutive windows with maximamiprevm\_\{i\}^\{\\text\{prev\}\}andmicurrm\_\{i\}^\{\\text\{curr\}\}, the recovery rate is:

Dead neuron recovery rate=∑i𝟏​\{miprev≤ϵ\}​𝟏​\{micurr\>ϵ\}∑i𝟏​\{miprev≤ϵ\},\\text\{Dead neuron recovery rate\}=\\frac\{\\sum\_\{i\}\\mathbf\{1\}\\\{m\_\{i\}^\{\\text\{prev\}\}\\leq\\epsilon\\\}\\mathbf\{1\}\\\{m\_\{i\}^\{\\text\{curr\}\}\>\\epsilon\\\}\}\{\\sum\_\{i\}\\mathbf\{1\}\\\{m\_\{i\}^\{\\text\{prev\}\}\\leq\\epsilon\\\}\},with the convention that the value is 0 when the denominator is 0\. The window size is set to 10,000 steps in our experiments\. This leads to between ten million and fifty million tokens seen per window, allowing rare features to fire\.

For LLM batches, we additionally log the percentage of features that never activate within the batch—though this tends to be a noisy metrics as rare semantic features are often marked as dead\.

Dead neurons batch pct=100⋅1ddict​∑i=1ddict𝟏​\{maxb⁡\|hb,i\|≤ϵ\}\.\\text\{Dead neurons batch pct\}=100\\cdot\\frac\{1\}\{d\_\{\\text\{dict\}\}\}\\sum\_\{i=1\}^\{d\_\{\\text\{dict\}\}\}\\mathbf\{1\}\\\!\\left\\\{\\max\_\{b\}\|h\_\{b,i\}\|\\leq\\epsilon\\right\\\}\.

### C\.3Adaptive\-Weight Metrics

For adaptive LASSO and adaptive elastic net SAEs, measuring the concentration of the weights is useful to observe degeneracy\. Using the same notation as in Section[3](https://arxiv.org/html/2605.05341#S3), the activation effective sample size \(ESS\) is:

ESS=\(∑iai\)2∑iai2,ai=max⁡\(h¯i,0\)\.\\text\{ESS\}=\\frac\{\\left\(\\sum\_\{i\}a\_\{i\}\\right\)^\{2\}\}\{\\sum\_\{i\}a\_\{i\}^\{2\}\},\\qquad a\_\{i\}=\\max\(\\bar\{h\}\_\{i\},0\)\.We also report summary statistics of the adaptive weights, together with the fractions pinned near the bounds, which can be used to adjust the weight clipping in the case of degeneracy:

Pinned max weight pct=100⋅1ddict​∑i𝟏​\{wi≤wmin\+ε\},\\displaystyle=100\\cdot\\frac\{1\}\{d\_\{\\text\{dict\}\}\}\\sum\_\{i\}\\mathbf\{1\}\\\{w\_\{i\}\\leq w\_\{\\min\}\+\\varepsilon\\\},Pinned min weight pct=100⋅1ddict​∑i𝟏​\{wi≥wmax−ε\},\\displaystyle=100\\cdot\\frac\{1\}\{d\_\{\\text\{dict\}\}\}\\sum\_\{i\}\\mathbf\{1\}\\\{w\_\{i\}\\geq w\_\{\\max\}\-\\varepsilon\\\},

### C\.4Feature\-Utilization Metrics

We argue that the distribution of firing features is a proxy for the health of an SAE\. Letrir\_\{i\}be the per\-feature firing rate, estimated either from the EMA firing rate or from batch frequencies\. The normalized entropy is:

Normalized entropy=−∑ipi​log⁡pilog⁡ddict,pi=ri∑jrj\.\\text\{Normalized entropy\}=\\frac\{\-\\sum\_\{i\}p\_\{i\}\\log p\_\{i\}\}\{\\log d\_\{\\text\{dict\}\}\},\\qquad p\_\{i\}=\\frac\{r\_\{i\}\}\{\\sum\_\{j\}r\_\{j\}\}\.The concentration of features firing can also be measured\. The top\-nn% feature\-firing mass is

Top\-n% mass pct=100⋅∑i∈top​1%pi\.\\text\{Top\-$n$\\% mass pct\}=100\\cdot\\sum\_\{i\\in\\mathrm\{top\}\\,1\\%\}p\_\{i\}\.The Gini coefficent, as a measure of computing inequality in firing rates, is computed from the sorted ratesr\(1\)≤⋯≤r\(n\)r\_\{\(1\)\}\\leq\\cdots\\leq r\_\{\(n\)\}as:

Gini=2​∑i=1ni​r\(i\)n​∑iri−n\+1n\.\\text\{Gini\}=\\frac\{2\\sum\_\{i=1\}^\{n\}i\\,r\_\{\(i\)\}\}\{n\\sum\_\{i\}r\_\{i\}\}\-\\frac\{n\+1\}\{n\}\.

### C\.5Dictionary Geometry Metrics

Let𝒜\\mathcal\{A\}be the active set for a given batch, with decoder blocksD𝒜D\_\{\\mathcal\{A\}\}andD𝒜cD\_\{\\mathcal\{A\}^\{c\}\}\. Since the Frobenius norm upper bounds the spectral norm \(‖D‖2≤‖D‖F\\\|D\\\|\_\{2\}\\leq\\\|D\\\|\_\{F\}\), a proxy for the interaction leakage, which is one of the factors controlling the stability of LASSO solution maps, can be measured as

Frobenius interaction=‖D𝒜c⊤​D𝒜‖F\.\\text\{Frobenius interaction\}=\\\|D\_\{\\mathcal\{A\}^\{c\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}\\\|\_\{F\}\.The other factor in stability is the curvature of the active Gram matrix isG𝒜=D𝒜⊤​D𝒜G\_\{\\mathcal\{A\}\}=D\_\{\\mathcal\{A\}\}^\{\\top\}D\_\{\\mathcal\{A\}\}, from which we report:

Active min eigenvalue=λmin​\(G𝒜\),\\displaystyle=\\lambda\_\{\\min\}\(G\_\{\\mathcal\{A\}\}\),Active max eigenvalue=λmax​\(G𝒜\),\\displaystyle=\\lambda\_\{\\max\}\(G\_\{\\mathcal\{A\}\}\),Active condition number=λmax​\(G𝒜\)max⁡\(λmin​\(G𝒜\),ϵ\)\.\\displaystyle=\\frac\{\\lambda\_\{\\max\}\(G\_\{\\mathcal\{A\}\}\)\}\{\\max\(\\lambda\_\{\\min\}\(G\_\{\\mathcal\{A\}\}\),\\epsilon\)\}\.Although metrics for the full Gram matrixG=D⊤​DG=D^\{\\top\}Dwill be intractable for large models, they are not prohibitive in the active space since it aims to be sparse with\|𝒜\|\|\\mathcal\{A\}\|on the scale of a few hundred features at most\.

As demonstrated, dictionary coherence can affect the choice of appropriate SAE architecture greatly, as predicted in the classical theory of LASSO\[[38](https://arxiv.org/html/2605.05341#bib.bib12)\]\. Letd~j=dj/‖dj‖2\\tilde\{d\}\_\{j\}=d\_\{j\}/\\\|d\_\{j\}\\\|\_\{2\}\. The nearest\-neighbour coherence of featurejjiscj=maxk≠j⁡\|d~j⊤​d~k\|c\_\{j\}=\\max\_\{k\\neq j\}\\left\|\\tilde\{d\}\_\{j\}^\{\\top\}\\tilde\{d\}\_\{k\}\\right\|\. We report:

Max mean cosine similarity=1ddict​∑jcj,\\displaystyle=\\frac\{1\}\{d\_\{\\text\{dict\}\}\}\\sum\_\{j\}c\_\{j\},Dictionary coherence nearest\-neighbour max=maxj⁡cj,\\displaystyle=\\max\_\{j\}c\_\{j\},Dictionary coherence max=maxj≠k⁡\|d~j⊤​d~k\|,\\displaystyle=\\max\_\{j\\neq k\}\\left\|\\tilde\{d\}\_\{j\}^\{\\top\}\\tilde\{d\}\_\{k\}\\right\|,Dictionary coherence mean=1ddict​\(ddict−1\)​∑j≠k\|d~j⊤​d~k\|\.\\displaystyle=\\frac\{1\}\{d\_\{\\text\{dict\}\}\(d\_\{\\text\{dict\}\}\-1\)\}\\sum\_\{j\\neq k\}\\left\|\\tilde\{d\}\_\{j\}^\{\\top\}\\tilde\{d\}\_\{k\}\\right\|\.

### C\.6LLM Downstream\-Fidelity Metrics

For validation on frozen language models, we evaluate next\-token cross\-entropy under three forward\-pass conditions: the original uninterrupted model \(clean\), the model with SAE reconstructions patched into the residual stream \(patched\), and an ablated baseline \(baseline\)\.

Letzzdenote logits andttdenote tokens\. The next\-token cross\-entropy is:

CE​\(z,t\)=−1N​∑n=1Nlog⁡pz​\(tn\+1∣t≤n\)\.\\mathrm\{CE\}\(z,t\)=\-\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\log p\_\{z\}\(t\_\{n\+1\}\\mid t\_\{\\leq n\}\)\.
We use a batch\-mean ablation for our baseline logits \(zbaselinez\_\{\\mathrm\{baseline\}\}\) rather than standard zero\-ablation\. As noted in recent mechanistic interpretability literature, zero\-ablating intermediate residual streams in modern architectures pushes the network off the valid activation manifold\[[28](https://arxiv.org/html/2605.05341#bib.bib33)\]\. Specifically, substituting zero vectors can destabilize rotary position embeddings \(RoPE\)\[[36](https://arxiv.org/html/2605.05341#bib.bib36)\]and induce uncalibrated out\-of\-distribution shifts that artificially inflate the apparent impact of the ablation\[[31](https://arxiv.org/html/2605.05341#bib.bib35)\]\. Mean ablation mitigates this geometric instability by keeping the baseline interventions closer to the true data distribution\[[23](https://arxiv.org/html/2605.05341#bib.bib34)\]\. While it constitutes a strictly stronger, more challenging baseline—as the mean activation inherently reconstructs high\-frequency, common features and lacks only fine\-grained semantic nuance—it provides a much cleaner, causally rigorous isolation of the SAE’s ability to recover token\-specific representations\.

The batch\-level metrics are:

Clean CE=CE​\(zclean,t\),\\displaystyle=\\mathrm\{CE\}\(z\_\{\\mathrm\{clean\}\},t\),Patched CE=CE​\(zpatched,t\),\\displaystyle=\\mathrm\{CE\}\(z\_\{\\mathrm\{patched\}\},t\),Baseline CE=CE​\(zbaseline,t\),\\displaystyle=\\mathrm\{CE\}\(z\_\{\\mathrm\{baseline\}\},t\),CE degradation=patched​\_​loss−clean​\_​loss,\\displaystyle=\\mathrm\{patched\\\_loss\}\-\\mathrm\{clean\\\_loss\},CE recovered=1−Patched CE−Clean CEBaseline CE−Clean CE,\\displaystyle=1\-\\frac\{\\text\{Patched CE\}\-\\text\{Clean CE\}\}\{\\text\{Baseline CE\}\-\\text\{Clean CE\}\},We also compute the KL divergence between clean and patched logits:

KL​\(pclean∥ppatched\)=∑vpclean​\(v\)​log⁡pclean​\(v\)ppatched​\(v\)\.\\mathrm\{KL\}\(p\_\{\\mathrm\{clean\}\}\\,\\\|\\,p\_\{\\mathrm\{patched\}\}\)=\\sum\_\{v\}p\_\{\\mathrm\{clean\}\}\(v\)\\log\\frac\{p\_\{\\mathrm\{clean\}\}\(v\)\}\{p\_\{\\mathrm\{patched\}\}\(v\)\}\.In implementation, logits are sanitized to finite values before computing the softmax and KL\. A set of final test metrics are reported for all metrics on a corpus of held\-out documents\.

### C\.7Training\-Health and Throughput Metrics

Finally, we log gradient norms and update ratios for the global network, as well as decomposed at the encoder/decoder level\. This is used for validating the timescale separation assumption\.

Global grad norm=‖∇θ‖2,\\displaystyle=\\\|\\nabla\\theta\\\|\_\{2\},Encoder grad norm=‖∇θenc‖2,\\displaystyle=\\\|\\nabla\\theta\_\{\\text\{enc\}\}\\\|\_\{2\},Decoder grad norm=‖∇θdec‖2,\\displaystyle=\\\|\\nabla\\theta\_\{\\text\{dec\}\}\\\|\_\{2\},The normalized update magnitudes are thus:

Encoder update ratio=η​‖∇θenc‖2‖θenc‖2,\\displaystyle=\\eta\\frac\{\\\|\\nabla\\theta\_\{\\mathrm\{enc\}\}\\\|\_\{2\}\}\{\\\|\\theta\_\{\\mathrm\{enc\}\}\\\|\_\{2\}\},Decoder update ratio=η​‖∇θdec‖2‖θdec‖2\.\\displaystyle=\\eta\\frac\{\\\|\\nabla\\theta\_\{\\mathrm\{dec\}\}\\\|\_\{2\}\}\{\\\|\\theta\_\{\\mathrm\{dec\}\}\\\|\_\{2\}\}\.We also profile FLOPs per step and cumulatively in order to compare the computational cost of different architectures\.

## Appendix DExtended Spike Model Analysis

### D\.1Data Generation and Training Protocol

To study the effect of dictionary coherence in a controlled setting, we consider a spiked covariance model followingWainwright \[[38](https://arxiv.org/html/2605.05341#bib.bib12)\]\. Each dictionary atomdj∈ℝdmodeld\_\{j\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}is constructed as a mixture of an independent componentuju\_\{j\}and a shared spikevv:

dj=1−ρ​uj\+ρ​v‖1−ρ​uj\+ρ​v‖2,d\_\{j\}=\\frac\{\\sqrt\{1\-\\rho\}\\,u\_\{j\}\+\\sqrt\{\\rho\}\\,v\}\{\\\|\\sqrt\{1\-\\rho\}\\,u\_\{j\}\+\\sqrt\{\\rho\}\\,v\\\|\_\{2\}\},whereuj∼𝒩​\(0,I\)u\_\{j\}\\sim\\mathcal\{N\}\(0,I\)are independent directions andv∼𝒩​\(0,I\)v\\sim\\mathcal\{N\}\(0,I\)is shared across all atoms\. The parameterρ∈\[0,1\]\\rho\\in\[0,1\]controls the expected pairwise coherence between atoms, interpolating between independent dictionaries \(ρ=0\\rho=0\) and highly collinear ones \(ρ→1\\rho\\to 1\)\.

For each coherence levelρ∈\{0,0\.3,0\.5,0\.7,0\.9\}\\rho\\in\\\{0,0\.3,0\.5,0\.7,0\.9\\\}, we construct a fixed teacher dictionaryD∗∈ℝdmodel×ddictD^\{\*\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{dict\}\}\}withdmodel=256d\_\{\\text\{model\}\}=256andddict=1024d\_\{\\text\{dict\}\}=1024\(a4×4\\timesovercomplete representation\)\.

Synthetic data is generated by sampling sparse codesh∗∈ℝddicth^\{\*\}\\in\\mathbb\{R\}^\{d\_\{\\text\{dict\}\}\}with exactlyk=16k=16nonzero entries\. Active indices are drawn uniformly without replacement, and nonzero magnitudes are sampled independently fromUniform​\[1,3\]\\mathrm\{Uniform\}\[1,3\]\. Observations are then formed asx∗=D∗​h∗\.x^\{\*\}=D^\{\*\}h^\{\*\}\.

All SAE variants are trained on the synthetic data using Adam for50,00050\{,\}000steps with batch size256256and learning rate10−310^\{\-3\}\. Hyperparameters are swept to target specific sparsity levels, and performance is evaluated in terms of reconstruction accuracy, sparsity, and feature utilization\.

### D\.2Low\-Coherence Regime

Whenρ=0\\rho=0, dictionary atoms are approximately orthogonal, and the sparse recovery problem is well\-conditioned\. In this regime, all architectures perform comparably in terms of reconstruction, and feature starvation is largely absent for continuous relaxation methods\.

Hard\-masking methods such as TopK perform well because feature selection is trivial in the absence of correlation\. AEN\-SAE maintains full dictionary utilization \(no dead features\) while incurring a modest reduction in explained variance due to theℓ2\\ell\_\{2\}regularization term\.

### D\.3High\-Coherence Regime

As coherence increases, the geometry of the sparse recovery problem deteriorates\. Atρ=0\.9\\rho=0\.9, atoms become highly collinear, violating the irrepresentability condition and inducing instability inℓ1\\ell\_\{1\}\-based methods\.

#### Continuous Relaxations\.

Standardℓ1\\ell\_\{1\}\-regularized SAEs exhibit strong sensitivity to hyperparameters, oscillating between dense, high\-shrinkage solutions and degenerate sparse representations\. Adaptive LASSO, while theoretically appealing, becomes numerically unstable in this regime; in our experiments, the majority of hyperparameter configurations collapse to degenerate solutions \(ℓ0<2\\ell\_\{0\}<2\), which is consistent with known limitations ofℓ1\\ell\_\{1\}methods under high collinearity\.

#### Hard\-Masking Methods\.

TopK exhibits extremely severe feature starvation in the high\-coherence regime\. Because feature selection is based on raw activation magnitude, highly correlated features compete, which leads to repeated selection of a small subset of dominant atoms\. As a result, over95%95\\%of dictionary elements become permanently inactive across a range of sparsity levels\.

#### AEN\-SAE Behavior\.

AEN\-SAE mitigates the effects of hard\-masking methods by combining adaptive reweighting with anℓ2\\ell\_\{2\}regularization term\. Theℓ2\\ell\_\{2\}term stabilizes the curvature of the optimization landscape, while adaptive weights suppress redundant features and encourage diversification of the active set\.

Empirically, AEN reduces dead features to approximately40%40\\%at typical sparsity levels\. While feature starvation is not completely eliminated, the reduction is substantial compared to hard\-masking methods\. Additionally, AEN achieves improved geometric conditioning, reducing the active\-set condition number \(e\.g\.,κ≈21\\kappa\\approx 21compared toκ≈769\\kappa\\approx 769for standard elastic net\) and lowering interaction leakage between active and inactive features\.

### D\.4Discussion

Our results highlight a fundamental limitation ofℓ1\\ell\_\{1\}\-based sparse coding under coherence: as dictionary atoms become increasingly correlated, the associated solution map becomes unstable and difficult to approximate with shallow encoders\. Even in this controlled setting, feature starvation remains significant, suggesting that it is not purely an optimization artifact but a structural consequence of coherence\.

AEN\-SAE provides a principled mechanism for mitigating these effects by directly controlling the geometric factors governing stability\. While it does not fully eliminate feature starvation, it substantially improves feature utilization and robustness without relying on auxiliary heuristics such as resampling or proxy gradients\.

## Appendix EExtended Analysis on Pythia 70M

### E\.1Experimental Setup

We evaluate SAE variants on activations extracted from layer 3 of a frozen Pythia 70M model\[[3](https://arxiv.org/html/2605.05341#bib.bib10)\]\. The residual stream has dimensiondmodel=512d\_\{\\text\{model\}\}=512and we train an8×8\\timesovercomplete dictionary withddict=4096d\_\{\\text\{dict\}\}=4096\.

Training data is drawn from a deduplicated version of the Pile\[[13](https://arxiv.org/html/2605.05341#bib.bib37)\], with sequence length 128 and per\-tokenℓ2\\ell\_\{2\}normalization applied to activations\. Optimization proceeds using Adam for 25,000 steps \(approximately204\.8204\.8M tokens\), with batch size 8,192 and learning rate10−310^\{\-3\}\. AEN\-SAEs employ a 4,000\-step warmup for adaptive weights\. Validation is performed on a held\-out shard of 5,000 documents\.

### E\.2Hyperparameter Tuning and Timescale Separation

AEN\-SAEs introduce several hyperparameters, including theℓ2\\ell\_\{2\}regularization strengthλ2\\lambda\_\{2\}, the adaptive weighting exponentγ\\gamma, the EMA momentumβ\\beta, weight bounds\[wmin,wmax\]\[w\_\{\\min\},w\_\{\\max\}\], and the reference percentilepp\. A full hyperparameter sweep is provided in the next section, Appendix[E\.3](https://arxiv.org/html/2605.05341#A5.SS3)\.

In practice, these parameters exhibit strong stability across runs\. Following classical adaptive elastic net protocol, we setγ∈\{0\.5,1\.0\}\\gamma\\in\\\{0\.5,1\.0\\\}and chooseλ2≤λ1\\lambda\_\{2\}\\leq\\lambda\_\{1\}, typically setting it to a small value and treating it as a perturbation for structuralℓ2\\ell\_\{2\}regularization\. Adaptive weights are clipped to\[0\.01,10\.0\]\[0\.01,10\.0\]for numerical stability\.

A key theoretical requirement is timescale separation between the EMA buffer and dictionary updates\. Withβ∈\{0\.999,0\.9999\}\\beta\\in\\\{0\.999,0\.9999\\\}, the EMA update rate\(1−β\)\(1\-\\beta\)is10−310^\{\-3\}–10−410^\{\-4\}, corresponding to a memory horizon spanning tens of millions of tokens\. Empirically, we observe that parameter updates occur at magnitude𝒪​\(10−7\)\\mathcal\{O\}\(10^\{\-7\}\)per step, which implies that the activation statistics evolve several orders of magnitude faster than the dictionary\. This satisfies the timescale separation assumption required for stable adaptive weighting\.

To initialize the EMA buffer, we couple the warmup duration to the EMA horizon viaTwarmup≈\(1−β\)−1T\_\{\\text\{warmup\}\}\\approx\(1\-\\beta\)^\{\-1\}, ensuring that adaptive weights are only applied after sufficient signal accumulation\. Finally, because our adaptive weights are scaled relative to the topp=0\.05p=0\.05percentile of features, the entire AEN\-SAE architecture is naturally scale\-invariant; good hyperparameter choices should generalize to large models\. Overall, AEN\-SAEs introduce zero meaningful computational overhead—measuring less than a 0\.1% increase in FLOPs per step compared to a purely TopK implementation—and only require tuning the base penaltyλ1\\lambda\_\{1\}to hit target sparsity bottlenecks just as in vanilla SAEs\.

### E\.3Hyperparameter Sweep for Pythia 70M

For a robust comparison, we conducted hyperparameter sweeps for all SAE architectures\. The complete grid of hyperparameters evaluated in our Pythia 70M experiments is detailed in Table[5](https://arxiv.org/html/2605.05341#A5.T5)\.

For our adaptive architectures \(adaptive LASSO and AEN\-SAE\), several structural hyperparameters were held constant across runs to isolate the effect of the sparsity penalty\. The bounds on the adaptive weights were fixed to\[wmin,wmax\]=\[0\.01,10\.0\]\[w\_\{\\min\},w\_\{\\max\}\]=\[0\.01,10\.0\]to ensure numerical stability without causing gradient explosion\. To prevent premature feature death during the noisy initial phase of optimization, the adaptive weighting mechanism utilized a delayed warmup schedule: the effective penalty was held strictly uniform \(ρ​\(t\)=0\\rho\(t\)=0\) for the first 4,000 training steps, followed by a linear interpolation to full adaptive weighting \(ρ​\(t\)=1\\rho\(t\)=1\) over the subsequent 2,000 steps\.

Table 5:Pythia 70M hyperparameter sweep grid\.The combinatorial sweep of hyperparameter configurations evaluated for each sparse autoencoder architecture\.
### E\.4Coherence\-Induced Failure Modes

Unlike synthetic sparse coding problems, LLM activations exhibit strong contextual correlations, inducing a highly coherent dictionary as characterized by high mutual cosine similarity between atoms\[[43](https://arxiv.org/html/2605.05341#bib.bib20),[11](https://arxiv.org/html/2605.05341#bib.bib41)\]\. This violates standard sparse recovery assumptions such as the irrepresentability condition and leads to predictable failure modes across SAE architectures\.

#### Vanillaℓ1\\ell\_\{1\}\.

To suppress correlated noise features, theℓ1\\ell\_\{1\}penalty must be increased, which induces significant shrinkage bias\. Atℓ0≈22\\ell\_\{0\}\\approx 22, we observe magnitude suppression of approximately25%25\\%, which degrades both reconstruction fidelity and downstream performance\. The method is also highly sensitive to hyperparameter choice, reflecting the instability ofℓ1\\ell\_\{1\}regularization under collinearity\.

#### Elastic net\.

The addition of anℓ2\\ell\_\{2\}penalty stabilizes optimization but leads to dense and poorly conditioned solutions\. In our experiments, elastic net produces representations withℓ0\>200\\ell\_\{0\}\>200and extremely high condition numbers \(medianκ\>1013\\kappa\>10^\{13\}\), indicating that it fails to meaningfully separate features in coherent regimes\.

#### Adaptive LASSO\.

Although adaptive LASSO benefits from strong theoretical guarantees in low\-coherence settings, it becomes unstable in this regime due to the absence of a structural curvature term\. Across our sweeps, a large fraction of configurations collapse to degenerate solutions, failing to capture meaningful structure\. This behavior is consistent with known limitations ofℓ1\\ell\_\{1\}\-based methods under strong feature correlations\[[42](https://arxiv.org/html/2605.05341#bib.bib22)\]; see also Theorem[2](https://arxiv.org/html/2605.05341#Thmtheorem2)and the subsequent discussion\.

### E\.5Reconstruction–Utilization Trade\-off

Figure[3](https://arxiv.org/html/2605.05341#S4.F3)and Table[2](https://arxiv.org/html/2605.05341#S4.T2)demonstrate a consistent trade\-off between reconstruction accuracy and feature utilization\.

TopK achieves higher explained variance across sparsity levels \(e\.g\.,0\.760\.76atℓ0≈32\\ell\_\{0\}\\approx 32\), but it does so by concentrating activation mass into a small number of highly redundant features\. This phenomenon is reflected in high maximum coherence \(≈0\.99\\approx 0\.99\) and moderate levels of feature starvation\.

In contrast, AEN\-SAEs distribute activation mass across correlated feature groups\. While this leads to a modest reduction in explained variance, it significantly improves feature utilization, reduces redundancy, and produces a more balanced dictionary\.

### E\.6Feature Utilization and Activation Concentration

At relaxed sparsity levels \(ℓ0≈128\\ell\_\{0\}\\approx 128\), the differences between methods become more evident\. TopK concentrates approximately86%86\\%of activation mass in the top10%10\\%of features, yielding a high Gini coefficient \(0\.9030\.903\) and persistent feature starvation\. AEN\-SAEs mitigate this concentration, reducing the Gini coefficient to0\.6700\.670and increasing entropy, while lowering dead features to approximately3%3\\%\. This reflects a more uniform utilization of the dictionary\.

### E\.7Mechanism: Adaptive Reweighting

The improved utilization arises from the adaptive weighting mechanism\. Frequently activated features accumulate larger EMA values, increasing their effective penalty and preventing them from dominating the representation\. Conversely, underutilized features receive relatively lower penalties, allowing them to become active\. This induces a self\-balancing effect that redistributes activation mass away from redundant hubs and toward underutilized features, which improves coverage of the representation space without requiring explicit resampling or auxiliary losses\.

### E\.8Discussion

Overall, our results demonstrate that coherence is a primary driver of SAE failure in real\-world settings\. Hard\-masking methods achieve strong reconstruction performance by collapsing onto a small subset of features, while AEN\-SAEs provide a principled mechanism for improving feature utilization and geometric stability with minimal computational overhead and no auxiliary heuristics\.