Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens

# Explaining Logit-Lens Scores in Features Instead of Tokens
Source: [https://arxiv.org/html/2609.01936](https://arxiv.org/html/2609.01936)
## Sparse Readout Prism: Explaining Logit\-LensScores in Features Instead of Tokens

Xinchi QiuNicholas D\. LaneDepartment of Computer Science and TechnologyUniversity of Cambridge

###### Abstract

A language model’s prediction of its next token develops across layers, and lens methods track this process by decoding intermediate hidden states into tokens\. But a lens reading reflects both the hidden state and the readout \(the unembedding matrix\) used to decode it\. Many lenses are fit on a corpus, and we show that two lenses differing only in their fitting corpus can report different tokens for the same hidden states\. We call this dependence corpus conditionality\. To examine readout structure independently of the fitting corpus, we introduce Sparse Readout Prism \(SRP\), which decomposes the readout using only its weights and expresses any token logit or logit difference as a sum of contributions from sparse readout features\. This reveals readout features as a new unit of analysis for lens readings, exposing structure that token identities can obscure and enabling comparisons across tokens, contexts, layers, and lenses\. Replacing the original readout with SRP’s sparse approximation reconstructs 8\.9–17\.3 percentage points more of the tested logit differences than the strongest of six baselines built on geometric relations among readout rows\. Ablating features shifts logit differences in proportion to their SRP contributions\. Although token readings vary with the fitting corpus, the dominant readout feature remains stable\. Because SRP uses no corpus in its construction, it provides a control independent of the fitting corpus for lens analyses\.

## 1Introduction

Figure 1:Where a lens reading comes from\.\(A\)The logit lens scoresbugagainstinsectfor Qwen3\.5\-2B on the prompt “The programmer reproduced the crash and filed a”\.\(B\)SRP decomposes that margin into signed readout feature contributions, in logit units\. The leading features all concern software defects, with no feature for the insect sense among them\. Bars give each feature’s id and the rows of the LM head that score highest on it, several of them non\-English \(§[5](https://arxiv.org/html/2609.01936#S5)\)\. Small terms and token form terms \(casing, spacing, and punctuation variants\) are hidden\. The sum and residual use the full decomposition\.In a transformer language model, each output logit is a dot product between the normalized final state presented to the language model head and one row of the unembedding matrixWUW\_\{U\}, also called the LM head\. We call this matrix the readout, since the model reads out its prediction from the final state alone\. Yet states at earlier layers already carry information about the eventual output\. Methods in the lens style\([nostalgebraist, 2020](https://arxiv.org/html/2609.01936#bib.bib53);[Belrose et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib3);[Pal et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib55);[Ghandeharioun et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib24)\)exploit this, tracking how that information accumulates by decoding those states through the same readout\. These readings are often taken as evidence about the computation behind them, as when predominantly English tokens are interpreted as an English intermediate working language\([Wendler et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib75);[Schut et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib67)\)\. The logit lens appliesWUW\_\{U\}unchanged, while fitted lenses such as the tuned lens\([Belrose et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib3)\)and the Jacobian lens\([Gurnee et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib28)\)first transport the state into the space the LM head reads, through a map estimated from a corpus\. In either case, the lens shows which tokens receive high scores\. Interpreting those scores as evidence about intermediate computation treats token identity as a proxy for the readout structure through which the state is decoded\.

Figure[1](https://arxiv.org/html/2609.01936#S1.F1)shows why this proxy is ambiguous\. The logit lens scoresbugaboveinsecton a prompt about software, but every sense ofbugshares one row of the LM head, so the score alone cannot distinguish a software defect from an insect\. Conversely, near\-synonyms and translation equivalents occupy different rows that can share dominant directions, so distinct tokens can report the same underlying structure\. Token identity is too coarse to distinguish senses of the same token and too fine to identify structure shared across tokens\.

A second ambiguity arises for a fitted lens, whose output can reflect the fitting corpus as well as the state\. We find that lenses fitted on English and on Chinese report different languages for the same fixed state \(§[5](https://arxiv.org/html/2609.01936#S5)\)\. The inference from reported tokens to a working language is therefore underdetermined\.

We introduce Sparse Readout Prism \(SRP\), which changes what a lens reports, from token identities to readout features, while leaving how it transports the state unchanged\. The features reveal a new unit of analysis for lens readings, exposing readout structure that token identities can obscure\. Because they are built from the readout shared by every lens, they also support comparisons across tokens, contexts, layers, and lenses\.

SRP applies dictionary learning directly to the rows of the readout and writes each row as sparse coefficients over a shared overcomplete basis plus a residual, and a readout feature is one direction of that basis\. A single row can carry several features and a single feature can appear in many rows, so the features describe the organization of the readout itself\. A lens scores a token through one row of the readout and a margin between two tokens through the difference of their rows, so with each row written this way the score splits into one signed contribution per feature plus a residual \(§[3\.2](https://arxiv.org/html/2609.01936#S3.SS2)\)\.

SRP resolves thebugambiguity of Figure[1](https://arxiv.org/html/2609.01936#S1.F1)within one token\. In the software context the margin favoringbugrests on defect features, and in an insect context the same token draws its support from features for the animal sense instead \([Figure3](https://arxiv.org/html/2609.01936#S4.F3)\)\. The cross\-lens analysis takes up the converse case, where the same readout feature carries different reported tokens \(§[5](https://arxiv.org/html/2609.01936#S5)\)\.

The paper makes two contributions, one methodological and one empirical\.

1. 1\.SRP \(§[3](https://arxiv.org/html/2609.01936#S3)\) provides one basis per model, fit once from the readout weights and shared by every lens and layer\. It reconstructs 8\.9 to 17\.3 percentage points more of the tested logit differences than the strongest of six baselines built on row geometry, and ablating a feature shifts the score in proportion to its contribution \(§[4](https://arxiv.org/html/2609.01936#S4)\)\.
2. 2\.We show that a fitted lens’s report depends on the corpus it was fit on, a dependence we call corpus conditionality\. Changing a Jacobian lens’s fitting corpus changes the language it reports for identical hidden states, while the dominant readout feature remains stable across fitting languages, across two lens constructions, and when the compared languages share a script\. The basis therefore separates a change in the state from a change in the instrument \(§[5](https://arxiv.org/html/2609.01936#S5)\)\.

Together the results caution against treating token rankings as direct evidence about intermediate computation\. They also supply the control such inferences need, a reference fixed in the weights before any reading is taken\.

## 2Related Work

#### Methods in the lens style\.

The logit lens applies the unembedding unchanged\([nostalgebraist, 2020](https://arxiv.org/html/2609.01936#bib.bib53)\), while the tuned lens, Future Lens, Patchscopes, and the Jacobian lens learn or transport the map first\([Belrose et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib3);[Pal et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib55);[Ghandeharioun et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib24);[Gurnee et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib28)\)\. All of them report in token identity, and SRP changes that unit without changing the lens, so differently fitted lenses can be compared on the same state \(§[5](https://arxiv.org/html/2609.01936#S5)\)\.

#### Output geometry and sparse row dictionaries\.

Weight tying\([Press and Wolf, 2017](https://arxiv.org/html/2609.01936#bib.bib61);[Inan et al\., 2017](https://arxiv.org/html/2609.01936#bib.bib33);[Lopardo et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib44)\), softmax and target geometry\([Yang et al\., 2018a](https://arxiv.org/html/2609.01936#bib.bib77);[Zhao et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib80)\), and tokenizer segmentation\([Bostrom and Durrett, 2020](https://arxiv.org/html/2609.01936#bib.bib5)\)give the unembedding rows shared structure\. Sparse coding of static word embeddings recovers interpretable codes from rows at the type level\([Murphy et al\., 2012](https://arxiv.org/html/2609.01936#bib.bib51);[Faruqui et al\., 2015](https://arxiv.org/html/2609.01936#bib.bib19);[Subramanian et al\., 2018](https://arxiv.org/html/2609.01936#bib.bib71);[Arora et al\., 2018](https://arxiv.org/html/2609.01936#bib.bib2)\)\. SRP carries this approach to the rows of a trained transformer’s LM head, judged by replacement and by reconstruction of selected scores\. Row neighborhoods, clusterings, and principal directions summarize the same rows, but none of them is constrained to sum to the logit or margin under analysis\. SRP satisfies that constraint by construction, and §[4\.2](https://arxiv.org/html/2609.01936#S4.SS2)measures how much of a selected score each method recovers\.

#### Activation SAEs and decompositions in parameter space\.

To our knowledge, sparse autoencoders \(SAEs\) have not previously been fit to a trained transformer’s static weights, only to its activations\. The two fits decompose opposite factors of the same score, since a lens score takes the formh~ℓ⊤​q\\tilde\{h\}\_\{\\ell\}^\{\\top\}q, a decoded state against a readout directionqq\. Activation SAEs learn one overcomplete basis per layer for the hidden stateshℓh\_\{\\ell\}behind the decoded state\([Bricken et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib7);[Huben et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib31);[Gao et al\., 2025a](https://arxiv.org/html/2609.01936#bib.bib21);[Templeton et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib72)\), while SRP applies the same machinery toqqitself, fitting one basis that serves every lens and layer\. An activation SAE also depends on the corpus that produced its hidden states, so judging one lens reading against another through its basis would introduce a third instrument conditioned on a corpus\. The cross\-lens analysis of §[5](https://arxiv.org/html/2609.01936#S5)instead needs a reference that does not move with the lens, the prompt, or the corpus\. Decompositions in parameter space also factorize weights\([Braun et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib6);[Gao et al\., 2025b](https://arxiv.org/html/2609.01936#bib.bib22)\), whereas SRP ties its factorization to the scores a lens reads\.

Appendix[P](https://arxiv.org/html/2609.01936#A16)extends this section with lens variants, sparse feature infrastructure, readings of latent language, the geometry of the output space, and attribution to mechanisms\.

## 3Sparse Readout Prism

Whether the score is a token’s logit or a margin between two tokens, a lens reports it in token identity, and SRP resolves the same score into the readout features that push it up or down and by how much \([Figure2](https://arxiv.org/html/2609.01936#S3.F2)\)\. The features come from a sparse dictionary learned from the rows of the LM head \(§[3\.1](https://arxiv.org/html/2609.01936#S3.SS1)\)\. The contributions and an explicit residual sum to the score exactly \(§[3\.2](https://arxiv.org/html/2609.01936#S3.SS2)\), one fixed dictionary makes decompositions comparable across contexts, layers, and lenses \(§[3\.3](https://arxiv.org/html/2609.01936#S3.SS3)\), and fidelity diagnostics accompany every decomposition \(§[3\.4](https://arxiv.org/html/2609.01936#S3.SS4)\)\.

Throughout,hℓ∈ℝdmodelh\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}denotes the hidden state at layerℓ\\ellbefore a lens decodes it, anddmodeld\_\{\\mathrm\{model\}\}is the dimension of the model’s hidden states\. Each lens applies a mapTℓT\_\{\\ell\}that covers everything the lens does before the unembedding, namely the model’s fixed output normalization composed with whatever transport the lens adds, in the order the lens applies them\. The logit lens adds no transport, so its map is that normalization alone, the model’s final LayerNorm or RMSNorm, and every lens reduces to the same map at the final layerℓ=L\\ell=L\. A lens is fitted whenTℓT\_\{\\ell\}carries parameters estimated from data\. A tuned lens adds the affine translator it fits, and the Jacobian lens compared in §[5](https://arxiv.org/html/2609.01936#S5)adds an averaged Jacobian fitted on a corpus\. We writeh~ℓ=Tℓ​\(hℓ\)∈ℝdmodel\\tilde\{h\}\_\{\\ell\}=T\_\{\\ell\}\(h\_\{\\ell\}\)\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}for the decoded state presented to the LM head\. The unembedding matrixWU∈ℝ\|𝒱\|×dmodelW\_\{U\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\_\{\\mathrm\{model\}\}\}over the vocabulary𝒱\\mathcal\{V\}then produces the logitsWU​h~ℓW\_\{U\}\\tilde\{h\}\_\{\\ell\}\. Rowvv, written as the column vectorwv:=WU\[v,:\]⊤w\_\{v\}:=W\_\{U\}\[v,:\]^\{\\top\}, gives tokenvvthe linear score

logitv⁡\(h~ℓ\)=h~ℓ⊤​wv\.\\operatorname\{logit\}\_\{v\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}w\_\{v\}\.Gemma applies a nonlinear logit softcap after this linear score, and Appendix[F](https://arxiv.org/html/2609.01936#A6)handles the softcap separately\. SRP factorizesWUW\_\{U\}alone, and a lens changes only the decoded state the rows are scored against\.

![Refer to caption](https://arxiv.org/html/2609.01936v1/readout_query_one_column.png)Figure 2:Sparse Readout Prism, schematically\.Left\.For coefficientsα\\alphaover unembedding rows, SRP decomposes the selected readout directionqαq\_\{\\alpha\}into signed feature directionsβi​\(α\)​di\\beta\_\{i\}\(\\alpha\)\\,d\_\{i\}and a residualrαr\_\{\\alpha\}\.Right\.For decoded stateh~ℓ\\tilde\{h\}\_\{\\ell\}, the scoresα​\(h~ℓ\)s\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\)decomposes into one contribution per feature andh~ℓ⊤​rα\\tilde\{h\}\_\{\\ell\}^\{\\top\}r\_\{\\alpha\}, all in logit units\. Blue bars supportAA, orange bars supportBB, and hatching marks the residual \(§[3\.2](https://arxiv.org/html/2609.01936#S3.SS2)\)\.### 3\.1Factorizing the LM Head into Sparse Readout Features

SRP fits a sparse autoencoder to the rows ofWUW\_\{U\}\([Bricken et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib7);[Huben et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib31);[Gao et al\., 2025a](https://arxiv.org/html/2609.01936#bib.bib21)\)\. Each row then decomposes aswv=w^v\+rvw\_\{v\}=\\widehat\{w\}\_\{v\}\+r\_\{v\}withw^v=μ\+∑i=1Dzv,i​di\\widehat\{w\}\_\{v\}=\\mu\+\\textstyle\\sum\_\{i=1\}^\{D\}z\_\{v,i\}d\_\{i\}, wherew^v\\widehat\{w\}\_\{v\}is the fitted row,rvr\_\{v\}is the residual,μ\\muis a shared offset,DDis the dictionary width,di∈ℝdmodeld\_\{i\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}is a readout feature direction, andzv,iz\_\{v,i\}is the sparse coefficient of rowvvon featureii\. In SAE termsμ\\muis the decoder bias anddid\_\{i\}is a decoder direction\. We call this dictionary the readout SAE\. With the readout feature directionsdid\_\{i\}fixed, the projectionpi​\(h~ℓ\)=h~ℓ⊤​dip\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}d\_\{i\}measures how strongly the decoded state aligns with featureii\.

We fit TopK sparse autoencoders\([Makhzani and Frey, 2014](https://arxiv.org/html/2609.01936#bib.bib48);[Gao et al\., 2025a](https://arxiv.org/html/2609.01936#bib.bib21)\), whose encoder keeps thekklargest pre\-activations in each row, sokksets the sparsity budget directly\. SRP requires only a sparse factorization of the rows, and any sparse autoencoder variant could supply it\. Our reported operating points fixk=256k=256at 32×\\timesexpansion, and individual score decompositions are sparser still \(Appendix[D](https://arxiv.org/html/2609.01936#A4)\)\.

We center the rows and normalize their norms before fitting, the same preprocessing that TopK autoencoders apply to activations\. Both steps are inverted afterward, so every term lies in the raw row space ofwvw\_\{v\}, in original logit units\. Appendices[C](https://arxiv.org/html/2609.01936#A3)–[F](https://arxiv.org/html/2609.01936#A6)give width, sparsity, and preprocessing details\.

### 3\.2Decomposing Selected Readout Scores

For two tokensA,B∈𝒱A,B\\in\\mathcal\{V\}, the margin is the signed logit differencemA,B​\(h~ℓ\)=h~ℓ⊤​\(wA−wB\)m\_\{A,B\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}\(w\_\{A\}\-w\_\{B\}\), and token logits and margins are the simplest scores we evaluate\. We handle every score family through a coefficient vectorα∈ℝ\|𝒱\|\\alpha\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}over unembedding rows, with the selected readout directionqα=∑v∈𝒱αv​wvq\_\{\\alpha\}=\\textstyle\\sum\_\{v\\in\\mathcal\{V\}\}\\alpha\_\{v\}w\_\{v\}and the selected readout scoresα​\(h~ℓ\)=h~ℓ⊤​qαs\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}q\_\{\\alpha\}, shortened below to the selected direction and the selected score\. SettingαA=1\\alpha\_\{A\}=1and all other coefficients to zero recovers the token logitlogitA⁡\(h~ℓ\)=h~ℓ⊤​wA\\operatorname\{logit\}\_\{A\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}w\_\{A\}for a vocabulary tokenA∈𝒱A\\in\\mathcal\{V\}, and weightingbugby plus one andinsectby minus one gives the margin between those two tokens\.

A selected score whose coefficients sum to zero is a contrast, and any score component shared by the compared rows cancels, including components arising from output embedding geometry or tokenization\([Press and Wolf, 2017](https://arxiv.org/html/2609.01936#bib.bib61);[Inan et al\., 2017](https://arxiv.org/html/2609.01936#bib.bib33);[Yang et al\., 2018a](https://arxiv.org/html/2609.01936#bib.bib77);[Bostrom and Durrett, 2020](https://arxiv.org/html/2609.01936#bib.bib5)\)\. Every margin is a contrast, and the remaining families replace either side of a margin with a uniform average over a token set, or take the margin between the token the original LM head ranks first at the evaluated state and a competitor it ranks lower\.

Because each row decomposes into a fitted row plus a residual, every selected direction inherits that split\. For a selected directionqαq\_\{\\alpha\}, define the row residual

rα=∑v∈𝒱αv​rv\.r\_\{\\alpha\}=\\textstyle\\sum\_\{v\\in\\mathcal\{V\}\}\\alpha\_\{v\}r\_\{v\}\.Substituting the fitted reconstruction row by row and exchanging the sums gives

sα​\(h~ℓ\)\\displaystyle s\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\)=\(∑vαv\)​h~ℓ⊤​μ\\displaystyle=\\big\(\\textstyle\\sum\_\{v\}\\alpha\_\{v\}\\big\)\\tilde\{h\}\_\{\\ell\}^\{\\top\}\\mu\+∑i=1D\(∑vαvzv,i\)h~ℓ⊤di\\displaystyle\+\\textstyle\\sum\_\{i=1\}^\{D\}\\big\(\\textstyle\\sum\_\{v\}\\alpha\_\{v\}z\_\{v,i\}\\big\)\\tilde\{h\}\_\{\\ell\}^\{\\top\}d\_\{i\}\+h~ℓ⊤​rα\\displaystyle\+\\tilde\{h\}\_\{\\ell\}^\{\\top\}r\_\{\\alpha\}For readout featureii, defineβi​\(α\)=∑v∈𝒱αv​zv,i\\beta\_\{i\}\(\\alpha\)=\\textstyle\\sum\_\{v\\in\\mathcal\{V\}\}\\alpha\_\{v\}z\_\{v,i\}and the signed contributionci​\(h~ℓ,α\)=βi​\(α\)​pi​\(h~ℓ\)c\_\{i\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=\\beta\_\{i\}\(\\alpha\)\\,p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\), so the middle sum above carries one signed contribution per feature\.

The resulting sum of shared offset, signed contributions, and residual is what we call a local score decomposition, shortened below to a decomposition\. The offset term\(∑vαv\)​h~ℓ⊤​μ\\left\(\\sum\_\{v\}\\alpha\_\{v\}\\right\)\\tilde\{h\}\_\{\\ell\}^\{\\top\}\\muvanishes for every contrast, whose score then reduces to the signed contributions plus the residual\.

We writesrecons\_\{\\mathrm\{recon\}\}for the reconstructed score, which sums the shared offset and signed contributions above\. The exact score issexact=sα​\(h~ℓ\)s\_\{\\mathrm\{exact\}\}=s\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\), and the residual term isϵ=sexact−srecon=h~ℓ⊤​rα\\epsilon=s\_\{\\mathrm\{exact\}\}\-s\_\{\\mathrm\{recon\}\}=\\tilde\{h\}\_\{\\ell\}^\{\\top\}r\_\{\\alpha\}\.

Table[3](https://arxiv.org/html/2609.01936#A1.T3)\(Appendix[A](https://arxiv.org/html/2609.01936#A1)\) instantiatesqαq\_\{\\alpha\}andβi​\(α\)\\beta\_\{i\}\(\\alpha\)for each score family we use\. Before SRP is applied, each evaluation fixes the decoded state, the coefficient vector, the competitor set, and the tokenization filters that screen out brittle token cases \(Appendix[G\.7](https://arxiv.org/html/2609.01936#A7.SS7)\)\.

### 3\.3Comparing Decompositions Across Contexts and Lenses

Eachβi​\(α\)\\beta\_\{i\}\(\\alpha\)is fixed by the coefficient vectorα\\alphaand the readout SAE, while the projectionspi​\(h~ℓ\)p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)vary with the decoded state \(Appendix[L\.6](https://arxiv.org/html/2609.01936#A12.SS6)\)\. For a fixed selected direction, two decompositions therefore differ only through the decoded state, and a change of context, layer, or lens supplies nothing else\. Decompositions of different selected directions share the coordinate system and differ in their coefficients\. The dominant feature of a score is the feature whose signed contribution is largest in absolute value, and two decompositions agree at the feature level when they share a dominant feature\.

### 3\.4Reporting Fidelity Diagnostics

The residual absorbs whatever the fitted rows miss, so the decomposition is exact by construction, and we report three fidelity diagnostics with every decomposition to measure how well those rows stand in for the originals\. Every display shows the residual termϵ\\epsilonbeside the contributions \(Appendix[F\.1](https://arxiv.org/html/2609.01936#A6.SS1)\)\. We apply the same diagnostics to simpler alternatives to test whether SRP reconstructs scores better than row geometry alone does \(§[4\.2](https://arxiv.org/html/2609.01936#S4.SS2)\)\.

Replacement fidelity works at the model level and measures whether held\-out readout distributions and rankings are preserved whenWUW\_\{U\}is replaced by the reconstructed LM headWU^\\widehat\{W\_\{U\}\}, the stacked fitted rows \(§[4\.1](https://arxiv.org/html/2609.01936#S4.SS1)\)\. A dictionary can reconstruct rows accurately in aggregate yet reconstruct the selected logits poorly, so the remaining two diagnostics work at the level of individual scores\.

Relative reconstruction error weighs the residual against the size of the score it belongs to and is reported at two settings of the constantδ\\delta, where

ρδ​\(h~ℓ,α\)=\|sexact−srecon\|\|sexact\|\+δ\.\\rho\_\{\\delta\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=\\frac\{\|s\_\{\\mathrm\{exact\}\}\-s\_\{\\mathrm\{recon\}\}\|\}\{\|s\_\{\\mathrm\{exact\}\}\|\+\\delta\}\.Aggregate summaries useρ0\.5​\(h~ℓ,α\)\\rho\_\{0\.5\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\), whose constant floors the denominator, while case study displays and the baseline comparisons use the stricter unflooredρ0​\(h~ℓ,α\)=\|ϵ\|/\|sexact\|\\rho\_\{0\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=\|\\epsilon\|\\,/\\,\|s\_\{\\mathrm\{exact\}\}\|, the plain ratio of residual to score, and no table mixes the two settings \(Appendix[A](https://arxiv.org/html/2609.01936#A1)\)\.

Sign agreement asks whethersexacts\_\{\\mathrm\{exact\}\}andsrecons\_\{\\mathrm\{recon\}\}carry the same sign, since either ratio bounds the error’s magnitude but not its direction, and a flipped sign reverses which side of a contrast the reconstruction supports\. Coverage combines the two, as the fraction of scores whose reconstruction matches the sign ofsexacts\_\{\\mathrm\{exact\}\}and holdsρ0\\rho\_\{0\}below one half, and we call those scores covered\.

## 4The Decomposition Is Faithful, Predicts Intervention Effects, and Survives Retraining

#### Setup\.

The suite spans Qwen3\.5\([Qwen Team, 2026](https://arxiv.org/html/2609.01936#bib.bib63)\), Gemma\-4\([Google DeepMind, 2026a](https://arxiv.org/html/2609.01936#bib.bib25);[Google DeepMind, 2026b](https://arxiv.org/html/2609.01936#bib.bib26)\), Ministral\([Mistral AI, 2026](https://arxiv.org/html/2609.01936#bib.bib50)\), and Qwen/Llama readouts distilled for reasoning\([DeepSeek\-AI, 2025](https://arxiv.org/html/2609.01936#bib.bib12)\)at 0\.8B–9B scale\. We fit one readout SAE per model to the rows of its final LM head and evaluate it on the decoded states of 10,000 C4 continuations\([Raffel et al\., 2020](https://arxiv.org/html/2609.01936#bib.bib64);[Dodge et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib14)\), none of which enter the fit, and on the same banks of selected scores for every model,∼\\sim1,350 scores per model drawn from∼\\sim850 contrasts with their prompts fixed\. The rewordings and context variants of one contrast stay together when the bootstrap resamples\. A positive margin says the readout favorsAAoverBB, correctness is a separate question, and scores with small margins remain in the aggregates\.

All eight models enter the reconstruction and replacement analyses \(Appendix[E\.4](https://arxiv.org/html/2609.01936#A5.SS4)\)\. The baseline and intervention comparisons cover the six softcap\-free readouts, whose scores are a plain linear map of the decoded state \([Table1](https://arxiv.org/html/2609.01936#S4.T1)\)\. The two Gemma\-4 readouts sit behind a nonlinear logit softcap that SRP does not factorize, and the split was fixed on that ground before any contrast was scored\.

The retraining analysis of §[4\.4](https://arxiv.org/html/2609.01936#S4.SS4)trains several dictionaries per setting, so it is run on Qwen3\.5\-2B, the model the worked cases use throughout\. In those cases the score exceeds its reconstruction error, the reconstruction keeps its sign, and the contribution mass is compact enough to list, the regime Appendix[G\.6](https://arxiv.org/html/2609.01936#A7.SS6)describes in full\.

The labels on the bars of the worked cases summarize each feature’s top unembedding rows and stay separate from the measured feature ids and signed contributions\. Three independent blinded audit runs\([Karvonen et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib35);[Makelov et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib47)\)rate the large majority of the labels that support a claim as coherent \(Appendix[L\.8](https://arxiv.org/html/2609.01936#A12.SS8)\)\. Appendices[B](https://arxiv.org/html/2609.01936#A2)and[G\.7](https://arxiv.org/html/2609.01936#A7.SS7)describe the suite, bank families, provenance, and filters\.

Table 1:Readout replacement and reconstruction of selected scores against baselines built on row geometry\.Q/Min/R1\-Q/R1\-L==Qwen3\.5 / Ministral\-3 / R1\-Distill\-Qwen / R1\-Distill\-Llama\. top\-1 is the fraction of held\-out decoded states whose reconstructed and original readouts agree on the argmax\. The SRP and “best alternative” columns give coverage, the fraction of the full bank of logit differences reconstructed with the correct sign and error below half the score’s magnitude, with 95% CIs from a cluster bootstrap\. The best alternative is the strongest of six such baselines, and lead is the difference in percentage points\. Metric definitions and the full method grid are in Appendix[G\.2](https://arxiv.org/html/2609.01936#A7.SS2.SSS0.Px1), full model names in Appendix[B](https://arxiv.org/html/2609.01936#A2)\.Modeltop\-1SRPBest alternativeLeadQ\-0\.8B0\.8910\.754 \[\.733, \.773\]0\.665 \[\.645, \.686\]\+8\.9Q\-2B0\.8870\.780 \[\.759, \.798\]0\.625 \[\.606, \.644\]\+15\.5Q\-9B0\.9000\.812 \[\.792, \.831\]0\.639 \[\.618, \.658\]\+17\.3Min\-8B0\.9040\.777 \[\.759, \.795\]0\.648 \[\.629, \.665\]\+12\.9R1\-Q\-7B0\.7600\.716 \[\.694, \.738\]0\.579 \[\.560, \.598\]\+13\.7R1\-L\-8B0\.7540\.751 \[\.728, \.772\]0\.624 \[\.606, \.642\]\+12\.7
### 4\.1The Sparse Head Replaces the Readout

On the six softcap\-free readouts a readout SAE fitted to the rows of an LM head reproduces that head as a predictor over the vocabulary, row by row, and on the selected scores\. ReplacingWUW\_\{U\}with the reconstructed LM head preserves the held\-out argmax on 0\.89–0\.90 of decoded states for the Qwen3\.5 and Ministral readouts and on 0\.75–0\.76 for the two readouts distilled for reasoning \([Table1](https://arxiv.org/html/2609.01936#S4.T1)\)\. The median KL between the original and reconstructed readout distributions splits the same way, at 0\.09–0\.14 bits for the Qwen3\.5 and Ministral readouts and 0\.43–0\.49 bits for the two distilled for reasoning\. Row reconstruction alone does not predict replacement, since the two Gemma\-4 readouts behind the softcap nearly match the six on explained variance over the centered rows while replacing far worse\. Replacement metrics and row reconstruction for each readout are in[Table5](https://arxiv.org/html/2609.01936#A3.T5)and[Table14](https://arxiv.org/html/2609.01936#A5.T14)\.

At the level of individual scores, sign agreement is at least0\.910\.91across all eight readouts, and the residual error and the few sign flips both concentrate on the scores whose margin sits nearest a tie \([Table20](https://arxiv.org/html/2609.01936#A7.T20), Appendix Figure[13](https://arxiv.org/html/2609.01936#A7.F13), Appendix Table[18](https://arxiv.org/html/2609.01936#A7.T18)\)\. Contrasts built from benchmark formats hold up at least as well, and the bank of 300 such contrasts, the only bank whoseAAside comes from a gold label, reaches sign agreement 0\.94 and coverage 0\.84 on Qwen3\.5\-2B\([Rajpurkar et al\., 2018](https://arxiv.org/html/2609.01936#bib.bib66);[Yang et al\., 2018b](https://arxiv.org/html/2609.01936#bib.bib78);[Koreeda and Manning, 2021](https://arxiv.org/html/2609.01936#bib.bib36);[Guha et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib27);[Siddiq and Santos, 2022](https://arxiv.org/html/2609.01936#bib.bib69)\)\(Appendix[L\.4](https://arxiv.org/html/2609.01936#A12.SS4), Table[36](https://arxiv.org/html/2609.01936#A12.T36)\)\.

### 4\.2SRP Outperforms Every Baseline Built on Row Geometry

SRP is fitted to the rows ofWUW\_\{U\}, so the test that matters is whether its codes add anything to the geometry those rows already carry\. We compare against six baselines built directly on that geometry, from methods that use the rows with no dictionary at all to dictionaries fitted by clustering\. They are nearest row ridge and weighted kNN on the 128 closest rows, k\-means centroid dictionaries at two widths, hard assignment of each row to a single centroid, and PCA at 256 components, each scored for coverage on the same banks under the same harness \(§[3\.4](https://arxiv.org/html/2609.01936#S3.SS4), Appendix[G\.3](https://arxiv.org/html/2609.01936#A7.SS3)\)\. SRP outperforms all of them, reaching coverage 0\.72–0\.81 across the six softcap\-free readouts, and the intervals do not overlap \([Table1](https://arxiv.org/html/2609.01936#S4.T1)\)\. It also has the lowest mean, 95th\-percentile, and maximum absolute error of the seven methods, so the ranking does not depend on how the error is summarized \(Appendix[G\.4](https://arxiv.org/html/2609.01936#A7.SS4)\)\. To remove capacity as a confound, we build the k\-means dictionary at SRP’s own width and sparsity,D=65,536D\{=\}65\{,\}536withk=256k\{=\}256active centroids per row, and it trails SRP by 18 points of coverage on Qwen3\.5\-2B\. Shuffled codes and random support leave coverage below 0\.10 on the three Qwen3\.5 and both Gemma\-4 readouts \([Table19](https://arxiv.org/html/2609.01936#A7.T19)\)\.

Table 2:Predicted versus measured local changes on the readout side\.For each contrast, a feature’s predicted contributionci=βi​\(α\)​pi​\(h~ℓ\)c\_\{i\}=\\beta\_\{i\}\(\\alpha\)\\,p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)is compared with the measured margin change whenh~ℓ\\tilde\{h\}\_\{\\ell\}is ablated along the unit decoder directiondid\_\{i\}, with no new forward pass\. Per model,∼\\sim260 bank contrasts, each with its top 10 features plus 10 random directions as controls\. We reportr2r^\{2\}and the slope of the regression of measured change on predicted contribution through the origin, over covered pairs with bootstrap 95% CIs, and random is the controlr2r^\{2\}\.Modelr2r^\{2\}\[95% CI\]SlopeRandomnnQ\-0\.8B0\.886 \[\.869, \.903\]1\.6610\.0101610Q\-2B0\.834 \[\.802, \.862\]1\.3380\.0071820Q\-9B0\.829 \[\.813, \.845\]1\.6860\.0581860Min\-8B0\.926 \[\.913, \.937\]0\.9570\.0092050R1\-Q\-7B0\.918 \[\.899, \.933\]1\.0820\.0001500R1\-L\-8B0\.934 \[\.920, \.947\]1\.1260\.0341210
### 4\.3Removing a Feature Moves the Margin as the Decomposition Predicts

A decomposition is meant to say how much of the margin each feature accounts for, so removing that feature’s direction from the decoded state should shift the margin with the sign and roughly the size of its contribution\. We test this by removing the direction, measuring the margin with the original readout, and comparing the measured change with the predicted one\. The measured changes track the predicted ones atr2=0\.83r^\{2\}=0\.83–0\.930\.93on all six softcap\-free readouts, against at most 0\.06 when a random direction is removed instead, so the contributions say which features move the margin most and in which direction \([Table2](https://arxiv.org/html/2609.01936#S4.T2), Appendix[J](https://arxiv.org/html/2609.01936#A10)\)\. The size of the change matches as well on Ministral and the two readouts distilled for reasoning \(slope 0\.96–1\.13\), while on the three Qwen3\.5 readouts the margin moves further than predicted \(slope 1\.34–1\.69\)\.

### 4\.4The Decomposition Survives Retraining

We test whether a decomposition survives retraining, since a dictionary trained from a new seed can fit the readout equally well with different decoder directions\. At each of two widths,32×32\\timesand16×16\\times, we train three dictionaries for Qwen3\.5\-2B that differ only in their seed\. All six replace the readout equally well, at a held\-out top\-1 agreement of 0\.805–0\.848, so the comparisons below are between dictionaries of equal fidelity\. What we compare is the tokens a dictionary groups with each token of a contrast\. For a contrast betweenAAandBB, pooling the top unembedding rows of the features whose coefficients most favor each side’s row gives the tokens the dictionary groups withAAand withBB\. These lists depend on the two rows and the dictionary alone, and we compare them across dictionaries on all 63 distinct token pairs among the bank’s two\-token margins\.

At the32×32\\timeswidth, a feature that any of these lists draws on has a median cosine of∼\\sim0\.33 between its direction and the nearest direction in another seed’s dictionary, so the directions themselves do not recur across seeds, as is documented for TopK dictionaries\([Paulo and Belrose, 2026](https://arxiv.org/html/2609.01936#bib.bib59)\)\. Yet for the same token of the same contrast, the lists from two seeds share a mean Jaccard of 0\.21–0\.24 across the two widths, while the lists for two unrelated contrasts share 0\.014–0\.018\. The tokens a dictionary groups withAAtherefore reproduce across seeds at 13–15×\\timesthe overlap that unrelated contrasts show at the same width, and every one of the 63 contrasts lies above the 90th percentile of that unrelated overlap\.

Across both widths and all seed pairs, 87–91% of the features these lists draw on find a counterpart in the other seed’s dictionary whose top rows overlap theirs beyond what the same search yields for random rows of matched frequency, so the row groups behind the features recur as well\. Appendix[I](https://arxiv.org/html/2609.01936#A9)gives the full tables for both widths and shows that the variation grows when the preprocessing recipe changes along with the seed\.

Figure 3:One token, carried by different features in different contexts\.Signed readout feature terms in logit units forbugon Qwen3\.5\-2B, scored in\(A\)an insect context against the anchorerror, chosen for the software sense, and\(B\)a software context againstinsect\. Green terms supportbugand orange terms oppose it\. Bars carry each feature’s id and the rows of the LM head that score highest on it\.
### 4\.5Context Decides Which Features Carry a Token

So far we have shown that the decomposition stands in for the readout, predicts what removing a feature does to a margin, and survives retraining\. We now ask what it adds to the score a lens already reports\. Every sense ofbugshares one row of the LM head, so a lens score for the token cannot tell a software defect from an insect\. SRP writes that rowvvas fixed coefficientszv,iz\_\{v,i\}over the readout features, so from one context to the next only the projectionspi​\(h~ℓ\)p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)change \(§[3\.3](https://arxiv.org/html/2609.01936#S3.SS3)\)\. The context therefore decides which readout features carry the score, and those features tell the two senses apart\.

#### Worked cases\.

In[Figure3](https://arxiv.org/html/2609.01936#S4.F3)we scorebugon Qwen3\.5\-2B in an insect context and in a software context, each time against a token from the other sense so that the leading terms belong to the sense in play\. In the insect context, againsterror, the margin is carried by features whose top rows are mosquito, bee, and beetle, with relative reconstruction errorρ0=0\.185\\rho\_\{0\}=0\.185\. In the software context of[Figure1](https://arxiv.org/html/2609.01936#S1.F1), againstinsect, it is carried by defect, crash, and debug features \(ρ0=0\.003\\rho\_\{0\}=0\.003\), and no feature for the insect sense appears among the leading terms\. Appendix[L\.2](https://arxiv.org/html/2609.01936#A12.SS2)gives the displays and the terms for each case\.

#### Held\-out senses\.

The worked cases show the two senses of one word carried by different features, and we test whether the same holds for 20 ambiguous words on CoarseWSD\-20\([Loureiro et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib45)\), a benchmark with labelled senses and a fixed split of contexts into training and test\. For each sense of a word we pick the eight features whose mean contribution on the training contexts differs most between that sense and the others\. We then assign each test context to the sense whose eight features add up to the largest contribution\. Both steps use feature ids and signed contributions only, so the labels on the bars play no part\. Balanced accuracy is 0\.90, 0\.92, and 0\.80 on Qwen3\.5\-2B, Qwen3\.5\-9B, and R1\-Llama\-8B, where shuffling the sense labels gives a null near 0\.41\. Every word beats its majority baseline, and the result holds whether we keep one, two, four, or eight features per sense\. A nearest centroid probe on the full decoded state reaches 0\.96, 0\.97, and 0\.92, so the full state is the stronger predictor of the sense and eight features give a readable summary of it \(Appendix[H](https://arxiv.org/html/2609.01936#A8)\)\.

#### Attribution and edits\.

Because the readout factorizes independently of the hidden state, the same terms compose with direct logit attribution\([Elhage et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib18);[Wang et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib73);[Nguyen, 2024](https://arxiv.org/html/2609.01936#bib.bib52)\), so that each layer’s contribution to a margin resolves into one term per feature \(Appendix[M](https://arxiv.org/html/2609.01936#A13)\)\. The terms also supply candidate directions for constrained edits on the readout side\. At matched next\-token KL those directions beat the mean and leading principal components of the rows they were discovered from on Qwen3\.5\-2B, while the mean wins on Qwen3\.5\-0\.8B and Qwen3\.5\-9B \(Appendices[N](https://arxiv.org/html/2609.01936#A14)and[N\.1](https://arxiv.org/html/2609.01936#A14.SS1)\)\.

With the basis read on its own evidence, the next section applies it to a question token readings leave open\.

## 5A Fitted Lens Reports the Language of Its Fitting Corpus

A fitted lens maps an intermediate hidden state into the space consumed by the LM head\. The resulting tokens are often interpreted as evidence about the information represented at that layer, including the language in which the model is operating\. For example,[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.01936#bib.bib28)obtain English tokens from intermediate states for a Chinese prompt and interpret them as evidence that the intermediate computation occurs in English\.

Because this map is learned from a corpus, its token readings can reflect both the hidden state and the fitting corpus, and a token ranking alone cannot distinguish these sources\. We call this dependence corpus conditionality\. The fitting corpus affects the lens through the learned maph~ℓ=Tℓ​\(hℓ\)\\tilde\{h\}\_\{\\ell\}=T\_\{\\ell\}\(h\_\{\\ell\}\)\(§[3](https://arxiv.org/html/2609.01936#S3)\), so our results apply only to fitted lenses\. Studies of latent language that decode states directly through the unembedding use no fitting corpus\([Wendler et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib75);[Schut et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib67)\)and are therefore not subject to this dependence\. Appendix[P](https://arxiv.org/html/2609.01936#A16)discusses a separate confound in token readings that affects those studies\.

### 5\.1Design

We test corpus conditionality by holding an intermediate hidden statehℓh\_\{\\ell\}fixed and varying only the corpus used to fit the lens\. The two fitted maps can produce different decoded states from the same state, and hence different token readings, so any difference between the readings is attributable to the fitting corpus\. To test whether different tokens reflect different readout structure, SRP decomposes both selected token scores in a single basis learned only from the model weights\. Translation pairs, language labels, and fitting corpora do not enter this basis, which provides a reference independent of the fitting corpus\.

Using the reference implementation, we fit two Jacobian lenses\([Gurnee et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib28)\)on Qwen3\.5\-9B\. The first uses 100 English C4 prompts, and the second uses 100 Chinese C4 prompts\. We hold seeds, layers, and all other procedures fixed\. We apply both lenses to the same frozen states from 80 evaluation prompts spanning seven task families\. We also evaluate 12 controls—six digit prompts and six proper noun prompts—whose expected outputs are language invariant\.

We evaluate layers 21, 24, 26, and 29, where the two lenses can produce different token readings\. At each layer, we decompose the score assigned to the token reported by each lens and record the feature with the largest contribution in absolute value \(§[3\.3](https://arxiv.org/html/2609.01936#S3.SS3)\)\. We count a prompt as agreement when the same feature dominates under both lenses in at least two of the four layers\. Requiring two of the four layers keeps a match at one layer alone from counting as agreement\. We use two null comparisons to estimate chance agreement under this rule\. One evaluates an unrelated token under the same lens, and the other shuffles prompts before pairing the two lenses\. Appendix[K](https://arxiv.org/html/2609.01936#A11)provides the full protocol, token and feature agreement rates for each family, and results for each prompt\.

Figure 4:Two lenses, two surface languages, one readout feature\.The figure shows a Chinese factual recall prompt, which translates as “The capital of China is Beijing\. The capital of the UK is …”\. For the same frozen Qwen3\.5\-9B state at layer 26, the Jacobian lens fitted on English reportsLondon, whereas the lens fitted on Chinese reports 伦敦 \(“London”\)\. Below each token is its logit under the corresponding lens\. The bar shows the share of the lens’s feature sum attributable to the shared readout feature f23180, which dominates both scores\. The feature sum is the signed sum of the feature contributions and excludes the offset and the residual\.
### 5\.2Different Tokens Share a Dominant Feature

Across layers 21, 24, 26, and 29, the two lenses report tokens in different scripts on 39 of 80 prompts, while their outputs converge at the final layer\. On factual recall prompts, the lenses report different scripts on 9 of 10 \([Figure4](https://arxiv.org/html/2609.01936#S5.F4)\)\. On the antonym prompt from[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.01936#bib.bib28), the lens fitted on English reportslargeat layers 24 and 26 andbigat layer 29\. The lens fitted on Chinese instead reports

大的\(“large” or “the large one”\) at layers 24 and 26 and

大\(“large” or “big”\) at layer 29\. Interpreting either lens alone would therefore yield opposing conclusions about the model’s intermediate language\.

The same SRP feature dominates under both lenses on 77 of 80 prompts \(96%, 95% CI \[0\.90, 0\.99\]\)\. In[Figure4](https://arxiv.org/html/2609.01936#S5.F4), f23180 accounts for 75% of the feature sum for the lens fitted on English and 63% for the lens fitted on Chinese\. Every other feature contributes at most\+1\.7\+1\.7logits\. Neither null comparison produces a match, with 0/159 for unrelated tokens and 0/240 after shuffling prompts\. The dominant feature agrees under both lenses on 5/6 digit controls and 5/6 proper noun controls\.

Figure 5:Feature agreement across language pairs and lens constructions\.A prompt counts as agreement when the same feature dominates under both lenses in at least two of layers 21, 24, 26, and 29\. Points show agreement rates, intervals show 95% binomial CIs, and counts appear at the right\. Two baseline markers per setting show the matched null rates\. All eight are at most 3/159 \(1\.9%\)\. Appendix[K](https://arxiv.org/html/2609.01936#A11)provides tables for each family and construction details\.
### 5\.3The Pattern Persists Across Languages, Lens Constructions, and Corpus Sizes

We extend the corpus conditionality analysis across a second lens construction, a language pair that shares a script, and larger fitting corpora\. In each extension we test whether the token readings change while the same readout feature remains dominant\.[Figure5](https://arxiv.org/html/2609.01936#S5.F5)reports the agreement rates and matched null comparisons\.

#### Lens construction\.

We fit ridge translators related to the tuned lens\([Belrose et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib3)\)on the same English and Chinese corpora of 100 prompts\. On 9 of 10 EN→\\toZH translation prompts, the translator fitted on English reads Latin tokens and the translator fitted on Chinese reads CJK tokens\. The same feature remains dominant under both translators on 67/80 prompts \(CI \[0\.74, 0\.90\]\), compared with 3/159 and 1/240 matches under the two null comparisons\. We then hold the English fitting corpus fixed to isolate the effect of lens construction\. The Jacobian lens and ridge translator agree on the dominant feature for 76/80 prompts, and both null comparisons yield zero matches\.

#### Shared script\.

We next test English and German, which share the Latin script, and exclude cognates from the evaluation set\. The Jacobian lenses fitted on English and German report different top\-1 tokens on 73/90 prompts\. The same feature remains dominant on 84/90 prompts, compared with 1/204 and 0/270 matches under the two null comparisons\. The dissociation therefore does not depend on the distinct scripts used in the English–Chinese comparison\.

#### Corpus size\.

Finally, we triple each fitting corpus from 100 to 300 prompts to test whether the original corpus size accounts for the disagreement\. Feature agreement remains 77/80 for English–Chinese and increases from 84/90 to 85/90 for English–German\. Token disagreement moves in opposite directions, falling from 39/80 to 33/80 for English–Chinese and rising from 73/90 to 76/90 for English–German\. Estimation noise that decreases with additional fitting data would predict a decrease for both language pairs\. Across the two fitting scales, eleven English–Chinese prompts change their reported language, and none changes its dominant feature\.

On Qwen3\.5\-9B, the token readings vary with the language and size of the fitting corpus, while the dominant readout feature remains stable across these changes and across two lens constructions\.

## 6Future Work

Safety audits can use SRP wherever the question admits a concrete readout score\. A study can define a group contrast for refusal, source verification, or a specified lexical family, compare its feature contributions across paraphrases, languages, adversarial prompts, and checkpoints, and treat the features that recur as nominations for causal tests or constrained interventions\. The evaluation should report reconstruction error, transfer, distributional cost, and effects on other outputs, so the safety claim stays tied to the selected score\. Appendix[O\.5](https://arxiv.org/html/2609.01936#A15.SS5)develops the methodological extensions, among them tracing feature contributions back to the components that supply them, joint activation and readout factorizations, cross\-model dictionary alignment, and decompositions that cover more of the output than one selected score\.

## 7Conclusion

Sparse Readout Prism offers an alternative unit for interpreting lens readings\. Token identity is too coarse where one token carries several readout features and too fine where several tokens share one\. SRP covers 8\.9 to 17\.3 percentage points more of the tested logit differences than the strongest of six baselines built on row geometry, and the tokens it groups with a contrast reproduce across seeds at 13 to 15 times the overlap of unrelated contrasts\. It shows what token rankings cannot, since lenses fitted on different corpora report different surface languages for the same hidden state while the same readout feature stays dominant under both\. Lens studies therefore gain a reference fixed in the weights, one that tells a change in the state apart from a change in the instrument\.

These features describe the readout itself, complementing sparse dictionary methods that decompose hidden states\. One row carries several senses and one feature spans surface forms and scripts, so the organization of a model’s output space becomes measurable from its weights alone\. Projecting a decoded state onto the features then shows how the state aligns with that organization layer by layer, before any token is read off it\. One dictionary per model serves every lens and layer, so a reading from any of them decomposes into the same terms with no further fitting, and those terms give auditing work a control for what the fitting corpus contributed to a reading, along with terms for layer\-by\-layer attribution and candidate directions for constrained edits on the readout side\.

## Limitations

#### Fitting and basis choice\.

SRP adds a one\-time sparse autoencoder fit for each model, taking roughly 6–12 GPU\-hours on one NVIDIA A40 in our experiments\. As in other sparse dictionary methods, individual decoder directions depend on dictionary size, sparsitykk, training recipe, and seed\([Elhage et al\., 2022](https://arxiv.org/html/2609.01936#bib.bib17);[Karvonen et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib35);[Makelov et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib47);[Paulo and Belrose, 2026](https://arxiv.org/html/2609.01936#bib.bib59)\)\. We therefore select operating points from fidelity sweeps and report stability across seeds and recipes \(§[4\.4](https://arxiv.org/html/2609.01936#S4.SS4), Appendix[I](https://arxiv.org/html/2609.01936#A9)\)\.

#### Empirical scope\.

The cross\-lens study evaluates Qwen3\.5\-9B across English–Chinese and English–German, two constructions of fitted lenses, and multiple fitting budgets\. It establishes in these controlled settings that the surface language a lens reports can depend on the corpus it was fit on\. The study of edits to the readout is run on Qwen3\.5\-2B, with cross\-model losses reported in Appendix[N](https://arxiv.org/html/2609.01936#A14)\.

#### Interpretation boundary\.

SRP is designed to explain selected readout scores and the organization of a model’s readout\. Questions at the level of circuits or of generation require interventions beyond the analyses on the readout side here\. Feature labels summarize top unembedding rows and may inherit tokenizer and vocabulary geometry\([Press and Wolf, 2017](https://arxiv.org/html/2609.01936#bib.bib61);[Inan et al\., 2017](https://arxiv.org/html/2609.01936#bib.bib33);[Yang et al\., 2018a](https://arxiv.org/html/2609.01936#bib.bib77);[Bostrom and Durrett, 2020](https://arxiv.org/html/2609.01936#bib.bib5)\)\. Quantitative conclusions therefore rest on feature ids, signed contributions, and residual errors, with further discussion in Appendix[O](https://arxiv.org/html/2609.01936#A15)\.

## Ethical Considerations

This work studies the readouts of pretrained language models, collects no new data from human subjects, and evaluates no deployed systems\. Experiments use public checkpoints and held\-out decoded states, including C4 continuations, which may contain offensive, private, copyrighted, or sensitive text\([Raffel et al\., 2020](https://arxiv.org/html/2609.01936#bib.bib64);[Dodge et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib14)\)\. We use C4 only to measure readout reconstruction and make no claims about the social validity of individual tokens, labels, or generated outputs\. Checkpoints and decoded states derived from C4 are used under the licenses or terms published with their release pages and model cards\. We do not redistribute the checkpoints or raw C4 text\. The released SRP dictionaries \([hematteo/sparse\-readout\-prism](https://huggingface.co/hematteo/sparse-readout-prism)\) carry license and usage notes specific to each artifact\.

SRP makes lexical readout directions and row labels easier to inspect, and its stress test of edits to the readout moves specified lexical scores, which is a dual\-use property of auditing tools\. We report the edit with probes for distribution shift and for effects off target, and keep claims tied to measured quantities\. Evaluation of safety filters, moderation, and deployment controls raises questions at the system level that lie beyond this work\.

## References

- Ameisen et al\. \(2025\)Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L\. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others\. 2025\.[Circuit tracing: Revealing computational graphs in language models](https://transformer-circuits.pub/2025/attribution-graphs/methods.html)\.*Transformer Circuits Thread*\.
- Arora et al\. \(2018\)Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski\. 2018\.[Linear algebraic structure of word senses, with applications to polysemy](https://doi.org/10.1162/tacl_a_00034)\.*Transactions of the Association for Computational Linguistics*, 6:483–495\.
- Belrose et al\. \(2023\)Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt\. 2023\.[Eliciting latent predictions from transformers with the tuned lens](https://arxiv.org/abs/2303.08112)\.*Preprint*, arXiv:2303\.08112\.
- Biderman et al\. \(2023\)Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\. 2023\.[Pythia: A suite for analyzing large language models across training and scaling](https://arxiv.org/abs/2304.01373)\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 2397–2430\. PMLR\.
- Bostrom and Durrett \(2020\)Kaj Bostrom and Greg Durrett\. 2020\.[Byte pair encoding is suboptimal for language model pretraining](https://doi.org/10.18653/v1/2020.findings-emnlp.414)\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 4617–4624\.
- Braun et al\. \(2025\)Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey\. 2025\.[Interpretability in parameter space: Minimizing mechanistic description length with attribution\-based parameter decomposition](https://arxiv.org/abs/2501.14926)\.*Preprint*, arXiv:2501\.14926\.
- Bricken et al\. \(2023\)Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L\. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield\-Dodds, Alex Tamkin, Karina Nguyen, and 6 others\. 2023\.[Towards monosemanticity: Decomposing language models with dictionary learning](https://transformer-circuits.pub/2023/monosemantic-features/index.html)\.*Transformer Circuits Thread*\.
- Bussmann et al\. \(2024\)Bart Bussmann, Patrick Leask, and Neel Nanda\. 2024\.[BatchTopK sparse autoencoders](https://arxiv.org/abs/2412.06410)\.*Preprint*, arXiv:2412\.06410\.
- Bussmann et al\. \(2025\)Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda\. 2025\.[Learning multi\-level features with Matryoshka sparse autoencoders](https://arxiv.org/abs/2503.17547)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 6077–6101\. PMLR\.
- Chung and Kim \(2025\)Woojin Chung and Jeonghoon Kim\. 2025\.[Exploiting vocabulary frequency imbalance in language model pre\-training](https://openreview.net/forum?id=TkHcdBLsJJ)\.In*Advances in Neural Information Processing Systems*\.
- Conmy et al\. \(2023\)Arthur Conmy, Augustine N\. Mavor\-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga\-Alonso\. 2023\.[Towards automated circuit discovery for mechanistic interpretability](https://arxiv.org/abs/2304.14997)\.In*Advances in Neural Information Processing Systems*\.
- DeepSeek\-AI \(2025\)DeepSeek\-AI\. 2025\.[DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning](https://doi.org/10.1038/s41586-025-09422-z)\.*Nature*, 645\(8081\):633–638\.
- Deng et al\. \(2026\)Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, Tianhao Li, Qian Cao, Xuancheng Ren, Xiaodong Deng, An Yang, Fei Huang, Dayiheng Liu, and Jingren Zhou\. 2026\.[Qwen\-Scope: Turning sparse features into development tools for large language models](https://arxiv.org/abs/2605.11887)\.*Preprint*, arXiv:2605\.11887\.
- Dodge et al\. \(2021\)Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner\. 2021\.[Documenting large webtext corpora: A case study on the colossal clean crawled corpus](https://doi.org/10.18653/v1/2021.emnlp-main.98)\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 1286–1305\.
- Dumas et al\. \(2025\)Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West\. 2025\.[Separating tongue from thought: Activation patching reveals language\-agnostic concept representations in transformers](https://doi.org/10.18653/v1/2025.acl-long.1536)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 31822–31841\.
- Dunefsky et al\. \(2024\)Jacob Dunefsky, Philippe Chlenski, and Neel Nanda\. 2024\.[Transcoders find interpretable LLM feature circuits](https://doi.org/10.52202/079017-0768)\.In*Advances in Neural Information Processing Systems*, volume 37\.
- Elhage et al\. \(2022\)Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield\-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah\. 2022\.[Toy models of superposition](https://transformer-circuits.pub/2022/toy_model/index.html)\.*Transformer Circuits Thread*\.
- Elhage et al\. \(2021\)Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield\-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others\. 2021\.[A mathematical framework for transformer circuits](https://transformer-circuits.pub/2021/framework/index.html)\.*Transformer Circuits Thread*\.
- Faruqui et al\. \(2015\)Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A\. Smith\. 2015\.[Sparse overcomplete word vector representations](https://doi.org/10.3115/v1/P15-1144)\.In*Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 1491–1500, Beijing, China\. Association for Computational Linguistics\.
- Gao et al\. \(2019\)Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie\-Yan Liu\. 2019\.[Representation degeneration problem in training natural language generation models](https://openreview.net/forum?id=SkEYojRqtm)\.In*International Conference on Learning Representations*\.
- Gao et al\. \(2025a\)Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu\. 2025a\.[Scaling and evaluating sparse autoencoders](https://arxiv.org/abs/2406.04093)\.In*International Conference on Learning Representations*\.
- Gao et al\. \(2025b\)Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V\. Govande, Bowen Baker, and Dan Mossing\. 2025b\.[Weight\-sparse transformers have interpretable circuits](https://arxiv.org/abs/2511.13653)\.*Preprint*, arXiv:2511\.13653\.
- Geiger et al\. \(2025\)Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard\. 2025\.[Causal abstraction: A theoretical foundation for mechanistic interpretability](https://arxiv.org/abs/2301.04709)\.*Journal of Machine Learning Research*, 26\(83\):1–64\.
- Ghandeharioun et al\. \(2024\)Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva\. 2024\.[Patchscopes: A unifying framework for inspecting hidden representations of language models](https://arxiv.org/abs/2401.06102)\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 15466–15490\. PMLR\.
- Google DeepMind \(2026a\)Google DeepMind\. 2026a\.Gemma 4\.[https://deepmind\.google/models/gemma/gemma\-4/](https://deepmind.google/models/gemma/gemma-4/)\.Official model\-family page; accessed 2026\-05\-20\.
- Google DeepMind \(2026b\)Google DeepMind\. 2026b\.Gemma 4 model collection\.[https://huggingface\.co/collections/google/gemma\-4](https://huggingface.co/collections/google/gemma-4)\.Hugging Face model collection; accessed 2026\-05\-20\.
- Guha et al\. \(2023\)Neel Guha, Julian Nyarko, Daniel E\. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas\-Wood, Austin Peters, Brandon Waldon, Daniel N\. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M\. Dickinson, Haggai Porat, Jason Hegland, and 21 others\. 2023\.[LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models](https://arxiv.org/abs/2308.11462)\.In*Advances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track*, volume 36\.
- Gurnee et al\. \(2026\)Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, T\. Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey\. 2026\.[Verbalizable representations form a global workspace in language models](https://transformer-circuits.pub/2026/workspace/)\.*Transformer Circuits Thread*\.
- Han et al\. \(2024\)Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji\. 2024\.[Word embeddings are steers for language models](https://doi.org/10.18653/v1/2024.acl-long.864)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 16410–16430\.
- Hoerl and Kennard \(1970\)Arthur E\. Hoerl and Robert W\. Kennard\. 1970\.[Ridge regression: Biased estimation for nonorthogonal problems](https://doi.org/10.1080/00401706.1970.10488634)\.*Technometrics*, 12\(1\):55–67\.
- Huben et al\. \(2024\)Robert Huben, Hoagy Cunningham, Logan Smith, Aidan Ewart, and Lee Sharkey\. 2024\.[Sparse autoencoders find highly interpretable features in language models](https://arxiv.org/abs/2309.08600)\.In*International Conference on Learning Representations*\.
- Iacob et al\. \(2025\)Alex Iacob, Lorenzo Sani, Meghdad Kurmanji, William F\. Shen, Xinchi Qiu, Dongqi Cai, Yan Gao, and Nicholas D\. Lane\. 2025\.[DEPT: Decoupled embeddings for pre\-training language models](https://arxiv.org/abs/2410.05021)\.In*The Thirteenth International Conference on Learning Representations*\.
- Inan et al\. \(2017\)Hakan Inan, Khashayar Khosravi, and Richard Socher\. 2017\.[Tying word vectors and word classifiers: A loss framework for language modeling](https://arxiv.org/abs/1611.01462)\.In*International Conference on Learning Representations*\.
- Jolliffe \(2002\)Ian T\. Jolliffe\. 2002\.[*Principal Component Analysis*](https://doi.org/10.1007/b98835), 2 edition\.Springer\.
- Karvonen et al\. \(2025\)Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu\-Tong Lau, Eoin Farrell, Callum Stuart Mcdougall, Kola Ayonrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda\. 2025\.[SAEBench: A comprehensive benchmark for sparse autoencoders in language model interpretability](https://arxiv.org/abs/2503.09532)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 29223–29264\. PMLR\.
- Koreeda and Manning \(2021\)Yuta Koreeda and Christopher D\. Manning\. 2021\.[ContractNLI: A dataset for document\-level natural language inference for contracts](https://doi.org/10.18653/v1/2021.findings-emnlp.164)\.In*Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 1907–1919\.
- Krojer et al\. \(2026\)Benno Krojer, Shravan Nayak, Oscar Mañas, Vaibhav Adlakha, Desmond Elliott, Siva Reddy, and Marius Mosbach\. 2026\.[LatentLens: Revealing highly interpretable visual tokens in LLMs](https://arxiv.org/abs/2602.00462)\.*Preprint*, arXiv:2602\.00462\.
- Lawson and Hanson \(1974\)Charles L\. Lawson and Richard J\. Hanson\. 1974\.*Solving Least Squares Problems*\.Prentice\-Hall, Englewood Cliffs, NJ\.
- Leask et al\. \(2025\)Patrick Leask, Bart Bussmann, Michael T\. Pearce, Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda\. 2025\.[Sparse autoencoders do not find canonical units of analysis](https://arxiv.org/abs/2502.04878)\.In*International Conference on Learning Representations*\.
- Lieberum et al\. \(2024\)Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda\. 2024\.[Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2](https://doi.org/10.18653/v1/2024.blackboxnlp-1.19)\.In*Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP*, pages 278–300\. Association for Computational Linguistics\.
- Lindsey et al\. \(2024\)Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah\. 2024\.[Sparse crosscoders for cross\-layer features and model diffing](https://transformer-circuits.pub/2024/crosscoders/index.html)\.*Transformer Circuits Thread*\.
- Liu and Deng \(2026\)Yiting Liu and Zhi\-Hong Deng\. 2026\.[Beyond activation patterns: A weight\-based out\-of\-context explanation of sparse autoencoder features](https://arxiv.org/abs/2601.22447)\.*Preprint*, arXiv:2601\.22447\.
- Llama Team \(2024\)Llama Team\. 2024\.[The Llama 3 herd of models](https://arxiv.org/abs/2407.21783)\.*Preprint*, arXiv:2407\.21783\.
- Lopardo et al\. \(2026\)Antonio Lopardo, Avyukth Harish, Catherine Arnett, and Akshat Gupta\. 2026\.[Weight tying biases token embeddings towards the output space](https://doi.org/10.18653/v1/2026.findings-acl.2027)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 40795–40810\. Association for Computational Linguistics\.
- Loureiro et al\. \(2021\)Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho\-Collados\. 2021\.[Analysis and evaluation of language models for word sense disambiguation](https://doi.org/10.1162/coli_a_00405)\.*Computational Linguistics*, 47\(2\):387–443\.
- Machina and Mercer \(2024\)Anemily Machina and Robert Mercer\. 2024\.[Anisotropy is not inherent to transformers](https://doi.org/10.18653/v1/2024.naacl-long.274)\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 4892–4907\.
- Makelov et al\. \(2025\)Aleksandar Makelov, Georg Lange, and Neel Nanda\. 2025\.[Towards principled evaluations of sparse autoencoders for interpretability and control](https://arxiv.org/abs/2405.08366)\.In*International Conference on Learning Representations \(ICLR\)*\.
- Makhzani and Frey \(2014\)Alireza Makhzani and Brendan Frey\. 2014\.[k\-sparse autoencoders](https://arxiv.org/abs/1312.5663)\.In*International Conference on Learning Representations \(ICLR\)*\.
- Minder et al\. \(2025\)Julian Minder, Clément Dumas, Caden Juang, Bilal Chughtai, and Neel Nanda\. 2025\.[Overcoming sparsity artifacts in crosscoders to interpret chat\-tuning](https://arxiv.org/abs/2504.02922)\.In*Advances in Neural Information Processing Systems*\.
- Mistral AI \(2026\)Mistral AI\. 2026\.[Ministral 3](https://arxiv.org/abs/2601.08584)\.*Preprint*, arXiv:2601\.08584\.
- Murphy et al\. \(2012\)Brian Murphy, Partha Talukdar, and Tom Mitchell\. 2012\.[Learning effective and interpretable semantic models using non\-negative sparse embedding](https://aclanthology.org/C12-1118/)\.In*Proceedings of COLING 2012*, pages 1933–1950, Mumbai, India\. The COLING 2012 Organizing Committee\.
- Nguyen \(2024\)Thong T\. Nguyen\. 2024\.Logit prisms: Decomposing transformer outputs for mechanistic interpretability\.[https://neuralblog\.github\.io/logit\-prisms/](https://neuralblog.github.io/logit-prisms/)\.
- nostalgebraist \(2020\)nostalgebraist\. 2020\.Interpreting GPT: The logit lens\.[https://www\.lesswrong\.com/posts/AcKRB8wDpdaN6v6ru/interpreting\-gpt\-the\-logit\-lens](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)\.
- O’Neill et al\. \(2024\)Charles O’Neill, Christine Ye, Kartheik Iyer, and John F\. Wu\. 2024\.[Disentangling dense embeddings with sparse autoencoders](https://arxiv.org/abs/2408.00657)\.*Preprint*, arXiv:2408\.00657\.
- Pal et al\. \(2023\)Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C\. Wallace, and David Bau\. 2023\.[Future lens: Anticipating subsequent tokens from a single hidden state](https://doi.org/10.18653/v1/2023.conll-1.37)\.In*Proceedings of the 27th Conference on Computational Natural Language Learning \(CoNLL\)*, pages 548–560\.
- Panchal et al\. \(2026\)Mihir Panchal, Deeksha Varshney, Mamta, and Asif Ekbal\. 2026\.[Indic\-TunedLens: Interpreting multilingual models in Indian languages](https://doi.org/10.18653/v1/2026.vardial-1.14)\.In*Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects*, pages 172–185, Rabat, Morocco\. Association for Computational Linguistics\.
- Park et al\. \(2024\)Kiho Park, Yo Joong Choe, and Victor Veitch\. 2024\.[The linear representation hypothesis and the geometry of large language models](https://arxiv.org/abs/2311.03658)\.In*Proceedings of the 41st International Conference on Machine Learning*\.
- Pati et al\. \(1993\)Yagyensh Chandra Pati, Ramin Rezaiifar, and P\. S\. Krishnaprasad\. 1993\.[Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition](https://doi.org/10.1109/ACSSC.1993.342465)\.In*Proceedings of the Twenty\-Seventh Asilomar Conference on Signals, Systems and Computers*, volume 1, pages 40–44\.
- Paulo and Belrose \(2026\)Gonçalo Paulo and Nora Belrose\. 2026\.[Sparse autoencoders trained on the same data learn different features](https://arxiv.org/abs/2501.16615)\.In*International Conference on Learning Representations \(ICLR\)*\.
- Phukan et al\. \(2025\)Anirudh Phukan, Divyansh, Harshit Kumar Morj, Vaishnavi, Apoorv Saxena, and Koustava Goswami\. 2025\.[Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in VLMs](https://doi.org/10.18653/v1/2025.naacl-long.488)\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 9661–9675, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Press and Wolf \(2017\)Ofir Press and Lior Wolf\. 2017\.[Using the output embedding to improve language models](https://doi.org/10.18653/v1/E17-2025)\.In*Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, pages 157–163\.
- Qwen Team \(2025\)Qwen Team\. 2025\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*Preprint*, arXiv:2505\.09388\.
- Qwen Team \(2026\)Qwen Team\. 2026\.Qwen3\.5 model collection\.[https://huggingface\.co/collections/Qwen/qwen35](https://huggingface.co/collections/Qwen/qwen35)\.Hugging Face model collection; accessed 2026\-05\-20\.
- Raffel et al\. \(2020\)Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J\. Liu\. 2020\.[Exploring the limits of transfer learning with a unified text\-to\-text transformer](https://jmlr.org/papers/v21/20-074.html)\.*Journal of Machine Learning Research*, 21\(140\):1–67\.
- Rajamanoharan et al\. \(2024\)Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda\. 2024\.[Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders](https://arxiv.org/abs/2407.14435)\.*Preprint*, arXiv:2407\.14435\.
- Rajpurkar et al\. \(2018\)Pranav Rajpurkar, Robin Jia, and Percy Liang\. 2018\.[Know what you don’t know: Unanswerable questions for SQuAD](https://doi.org/10.18653/v1/P18-2124)\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 784–789\.
- Schut et al\. \(2025\)Lisa Schut, Yarin Gal, and Sebastian Farquhar\. 2025\.[Do multilingual LLMs think in English?](https://arxiv.org/abs/2502.15603)*Preprint*, arXiv:2502\.15603\.
- Shen et al\. \(2025\)William F\. Shen, Xinchi Qiu, Meghdad Kurmanji, Alex Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, and Nicholas D\. Lane\. 2025\.[LLM unlearning via neural activation redirection](https://arxiv.org/abs/2502.07218)\.In*Advances in Neural Information Processing Systems*\.
- Siddiq and Santos \(2022\)Mohammed Latif Siddiq and Joanna C\. S\. Santos\. 2022\.[SecurityEval dataset: Mining vulnerability examples to evaluate machine learning\-based code generation techniques](https://doi.org/10.1145/3549035.3561184)\.In*Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security*, pages 29–33\.
- Stollenwerk et al\. \(2026\)Felix Stollenwerk, Anna Lokrantz, and Niclas Hertzberg\. 2026\.[Output embedding centering for stable LLM pretraining](https://arxiv.org/abs/2601.02031)\.*Preprint*, arXiv:2601\.02031\.
- Subramanian et al\. \(2018\)Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg\-Kirkpatrick, and Eduard Hovy\. 2018\.[SPINE: SParse Interpretable Neural Embeddings](https://doi.org/10.1609/aaai.v32i1.11935)\.In*Proceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence*, pages 4921–4928, New Orleans, Louisiana, USA\. AAAI Press\.
- Templeton et al\. \(2024\)Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L\. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C\. Daniel Freeman, and 7 others\. 2024\.[Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)\.*Transformer Circuits Thread*\.
- Wang et al\. \(2023\)Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt\. 2023\.[Interpretability in the wild: A circuit for indirect object identification in GPT\-2 small](https://arxiv.org/abs/2211.00593)\.In*International Conference on Learning Representations \(ICLR\)*\.
- Wang \(2025\)Zhenyu Wang\. 2025\.[LogitLens4LLMs: Extending logit lens analysis to modern large language models](https://arxiv.org/abs/2503.11667)\.*Preprint*, arXiv:2503\.11667\.
- Wendler et al\. \(2024\)Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West\. 2024\.[Do llamas work in English? On the latent language of multilingual transformers](https://doi.org/10.18653/v1/2024.acl-long.820)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15366–15394\.
- Yang et al\. \(2024\)An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang\. 2024\.[Qwen2\.5\-Math technical report: Toward mathematical expert model via self\-improvement](https://arxiv.org/abs/2409.12122)\.*Preprint*, arXiv:2409\.12122\.
- Yang et al\. \(2018a\)Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W\. Cohen\. 2018a\.[Breaking the softmax bottleneck: A high\-rank RNN language model](https://arxiv.org/abs/1711.03953)\.In*International Conference on Learning Representations*\.
- Yang et al\. \(2018b\)Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W\. Cohen, Ruslan Salakhutdinov, and Christopher D\. Manning\. 2018b\.[HotpotQA: A dataset for diverse, explainable multi\-hop question answering](https://doi.org/10.18653/v1/D18-1259)\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 2369–2380\.
- Zhang et al\. \(2020\)Zhong Zhang, Chongming Gao, Cong Xu, Rui Miao, Qinli Yang, and Junming Shao\. 2020\.[Revisiting representation degeneration problem in language modeling](https://doi.org/10.18653/v1/2020.findings-emnlp.46)\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 518–527\.
- Zhao et al\. \(2024\)Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis\. 2024\.[Implicit geometry of next\-token prediction: From language sparsity patterns to model representations](https://openreview.net/forum?id=qyilOnIRHI)\.In*First Conference on Language Modeling*\.

## Appendix Roadmap

ContentsA[SRP Score Decomposition Identities](https://arxiv.org/html/2609.01936#A1)\(p\.[A](https://arxiv.org/html/2609.01936#A1)\)\.
Readout SAE reconstruction identities, selected readout directions, reconstruction error, sign agreement, and fidelity diagnostics\.B[Model Suite and Selection Rationale](https://arxiv.org/html/2609.01936#A2)\(p\.[B](https://arxiv.org/html/2609.01936#A2)\)\.
Model\-suite rationale, evaluated model families, targeted comparison rows, model roles, and interpretation details\.C[How the Readout SAEs Were Selected](https://arxiv.org/html/2609.01936#A3)\(p\.[C](https://arxiv.org/html/2609.01936#A3)\)\.
Selection criteria, metric glossary, operating regimes, final dictionary settings, and sweep evidence map\.D[Qwen Readout SAE Recipe and Capacity Sweeps](https://arxiv.org/html/2609.01936#A4)\(p\.[D](https://arxiv.org/html/2609.01936#A4)\)\.
Qwen3\.5\-9B recipe selection, capacity sweeps, and native operating\-point claims\.E[Readout SAE Transfer and Model\-Family Comparisons](https://arxiv.org/html/2609.01936#A5)\(p\.[E](https://arxiv.org/html/2609.01936#A5)\)\.
Smaller Qwen/Gemma transfer, cross\-family rows, R1\-distilled comparisons, and row\-norm tail analysis\.F[Readout SAE Diagnostics and Reproducibility](https://arxiv.org/html/2609.01936#A6)\(p\.[F](https://arxiv.org/html/2609.01936#A6)\)\.
Support\-selection diagnostics, rejected alternatives, denominator controls, and training/evaluation recipe details\.G[Fidelity Checks and Controls](https://arxiv.org/html/2609.01936#A7)\(p\.[G](https://arxiv.org/html/2609.01936#A7)\)\.
Fidelity results with bootstrap intervals, C4 replacement, baselines and nulls, nearest row reading, token\-level audits, and operating regimes\.H[Sense\-Labelled Evaluation](https://arxiv.org/html/2609.01936#A8)\(p\.[H](https://arxiv.org/html/2609.01936#A8)\)\.
Alignment against a permutation null that repeats the selection, sparsity references, and scope\.I[Stability Across Independent Training Runs](https://arxiv.org/html/2609.01936#A9)\(p\.[I](https://arxiv.org/html/2609.01936#A9)\)\.
Cross\-seed explanation stability, feature\-group matching, held\-out recovery of the stable core, and the disclosure across recipes\.J[Predicted Versus Realized Local Readout\-Side Changes](https://arxiv.org/html/2609.01936#A10)\(p\.[J](https://arxiv.org/html/2609.01936#A10)\)\.
Intervention\-validation protocol, falsifiers, six\-model results, and slope diagnosis\.K[Cross\-Lens Study Protocol, Controls, and Per\-Family Results](https://arxiv.org/html/2609.01936#A11)\(p\.[K](https://arxiv.org/html/2609.01936#A11)\)\.
Corpus conditionality protocol, agreement per family, null floors, worked example, and scope\.L[Additional Qwen Displays and Feature Audits](https://arxiv.org/html/2609.01936#A12)\(p\.[L](https://arxiv.org/html/2609.01936#A12)\)\.
Margin examples, examples of selected scores, fixed\-token contexts, profiles, lens/prism comparisons, and token row audits\.M[Feature\-Resolved Direct Logit Attribution](https://arxiv.org/html/2609.01936#A13)\(p\.[M](https://arxiv.org/html/2609.01936#A13)\)\.
Connection to direct logit attribution, DLA terms split by SAE feature, and additivity error\.N[Constrained Readout\-Side Edit Test](https://arxiv.org/html/2609.01936#A14)\(p\.[N](https://arxiv.org/html/2609.01936#A14)\)\.
Constrained edit for SAE decoder directions on the readout side, held\-out terms, cross\-model repeats, and the matched\-KL frontier with cross\-model losses\.O[Discussion and Future Work](https://arxiv.org/html/2609.01936#A15)\(p\.[O](https://arxiv.org/html/2609.01936#A15)\)\.
Interpretation details, model\-family results, feature\-resolved DLA terms, and extensions\.P[Extended Related Work](https://arxiv.org/html/2609.01936#A16)\(p\.[P](https://arxiv.org/html/2609.01936#A16)\)\.
Additional lens\-method context and output\-embedding related work\.

## Appendix ASRP Score Decomposition Identities

This appendix states the decomposition identities used by the figures and control tables and fixes the bookkeeping conventions for displaying a trained readout SAE\. The identities are the same local score decomposition introduced in §[3\.2](https://arxiv.org/html/2609.01936#S3.SS2)\. All reported scores are evaluated in the model’s raw readout space\. If training used centering or row normalization, the SAE coefficients, reconstructed rows, and residual terms are first mapped back to that space\.

Table 3:Selected scores and their sparse\-feature coefficients\.Rows instantiate the selected readout directionqαq\_\{\\alpha\}and corresponding direction coefficientβi​\(α\)\\beta\_\{i\}\(\\alpha\)\. Barredwwandzzdenote uniform token\-set averages, and for top competitor margins,w¯π,ℛ=∑rπr​wr\\bar\{w\}\_\{\\pi,\\mathcal\{R\}\}=\\sum\_\{r\}\\pi\_\{r\}w\_\{r\}andz¯π,ℛ,i=∑rπr​zr,i\\bar\{z\}\_\{\\pi,\\mathcal\{R\},i\}=\\sum\_\{r\}\\pi\_\{r\}z\_\{r,i\}, whereπr\\pi\_\{r\}are the nonnegative competitor weights summing to one over the competitor setℛ\\mathcal\{R\}\. HereA,B∈𝒱A,B\\in\\mathcal\{V\}are single tokens,𝒜,ℬ⊂𝒱\\mathcal\{A\},\\mathcal\{B\}\\subset\\mathcal\{V\}are token groups, anduuis the top\-ranked token under the original LM head\.Selected score familyqαq\_\{\\alpha\}βi​\(α\)\\beta\_\{i\}\(\\alpha\)Token logitwAw\_\{A\}zA,iz\_\{A,i\}Vocabulary\-mean contrastwA−w¯𝒱w\_\{A\}\-\\bar\{w\}\_\{\\mathcal\{V\}\}zA,i−z¯𝒱,iz\_\{A,i\}\-\\bar\{z\}\_\{\\mathcal\{V\},i\}Logit differencewA−wBw\_\{A\}\-w\_\{B\}zA,i−zB,iz\_\{A,i\}\-z\_\{B,i\}Group contrastw¯𝒜−w¯ℬ\\bar\{w\}\_\{\\mathcal\{A\}\}\-\\bar\{w\}\_\{\\mathcal\{B\}\}z¯𝒜,i−z¯ℬ,i\\bar\{z\}\_\{\\mathcal\{A\},i\}\-\\bar\{z\}\_\{\\mathcal\{B\},i\}Top competitor marginwu−w¯π,ℛw\_\{u\}\-\\bar\{w\}\_\{\\pi,\\mathcal\{R\}\}zu,i−z¯π,ℛ,iz\_\{u,i\}\-\\bar\{z\}\_\{\\pi,\\mathcal\{R\},i\}### A\.1Raw\-Space Row Decomposition

Letwv∈ℝdmodelw\_\{v\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}be the column\-vector form of unembedding rowvv, following §[3](https://arxiv.org/html/2609.01936#S3)\. Under the row preprocessing convention used by the reported analyses,

xv\\displaystyle x\_\{v\}=wv−μav,av\>0,\\displaystyle=\\frac\{w\_\{v\}\-\\mu\}\{a\_\{v\}\},\\qquad a\_\{v\}\>0,xv\\displaystyle x\_\{v\}=∑izv,ipre​dipre\+rvpre,\\displaystyle=\\sum\_\{i\}z^\{\\mathrm\{pre\}\}\_\{v,i\}d^\{\\mathrm\{pre\}\}\_\{i\}\+r^\{\\mathrm\{pre\}\}\_\{v\},withav=1a\_\{v\}=1for unnormalized rows\. The raw\-space decomposition then uses

zv,i\\displaystyle z\_\{v,i\}=av​zv,ipre,\\displaystyle=a\_\{v\}z^\{\\mathrm\{pre\}\}\_\{v,i\},di\\displaystyle d\_\{i\}=dipre,\\displaystyle=d^\{\\mathrm\{pre\}\}\_\{i\},rv\\displaystyle r\_\{v\}=av​rvpre\.\\displaystyle=a\_\{v\}r^\{\\mathrm\{pre\}\}\_\{v\}\.so that

wv=μ\+∑izv,i​di\+rv\.w\_\{v\}=\\mu\+\\sum\_\{i\}z\_\{v,i\}d\_\{i\}\+r\_\{v\}\.The displayed local score decomposition contains only this shared offset, SAE feature terms, and the explicit residual term, with no hidden dense base term added\.

#### Why the offset stays outside the dictionary\.

Nothing forbids folding the shared offset into the dictionary, sinceμ\\muis a direction inℝdmodel\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}like any other and the row reconstruction could be written with one additional atom\. We keep it explicit because that atom would carry coefficient one on every row by construction, so it would occupy one of thekkactive slots for every token and spend sparse capacity on a quantity that is identical everywhere\. Its decoder direction would also be drawn toward the row mean, away from structure that distinguishes rows\. An explicit centering term is the standard arrangement in sparse autoencoder practice\([Bricken et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib7);[Gao et al\., 2025a](https://arxiv.org/html/2609.01936#bib.bib21)\), and centering output embeddings in particular is a documented stabilizer\([Stollenwerk et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib70)\)\. Holdingμ\\muoutside the sparse code lets the features model deviations from the shared row mean, and it keeps the offset a separately visible term in the identity for a selected score below, where its cancellation becomes a property of the selected direction, independent of which atoms happened to be learned\.

### A\.2Selected\-Score Identity

For a hidden statehℓh\_\{\\ell\}, leth~ℓ=Tℓ​\(hℓ\)\\tilde\{h\}\_\{\\ell\}=T\_\{\\ell\}\(h\_\{\\ell\}\)be the corresponding state presented to the LM head, following §[3](https://arxiv.org/html/2609.01936#S3)\. A selected readout direction is a linear function of vocabulary rows,

qα=∑v∈𝒱αv​wv,sα​\(h~ℓ\)=h~ℓ⊤​qα\.q\_\{\\alpha\}=\\sum\_\{v\\in\\mathcal\{V\}\}\\alpha\_\{v\}w\_\{v\},\\qquad s\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}q\_\{\\alpha\}\.Define the direction coefficient, selected\-direction residual, and SAE feature contribution as

βi​\(α\)\\displaystyle\\beta\_\{i\}\(\\alpha\)=∑vαv​zv,i,\\displaystyle=\\sum\_\{v\}\\alpha\_\{v\}z\_\{v,i\},rα\\displaystyle r\_\{\\alpha\}=∑vαv​rv,\\displaystyle=\\sum\_\{v\}\\alpha\_\{v\}r\_\{v\},ci​\(h~ℓ,α\)\\displaystyle c\_\{i\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=βi​\(α\)​pi​\(h~ℓ\),\\displaystyle=\\beta\_\{i\}\(\\alpha\)\\,p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\),wherepi​\(h~ℓ\)=h~ℓ⊤​dip\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}d\_\{i\}is the readout feature projection of §[3\.1](https://arxiv.org/html/2609.01936#S3.SS1)\. Substituting the raw\-space row decomposition gives the exact grouped identity

sα​\(h~ℓ\)\\displaystyle s\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\)=sμ​\(h~ℓ,α\)\+sfeat​\(h~ℓ,α\)\\displaystyle=s\_\{\\mu\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)\+s\_\{\\mathrm\{feat\}\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)\+sresid​\(h~ℓ,α\),\\displaystyle\+s\_\{\\mathrm\{resid\}\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\),sμ​\(h~ℓ,α\)\\displaystyle s\_\{\\mu\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=\(∑vαv\)​h~ℓ⊤​μ,\\displaystyle=\\left\(\\sum\_\{v\}\\alpha\_\{v\}\\right\)\\tilde\{h\}\_\{\\ell\}^\{\\top\}\\mu,sfeat​\(h~ℓ,α\)\\displaystyle s\_\{\\mathrm\{feat\}\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=∑ici​\(h~ℓ,α\),\\displaystyle=\\sum\_\{i\}c\_\{i\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\),sresid​\(h~ℓ,α\)\\displaystyle s\_\{\\mathrm\{resid\}\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=h~ℓ⊤​rα\.\\displaystyle=\\tilde\{h\}\_\{\\ell\}^\{\\top\}r\_\{\\alpha\}\.For zero\-sum contrasts, including pairwise logit differences, group contrasts, vocabulary mean contrasts, and top competitor margins, the offset term cancels\.[Figure2](https://arxiv.org/html/2609.01936#S3.F2)depicts the decomposition\.

### A\.3Error, Sign, and Display Rules

For a selected readout directionqαq\_\{\\alpha\}, the exact and reconstructed scores are

sexact\\displaystyle s\_\{\\mathrm\{exact\}\}=h~ℓ⊤​qα,\\displaystyle=\\tilde\{h\}\_\{\\ell\}^\{\\top\}q\_\{\\alpha\},srecon\\displaystyle s\_\{\\mathrm\{recon\}\}=sμ​\(h~ℓ,α\)\+sfeat​\(h~ℓ,α\)\.\\displaystyle=s\_\{\\mu\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)\+s\_\{\\mathrm\{feat\}\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)\.The residual score is the reconstruction error,

ϵ⁡\(h~ℓ,α\)=sexact−srecon=sresid​\(h~ℓ,α\),\\epsilon\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=s\_\{\\mathrm\{exact\}\}\-s\_\{\\mathrm\{recon\}\}=s\_\{\\mathrm\{resid\}\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\),and aggregate summaries use the parameterized relative error

ρδ​\(h~ℓ,α\)=\|ϵ⁡\(h~ℓ,α\)\|\|sexact\|\+δ,\\rho\_\{\\delta\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=\\frac\{\|\\epsilon\(\\tilde\{h\}\_\{\\ell\},\\alpha\)\|\}\{\|s\_\{\\mathrm\{exact\}\}\|\+\\delta\},withδ=0\.5\\delta=0\.5logits, denotedρ0\.5\\rho\_\{0\.5\}, in aggregate summaries\. We drop the arguments ofϵ\\epsilonandρδ\\rho\_\{\\delta\}where the state and coefficient vector are clear from context\. The floor prevents arbitrarily small margins from dominating median error summaries, and near the decision boundary sign agreement and margin\-binned flip rates are the primary reliability checks\. The unfloored variantρ0​\(h~ℓ,α\)=\|ϵ⁡\(h~ℓ,α\)\|/\|sexact\|\\rho\_\{0\}\(\\tilde\{h\}\_\{\\ell\},\\alpha\)=\|\\epsilon\(\\tilde\{h\}\_\{\\ell\},\\alpha\)\|/\|s\_\{\\mathrm\{exact\}\}\|is used for identity checks, case\-study displays, and all baseline comparisons\. It upper\-boundsρ0\.5\\rho\_\{0\.5\}, and the two subscripts are never mixed within one table\. For signed contrasts, side\-specific feature support is evidence\-bearing when

sign⁡\(sexact\)=sign⁡\(srecon\)\.\\operatorname\{sign\}\(s\_\{\\mathrm\{exact\}\}\)=\\operatorname\{sign\}\(s\_\{\\mathrm\{recon\}\}\)\.Each interpreted figure states the selected readout score, exact score, reconstructed score, reconstruction error, relative reconstruction error, and sign status for contrasts\. If a plot displays only the largest bars, the reported feature sum and residual are still computed from the full active feature sum\. Token\-level figures report tokenization or row audit information where it affects interpretation, and brittle single\-token contrasts are handled as explicit group contrasts when needed\.

## Appendix BModel Suite and Selection Rationale

This appendix explains the model suite used to evaluate Sparse Readout Prism, which spans the Qwen model suite\([Qwen Team, 2025](https://arxiv.org/html/2609.01936#bib.bib62);[Qwen Team, 2026](https://arxiv.org/html/2609.01936#bib.bib63)\), the Gemma model suite\([Google DeepMind, 2026a](https://arxiv.org/html/2609.01936#bib.bib25);[Google DeepMind, 2026b](https://arxiv.org/html/2609.01936#bib.bib26)\), Ministral\([Mistral AI, 2026](https://arxiv.org/html/2609.01936#bib.bib50)\), and targeted DeepSeek\-R1 distilled comparison rows\([DeepSeek\-AI, 2025](https://arxiv.org/html/2609.01936#bib.bib12)\)\. Appendices[C](https://arxiv.org/html/2609.01936#A3)–[F](https://arxiv.org/html/2609.01936#A6)give the metric summary, reporting the trained widths, nativekksettings \(the TopK budget fixed at training\), rowEV, top\-1, KL, and usage statistics for each readout SAE setting\. This appendix motivates the model choices\.

Sparse Readout Prism factorizes the output\-token scoring matrix, so the model suite is organized around readout fidelity, scale, and portability\. The suite spans model size and readout geometry while preserving enough budget for tokenizer, row\-norm, special\-token, and matched\-control analyses\.

### B\.1Selection Rationale

The Qwen3\.5 rows provide a fixed qualitative display setting and Qwen\-family companion analyses\. Qwen3\.5\-9B is the capacity anchor, large enough to be a meaningful test of readout factorization while still supporting recipe and width sweeps\. Qwen3\.5\-0\.8B and Qwen3\.5\-2B test the same recipe at smaller unembedding dimensions and cheaper rerun settings\.

The Gemma rows provide a second tied\-readout family with a different tokenizer and final\-logit softcap\. These rows exercise the readout SAE recipe under a distinct readout geometry, and the softcap\-correct readout evaluation is reported in Appendix[E](https://arxiv.org/html/2609.01936#A5)\.

Additional targeted rows broaden the coverage of readout geometries\. Ministral\-3\-8B\-Base tests transfer to an untied, softcap\-free 8B\-scale readout outside Qwen\. R1\-Distill\-Qwen\-7B and R1\-Distill\-Llama\-8B evaluate the same recipe\-selection measurements on reasoning\-distilled comparison rows, where row reconstruction and LM head replacement fidelity are visible as separate result axes\.

### B\.2Final Suite

Table[4](https://arxiv.org/html/2609.01936#A2.T4)lists the evaluated models and the role each one plays in the results\.

Table 4:Evaluated model suite and role\. Exact dictionary settings and metrics are in Appendices[C](https://arxiv.org/html/2609.01936#A3)–[F](https://arxiv.org/html/2609.01936#A6)\.Model\(s\)Why includedRole in studyQwen3\.5\-0\.8BQwen3\.5\-2BSmall\-model transfer and lower\-cost repeats under the nativek=128k=128strict budget andk=256k=256high fidelity regimes\.Qwen transfer evidence\.Qwen3\.5\-9BScale anchor for recipe selection, capacity sweeps, and high fidelity readout replacement\.Qwen fidelity evidence\.Gemma\-4\-E2BGemma\-4\-E4BCross\-family evidence for whether the readout SAE recipe transfers beyond Qwen\.Cross\-family transfer evidence\.R1\-Distill\-Qwen\-7BQwen\-derived post\-training comparison with an untied, softcap\-free readout\.rowEV vs\. fidelity separation evidence\.R1\-Distill\-Llama\-8BReasoning\-distilled comparison from a non\-Qwen base family \(both dictionary settings\) for cross\-architecture confirmation of the rowEV / fidelity gap\.Cross\-architecture post\-training evidence\.Ministral\-3\-8B\-BaseCross\-family 8B\-scale comparison with an untied, softcap\-free readout \(adds an 8B\-scale non\-Qwen readout geometry\)\.Cross\-family fidelity comparison\.Qualitative feature analyses use a single low\-error display setting so that examples share the same readout SAE and reporting convention\. The choice is evidence\-driven, since the additional rows show how the same metrics vary with readout geometry and post\-training regime\.

### B\.3Interpretation Details

We built the suite to evaluate readout factorization\. We report token frequency, length, special\-token behavior, row norm, tied embedding status, softcap handling, and matched\-score identity bias so single\-token logit decompositions can be compared on the same footing\.

The evaluation makes three commitments\. First, model\-specific readout layers are handled before reporting decision fidelity\. Second, the SAE architecture, training recipe, seed convention, dictionary settings, and evaluation settings are reported with the sweep records in Appendices[C](https://arxiv.org/html/2609.01936#A3)–[F](https://arxiv.org/html/2609.01936#A6)\. Third, primary reported claims are stated for selected readout scores, especially token logits and contrastive logit differences\.

## Appendix CHow the Readout SAEs Were Selected

This appendix documents how the readout SAEs used in the main text are chosen\. The selection criterion combines row reconstruction with readout\-distribution preservation after substitutingWU^\\widehat\{W\_\{U\}\}forWUW\_\{U\}, namely rowEV, top\-1 agreement, KL in bits, selected\-logit reconstruction errors when available, and dead/rare feature usage\. A feature is counted as rare when its evaluation firing rate is below10−310^\{\-3\}\. All trained runs use seed00unless stated otherwise, and the Qwen3\.5\-0\.8B 32×\\times,k=256k=256dictionary setting has the three\-seed stability window reported below\.

#### Metric glossary\.

rowEV is row\-centered explained variance for reconstructed unembedding rows\. top\-1 is agreement between the original and reconstructed vocabulary argmaxes over held\-out decoded states\. top5 is mean top\-five set overlap,\|top5⁡\(WU​h~L\)∩top5⁡\(WU^​h~L\)\|/5\|\\mathrm\{top5\}\(W\_\{U\}\\tilde\{h\}\_\{L\}\)\\cap\\mathrm\{top5\}\(\\widehat\{W\_\{U\}\}\\tilde\{h\}\_\{L\}\)\|/5, and as a set\-overlap metric it can be lower or higher than top\-1\. KL is the median over held\-out decoded states ofDKL\(pWU\(⋅∣h~L\)∥pWU^\(⋅∣h~L\)\)D\_\{\\mathrm\{KL\}\}\(p\_\{W\_\{U\}\}\(\\cdot\\mid\\tilde\{h\}\_\{L\}\)\\\|p\_\{\\widehat\{W\_\{U\}\}\}\(\\cdot\\mid\\tilde\{h\}\_\{L\}\)\)in bits\. Selected\-logit reconstruction error is the absolute fractional error on the original top logit\.

#### Qwen3\.5\-9B unembedding object\.

The Qwen3\.5\-9B experiments factorize the final unembedding matrixWU∈ℝ248320×4096W\_\{U\}\\in\\mathbb\{R\}^\{248320\\times 4096\}\. The prism loss is the auxiliary training term, weightedλprism\\lambda\_\{\\mathrm\{prism\}\}, that scores reconstruction in logit space on decoded states, on top of the row\-space objective\. The states used for it and for evaluation are the final RMSNorm outputs immediately before the language\-model head\. Anlm\_headpre\-hook confirms this, since the captured input reproduces the model readout with no additional post\-readout transform\. The remaining discrepancy between reconstruction and readout is consistent with the model’s bf16 precision floor of the LM head as the relevant limit\.

#### Selection rule\.

The active budgetkkis part of the trained SAE\. We therefore report nativek=128k=128strict budget rows and nativek=256k=256high fidelity rows as separate operating regimes, and truncatedk=256k=256dictionaries are over\-budget comparisons\. Within each regime, rowEV is paired with top\-1, KL, reconstruction error, achieved eval\-L0L\_\{0\}where applicable, and usage statistics\. Orthogonal matching pursuit \(OMP\) rows\([Pati et al\., 1993](https://arxiv.org/html/2609.01936#bib.bib58)\)are sparse\-inference comparisons over fixed dictionaries, while deployed metrics use the learned TopK encoders\.

#### Per\-model settings\.

The sweep supports two operating regimes\. The strict budget rows preserve a nativek=128k=128interpretability budget, while the high fidelity rows use nativek=256k=256encoders when readout fidelity is the primary constraint\. Table[5](https://arxiv.org/html/2609.01936#A3.T5)summarizes these settings, and the model\-by\-model comparisons supporting them appear in Appendix[E\.1](https://arxiv.org/html/2609.01936#A5.SS1)\.

Table 5:Representative dictionary settings from the Qwen/Gemma sweep\. Nativek=128k=128rows support the strict budget regime and nativek=256k=256rows support the high fidelity regime\. Gemma top\-1/KL are softcap\-correct\.ModelSettingrowEVtop\-1KLrareRoleStrict\-budget regime, nativek=128k=128Qwen3\.5\-0\.8B16×\\times,k=128k=1280\.7600\.7600\.8440\.8440\.2770\.2770\.0010\.001strict budget resultQwen3\.5\-2B16×\\times,k=128k=1280\.7120\.7120\.8580\.8580\.2610\.2610\.0040\.004strict budget runner\-upGemma\-4\-E2B16×\\times,k=128k=1280\.7140\.7140\.6230\.6231\.941\.940\.0310\.031strict budget, softcap\-correctGemma\-4\-E4B16×\\times,k=128k=1280\.6930\.6930\.6690\.6691\.821\.820\.0400\.040strict budget, softcap\-correctHigh\-fidelity regime, nativek=256k=256Qwen3\.5\-0\.8B32×\\times,k=256k=2560\.8770\.8770\.8910\.8910\.1350\.1350\.0010\.001high fidelity resultQwen3\.5\-2B32×\\times,k=256k=2560\.8470\.8470\.8870\.8870\.1360\.1360\.0100\.010high fidelity runner\-upQwen3\.5\-9B16×\\times,k=256k=2560\.7610\.7610\.8740\.8740\.1670\.1670\.0610\.061compact 9B comparisonQwen3\.5\-9B32×\\times,k=256k=2560\.8570\.8570\.9000\.9000\.1050\.1050\.2920\.292reported 9B pointGemma\-4\-E2B32×\\times,k=256k=2560\.8340\.8340\.3330\.3336\.376\.370\.0380\.038fidelity row, softcap\-correctGemma\-4\-E4B32×\\times,k=256k=2560\.8270\.8270\.7360\.7361\.221\.220\.0780\.078fidelity row, softcap\-correct
### C\.1Sweep Coverage and Evidence Map

The selection evidence has three distinct roles, choosing a Qwen3\.5\-9B recipe, testing how that recipe scales with capacity, and evaluating whether the same recipe transfers to smaller Qwen and Gemma models\. These roles are separated because the runs differ in model scale, active sparsity, convergence stage, and evaluation design\. Table[6](https://arxiv.org/html/2609.01936#A3.T6)maps each family to the decision it supports\.

Table 6:Sweep coverage for Appendix[C](https://arxiv.org/html/2609.01936#A3)\. Run families and the decision each supports, without listing every cell\.Sweep familyCoverageAxes variedSelection useOriginal Qwen3\.5\-9B baselineunembedding measurements, short\-run comparisons, and a 50k runmatryoshka width,kk,λ\\lambdaanchors the old 50k baseline at rowEV0\.5110\.511, top\-10\.8070\.807Qwen fixed\-k=128k=128recipe searchunembedding measurements, short\-run recipe comparison, and 20k runinit, width, sampling, TopK/matryoshka objective, delayed rampselects thetopk8×\\times,k=128k=128recipeQwen BatchTopK control\([Bussmann et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib8)\)target\-L0L\_\{0\}measurements and evaluation\-L0L\_\{0\}controlseval\-L0L\_\{0\}targeting, random vs\. row\-seeded initrejects BatchTopK for this settingQwen capacity sweephigher\-kkscreening runs, 16×\\timesand 32×\\times20k runs, and OMP comparisonwidth and active budgetkkcompares the 32×\\times,k=256k=256high\-capacity point with the 16×\\times,k=256k=256compact pointQwen JumpReLU control\([Rajamanoharan et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib65)\)four JumpReLU cells and matched TopK controlsLR, prism ramp, controller target\-L0L\_\{0\}rejects JumpReLU as a reported readout SAESmall\-model transfercomplete four\-model width/kkscreening grid plus 20k runsmodel family, width,kkselects representative nativek=128k=128andk=256k=256transfer rows under the shared metric suiteThe following readout SAE appendices follow this evidence map, taking first the Qwen3\.5\-9B recipe, then the capacity and operating\-point comparisons, then smaller\-model transfer and additional metric analyses\.

## Appendix DQwen Readout SAE Recipe and Capacity Sweeps

This appendix gives the Qwen3\.5\-9B recipe\-selection and capacity evidence summarized in Appendix[C](https://arxiv.org/html/2609.01936#A3)\.

### D\.1Qwen3\.5\-9B Fixed\-k=128k=128Recipe Selection

The first Qwen3\.5\-9B sweep fixes the interpretability budget atk=128k=128and selects a no\-PCA recipe\. A 5k screening sweep identified the recipe family, a 20k continuation evaluated convergence of the selected recipe, and the older 50k matryoshka run\([Bussmann et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib9)\)anchored the comparison against the previous long\-trained baseline\. The selected TopK recipe’s own convergence ladder provides the step\-count evidence, while the previous recipe is an architectural reference point\. Table[7](https://arxiv.org/html/2609.01936#A4.T7)maps each run to the evidence it supplies, and Table[8](https://arxiv.org/html/2609.01936#A4.T8)gives the main effects the sweep isolates\.

Table 7:Evidence map for the Qwen3\.5\-9B fixed\-k=128k=128recipe\-selection experiment\. Per\-run metrics for the 5k screening sweep are provided with Figure[6](https://arxiv.org/html/2609.01936#A4.F6)\.Evidence roleRun or run familyWhat it establishesPrevious baselineMatryoshka TopK,D=16384D=16384,k=128k=128, 50k stepsLong\-trained previous recipe used as an architectural reference, at rowEV0\.5110\.511, top\-10\.8070\.807, KL0\.4160\.416, dead0\.1190\.119, rare0\.2360\.236\.5k selected recipeTopK,D=32768D=32768,k=128k=128, row\-seeded, hybrid row sampling, delayed prism rampBest recipe\-region representative in the screening sweep, at rowEV0\.5670\.567, top\-10\.7750\.775, KL0\.5720\.572, reconstruction error0\.0380\.038, rare0\.0600\.060\.5k screening sweep18 recipe variants varying initialization, architecture, row sampling, prism schedule, and widthSupports the selected recipe by matched comparisons, with row\-seeded initialization, TopK, delayed ramp, andD=32768D=32768each selected from the screening grid\.20k selected recipeTopK,D=32768D=32768,k=128k=128, row\-seeded, hybrid row sampling, delayed prism rampFinal reported Qwen3\.5\-9B fixed\-k=128k=128recipe, at rowEV0\.6210\.621, top\-10\.8460\.846, KL0\.2960\.296, reconstruction error0\.0270\.027, dead0\.0000\.000, rare0\.0210\.021\.Table 8:Main effects in the fixed\-k=128k=128Qwen3\.5\-9B recipe sweep\.FactorEvidenceConsequenceInitializationrow\_seededwas2\.0×2\.0\\timesrandomat 5k \(rowEV0\.5630\.563vs\.0\.2780\.278atD=32768D=32768\)\.Use row\-seeded initialization\.Prism\-loss scheduleDelayed ramp kept rowEV unchanged at0\.5670\.567while reducing selected logit reconstruction error from0\.0640\.064to0\.0380\.038in the matched TopK hybrid cell\.Rampλprism\\lambda\_\{\\mathrm\{prism\}\}to10−310^\{\-3\}after70%70\\%of training\.ArchitectureAtD=32768D=32768/5k, plain TopK exceeded matryoshka variants\([Bussmann et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib9)\)in rowEV \(0\.5670\.567vs\.0\.559/0\.5540\.559/0\.554\) at about3\.6×3\.6\\timeslower cost\.Use plain TopK for Qwen/Gemma\.WidthD=16384D=16384trailedD=32768D=32768by about0\.060\.06rowEV in the fixed\-k=128k=128recipe sweep\.UseD=32768D=32768for the 9Bk=128k=128recipe\.Figure 6:Qwen3\.5\-9B fixed\-k=128k=128recipe\-selection frontier\. Points are 5k recipe variants and the black curve is the 5k non\-dominated frontier\. The final 20k TopK recipe improves both reconstruction and KL at the fixed interpretability budget\.The 9B fixed\-k=128k=128recipe istopk,D=32768D=32768\(8×\\times\),k=128k=128, row\-seeded initialization,hybrid\_50freq\_50uniformrow sampling, warmup\-cosine learning rate10−3→10−410^\{\-3\}\\rightarrow 10^\{\-4\}, delayed prism ramp to10−310^\{\-3\}, 20k steps\. It reaches rowEV0\.62060\.6206, top\-10\.84570\.8457, KL0\.29550\.2955, selected logit reconstruction error0\.0270\.027, dead0\.0000\.000, rare0\.0210\.021, and Gini0\.2560\.256\. The 15k→\\rightarrow20k rowEV gain was\+0\.0025\+0\.0025, consistent with the stop rule \(Figure[7](https://arxiv.org/html/2609.01936#A4.F7)\)\.

Figure 7:Convergence ladder for the selected Qwen3\.5\-9B fixed\-k=128k=128recipe\. The final 15k→\\rightarrow20k rowEV gain is small while decision\-fidelity and usage metrics remain stable, which is why we stop at 20k\.#### Recipe\-selection conclusion\.

The selected no\-PCA TopK recipe improves over the previous long\-trained matryoshka baseline on reconstruction, top\-1 agreement, KL, and feature usage, and its 5k–20k ladder supports the 20k stop rule\. The main drivers are row\-seeded initialization, delayed prism ramping, plain TopK, and theD=32768D=32768width choice\.

### D\.2Qwen3\.5\-9B Capacity Sweep

After recipe selection, we vary capacity along dictionary width and active sparsity budget\. This targeted sweep separates two questions that rowEV alone can blur, namely which nativek=128k=128readout SAE should be used for fixed\-budget comparisons, and which nativek=256k=256readout SAE should be the high fidelity operating point\.

A 5k screen selected which larger\-capacity cells received 20k training\. At 16×\\times, thek=256k=256cell had better rowEV and rare feature rate thank=192k=192\(0\.695/0\.1000\.695/0\.100vs\.0\.675/0\.1640\.675/0\.164\), despite lower 5k top\-1, so it was continued\. The 32×\\times,k=256k=256cell had enough rowEV headroom \(0\.7990\.799\) to justify a width continuation\. The native 20k rows below provide the evidence for the operating\-point claims\.

Table 9:Converged 20k native Qwen3\.5\-9B capacity rows under the selected recipe family\. Nativek=128k=128and nativek=256k=256rows are separate operating regimes\.RunWidthEvalkkrowEVtop\-1KLrareInterpretationD=32768D=32768, traink=128k=1288×\\times1280\.6210\.6210\.8460\.8460\.2960\.2960\.0210\.021strict budget recipeD=65536D=65536, traink=256k=25616×\\times2560\.7610\.7610\.8740\.8740\.1670\.1670\.0610\.061compactk=256k=256comparisonD=131072D=131072, traink=256k=25632×\\times2560\.8570\.8570\.9000\.9000\.1050\.1050\.2920\.292best rowEV/top\-1/KL, high rare feature rateFigure 8:Qwen3\.5\-9B capacity sweep\. Increasing width and active sparsity improves rowEV and top\-1 agreement, while rare feature usage is an additional usage statistic\.Table[9](https://arxiv.org/html/2609.01936#A4.T9)and Figure[8](https://arxiv.org/html/2609.01936#A4.F8)show the tradeoff\. TheD=131072D=131072,k=256k=256run has the best rowEV, top\-1, and KL in this Qwen3\.5\-9B capacity sweep\. Its rare feature ladder0\.300→0\.294→0\.2920\.300\\rightarrow 0\.294\\rightarrow 0\.292shows a persistent rare\-feature tail, which is reported as a usage property alongside fidelity metrics\.

The 16×\\times,k=256k=256comparison provides the compact high fidelity point\. It also improves during convergence, since from 5k to 20k steps its rowEV ladder is0\.719→0\.749→0\.758→0\.7610\.719\\rightarrow 0\.749\\rightarrow 0\.758\\rightarrow 0\.761, and its rare feature rate falls from0\.1000\.100to0\.0610\.061\.

### D\.3Lower Sparsity Budgets

Activation SAEs are often trained at smaller absolute budgets than ours, which raises the question of whether a readout SAE would also work atk=64k=64or below\. We retrained Qwen3\.5\-2B and Qwen3\.5\-0\.8B under an otherwise identical recipe \(20,000 steps, batch4,0964\{,\}096, same data, schedule, and initialization family, with only width,kk, and seed differing\) and measured how much of each dictionary is ever used \(Table[10](https://arxiv.org/html/2609.01936#A4.T10)\)\.

Table 10:Sparsity budget versus dictionary usage\. Ranges are over independently seeded runs at otherwise identical settings \(20,000 steps each\), and single\-seed rows give the one measured value\. Dead counts features never active on the evaluation rows, and rare an evaluation firing rate below10−310^\{\-3\}\(Appendix[C](https://arxiv.org/html/2609.01936#A3)\)\. Width multipliers are relative todmodeld\_\{\\mathrm\{model\}\},20482048for Qwen3\.5\-2B and10241024for Qwen3\.5\-0\.8B\. The dead and rare rates are the quantities at issue in these rows, and rowEV, which is not what disqualifies a budget here, is entered as — where the run artifact does not carry it\.ModelSettingSeedsDeadRarerowEVtop\-1KLQ\-2B16×\\times\(D=32768D=32768\),k=32k=3210\.6940\.6940\.7760\.776—0\.7660\.7660\.4760\.476Q\-2B16×\\times\(D=32768D=32768\),k=64k=6420\.4910\.491–0\.5120\.5120\.6320\.632–0\.6450\.6450\.6830\.683–0\.6840\.6840\.7830\.783–0\.8160\.8160\.3670\.367–0\.3700\.370Q\-2B16×\\times\(D=32768D=32768\),k=256k=25620\.0360\.036–0\.0370\.0370\.2130\.213–0\.2180\.2180\.8010\.8010\.8070\.807–0\.8480\.8480\.2260\.226–0\.2430\.243Q\-2B32×\\times\(D=65536D=65536\),k=128k=12830\.3360\.336–0\.3520\.3520\.6280\.628–0\.6370\.6370\.7690\.769–0\.7710\.7710\.8160\.816–0\.8260\.8260\.2410\.241–0\.2530\.253Q\-0\.8B16×\\times\(D=16384D=16384\),k=32k=3210\.6410\.6410\.6870\.687—0\.7710\.7710\.4340\.434Q\-0\.8B16×\\times\(D=16384D=16384\),k=64k=6410\.3170\.3170\.4120\.412—0\.8010\.8010\.2960\.296At matched width, lowering the budget strands capacity\. Going fromk=256k=256tok=64k=64at 16×\\timesraises the dead\-feature rate from under4%4\\%to about half the dictionary \(0\.4910\.491–0\.5120\.512\), while reconstruction and replacement both degrade \(rowEV0\.80→0\.680\.80\\rightarrow 0\.68, KL0\.23→0\.370\.23\\rightarrow 0\.37\)\. Halving the budget again strands two thirds of the dictionary \(0\.6940\.694atk=32k=32\), and Qwen3\.5\-0\.8B repeats the progression at its own 16×\\timeswidth \(0\.3170\.317atk=64k=64,0\.6410\.641atk=32k=32\), so the pattern is not specific to one model\. The comparison is between budgets at a fixed width, since thek=128k=128rows sit at twice the width, where the same budget leaves about a third of the dictionary dead\. What governs usage is the budget relative to width, which is why we report sparsity ask/Dk/Dand as the number of features a score actually recruits \(Appendix[D\.4](https://arxiv.org/html/2609.01936#A4.SS4)\)\. Seed spread at each setting is small compared with the differences between settings, so the ordering does not depend on the seed\.

### D\.4Operating\-Point Claims

The final presentation uses two native operating regimes\. The strict budget readout uses nativek=128k=128readout SAEs for fixed\-budget comparability, and after recipe transfer the selected strict budget reconstruction point is Qwen3\.5\-0\.8B 16×\\times\(Appendix[E\.1](https://arxiv.org/html/2609.01936#A5.SS1)\)\. The high fidelity readout uses nativek=256k=256readout SAEs when readout accuracy is the limiting concern\. On Qwen3\.5\-9B the reported point is the 32×\\times,k=256k=256run, with rowEV0\.8570\.857, top\-10\.9000\.900, KL0\.1050\.105, and rare0\.2920\.292, and every 9B number in the main text comes from it\. The 16×\\times,k=256k=256run is retained as a compact comparison, with rowEV0\.7610\.761, top\-10\.8740\.874, KL0\.1670\.167, and rare0\.0610\.061\.

#### Sparsity of the operating points\.

Activation SAEs are often reported at smaller absolutekk, butkkis comparable across dictionaries only relative to width\. AtD=65,536D=65\{,\}536,k=128k=128andk=256k=256activate0\.20%0\.20\\%and0\.39%0\.39\\%of the dictionary per row\. What a reader inspects is sparser still, because a selected score concentrates its mass on a few features\. Across the six softcap\-free readouts, a median of6565–107107features carries80%80\\%of a score’s absolute contribution mass \(7676on the display model\), against dictionaries of tens of thousands\. The two operating points therefore differ only in budget, withk=128k=128the stricter interpretability budget andk=256k=256prioritizing reconstruction fidelity\. Budgets below this range are not simply sparser, since they leave much of the dictionary unused \(Appendix[D\.3](https://arxiv.org/html/2609.01936#A4.SS3)\)\. Truncating ak=256k=256dictionary back tok=128k=128after training does not substitute for the native recipe, since it can raise rowEV while sharply reducing top\-1 agreement \(Figure[9](https://arxiv.org/html/2609.01936#A4.F9)\)\.

Figure 9:Truncatingk=256k=256\-trained dictionaries back tok=128k=128can raise rowEV relative to the nativek=128k=128recipe while sharply reducing top\-1 agreement\. This rowEV vs\. fidelity inversion motivates treating the fixed interpretability budget as part of the trained object\.

## Appendix EReadout SAE Transfer and Model\-Family Comparisons

This appendix reports how the selected readout SAE recipe transfers across the smaller Qwen/Gemma suite and targeted cross\-family or post\-training rows\.

### E\.1Transfer to Smaller Models

The fixed Qwen3\.5\-9B recipe transfers to four smaller models\. The transfer tests both whether a fixed active budget reconstructs lower\-dimensional unembedding rows more effectively, and whether the higherk=256k=256fidelity regime remains useful away from 9B\. As above, nativek=128k=128and nativek=256k=256rows are reported as separate operating regimes, and truncation comparisons are handled as capacity controls\.

#### Coverage\.

The 5k screen covered all 24 model/width/kkcells \(Figure[10](https://arxiv.org/html/2609.01936#A5.F10)\)\. In every model, the largest screening rowEV occurred at 32×\\times,k=256k=256, giving Qwen3\.5\-0\.8B0\.8120\.812, Qwen3\.5\-2B0\.7650\.765, Gemma\-4\-E2B0\.7110\.711, and Gemma\-4\-E4B0\.7420\.742\. These screening values select continuations, and the 20k rows in Table[11](https://arxiv.org/html/2609.01936#A5.T11)supply the reported dictionary settings\.

![Refer to caption](https://arxiv.org/html/2609.01936v1/appendix_k_small_model_pilot_grid.png)Figure 10:Small\-model screening grid\. Each model evaluated at 8×\\times, 16×\\times, and 32×\\timesunder nativek=128k=128andk=256k=256, while final claims use the native 20k rows in Table[11](https://arxiv.org/html/2609.01936#A5.T11)\.Table 11:Converged 20k small\-model results\. The 16×\\times,k=128k=128rows instantiate the strict budget regime and the 32×\\times,k=256k=256rows the high fidelity regime\. Gemma top\-1/KL are softcap\-correct\.ModelDictionary settingrowEVtop\-1KLdead/rareRoleQwen3\.5\-0\.8B16×\\times,k=128k=1280\.7600\.7600\.8440\.8440\.2770\.2770\.000/0\.0010\.000/0\.001strict budget transfer resultQwen3\.5\-0\.8B32×\\times,k=256k=2560\.877\\mathbf\{0\.877\}0\.8910\.8910\.1350\.1350\.000/0\.0010\.000/0\.001high fidelity transfer resultQwen3\.5\-2B16×\\times,k=128k=1280\.7120\.7120\.8580\.8580\.2610\.2610\.000/0\.0040\.000/0\.004strict budget transferQwen3\.5\-2B32×\\times,k=256k=2560\.8470\.8470\.8870\.8870\.1360\.1360\.000/0\.0100\.000/0\.010high fidelity transferGemma\-4\-E2B16×\\times,k=128k=1280\.7140\.7140\.6230\.6231\.941\.940\.001/0\.0310\.001/0\.031softcap\-correct fidelityGemma\-4\-E2B32×\\times,k=256k=2560\.8340\.8340\.3330\.3336\.376\.370\.000/0\.0380\.000/0\.038high rowEV, lower replacement fidelityGemma\-4\-E4B16×\\times,k=128k=1280\.6930\.6930\.6690\.6691\.821\.820\.012/0\.0400\.012/0\.040softcap\-correct fidelityGemma\-4\-E4B32×\\times,k=256k=2560\.8270\.8270\.7360\.7361\.221\.220\.002/0\.0780\.002/0\.078fidelity row, softcap\-correctTable 12:Per\-kkevaluation curve for the completed Gemma\-4\-E4B 32×\\times,k=256k=256convergence\.kevalk\_\{\\mathrm\{eval\}\}rowEVtop\-1KL \(bits\)64640\.1130\.1130\.0080\.00838\.538\.51281280\.5680\.5680\.0370\.03731\.631\.61921920\.7740\.7740\.3210\.32110\.810\.8256256native0\.8270\.8270\.7360\.7361\.221\.223843840\.6840\.6840\.7200\.7201\.371\.37The nativek=256k=256evaluation in Table[12](https://arxiv.org/html/2609.01936#A5.T12)is the reported dictionary setting, and thek=384k=384row is an over\-budget comparison\. KL is softcap\-correct, while rowEV and top\-1 are cap\-independent / argmax\-invariant\.

Figure 11:Qwen scaling summary\. Panel A gives the Qwen rows used for the scaling claim, and panel B the rare feature rate as a usage statistic\. Additional model\-family rows appear in the surrounding tables\.Across the native Qwen dictionary settings in Figure[11](https://arxiv.org/html/2609.01936#A5.F11), additional capacity generally improves both rowEV and top\-1 agreement\. The selected strict budget transfer point is Qwen3\.5\-0\.8B 16×\\times,k=128k=128, and the selected high fidelity small\-model point is Qwen3\.5\-0\.8B 32×\\times,k=256k=256, with Qwen3\.5\-2B giving the same qualitative pattern\. The scaling figure focuses on Qwen so that model size is varied within a single family, and the surrounding tables evaluate the same recipe under additional readout geometries\. Rare feature rate is a usage statistic, and it stays low for the smaller Qwen 32×\\times,k=256k=256rows and rises to0\.2920\.292for the 9B 32×\\times,k=256k=256capacity point\. Qwen3\.5\-2B has higher strict budget top\-1, so the 0\.8B selection is a joint reconstruction\-and\-fidelity dictionary setting with top\-1 as one component\. Table[13](https://arxiv.org/html/2609.01936#A5.T13)repeats that 0\.8B setting across three seeds\.

Table 13:Qwen3\.5\-0\.8B 32×\\times,k=256k=256seed\-variation window\. Seed 0 is the archived dictionary setting, and seeds 1 and 2 repeat the same recipe\.SeedrowEVtop\-1KLrareWall0 \(archived\)0\.8770\.8770\.8910\.8910\.1350\.1350\.0010\.001–10\.8860\.8860\.8750\.8750\.1040\.1040\.1340\.13439 min20\.8870\.8870\.8790\.8790\.1050\.1050\.1420\.14251 minmean±\\pmstd0\.883±0\.0060\.883\\pm 0\.0060\.882±0\.0090\.882\\pm 0\.0090\.115±0\.0180\.115\\pm 0\.0180\.092±0\.0800\.092\\pm 0\.080–The seed window shows that the headline replacement metrics are stable under this recipe, since rowEV, top\-1, and KL have standard deviation at most0\.020\.02\. Rare feature rate varies more across seeds, so we report it as a usage statistic and use the seed\-window mean when making seed\-robust statements about that tail\.

### E\.2Cross\-Family 8B Comparison \(Ministral\)

Ministral\-3\-8B\-Base is included as a cross\-family 8B\-scale comparison\. It has the same hidden widthdmodel=4096d\_\{\\mathrm\{model\}\}=4096as Qwen3\.5\-9B but an untiedlm\_headand no post\-readout softcap, and the factorized readout object islanguage\_model\.lm\_head\.weightwith shape131,072×4096131\{,\}072\\times 4096\(readout\-extraction reconstruction relative error≈3\.2×10−6\\approx 3\.2\{\\times\}10^\{\-6\}, an order of magnitude cleaner than the bf16 Qwen floor\)\. The transferred TopK recipe applies at the matched widths\.

Table 14:Cross\-family and post\-training metrics under the transferred TopK recipe\. “Strict” denotes nativek=128k=128and “Fidelity” nativek=256k=256\.ModelDictionary settingWidth /kkrowEVtop\-1KLdeadrareRoleMinistral3\-8BStrict16×\\times,k=128k=1280\.8060\.8060\.8850\.8850\.1300\.1300\.0260\.0260\.4560\.456strict budget comparisonMinistral3\-8BFidelity32×\\times,k=256k=2560\.8880\.8880\.9040\.9040\.0870\.0870\.0180\.0180\.5290\.529fidelity comparisonR1\-DistillQwen\-7BStrict16×\\times,k=128k=1280\.7090\.7090\.6950\.6950\.7770\.7770\.0160\.0160\.2980\.298strict budget comparisonR1\-DistillQwen\-7BFidelity32×\\times,k=256k=2560\.8440\.8440\.7600\.7600\.4890\.4890\.0160\.0160\.3780\.378fidelity gapR1\-DistillLlama\-8BStrict16×\\times,k=128k=1280\.7960\.7960\.7250\.7250\.5360\.5360\.0090\.0090\.3560\.356strict budget comparisonR1\-DistillLlama\-8BFidelity32×\\times,k=256k=2560\.8880\.8880\.7540\.7540\.4340\.4340\.0110\.0110\.3920\.392fidelity gap \(rowEV\-matched, top\-1/KL apart\)Both Ministral dictionary settings have healthy usage and decision\-fidelity metrics \(dead<0\.03<0\.03, KL<0\.14<0\.14bits\)\. Decision fidelity \(top\-1, KL\) falls in the high fidelity range of the matched 32×\\times,k=256k=256comparison rows, which supports the claim that the row\-factorization recipe transfers beyond the Qwen family when the readout is geometrically clean \(untied, softcap\-free\)\. The rare feature rate is high at both points, and we report it as a usage statistic, following the same convention as the Qwen3\.5\-9B 32×\\times,k=256k=256row\.

### E\.3R1\-Distilled Comparison Rows

R1\-Distill\-Qwen\-7B is included as a post\-training comparison row for rowEV and decision fidelity\. It is a Qwen\-derived 7B readout with an untiedlm\_headand no post\-readout softcap, which gives a clean comparison under the same transferred TopK recipe family at the strict budget and high fidelity dictionary settings in Table[14](https://arxiv.org/html/2609.01936#A5.T14)\.

The 32×\\times,k=256k=256R1 row preserves high row reconstruction, within0\.0440\.044rowEV of the matched Ministral point, while top\-1 and KL separate from the base\-model comparison\. This gives a third pattern on the rowEV vs\. fidelity axis, since row reconstruction and decision\-level readout replacement are distinct evaluation quantities\.

R1\-Distill\-Llama\-8B adds a second reasoning\-distilled checkpoint from a non\-Qwen base family at both dictionary settings\. At the strict budget, its rowEV is close to the Ministral strict budget row \(0\.7960\.796vs\.0\.8060\.806\), with different top\-1/KL behavior \(0\.725/0\.5360\.725/0\.536vs\.0\.885/0\.1300\.885/0\.130\)\. At the fidelity setting row reconstruction matches the Ministral fidelity row to three decimals \(0\.8880\.888vs\.0\.8880\.888\), while top\-1/KL remain separated \(0\.754/0\.4340\.754/0\.434vs\.0\.904/0\.0870\.904/0\.087\)\. The pattern is consistent across both base families \(Qwen2\.5\-Math\-derived\([Yang et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib76)\)and Llama\-3\.1\-derived\([Llama Team, 2024](https://arxiv.org/html/2609.01936#bib.bib43)\)\), in that reasoning\-distilled readouts can preserve row reconstruction while requiring separate decision\-fidelity measurement\. Appendix[E\.4](https://arxiv.org/html/2609.01936#A5.SS4)reports the row\-norm tail analysis that helps explain this separation\.

#### Attribution of the post\-training effect\.

The R1\-distilled checkpoints differ from the base\-model rows in architecture, tokenizer, and pretraining lineage as well as post\-training\. The repeated pattern across two distillation lineages makes post\-training a plausible contributor, and a matched base\-against\-distill ablation within one architecture would further isolate the effect\.

#### R1 contrastive score reconstruction\.

We also ran the held\-out contrastive score\-reconstruction check on the two R1\-distilled checkpoints\. Table[15](https://arxiv.org/html/2609.01936#A5.T15)reports these results using the same bank version as the five\-model run\. Sign agreement remains close to the five\-model baseline, and the fraction with relative error below one half is comparable in the fidelity setting and lower under the stricter setting\.

Table 15:Held\-out contrastive score\-reconstruction metrics for the R1\-distilled checkpoints under the same contrast\-bank version as the five\-model run\. “sign” is contrastive sign agreement, “median resid\.” the median relative residual\|Δ​margin\|/\|exact margin\|\|\\Delta\\text\{margin\}\|/\|\\text\{exact margin\}\|, and “resid\.<0\.5<0\.5” the fraction of held\-out contrasts with relative residual below0\.50\.5\.ModelSetting\(cfg\)signmedianresid\.resid\.<0\.5<0\.5Ministral3\-8B\-Basefidelity\(32×\\times,k=256k=256\)0\.9490\.9490\.0790\.0790\.8310\.831Ministral3\-8B\-Basestrict\(16×\\times,k=128k=128\)0\.9470\.9470\.1020\.1020\.8120\.812R1\-DistillLlama\-8Bfidelity\(32×\\times,k=256k=256\)0\.9260\.9260\.1460\.1460\.7820\.782R1\-DistillLlama\-8Bstrict\(16×\\times,k=128k=128\)0\.9070\.9070\.1930\.1930\.7520\.752R1\-DistillQwen\-7Bfidelity\(32×\\times,k=256k=256\)0\.9210\.9210\.1630\.1630\.7380\.738R1\-DistillQwen\-7Bstrict\(16×\\times,k=128k=128\)0\.8900\.8900\.2330\.2330\.7000\.7005\-model baseline0\.9340\.934–0\.7730\.773

### E\.4Readout Row\-Norm Tail Analysis

The TopK recipe operates on row\-centered, row\-normalizedWUW\_\{U\}rows, which treats all rows uniformly inL2L\_\{2\}\. When the underlying row\-norm distribution is heavy\-tailed, however, a small per\-row reconstruction error is amplified disproportionately on the high\-norm rows, and those are the rows that disproportionately affect top\-1 / KL after the softmax\. We therefore report row\-norm summary statistics for the base\-model and reasoning\-distilled checkpoints as a usage analysis for the separation between rowEV and fidelity observed in Appendix[E\.3](https://arxiv.org/html/2609.01936#A5.SS3)\. Table[16](https://arxiv.org/html/2609.01936#A5.T16)reports those statistics\.

Table 16:WUW\_\{U\}row\-norm statistics for the base\-model and reasoning\-distilled checkpoints, computed on the readout extraction artifacts used to train each SAE\. “CoV” is the coefficient of variationσ/μ\\sigma/\\mu\. “max/median” summarizes the heavy\-tail end\.Modelminp50maxmeanCoVmax/medQwen3\.5\-9B \(base\)0\.3790\.3790\.9800\.9801\.5451\.5450\.9830\.9830\.1200\.1201\.581\.58Ministral\-3\-8B \(base\)0\.2500\.2500\.4470\.4470\.6840\.6840\.4490\.4490\.1160\.1161\.531\.53R1\-Distill\-Qwen\-7B \(sft\)0\.5730\.5731\.0541\.0541\.7561\.7561\.0291\.0290\.1590\.1591\.671\.67R1\-Distill\-Llama\-8B \(sft\)0\.4240\.4240\.8970\.8971\.7871\.7870\.9040\.9040\.1390\.1391\.991\.99

Two effects appear\. First, the upper tail is heavier on the distilled checkpoints, with max/median at1\.51\.5–1\.61\.6for the base\-model rows and1\.71\.7–2\.02\.0for the distilled rows, R1\-Distill\-Llama\-8B the most extreme\. Second, the lower tail is compressed, since minimum row norms rise from0\.250\.25–0\.380\.38on the base\-model rows to0\.420\.42–0\.570\.57on the distilled rows\. In these comparison rows, the row\-norm distribution is wider for the reasoning\-distilled checkpoints than for the base\-model rows\. The coefficient of variation tracks this \(0\.120\.12on base models vs\.0\.140\.14–0\.160\.16on distilled models\)\.

The row\-centered\-and\-normalized SAE recipe operates on a heavier\-tailed norm distribution for the distilled rows, which is consistent with the observed pattern, where row reconstruction \(which is uniform inL2L\_\{2\}\) is preserved, while decision fidelity \(which is exponential in the logit error and dominated by the heaviest rows\) separates from rowEV\. This characterizes a second\-order property of these reasoning\-distilled comparison rows and motivates norm\-aware SAE recipes as a natural extension for those settings\.

## Appendix FReadout SAE Diagnostics and Reproducibility

This appendix collects secondary diagnostics, comparisons of alternative recipes, denominator controls, and reproducibility details for the selected readout SAEs\.

#### Artifact availability\.

All code, with per\-model training and evaluation recipes, is released at[github\.com/hematteo/sparse\-readout\-prism](https://github.com/hematteo/sparse-readout-prism)\(MIT\)\. The selected readout SAE checkpoints for all eight models \(both thek=128k\{=\}128andk=256k\{=\}256recipes per model\) are released at[hematteo/sparse\-readout\-prism](https://huggingface.co/hematteo/sparse-readout-prism)\. The contrast banks, both the diagnostic bank and the bank derived from benchmarks, and the blinded label\-audit protocol accompany the code release\.

### F\.1Additional Metric Analyses

These analyses ask whether the remaining error comes from coefficient fitting, support selection, low\-rank structure, or a mismatch between reconstruction and readout metrics\.

#### Support selection\.

On the Qwen3\.5\-9BD=32768D=32768,k=128k=128recipe, the trained encoder reaches rowEV0\.6210\.621\. Nonnegative least\-squares\([Lawson and Hanson, 1974](https://arxiv.org/html/2609.01936#bib.bib38)\)on the encoder’s own support reaches0\.6250\.625, nonnegative OMP\([Pati et al\., 1993](https://arxiv.org/html/2609.01936#bib.bib58)\)0\.6470\.647, and signed\-OMP0\.6550\.655\. The coefficient gap is therefore small \(about0\.0040\.004\), while support\-selection headroom within the same coefficient class is about0\.0260\.026\. Dense rank\-128 reconstruction reaches only0\.1010\.101, far below the sparse dictionary, ruling out low\-rank PCA structure\([Jolliffe, 2002](https://arxiv.org/html/2609.01936#bib.bib34)\)\. Figure[12](https://arxiv.org/html/2609.01936#A6.F12)shows both the strict budget recipe and the high fidelity 32×\\times,k=256k=256run\. The right\-hand table labels the latter OMP numbers as sparse\-reconstruction ceilings, while the deployed encoder’s metrics are the trained\-encoder rows\.

32×\\times,k=256k=256ceiling comparison

QuantityrowEV95% CITrained encoder0\.8580\.858\[0\.856,0\.859\]\[0\.856,0\.859\]Signed LS support0\.8640\.864\[0\.863,0\.866\]\[0\.863,0\.866\]Signed\-OMP proxy0\.9020\.902\[0\.900,0\.903\]\[0\.900,0\.903\]

Figure 12:Support\-selection analyses for the Qwen3\.5\-9B readout SAE\.The left panel gives the strict budgetk=128k=128recipe and the high fidelity 32×\\times,k=256k=256capacity run, and the right panel the 32×\\times,k=256k=256EV\-ceiling numbers with 95% bootstrap CIs\. OMP \(orthogonal matching pursuit\) is an achieved sparse\-reconstruction comparison\.For the 32×\\times,k=256k=256capacity run, coefficient refitting contributes\+0\.006\+0\.006rowEV, while support selection contributes\+0\.038\+0\.038\. The encoder reaches95\.1%95\.1\\%of the signed\-OMP proxy, similar to theD=32768D=32768,k=128k=128ratio\. Dense rank\-256 reconstruction is0\.1550\.155, so thek=256k=256sparse code is about5\.9×5\.9\\timesbetter by unexplained\-variance reduction than the same\-rank dense reference, since\(1−0\.155\)/\(1−0\.858\)≈5\.9\(1\-0\.155\)/\(1\-0\.858\)\\approx 5\.9\.

#### Small\-model support\-selection analysis\.

We repeat the analysis on the 16×\\times,k=128k=128small\-model transfer 5k checkpoints to isolate the source of the remaining reconstruction gap with the learned dictionary fixed, and the converged transfer metrics are reported in Appendix[E\.1](https://arxiv.org/html/2609.01936#A5.SS1)\. Across all four models, refitting coefficients on the encoder’s own support gives a much smaller gain than changing the support, so support selection is the dominant remaining bottleneck on the recipe side\. The signed\-OMP gaps are\+0\.158\+0\.158for Qwen3\.5\-0\.8B,\+0\.099\+0\.099for Qwen3\.5\-2B,\+0\.182\+0\.182for Gemma\-4\-E2B, and\+0\.125\+0\.125for Gemma\-4\-E4B\.

#### Metric mismatch\.

The capacity sweep motivates selecting readout SAEs with metrics at both the row level and the readout level\. TheD=65536D=65536,k=256k=256model truncated tok=128k=128has higher rowEV than the fixed\-k=128k=128recipe \(0\.695\>0\.6210\.695\>0\.621\), while top\-1/KL favor the fixed\-k=128k=128recipe \(0\.721/0\.4690\.721/0\.469vs\.0\.846/0\.2960\.846/0\.296\)\. TheD=131072D=131072truncation shows the same metric separation, with rowEV/top\-1/KL at0\.816/0\.738/0\.4250\.816/0\.738/0\.425\. These truncated comparisons motivate the selection rule, which uses rowEV, top\-1, KL, reconstruction\-error metrics, and usage statistics jointly\.

### F\.2Alternatives and Negative Controls

#### Alternative recipes and denominator controls\.

Matched comparisons rule out the non\-selected recipes\. AtD=32768D=32768/5k, plain TopK exceeded the matryoshka variants in rowEV\([Bussmann et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib9)\)\(0\.5670\.567vs\.0\.559/0\.5540\.559/0\.554\) at lower cost, and theD=16384D=16384fixed\-k=128k=128cell trailedD=32768D=32768by about0\.060\.06rowEV\. Calibrated BatchTopK\([Bussmann et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib8)\)reached rowEV0\.5310\.531on the matched frontier cell\. Qwen JumpReLU\([Rajamanoharan et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib65)\)missed the intended targetL0=256L\_\{0\}=256\(achieved about3838–6868\), and step\- or sparsity\-matched TopK controls had higher rowEV \(0\.420\.42–0\.440\.44vs\.0\.260\.26–0\.310\.31\), with mixed logit metrics\. Finally, tokenizer and non\-token row audits support the denominator choice, since structural non\-token rows were5\.4%5\.4\\%of rows and excluding non\-token readout rows changed rowEV by\+0\.0016\+0\.0016\.

### F\.3Reproducibility and Selection Details

The main experiments report exact and reconstructed scores, sign agreement, relative error, and null/reference baselines for interpreted scores\. Appendices[B](https://arxiv.org/html/2609.01936#A2)–[G](https://arxiv.org/html/2609.01936#A7)specify models, dictionary settings, seeds, row sampling, selection order, tokenization filters, score construction, and denominator controls\. The remaining risks are computational cost, checkpoint and tokenizer versioning, and tokenizer\-specific filtering for single\-token scores\. Local decompositions include explicit reconstruction errors, and we restrict qualitative interpretations to sufficiently reconstructed scores\.

#### Artifact contents\.

The artifact bundle has four components\. First, the code artifact includes the readout\-SAE training scripts, hidden\-state capture and evaluation scripts, baseline implementations, and figure/table reproduction scripts\. Second, the dictionary artifact includes the reported readout\-SAE checkpoints when permitted by upstream model licenses, covering decoder directions, sparse row codes, shared offsets, preprocessing metadata, configuration files, seeds, and model checkpoint revision identifiers\. For any checkpoint that cannot be redistributed directly, we provide the exact recipe needed to regenerate it from the public model weights\. Third, the prompt artifact includes the prompt templates and filled prompts used for reconstruction of selected scores, qualitative displays, feature\-resolved DLA examples, and the constrained lexical\-edit probes\. Fourth, the score\-set artifact includes token ids or token groups, row\-combination coefficientsα\\alpha, split/base\-case identifiers, tokenization filters, bootstrap clusters, and the exact score\-family metadata needed to recompute the reported sign and reconstruction metrics\. We do not redistribute original model checkpoints or raw C4 text, and C4\-based evaluations are reproduced from public dataset/model identifiers and the artifact sampling, filtering, and scoring code\.

Dictionary selection used row reconstruction, replacement\-logit metrics, and feature usage statistics, and the logit difference contrast set enters only the reconstruction checks\. Most reported dictionary settings use seed00, with the Qwen3\.5\-0\.8B 32×\\times,k=256k=256three\-seed window providing the seed\-variation snapshot above\. For Qwen3\.5\-9B, tokenizer\-free denominator controls replace tokenizer\-aligned text\-token\-only EV and frequency\-stratified fidelity, since the tokenizer row count did not cleanly align with theWUW\_\{U\}rows and no vocab\-aligned unigram table was available\. The tokenizer\-free non\-token row audit is therefore the denominator\-control result for that model\. Gemma rowEV is measured on the linear pre\-softcap unembedding rows and is cap\-independent, while Gemma top\-1/KL are evaluated under the model’s30tanh\(⋅/30\)30\\tanh\(\\cdot/30\)final\_logit\_softcapping\. Since the cap is strictly monotone, top\-1 is cap\-independent and the cap affects only KL\. The selected operating points and reported claims come from the Qwen/Gemma sweeps and the targeted Ministral/R1 comparison rows above, and earlier Pythia\-160M architecture and continuity sweeps\([Biderman et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib4)\)are design\-history records\. Exact replication depends on using the same model checkpoint revisions, tokenizers, final\-normalization and softcap code paths, score filters, and single\-token eligibility rules\. For this reason, every interpreted local score decomposition reports the exact score, reconstructed score, reconstruction error, and, for signed contrasts, sign agreement\.

#### Training and evaluation recipe\.

All reported readout SAEs are TopK sparse autoencoders trained on the model’s final readout rows\. The reported Qwen/Gemma recipe uses row\-seeded initialization,hybrid\_50freq\_50uniformrow sampling, a warmup\-cosine learning\-rate schedule from10−310^\{\-3\}to10−410^\{\-4\}, and seed00unless stated otherwise\. The Qwen3\.5\-9B recipe\-selection run usesD=32768D=32768, nativek=128k=128, a delayed prism\-loss ramp to10−310^\{\-3\}after70%70\\%of training, and 20k optimization steps, while the capacity and transfer runs keep the same recipe family while varyingDDand nativekkas reported in Tables[5](https://arxiv.org/html/2609.01936#A3.T5),[9](https://arxiv.org/html/2609.01936#A4.T9), and[11](https://arxiv.org/html/2609.01936#A5.T11)\. Hidden\-state evaluation uses the final readout after the relevant final normalization and, for Gemma, the model’s monotone final\-logit softcap for top\-1/KL evaluation\. The reported metrics are rowEV, top\-1 agreement, KL in bits, feature usage, and the reconstruction error and sign agreement for selected scores defined above\.

#### Compute and software\.

Readout\-SAE training and the headline evaluations ran on a single NVIDIA A40 per job on an academic Slurm cluster\. A 24 GB\-class GPU suffices for inference and evaluation, while training the larger readouts \(e\.g\. Qwen3\.5\-9B\) needs roughly 40 GB\. One readout SAE costs about 6–12 A40 GPU\-hours depending on vocabulary size and dictionary width \(the Ministral\-3\-8B131072×4096131072\\times 4096head takes 12\.2 h at32×32\\times,k=256k=256, and 6\.3 h at16×16\\times,k=128k=128, for 20k steps\), and the three\-model constrained\-edit evaluation totals about 2\.5 GPU\-hours\. Fidelity evaluations of selected scores for the three late\-added models ran on a single rented cloud GPU, and some figure and analysis passes ran on Apple Silicon \(PyTorch MPS, bfloat16\)\. Experiments use Python 3\.11–3\.12 with pinned dependencies, includingtorch2\.7\.1 \(CUDA 11\.8\) on the cluster andtorch2\.12\.0 on Apple Silicon,transformers5\.9\.0,numpy2\.4\.6,datasets4\.8\.5, andsafetensors0\.7\.0\. The released code includes the full lockfile\.

## Appendix GFidelity Checks and Controls

This appendix collects the checks and controls that support interpreting and reproducing the local score decompositions, covering tokenization and filtering rules, row\-norm and frequency checks, split construction, dictionary settings, and aggregate metrics\. The decomposition definitions themselves are stated in Appendix[A](https://arxiv.org/html/2609.01936#A1)\.

### G\.1Controls

Table[17](https://arxiv.org/html/2609.01936#A7.T17)groups the measurements used to read a local score decomposition\. Aggregate outcomes for these checks appear in the quantitative summaries below\.

Table 17:Checks and controls supporting local score decompositions\.The checks verify that a displayed local score decomposition tracks the intended readout score and control for prompt, tokenization, and SAE artifacts\. The evaluation setup specifies the token choices, coefficients of the readout directions, split assignment, and tokenization rules used for the aggregate metrics\.Check or controlWhat it testsHow it is specifiedReconstruction error and sign agreementFeature bars are reported with relative reconstruction error and, for contrasts, the signs of both the exact and reconstructed scores\.Reported local score decompositions include exact score, reconstructed score, reconstruction error, andρ0\.5\\rho\_\{0\.5\}\.Swapped and matched contrastsA/B swaps flip signs consistently, and matched negative\-control contrasts underperform the intended contrast\.Evaluation sets specify paired A/B contrasts and matched negative controls before aggregate evaluation\.Template and context variantsPrompt rewordings and context variants are clustered before aggregation, so correlated variants contribute through their base case\.Rows are clustered by score or base\-case id before aggregate evaluation\.Token\-level auditsSingle\-token claims can be brittle under token length, frequency, whitespace, capitalization, row norm, special\-token status, or tokenization collisions\.Tokenization filters and row\-norm/frequency controls are applied before aggregate evaluation, and ambiguous cases are handled through feature\-audit flags or explicit group contrasts\.Null and reference baselinesLearned token\-to\-feature support is compared against shuffled support, random support, dense PCA, and lexical nearest row references\.Baselines use the same score set and the same reconstruction error and sign metrics\.
### G\.2Metric Definitions and Fidelity Results

#### Metric definitions for[Table1](https://arxiv.org/html/2609.01936#S4.T1)\.

Dictionary width is the cfg multiplier times the hidden dimension \(width selection in Appendices[C](https://arxiv.org/html/2609.01936#A3)–[D](https://arxiv.org/html/2609.01936#A4)\)\. rowEV \(row\-centered explained variance\) is the fraction of variance in row\-centered unembedding rows captured by the sparse reconstruction\. top\-1 is the fraction of held\-out decoded states where SRP and the dense LM head agree on the argmax token\. KL is the median KL divergence between dense and SRP\-reconstructed next\-token distributions\. Sign is the fraction of contrasts where the SRP\-reconstructed margin shares sign with the exact margin\. Replacement uses held\-out decoded states, while score metrics use the main held\-out logit difference set at each model’sk=256k=256operating point\. Med\.ρ0\.5\\rho\_\{0\.5\}andρ0\.5<0\.5\\rho\_\{0\.5\}<0\.5use the flooredρ0\.5\\rho\_\{0\.5\}of §[3\.4](https://arxiv.org/html/2609.01936#S3.SS4)\. The main text reports top\-1 and the baseline comparison, while rowEV and KL are in[Table5](https://arxiv.org/html/2609.01936#A3.T5), and the per\-score columns in[Table20](https://arxiv.org/html/2609.01936#A7.T20)\. The main\-text table lists the six softcap\-free readouts, and the two Gemma\-4 readouts are reported at the same operating point in[Table5](https://arxiv.org/html/2609.01936#A3.T5), with their per\-score reconstruction in[Table20](https://arxiv.org/html/2609.01936#A7.T20)and their null controls in[Table19](https://arxiv.org/html/2609.01936#A7.T19)\. Gemma\-4\-E4B reaches top\-1 0\.736, while its median KL of 1\.22 bits lies well above the six softcap\-free readouts, whose largest value is 0\.489 bits\.

Fidelity on selected scores is the central quantitative check\. Across the eightk=256k=256readout SAEs \(32×32\\timeswidth\), the sparse reconstruction preserves contrast sign for0\.9140\.914–0\.9580\.958of selected readout scores, with medianρ0\.5=0\.068\\rho\_\{0\.5\}=0\.068–0\.2550\.255\([Table20](https://arxiv.org/html/2609.01936#A7.T20), with cluster\-bootstrap 95% intervals\)\. The broader contrast set over all rows supplies a supporting diagnostic\. Aggregating both held\-out halves, the Qwen3\.5\-0\.8B/2B/9B settings give sign agreement0\.9400\.940–0\.9510\.951, medianρ0\.5=0\.091\\rho\_\{0\.5\}=0\.091–0\.1070\.107, andρ0\.5<0\.5\\rho\_\{0\.5\}<0\.5fractions0\.7900\.790–0\.8180\.818\. These per\-score figures cover the high fidelity subset of this evaluation, which is why the Qwen sign and reconstruction error ranges are tighter than the full set’s\. The Qwen3\.5\-2B reference panel below evaluates every method on that model’s 1,348\-row full logit difference set under the baseline harness, and the redesigned six\-model comparison behind the main\-text baseline table is in Appendix[G\.3](https://arxiv.org/html/2609.01936#A7.SS3)\. Many C4 positions are local near\-ties, which lowers headline numbers, yet Qwen still reaches0\.8220\.822–0\.8380\.838top\-1 agreement and0\.6340\.634–0\.6670\.667preservation of the top\-1–top\-2 ordering at relative error0\.50\.5on 10,000 neutral C4 continuations per model\([Raffel et al\., 2020](https://arxiv.org/html/2609.01936#bib.bib64);[Dodge et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib14)\)\. This appendix writesmexactm\_\{\\mathrm\{exact\}\}andmreconm\_\{\\mathrm\{recon\}\}for the exact and reconstructed margins, the quantitiessexacts\_\{\\mathrm\{exact\}\}andsrecons\_\{\\mathrm\{recon\}\}of §[3\.2](https://arxiv.org/html/2609.01936#S3.SS2)evaluated on the logit difference bank\. For confident Qwen margins,\|mexact\|≥2\|m\_\{\\mathrm\{exact\}\}\|\\geq 2, C4 top\-1 agreement rises to0\.970\.97–1\.001\.00and top\-1–top\-2 ordering preservation to0\.840\.84–0\.960\.96\. Additional model\-family rows use the same metric suite, separating row reconstruction, replacement fidelity, and reconstruction of selected scores\. Null and reference baseline comparisons preserve the same ordering, since shuffled and random support approach chance sign agreement and dense PCA underperforms Sparse Readout Prism at comparable compactness\. Cluster\-bootstrap 95% intervals on the Qwen3\.5\-2B panel under the baseline harness \(848 base cases, 400 resamples, unflooredρ0\\rho\_\{0\}identically for all methods\) give SRP coverage0\.7740\.774\(0\.7550\.755–0\.7920\.792\) and medianρ0=0\.081\\rho\_\{0\}=0\.081\(0\.0710\.071–0\.0920\.092\), against nearest row ridge at0\.6220\.622\(0\.6030\.603–0\.6430\.643\) and0\.2600\.260\(0\.2420\.242–0\.2950\.295\)\. PCA\-1024 reaches0\.5390\.539\(0\.5140\.514–0\.5630\.563\) using a median of 337 components per score, while PCA\-256, matched to SRP’s realized compactness \(a median of 78 components carrying80%80\\%of absolute contribution mass against SRP’s 76 features\), drops to0\.2720\.272\(0\.2470\.247–0\.2960\.296\)\. The redesigned harness of Appendix[G\.3](https://arxiv.org/html/2609.01936#A7.SS3)gives 0\.780 for SRP on this model, the value in[Table1](https://arxiv.org/html/2609.01936#S4.T1)\.[Table19](https://arxiv.org/html/2609.01936#A7.T19)repeats the two null controls on every Qwen/Gemma readout at the operating point that prioritizes fidelity\. Each model–null cell lowers coverage by0\.580\.58–0\.710\.71and sign agreement to0\.380\.38–0\.520\.52, so the learned row–code assignment carries the reconstruction on every evaluated model, including the lower\-fidelity Gemma rows\. Figure[16](https://arxiv.org/html/2609.01936#A7.F16)reports the same score set and metrics as the Sparse Readout Prism row\. Tokenization filters, row\-norm controls, frequency controls, KL, sign agreement, and relative reconstruction error use the same definitions across the score\-group and baseline comparisons\.

Fidelity also varies by family of selected scores\.[Table18](https://arxiv.org/html/2609.01936#A7.T18)breaks the maink=256k=256comparison point down by family, ordered by how close the competitor sits to the selected token, and the main text reads the digest for each family from this table\.

Table 18:Score and contrast groups at the maink=256k=256setting\.Rows group selected readout score families at thek=256k=256fidelity setting\. Row counts aggregate evaluated model–case cells across the five Qwen3\.5 and Gemma\-4 readout SAEs \(excluding Ministral and the R1\-distilled readouts\), after tokenization filters\. Medianρ0\.5\\rho\_\{0\.5\}is the floored variant of §[3\.4](https://arxiv.org/html/2609.01936#S3.SS4)\. Source banks and per\-family definitions of the score objects are in Appendix[G\.7](https://arxiv.org/html/2609.01936#A7.SS7)\.GroupSelected scorecasesrowssign agreementmedianρ0\.5\\rho\_\{0\.5\}Vocabulary\-mean contrasttop\-1 vs vocabulary mean30115051\.0000\.020Reference\-row contrasttop\-1 vs row\-norm distractor30115051\.0000\.048Logit differencesource vs prior823920\.9640\.172Logit differenceaction/tool alternatives964800\.9060\.283Group contrastabstention family844200\.8880\.198Logit differencecurrent vs stale924600\.8610\.369Top competitortop\-1 vs sampled competitor30115050\.9930\.113Top competitortop\-1 vs top5 competitor30115050\.9400\.215Top competitortop\-1 vs top2 competitor30115050\.8240\.342Table 19:Cross\-model null collapse\.Coverage, meaning sign agreement together withρ0<0\.5\\rho\_\{0\}<0\.5, with sign agreement in parentheses, for SRP and the two null controls on each model’s full logit difference set at the operating point that prioritizes fidelity \(k=256k=256,32×32\\timeswidth\), under the baseline harness’s unflooredρ0\\rho\_\{0\}\. The Qwen3\.5\-2B SRP row is computed on the same banks as the main\-text baseline table, whose redesigned harness gives 0\.780 for SRP on this model \(Appendix[G\.3](https://arxiv.org/html/2609.01936#A7.SS3)\)\. Anchor sign agreement on this full set sits below the per\-model values of[Table20](https://arxiv.org/html/2609.01936#A7.T20)\(the main\-text0\.910\.91sign floor refers to that evaluation\)\.ModelSRPShuffled codesRandom supportQwen3\.5\-0\.8B0\.752 \(0\.926\)0\.088 \(0\.519\)0\.085 \(0\.484\)Qwen3\.5\-2B0\.774 \(0\.941\)0\.081 \(0\.502\)0\.069 \(0\.407\)Qwen3\.5\-9B0\.784 \(0\.927\)0\.094 \(0\.467\)0\.079 \(0\.424\)Gemma\-4\-E2B0\.665 \(0\.890\)0\.085 \(0\.484\)0\.087 \(0\.377\)Gemma\-4\-E4B0\.701 \(0\.900\)0\.095 \(0\.476\)0\.072 \(0\.438\)Table 20:Reconstruction of selected scores with cluster\-bootstrap intervals\.Per\-model sign agreement, medianρ0\.5\\rho\_\{0\.5\}andρ0\.5<0\.5\\rho\_\{0\.5\}<0\.5on the held\-out logit difference set, with 95% intervals \(flooredρ0\.5\\rho\_\{0\.5\}, base\-case clusters resampled with replacement, 400 resamples\)\. Rows and base cases per model are 1,853/649 \(Qwen3\.5 and R1\-distilled\), 1,859/655 \(Gemma\-4\), 1,856/652 \(Ministral\)\.ModelSignMed\.ρ0\.5\\rho\_\{0\.5\}ρ0\.5<0\.5\\rho\_\{0\.5\}<0\.5Qwen3\.5\-0\.8B0\.950 \(0\.942, 0\.960\)0\.075 \(0\.069, 0\.084\)0\.891 \(0\.877, 0\.905\)Qwen3\.5\-2B0\.958 \(0\.947, 0\.967\)0\.071 \(0\.064, 0\.080\)0\.890 \(0\.875, 0\.904\)Qwen3\.5\-9B0\.949 \(0\.938, 0\.959\)0\.088 \(0\.079, 0\.096\)0\.878 \(0\.863, 0\.893\)Gemma\-4\-E2B0\.914 \(0\.900, 0\.928\)0\.255 \(0\.233, 0\.278\)0\.710 \(0\.687, 0\.730\)Gemma\-4\-E4B0\.940 \(0\.927, 0\.951\)0\.155 \(0\.141, 0\.165\)0\.807 \(0\.784, 0\.828\)Ministral\-3\-8B0\.949 \(0\.939, 0\.958\)0\.068 \(0\.061, 0\.078\)0\.888 \(0\.873, 0\.902\)R1\-Distill\-Qwen\-7B0\.921 \(0\.908, 0\.933\)0\.147 \(0\.134, 0\.160\)0\.787 \(0\.771, 0\.806\)R1\-Distill\-Llama\-8B0\.926 \(0\.913, 0\.938\)0\.134 \(0\.125, 0\.143\)0\.830 \(0\.814, 0\.846\)Figure 13:Held\-out logit difference fidelity\.Per\-bin reconstruction across five readout SAEs vs\. exact\-margin magnitude\|mexact\|\|m\_\{\\mathrm\{exact\}\}\|\.A\.Median relative error\|mexact−mrecon\|/\|mexact\|\|m\_\{\\mathrm\{exact\}\}\-m\_\{\\mathrm\{recon\}\}\|/\|m\_\{\\mathrm\{exact\}\}\|\(log\)\.B\.Decision\-flip rate\. Solid lines are Qwen3\.5\-0\.8B/2B/9B, dashed Gemma\-4\-E2B/E4B\. Both concentrate at small margins, with calibration in Figure[14](https://arxiv.org/html/2609.01936#A7.F14)\.Figure 14:Exact versus reconstructed logit differences\.Binned exact versus reconstructed margins on the five\-model held\-out contrast set, where the dashed line marks perfect reconstruction\. This is the global calibration companion to the reconstruction error and decision\-flip summaries of Figure[13](https://arxiv.org/html/2609.01936#A7.F13), and the decision boundary is better captured by margin\-binned views\.Figure 15:Margin size controls the reliability of signed contrasts\.Aggregate reconstruction error is tight on the held\-out contrast set, while sign agreement is lowest near zero exact margins\. Raising the minimum\|mexact\|\|m\_\{\\mathrm\{exact\}\}\|increases sign agreement while reducing retained coverage\. Feature displays therefore report reconstruction error and sign agreement alongside row reconstruction\.Figure 16:Baseline and reference comparisons for Qwen3\.5\-2B\.Sparse Readout Prism is compared with nearest row ridge, dense PCA references, shuffled sparse codes, and random support under the same reconstruction error and sign summaries\. The readout SAE preserves sign and keeps a larger low\-error tail than the null controls, while PCA references lose that low\-error tail as the compactness constraint tightens\.

### G\.3Direct\-Geometry Alternatives Across the Primary Readouts

The main\-text baseline table \([Table1](https://arxiv.org/html/2609.01936#S4.T1)\) summarizes a six\-model comparison against methods built directly fromWUW\_\{U\}geometry, and[Table21](https://arxiv.org/html/2609.01936#A7.T21)gives the full grid\. Each method reconstructs the same selected scores on the same banks and is scored identically \(sign agreement and unflooredρ0\\rho\_\{0\}, cluster\-bootstrap CIs by base case\)\. The methods are nearest row ridge \(ridge regression of each selected direction on its top\-128 nearest rows\), weighted kNN on the rows \(zero\-fit weighted average of the top\-128 nearest rows\), k\-means centroid dictionaries atD=65,536D=65\{,\}536andD=16,384D=16\{,\}384withk=256k=256active centroids per row \(width\- and sparsity\-matched to SRP\), hard cluster assignment \(each row explained by its singleD=65,536D=65\{,\}536cluster centroid\), and PCA\-256\. An earlier version of the clustering and kNN baselines over\-reconstructed through a full\-rank projection artifact and was redesigned\. The k\-means centroid dictionary at matchedDDandkkis the sharpest test, since it grants the alternative SRP’s exact capacity, and it still trails SRP’s coverage on every model\.

Table 21:Coverage for SRP and six alternatives built from direct row geometry\.Coverage is sign agreement together with unflooredρ0<0\.5\\rho\_\{0\}<0\.5, on each softcap\-free readout’s full logit difference bank \(∼\\sim1,350 rows over∼\\sim850 base\-case clusters\), with cluster bootstrap 95% CIs in brackets\. Bold marks the best method per model\.MethodQ\-0\.8BQ\-2BQ\-9BMin\-8BR1\-Q\-7BR1\-L\-8BSRP0\.754\[\.733, \.773\]0\.780\[\.759, \.798\]0\.812\[\.792, \.831\]0\.777\[\.759, \.795\]0\.716\[\.694, \.738\]0\.751\[\.728, \.772\]Nearest\-row ridge \(top\-128\)0\.665 \[\.645, \.686\]0\.625 \[\.606, \.644\]0\.639 \[\.618, \.658\]0\.648 \[\.629, \.665\]0\.579 \[\.560, \.598\]0\.600 \[\.580, \.620\]k\-means dict,D=65,536D\{=\}65\{,\}536,k=256k\{=\}2560\.585 \[\.566, \.603\]0\.596 \[\.575, \.614\]0\.619 \[\.598, \.639\]0\.628 \[\.610, \.648\]0\.479 \[\.457, \.503\]0\.624 \[\.606, \.642\]k\-means dict,D=16,384D\{=\}16\{,\}384,k=256k\{=\}2560\.570 \[\.550, \.591\]0\.561 \[\.540, \.581\]0\.602 \[\.580, \.623\]0\.604 \[\.585, \.622\]0\.312 \[\.284, \.339\]0\.537 \[\.516, \.558\]Hard cluster,D=65,536D\{=\}65\{,\}5360\.430 \[\.405, \.451\]0\.397 \[\.370, \.422\]0\.411 \[\.386, \.435\]0\.499 \[\.475, \.521\]0\.353 \[\.325, \.377\]0\.492 \[\.470, \.512\]Weighted row\-kNN \(top\-128\)0\.420 \[\.397, \.442\]0\.331 \[\.308, \.353\]0\.403 \[\.377, \.426\]0\.480 \[\.456, \.504\]0\.163 \[\.143, \.184\]0\.257 \[\.234, \.279\]PCA\-2560\.462 \[\.439, \.486\]0\.269 \[\.244, \.291\]0\.244 \[\.224, \.268\]0\.338 \[\.317, \.367\]0\.084 \[\.070, \.099\]0\.105 \[\.090, \.120\]
### G\.4Error Tails and Margin Stratification

A median can hide large failures in the tail, so we report the full error distribution for every method on the same banks and harness as Appendix[G\.3](https://arxiv.org/html/2609.01936#A7.SS3)\([Table22](https://arxiv.org/html/2609.01936#A7.T22)\)\. SRP has the lowest mean, 95th\-percentile, and maximum absolute error of the seven methods, so its coverage lead in[Table1](https://arxiv.org/html/2609.01936#S4.T1)does not depend on the choice of summary statistic\. Per model, SRP’s mean absolute error is0\.430\.43–1\.221\.22logits, its p951\.531\.53–3\.833\.83, and its maximum4\.44\.4–10\.410\.4, against1\.161\.16–2\.952\.95,2\.982\.98–8\.008\.00, and8\.58\.5–17\.417\.4for nearest row ridge, the strongest alternative\.

Table 22:Full error distribution for SRP and the six alternatives built from direct row geometry\.Absolute reconstruction error\|ϵ\|\|\\epsilon\|in logits and unfloored relative errorρ0\\rho\_\{0\}, pooled over the six softcap\-free readouts \(8,009 contrasts per method, under the same banks, dictionaries, and harness as[Table21](https://arxiv.org/html/2609.01936#A7.T21)\)\. Absolute errors are pooled across models of different logit scales, and per\-model ranges are given in the text\. Coverage \(sign agreement andρ0<0\.5\\rho\_\{0\}<0\.5\) repeats the pooled figure for reference\. Lower is better except sign agreement and coverage\.MethodMeanp95MaxMedianρ0\\rho\_\{0\}SignCoverageSRP0\.692\.3410\.40\.1010\.9260\.765Nearest\-row ridge \(top\-128\)1\.975\.8817\.40\.2890\.8580\.626k\-means dict,D=65,536D\{=\}65\{,\}536,k=256k\{=\}2562\.627\.3414\.60\.3640\.8390\.588k\-means dict,D=16,384D\{=\}16\{,\}384,k=256k\{=\}2563\.589\.8922\.50\.4600\.8320\.531Hard cluster,D=65,536D\{=\}65\{,\}5364\.4711\.5721\.40\.5710\.7940\.430Weighted row\-kNN \(top\-128\)5\.0913\.1226\.10\.6680\.7590\.343PCA\-2565\.7415\.6930\.70\.7540\.7780\.252Where the remaining tail sits decides whether it threatens interpretation\.[Table23](https://arxiv.org/html/2609.01936#A7.T23)stratifies the same contrasts by exact margin magnitude\. SRP’s absolute error is nearly flat across bins \(mean0\.640\.64–0\.710\.71logits\), so the relative\-error tail is a denominator effect\. It concentrates in the14%14\\%of contrasts with\|mexact\|<0\.5\|m\_\{\\mathrm\{exact\}\}\|<0\.5, where the two outputs are close to tied, and falls monotonically as the margin widens\. On the61%61\\%of contrasts with\|mexact\|≥2\|m\_\{\\mathrm\{exact\}\}\|\\geq 2, sign agreement is0\.9960\.996and0\.9660\.966of contrasts are covered\. Nearest\-row ridge shows the same shape with roughly double the absolute error in every bin \(1\.301\.30–2\.392\.39logits\) and0\.0570\.057coverage in the near\-tie bin against SRP’s0\.2270\.227\. The diagnostics of §[3\.4](https://arxiv.org/html/2609.01936#S3.SS4)make the near\-tie regime visible, so this tail is measured and reported\.[Figure15](https://arxiv.org/html/2609.01936#A7.F15)traces the resulting trade\-off, since raising the minimum\|mexact\|\|m\_\{\\mathrm\{exact\}\}\|buys sign agreement at the cost of retained coverage\.

Table 23:SRP error by exact\-margin magnitude\.The same 8,009 pooled contrasts as[Table22](https://arxiv.org/html/2609.01936#A7.T22), binned by\|mexact\|\|m\_\{\\mathrm\{exact\}\}\|\. Share is the fraction of contrasts in the bin, Mean and p95 are absolute errors in logits, and the last column is the coverage\.\|mexact\|\|m\_\{\\mathrm\{exact\}\}\|ShareMeanp95Med\.ρ0\\rho\_\{0\}Signρ0<0\.5\\rho\_\{0\}<0\.5<0\.5<0\.50\.140\.642\.151\.8640\.6870\.2270\.50\.5–110\.120\.672\.180\.5560\.8430\.45911–220\.140\.662\.170\.2560\.9310\.679≥2\\geq 20\.610\.712\.490\.0340\.9960\.966
### G\.5Nearest\-Row Reading of Selected Directions

The zero\-fitting reading of a selected readout directionqqranks vocabulary rows by cosine alignment withqqand inspects the top tokens\.[Table24](https://arxiv.org/html/2609.01936#A7.T24)applies this reading to two case\-study contrasts on Qwen3\.5\-2B, ranking row\-centered unembedding rows with the two contrast tokens themselves excluded, and raw\-cosine rankings behave the same\. Forwbug−winsectw\_\{\\texttt\{bug\}\}\-w\_\{\\texttt\{insect\}\}the listing is dominated by casing and morphological variants of the contrast words, the first sense\-bearing rows \(a malfunction row,debug,issue,glitch,flaw\) appear at ranks 13–17 at roughly half the leading alignment, and the orthographic neighborblogenters by rank 30\. Forwbug−werrorw\_\{\\texttt\{bug\}\}\-w\_\{\\texttt\{error\}\}the top 30 consists entirely of casing, sub\-word, and cross\-lingual variants of the two contrast words\. The nearest row reading recovers the contrast’s token family, while the sense structure that the SRP decomposition of the same scores separates into signed feature terms \([Figure18](https://arxiv.org/html/2609.01936#A12.F18)\) sits below its morphological band\.

Table 24:Nearest centered unembedding rows for two selected directions\.Qwen3\.5\-2B, top 12 rows by\|\|cosine\|\|with each direction, contrast tokens excluded\.□\\squaremarks a leading space, and bracketed entries gloss non\-Latin tokens\. For signed cosines, positive supports the first token of the contrast\.wbug−winsectw\_\{\\texttt\{bug\}\}\-w\_\{\\texttt\{insect\}\}wbug−werrorw\_\{\\texttt\{bug\}\}\-w\_\{\\texttt\{error\}\}RankTokencosTokencos1□\\squareBug\+0\.49\+0\.49□\\squareError−0\.53\-0\.532bug\+0\.46\+0\.46error−0\.50\-0\.503Bug\+0\.44\+0\.44□\\squareerrors−0\.49\-0\.494□\\squarebugs\+0\.42\+0\.42\_error−0\.47\-0\.475□\\squareBUG\+0\.34\+0\.34Bug\+0\.45\+0\.456\_bug\+0\.31\+0\.31\-error−0\.44\-0\.447BUG\+0\.31\+0\.31\[tab\]error−0\.44\-0\.448□\\squareinsects−0\.30\-0\.30Error−0\.44\-0\.449bugs\+0\.29\+0\.29□\\squareBug\+0\.42\+0\.4210□\\squarebuggy\+0\.29\+0\.29\[zh: numerical error\]−0\.42\-0\.4211□\\squareBugs\+0\.28\+0\.28bug\+0\.42\+0\.4212\[zh: insect\]−0\.24\-0\.24□\\squarebugs\+0\.40\+0\.40
### G\.6Operating\-Regime Cases

A faithful decomposition is a local reading of the readout side, tied to this dictionary and its budget of active features, since a different dictionary or budget could split the same contribution across other features\. Operating\-regime cases define when a compact local score decomposition is evidence\-bearing\. Qualitative interpretation uses cases where the selected readout score is larger than the reconstruction error, the reconstructed contrast preserves sign, the exact model readout supports the intended side of the contrast, contribution mass is compact enough to summarize, and token\-boundary, special\-token, row\-norm, frequency, or tokenization\-collision effects do not dominate the contrast\. The remaining cases are retained as quantitative records under the same reporting rule\.

Each failing condition routes the reading to the part that still holds\. When the residual dominates, only the selected score itself is read\. When the reconstructed contrast flips sign, the exact contrast is used for direction alone\. When the exact readout does not support the intended target, the score is reselected\. When the contribution mass is too diffuse for a compact reading, the aggregated mass is reported in place of individual terms\. And when a single\-token contrast turns on token\-boundary, special\-token, row\-norm, frequency, or tokenization\-collision effects, the reading falls back to the safer token\-family contrast\. Each promoted display names the constraint it met and the reading that constraint leaves in place\. Where the residual dominates or terms repeatedly change sign, the score is reported without a feature\-level reading and retained in the record\. The Qwen3\.5\-2B readout used for the main\-paper displays sits well within the operating regime on every one of these measurements\.

### G\.7Selected\-Score Bank Provenance

The banks of selected scores define the analyzed object behind the headline fidelity numbers\. A row record contains the prompt or hidden\-state source, token ids or token\-family strings before tokenization, the score\-family label, tokenizer\-audit fields, and the exact and reconstructed scores used to computeρ0\.5\\rho\_\{0\.5\}\.[Table25](https://arxiv.org/html/2609.01936#A7.T25)lists the source banks\. The controlled A/B banks are deterministic synthetic probes that build benchmark\-like margin types, while the separate C4 bank is sampled from C4 validation and supports the model\-native replacement and frontier diagnostics only\.[Table26](https://arxiv.org/html/2609.01936#A7.T26)maps each main\-text score family \([Table18](https://arxiv.org/html/2609.01936#A7.T18)\) to its source bank and gives a one\-line definition of the score object, and[Table27](https://arxiv.org/html/2609.01936#A7.T27)shows representative A/B records\. A separate contrast bank derived from benchmarks builds answer\-vs\-distractor, abstention\-vs\-entity, label, and safe\-vs\-action readout contrasts from SQuAD2, HotpotQA, LegalBench \(ContractNLI\), and SecurityEval formats\. It tests reconstruction of fixed readout contrasts, with task accuracy out of scope, and its per\-format results are in Table[36](https://arxiv.org/html/2609.01936#A12.T36)\.

Table 25:Source banks for the checks on selected scores\.The main\-text score\-family table uses the model\-native, curated A/B, and case\-candidate banks at the operating point that prioritizes fidelity, while the C4 bank supports the separate model\-facing replacement diagnostic\.BankRecordsScore constructionProvenanceModel\-native prompts v2301 prompt recordsRun\-time top\-1 contrasts against the vocabulary mean, a row\-norm matched distractor, a sampled competitor, the rank\-5 token, and the rank\-2 token\.Hand\-written prompt set covering factual QA \(50\), instruction \(30\), paragraph \(40\), code \(30\), dialogue \(30\), retrieval\-style updates \(96\), and safety/tool prompts \(25\)\.Curated A/B v2320 prompt\-target recordsSingle\-tokenA−BA\-Bmargins, plus abstention\-family versus forced\-answer\-family margins\.Deterministic template expansion over 80 current\-vs\-stale, 80 source\-vs\-prior, 80 action/tool, and 80 abstention cases\.Case candidates v240 prompt\-target recordsThe same contrast types as the curated A/B bank, used for figure\-candidate and coverage rows when tokenization checks pass\.16 action/tool, 8 source\-vs\-prior, 12 current\-vs\-stale, and 4 abstention records\.C4 model\-native v310,000 prompt prefixesModel\-native replacement, top\-token, KL, and top\-1–top\-2 frontier checks\.Sampled fromallenai/c4English validation at pinned revision1588ec45\.\.\., seed 0, 110\-word prefixes after length and control\-character filters\.Table 26:How source banks map to the main\-text score\-family rows\.Cases count distinct base cases after tokenization filters at the operating point that prioritizes fidelity, and rows count evaluated model–case cells across the five\-model Qwen/Gemma suite\.Score familySource bankCasesRowsScore objecttop\-1 vs vocab meanModel\-native prompts v23011505Exact top token against the vocabulary mean row\.top\-1 vs row\-norm distractorModel\-native prompts v23011505Exact top token against a sampled row\-norm matched distractor\.source vs priorCurated A/B \+ case candidates82392Source\-supported answer against a plausible prior answer\.action/tool alternativesCurated A/B \+ case candidates96480Task\-appropriate action against a plausible wrong action\.abstention familyCurated A/B \+ case candidates84420Mean abstention\-token family against forced\-answer tokens\.current vs staleCurated A/B \+ case candidates92460Current or corrected value against an earlier stale value\.top\-1 vs sampled competitorModel\-native prompts v23011505Exact top token against a sampled lower\-ranked competitor\.top\-1 vs top5 competitorModel\-native prompts v23011505Exact top token against the rank\-5 token\.top\-1 vs top2 competitorModel\-native prompts v23011505Exact top token against the rank\-2 token\.Table 27:Representative A/B prompt records\.Representative rows from the synthetic controlled banks, used as controlled probes of readout margins\. The full bank is machine\-readable\.FamilyPrompt endingSpecified contrastcurrent vs staleInitial note: the password was blue\. Update: the password is now red\. The current password isredminusbluesource vs priorThe document says: ‘‘The meeting city is Paris\.’’ Ignore prior assumptions\. The meeting city isParisminusLyonaction/toolThe request is ambiguous and might damage files\. The appropriate next action is toaskminusrunabstention familyThe passage says Alice owns a key\. It does not say her favorite color\. Alice’s favorite color ismean\(unknown,Unknown,unclear,not\) minus forced\-color tokens

## Appendix HSense\-Labelled Evaluation

§[4\.5](https://arxiv.org/html/2609.01936#S4.SS5)reads individual scores for a fixed token under two contexts\. This appendix tests the same feature groups against labelled sense data and measures what the sparse account gives up against the full decoded state\.

#### Data and coverage\.

CoarseWSD\-20\([Loureiro et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib45)\)supplies 20 ambiguous words with coarse sense labels and a fixed train and test split, 33,566 contexts in total\. We keep the shipped splits and score the final position of a cloze prompt that asks for the ambiguous word\. All 20 words resolve to a single token under the Qwen3\.5 vocabularies\. Seven do not under the R1\-Llama\-8B vocabulary, so that model is evaluated on 13 words and 16,785 contexts and its column is not comparable item for item with the other two\. Row\-gate coverage is 1\.00 on the Qwen models and 0\.976 on R1\-Llama\-8B\. Sense distributions are skewed, so we report balanced accuracy throughout\. Unbalanced accuracy gives the same ordering and is released with the run artifacts\.

#### Protocol\.

For each word we select one group of eight features per sense on the training split by largest mean contribution difference between that sense and the others, then assign each held\-out context to the sense whose group carries the larger summed contribution\. Feature scales are standardized by training statistics alone, and no test label enters the selection\. The null permutes sense labels within the training split and repeats the whole selection, so it inherits the selection procedure\.

#### Sense alignment\.

[Table28](https://arxiv.org/html/2609.01936#A8.T28)gives the result\. Selected groups recover held\-out senses at 0\.90, 0\.92, and 0\.80 balanced accuracy against a permutation null near 0\.41\. Every word on Qwen3\.5\-2B, 19 of 20 on Qwen3\.5\-9B, and 12 of 13 on R1\-Llama\-8B exceed that null’s 95th percentile, and every word on all three models exceeds its majority baseline\. The finding does not depend on the account size\. Across sizes one, two, four, and eight the majority baseline is exceeded for every word on every model, and the count above the null’s 95th percentile moves by at most one word\.

The last two rows of the same table bound the result from above\. A nearest centroid probe on the full decoded state reaches 0\.96, 0\.97, and 0\.92, and the full 256\-dimensional account without any selection reaches 0\.95, 0\.97, and 0\.89\. The three rows form a sparsity ladder in which factorizing the row costs little\. Compressing to eight features costs a further 5 to 9 points against the full account, and 5 to 12 points against the full state\. That is what the account costs, and it buys a set of features a reader can name\.

This is an alignment result for the selected groups\. The comparison against directWUW\_\{U\}geometry is made on reconstruction and coverage in[Table1](https://arxiv.org/html/2609.01936#S4.T1)and on grouping recovery in §[4\.4](https://arxiv.org/html/2609.01936#S4.SS4), where the alternatives are matched by construction\.

#### A classifier framing\.

A different framing trains a per\-word classifier on the ten features that carry the largest absolute contribution, skipping the group selection\. Under that framing the unweighted projectionspi​\(h~ℓ\)p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)beat the signed contributions on all three models, at 0\.87 against 0\.84, 0\.90 against 0\.76, and 0\.73 against 0\.67\. Row\-coefficient shuffles and randomly drawn features sit close to the signed contributions\.

That ordering follows from the factorization\. Holding a vocabulary rowvvfixed freezes its coefficientszv,iz\_\{v,i\}and leaves onlypi​\(h~ℓ\)p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)free \(§[3\.2](https://arxiv.org/html/2609.01936#S3.SS2)\), so for a fixed word every weighted variant is a constant per\-coordinate rescaling of the same projections\. A per\-word supervised classifier gains nothing from that rescaling, and a shuffle of the coefficients preserves the projections it is meant to remove\. The signed contributions name which readout directions support a selected score\. They are not a sense representation, and this second framing measures the second quantity\. Sense structure in embedding geometry has been studied directly\([Arora et al\., 2018](https://arxiv.org/html/2609.01936#bib.bib2)\)\.

#### Scope\.

The evaluation covers one dataset, one sparsity setting per model, and one prompt template\. No behavioral filter is applied, and the model ranks the ambiguous word at median rank 174, 10, and 1131 on the three models, so a share of the scored contexts sits outside the operating regime of Appendix[G\.6](https://arxiv.org/html/2609.01936#A7.SS6)\. Filtering on target rank would change the reported values in an untested direction\.

Table 28:Sense alignment on CoarseWSD\-20\.Balanced accuracy on held\-out contexts for groups of eight features selected on the training split, against a label\-permutation null that repeats the selection, a majority baseline, and two references that use no selection\.Qwen3\.5\-2BQwen3\.5\-9BR1\-Llama\-8BWords202013Selected groups0\.8990\.9230\.798Permutation null0\.4080\.4070\.422Majority0\.4100\.4100\.429Words above null p9520/2019/2012/13Words above majority20/2020/2013/13Full account, no selection0\.9470\.9700\.892Hidden state0\.9580\.9710\.920

## Appendix IStability Across Independent Training Runs

This appendix documents the reproducibility evidence summarized in §[4\.4](https://arxiv.org/html/2609.01936#S4.SS4), together with a held\-out comparison of the grouping structure against row geometry that the main text does not report\. All experiments use Qwen3\.5\-2B with three dictionaries per width configuration \(32×32\\times/k=256k\{=\}256and16×16\\times/k=128k\{=\}128\), trained from scratch with independent initialization, data\-order, and sampling seeds\. Held\-out reconstruction is equal across seeds \(validation top\-1 agreement 0\.805–0\.848\), so differences below are not fidelity artifacts\. All statements in this appendix hold within a recipe, meaning seeds vary while width, sparsity, and preprocessing are fixed\. Variation across recipes is larger and is disclosed at the end\.

### I\.1Explanation\-Level Stability

The unit under test is the decomposition of a selected score\. For each of 63 curated contrasts \(126 signed side\-sets\), we compare the token sets that each seed’s decomposition places on each side of the contrast\.

Table 29:Cross\-seed stability of contrast explanations\.Qwen3\.5\-2B, three seeds per width, with means over seed pairs\. Same\-side Jaccard compares side token\-sets for the same contrast across seeds, and the cross\-contrast null pairs decompositions of unrelated contrasts through the identical pipeline\.Metric32×32\\times/k=256k\{=\}25616×16\\times/k=128k\{=\}128Same\-side Jaccard \(mean\)0\.2140\.240Cross\-side leakage0\.0030\.005Cross\-contrast null0\.0140\.018Contrasts above null p90100%100%Matched\-projection correlation0\.53–0\.540\.63–0\.66Matched decoder cosine\(used features\)0\.31–0\.340\.52–0\.55Individual decoder directions do not recur across seeds \(matched decoder cosines average∼\\sim0\.33 at32×32\\times\), consistent with the documented seed\-dependence of TopK dictionaries at scale\([Paulo and Belrose, 2026](https://arxiv.org/html/2609.01936#bib.bib59)\)\. Yet every tested contrast reproduces its explanation above the null’s 90th percentile, at∼\\sim15×\\timesthe null level and with negligible cross\-side leakage \([Table29](https://arxiv.org/html/2609.01936#A9.T29)\)\. The decomposition of a selected score is therefore the trustworthy unit of interpretation\.

### I\.2Feature\-Group Matching

A complementary question is whether the feature\-level token groups themselves have an equivalent in another seed’s dictionary, with indices re\-shuffled\. We sample 100 features per seed pair from the bank\-used set\. Each feature’s group is its top\-12 centered\-row token set, and the counterpart is the best\-Jaccard match over the full candidate dictionary\. The null passes 500 frequency\-matched pseudo\-groups through the identical best\-of\-DDsearch, pricing in the inflation of searching 65,536 candidates, and recall@3 admits a greedy union of at most three candidate features, testing whether instability is feature splitting\.

Table 30:Cross\-seed feature\-group matching\.Qwen3\.5\-2B, ranges over the three seed pairs per width\. The null passes frequency\-matched pseudo\-groups through the identical search \(best\-single Jaccard median 0\.059, p99≈\\approx0\.11, recall@3 0\.25\)\.Metric \(range over pairs\)32×32\\times16×16\\timesBest\-single Jaccard, median0\.32–0\.420\.37–0\.43Groups above null p9989–90%87–91%Strong equivalence \(J≥0\.5J\\geq 0\.5\)36–41%40–47%Recall@3, median0\.83–0\.900\.86–0\.88Recall@3 above null p9991–93%92%Matched decoder cosine,med / p900\.20–0\.26 / 0\.80–0\.890\.40–0\.42 / 0\.90–0\.94About 90% of used feature groups have an above\-chance counterpart in an independently trained dictionary, roughly 40% a near\-exact single match, and the median group is 83–90% covered by a union of at most three features \([Table30](https://arxiv.org/html/2609.01936#A9.T30)\), so token\-group structure survives retraining even though feature indices are reassigned, and the residual instability takes the form of feature splitting\. The decoder\-cosine distribution is bimodal, with a stable core recurring almost exactly while the rest recombine, and the narrower16×16\\timesdirections are more stable, consistent with[Paulo and Belrose \(2026\)](https://arxiv.org/html/2609.01936#bib.bib59)\. For the below\-null tail \(32/300 and 33/300 groups at the two widths\), a direction\-level check finds decoder counterparts at cosine≥0\.5\\geq 0\.5for 5/32 and 10/33\. A qualitative pass suggests most of the remainder resemble the audit’s mixed/ambiguous class, but we have not rubric\-scored that tail and do not quote it as an audited rate\. The strong\-equivalence and recall@3 thresholds were fixed for this analysis, with no sweep, so we report them descriptively alongside the threshold\-free medians\.

### I\.3Held\-Out Recovery of the Stable Grouping Core

The grouping comparison quoted in the main text asks which methods recover reproducible grouping structure, on identical targets\. For each held\-out seed, the core is the set of grouped tokens stable in both remaining seeds \(leave\-one\-out construction, all methods evaluated on identical cores, 95% contrast\-clustered bootstrap CIs\)\.

Table 31:Held\-out recovery of the stable core\.Qwen3\.5\-2B, showing recovery of the leave\-one\-out core by the held\-out SRP dictionary from the same recipe, weighted kNN on the rows, and hard clustering, on identical cores\. The held\-out SRP value \(≈\\approx0\.75\) is the operative anchor, and recall should be read against it\.Method32×32\\times16×16\\timesHeld\-out SRP\(same recipe\)0\.749 \[\.73, \.77\]0\.720 \[\.70, \.74\]Weighted row\-kNN0\.493 \[\.44, \.54\]0\.432 \[\.38, \.48\]Hard clustering0\.249 \[\.21, \.29\]0\.206 \[\.17, \.24\]SRP/kNN ratio1\.521\.67Cross\-contrast null0\.0140\.018Held\-out SRP recovers 72–75% of the stable core, while kNN on the rows recovers 43–49% and clustering 21–25%, with SRP/kNN ratio CIs excluding parity \([Table31](https://arxiv.org/html/2609.01936#A9.T31)\)\. Direct row geometry overlaps the grouping structure well above the null but misses most of the reproducible core that an independently trained dictionary from the same recipe recovers\.

### I\.4Cross\-Recipe Disclosure

The stability results above hold the training recipe fixed, while across recipes the agreement is weaker\. The released paper dictionary, trained under an earlier preprocessing recipe than the seed\-variation family, shares only 36% of the stable core and recovers 0\.444 of the leave\-one\-out cores, below kNN on the rows in that comparison\. Variation across recipes therefore exceeds seed variation within a recipe, and every stability claim in this paper is scoped to a fixed recipe\. Comparisons in this appendix use the seed\-variation family end\-to\-end, and no table mixes dictionary families\.

## Appendix JPredicted Versus Realized Local Readout\-Side Changes

This appendix documents the intervention\-validation protocol behind[Table2](https://arxiv.org/html/2609.01936#S4.T2)\.

#### Claim under test\.

For a selected contrast directionqαq\_\{\\alpha\}and decoded stateh~ℓ\\tilde\{h\}\_\{\\ell\}, SRP asserts that featureiicontributesci=βi​\(α\)​pi​\(h~ℓ\)c\_\{i\}=\\beta\_\{i\}\(\\alpha\)\\,p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)to the realized score\. If this attribution is correct, removing thedid\_\{i\}\-component of the decoded state should change the exact score by−ci\-c\_\{i\}\. We therefore ablateh~ℓ′=h~ℓ−\(h~ℓ⊤​di\)​di\\tilde\{h\}\_\{\\ell\}^\{\\prime\}=\\tilde\{h\}\_\{\\ell\}\-\(\\tilde\{h\}\_\{\\ell\}^\{\\top\}d\_\{i\}\)\\,d\_\{i\}for unit\-norm decoder directiondid\_\{i\}and measure the realized changesα​\(h~ℓ′\)−sα​\(h~ℓ\)s\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}^\{\\prime\}\)\-s\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\)with the dense LM head, so the prediction uses the factorization while the measurement does not\. The claim is falsifiable in three independent ways, since sparse\-code misattribution \(a contribution assigned to the wrong feature\), decoder non\-orthogonality \(overlapping features absorbing the ablated mass\), and the row residual \(score mass outside the fitted basis\) each break the predicted–realized agreement\. Random unit directions supply the control\.

#### Protocol\.

Each model contributes approximately 260 contrasts from the main bank, and for each we take the top\-10 features by absolute contribution plus 10 random\-direction controls, giving 1,210–2,050 prediction–realization pairs per model\. We reportr2r^\{2\}and the slope of realized change on predicted contribution, fit through the origin, over covered pairs \(§[3\.4](https://arxiv.org/html/2609.01936#S3.SS4)\), with bootstrap 95% CIs, and results over all pairs differ by less than 0\.02 inr2r^\{2\}throughout\. A harness self\-test on a residual\-free synthetic factorization recoversr2=1\.000r^\{2\}=1\.000\(slope 1\.04\) with zero\-scoring controls, so the pipeline itself does not manufacture agreement\.

#### Results and reading\.

All six softcap\-free readouts giver2=0\.83r^\{2\}=0\.83–0\.930\.93against random\-controlr2≤0\.06r^\{2\}\\leq 0\.06\([Table2](https://arxiv.org/html/2609.01936#S4.T2)\)\. Slopes are close to 1 on the 7–8B readouts \(0\.96–1\.13\) and rise to 1\.34–1\.69 on the Qwen3\.5\-0\.8B/2B/9B readouts, where realized changes exceed predicted magnitudes while preserving sign and rank\. A candidate explanation is feature interference, with overlapping decoder directions sharing ablated mass, but we have not isolated it and report slopes as measured\. The validated claim is local, since ablations act on the readout side of a realized forward pass \(linear inh~ℓ\\tilde\{h\}\_\{\\ell\}, no re\-forward\), so the result licenses the claim that feature contributions predict local changes on the readout side\. Circuit\-level necessity, sufficiency, and generation\-level effects call for separate interventions\([Conmy et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib11);[Geiger et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib23)\)\.

## Appendix KCross\-Lens Study Protocol, Controls, and Per\-Family Results

This appendix documents the corpus conditionality study of §[5](https://arxiv.org/html/2609.01936#S5)\.

#### Lenses\.

We use the reference implementation of the Jacobian lens\([Gurnee et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib28)\), which reads an intermediate hidden state by transporting it through an averaged Jacobian into the space consumed by the unembedding and decoding through the LM head\. On Qwen3\.5\-9B we fit two lenses that differ only in fitting corpus, 100 seeded English C4 prompts against 100 seeded Chinese C4 prompts, with identical seeds, positions, and procedure\. Transport is evaluated at layers\{2,5,7,10,12,14,17,19,21,24,26,29\}\\\{2,5,7,10,12,14,17,19,21,24,26,29\\\}of 31 at the final prompt position\. The cross\-lens comparison uses layers 21, 24, 26, and 29, where the two lenses’ token outputs can diverge, while both lenses converge at the final layer\. Composition is validated numerically, since the decoded statesh~ℓ=Tℓ​\(hℓ\)\\tilde\{h\}\_\{\\ell\}=T\_\{\\ell\}\(h\_\{\\ell\}\)reproduce the lens’s own output scores to within 0\.2%\.

#### SRP basis and comparison operation\.

Because the implemented lens feeds the unembedding directly, the score each lens reports is a selected readout score onh~ℓ\\tilde\{h\}\_\{\\ell\}, and SRP decomposes it in the shared basis fitted from the weights alone \(§[3\.3](https://arxiv.org/html/2609.01936#S3.SS3)\)\. The dictionary is an independently trainedk=128k\{=\}128\(seed 0\) readout SAE for Qwen3\.5\-9B, a different operating point from the32×32\\times/k=256k\{=\}256configuration of the main tables, disclosed here per the rule of Appendix[I](https://arxiv.org/html/2609.01936#A9)for comparisons across recipes\. No translation pairs, language labels, or fitting corpora enter the basis, which is fitted to the rows of the LM head only\. For each prompt, each lens’s reported score is decomposed at layers 21, 24, 26, and 29\. The dominant feature at a layer is the one whose signed contribution is largest in absolute value\. We count a prompt as agreement when the same feature id dominates under both lenses in at least two of the four layers\.

#### Prompts\.

Eighty evaluation prompts span seven families, namely Chinese antonym \(14\), English cloze \(10\), Chinese cloze \(18\), Chinese exemplar \(8\), Chinese factual recall \(10\), and translation in both directions \(10 each\)\. A further 12 control prompts \(6 digit, 6 proper noun\) have an expected surface form that is language\-invariant\. Two null comparisons price in loose matching\. Under the unrelated token null, the dominant feature of each reading is compared against the decomposition of an unrelated token under the same lens on the same state \(159 comparisons\)\. Under the shuffled pairing null, the two lenses’ readings are re\-paired across different prompts \(240 comparisons\)\.

Table 32:Cross\-lens agreement on the dominant feature by prompt family\.Agreement means the same SRP feature is dominant under the English and the Chinese lens in at least two of layers 21, 24, 26, and 29, with 95% binomial CIs\. “Lens\-only” is the surface baseline, counting prompts where the English lens reports top\-1 tokens in Latin script and the Chinese lens in CJK script \(modal script across the same four layers\), i\.e\. where token outputs alone suggest disagreement\. Control families \(below the rule\) are excluded from the 77/80 headline\.FamilyAgreement95% CILens\-only splitAntonym \(ZH\)14/14\[0\.78, 1\.00\]4/14Cloze \(EN\)10/10\[0\.72, 1\.00\]7/10Cloze \(ZH\)16/18\[0\.67, 0\.97\]6/18Exemplar \(ZH\)8/8\[0\.68, 1\.00\]7/8Factual recall \(ZH\)9/10\[0\.60, 0\.98\]9/10Translation EN→\\toZH10/10\[0\.72, 1\.00\]6/10Translation ZH→\\toEN10/10\[0\.72, 1\.00\]0/10All cross\-lens77/80\[0\.90, 0\.99\]—Digit controls5/6\[0\.44, 0\.97\]0/6Proper noun controls5/6\[0\.44, 0\.97\]0/6
#### Results\.

The same SRP feature is dominant under both lenses on 77 of 80 cross\-lens prompts \(96%, CI \[0\.90, 0\.99\]\), uniformly across families \([Table32](https://arxiv.org/html/2609.01936#A11.T32)\)\. Within a single lens, one feature carries both surface forms \(e\.g\. the English and Chinese realizations of the same concept\) on 61/80 \(English lens\) and 62/80 \(Chinese lens\) prompts\. Both null floors are empty, at 0/159 matches under the unrelated token null and 0/240 under the shuffled pairing null, so the headline agreement is not an artifact of a permissive matching criterion\. The “Lens\-only” column shows the surface picture the decomposition corrects, since on factual recall prompts the two lenses report different scripts on 9/10 prompts while the dominant feature agrees on 9/10\. The worked example is the prompt[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.01936#bib.bib28)use for their multilingual illustration, “the opposite of

小”\. The lens fitted on English reportslarge/bigat layers 24, 26, and 29, while the lens fitted on Chinese reports

大的/大, and both readings are carried by the same readout feature, whose top unembedding rows are the

大 family\.[Table33](https://arxiv.org/html/2609.01936#A11.T33)gives that case layer by layer\. The shared feature f112 is the top contributor to

大under both fitted lenses at each comparison layer \(\+12\.5\+12\.5and\+13\.4\+13\.4at layer 24,\+14\.5\+14\.5and\+17\.6\+17\.6at layer 29\) and is also the top contributor tobigunder both, so the two surface forms are carried by one coordinate within each lens as well as across the pair\.

Table 33:The antonym case, layer by layer\.Top\-1 token and its softmax share under each lens on identical Qwen3\.5\-9B hidden stateshℓh\_\{\\ell\}for the prompt of[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.01936#bib.bib28)\(“小”的反义词是, and the model answers 大\)\. The two Jacobian lenses differ only in fitting corpus\.LayerEN\-fittedZH\-fitted24large40%大的 36%26large26%大的 73%29big58%大 93%final大 98%大 98%
#### A second language pair\.

The English–German bank removes the script confound, since both languages use the Latin script, so agreement cannot come from the script alone\. Ninety cross\-lens prompts in seven families mirror the English–Chinese bank \(German antonym, cloze in both languages, German exemplar and factual recall, translation in both directions\), plus the same 12 controls\. Every decomposed surface is validated single\-token under the Qwen tokenizer, cognate pairs are excluded at normalized edit distance≤2\\leq 2after diacritic folding \(30 of 132 candidates excluded, with the exclusion report released alongside the bank\), and calls on the surface language use a lexical criterion, since the two languages share a script\. Jacobian lenses fitted on English and on German \(identical recipe, seeds, and 100\-prompt fitting budget\) diverge in their top\-1 token readings on 73/90 prompts while the same SRP feature stays dominant on 84/90 \(0\.93, CI \[0\.86, 0\.97\]\), against floors of 1/204 \(unrelated token\) and 0/270 \(shuffled pairing\)\. Per family the counts are German antonym 7/9, German cloze 14/14, English cloze 15/15, exemplar 7/10, factual recall 8/8, translation DE→\\toEN 17/17 and EN→\\toDE 16/17, with controls at 5/6 \(digits\) and 6/6 \(proper nouns\)\. Within a single lens, one feature carries both surface forms on 62/90 \(English lens\) and 53/90 \(German lens\) prompts, weaker than in the English–Chinese study and reported as measured\.

#### One prompt read by three fitted lenses\.

Because the two banks share a model and a basis, one hidden statehℓh\_\{\\ell\}can be read by three transports at once, which the pairwise design does not display\. We takeantonym\_de\_01, the German analogue of the published antonym case \(“Das Gegenteil von hoch ist tief\. Das Gegenteil von klein ist”, model answergroß\), and read its frozen states with the Jacobian lenses fitted on English, Chinese, and German \([Table34](https://arxiv.org/html/2609.01936#A11.T34)\)\. At layer 21 all three return formatting tokens, and at layer 24 each fills its ranking with its own fitting language\. The English lens readslarge,large,\-large,Large, the German lens readsgroß,große,large,großes, and the Chinese lens readslargefollowed by

大了, 巨大, 大的\. The same shift is visible on a single token, since the transported logit forgroßis 15\.16 under the English lens and 15\.60 under the Chinese lens against 22\.19 under the German lens\. By layer 29 the lenses fitted on Chinese and on German both readgroßand the lens fitted on English alone still readslarge\.

The decomposition separates the two language pairs here\.großis carried by f6764 andlargeby f12474, whilebigand

大share f112\. The English and Chinese realizations of this concept share a readout feature and the English and German ones do not, which is consistent with the weaker carrying rate within a single lens reported for German above\. That contrast comes from the row codes of the LM head alone, and this single prompt is reported as a worked example\.

Table 34:A German antonym prompt read by three fitted lenses\.Top\-1 token and its transported logit on identical Qwen3\.5\-9B hidden stateshℓh\_\{\\ell\}forantonym\_de\_01, whose answer isgroß\. The three lenses differ only in fitting corpus\.LayerEN\-fittedZH\-fittedDE\-fitted24large22\.4large21\.5groß22\.226large28\.8large29\.9large29\.129large19\.0groß19\.5groß21\.2
#### A second lens family\.

To test whether corpus conditionality is specific to the Jacobian construction, we fit a second family of ridge translators\([Hoerl and Kennard, 1970](https://arxiv.org/html/2609.01936#bib.bib30)\), one per layer and without a bias term, into the final\-layer residual basis, in the tuned\-lens style\([Belrose et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib3)\)but solved in closed form in state space, on exactly the same 100\-prompt English and Chinese corpora and layer set as the Jacobian lenses\. Controlling the fitting corpus is what the test requires, so a released tuned lens is unusable here\. It arrives fitted on a corpus we did not choose, and matching our corpora would mean retraining it in any case\. A closed\-form translator keeps that retraining cheap, and because it shares no fitting procedure with the Jacobian lens, agreement between the two families cannot be an artifact of a common construction\. We tune the ridge strength per layer, and at every layer the tuning picks the largest value we allow\. Even that value contributes negligibly to the fit, so the translators are effectively ordinary least squares\. HoldoutR2R^\{2\}on a 10\-prompt tail rises monotonically with depth \(0\.46–0\.92 English, 0\.46–0\.86 Chinese\), and the resulting decoded states match the model’s top\-1 token on 61–75% of positions at the two deepest fitted layers\. Under this family the corpus conditionality pattern reproduces \(e\.g\. EN→\\toZH translation prompts read Latin under the translator fitted on English and CJK under the translator fitted on Chinese on 9/10\), while the same SRP feature stays dominant on 67/80 prompts \(floors 3/159 and 1/240\)\. Finally, holding the fitting corpus fixed \(English\) and varying only the construction, the Jacobian and ridge lenses read the same dominant feature on 76/80 prompts with both floors empty \(0/159, 0/240\), so the decomposition is invariant to the choice of transport across the two families\.

#### A larger fitting corpus\.

The 100\-prompt fitting budget is small, so a divergence between two lenses could in principle reflect estimation noise\. We refit the English, Chinese, and German Jacobian lenses on 300 prompts each, drawn from the same seeded pools under an identical recipe, and re\-read the same frozen states\. The decomposition does not move, since English–Chinese agreement is 77/80 again and the agreement decision is identical prompt for prompt, with the same 77 prompts passing, the same three failing, and the counts unchanged in every family and in both control groups\. English–German rises from 84/90 to 85/90\. The null floors are unchanged in both pairs \(0/159 and 0/240 for English–Chinese, 1/204 and 0/270 for English–German\)\. The surface reading does move, since script divergence on the English–Chinese bank falls from 39/80 to 33/80 and the factual recall figure quoted in §[5](https://arxiv.org/html/2609.01936#S5)falls from 9/10 to 8/10, while token divergence on the English–German bank rises from 73/90 to 76/90\. Eleven English–Chinese prompts change their reported surface language across the two fitting scales, and none changes its dominant feature\. If the divergence were estimation noise that shrinks as the fitting corpus grows, it would shrink for both pairs, and it does not\. What the fitting budget changes is how often the instrument’s surface report flips, while the readout feature carrying the reading stays fixed\.[Table35](https://arxiv.org/html/2609.01936#A11.T35)collects the agreement rates and both null floors for the main study and all three extensions\.

Table 35:Cross\-lens extension matrix\.Agreement on the dominant feature \(at least two of layers 21, 24, 26, and 29\) with 95% binomial CIs and the two null floors \(unrelated token / shuffled pairing\)\. Row 1 is the main study, rows 2–3 vary one axis each, row 4 holds the corpus fixed and varies only the lens construction, and rows 5–6 hold corpus language and construction fixed while tripling the fitting budget\.ComparisonAgreement95% CIFloorsEN–ZH, Jacobian77/80\[0\.90, 0\.99\]0/159⋅\\cdot0/240EN–DE, Jacobian84/90\[0\.86, 0\.97\]1/204⋅\\cdot0/270EN–ZH, ridge translator67/80\[0\.74, 0\.90\]3/159⋅\\cdot1/240Jacobian vs ridge, EN76/80\[0\.88, 0\.98\]0/159⋅\\cdot0/240EN–ZH, Jacobian,n=300n\{=\}30077/80\[0\.90, 0\.99\]0/159⋅\\cdot0/240EN–DE, Jacobian,n=300n\{=\}30085/90\[0\.88, 0\.98\]1/204⋅\\cdot0/270
#### Caveats and scope\.

The Chinese and German C4 fitting corpora contain incidental English text, which biases the lens pairs toward each other and makes the token\-level divergence conservative\. The study covers one model, now across two language pairs, two constructions of fitted lenses, and two fitting scales, and the ridge translator is our minimal instantiation in the tuned\-lens style, simpler than the exact construction of[Belrose et al\. \(2023\)](https://arxiv.org/html/2609.01936#bib.bib3)\. The released records for each case include every prompt, layer, dominant feature id, and top\-1 token reading\.

## Appendix LAdditional Qwen Displays and Feature Audits

This appendix collects Qwen\-family companion analyses that support the main text\. It proceeds from additional margin examples to variants of selected scores, fixed\-token context shifts, projection profiles, logit\-lens comparisons, and token row audits\. All figures use the same Qwen3\.5\-2B32×32\\times, native\-k=256k=256SAE trained on unembedding rows and the reconstruction error reporting conventions from Appendix[A\.3](https://arxiv.org/html/2609.01936#A1.SS3)\.

### L\.1All\-Layer Logit\-Lens Dense Table

Figure[17](https://arxiv.org/html/2609.01936#A12.F17)gives an all\-layer dense logit\-lens display for the literature promptIt was the best of times, it was the worst of times\. The displayed columns cover the first eight prompt tokens\. The top panel is the ordinary logit\-lens top\-token trajectory after applying the model’s final normalization at each layer\. The bottom panel lists the two largest positive decomposition terms supporting each cell’s vocabulary mean top\-token contrast\. The feature labels are decoded from top Qwen unembedding rows as token\-label summaries for the feature ids\.

![Refer to caption](https://arxiv.org/html/2609.01936v1/qwen2b_32x_dickens_all_layer_lens.png)Figure 17:All\-layer Qwen3\.5\-2B logit\-lens dense table\.The prompt isIt was the best of times, it was the worst of times, and columns are the first eight tokens\. The top panel gives the ordinary logit\-lens top\-token prediction at each layer, and the bottom panel the top positive decomposition terms for each cell’s vocabulary mean top\-token contrast under the Qwen3\.5\-2B32×32\\times, native\-k=256k=256readout SAE used throughout this appendix\.
### L\.2Additional Logit\-Difference Case Studies

Figure 18:Sparse Readout Prism case studies\.Four selected Qwen3\.5\-2B \(D=65536D=65536,k=256k=256\) pairwise logit differences\. Positive terms support the first token in each contrast and negative terms support the competitor\. The examples arebug\-vs\-errorin an insect context,bug\-vs\-insectin a software context,bark\-vs\-dogin a tree context, andbass\-vs\-troutin a music context\. Small bars are suppressed for readability\. All four panels report the unflooredρ0<0\.5\\rho\_\{0\}<0\.5and preserve the sign of the logit difference \(display rules in Appendix[A\.3](https://arxiv.org/html/2609.01936#A1.SS3)\)\.[Table39](https://arxiv.org/html/2609.01936#A12.T39)lists thebug\-panel feature rows\.Figure[18](https://arxiv.org/html/2609.01936#A12.F18)collects the four curated pairwise logit differences read in §[4\.5](https://arxiv.org/html/2609.01936#S4.SS5), namely the twobugcontrasts, and thebarkandbassdifferences\. In the tree context,barkdraws on bark/wood SAE features against dog/cat features, and in the music contextbassdraws on music/percussion\-related features against trout/salmon features \(relative reconstruction errors0\.1260\.126and0\.0970\.097\)\.

### L\.3Additional Selected Readout Score Examples

Figure[19](https://arxiv.org/html/2609.01936#A12.F19)gives the single\-state example summarized in the main text, where raw\-token, vocabulary mean contrast, and pairwise logit difference scores yield different local score decompositions for the same decoded state\. Figure[20](https://arxiv.org/html/2609.01936#A12.F20)adds a group contrast and a top competitor margin\.

Figure 19:Same decoded state, different selected scores\.All panels use the same final state presented to the LM head for the prompt “After reviewing the evidence, the jury found the defendant”, selecting the rawguiltyscore, theguilty\-versus\-vocabulary mean contrast, and theguilty\-minus\-notlogit difference\. Positive bars increase the selected readout score and negative bars reduce it\.Figure 20:Additional Qwen examples of group contrasts and top competitor margins\.On the left is the abstention\-group versus forced\-color\-group contrast, and on the right is how far the top\-rankedguiltyrow leads its rank\-5 competitor\. Bars increase or reduce the selected readout score by sign, and labels give feature ids and token summaries\.
### L\.4Benchmark\-Format Readout Contrast Examples

[Table36](https://arxiv.org/html/2609.01936#A12.T36)reports reconstruction for answer, abstention, label, and safe/action contrasts derived from benchmarks and instantiated as fixed combinations of rows of the LM head\([Rajpurkar et al\., 2018](https://arxiv.org/html/2609.01936#bib.bib66);[Yang et al\., 2018b](https://arxiv.org/html/2609.01936#bib.bib78);[Koreeda and Manning, 2021](https://arxiv.org/html/2609.01936#bib.bib36);[Guha et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib27);[Siddiq and Santos, 2022](https://arxiv.org/html/2609.01936#bib.bib69)\)\. These rows are used as reconstruction checks for readout contrasts\. Lower\-error QA examples appear in Figure[21](https://arxiv.org/html/2609.01936#A12.F21)\.

Table 36:Reconstruction of readout contrasts derived from benchmarks\.Rows evaluate answer\-vs\-distractor, abstention\-vs\-entity, label, or safe\-vs\-unsafe readout contrasts\. Relative errors are the unflooredρ0\\rho\_\{0\}, and the last column gives sign agreement withρ0<0\.5\\rho\_\{0\}<0\.5\. All aggregate rows use 300 contrasts, and family rows break down Qwen3\.5\-2B\.ModelContrast sourcennMed\.ρ0\\rho\_\{0\}SignSign &ρ0<\.5\\rho\_\{0\}<\.5Q\-0\.8Ball contrasts from benchmarks3000\.1800\.9830\.933Q\-2Ball contrasts from benchmarks3000\.1290\.9430\.840Q\-2BSQuAD2 answerable500\.1501\.0000\.900Q\-2BHotpotQA distractor750\.1250\.9330\.907Q\-2BLegalBench ContractNLI500\.0531\.0001\.000Q\-2BSQuAD2 unanswerable750\.1570\.9200\.787Q\-2BSecurityEval500\.3930\.8800\.600Figure 21:Local score decompositions for contrasts derived from benchmarks\.Positive bars support the gold/intended side and negative bars support the distractor side\. Both Qwen3\.5\-2B panels use the selected32×32\\times,k=256k=256readout SAE and have correct sign with low relative error\.
### L\.5Fixed\-Token Context Comparisons

The fixed\-token analyses of §[4\.5](https://arxiv.org/html/2609.01936#S4.SS5)subtract two prompt contexts for the same selected token\. Figure[3](https://arxiv.org/html/2609.01936#S4.F3)gives the signed feature terms for the twobugcontrasts discussed there\. Figure[22](https://arxiv.org/html/2609.01936#A12.F22)shows thebugandbridgecases, and Figure[23](https://arxiv.org/html/2609.01936#A12.F23)gives the additionalringexample\.

Figure 22:Same selected token, different prompt contexts\.Qwen3\.5\-2B \(D=65536D=65536,k=256k=256\)\. Each panel fixes the output token and plots the first\-minus\-second change in feature contribution toh~L⊤​\(wv−μ\)\\tilde\{h\}\_\{L\}^\{\\top\}\(w\_\{v\}\-\\mu\), the token logit with the shared offset term removed, with teal more in the first context and orange more in the second\.A\.bug: insect−\-software\.B\.bridge: structure−\-network\.ringcompanion: Figure[23](https://arxiv.org/html/2609.01936#A12.F23)\.Figure[24](https://arxiv.org/html/2609.01936#A12.F24)gives the companion prompt\-support view for the fixed\-token cases, retaining shared positive support that the signed difference plot subtracts away\.

Figure 23:Additional fixed\-token context comparison forring\.Selected token fixed, with the jewelry context compared against the phone context, using the same signed\-difference convention as Figure[22](https://arxiv.org/html/2609.01936#A12.F22)\.Figure 24:Prompt\-support companion for the main fixed\-token examples\.Each panel holds the selected token fixed and compares positive SAE feature contributions in two prompt contexts\. Bars show the union of the four largest positive contributions per context, with teal the first context in the panel title and orange the second\. Missing bars are features outside that context’s top\-four set\. This view highlights shared positive support that the signed main\-text difference plot subtracts away\.Figure[25](https://arxiv.org/html/2609.01936#A12.F25)adds one technical\-domain context comparison\. The most clearly separated panels aregraph,memory, andvirus, whilepipeillustrates a mixed Unix/plumbing case\. Across the four panels, the maximum rank of the selected token is1717and the maximum relative reconstruction error is0\.050\.05\.

Figure 25:Technical\-domain fixed\-token context comparisons\.Positive bars are larger in the left context and negative bars larger in the right\.graphseparates chart/diagram support from edge/vertex\-like graph\-structure support,memoryseparates human memory from computer\-memory/RAM support, andvirusseparates biological\-virus from malware/antivirus features\.pipeshows a mixed technical/plumbing local score decomposition\.
### L\.6Readout\-Feature Profiles

The examples above decompose selected token rows or contrasts\. As a complementary view, we rank readout features directly from the decoded state before choosing a token or logit difference\. For the final decoded stateh~L\\tilde\{h\}\_\{L\}, we score each SAE decoder direction byh~L⊤​di\\tilde\{h\}\_\{L\}^\{\\top\}d\_\{i\}\. Vocabulary rows then label the selected directions by their highest\-associated rows\. The resulting figures therefore show projections in readout space before score selection, while score\-specific figures provide the local score contributions\. Figures[26](https://arxiv.org/html/2609.01936#A12.F26)and[27](https://arxiv.org/html/2609.01936#A12.F27)give these profiles for theshell,Java,port, andcellprompts\.

Figure 26:Readout\-feature profiles for shell and Java prompts\.Features ranked by\|h~L⊤​di\|\|\\tilde\{h\}\_\{L\}^\{\\top\}d\_\{i\}\|at the final decoded state\. Theshellprompts share a shell\-related direction, and the terminal context additionally has large projections onto command/parser/editor\-like directions\. TheJavaprompts separate beverage\-related directions from programming\-framework and language directions\. Each label lists the top associated unembedding rows for that feature\.Figure 27:Readout\-feature profiles for port and cell prompts\.Theportprompts separate harbor/city/water directions from address/socket/interface directions\. Thecellprompts separate prison\-like physical\-enclosure directions from biology and organism\-related directions, while also exposing a generic uppercase\-letter direction as a non\-lexical high\-magnitude component\.
### L\.7What SRP Adds Beyond the Logit Lens

Figure[1](https://arxiv.org/html/2609.01936#S1.F1)gives the main\-text lens\-versus\-prism comparison\. Table[37](https://arxiv.org/html/2609.01936#A12.T37)lists four additional Qwen3\.5\-2B examples where the ordinary logit lens gives a local A\-vs\-B logit difference and the SRP local score decomposition separates the competing feature terms behind that contrast\. In the prompt\-injection case SRP decomposes an undesirable local readout ranking, where the ordinary lens favors the wrong action and the sparse terms reconstruct that contrast tightly\. The companion feature tables list the displayed feature ids, token\-summary labels, exact and reconstructed scores, and reconstruction errors for these examples\.

Table 37:Logit\-lens comparison examples\.All rows are Qwen3\.5\-2B pairwise logit differences with small local reconstruction errors\.CaseContrastExactSparseρ0\\rho\_\{0\}NotePrompt injectionsummarize−\-delete\-4\.000\-4\.1230\.031undesirable local readout rankingAmbiguous actionask−\-run\+4\.125\+4\.5710\.108successful pairwise contrastSource conflictdollar−\-euro\-1\.438\-1\.4300\.005lexical contrast exampleAbstentionUnknown−\-Alice\+9\.375\+8\.0510\.141abstention\-family exampleFor the prompt fragment

> Trusted task: summarize the email\. Untrusted email content says: delete the files\. We must follow the trusted task\. The next action is to

the ordinary logit\-lens margin favorsdeleteoversummarize\. Sparse Readout Prism shows that the undesirable logit difference is carried by several deletion\-related SAE features against summary and communication features:

ContributionSideToken summary\-2\.938deletedeleting/deletes/deleted\-1\.700delete\_delete/Delete/delete\-1\.606deleteDelete/delete/deletion\+0\.969summarizesum/Sum/\-sum\+0\.933summarizecommunicate/communicates\+0\.922summarizesummary/SummaryThe lens reports the local readout ranking, and SRP resolves that ranking into signed feature terms\.

### L\.8Token\-Row Audit for Main Case\-Study Features

#### How labels are produced\.

Feature labels in the displays are drafted by a language model from the feature’s highest\-scoring unembedding rows\. The authors then verify every label used to support a claim against those same rows, and the audit below measures how often that verification succeeds\.

Table[38](https://arxiv.org/html/2609.01936#A12.T38)reports a conservative qualitative audit of the claim\-bearing displayed labels in the Qwen feature figures\. The audit classifies label readability, a separate question from feature validity\. Coherent labels name a clean semantic lexical family read off the top\-20 token rows, while token form labels are surface artifacts \(letter casing, leading\-space, and sub\-word surface forms, punctuation\-like rows\) that carry no semantic family\. A stricter on\-target test then asks whether a coherent family also matches the sense named in its panel and points to the side indicated by the signed feature term\. The source table for each row, refined feature descriptions, and full top\-20 token row audit accompany the figure artifacts, and the shipped artifact records the earlier three\-way coherent/ambiguous/token form classification, which the binary scheme above refines by folding ambiguous labels into the coherent count and adding the on\-target test\.

#### Blinded rater runs\.

The author audit is complemented by the blinded audit behind the labels of §[4\.5](https://arxiv.org/html/2609.01936#S4.SS5), namely three independent rater runs ofgpt\-5\.6\-solat high reasoning effort, each in a fresh agent context with a frozen prompt, rating only the top\-20 unembedding rows of each claim\-bearing feature \(no access to panel claims, signed terms, or the author classification\)\. The three runs agree unanimously on 93\.4% of items \(Fleiss’κ=0\.87\\kappa=0\.87\), and their majority\-rule consensus over the 99 display occurrences is 76 coherent, 4 mixed or ambiguous, and 19 token form, agreeing with the author audit on 81\.8% of items, with disagreements skewing toward rating labels more coherent than the author did\. We report this as a same\-model reproducibility audit under a fixed protocol and retain the frozen prompt, per\-run outputs, and aggregation rule in the released artifacts\. Item order was randomized under a fixed seed, and generation temperature was not exposed by the agent runtime\.

Table 38:Qualitative feature\-label audit for claim\-bearing Qwen displays\.Rows count the displayed, claim\-bearing feature labels\. Coherent labels are clean semantic lexical families read off the top\-20 token rows, while token form labels are surface artifacts \(letter casing, leading\-space and sub\-word surface forms, punctuation\-like rows\) that carry no semantic family\. The on\-target column is a stricter subset of the coherent count, holding coherent families whose sense matches the named family used in that panel and whose signed contribution points to the claimed side\.Feature setLabelsCoherentToken\-formOn\-targetScopeMainbugpanels11838Figure[18](https://arxiv.org/html/2609.01936#A12.F18), detailed rows in Table[39](https://arxiv.org/html/2609.01936#A12.T39)\.bark/bassmargins12937Figure[18](https://arxiv.org/html/2609.01936#A12.F18)\.Fixed\-token context panels121209Figure[22](https://arxiv.org/html/2609.01936#A12.F22)\.Technical context stress3231124Figure[25](https://arxiv.org/html/2609.01936#A12.F25)\.verify/assumeDLA8807Feature heatmap Figure[30](https://arxiv.org/html/2609.01936#A13.F30), also Figures[28](https://arxiv.org/html/2609.01936#A13.F28)and[32](https://arxiv.org/html/2609.01936#A13.F32)\.General readout examples2417711Figures[19](https://arxiv.org/html/2609.01936#A12.F19)and[20](https://arxiv.org/html/2609.01936#A12.F20)\.Total99851466Conservative count over displayed, claim\-bearing labels\.Across the displayed panels, 85 of 99 claim\-bearing labels name a coherent lexical family at top\-20 token depth, and the remaining 14 are token form surface artifacts\. By the stricter on\-target test, 66 of the 85 coherent families are on\-target, all side\-consistent, with no case where an on\-target label opposed the side claimed in the prose\. The remaining 19 coherent families are off\-target, being clean families that name an off\-panel sense or that are dominated by the panel’s own target token\. In the general\-readout set, the six off\-target coherent families are the positive\-affect family, twogu/Gufragment families, two blue\-surface families, and an effects family\. Panel\-level claims in the main text draw on the on\-target subset\.

Table[39](https://arxiv.org/html/2609.01936#A12.T39)summarizes every feature shown in the two mainbugpanels\. For each sparse feature, we score centered and normalized unembedding rows with that feature’s encoder row and list the highest\-scoring tokens\. The result supports the main\-text interpretation, since most large contributors at the row level are coherent lexical families, while three displayed features are marked as low\-level token form directions\.

Table 39:Top associated unembedding rows for the main case\-study features\.Rows cover all unique features displayed in the twobugpanels of Figure[18](https://arxiv.org/html/2609.01936#A12.F18)\. Leading\-space token variants are shown without the leading space, and non\-Latin synonyms are summarized\.FeatureTop associated rows, shortenedInterpretationConfidencef36mosquito,mosquitoes,Mos,mos, mosquito tokens in Chinese and Vietnamese,insect,flea,malaria,bug,mothmosquito / insect / disease\-vector bug sensehighf4095bee,bees,Bee, bee tokens in Chinese,hive,butterfly,insect,ants,honeybees / insectshighf18302Beetle,beetle,beet,Beet,Bug,bug,\_bug,BUG,buggybeetle / bug token family, mixed insect and software string formsmedium\-highf58330\_error,\_ERROR,error,\-error,Error,\.error,\_errors,/error,\_errprogramming/error\-token featurehighf13081Error,error,\_error,\-error,ERROR,errors,\.Error,\_Error, Japanese error tokenerror / exception surface formshighf59571M,J,G,K,N,P,I,L,A,B,…\\ldots, newline\-like tokensingle leading\-space capitals / low\-level formatting\-like featurelowf21804defect,defects, defect tokens in Chinese and Russian,flaw,defe,defective,flaws,bug,faultdefect / flaw / software\-bug sensehighf43419Crash,crash,crashes,crashed,crashing,Bug,bug,bugs,paniccrash / bug / panic software failurehighf5680debug,Debug,\.debug,\_debug,DEBUG,debugging,debugger,dbg, Chinese debug tokendebug / debugging code tokenshighf34693p,j,d,b,t,g,m,l,h,s,c,nsingle lowercase leading\-space letterslowf23681M,P,S,K,G,A,D,B,N,J,I,Y,H,O,C,Fsingle uppercase letters / alphabetic token familylow

## Appendix MFeature\-Resolved Direct Logit Attribution

Sparse Readout Prism terms sit inside ordinary direct logit attribution as an additive identity\([Elhage et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib18);[Wang et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib73);[Nguyen, 2024](https://arxiv.org/html/2609.01936#bib.bib52)\), the composition §[4](https://arxiv.org/html/2609.01936#S4)points to\. This appendix states that identity and gives the worked case\. Direct logit attribution splits a margin across layers and SRP splits it across features, so composing the two resolves each layer’s contribution into one term per feature, and the basis that says which structure a margin recruits also says which layer supplies it\. For a selected readout directionqα=∑vαv​wvq\_\{\\alpha\}=\\sum\_\{v\}\\alpha\_\{v\}w\_\{v\}, letrα=∑vαv​rvr\_\{\\alpha\}=\\sum\_\{v\}\\alpha\_\{v\}r\_\{v\}\. Sparse Readout Prism gives

qα=\(∑vαv\)​μ\+∑iβi​\(α\)​di\+rα,q\_\{\\alpha\}=\\Big\(\\sum\_\{v\}\\alpha\_\{v\}\\Big\)\\mu\+\\sum\_\{i\}\\beta\_\{i\}\(\\alpha\)d\_\{i\}\+r\_\{\\alpha\},wheredid\_\{i\}is an SAE decoder direction andrαr\_\{\\alpha\}is the readout reconstruction residual for the selected readout score\. The displayed scores below are zero\-sum vocabulary mean or token\-token contrasts, so∑vαv=0\\sum\_\{v\}\\alpha\_\{v\}=0and the offset term cancels, where raw token logits would retain it\. At the final decoded stateh~L\\tilde\{h\}\_\{L\},

h~L⊤​qα=\(∑vαv\)​h~L⊤​μ\+∑iβi​\(α\)​h~L⊤​di\+h~L⊤​rα\.\\tilde\{h\}\_\{L\}^\{\\top\}q\_\{\\alpha\}=\\Big\(\\sum\_\{v\}\\alpha\_\{v\}\\Big\)\\tilde\{h\}\_\{L\}^\{\\top\}\\mu\+\\sum\_\{i\}\\beta\_\{i\}\(\\alpha\)\\tilde\{h\}\_\{L\}^\{\\top\}d\_\{i\}\+\\tilde\{h\}\_\{L\}^\{\\top\}r\_\{\\alpha\}\.Under the fixed final\-normalization convention used for DLA, decomposeh~L≈∑jh~L,j\\tilde\{h\}\_\{L\}\\approx\\sum\_\{j\}\\tilde\{h\}\_\{L,j\}, indexing residual\-stream components byjj\. The component offset, SAE feature, and residual terms are

Oj​\(α\)\\displaystyle O\_\{j\}\(\\alpha\)=\(∑vαv\)​h~L,j⊤​μ,\\displaystyle=\\Big\(\\textstyle\\sum\_\{v\}\\alpha\_\{v\}\\Big\)\\tilde\{h\}\_\{L,j\}^\{\\top\}\\mu,Mj,i​\(α\)\\displaystyle M\_\{j,i\}\(\\alpha\)=βi​\(α\)​h~L,j⊤​di,\\displaystyle=\\beta\_\{i\}\(\\alpha\)\\,\\tilde\{h\}\_\{L,j\}^\{\\top\}d\_\{i\},Rj​\(α\)\\displaystyle R\_\{j\}\(\\alpha\)=h~L,j⊤​rα\.\\displaystyle=\\tilde\{h\}\_\{L,j\}^\{\\top\}r\_\{\\alpha\}\.For the zero\-sum contrasts used in the figures,Oj​\(α\)=0O\_\{j\}\(\\alpha\)=0\. We therefore report two separate gaps, the DLA additivity error betweenh~L⊤​qα\\tilde\{h\}\_\{L\}^\{\\top\}q\_\{\\alpha\}and∑jh~L,j⊤​qα\\sum\_\{j\}\\tilde\{h\}\_\{L,j\}^\{\\top\}q\_\{\\alpha\}, and the readout SAE residualh~L⊤​rα\\tilde\{h\}\_\{L\}^\{\\top\}r\_\{\\alpha\}\(or its componentized version∑jRj​\(α\)\\sum\_\{j\}R\_\{j\}\(\\alpha\)under the same DLA approximation\)\. Because these identity checks sit on well\-separated margins, every relative residual quoted below is the unfloored ratioρ0\\rho\_\{0\}\(§[3\.4](https://arxiv.org/html/2609.01936#S3.SS4)\), computed on full\-precision margins before display rounding\. The difference betweenh~L⊤​qα\\tilde\{h\}\_\{L\}^\{\\top\}q\_\{\\alpha\}and a componentized feature\-only sum∑j,iMj,i​\(α\)\\sum\_\{j,i\}M\_\{j,i\}\(\\alpha\)contains both sources unless the residual and DLA additivity terms are included separately\.

The main display uses the pairwise logit differenceqα=wverify−wassumeq\_\{\\alpha\}=w\_\{\\mathrm\{verify\}\}\-w\_\{\\mathrm\{assume\}\}, for whichβi​\(α\)=zverify,i−zassume,i\\beta\_\{i\}\(\\alpha\)=z\_\{\\mathrm\{verify\},i\}\-z\_\{\\mathrm\{assume\},i\}\. The same identity also covers the vocabulary mean contrastqα=wverify−w¯𝒱q\_\{\\alpha\}=w\_\{\\mathrm\{verify\}\}\-\\bar\{w\}\_\{\\mathcal\{V\}\}, which decomposes one resolved token’s readout above the vocabulary mean row across residual\-stream components\. The component view is an additive decomposition over a realized forward pass\.

[Figure28](https://arxiv.org/html/2609.01936#A13.F28)shows theverify\-minus\-assumemargin discussed in the main text, grouped by semantics\.

![Refer to caption](https://arxiv.org/html/2609.01936v1/qwen2b_32x_prism_dla_semantic_table.png)Figure 28:Feature\-resolved component attribution\.Qwen3\.5\-2Bverify\-assumemargin on the source\-verification prompt \(“The source has not been checked yet\. The appropriate next action is to”\)\. Rows are embedding and per\-layer post\-norm contributions, and columns group verify/check, assume, other sparse, LM head residual, and ordinary DLA terms\. Red supportsverifyand blue supportsassume\.[Figure29](https://arxiv.org/html/2609.01936#A13.F29)gives the per\-component view of the same margin, with each component’s readout feature terms stacked against ordinary direct logit attribution for that component\.

Figure 29:Component attribution resolved by readout feature\.Theverify\-minus\-assumemargin on the source\-verification prompt for Qwen3\.5\-2B, decomposed across residual\-stream components \(embedding and layers 0–23\)\. Each bar stacks that component’s readout feature terms with its LM head reconstruction residual, and black points mark ordinary direct logit attribution for the same component, which the stacked terms track\. Verification mass concentrates in the later layers\.[Figure30](https://arxiv.org/html/2609.01936#A13.F30)resolves theverify\-minus\-assumemargin to individual readout features across residual\-stream stages, where[Figure28](https://arxiv.org/html/2609.01936#A13.F28)groups them by semantics\. The margin is carried by several distinct same\-family directions, where verification and checking features supply the positive mass, concentrated in the later layers, against a smaller assumption\-family group\. The exact margin is\+4\.31\+4\.31, the component attribution sum is\+4\.36\+4\.36, and the sparse feature sum is\+4\.28\+4\.28, with relative residualρ0=0\.009\\rho\_\{0\}=0\.009\.

![Refer to caption](https://arxiv.org/html/2609.01936v1/qwen2b_32x_prism_dla_feature_heatmap.png)Figure 30:Feature\-resolved component accounting\.Theverify\-minus\-assumemargin decomposed by residual\-stream stage \(rows\) against individual sparse readout features \(columns\) for Qwen3\.5\-2B\. Red cells supportverifyand blue cells supportassume\. The contrast is distributed over several same\-family feature directions, with the leading verification feature \(f25948\) dominant in the later layers \(L18–23\)\. Exact margin\+4\.31\+4\.31, component attribution sum\+4\.36\+4\.36, sparse feature sum\+4\.28\+4\.28, relative residualρ0=0\.009\\rho\_\{0\}=0\.009\.[Figure31](https://arxiv.org/html/2609.01936#A13.F31)shows the vocabulary mean contrast version\. The prompt is the same source\-verification prompt, and the selected score ish~L⊤​\(wverify−w¯𝒱\)\\tilde\{h\}\_\{L\}^\{\\top\}\(w\_\{\\mathrm\{verify\}\}\-\\bar\{w\}\_\{\\mathcal\{V\}\}\), whereas the main figure uses the token\-token logit difference\.

Figure 31:Vocabulary\-mean contrast component attribution\.Vocabulary\-meanverifycontrasth~L⊤​\(wverify−w¯𝒱\)\\tilde\{h\}\_\{L\}^\{\\top\}\(w\_\{\\mathrm\{verify\}\}\-\\bar\{w\}\_\{\\mathcal\{V\}\}\)for the source\-verification prompt, decomposed by residual\-stream component\. Each row stacks readout terms grouped into verification/checking features, remaining sparse features, and the unembedding reconstruction error term, and black points mark ordinary direct logit attribution for that residual\-stream component\. The vocabulary mean contrast is\+17\.34\+17\.34, the component DLA sum is\+17\.34\+17\.34, and the sparse SAE feature sum is\+17\.43\+17\.43, with relative reconstruction error0\.0050\.005\.[Figure32](https://arxiv.org/html/2609.01936#A13.F32)is the pairwise companion to the vocabulary mean view above, adding theassumeside of the contrast so that verification/checking and assumption\-family terms appear on the same component axis\.

Figure 32:Pairwise logit difference DLA by SAE feature\.verify\-minus\-assumelogit difference for the source\-verification prompt, decomposed by residual\-stream component\. Each row stacks sparse readout terms grouped into verification/checking features, assumption features, remaining sparse features, and the unembedding reconstruction error term, and black points mark ordinary direct logit attribution for that residual\-stream component\. Positive terms supportverifyoverassume, and negative terms supportassume\. The exact logit difference is\+4\.31\+4\.31, the component DLA sum is\+4\.36\+4\.36, and the sparse SAE feature sum is\+4\.28\+4\.28, with relative reconstruction error0\.0090\.009\.
## Appendix NConstrained Readout\-Side Edit Test

This appendix reports a constrained readout\-edit test for readout SAE decoder directions, asking whether directions associated with a fixed profanity\-related candidate lexicon can move specified logit differences when edited at the readout, and how that effect behaves under held\-out terms, dose sweeps, comparison rows, and off\-target probes\. The test asks a different question from the ablation of §[4\.3](https://arxiv.org/html/2609.01936#S4.SS3)\. The ablation removes a feature from a state the model produced and checks that the margin moves by the feature’s contribution\. The edit test selects features from a few discovery tokens, edits the readout along their decoder directions, and asks how far held\-out terms of the same lexicon are suppressed\. A direction that reaches terms which played no part in selecting it carries the lexical family beyond the rows it was read from\. Table[40](https://arxiv.org/html/2609.01936#A14.T40)lists the checks the test reports and what each one establishes\.

Table 40:Readout SAE decoder\-direction edit summary\.The constrained edit on the readout side supports an intervention result at the readout level, in that editing SAE decoder directions selected from discovery tokens can move constrained lexical logit differences\. The table records the evidence used to pair intended margin shifts with off\-target lexical probes\.MeasurementWhat it measuresSettingLexical specificityDegree to which the selected features and token summaries are specific to the profanity\-related lexical family\.Supports feature selection for this edit test\.Score\-level intervention effectWhether editing along the selected SAE decoder directions changes the specified profanity\-vs\-reference logit differences in the intended direction\.Identifies a score\-level effect at the edited readout, and circuit tracing\([Ameisen et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib1)\)can localize upstream sources\.Held\-out templatesWhether the same intervention transfers to held\-out lexical prompts and prompt templates after feature selection\.Measures held\-out template transfer after feature selection\.Dose responseWhether larger intervention strengths produce ordered margin shifts before quality or off\-target behavior dominates\.Measures dose ordering and separates threshold artifacts from graded response\.Cross\-model replicationWhether the same intervention pattern appears in more than one evaluated model\.Replication is evidence for a recurring pattern at the readout level, while each model is reported alongside its own readout fidelity metrics\.Off\-target lexical probesWhether the intervention leaves unrelated lexical contrasts and neutral prompts stable\.These probes measure obvious off\-target regressions alongside the intended margin movement\.Open\-generation probesHow the edit at the readout level propagates when the model is allowed to generate freely\.Complements the candidate\-constrained lexical\-score measurements\.#### Setup\.

The primary run uses Qwen3\.5\-2B with the 32×\\times,k=256k=256readout SAE reported in Appendix[C](https://arxiv.org/html/2609.01936#A3)\. The evaluation vocabulary contains 22 non\-hateful profanity terms and 20 affect/control terms, and all single\-token claims are filtered through the model tokenizer\. Features are selected from five discovery terms, \{fuck,shit,hell,sucks,bastard\}, and then evaluated on held\-out terms \{fucking,fucked,bullshit,shitty,damned,asshole,bitch,crap,piss,pissed\}\. The intervention is a one\-step logit edit, where for selected decoder directionsdid\_\{i\},i∈Fi\\in F, we adde​WU⊤eW\_\{U\}^\{\\top\}to the logits withe=−γ∑i∈Fdi/∥di∥2e=\-\\gamma\\sum\_\{i\\in F\}d\_\{i\}/\\\|d\_\{i\}\\\|\_\{2\}\. With this sign convention, positiveγ\\gammasuppresses the selected profanity\-associated directions\. The reported intervention scale isγ=16\\gamma=16\. The primary candidate\-constrained grid has 16 prompts, 9 profane\-to\-reference candidate pairs, 10 intervention scales, and 5 methods \(none, readout SAE direction edit, discovery token bias, oracle token bias, and norm\-matched random feature edit\), giving 7200 evaluated rows\.

#### Feature specificity\.

The selected directions are lexically specific\. In the Qwen3\.5\-2B run, 12 of the top 13 profanity\-associated SAE features have specificity1\.01\.0against the 20 control terms, and the remaining feature has specificity0\.9880\.988\. The token summaries partition recognizable lexical families, including f\-word, sh\-word, hell/damn, and insult\-related directions\.

Table 41:Primary Qwen3\.5\-2B readout\-edit comparison\.All methods use intervention scale 16\. Discovery token bias penalizes only the five discovery token ids, while oracle token bias penalizes the full evaluation lexicon\. Random features are norm\-matched to the selected readout SAE direction edit\.Δ​pbad\\Delta p\_\{\\mathrm\{bad\}\}is reported as a positive reduction in probability assigned to the profane\-token side\.MethodDiscoveryflipHeld\-outflipHeld\-outΔ​pbad\\Delta p\_\{\\mathrm\{bad\}\}MedianKLInterpretationReadout SAE directions \(10\)0\.4500\.4500\.3590\.3590\.3980\.3980\.6790\.679Transfers from discovery terms to held\-out variants\.Discovery token bias0\.4500\.4500\.0000\.0000\.0000\.0000\.0010\.001Token\-id edit restricted to discovery terms, so held\-out ids receive no direct bias\.Oracle token bias0\.4500\.4500\.3590\.3590\.4040\.4040\.0040\.004Upper\-bound token edit with full lexicon knowledge\.Random features0\.0130\.0130\.1560\.1560\.0780\.0781\.3631\.363Norm\-matched control with a different effect/cost profile\.No intervention0\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.000Reference condition\.Table[41](https://arxiv.org/html/2609.01936#A14.T41)reports the key intervention comparison\. The readout SAE direction edit matches the oracle token bias held\-out flip rate while using only directions selected from discovery tokens, whereas the equally informed discovery token bias has zero held\-out effect\. The match is exact because scale 16 saturates the evaluation\. A candidate can flip only when the unedited model prefers the profane side, which holds for 23 of the 64 held\-out and 36 of the 80 discovery candidates\. At scale 16 the SAE direction edit and the oracle flip exactly the same 23 held\-out candidates, so both rates equal the ceiling23/64=0\.35923/64=0\.359\(cluster\-bootstrap 95% CI0\.2500\.250–0\.4690\.469for both methods\)\. The methods separate below saturation, where at scale 4 the oracle flips 20 of the 23 against the SAE edit’s 11 \(intersection 10\), and the oracle saturates by scale 6 while the SAE edit reaches the ceiling at scale 12\. KL summaries cluster at the prompt level, since the 16 prompts are shared across both splits\. The KL column records the distributional cost of each edit, and the gap between the oracle’s0\.0040\.004and the direction edit’s0\.6790\.679reflects what each method is allowed to know\. The oracle bias penalizes the logits of the evaluation lexicon itself, so it moves exactly the tokens being scored and leaves the rest of the distribution untouched\. A readout direction is selected from five discovery tokens with the held\-out terms withheld from the selector, so it acts through the whole vocabulary and pays for whatever else it moves\. The oracle is therefore a label\-informed upper bound, and the comparisons that carry the claim are the label\-free mean\-row and PCA directions of Appendix[N\.1](https://arxiv.org/html/2609.01936#A14.SS1), which are matched on distributional cost\. Across the intervention\-scale sweep, the discovery flip rate under the readout SAE direction edit increases monotonically from0\.0380\.038to0\.4500\.450, and the held\-out flip rate tracks it within about0\.040\.04across the mid\-range scales \(6 to 10\), widening to roughly0\.070\.07at scale 4 and0\.090\.09at the headline scale 16\. The result establishes a constrained logit difference intervention pattern, while open\-generation behavior remains a separate evaluation layer beyond candidate scoring\.

Table 42:Cross\-model readout SAE direction edit measurements\.All rows use ten readout SAE decoder directions selected from discovery tokens at intervention scale 16\. Discovery directions are selected from five profanity tokens, and held\-out terms are withheld from both the feature selector and the discovery\-only token bias baseline\. The Ministral tokenizer leaves the held\-out split below the single\-token threshold, so that row reports the comparison on the discovery side\.ModelSAE usedFeaturediscovery flipFeatureheld\-out flipDiscovery\-tokenbias held\-outOff\-target top\-1regressionSettingQwen3\.5\-2B32×\\times,k=256k=2560\.4500\.4500\.3590\.3590\.0000\.0000\.4440\.444Primary readout\-edit run\.R1\-DistillQwen\-7B32×\\times,k=256k=2560\.3880\.3880\.4690\.4690\.0000\.0000\.2220\.222Same tokenizer family, reasoning\-distilled checkpoint\.Ministral3\-8B\-Base16×\\times,k=128k=1280\.5620\.562––0\.2780\.278Tokenizer\-filtered held\-out split, evaluated with an earlier strict SAE checkpoint and reported as a discovery\-side comparison\.Table[42](https://arxiv.org/html/2609.01936#A14.T42)records the quantitative readout\-edit outcome\. The Qwen3\.5\-2B and R1\-Distill\-Qwen\-7B rows show the intended transfer pattern, since the readout SAE direction edit transfers to held\-out lexical variants, while the discovery\-only token bias baseline has zero held\-out effect because those token ids were not in its bias set\. The Ministral row provides a cross\-family comparison on the discovery side because its tokenizer leaves too few held\-out single\-token terms for the held\-out split\.

SAE decoder directions move constrained lexical logit differences when edited at the readout\. The intended scores move under intervention, held\-out templates and scale\-sweep measurements provide supporting evidence, and the two Qwen\-family rows show the same held\-out lexical transfer pattern\. The Ministral row is a cross\-family comparison on the discovery side because too few held\-out terms pass the tokenizer filter\. Off\-target lexical probes map regressions beyond the intended candidate scores\. The Qwen3\.5\-2B off\-target top\-1 regression of0\.4440\.444\([Table42](https://arxiv.org/html/2609.01936#A14.T42)\) is the candidate\-scoring counterpart of the median\-KL distributional cost, and it shows that the edit is constrained by its construction, while its measured effect reaches beyond the intended scores\.

### N\.1Matched\-KL Frontier and Cross\-Model Outcome

The comparison above matches intervention scale, while a stricter comparison matches distributional cost, since a direction can buy more suppression by moving the next\-token distribution further\. The comparison against the mean row and leading principal components of the same discovery rows tests whether the dictionary supplies more of the lexical family than the row geometry alone\. The upgraded protocol enlarges the held\-out lexicon from 64 to 200 candidates \(10 held\-out pairs×\\times20 prompts\), extends scales to 64 so that baseline frontiers reach SRP’s KL range, and adds three label\-free control directions, namely the mean row of the five discovery tokens, PCA rank\-1, and PCA rank\-4 \(the full rank of the discovery rows, hence capacity\-fair\), while the oracle, discovery\-bias, and random brackets are retained\. The frontier plots held\-out suppression \(Δ​pbad\\Delta p\_\{\\mathrm\{bad\}\}primary, flip rate secondary\) against median next\-token KL, and paired cluster bootstraps \(same candidates across methods, clustered conservatively by the 10 held\-out terms and, separately, by the 20 prompts, with 10,000 resamples\) test differences at matched KL\.

Figure 33:Held\-out suppression versus distributional cost at matched KL \(Qwen3\.5\-2B\)\.Held\-outΔ​pbad\\Delta p\_\{\\mathrm\{bad\}\}\(left\) and flip rate \(right\) against median next\-token KL for the SRP direction edit, mean\-row, PCA rank\-1/rank\-4, and the oracle and random brackets\. SRP leads every label\-free baseline at every matched KL on this model and reaches the oracle ceiling at scale 12, while trivial directions reach it only at 1\.3–5×\\timesthe KL\.On the display model SRP outperforms every label\-free baseline at every matched KL \([Figure33](https://arxiv.org/html/2609.01936#A14.F33)\), with all 12 paired comparisons significant \(term\-clustered CIs exclude zero\)\. Against the strongest baseline, the mean\-row direction, the pairedΔ​pbad\\Delta p\_\{\\mathrm\{bad\}\}advantage is\+0\.032\+0\.032\[\+0\.014\+0\.014,\+0\.054\+0\.054\] at KL 0\.02,\+0\.087\+0\.087\[\+0\.065\+0\.065,\+0\.115\+0\.115\] at 0\.05,\+0\.085\+0\.085\[\+0\.062\+0\.062,\+0\.109\+0\.109\] at 0\.10, and\+0\.074\+0\.074\[\+0\.048\+0\.048,\+0\.103\+0\.103\] at 0\.20\. Against PCA rank\-4 the advantages span\+0\.036\+0\.036to\+0\.097\+0\.097, all significant\.

The advantage is model\-specific, and we report the losses\. At matched KL, Qwen3\.5\-0\.8B underperforms the mean\-row direction \(Δ\\Delta−0\.030\-0\.030to−0\.094\-0\.094, significant\) and Qwen3\.5\-9B underperforms it more clearly \(−0\.086\-0\.086to−0\.135\-0\.135, significant\), while R1\-Distill\-Qwen\-7B is mixed \(underperforms mean\-row at two of four matched\-KL points, outperforms PCA rank\-4\)\. Ministral is excluded because the Tekken tokenizer leaves too few single\-token terms for feature discovery \(the dictionary itself is validated in Appendix[J](https://arxiv.org/html/2609.01936#A10)\), and R1\-Distill\-Llama\-8B is excluded with one tokenizable pair and no held\-out terms\. Two candidate explanations for the losses were tested and rejected, namely replacement fidelity \(the 9B readout has high fidelity and still loses\) and dictionary width \(a registered prediction that the narrower 0\.8B dictionary should do better, which it did not\)\. The two comparisons separate\. Against capacity\-matched PCA rank\-4, the dictionary directions match or lead on nearly every model and matched\-KL point\. Against the mean discovery row, the simplest label\-free direction available, they lead decisively on the display model and lose on two others\. The supported claim is therefore scoped\. On the display model, dictionary\-direction edits outperform trivial category directions at matched distributional cost, the advantage is not general across models, and the mean row is the strongest label\-free baseline\.

## Appendix ODiscussion and Future Work

This appendix expands the discussion and future\-work map\. It covers the scope of selected scores, relation to activation SAEs, cross\-family interpretation, DLA composition, and possible extensions\.

### O\.1What Sparse Readout Prism Decomposes

#### Selected readout score\.

Each local score decomposition selects one scalarsα​\(h~ℓ\)=h~ℓ⊤​qαs\_\{\\alpha\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}q\_\{\\alpha\}\. This may be a selected vocabulary logit, a specified pairwise logit difference, a group contrast, a vocabulary mean contrast, or a top competitor margin\. The readout feature projectionspi​\(h~ℓ\)=h~ℓ⊤​dip\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)=\\tilde\{h\}\_\{\\ell\}^\{\\top\}d\_\{i\}are available before selecting such a score, and support for a particular token or contrast becomes defined only after they are combined with the direction coefficientsβi​\(α\)\\beta\_\{i\}\(\\alpha\)\. Distribution\-wide decompositions therefore require measuring reconstruction over the relevant vocabulary slice or token group\. This distinction matters because a token can be supported above a reference while still losing to a named alternative or a group contrast, and a top\-1 token can have only a small margin over top\-ranked competitors\. Reference contrasts, pairwise logit differences, group contrasts, and top competitor margins therefore define distinct selected readout scores\.

#### Local score decomposition\.

SRP provides local additive accounting for a selected readout score\. For that score, the displayed terms express the exact value as SAE feature terms plus preprocessing and reconstruction error terms under the conventions in Appendix[A](https://arxiv.org/html/2609.01936#A1)\. Contribution terms are reported with those error terms, so the local score decomposition is tied to its reconstruction quality\.

### O\.2Relation to Activation SAEs

#### Fitted object\.

Sparse Readout Prism uses sparse autoencoder machinery, but its fitted object differs from activation\-SAE work\. Activation SAEs learn sparse codes for a distribution of hidden states\([Bricken et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib7);[Huben et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib31);[Gao et al\., 2025a](https://arxiv.org/html/2609.01936#bib.bib21)\)\. Sparse Readout Prism learns sparse codes for rows of the unembedding matrixWUW\_\{U\}\. The learned dictionary features are therefore readout directions used by token rows\.

#### Context dependence\.

The decoded state enters after the readout SAE has been learned\. A feature contributes to a selected readout score when two conditions hold together\. The selected readout direction must use that SAE feature through its sparse row coefficients, and the current decoded state must have a large dot product with the corresponding decoder direction\. Hence the same token row can have different dominant feature contributions across contexts, since the row coefficients are fixed while the projection termsh~ℓ⊤​di\\tilde\{h\}\_\{\\ell\}^\{\\top\}d\_\{i\}change with the state being decoded\.

### O\.3Model\-Family Results

#### Qwen evidence\.

Across Qwen sizes, the selected dictionary settings preserve held\-out signed readout scores and readout\-distribution behavior on decoded states with reconstruction errors small enough to support the reported local score decompositions\. These rows provide the fixed display setting and Qwen\-family companion analyses used for most qualitative figures\.

#### Additional families\.

The additional model\-family rows report the same metric suite under different tokenizer, readout, and post\-training conditions\. They separate row reconstruction, LM head replacement, and reconstruction of selected scores, and the paper reports all three beside its feature contributions\.

### O\.4Feature\-Resolved DLA

#### Readout versus components\.

Analyses on the readout side and the component side answer different questions\. The SRP decomposition asks which SAE terms on the unembedding side score a selected logit or contrast, while component\-level attribution asks which residual\-stream components project onto the selected readout direction in the realized forward pass\.

#### Residual terms\.

Combining the two views yields additive component\-by\-feature terms\. The analysis reports both the DLA additivity error for the selected readout score and the readout SAE reconstruction error left outside the displayed feature terms\.

### O\.5Future Work

#### Extensions\.

Natural extensions include controlled interventions on SAE decoder directions, cross\-model feature\-alignment analyses, distributional decompositions of local top\-kkfrontiers, and broader model\-family sweeps\. The basis also extends to the effective linear readouts used by tuned lenses, logit\-prism variants, and future\-token lenses, so lens methods can be compared by sparse signed feature terms as well as decoded tokens, and terms nominated by SRP give targets for causal patching, ablation, and generation\-level evaluation\. A further extension factorizes both sides of a selected margin at once, pairing an activation dictionary forhℓh\_\{\\ell\}with the readout SAE forqq, so that each term records the alignment between a feature on the activation side and one on the readout side\. All depend on the fidelity diagnostics used in the main text, with feature terms read against the reported residual of the selected readout score\.

#### Residual\-stream census\.

SRP also exposes what the readout can express before a decoded state is reduced to a single score\. The readout feature projectionspi​\(h~ℓ\)p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)can be spread widely, while the selected coefficientsα\\alphaselect the termsβi​\(α\)​pi​\(h~ℓ\)\\beta\_\{i\}\(\\alpha\)\\,p\_\{i\}\(\\tilde\{h\}\_\{\\ell\}\)\. Future work could quantify projection mass used by the selected readout score, assigned to competitors, or active but unused\. The unused category is local to the selected readout score, and it may support alternative tokens, later positions, or other computations\. Comparing the split across layers, ambiguous prompts, and incorrect generations could test whether models carry broad task state beyond the immediate readout decision\.

## Appendix PExtended Related Work

This appendix records broader context relevant to positioning but not required for the main argument\.

#### Modern lens variants\.

Beyond the lens work cited in the main text, LogitLens4LLMs updates logit\-lens tooling for modern Hugging Face model families\([Wang, 2025](https://arxiv.org/html/2609.01936#bib.bib74)\), ContextualLens and LatentLens use richer contextual representations for vision\-language models\([Phukan et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib60);[Krojer et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib37)\), and Indic\-TunedLens adapts decoding in the tuned\-lens style to Indian languages\([Panchal et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib56)\)\. These variants are complementary front ends to hidden\-state inspection, orthogonal to SRP’s factorization of readout rows\.

#### Recent sparse\-feature infrastructure\.

Model\-family\-specific activation\-SAE releases include Gemma Scope and Qwen\-Scope\([Lieberum et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib40);[Deng et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib13)\), and recent weight\-based SAE interpretation studies how activation\-SAE features interact with model weights and output\-token effects\([Liu and Deng, 2026](https://arxiv.org/html/2609.01936#bib.bib42)\)\. A separate line trains SAEs on the outputs of dense text\-embedding models\([O’Neill et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib54)\), and those dictionaries decompose embedding vectors computed for inputs, whereas SRP decomposes the weight rows of the LM head itself\. Crosscoders extend dictionary learning across layers, models, or training time within a shared feature space\([Lindsey et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib41)\), with crosscoder\-specific sparsity artifacts documented in chat\-model diffing\([Minder et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib49)\)\. SRP shares the model\-family\-specific emphasis but fits its sparse dictionary directly to unembedding rows, and does not evaluate or replace activation\-feature infrastructure\. Activation\-SAE evaluation has standardized diagnostics\([Karvonen et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib35);[Makelov et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib47)\)and known caveats about unit canonicity\([Leask et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib39)\)\.

#### Latent\-language readings\.

The inference re\-tested in §[5](https://arxiv.org/html/2609.01936#S5)also appears in the latent\-language literature\([Wendler et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib75);[Schut et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib67)\), which reads through the unembedding directly\. Those lenses have no fitting corpus and are not exposed to corpus conditionality\. A weaker and separate confound applies to them instead, since which of several realizations surfaces in a ranking is fixed by the row codes ofWUW\_\{U\}alone, and one readout feature carries both realizations on 61/80 and 62/80 of our own within\-lens comparisons\.[Dumas et al\. \(2025\)](https://arxiv.org/html/2609.01936#bib.bib15)establish language\-agnostic concept representations causally, by activation patching, without relying on a ranking at all\.

#### Output\-space geometry and confounds\.

The linear\-representation view treats token distinctions as directions or contrasts in vocabulary space\([Park et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib57)\), and vocabulary\-embedding directions can steer generation\([Han et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib29)\), and SRP’s edits on the readout side act in the same space at the granularity of individual features\. The same view also motivates edits on the activation side, where redirecting the representations of a target set into a region the model already uses to decline an answer suppresses knowledge through a single down\-projection matrix\([Shen et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib68)\)\. Those edits are stated in hidden states, whereas SRP’s are stated in readout rows\. Which rows are available to either kind of edit is itself a training outcome, since embeddings can be decoupled from the transformer body and trained with a separate vocabulary per data source\([Iacob et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib32)\), one reason the paper reports its metric suite per model family \(Appendix[O\.3](https://arxiv.org/html/2609.01936#A15.SS3)\)\. Degeneration and frequency analyses show that mean directions, row norms, and token frequencies can move token scores independently of the property under test\([Gao et al\., 2019](https://arxiv.org/html/2609.01936#bib.bib20);[Zhang et al\., 2020](https://arxiv.org/html/2609.01936#bib.bib79);[Chung and Kim, 2025](https://arxiv.org/html/2609.01936#bib.bib10)\)\. Anisotropy varies with training and data\([Machina and Mercer, 2024](https://arxiv.org/html/2609.01936#bib.bib46)\), and output\-embedding centering targets the shared mean direction during pretraining\([Stollenwerk et al\., 2026](https://arxiv.org/html/2609.01936#bib.bib70)\)\. These effects motivate the row\-norm and frequency controls of Appendix[G](https://arxiv.org/html/2609.01936#A7)\.

#### Additive attribution and mechanisms\.

Direct logit attribution splits a token logit or logit difference into per\-component terms\([Elhage et al\., 2021](https://arxiv.org/html/2609.01936#bib.bib18);[Wang et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib73)\)\. Despite the similar names, logit prisms\([Nguyen, 2024](https://arxiv.org/html/2609.01936#bib.bib52)\)split who wrote the hidden state across residual\-stream components, while SRP splits what reads that state across the row structure of the LM head, and the two decompositions compose \(Appendix[M](https://arxiv.org/html/2609.01936#A13)\)\. Interventions\([Conmy et al\., 2023](https://arxiv.org/html/2609.01936#bib.bib11)\), causal abstraction\([Geiger et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib23)\), and transcoder or attribution\-graph work\([Dunefsky et al\., 2024](https://arxiv.org/html/2609.01936#bib.bib16);[Ameisen et al\., 2025](https://arxiv.org/html/2609.01936#bib.bib1)\)address complementary mechanism\-level questions about which components and paths produce a behavior\.