Latent Mechanisms of Language Control in Multilingual Language Models

# Latent Mechanisms of Language Control in Multilingual Language Models
Source: [https://arxiv.org/html/2609.00325](https://arxiv.org/html/2609.00325)
Ryo Mitsuhashi††thanks:The first author was an intern at QCRI at the time this work was conducted\.Sabri BoughorbelAffiliation:Prince Sattam binAffiliation:Abdulaziz UniversityAffiliation:Riyadh, Saudi ArabiaMajd HawaslyAffiliation:QCRI, Hamad Bin KhalifaAffiliation:University, Doha, QatarEmail:[mhawasly@hbku\.edu\.qa](mailto:)

###### Abstract

Multilingual large language models can exhibitunintended code\-switching– unnecessarily alternating between languages during generation\. We present a comparative study of three methods that identify language\-controlling latents in cross\-layer transcoders: activation value\-based selection \(ValSel\), activation frequency\-based selection \(FreqSel\), and LLM\-generated latent annotation\-based selection \(AnnSel\)\. To evaluate the efficacy of these methods in identifying language\-controlling latents, we introduce two multilingual benchmarks that exhibit code\-switching for fine\-grained analysis of language steering across seven languages\. Through targeted intervention experiments onGemma\-2\-2BandQwen3\-4B, we find that all three methods effectively manipulate generation language, withFreqSelachieving the strongest overall performance, whileAnnSeloffering interpretable latent selection through explicit language annotations\. A knock\-out analysis suggests the methods select non\-overlapping but each\-functional latent subsets, indicating redundancy rather than a single canonical language direction\. Code and data can be found at[https://github\.com/rm\-3284/Latent\-Mechanism\-Multilingual](https://github.com/rm-3284/Latent-Mechanism-Multilingual)\.

## 1Introduction

Large language models have achieved remarkable multilingual capabilities, but this flexibility comes with an unintended consequence:code\-switching, where models inappropriately alternate between languages mid\-generation\([Marchisio et al\., 2024](https://arxiv.org/html/2609.00325#bib.bib12)\)\. This problem motivated explicit language consistency rewards during training of models such as DeepSeek\-R1\([DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib5)\)\.

Recent mechanistic interpretability work has identified neurons and latents that causally influence output language\([Tang et al\., 2024](https://arxiv.org/html/2609.00325#bib.bib14);[Deng et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib6)\), suggesting that targeted interventions could provide fine\-grained language control in generation\. However, to our knowledge, no systematic comparison exists of language\-controlling mechanism discovery, and it remains unclear whether different approaches uncover the same underlying latents or redundant representations\.

We compare three approaches that identify language\-specific latents in cross\-layer transcoders \(CLTs\)\([Ameisen et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib1)\):ValSel, based on activation value differences\([Deng et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib6)\);FreqSel, based on language\-specific activation frequency\([Andrylie et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib2)\); andAnnSelthat utilizes LLM\-generated annotations to find latents whose descriptions explicitly reference the target language from an extracted circuit graph\. Unlike prior work on single\-layer sparse autoencoders \(SAEs\), CLTs provide a unified sparse dictionary across all layers, enabling holistic analysis of language representations in latent space\.

The contributions of this work include: a comparative study in controlled settings of language\-steering latent selection methods; two controlled benchmarks; an analysis of intervention mechanism efficacy in 7 languages; and further, we highlight the potential redundancy of language representation in two families of models\.

## 2Preliminaries

##### Mechanistic Approaches to Language Control

Prior work has investigated language control at multiple levels of abstraction\. At the neuron level,[Tang et al\. \(2024\)](https://arxiv.org/html/2609.00325#bib.bib14)identified language neurons whose selective activation steers output language\. At the representation level,[Goncharov et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib7)extracted language directions via PCA, enabling steering with directional ablation\. More recently, sparse autoencoders \(SAEs\), which addresspolysemanticityby learning an overcomplete sparse basis of monosemantic features\([Bricken et al\., 2023](https://arxiv.org/html/2609.00325#bib.bib3)\), have been applied to isolate language\-controlling features through activation magnitude\([Deng et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib6)\)or frequency patterns\([Andrylie et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib2)\)\.

##### Cross\-Layer Transcoders \(CLTs\)

extend SAEs by replacing MLP blocks across all layers with a jointly\-trained sparse dictionary\([Ameisen et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib1)\), capturing features spanning model depth\. In this paper, we use CLTs trained forGemma\-2\-2B\([Lieberum et al\., 2024](https://arxiv.org/html/2609.00325#bib.bib11)\)andQwen3\-4B[Hanna \(2025\)](https://arxiv.org/html/2609.00325#bib.bib9)\.

## 3Manipulating Generation Language through Latent Interventions

We study three language\-specificlatent selection methods, and fourlatent intervention strategiesthat steer generation toward a target language\.

### 3\.1Latent Selection Methods

Given a CLT with latent space𝒮\\mathcal\{S\}and a collection of languagesLL, we identify for languagel∈Ll\\in La subset of latents𝒮l⊂𝒮\\mathcal\{S\}^\{l\}\\subset\\mathcal\{S\}that are specifically associated with the generation ofll\. We compare three selection criteria based on activation magnitude, activation frequency, and semantic annotation\.

#### 3\.1\.1Value\-Based Selection \(ValSel\)

Inspired by[Deng et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib6), this method identifies latents whose activation magnitude is distinctively higher for a target language compared to others\. For a latentssand a language\-specific corpusDlD^\{l\}, we compute the mean activation:

μsl=1\|Dl\|​∑t∈Dlas​\(t\)\\mu^\{l\}\_\{s\}=\\frac\{1\}\{\|D^\{l\}\|\}\\sum\_\{t\\in D^\{l\}\}a\_\{s\}\(t\)\(1\)whereas​\(t\)a\_\{s\}\(t\)denotes the activation of latentssfor tokentt\. Thel\-monolinguality scoreof latentssfor languagellis then defined as:

vsl=μsl−1\|L\|−1​∑j∈L∖\{l\}μsjv\_\{s\}^\{l\}=\\mu^\{l\}\_\{s\}\-\\frac\{1\}\{\|L\|\-1\}\\sum\_\{j\\in L\\setminus\\\{l\\\}\}\\mu^\{j\}\_\{s\}\(2\)We select the top\-KKlatents with the highestll\-monolinguality scores for each languagell\.

#### 3\.1\.2Frequency\-Based Selection \(FreqSel\)

Following[Andrylie et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib2), this method identifies latents that activate more frequently for a specific language\. We compute theactivation probabilityfor latentssin languagellas:

Psl=1\|Dl\|​\|\{t∈Dl:as​\(t\)\>0\}\|P^\{l\}\_\{s\}=\\frac\{1\}\{\|D^\{l\}\|\}\{\|\\\{t\\in D^\{l\}:a\_\{s\}\(t\)\>0\\\}\|\}\(3\)A latent is consideredspecificto languagellif: \(1\)Psl=maxj∈L⁡PsjP^\{l\}\_\{s\}=\\max\_\{j\\in L\}P^\{j\}\_\{s\}and \(2\) no languagel′≠ll^\{\\prime\}\\neq lsatisfiesPsl′≥T⋅PslP^\{l^\{\\prime\}\}\_\{s\}\\geq T\\cdot P^\{l\}\_\{s\}for a thresholdT∈\[0,1\]T\\in\[0,1\]\. We select the language\-specific latents that are active in at leastN%N\\%of the examples and at leastMM% of the tokens in the corpus of the languagell\.

#### 3\.1\.3Annotation\-Based Selection \(AnnSel\)

We propose an approach that supports interpretability by leveraging LLM\-generated latent annotations\. This method operates in three stages: circuit tracing, path pruning, and annotation filtering\.

For each sentence inDlD^\{l\}, we trace the attribution graph from input tokens to the predicted output token using Circuit Tracer\([Hanna et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib10)\)\. We truncate sentences at a midpoint, ensuring the target token is a valid word inll\. We then greedily prune the graph, retaining only its high\-importance paths from each token embedding to the top predicted logit based on maximum bottleneck edge weight\. Finally, we extract latents contributing to the prediction, filtering for those with annotations explicitly referencing languagell\(e\.g\., containing “Japanese” or “Japan” forl=l=‘ja’\) to ensure interpretable, language\-related selection\. Refer to Appendix[A](https://arxiv.org/html/2609.00325#A1)for more detail\.

ContextTargetAntonymsThe opposite of“\[adj\]”is “\[antonym\]ex1 \(en, fr\)The opposite of “grand” is “petitex2 \(de, ja\)Das Gegenteil von “大きい” ist “小さいEnumerationsThe \[enum\] are:\[item 1\], \[item 2\],\[item 3\], \[item 4\], …ex3 \(en, es\)The four seasons are:el verano,el otoño,el invierno, la primaveraex4 \(fr, zh\)Les jours de la semaine sont:星期日,星期一,星期二,星期三,星期四, …Table 1:Data format and examples of the two evaluation datasets\. The sentence frame \(insans\-serif\) is in the context languagel0l\_\{0\}, while the target languagellis inbold\. The examples show context/target languages \(l0l\_\{0\},ll\)\.

### 3\.2Intervention Strategies

For the sets of discovered language\-specific latents\{𝒮l\}l∈L\\\{\\mathcal\{S\}^\{l\}\\\}^\{l\\in L\}, we apply inference\-time interventions to steer generation from a context languagel0l\_\{0\}toward a target languagell\. We evaluate four strategies:

- •Target Amplification: Set each latents∈𝒮ls\\in\\mathcal\{S\}^\{l\}to its mean nonzero activation computed overDlD^\{l\}\. This naturally amplifies the signal for the target languagellwhile avoiding out\-of\-distribution effects, without affecting other languages\.
- •Distractor Zero Ablation: Suppress competing signals from other languages by zeroing all latents associated with the distractorl0l\_\{0\}, or the latents of all non\-target languages⋃j∈L∖\{l\}𝒮j\\bigcup\_\{j\\in L\\setminus\\\{l\\\}\}\\mathcal\{S\}^\{j\}\.
- •Distractor Direction Ablation: Ablate the latents of the distractor language𝒮l0\\mathcal\{S\}^\{l\_\{0\}\}bydirection ablation, where the context language gets suppressed by removing the projection of the residual stream activation along a latent’s direction\. We do this either for the single layer with the most latents \(one\-layer\), or for all layers \(multi\-layer\)\.
- •Combined Interventions: Apply distractor ablation and target amplification simultaneously, boosting the signal of the target language while suppressing that of the context language\. We test distractor zero ablation \+ target amplification \(Zero\+Amp\), and distractor one\-layer direction ablation \+ target amplification \(1L\+Amp\)\.

## 4Experiments

All the experiments are conducted on the transcoders ofGemma\-2\-2B\-pt111https://hf\.co/mwhanna/gemma\-scope\-transcodersfrom Gemma\-scope[Lieberum et al\. \(2024\)](https://arxiv.org/html/2609.00325#bib.bib11)andQwen3\-4b222https://hf\.co/mwhanna/qwen3\-4b\-transcoders\.

ForValSel, we useK=50K=50\(to parallel the 2 latents per layer choice in[Deng et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib6)for the 26 layers ofGemma\-2\-2B\)\. ForFreqSel, we setT=0\.8T=0\.8,N=98N=98, andM=10M=10, following[Andrylie et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib2)\. ForAnnSel, we utilize the Circuit Tracer tool[Hanna et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib10)and Neuronpedia annotations333https://www\.neuronpedia\.org\.

To extract the latent sets, we use FLORES\+[NLLB Team et al\. \(2024\)](https://arxiv.org/html/2609.00325#bib.bib13)in seven languages: English \(en\), French \(fr\), German \(de\), Spanish \(es\), Chinese \(zh\), Korean \(ko\), and Japanese \(ja\)\. We apply the three selection methods separately to both CLTs and each language subset\. We report in the Appendix \([Table6](https://arxiv.org/html/2609.00325#A4.T6)\) the count of latents extracted per method and language\. We also report the computational cost of each method in Appendix[E](https://arxiv.org/html/2609.00325#A5)and hyperparameter sensitivity analysis in Appendix[F](https://arxiv.org/html/2609.00325#A6)\.

### 4\.1Datasets

We introduceAntonymsandEnumerations, two novel evaluation datasets for language\-steering interventions\.444Refer to Appendix[K](https://arxiv.org/html/2609.00325#A11)and[L](https://arxiv.org/html/2609.00325#A12)for the complete list\. The datesets can also be accessed from the project’s repository\.TheAntonymstask requires producing a target antonym in languagellgiven a context in languagel0l\_\{0\}, creating a controlled code\-switching scenario\.Enumerationstests open\-ended generation by requiring the continuation of a list in the target language\. Table[1](https://arxiv.org/html/2609.00325#S3.T1)displays the data formats and examples\.

##### Dataset construction\.

We collect 100 antonym pairs per language for the seven languages \(en, fr, de, es, zh, ja, ko\) forAntonymsand 7 ordered categories forEnumerations\. The complete lists of adjectives and words are provided in Appendix[K](https://arxiv.org/html/2609.00325#A11)and[L](https://arxiv.org/html/2609.00325#A12)respectively\. For each of the7×6=427\\times 6=42ordered pairs of context and target languages\(l0,l\)\(l\_\{0\},l\)wherel0≠ll\_\{0\}\\neq l, we generate a test example per antonym pair and per enumeration category\.

##### Evaluation protocol\.

Given a context, we evaluate whether the model prefers the target antonym or enumeration in languagellover its translations\. We record the target token’slogitforAntonymsand length\-normalizedlogprobforEnumerationsfor each example across all seven languages\. A successful intervention must increase these values relative to the baseline\.

Distractor \(l0l\_\{0\}\)L∖\{l\}L\\setminus\\\{l\\\}Distractor \(l0l\_\{0\}\)Distractor \(l0l\_\{0\}\)Target \(ll\)Zero\+Amp1L\+Ampzerozeroone\-layermulti\-layeramplificationablationablationdirection ablationdirection ablationValSel\-8\.0222\.391\.9913\.9869\.5779\.8369\.73FreqSel2\.0221\.39\-2\.95\-51\.6290\.8399\.0796\.16AnnSel\-0\.2011\.900\.3064\.0587\.0096\.7289\.33

Table 2:Intervention effectiveness per method usingAntonyms, computed as the total change in top\-1 logit margin under the intervention \(Equation[4](https://arxiv.org/html/2609.00325#S4.E4)\) averaged over all target languages \(bigger is better\) forGemma\-2\-2B\.

### 4\.2Findings

#### 4\.2\.1Intervention strategy effectiveness

To assess language manipulation efficacy, we intervene on the extracted latents, enforcing a target languagellin a sentence with context languagel0l\_\{0\}usingAntonyms, where the target is a single token\. We measure the target token’stop\-1 logit margin, which is the separation of the logit of the target adjective from its best competitor:ml=maxi∈L∖\{l\}⁡ai−alm\_\{l\}=\\max\_\{i\\in L\\setminus\\\{l\\\}\}a\_\{i\}\-a\_\{l\}, whereala\_\{l\}is the logit of the target adjective in languagell\. A more negativemlm\_\{l\}value indicates a higher probability for target languagell\. We measure the change inmlm\_\{l\}due to the intervention:

Δ​ml=−\(mla​f​t​e​r−mlb​e​f​o​r​e\)\\Delta m\_\{l\}=\-\(m\_\{l\}^\{after\}\-m\_\{l\}^\{before\}\)\(4\)Hence, a larger positive change indicates a more effective intervention\. Summary results forGemma\-2\-2Bare in[Table2](https://arxiv.org/html/2609.00325#S4.T2), with detailed results in the Appendix[Tables19](https://arxiv.org/html/2609.00325#A7.T19),[20](https://arxiv.org/html/2609.00325#A7.T20)and[21](https://arxiv.org/html/2609.00325#A7.T21), as well as results forQwen3\-4B\([Tables24](https://arxiv.org/html/2609.00325#A7.T24),[23](https://arxiv.org/html/2609.00325#A7.T23)and[22](https://arxiv.org/html/2609.00325#A7.T22)\)\.

The most effective intervention isZero\+Amp, combining zero ablation of distractor languagel0l\_\{0\}with target language amplification, followed by1L\+Amp, which instead uses single\-layer directional ablation ofl0l\_\{0\}\. Zero ablation seems to outperform the more powerful direction ablation, possibly due to the latent overlap of related languages, which weakens the target language when direction\-ablating the context language\.

#### 4\.2\.2Selection method efficacy

We now apply the most effective intervention \(Zero\+Amp\) to all selection methods and languages to contrast their performance per target language\. Table[3](https://arxiv.org/html/2609.00325#S4.T3)shows summary results forGemma\-2\-2BonAntonymsandEnumerations\(detailed results in[Tables25](https://arxiv.org/html/2609.00325#A8.T25),[26](https://arxiv.org/html/2609.00325#A8.T26)and[27](https://arxiv.org/html/2609.00325#A8.T27)in the Appendix\), and results forQwen3\-4Bare in[Table28](https://arxiv.org/html/2609.00325#A9.T28)\.

esenzhdejafrkoAvgAntons\.ValSel14\.1910\.1616\.0613\.564\.639\.0112\.1811\.40FreqSel8\.954\.3434\.507\.858\.656\.4528\.3014\.15AnnSel13\.864\.8724\.5219\.6711\.1213\.549\.1113\.81Enums\.ValSel1\.00\-0\.081\.130\.931\.540\.721\.871\.02FreqSel2\.08\-0\.332\.122\.692\.521\.403\.922\.06AnnSel0\.900\.301\.051\.831\.470\.850\.380\.99

Table 3:Selection method efficacy per language under theZero\+Ampintervention forGemma\-2\-2B\.Top: post\-intervention total change in top\-1 logit margin forAntonyms;Bottom: total change in normalized logprob of the target sequence forEnumerations\.In total across all settings and models, all the selection methods are effective\.FreqSelachieves the greatest total absolute logit and logprob change across the tasks, with significant performance gains in Asian languages \(especially Chinese and Korean\)\.AnnSel, on the other hand, shows decent performance across all languages but is hindered by its weak performance in Korean, which might be justified by the small number of extracted latents for Korean \([Table6](https://arxiv.org/html/2609.00325#A4.T6)\)\. We also present the effectiveness of the latents found in a free generation setting in Appendix[J](https://arxiv.org/html/2609.00325#A10)\.

#### 4\.2\.3Is there redundancy in latent language representation?

To evaluate the equivalence of latent sets from different selection methods, we conduct a cross\-method knock\-out experiment for each target language\. We apply the amplification intervention using the first method’s latents, then ablate the residual stream change vector’s projection onto the second method’s representative direction\. This isolates whether the first method’s latents can still drive a positive logit margin change independent of the second method\.

AmplificationAblationTotalmethodmethodΔ​ml\\Delta m\_\{l\}Gemma\-2\-2BAnnSelFreqSel5\.20AnnSelValSel38\.82FreqSelAnnSel90\.63FreqSelValSel92\.16ValSelAnnSel49\.52ValSelFreqSel\-1\.19Qwen3\-4BAnnSelFreqSel1\.28AnnSelValSel7\.58FreqSelAnnSel155\.29FreqSelValSel85\.82ValSelAnnSel107\.55ValSelFreqSel33\.86

Table 4:Aggregate knock\-out experiment results over all languages forGemma\-2\-2BandQwen3\-4B, where the latents of one selection method for the target language are amplified and the latents of the second method are used for direction ablation\.We report in[Table4](https://arxiv.org/html/2609.00325#S4.T4)the total logit margin change aggregated over the seven languages per method pair forGemma\-2\-2BandQwen3\-4BonAntonyms\. The generally positive numbers indicate a form of robustness, however, the asymmetry of the results for the same pair of methods requires further investigation, as it indicates latent sets that are not completely independent\. We provide in the Appendix[M](https://arxiv.org/html/2609.00325#A13)additional observations related to the cosine similarity of the latent set representative residual\-stream directions where language directions vary to different extents in their similarity, in addition to observations related to latent annotations where the intersection of latent sets of the same language identified by different methods is not empty\.

## 5Conclusion

We presented a comparative study of three methods for identifying language\-specific latents in cross\-layer transcoders: value\-based \(ValSel\), frequency\-based \(FreqSel\), and annotation\-based \(AnnSel\)\. Through experiments on the controlledAntonymsandEnumerationstasks in seven languages, we demonstrated that all three methods can effectively steer generation language\.FreqSelachieves the strongest overall performance, particularly for Asian languages, whileAnnSelprovides the added benefit of interpretable latent selection\.

Future work could look into generalizing the findings of this work to larger models and to more languages\. Real\-world code\-switching scenarios may present different challenges to the studied controlled settings\. A deeper understanding of to what extent different methods identify similar or different latents has implications for both understanding multilingual representations and developing robust language\-steering interventions\.

## 6Limitations

Our study has several limitations\. First, experiments were conducted on the CLTs of two small models \(Gemma\-2\-2BandQwen3\-4B\); findings may not generalize to other architectures or standard single\-layer SAEs\. Second, the evaluation was limited to seven languages, predominantly high\-resource, and two controlled code\-switching tasks \(single\-token and multi\-token\), although we also present an experiment in a less constrained setting in Appendix[J](https://arxiv.org/html/2609.00325#A10)\. Real\-world code\-switching scenarios may present different challenges\. Third,AnnSel’s effectiveness depends on the quality of Neuronpedia annotations, which may be inconsistent across languages, as evidenced by the low latent count for Korean \(7 vs\. 45\+ for other languages\)\. Also, keyword matching may not be the most effective selection method, as it does not consider annotation context \(e\.g\., it selects ”Non\-English characters” as an English latent\)\. Finally, we did not systematically evaluate computational costs, thoughAnnSelrequires additional circuit tracing compared to the statistics\-based methods\. We do present a comparison of computational costs in Appendix[E](https://arxiv.org/html/2609.00325#A5)\.

## Potential Risks

Language\-steering interventions could in principle be repurposed to force generation in a specific language for deceptive purposes or to suppress a language entirely\. We consider this risk to be low in practice, as the interventions require direct access to model internals \(CLT weights and latent activations\) and are demonstrated only on small\-scale models in controlled settings\. The benchmarks and methods presented in this work are intended to advance mechanistic understanding of multilingual representations and do not introduce new capabilities beyond what is already achievable through prompting or fine\-tuning\.

## References

- Ameisen et al\. \(2025\)Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L\. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others\. 2025\.[Circuit tracing: Revealing computational graphs in language models](https://transformer-circuits.pub/2025/attribution-graphs/methods.html)\.*Transformer Circuits Thread*\.
- Andrylie et al\. \(2025\)Lyzander Marciano Andrylie, Inaya Rahmanisa, Mahardika Krisna Ihsani, Alfan Farizki Wicaksono, Haryo Akbarianto Wibowo, and Alham Fikri Aji\. 2025\.[Sparse autoencoders can capture language\-specific concepts across diverse languages](https://arxiv.org/abs/2507.11230)\.*Preprint*, arXiv:2507\.11230\.
- Bricken et al\. \(2023\)Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield\-Dodds, Alex Tamkin, Karina Nguyen, and 6 others\. 2023\.[Towards monosemanticity: Decomposing language models with dictionary learning](https://transformer-circuits.pub/2023/monosemantic-features/index.html)\.*Transformer Circuits Thread*\.
- Comanici et al\. \(2025\)Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan\-Jiang Jiang, and 3416 others\. 2025\.[Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://arxiv.org/abs/2507.06261)\.*Preprint*, arXiv:2507\.06261\.
- DeepSeek\-AI et al\. \(2025\)DeepSeek\-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z\. F\. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others\. 2025\.[Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://arxiv.org/abs/2501.12948)\.*Preprint*, arXiv:2501\.12948\.
- Deng et al\. \(2025\)Boyi Deng, Yu Wan, Baosong Yang, Yidan Zhang, and Fuli Feng\. 2025\.[Unveiling language\-specific features in large language models via sparse autoencoders](https://doi.org/10.18653/v1/2025.acl-long.229)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 4563–4608, Vienna, Austria\. Association for Computational Linguistics\.
- Goncharov et al\. \(2025\)Andrey Goncharov, Nikolai Kondusov, and Alexey Zaytsev\. 2025\.[Language steering in latent space to mitigate unintended code\-switching](https://arxiv.org/abs/2510.13849)\.*Preprint*, arXiv:2510\.13849\.
- Google DeepMind \(2025\)Google DeepMind\. 2025\.Gemini 3 flash model card\.[https://storage\.googleapis\.com/deepmind\-media/Model\-Cards/Gemini\-3\-Flash\-Model\-Card\.pdf](https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card.pdf)\.Accessed: 2026\-03\-12\.
- Hanna \(2025\)Michael Hanna\. 2025\.[Qwen3\-4b transcoders](https://huggingface.co/mwhanna/qwen3-4b-transcoders)\.Huggingface\.
- Hanna et al\. \(2025\)Michael Hanna, Mateusz Piotrowski, Jack Lindsey, and Emmanuel Ameisen\. 2025\.circuit\-tracer\.[https://github\.com/safety\-research/circuit\-tracer](https://github.com/safety-research/circuit-tracer)\.The first two authors contributed equally and are listed alphabetically\.
- Lieberum et al\. \(2024\)Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda\. 2024\.[Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2](https://doi.org/10.18653/v1/2024.blackboxnlp-1.19)\.In*Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP*, pages 278–300, Miami, Florida, US\. Association for Computational Linguistics\.
- Marchisio et al\. \(2024\)Kelly Marchisio, Wei\-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder\. 2024\.[Understanding and mitigating language confusion in LLMs](https://doi.org/10.18653/v1/2024.emnlp-main.380)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 6653–6677, Miami, Florida, USA\. Association for Computational Linguistics\.
- NLLB Team et al\. \(2024\)NLLB Team, Marta R\. Costa\-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others\. 2024\.[Scaling neural machine translation to 200 languages](https://doi.org/10.1038/s41586-024-07335-x)\.*Nature*, 630\(8018\):841–846\.
- Tang et al\. \(2024\)Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Wayne Xin Zhao, Furu Wei, and Ji\-Rong Wen\. 2024\.Language\-specific neurons: The key to multilingual capabilities in large language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 5701–5715\.

## Appendix AAnnSelimplementation details

The implementation ofAnnSelcan be decomposed into three steps: attribution graph construction, path pruning, and annotation\-based latent selection\.

### A\.1Attribution graph construction

Since attribution graphs explain how the model predicts the next token, given a sentence from FLORES\+\([NLLB Team et al\., 2024](https://arxiv.org/html/2609.00325#bib.bib13)\), we randomly cut the sentence in the middle so that the next token the model should predict is a valid word in the target language, i\.e\., not a number or punctuation\.

We use implementation from Circuit Tracer[Hanna et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib10)with parameters:max\_n\_logits=5,desired\_logit\_prob=0\.95,max\_feature\_nodes=None\\texttt\{max\\\_n\\\_logits\}=5,\\texttt\{desired\\\_logit\\\_prob\}=0\.95,\\texttt\{max\\\_feature\\\_nodes\}=\\mathrm\{None\}\. It means the attribution graphs are computed to explain the topnnlogits, wheren=min⁡\(5,m\)n=\\min\(5,m\)andmmis the smallest integer that satisfies∑i=1mp​r​o​bi≥0\.95\\sum\_\{i=1\}^\{m\}prob\_\{i\}\\geq 0\.95\. We then prune the graph withnode\_threshold=0\.8,edge\_threshold=0\.98\\texttt\{node\\\_threshold\}=0\.8,\\texttt\{edge\\\_threshold\}=0\.98, meaning keeping nodes that explain 80% of total influence on the output logits and edges that explain 98% of total influence on the output logits\.

### A\.2Path Pruning

After computing the attribution graph, we prune the graphs to prevent unimportant features from being included in the language supernodes\. From each token to the top logit, we select important paths as follows\. We first order the edges by the edge weight in descending order and keep choosing edges that are incident to already chosen nodes, starting from the token\. When we reach the top logit, we can form a path from token to logit\. Then, we remove that chosen path and start again\. We keep doing this until either the edge weight chosen is less than or equal to 0\.1 or we choose 75 distinct paths\. We do this for every token position and obtain a set of paths\.

After obtaining important paths, we further choose more important paths by setting a threshold on the weights of the first edge and last edge, since they represent the importance of the chosen paths in terms of the token and logit, respectively, and thus are directly involved in the information flow from token to logit\. We choose paths whose first edge has more than 0\.5 edge weight and whose last edge has more than 0\.25 edge weight\. Then, out of the chosen features, we extract all the features in the last token position and count how many times each of them appeared\.

### A\.3Annotation\-based latent selection

After obtaining the set of features and the number of times they appear in the important paths, we first extract the description of each feature from Neuronpedia\. We then remove all the features that does not mention the language name or highly relevant country name, e\.g\., \[”Japanese”, ”japanese”, ”Japan”, ”japan”\] for Japanese\.

Then, what we have is a list of features that includes language names or country names in its description and their frequency\. Let us denote the list byℓ\\elland the frequency of featureffbyf​r​e​qffreq\_\{f\}\. We then chooseffthat satisfiesf​r​e​qf≥0\.1​maxk∈ℓ​f​r​e​qkfreq\_\{f\}\\geq 0\.1\\max\_\{k\\in\\ell\}freq\_\{k\}\.

## Appendix BAnnotation Quality ofAnnSel

As we mention in Limitations \([Section6](https://arxiv.org/html/2609.00325#S6)\), the effectiveness of latents found byAnnSeldepends on the quality of Neuronpedia annotations, and we consider this to be the primary reason why Korean latents found byAnnSeldo not perform well\. Also, theAnnSelcriterion is quite simple, as we utilize keyword matching\. In this paper, we presentAnnSelas an interpretable alternative to other methods\. However, more comprehensive validation would be necessary for other applications, as the quality and accuracy of latent annotations can substantially affect the coverage and reliability of the extracted latent sets\.

We list below some annotations from Neuronpedia of Gemma\-2\-2B CLT latents that were selected byAnnSelto show that our simple criterion manages to capture semantically meaningful latents:

- •German: “sentences in German that uses modal verbs to describe realistic situations”
- •English: “words that are common in conversational English grammar despite having little content\.”
- •Spanish: “news casts, TV programs, and names of Spanish\-speaking people in Panama”
- •French: “French language and words related to treaties\.”
- •Japanese: “Japanese text related to programming errors, particularly requests for help”
- •Korean: “Korean names, especially those with multiple parts separated by hyphens, in credits or medical contexts\.”
- •Chinese: “text and code snippets that occur in Chinese contexts”

We also observe that keyword matching may not be semantically correct\. For example, the algorithm selects a latent with the description ”Non\-English characters” as an English latent since it matches the keyword ”English”\. We do not remove these latents to keep the algorithm scalable\.

## Appendix CApplying SAE feature selection methods on CLT

We apply two methods,ValSelandFreqSel, from[Deng et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib6)and[Andrylie et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib2)respectively on the CLT\. Since the original paper executes these methods on SAE, in order to justify the effectiveness of the methods on CLT, we conduct two experiments proposed in[Deng et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib6)and[Andrylie et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib2)\.

### C\.1ValSeland code\-switching experiment

[Deng et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib6)confirms the specificity of the language\-specific features by comparing the activation values in the normal context and code\-switching context\. Following their experiments, we use their code\-switching dataset and plot the activation values of the top\-50 features found and used for steering in our experiments\. Keep in mind that we plot the activations for all the languages used in our experiment except German because[Deng et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib6)does not include German in the experiments\. Also, since we conduct the experiments on the CLT, the extracted features are not distributed uniformly across layers\. The trend is the same as what is claimed in[Deng et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib6), and the activation is generally highest in the original context and noun \(Lang A Prefix \+ Lang A Noun\), somewhat active in original context and modified noun \(Lang A Prefix \+ Lang B Noun\), and close to zero in the modified noun \(Lang B Noun\)\. The results for English as Lang A are shown in[Figure1](https://arxiv.org/html/2609.00325#A3.F1)\. For results in other languages, please refer to our codebase\.

![Refer to caption](https://arxiv.org/html/2609.00325v1/code_switch_activations_en.png)Figure 1:Activation values of English features on other language contexts and nouns ofGemma\-2\-2B\.
### C\.2FreqSeland text\-generation experiment

[Andrylie et al\. \(2025\)](https://arxiv.org/html/2609.00325#bib.bib2)confirms the downstream effectiveness of the features by steering and doing unconditional text\-generation\. Their steering is𝒙n​e​w=𝒙\+α​𝒛jm​a​x​𝒅j\\boldsymbol\{x\}\_\{new\}=\\boldsymbol\{x\}\+\\alpha\\boldsymbol\{z\}\_\{j\}^\{max\}\\boldsymbol\{d\}^\{j\}, whereα∈ℝ\\alpha\\in\\mathbb\{R\}is a scaling factor and𝒛jm​a​x\\boldsymbol\{z\}\_\{j\}^\{max\}is the maximum value of the feature of interest𝒅j\\boldsymbol\{d\}^\{j\}in the multilingual corpora\. We use the same multilingual corpora used to determine the language\-specific features when deciding on𝒛jm​a​x\\boldsymbol\{z\}\_\{j\}^\{max\}\. The model was prompted with \[BOS\] and decoded with top\-p sampling\. Some examples are shown in[Table5](https://arxiv.org/html/2609.00325#A3.T5)\. The output is steered toward the target languages, although the sentences might not be perfectly natural\.

Table 5:Examples of unconditional text generation given \[BOS\] \(p=0\.0p=0\.0means greedy decoding\) ofGemma\-2\-2BIntervenedLanguageα\\alphapOutputde0\.80\.9package Produktion Kleidung refroidissement nôtre enfanshindurchholen nôtre bissunter Bedürf Flüssigkeit geben Flüssigkeit Kleidungen0\.10\.0How to get the value of a variable in a function in python\\n\\nes0\.80\.9How convert Activity anuales in española y y argentina\\npre acuerdos efectivosfr0\.40\.8Que efficaces financières russes russes efficaces financièresfinancières efficaces financières efficaces financières efficacesja0\.30\.0How toget指定したフォルダ内のファイル一覧をスクスクスクスクko0\.50\.0How to 3D 렌샷 것이다 렌샷 것이다 렌zh0\.50\.9The特平阶在道路上有什么用呢？为什么不把柏的吗

## Appendix DLatent counts

[Table6](https://arxiv.org/html/2609.00325#A4.T6)shows the number of extracted token per method and language pair forGemma\-2\-2BandQwen3\-4B\. Note thatValSelextracts 50 latents by design, while the other two methods return a variable number of latents that relates to the CLT training data\.

Table 6:Count of extracted latents per method and language forGemma\-2\-2BandQwen3\-4B\.enfrdeeszhkojaGemmaValSel50505050505050FreqSel101423579509268938234AnnSel1741591941745QwenValSel50505050505050FreqSel11438500299136366293AnnSel1216131016813
## Appendix EComputational Cost of Each Method

We report the computational cost in[Table7](https://arxiv.org/html/2609.00325#A5.T7)\. The numbers reported are cumulative costs in GPU\-hours and kWh when processing 100 sentences×\\times7 languages on an NVIDIA L40 GPU\. We usekWh≈GPU\-hours×\(util/100\)×0\.3​kW\\texttt\{kWh\}\\approx\\texttt\{GPU\-hours\}\\times\(\\texttt\{util\}/100\)\\times 0\.3\\texttt\{kW\}to estimate kWh, as the TDP of the L40 is about 300 W\. Since power does not scale linearly with utilization, this is likely a lower bound of the actual energy consumption\.

Table 7:Comparison of energy spent for each method and each model in GPU\-hours and kWhGemma\-2\-2BQwen3\-4BAnnSel54 GPU\-h, 4\.4 kWh107 GPU\-h, 11\.8 kWhValSel5 GPU\-h, 0\.8 kWh19 GPU\-h, 3\.8 kWhFreqSel5 GPU\-h, 0\.8 kWh23 GPU\-h, 4\.2 kWhWhile the cost ofValSelandFreqSelmainly comes from the forward pass through the model and CLT,AnnSelrequires more extensive circuit tracing and graph pruning on top of that\. The additional cost of Qwen comes from the larger number of features in CLT\. Specifically, the CLT for Qwen has 163,840 features per layer and 36 layers, while the CLT for Gemma has 16,384 features per layer and 26 layers in total\.

## Appendix FHyperparameter Sensitivity Analysis

We conduct the hyperparameter sensitivity analysis using Gemma\-2\-2B and the Antonyms dataset\. All the reported numbers are the results of the Zero\+Amp intervention\.

### F\.1Fixed budget comparison

As the number of latents found by each method differs, we fixed the latent budget per language to the smallest number identified across the three methods\.[Table8](https://arxiv.org/html/2609.00325#A6.T8)shows the results\. As[Table6](https://arxiv.org/html/2609.00325#A4.T6)shows,AnnSelhas the fewest latent counts in all the languages except German\. As for German latents, we rank theAnnSellatents by frequency and take the top 50 latents to match the size ofValSellatents\.

Table 8:The average total change per language for the three selection methods for Gemma\-2\-2B and the Antonyms dataset\.deenesfrjakozhAvgAnnSel19\.454\.8713\.4813\.3611\.219\.0524\.6813\.73FreqSel10\.674\.1412\.7011\.034\.0612\.1019\.3210\.57ValSel13\.6312\.9214\.229\.374\.049\.0617\.2711\.50
### F\.2ValSelsensivity analysis

The only hyperparameter used forValSeliskkwhen choosing the top\-kkhighest latents\.[Table9](https://arxiv.org/html/2609.00325#A6.T9)shows the results, and as the number of latents chosen increases, the average total change increases\. We usek=50k=50in our study\.

Table 9:ValSelaverage total change with varying top\-kTop\-k102030405075100150200Latent count102030405075100150200Avg total change8\.7510\.8411\.2411\.4011\.2314\.1114\.2214\.2615\.36
### F\.3FreqSelsensivitity analysis

FreqSelhas three hyperparameters: cross\-lingual thresholdTT, active example percentageNN, and token thresholdMM, whereNNandMMare expressed as a percentage\. The default values areT=0\.8T=0\.8,N=98N=98, andM=10M=10\.[Tables10](https://arxiv.org/html/2609.00325#A6.T10),[11](https://arxiv.org/html/2609.00325#A6.T11)and[12](https://arxiv.org/html/2609.00325#A6.T12)show the sensitivity analysis results\. When changing one hyperparameter, we use the default values for other hyperparameters\.

Table 10:FreqSelaverage total change with varying cross lingual threshold \(T\)Cross lingual threshold \(T\)0\.60\.70\.750\.80\.850\.90\.951\.0Latent count11401359145115861707182520342269Avg total change14\.2514\.5614\.4714\.8215\.1615\.1415\.3815\.32Table 11:FreqSelaverage total change with varying active example percentage threshold \(N\)Active example percentage threshold \(N\)8085909598Latent count29282861272123031586Avg total change14\.2214\.2214\.1614\.4014\.82Table 12:FreqSelaverage total change with varying active token percentage threshold \(M\)Active token percentage threshold \(M\)57\.51012\.5152025Latent count167916451586146512971022818Avg total change15\.7415\.6414\.8314\.4414\.0712\.7512\.32
### F\.4AnnSelsensivity analysis

AnnSelhas three phases: attribution graph construction, path pruning, and annotation\-based latent selection as explained in[AppendixA](https://arxiv.org/html/2609.00325#A1)\. Attribution graph construction phase has two hyperparameters: node threshold and edge threshold\. The default values arenode\_threshold=0\.8\\texttt\{node\\\_threshold\}=0\.8andedge\_threshold=0\.98\\texttt\{edge\\\_threshold\}=0\.98, and the results are shown in[Tables13](https://arxiv.org/html/2609.00325#A6.T13)and[14](https://arxiv.org/html/2609.00325#A6.T14)\. We report the hyperparameter sweep on three hyperparameters in path pruning phase: first edge weight threshold, last edge weight threshold, and edge throughput threshold\. The default values arefirst\_edge\_weight\_threshold=0\.5\\texttt\{first\\\_edge\\\_weight\\\_threshold\}=0\.5,last\_edge\_weight\_threshold=0\.25\\texttt\{last\\\_edge\\\_weight\\\_threshold\}=0\.25, andedge\_throughput\_threshold=0\.1\\texttt\{edge\\\_throughput\\\_threshold\}=0\.1\.[Tables15](https://arxiv.org/html/2609.00325#A6.T15),[16](https://arxiv.org/html/2609.00325#A6.T16)and[17](https://arxiv.org/html/2609.00325#A6.T17)show the results\. The selection phase has one hyperparameter: frequency separation threshold, which defaults to 0\.1\.[Table18](https://arxiv.org/html/2609.00325#A6.T18)shows the result\. When doing a sweep on one hyperparameter, we use default values for all the other hyperparameters\.

Table 13:AnnSelaverage total change with varying node thresholdNode threshold0\.60\.70\.750\.80\.850\.90\.95Latent count221227227228228229231Avg total change13\.7913\.7813\.7613\.6813\.7013\.8413\.81Table 14:AnnSelaverage total change with varying edge thresholdEdge threshold0\.90\.920\.950\.980\.99Latent count228229228228227Avg total change13\.6813\.6713\.6813\.6813\.71Table 15:AnnSelaverage total change with varying first edge weight thresholdFirst edge weight threshold0\.30\.40\.50\.60\.7Latent count230229228226217Avg total change13\.6813\.6713\.6813\.6913\.59Table 16:AnnSelaverage total change with varying last edge weight thresholdLast edge weight threshold0\.10\.150\.20\.250\.30\.35Latent count347270243228188161Avg total change14\.0513\.6913\.8613\.6813\.7113\.30Table 17:AnnSelaverage total change with varying edge throughput thresholdEdge throughput threshold0\.050\.0750\.10\.150\.20\.25Latent count228228228228228228Avg total change13\.6813\.6813\.6813\.6813\.6813\.68Table 18:AnnSelaverage total change with varying frequency separation thresholdFrequency separation threshold0\.050\.0750\.10\.1250\.150\.20\.250\.30\.40\.5Latent count5643242291781399271594235Avg total change15\.7314\.8513\.7413\.4212\.9111\.7711\.019\.949\.409\.24

## Appendix GDetailed Intervention Strategy Comparison

Next, we provide detailed results for the different interventions underGemma\-2\-2Band theAntonymstask in[Tables19](https://arxiv.org/html/2609.00325#A7.T19),[20](https://arxiv.org/html/2609.00325#A7.T20)and[21](https://arxiv.org/html/2609.00325#A7.T21)\. Similarly, the detailed results for different interventions underQwen3\-4Band theAntonymstask are shown in[Tables22](https://arxiv.org/html/2609.00325#A7.T22),[23](https://arxiv.org/html/2609.00325#A7.T23)and[24](https://arxiv.org/html/2609.00325#A7.T24)\.

Table 19:Intervention effectiveness per target language forValSellatents ofGemma\-2\-2BDistractor \(l0l\_\{0\}\)L/\{l\}L/\\\{l\\\}Distractor \(l0l\_\{0\}\)Distractor \(l0l\_\{0\}\)Target \(ll\)Zero\+Amp1L\+Ampzerozeroone\-layermulti\-layeramplificationablationablationdirection ablationdirection ablationde\-3\.183\.12\-0\.46\-3\.1514\.5213\.613\.92en7\.495\.062\.7752\.075\.8810\.164\.24es0\.253\.06\-0\.06\-1\.5713\.3514\.1915\.08fr\-0\.771\.15\-0\.28\-9\.658\.749\.018\.71ja\-4\.61\-1\.57\-0\.41\-11\.593\.114\.625\.09ko\-0\.0812\.13\-1\.61\-9\.410\.3812\.197\.58zh\-7\.12\-0\.562\.04\-2\.7313\.5916\.0615\.11Total\-8\.0222\.391\.9913\.9869\.5779\.8369\.73Table 20:Intervention effectiveness per target language forFreqSellatents ofGemma\-2\-2BDistractor \(l0l\_\{0\}\)L/\{l\}L/\\\{l\\\}Distractor \(l0l\_\{0\}\)Distractor \(l0l\_\{0\}\)Target \(ll\)Zero\+Amp1L\+Ampzerozeroone\-layermulti\-layeramplificationablationablationdirection ablationdirection ablationde\-3\.450\.8\-1\.670\.585\.087\.866\.29en5\.865\.433\.6\-0\.21\-0\.554\.361\.94es\-0\.73\.39\-0\.6\-5\.126\.688\.957\.09fr\-1\.510\.65\-0\.78\-5\.334\.456\.455\.09ja\-0\.971\.38\-1\.45\-27\.918\.658\.659\.49ko4\.826\.62\-3\.30\.0629\.5428\.3127\.6zh\-2\.033\.121\.25\-13\.6936\.9834\.4938\.66Total2\.0221\.39\-2\.95\-51\.6290\.8399\.0796\.16Table 21:Intervention effectiveness per target language forAnnSellatents ofGemma\-2\-2BDistractor \(l0l\_\{0\}\)L/\{l\}L/\\\{l\\\}Distractor \(l0l\_\{0\}\)Distractor \(l0l\_\{0\}\)Target \(ll\)Zero\+Amp1L\+Ampzerozeroone\-layermulti\-layeramplificationablationablationdirection ablationdirection ablationde\-3\.870\.68\-0\.04\-3\.6220\.1519\.6820\.6en4\.354\.830\.8243\.071\.124\.881\.74es\-0\.041\.98\-0\.156\.613\.9113\.8614\.05fr\-1\.010\.620\.02\-3\.7113\.6913\.5413\.95ja\-2\.211\.5\-1\.01\-4\.9711\.1711\.1210\.53ko6\.765\.970\.4412\.194\.279\.134\.81zh\-4\.18\-3\.680\.2214\.4922\.6924\.5123\.65Total\-0\.2011\.900\.3064\.0587\.0096\.7289\.33Table 22:Intervention effectiveness per target language forValSellatents ofQwen3\-4BDistractor \(l0l\_\{0\}\)L/\{l\}L/\\\{l\\\}Distractor \(l0l\_\{0\}\)Distractor \(l0l\_\{0\}\)Target \(ll\)Zero\+Amp1L\+Ampzerozeroone\-layermulti\-layeramplificationablationablationdirection ablationdirection ablationde5\.240\.83\-4\.32\-21\.714\.588\.474\.99en1\.056\.6\-2\.812\.16\-14\.3\-6\.17\-14\.71es3\.351\.7\-1\.4\-20\.72\.797\.44\.44fr3\.51\-0\.23\-2\.79\-16\.112\.857\.514\.05ja3\.33\-4\.71\-6\.52\-19\.02\-6\.02\-1\.2\-4\.12ko18\.592\.5\-2\.11\-5\.643\.4411\.817\.29zh5\.478\.842\.6717\.17\.5117\.7413\.9Total40\.5415\.53\-17\.28\-63\.920\.8545\.5615\.84Table 23:Intervention effectiveness per target language forFreqSellatents ofQwen3\-4BDistractor \(l0l\_\{0\}\)L/\{l\}L/\\\{l\\\}Distractor \(l0l\_\{0\}\)Distractor \(l0l\_\{0\}\)Target \(ll\)Zero\+Amp1L\+Ampzerozeroone\-layermulti\-layeramplificationablationablationdirection ablationdirection ablationde\-2\.920\.15\-0\.28\-24\.444\.696\.355\.08en6\.678\.391\.19\-9\.42\-2\.016\.11\-1\.55es1\.413\.670\.35\-21\.372\.854\.922\.93fr\-0\.962\.51\-0\.3\-16\.975\.056\.885\.18ja\-3\.30\.18\-3\.63\-37\.57\-0\.270\.37\-1\.25ko3\.997\.29\-0\.25\-30\.1519\.9721\.2920\.5zh11\.6611\.867\.94\-25\.0926\.4729\.7627\.71Total16\.5534\.055\.02\-165\.0156\.7575\.6858\.60Table 24:Intervention effectiveness per target language forAnnSellatents ofQwen3\-4BDistractor \(l0l\_\{0\}\)L/\{l\}L/\\\{l\\\}Distractor \(l0l\_\{0\}\)Distractor \(l0l\_\{0\}\)Target \(ll\)Zero\+Amp1L\+Ampzerozeroone\-layermulti\-layeramplificationablationablationdirection ablationdirection ablationde\-0\.6700\.02\-2\.516\.736\.496\.89en4\.624\.90\.68\.84000es1\.200\.04\-2\.586\.317\.286\.41fr\-0\.710\-0\.02\-6\.216\.56\.356\.56ja0\.40\-0\.18\-13\.11\.761\.831\.72ko0\.120\-2\.42\-55\.67\.76\.08zh3\.4603\.08\-0\.46\.399\.789\.73Total8\.424\.91\.12\-20\.9633\.2939\.4337\.39
## Appendix HZero\+AmpDetailed Results

[Tables25](https://arxiv.org/html/2609.00325#A8.T25),[26](https://arxiv.org/html/2609.00325#A8.T26)and[27](https://arxiv.org/html/2609.00325#A8.T27)show the logit difference results ofValSel,FreqSelandAnnSel, respectively, before and after theZero\+Ampintervention onAntonymsinGemma\-2\-2B\. The numbers in the table are fortop\-1 logit marginto the target adjective, which is the separation in the logit space of the target token to its best competitor\. A value≤0\\leq 0\(green\) indicates that the target adjective is the most likely token, while a positive value \(red\) means it lags behind other undesired tokens\.

Specifically, rows show the context language and columns show the adjective language\. For each language pair, the ‘before’ value shows the logit margin before any intervention, while ‘after’ is its value after the latent amplification is applied\. The desired outcome is that the value becomes smaller \(greener\) after intervention\. Red cells indicate failures\.

While the three methods show reasonable efficacy in enforcing Spanish and Chinese,FreqSelis weaker in enforcing German and English, andValSelshows fragility when the context language is Chinese \(Table[25](https://arxiv.org/html/2609.00325#A8.T25)\)\. The situation of French adjectives is unique, whereValSelandFreqSelfail for some context languages whileAnnSelshows better robustness\. On the other hand,AnnSelshows fragility in Korean, where only 7 latents were identified \(Table[6](https://arxiv.org/html/2609.00325#A4.T6)\) — far fewer than for any other language, whileFreqSelshows the most robust performance\.

Table 25:Logit difference forValSelandGemma\-2\-2Bbefore and after the intervention usingAntonymsesenzhdejafrkobeforeafterbeforeafterbeforeafterbeforeafterbeforeafterbeforeafterbeforeafterde\-1\.79\-2\.93\-4\.57\-5\.19\-2\.97\-4\.88\-\-\-2\.62\-2\.69\-0\.24\-0\.98\-2\.38\-2\.21en\-2\.29\-3\.29\-\-\-3\.77\-4\.41\-1\.45\-2\.35\-2\.96\-2\.71\-0\.10\-0\.66\-2\.87\-3\.27es\-\-\-4\.15\-5\.83\-3\.77\-4\.56\-1\.30\-2\.68\-2\.27\-2\.58\-0\.08\-2\.03\-2\.30\-2\.25fr\-1\.22\-3\.19\-4\.13\-5\.08\-1\.89\-4\.22\-1\.02\-2\.26\-2\.48\-2\.65\-\-\-2\.22\-2\.18ja0\.95\-1\.87\-3\.54\-5\.900\.34\-4\.450\.87\-1\.73\-\-1\.650\.251\.69\-3\.29ko1\.54\-2\.37\-3\.28\-4\.40\-0\.44\-6\.040\.80\-2\.53\-1\.98\-2\.541\.75\-0\.61\-\-zh2\.62\-0\.73\-2\.29\-5\.72\-\-3\.38\-0\.772\.25\-1\.522\.800\.805\.79\-1\.27Total\-0\.19\-14\.38\-21\.96\-32\.12\-12\.50\-28\.561\.28\-12\.32\-10\.06\-14\.695\.78\-3\.23\-2\.29\-14\.47Change\+14\.19\+10\.16\+16\.06\+13\.6\+4\.63\+9\.01\+12\.18

Table 26:Logit difference forFreqSelandGemma\-2\-2Bbefore and after the intervention usingAntonymsesenzhdejafrkobeforeafterbeforeafterbeforeafterbeforeafterbeforeafterbeforeafterbeforeafterde\-1\.79\-1\.59\-4\.57\-4\.97\-2\.97\-7\.58\-\-\-2\.62\-3\.13\-0\.24\-0\.18\-2\.38\-5\.82en\-2\.29\-1\.55\-\-\-3\.77\-7\.99\-1\.45\-1\.17\-2\.96\-3\.34\-0\.100\.16\-2\.87\-5\.73es\-\-\-4\.15\-4\.62\-3\.77\-8\.23\-1\.30\-1\.03\-2\.27\-3\.21\-0\.08\-0\.26\-2\.30\-5\.61fr\-1\.22\-1\.75\-4\.13\-4\.70\-1\.89\-7\.79\-1\.02\-1\.04\-2\.48\-3\.30\-\-\-2\.22\-5\.90ja0\.95\-1\.45\-3\.54\-4\.060\.34\-7\.060\.87\-1\.05\-\-1\.650\.191\.69\-3\.99ko1\.54\-1\.85\-3\.28\-4\.26\-0\.44\-8\.350\.80\-1\.30\-1\.98\-3\.131\.75\-0\.59\-\-zh2\.62\-0\.95\-2\.29\-3\.69\-\-3\.38\-0\.982\.25\-2\.602\.800\.015\.79\-3\.54Total\-0\.19\-9\.14\-21\.96\-26\.30\-12\.50\-47\.001\.28\-6\.57\-10\.06\-18\.715\.78\-0\.67\-2\.29\-30\.59Change\+8\.95\+4\.34\+34\.5\+7\.85\+8\.65\+6\.45\+28\.3

Table 27:Logit difference forAnnSelandGemma\-2\-2Bbefore and after the intervention usingAntonymsesenzhdejafrkobeforeafterbeforeafterbeforeafterbeforeafterbeforeafterbeforeafterbeforeafterde\-1\.79\-3\.04\-4\.57\-5\.32\-2\.97\-5\.67\-\-\-2\.62\-3\.63\-0\.24\-1\.67\-2\.38\-2\.54en\-2\.29\-3\.11\-\-\-3\.77\-6\.46\-1\.45\-3\.01\-2\.96\-3\.69\-0\.10\-1\.04\-2\.87\-2\.57es\-\-\-4\.15\-4\.84\-3\.77\-6\.52\-1\.30\-3\.37\-2\.27\-3\.62\-0\.08\-2\.26\-2\.30\-2\.50fr\-1\.22\-3\.06\-4\.13\-4\.88\-1\.89\-5\.91\-1\.02\-3\.23\-2\.48\-3\.74\-\-\-2\.22\-2\.49ja0\.95\-1\.62\-3\.54\-4\.690\.34\-6\.000\.87\-2\.71\-\-1\.65\-0\.581\.69\-2\.07ko1\.54\-2\.36\-3\.28\-3\.33\-0\.44\-6\.460\.80\-3\.55\-1\.98\-3\.531\.75\-1\.61\-\-zh2\.62\-0\.86\-2\.29\-3\.77\-\-3\.38\-2\.522\.25\-2\.972\.80\-0\.605\.790\.77Total\-0\.19\-14\.05\-21\.96\-26\.83\-12\.50\-37\.021\.28\-18\.39\-10\.06\-21\.185\.78\-7\.76\-2\.29\-11\.40Change\+13\.86\+4\.87\+24\.52\+19\.67\+11\.12\+13\.54\+9\.11

## Appendix IDetailed Selection Method Comparison

[Table28](https://arxiv.org/html/2609.00325#A9.T28)presents summary results for selection method efficacy forQwen3\-4Blatents forAntonymsandEnumerations\.

Table 28:Selection method efficacy per language under theZero\+Ampintervention forQwen3\-4B\. Post\-intervention total change in top\-1 logit margin forAntonyms; Bottom: average change in normalized logprob of the target sequence forEnumerations\.esenzhdejafrkoAvgAntons\.ValSel7\.36\-5\.3117\.566\.38\-1\.177\.4911\.296\.23FreqSel4\.906\.0929\.874\.13\-7\.346\.9020\.869\.34AnnSel7\.355\.129\.844\.952\.236\.956\.796\.17Enums\.ValSel0\.720\.000\.190\.780\.300\.790\.400\.45FreqSel1\.200\.271\.091\.360\.901\.231\.271\.05AnnSel0\.510\.170\.100\.610\.120\.640\.350\.36

## Appendix JContinued Generation

To verify whether the latents we found are effective outside our evaluation datasets and in real\-life generation, we take sentences from FLORES\+\([NLLB Team et al\., 2024](https://arxiv.org/html/2609.00325#bib.bib13)\)that we do not use when finding the latents and do top\-p decoding withp=0\.5p=0\.5while applying Zero\+Amp intervention\. We show some examples of the generated sentences in[Table29](https://arxiv.org/html/2609.00325#A10.T29)\. For the full list of generated sentences, please refer to the GitHub repo\.

Table 29:Selected held\-out FLORES\+ continuations under feature intervention \(ablateAA, amplifyBB,A≠BA\\neq B\)\. Gemma\-2\-2B, default latent sets, top\-p=0\.5p=0\.5\. The generated column is the model suffix after the midpoint\-truncated prompt\.MethodAABBOriginal sentence \(languageAA\)Generated continuation \(languageBB\)AnnSelesfrLa cantidad de hablantes nativos será diferente según se computen o no dialectos que estén muy relacionados entre sí\.… situe la langue dans l’échelle des langues de la planète\.ValSelfresLe nombre de langues maternelles dépend de si oui ou non les dialectes très proches sont pris en compte\.…, si la lengua materna es una lengua de la familia o si es una lengua no de la familia\.FreqSelfresLe nombre de langues maternelles dépend de si oui ou non les dialectes très proches sont pris en compte\.… es la única lengua materna o si hay otras\.AnnSeldeesDie Anzahl der Muttersprachler variiert abhängig davon, ob sehr eng verwandte Dialekte mitgezählt werden oder nicht\.… en inglés o un español es la lengua materna de los estudiantes\.AnnSelzhes母语人数取决于紧密相关的方言是否计算在内。…mente de los diferentes factores que afectan la cantidad de hablantes de un idioma\.FreqSeljafr母語話者の人数は、言語的に非常に近い方言を数に入れるかどうかによって変わってきます。… diversifiez la culture de votre enfant\.AnnSelfresVous vous fatigueriez moins de cette façon\. N’oubliez pas qu’il n’est pas nécessaire de trop insister sur les touches pour qu’elles résonnent plus fort\. C’est le cas sur le piano\.… trabajar a fondo en una jornada\.AnnSeljaesそうすることで、疲れがたまりにくくなります。ピアノのように音量を出すために力を入れて鍵盤を叩く必要はないことを覚えておいてください。… instrumentos musicales en casa que no requieren de mucha fuerza\.AnnSelkoja이 방식을 통해 연주의 피로도를 최대한 낮출 수 있습니다\. 피아노와 달리, 아코디언은 더 큰 소리를 내기 위해 건반을 세게 칠 필요가 없다는 것을 기억하세요\.…3弦ヴァイオリンは2弦ヴァイオリンよりも弦の張力が強め、弦の弾き方にもよりますが、音響的にはより響きが良きです。AnnSelfresLe repérage et la collecte d’informations par David Headley ont facilité l’opération menée par les dix hommes armés du groupe militant pakistanais Laskhar\-e\-Taiba\.…ée en 2008 en Pakistan, a relatado el ministro de Defensa de los Estados Unidos, Robert Gates, durante un discurso en el Colegio de la Guerra de Estados Unidos, citado por la agencia de noticias AFP\.FreqSelzhen他又称：“这个病例很严重。但我们的医疗系统运作良好，请大家放心。”… fortunately, it’s not as serious as we thought\.”AnnSelkoja“그는 ‘이건 심각한 사건입니다\. 다행히도 우리 시스템이 잘 작동하고 있으니 안심하셔도 됩니다\.’라고 말을 이었다\.”…誤作動しただけで、100%確信し得る情報はありませんと語りました。サイバーセキュリティは、あらゆるものを守るための重要な手段です。FreqSeldefrEr fuhr fort: ,,Dieser Fall ist ernst\. Seien Sie versichert, dass unser System so gut funktioniert, wie es sollte\.“… très sérieux\.
## Appendix KAdjectives Used inAntonyms

We prepare 100 pairs of antonyms to use for the antonym\-structured sentences \(The opposite of ”adj1” is ”, adj2\)\. Keep in mind that there could be multiple suitable words foradj2\(e\.g\., the antonym of small could be either big or large\)\. A human annotator first compiled an initial set of 30 adjective antonym pairs in English\. An LLM \(Gemini 2\.5 Flash\([Comanici et al\., 2025](https://arxiv.org/html/2609.00325#bib.bib4)\)\) was then instructed to expand this list to 100 pairs, after which every pair was manually validated\. The validated pairs were subsequently translated into the remaining six languages using an LLM, and native speakers verified the translations for their respective languages\. All the antonyms are shown in[Table30](https://arxiv.org/html/2609.00325#A11.T30)\. It is worth mentioning that the words or tokens can be exactly the same in some cases across different languages, most commonly between French and Spanish and between Chinese and Japanese\.

Table 30:Multilingual Antonym Pairs in the 7 languagesEnglishFrenchGermanSpanishChineseJapaneseKoreangoodbongutbueno好良い좋은badmauvaisschlechtmalo坏悪い나쁜happyheureuxglücklichfeliz开心嬉しい행복한sad, unhappytriste, malheureuxtraurig, unglücklichtriste, infeliz难过, 不高兴悲しい슬픈, 불행한biggrandgroßgrande大大きい큰smallpetitkleinpequeño小小さい작은hotchaudheißcaliente热暑い더운coldfroidkaltfrío冷寒い, 冷たい추운, 차가운fastrapideschnellrápido快速い빠른slowlent, lentelangsamlento慢遅い느린lightlégerleichtligero轻軽い가벼운heavy, darklourdschwerpesado重重い무거운easyfacileeinfachfácil容易簡単な쉬운difficult, harddifficileschwierig, schwerdifícil难難しい어려운newnouveauneunuevo新新しい새로운oldvieux, ancienaltviejo旧古い오래된truevraiwahrverdadero真正しい진실한false, untruefauxfalsch, unwahrfalso, incorrecto假間違った거짓의, 틀린alivevivantlebendigvivo活生きている살아있는deadmorttotmuerto死死んでいる죽은fullpleinvolllleno满満杯가득 찬emptyvideleervacío空空っぽ빈brightbrillanthellbrillante亮明るい밝은dark, dimsombre, obscurdunkeloscuro, opaco暗暗い어두운, 흐릿한strongfortstarkfuerte强強い강한weakfaibleschwachdébil弱弱い약한cleanpropresauberlimpio干净きれいな깨끗한dirtysaleschmutzigsucio脏汚い더러운openouvertoffenabierto开開いた열린closedfermégeschlossencerrado关閉じた닫힌richrichereichrico富裕裕福な부유한poorpauvrearmpobre贫穷貧しい가난한beautifulbeauschönhermoso美美しい아름다운uglylaidhässlichfeo丑醜い못생긴longlonglanglargo长長い긴shortcourtkurzcorto短短い짧은widelargebreitancho宽広い넓은narrowétroitengestrecho窄狭い좁은harddurhartduro硬硬い딱딱한softmouweichsuave软柔らかい부드러운drysectrockenseco干乾いた마른wet, moisthumide, mouillénassmojado, húmedo湿濡れた, 湿った젖은, 축축한loudfortlautruidoso大声うるさい시끄러운quiet, silentsilencieux, douxleise, stilltranquilo, silencioso安静, 小声静かな조용한, 고요한earlytôtfrühtemprano早早い이른latetardspättarde晚遅い늦은nearprochenahcerca近近い가까운farloinfern, weitlejos远遠い먼deepprofondtiefprofundo深深い깊은shallowpeu profondflachsuperficial, poco profundo浅浅い얕은badmauvaisschlechtmalo坏悪い나쁜goodbongutbueno好良い좋은sadtristetraurigtriste难过悲しい슬픈happy, joyfulheureux, joyeuxglücklichfeliz, alegre开心嬉しい행복한, 기쁜smallpetitkleinpequeño小小さい작은big, largegrandgroßgrande大大きい큰coldfroidkaltfrío冷寒い추운hot, warmchaudheißcaliente, cálido热暑い, 熱い더운, 따뜻한slowlentlangsamlento慢遅い느린fast, quickrapideschnellrápido, veloz快速い, 早い빠른heavylourdschwerpesado重重い무거운lightlégerleichtligero轻軽い가벼운difficultdifficileschwierigdifícil难難しい어려운easy, simplefacileeinfachfácil, sencillo容易, 易簡単な, 簡単쉬운, 간단한oldvieuxaltviejo旧古い오래된newnouveau, jeuneneunuevo, joven新新しい새로운, 젊은falsefauxfalschfalso假間違った거짓의true, correctvrai, correctwahr, richtigverdadero, correcto真正しい진실한, 올바른deadmorttotmuerto死死んでいる죽은alive, livingvivantlebendigvivo, viviente活, 生生きている살아있는, 생생한emptyvideleervacío空空っぽ빈fullpleinvolllleno满満杯가득 찬darksombredunkeloscuro暗暗い어두운bright, lightbrillant, clairhellbrillante, claro亮, 光明明るい밝은weakfaibleschwachdébil弱弱い약한strongfortstarkfuerte强, 强壮強い, 丈夫な강한dirtysaleschmutzigsucio脏汚い더러운cleanpropresauberlimpio干净きれいな, きれい, 綺麗깨끗한closedfermégeschlossencerrado关閉じた닫힌openouvertoffenabierto开開いた열린poorpauvrearmpobre贫穷貧しい가난한rich, wealthyrichereichrico, adinerado富裕裕福な, 豊かな부유한, 풍부한uglylaidhässlichfeo丑醜い못생긴beautiful, prettybeau, jolischönhermoso, bonito美美しい아름다운, 예쁜shortcourtkurzcorto短短い짧은longlonglanglargo长長い긴narrowétroitengestrecho窄狭い좁은wide, broadlargebreitancho宽広い넓은softmouweichsuave软柔らかい부드러운hard, firmdurhartduro硬硬い딱딱한wethumidenassmojado湿濡れた젖은drysectrockenseco干乾いた마른quietsilencieuxleisetranquilo安静静かな조용한loud, noisyfort, bruyantlautruidoso大声, 吵闹うるさい시끄러운latetardspättarde晚遅い늦은earlytôtfrühtemprano早早い이른farloinfernlejos远遠い먼near, closeprochenahcerca近近い가까운shallowpeu profondflachsuperficial浅浅い얕은deepprofondtief, hochprofundo深深い깊은youngjeunejungjoven年轻若い젊은oldvieuxaltviejo老老いた늙은kindgentilfreundlichamable善良親切な친절한unkind, meanméchantunfreundlichdesagradable, malo不善良, 邪恶意地悪な, 不親切な, 失礼な불친절한, 못된bravecourageuxmutigvaliente勇敢勇敢な용감한cowardlylâchefeigecobarde懦弱臆病な겁많은wisesageweisesabio明智賢い현명한foolishinsensé, stupidedumm, törichttonto, necio愚蠢愚かな어리석은politepolihöflicheducado礼貌丁寧な예의 바른impolite, rudeimpoli, grossierunhöflich, grobdescortés, maleducado不礼貌失礼な, 無礼な, 粗略な, 粗雑な무례한patientpatientgeduldigpaciente耐心我慢強い인내심 있는impatientimpatientungeduldigimpaciente不耐烦せっかちな성급한, 참을성 없는honesthonnêteehrlichhonesto诚实正直な정직한dishonestmalhonnêteunehrlichdeshonesto不诚实不正直な부정직한safesûrsicherseguro安全安全な안전한dangerous, unsafedangereuxgefährlich, unsicherpeligroso, inseguro危险危険な위험한activeactifaktivactivo积极活動的な활동적인inactive, passiveinactif, passifinaktiv, passivinactivo, pasivo消极消極的な비활동적인, 소극적인straightdroitgeraderecto直まっすぐな곧은curved, bentcourbe, tordugebogen, krummcurvo, doblado弯曲がった굽은, 휘어진wholeentierganzentero完整全体の전체의part, brokenpartiel, casséteilweise, gebrochenparcial, roto部分, 破部分的な, 壊れた부분적인, 부서진calmcalmeruhigcalmado平静穏やかな차분한agitated, stormyagité, orageuxaufgeregt, stürmischagitado, tormentoso激动, 骚乱荒れた, 興奮した격앙된, 폭풍우치는correctcorrectrichtigcorrecto正确正しい정확한incorrect, wrongincorrect, fauxfalschincorrecto, equivocado不正确, 错误間違っている부정확한, 틀린complexcomplexekomplexcomplejo复杂複雑な복잡한simplesimpleeinfachsimple, sencillo简单単純な간단한, 단순한effectiveefficaceeffektivefectivo有效効果的な효과적인ineffectiveinefficaceineffektivineficaz无效非効果的な비효과적인famouscélèbreberühmtfamoso著名有名な유명한unknown, obscureinconnu, obscurunbekannt, unbedeutenddesconocido, oscuro无名, 不为人知無名の, 知られていない무명의, 잘 알려지지 않은generousgénéreuxgroßzügiggeneroso慷慨寛大な관대한stingy, meanavare, mesquingeizigtacaño, malo吝啬ケチな인색한, 못된contentcontentzufriedencontento高兴幸せな행복한unhappy, dissatisfiedmécontent, insatisfaitunzufriedeninfeliz, insatisfecho不高兴, 不满意不幸せな불행한, 불만족스러운healthysaingesundsaludable健康健康な건강한unhealthy, sickmalsain, maladeungesund, krankinsalubre, enfermo不健康, 生病不健康な, 病気の건강하지 않은, 아픈highhauthochalto高高い높은lowbasniedrigbajo低低い낮은importantimportantwichtigimportante重要重要な중요한unimportant, trivialsans importance, insignifiantunwichtig, trivialsin importance, trivial不重要, 琐碎重要でない, 取るに足らない중요하지 않은, 사소한innocentinnocentunschuldiginocente无辜無罪の무고한guiltycoupableschuldigculpable有罪有罪の유죄의knownconnubekanntconocido已知既知の알려진unknowninconnuunbekanntdesconocido未知未知の알려지지 않은malemasculinmännlichmasculino男性男性の남성의femalefémininweiblichfemenino女性女性の여성의normalnormalnormalnormal正常普通の정상적인abnormal, unusualanormal, inusuelabnormal, ungewöhnlichanormal, inusual异常, 不正常異常な비정상적인, 이상한possiblepossiblemöglichposible可能可能な가능한impossibleimpossibleunmöglichimposible不可能不可能な불가능한privateprivéprivatprivado私人個人の사적인publicpublicöffentlichpúblico公共公共の공적인rightjusterichtigcorrecto对正しい올바른wrongfauxfalschincorrecto, equivocado错間違っている틀린simplesimpleeinfachsencillo简单簡単な단순한complexcomplexekomplexcomplejo复杂複雑な복잡한sweetdouxsüßdulce甜甘い달콤한sour, bitteracide, amersauer, bitteragrio, amargo酸, 苦酸っぱい, 苦い신, 쓴visiblevisiblesichtbarvisible可见見える보이는invisibleinvisibleunsichtbarinvisible不可见見えない보이지 않는warmchaudwarmcálido暖和暖かい따뜻한cool, coldfrais, froidkühl, kaltfresco, frío凉, 冷涼しい, 冷たい시원한, 차가운smoothlisseglattliso光滑滑らかな매끄러운rough, bumpyrugueux, bosselérau, holprigáspero, irregular粗糙粗い거친, 울퉁불퉁한thinmincedünndelgado薄薄い얇은thick, fatépaisdickgrueso厚厚い두꺼운tallgrandgroßalto高背が高い키가 큰shortpetitkleinbajo矮背が低い키가 작은marriedmariéverheiratetcasado已婚既婚の결혼한single, unmarriedcélibataireledig, unverheiratetsoltero单身, 未婚独身の, 未婚の독신의, 미혼의optimisticoptimisteoptimistischoptimista乐观楽観的な낙관적인pessimisticpessimistepessimistischpesimista悲观悲観的な비관적인permanentpermanentpermanentpermanente永久恒久的な영구적인temporarytemporairetemporärtemporal临时一時的な일시적인presentprésentgegenwärtigpresente现在現在の현재의absent, pastabsent, passéabwesend, vergangenausente, pasado缺席, 过去不在の, 過去の부재의, 과거의publicpublicöffentlichpúblico公共公共の공공의privateprivéprivatprivado私人個人の사적인realréelechtreal真实本物の실제적인fake, unrealfaux, irréelfalsch, unrealfalso, irreal假, 虚假偽物の, 非現実の가짜의, 비현실적인responsibleresponsableverantwortlichresponsable负责責任책임 있는irresponsibleirresponsableunverantwortlichirresponsable不负责無責任な, 無責任무책임한singlecélibataireledigsoltero单身独身の독신의marriedmariéverheiratetcasado已婚既婚の, 結婚している결혼한souracidesaueragrio酸酸っぱい신sweetdouxsüßdulce甜甘い달콤한usefulutilenützlichútil有用役に立つ유용한uselessinutilenutzlosinútil没用役に立たない쓸모없는verticalverticalvertikalvertical垂直垂直な수직의horizontalhorizontalhorizontalhorizontal水平水平な수평의wellbiengutbien好良い잘poorlymalschlechtmal, pobremente差悪い못, 형편없이winninggagnantgewinnendganador获胜勝利の승리하는losingperdantverlierendperdedor失败敗北の패배하는unkindméchantunfreundlichdesagradable不善良意地悪な불친절한kindgentilfreundlichamable善良親切な친절한cowardlylâchefeigecobarde懦弱臆病な겁많은bravecourageuxmutigvaliente勇敢勇敢な용감한
## Appendix LListings Used inEnumerations

We prepare 7 ordered enumerations to use for the enumeration\-structured sentences \(The\{category\}are:\{choices\}\)\. The English dataset was created entirely by human annotators and then translated into the remaining languages following the same LLM\-assisted translation and native\-speaker validation process, using Gemini 3 Flash\([Google DeepMind, 2025](https://arxiv.org/html/2609.00325#bib.bib8)\)for translation\. All the categories, listings, and the split of the listed choices and continuation words are shown in[Table31](https://arxiv.org/html/2609.00325#A12.T31)\.

Table 31:Enumerations Dataset: categories, items, and split thresholds\.Category IDSplitLangCategory NameListingsmonths4—8enmonths of the yearJanuary, February, March, April, May, June, July, August, September, October, November, DecemberdeMonate des JahresJanuar, Februar, März, April, Mai, Juni, Juli, August, September, Oktober, November, Dezemberfrmois de l’annéejanvier, février, mars, avril, mai, juin, juillet, août, septembre, octobre, novembre, décembreesmeses del añoenero, febrero, marzo, abril, mayo, junio, julio, agosto, septiembre, octubre, noviembre, diciembrezh一年中的月份一月, 二月, 三月, 四月, 五月, 六月, 七月, 八月, 九月, 十月, 十一月, 十二月ja一年の月一月, 二月, 三月, 四月, 五月, 六月, 七月, 八月, 九月, 十月, 十一月, 十二月ko1년의 열두 달1월, 2월, 3월, 4월, 5월, 6월, 7월, 8월, 9월, 10월, 11월, 12월numbers4—6ennumbers 1 to 10one, two, three, four, five, six, seven, eight, nine, tendeZahlen von eins bis zehneins, zwei, drei, vier, fünf, sechs, sieben, acht, neun, zehnfrnombres de un à dixun, deux, trois, quatre, cinq, six, sept, huit, neuf, dixesnúmeros del uno al diezuno, dos, tres, cuatro, cinco, seis, siete, ocho, nueve, diezzh从一到十的数字一, 二, 三, 四, 五, 六, 七, 八, 九, 十ja一から十までの数字一, 二, 三, 四, 五, 六, 七, 八, 九, 十ko1부터 10까지의 숫자일, 이, 삼, 사, 오, 육, 칠, 팔, 구, 십days\_of\_week3—4endays of the weekSunday, Monday, Tuesday, Wednesday, Thursday, Friday, SaturdaydeWochentageSonntag, Montag, Dienstag, Mittwoch, Donnerstag, Freitag, Samstagfrjours de la semainedimanche, lundi, mardi, mercredi, jeudi, vendredi, samediesdías de la semanadomingo, lunes, martes, miércoles, jueves, viernes, sábadozh一星期中的日子星期日, 星期一, 星期二, 星期三, 星期四, 星期五, 星期六ja曜日日曜日, 月曜日, 火曜日, 水曜日, 木曜日, 金曜日, 土曜日ko요일일요일, 월요일, 화요일, 수요일, 목요일, 금요일, 토요일four\_seasons2—2enfour seasonsspring, summer, fall, winterdevier JahreszeitenFrühling, Sommer, Herbst, Winterfrquatre saisonsprintemps, été, automne, hiverescuatro estacionesprimavera, verano, otoño, inviernozh四季春, 夏, 秋, 冬ja四季春, 夏, 秋, 冬ko사계절봄, 여름, 가을, 겨울times\_of\_day2—2entimes of daymorning, afternoon, evening, nightdeTageszeitenMorgen, Nachmittag, Abend, Nachtfrmoments de la journéematin, après\-midi, soir, nuitesmomentos del díamañana, tarde, tarde, nochezh一天的时段早上, 下午, 晚上, 夜里ja一日の時間帯朝, 午後, 晩, 夜ko하루의 시간대아침, 오후, 저녁, 밤cardinal\_directions2—2encardinal directionsNorth, South, East, WestdeHimmelsrichtungenNorden, Süden, Osten, Westenfrpoints cardinauxnord, sud, est, ouestespuntos cardinalesnorte, sur, este, oestezh基本方位北, 南, 东, 西ja基本方位北, 南, 東, 西ko방위북, 남, 동, 서primary\_colors1—2enprimary colors of lightred, green, bluedePrimärfarben des LichtsRot, Grün, Blaufrcouleurs primaires de la lumièrerouge, vert, bleuescolores primarios de la luzrojo, verde, azulzh光的三原色红, 绿, 蓝ja光の三原色赤, 緑, 青ko빛의 삼원색빨강, 초록, 파랑
## Appendix MAdditional Redundancy Analysis

To assess the equivalence of the latent sets identified by the three selection methods, we compute representative per\-layer activation space directions that correspond to the decoder directions of these layer\-wise latents, then calculate the pairwise cosine similarity between per\-method directions\. If the latents of two selection methods for the same language are identical or equivalent in their representation in a certain layer, we expect their corresponding residual stream directions to be similar \(large cosine similarity\), while a small cosine similarity in a certain layer might indicate divergence\. We report in Table[32](https://arxiv.org/html/2609.00325#A13.T32)for each pair of selection methods the minimum cosine similarity, as well as the across\-layer average, for the language latents ofGemma\-2\-2BandQwen3\-4B\.

Table 32:Minimum and average cosine similarity between representative layer\-wise activation space directions per language forGemma\-2\-2B\(top\) andQwen3\-4B\(bottom\)\.esenzhdejafrkoGemma\-2\-2BAnnSel \- ValSelAvg0\.660\.060\.580\.590\.710\.450\.30Min0\.47\-0\.10\.440\.410\.530\.07\-0\.07FreqSel \- AnnSelAvg0\.570\.040\.590\.630\.470\.570\.41Min0\.09\-0\.150\.250\.240\.180\.120\.05ValSel \- FreqSelAvg0\.340\.040\.450\.290\.420\.250\.30Min\-0\.1\-0\.370\.110\.070\.08\-0\.05\-0\.44Qwen3\-4BAnnSel \- ValSelAvg0\.070\.260\.470\.340\.350\.30\.62Min\-0\.07\-0\.220\.06\-0\.040\.02\-0\.040\.31FreqSel \- AnnSelAvg0\.410\.370\.380\.480\.550\.320\.35Min\-0\.020\.040\.020\.230\.110\.000\.06ValSel \- FreqSelAvg0\.420\.140\.210\.270\.460\.360\.35Min\-0\.03\-0\.17\-0\.32\-0\.010\.00\-0\.15\-0\.35

Finally, we analyze the Neuronpedia annotations of the identified latent sets as a \(potentially noisy\) means to discern latent set overlap\. We compute as a proxy the percentage of language latents whose Neuronpedia annotation refers explicitly to the language\. While this is 100% forAnnSelby design, the results forValSelandFreqSelin[Table33](https://arxiv.org/html/2609.00325#A13.T33)reveal substantial variability\.

Table 33:The percentage \(%\) of extracted latents ofGemma\-2\-2B\(top\) andQwen3\-4B\(bottom\) per method and language whose Neuronpedia annotations explicitly refer to the language\.AnnSelis 100% by design\.enfrdeeszhkojaGemma\-2\-2BValSel272945458246FreqSel0\.171775339158AnnSel100100100100100100100Qwen3\-4BValSel22432241208FreqSel1\.132\.938\.228\.437\.54\.116\.7AnnSel100100100100100100100