Distribution-aware Language Neuron Identification in Multilingual Large Language Models

# Distribution-aware Language Neuron Identificationin Multilingual Large Language Models
Source: [https://arxiv.org/html/2609.10993](https://arxiv.org/html/2609.10993)
Inho WonAffiliation:KAISTEmail:[ktlim@kaist\.ac\.kr](mailto:[email protected])Junghun YukAffiliation:KAISTDongyeon KimAffiliation:KAISTJihyo Kim†Affiliation:KAIST InnoCORE PRISM\-AI CenterKyungTae Lim†Affiliation:KAIST

###### Abstract

Multilingual large language models \(mLLMs\) contain a small fraction of feed\-forward neurons that are sensitive to particular languages, commonly termed language\-specific neurons\. Existing work measures language specificity using the entropy of each neuron’s language\-wise probabilities of being active, where a neuron is considered active when its activation value is positive\. However, this approach may not fully capture the multilingual nature of mLLMs, where language representations are distributional and mutually related\. We proposeDistribution\-aware Language Neuronselection, which leverages pairwise relationships between per\-language activation distributions over the full activation range, including negative values\. Specifically, we quantify each neuron’s language specificity by clustering languages using pairwise overlap coefficients between their activation distributions\. Across two mLLMs and two held\-out corpora, our identifier more effectively isolates language\-specific causal effects, yielding up to 4\.9×\\timeshigher on\-target language damage per neuron while preserving off\-target language performance\.

22footnotetext:Corresponding Authors## 1Introduction

Multilingual large language models \(mLLMs\) exhibit strong capabilities across dozens of languages within a single shared parameter space\([Scao et al\., 2022](https://arxiv.org/html/2609.10993#bib.bib23);[Yang et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib8)\)\. However, how these models internally organize multilingual representations remains an open question\.

![Refer to caption](https://arxiv.org/html/2609.10993v1/figure1.png)Figure 1:Identifying structurally distinctive language distributions matters more than identifying the most frequently active language\. Here,aadenotes the activation value\. A distributional view reveals distinctions that activation probabilities alone miss\.One empirical approach to this question is neuron\-level interpretability based on activation values\. Previous studies identify language\-specific neurons \(LSNs\) as neurons that exhibit preferential activation for particular languages, based on activation profiles that distinguish one language from the others\([Tang et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib3);[Kojima et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib13);[Zhang et al\., 2026](https://arxiv.org/html/2609.10993#bib.bib4)\)\. They consider neurons with positive activation values to be activated for each language and quantify language specificity using the entropy of per\-language activation probabilities\. In this view, neurons exhibiting language specificity can be conceptualized as “neurons that activate predominantly for a single language\.” These identified neurons provide a useful analytic tool for studying the mechanisms of cross\-lingual transfer, intervening on language generation, and probing multilingual knowledge organization\([Rahmanisa et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib21);[Mondal et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib24);[Deng et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib19)\)\.

However, previous LSN identification methods may not fully reflect the multilingual nature of mLLMs\. Since mLLMs are trained on multiple languages under a shared parameter space, language representations are inherently distributional and mutually related\. Binarizing continuous activations and treating them as separate per\-language distributions introduces two critical limitations: first, it discards informative signals from negative activations since neurons with negative activation values are treated as inactive; second, because activation probabilities are computed from a binary state separately for each language, they do not preserve the unique shape of per\-language activation distributions and the relational structure among languages\.

Figure[1](https://arxiv.org/html/2609.10993#S1.F1)provides an example where the previous activation probability methods are insufficient to capture the language specificity\. As shown in the left plot, even when all activation values are positive, previous methods may fail to identify this neuron as language\-specific because the entropy remains high despite distinct activation distributions\. Furthermore, the right plot shows a scenario in which focusing solely on activation probabilities may fail to capture the structural distinctiveness: in this example, English forms an isolated distribution in a region that previous methods would treat as inactive\. These structural blind spots raise a fundamental question: given the multilingual nature of mLLMs, how can we identify language neurons by leveraging the activation distribution and its relational structure across languages while avoiding potential information loss?

To address this question, we proposeDistribution\-aware Language Neuron\(DLN\) selection, which estimates neuron\-level language specificity from pairwise overlap coefficients between per\-language activation distributions\. By modeling the full range of activation values and clustering languages based on their pairwise overlap coefficient, DLN captures the relational structure among activation distributions\. This formulation naturally identifies two types of language neurons, \(1\)*single\-language neurons*\(SLNs\), specific to one language, and \(2\)*multi\-language neurons*\(MLNs\), specific to multiple languages\. Consequently, rather than focusing solely on neurons that activate predominantly for a single language, our method reframes the neurons to be identified as “neurons whose activation distributions differ across languages\.”

Through evaluations on Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib7)\)and SmolLM3\-3B\-Base\([Bakouch et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib9)\)over two held\-out corpora, with additional validation on three further models in the appendix, we show that DLN more effectively isolates language\-specific causal effects than activation\-probability baselines\. Ablating DLN\-selected SLNs yields up to 4\.9×\\timeshigher on\-target language damage per neuron while preserving off\-target performance\. DLN also captures MLNs, uncovering shared structures such as a Chinese–Japanese coalition that yields 7\.12×\\timesgreater within\-cluster damage while using 2\.1×\\timesfewer neurons\. By modeling activation densities, DLN identifies neurons operating in the negative activation regime, including English SLNs that are invisible to positive\-rate metrics yet act as functional triggers for target\-language control\. Finally, DLN remains effective beyond the standard gate\-activation site, maintaining high selectivity at the gate\-up product site where activation\-probability baselines lose much of their effectiveness\. Our contributions are as follows:

- •A distribution\-aware language neuron identification framework\.By formulating neuron selection as a clustering problem using overlap coefficients of continuous activation densities, our method effectively leverages the distributional and inter\-language relational structures reflecting the nature of mLLMs\.
- •Effectiveness across SLNs and MLNs\.We evaluate the effectiveness of the neurons identified by our method for two types of language neurons\. SLNs significantly outperform the baselines in isolating language\-specific causal effects, while MLNs identify structurally coupled language coalitions\.
- •What full activation distributions reveal\.Analysis results demonstrate that our method achieves its design objectives: \(1\) capturing the importance of neurons identified in the negative activation regime, and \(2\) enabling flexible intervention locations by considering the full range of activation values\.

## 2Related Work

mLLMs share parameters across many languages while preserving language identity in generation\([Conneau et al\., 2020](https://arxiv.org/html/2609.10993#bib.bib14);[Grattafiori et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib7);[Yang et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib8)\)\. Prior work suggests that languages occupy partially overlapping representational subspaces\([Pires et al\., 2019](https://arxiv.org/html/2609.10993#bib.bib15);[Wu and Dredze, 2019](https://arxiv.org/html/2609.10993#bib.bib16)\)and that decoder\-only mLLMs may internally rely on English\-centric pivot representations\([Wendler et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib17);[Schut et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib20)\)\. These findings motivate a mechanistic question: how are language\-specific and language\-shared computations implemented inside a single mLLM?

![Refer to caption](https://arxiv.org/html/2609.10993v1/framework.png)Figure 2:Overall framework of the proposed method\.Neuron\-level analyses provide one way to study this question\. Prior work shows that individual neurons can encode factual, syntactic, semantic, or task\-relevant information in Transformer models\([Geva et al\., 2021](https://arxiv.org/html/2609.10993#bib.bib18);[Dai et al\., 2022](https://arxiv.org/html/2609.10993#bib.bib5);[Stanczak et al\., 2022](https://arxiv.org/html/2609.10993#bib.bib6)\)\. Extending this line to multilingual models,[Kojima et al\. \(2024\)](https://arxiv.org/html/2609.10993#bib.bib13)identify language\-specific neurons \(LSNs\) by measuring how precisely each neuron distinguishes a target language from others\.[Tang et al\. \(2024\)](https://arxiv.org/html/2609.10993#bib.bib3)introduce language activation probability entropy \(LAPE\), which measures language specificity from the entropy of per\-language activation probabilities, treating neurons with positive activation values as active\. Based on LAPE,[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.10993#bib.bib4)categorize neurons as LSNs, language\-related neurons \(LRNs\), and general neurons based on how many languages exceed a predefined activation\-probability threshold\.111LSNs and LRNs are closely related to our notions of single\-language neurons \(SLNs\) and multi\-language neurons \(MLNs\), respectively\. We use SLNs and MLNs to emphasize whether the identified neurons are associated with a single language or shared across multiple languages\.

However, these methods mainly rely on binary activation statistics, i\.e\., whether and how often a neuron is active for each language\. This abstraction can miss distributional distinctions: languages with similar activation probabilities may still induce different activation\-value distributions, and vice versa\. In contrast, we analyze the full per\-language activation distribution, enabling the identification of both single\-language and multi\-language neurons from distributional activation patterns\.

## 3Distribution\-aware Language Neurons

In this section, we introduce*distribution\-aware language neuron \(DLN\)*identification, which leverages the full range of per\-language activation distributions and quantifies their relationships using pairwise overlap coefficients\. Previous methods reduce each per\-language activation distribution to a binary activation statistic based on the sign of activation values, although negative activations may contain useful information for determining language specificity\. Moreover, multiple languages can have nearly identical binary activation statistics while occupying clearly separable activation regimes\. Such neurons may still be language\-specific, but entropy\-based methods can fail to identify them\. Figure[2](https://arxiv.org/html/2609.10993#S2.F2)illustrates the overall framework of our method\.

### 3\.1Distribution Overlap Coefficient Matrix

We study autoregressive Transformer language models with Gated Linear Unit \(GLU\) feed\-forward blocks\([Shazeer, 2020](https://arxiv.org/html/2609.10993#bib.bib12)\)\. We index each multilayer perceptron \(MLP\) neuron by the pairj=\(ℓ,n\)j=\(\\ell,n\), whereℓ\\ellis its layer andnnis its position within the layer’s intermediate dimensiondffnd\_\{\\mathrm\{ffn\}\}\. Letdddenote the residual\-stream hidden dimension\. Given the gate projectionWgateℓ∈ℝdffn×dW\_\{\\mathrm\{gate\}\}^\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{ffn\}\}\\times d\}, the activation of neuronjjon inputx∈ℝdx\\in\\mathbb\{R\}^\{d\}is

aj​\(x\)=\[σ⁡\(Wgateℓ​x\)\]n,a\_\{j\}\(x\)\\;=\\;\\bigl\[\\sigma\(W\_\{\\mathrm\{gate\}\}^\{\\ell\}x\)\\bigr\]\_\{n\},whereσ⁡\(⋅\)\\sigma\(\\cdot\)is the activation function \(e\.g\., SiLU\)\.

We determine each neuron’s language specificity by measuring the overlap coefficients, the shared areas, between per\-language activation distributions\. Letℒ\\mathcal\{L\}denote the set of languages, andpjk​\(v\)p\_\{j\}^\{k\}\(v\)denote the continuous probability density function of the activation valuev=aj​\(x\)v=a\_\{j\}\(x\)for neuronjjunder languagek∈ℒk\\in\\mathcal\{L\}\. The pairwise overlap coefficient matrix𝐒j∈\[0,1\]\|ℒ\|×\|ℒ\|\\mathbf\{S\}\_\{j\}\\in\[0,1\]^\{\|\\mathcal\{L\}\|\\times\|\\mathcal\{L\}\|\}is calculated as:

𝐒j​\[k,k′\]=∫min⁡\(pjk​\(v\),pjk′​\(v\)\)​𝑑v\.\\mathbf\{S\}\_\{j\}\[k,k^\{\\prime\}\]=\\int\\min\\left\(p\_\{j\}^\{k\}\(v\),\\,p\_\{j\}^\{k^\{\\prime\}\}\(v\)\\right\)dv\.This summarizes the relationship among\{pjk​\(v\)\}\.\\\{p\_\{j\}^\{k\}\(v\)\\\}\.

The overlap coefficient is one of several bounded, shape\-sensitive statistics for comparing per\-language distributions, and we adopt it as the most intuitive and computationally cheapest choice\. Appendix[17](https://arxiv.org/html/2609.10993#A5.T17)shows that alternatives such as Jensen\-Shannon, squared Hellinger, and Kolmogorov\-Smirnov yield near\-identical neuron sets, whereas scale\-bearing metrics \(Wasserstein\-1, energy distance\([Székely and Rizzo, 2013](https://arxiv.org/html/2609.10993#bib.bib26)\)\) diverge\.

### 3\.2Cluster\-based Neuron Classification

To distinguish languages with distinct activation patterns, we partitionℒ\\mathcal\{L\}into two clusters,\(Aj,Ajc\)\(A\_\{j\},A\_\{j\}^\{c\}\), by minimizing the maximum between\-cluster overlap coefficient,maxk∈Aj,k′∈Ajc⁡𝐒j​\[k,k′\]\\max\_\{k\\in A\_\{j\},\\,k^\{\\prime\}\\in A\_\{j\}^\{c\}\}\\,\\mathbf\{S\}\_\{j\}\[k,k^\{\\prime\}\], over all non\-trivial bipartitions\. This procedure is equivalent to single\-linkage clustering\([Johnson, 1967](https://arxiv.org/html/2609.10993#bib.bib22)\)with the number of clustersK=2K=2\. This is a selection decision rather than a bimodality assumption, as𝐒j\\mathbf\{S\}\_\{j\}retains all pairwise relations and a neuron with three or more separated groups is still detected through its most\-separated bipartition\. Appendix[17](https://arxiv.org/html/2609.10993#A5.T17)validates this design together with alternative linkage methods\.

We defineAjA\_\{j\}to be the smaller cluster and designate it as the neuron’s candidate language set\. This follows from the assumption of our method design that languages with distinct activation patterns are those most strongly associated with the neuron\. The larger clusterAjcA\_\{j\}^\{c\}can be regarded as a reference group of languages with relatively similar activation behavior\.

### 3\.3Neuron Selection Criterion

After obtainingAjA\_\{j\}andAjcA\_\{j\}^\{c\}, we select neuronjjas a language neuron only if all between\-cluster overlap coefficients do not exceed a thresholdτ\\tau:

𝒩=\{j:maxk∈Aj,k′∈Ajc⁡𝐒j​\[k,k′\]≤τ\}\.\\mathcal\{N\}=\\left\\\{j:\\max\\limits\_\{k\\in A\_\{j\},k^\{\\prime\}\\in A\_\{j\}^\{c\}\}\\mathbf\{S\}\_\{j\}\[k,k^\{\\prime\}\]\\leq\\tau\\right\\\}\.Otherwise, if any between\-cluster pair has an overlap coefficient greater thanτ\\tau, the neuron is not selected\.

We denote each selected language neuron asj⋆∈𝒩\{j^\{\\star\}\}\\in\\mathcal\{N\}and categorize it according to the size of its language setAj⋆A\_\{j^\{\\star\}\}as follows:

- •\|Aj⋆\|=1\|A\_\{j^\{\\star\}\}\|=1:single\-language neuron \(SLN\),
- •\|Aj⋆\|\>1\|A\_\{j^\{\\star\}\}\|\>1:multi\-language neuron \(MLN\)\.

Because overlap coefficients vary across models, we set the thresholdτ\\tauto the bottomPP\-th percentile of overlap coefficients computed over all neurons\.

## 4Experiments

### 4\.1Experimental Settings

#### Baselines\.

We employ two language neuron identification methods using the entropy of activation probabilities, LAPE\([Tang et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib3)\)and[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.10993#bib.bib4), as our baselines\. Specifically, LAPE identifies LSNs by their defined score computed as:

LAPEj=−∑k∈ℒr~jklogr~jk,\\mathrm\{LAPE\}\_\{j\}\\;=\\;\-\\sum\_\{k\\in\\mathcal\{L\}\}\\tilde\{r\}\_\{j\}^\{k\}\\log\\tilde\{r\}\_\{j\}^\{k\},wherer~jk\\tilde\{r\}\_\{j\}^\{k\}is L1 normalizedrjk=ℙ⁡\(aj​\(x\)\>0\|k\)r\_\{j\}^\{k\}=\\mathbb\{P\}\\bigl\(a\_\{j\}\(x\)\>0\|k\\bigr\)acrosskk\.

[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.10993#bib.bib4)additionally incorporate the maximum activation probability,maxk∈ℒ⁡rjk\\max\_\{k\\in\\mathcal\{L\}\}r\_\{j\}^\{k\}, to quantify the strongest degree to which neuronjjis active across languages to improve language neuron selection\. Accordingly, they revise the LAPE score as follows:

scorej=−∑k∈ℒr~jklogr~jk−λmaxk∈ℒrjk,\\mathrm\{score\}\_\{j\}\\;=\\;\-\\sum\_\{k\\in\\mathcal\{L\}\}\\tilde\{r\}\_\{j\}^\{k\}\\log\\tilde\{r\}\_\{j\}^\{k\}\\;\-\\;\\lambda\\max\_\{k\\in\\mathcal\{L\}\}r\_\{j\}^\{k\},whereλ\\lambdais a balancing coefficient\. Theλ\\lambdafor each evaluated model can be found in Table[6](https://arxiv.org/html/2609.10993#A1.T6)in Appendix[A](https://arxiv.org/html/2609.10993#A1)\. In addition, they distinguish LRNs from LSNs as neurons involved in multiple languages\.222Hereafter, for clarity, we refer to LSNs identified by[Tang et al\. \(2024\)](https://arxiv.org/html/2609.10993#bib.bib3)as LAPE, and to LSNs and LRNs identified by[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.10993#bib.bib4)as LSN and LRN, respectively\.Specifically, neuronjjis considered an LRN if the number of languages for whichrjkr\_\{j\}^\{k\}exceeds the predefined threshold is greater than11\.

#### Models\.

We study two pretrained base models with Gated Linear Unit feed\-forward design\([Shazeer, 2020](https://arxiv.org/html/2609.10993#bib.bib12)\)and SiLU activation: Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib7)\)as our primary model, with SmolLM3\-3B\-Base\([Bakouch et al\., 2025](https://arxiv.org/html/2609.10993#bib.bib9)\)\.

#### Corpora and languages\.

For language neuron selection, we sample100100k tokens per language fromwikipedia\-20231101\. For evaluation, we use FLORES\+ devtest \([Team et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib10); FLORES\) and a held\-out Wikipedia split \(wikipedia\-20230601; WIKI\), up to1​M1\\text\{M\}tokens per language, disjoint from the calibration source\.

Following[Tang et al\. \(2024\)](https://arxiv.org/html/2609.10993#bib.bib3), we use seven languages spanning four scripts \(Latin, Han, Kanji, and Kana\) and five language families \(Indo\-European, Sino\-Tibetan, Austroasiatic, Austronesian, and Japonic\): English \(en\), Chinese \(zh\), French \(fr\), Spanish \(es\), Vietnamese \(vi\), Indonesian \(id\), and Japanese \(ja\)\.

#### Density approximation\.

To compute the continuous overlap integral in practice, we approximate the densitypjk​\(v\)p\_\{j\}^\{k\}\(v\)using a histogram\-based approach with400400inner bins\. The bins are defined on a per\-neuron adaptive range\[μj−3​σj,μj\+3​σj\]\[\\mu\_\{j\}\-3\\sigma\_\{j\},\\,\\mu\_\{j\}\+3\\sigma\_\{j\}\], whereμj\\mu\_\{j\}andσj\\sigma\_\{j\}are pooled estimates over the seven languages\.

#### Neuron intervention\.

For neuronj⋆j^\{\\star\}identified for targetkk, we replace itsaj⋆a\_\{j^\{\\star\}\}with the mean activation value over the non\-target languages\. This mean\-patching intervention aligns with our method: we treatpjkp\_\{j\}^\{k\}as primitive and therefore call for replacingaj⋆a\_\{j^\{\\star\}\}with a non\-target reference value estimated from the other languages, whereas the baseline methods operationalize language specificity through the binary eventaj⋆​\(x\)\>0a\_\{j^\{\\star\}\}\(x\)\>0, for which zero serves as a natural counterfactual\. Mean\-patching realizes this intervention at the first\-moment level while keeping the replacement within the neuron’s typical activation range\([Heimersheim and Nanda, 2024](https://arxiv.org/html/2609.10993#bib.bib11)\)\. We verified that mean\-patching does not adversely affect the baselines compared with zero\-ablation, as detailed in Figures[7](https://arxiv.org/html/2609.10993#A2.F7),[8](https://arxiv.org/html/2609.10993#A2.F8)in Appendix[B](https://arxiv.org/html/2609.10993#A2)\.

#### Selectivity score\.

Let𝒜\\mathcal\{A\}denote the collection of all target language clusters under evaluation\. ForA∈𝒜A\\in\\mathcal\{A\}, letAcA^\{c\}denote its complementℒ∖A\\mathcal\{L\}\\setminus A\. Mean\-patching its neuron subset𝒩A\\mathcal\{N\}\_\{A\}induces a negative log\-likelihood \(NLL\) increaseΔ​NLLk​\(𝒩A\)\\Delta\\mathrm\{NLL\}\_\{k\}\(\\mathcal\{N\}\_\{A\}\)on evaluation languagekk\. The selectivity of neuron set𝒩\\mathcal\{N\}is:

Sel⁡\(𝒩\):=∑A∈𝒜\[1\|A\|​∑k∈AΔ​NLLk​\(𝒩A\)\]∑A∈𝒜\[1\|Ac\|​∑k′∈AcΔ​NLLk′​\(𝒩A\)\]\.\\mathrm\{Sel\}\(\\mathcal\{N\}\):=\\frac\{\\sum\_\{A\\in\\mathcal\{A\}\}\\\!\\left\[\\frac\{1\}\{\|A\|\}\\sum\_\{k\\in A\}\\Delta\\mathrm\{NLL\}\_\{k\}\(\\mathcal\{N\}\_\{A\}\)\\right\]\}\{\\sum\_\{A\\in\\mathcal\{A\}\}\\\!\\left\[\\frac\{1\}\{\|A^\{c\}\|\}\\sum\_\{k^\{\\prime\}\\in A^\{c\}\}\\Delta\\mathrm\{NLL\}\_\{k^\{\\prime\}\}\(\\mathcal\{N\}\_\{A\}\)\\right\]\}\.For instance, in Figure[3](https://arxiv.org/html/2609.10993#S4.F3),Sel⁡\(𝒩\)\\mathrm\{Sel\}\(\\mathcal\{N\}\)is computed as the ratio between the diagonal sum and the off\-diagonal sum\. HigherSel⁡\(𝒩\)\\mathrm\{Sel\}\(\\mathcal\{N\}\)means the intervention concentrates damage on the target languagesAAand spares the off\-target languagesAcA^\{c\}\.

### 4\.2Assessing the Neuron Selectivity

We evaluate the effectiveness of our method from two perspectives: \(1\) for SLNs, whether it yields more language\-selective per\-language interventions than the baselines; and \(2\) for MLNs, whether our clustering\-based framework recovers cluster\-internal intervention effects missed by single\-language identifiers and concentrates these effects more strongly per neuron than LRN\. To examine these two perspectives, we analyze changes in NLL across languages after intervening on neurons associated with the target language\.

![Refer to caption](https://arxiv.org/html/2609.10993v1/main_fig3_sln_llama_zhang.png)Figure 3:Δ\\DeltaNLL by language on Llama\-3\.1\-8B for LAPE \(top\) and LSN \(middle\), and our SLN atP=1%P=1\\%\(bottom\) on WIKI \(left\) and FLORES \(right\)\.\|𝒩\|\|\\mathcal\{N\}\|represents the number of ablated neurons\.#### Single\-language neurons\.

Figure[3](https://arxiv.org/html/2609.10993#S4.F3)compares the per\-languageΔ​NLL\\Delta\\mathrm\{NLL\}matrices for LAPE, LSN, and SLN on Llama\-3\.1\-8B\. The three identifiers select different pool sizes \(4,5874\{,\}587/1,8141\{,\}814/2,9722\{,\}972\); we therefore report on\-target damage normalized by\|𝒩\{k\}\|\|\\mathcal\{N\}\_\{\\\{k\\\}\}\|alongside the rawΔ​NLL\\Delta\\mathrm\{NLL\}\. Three observations follow\. \(i\) Averaged over the seven languages, SLN’s on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}per neuron is4\.9×4\.9\\times\(WIKI\) and5\.4×5\.4\\times\(FLORES\) larger than that of LAPE, and3\.13\.1/3\.2×3\.2\\timeslarger than LSN\. \(ii\) The advantage is most pronounced for English, which both baselines fail to identify as a language\-specific subset: their English\-assigned subsets produce no measurable damage when ablated, whereas our SLN deliversΔ​NLLen​\(𝒩\{en\}\)=1\.47\\Delta\\mathrm\{NLL\}\_\{\\texttt\{en\}\}\(\\mathcal\{N\}\_\{\\\{\\texttt\{en\}\\\}\}\)=1\.47\(WIKI\) and1\.871\.87\(FLORES\)\. \(iii\) The exception is Chinese: thezhdiagonal is weak under all three identifiers \(0\.330\.33/0\.100\.10/0\.010\.01on WIKI\)\. Chinese activations are largely captured by the\{\\\{zh,ja\}\\\}multi\-language cluster, and the damage is recovered under the MLN setting\.

![Refer to caption](https://arxiv.org/html/2609.10993v1/main_fig4_mln_llama_zhang.png)Figure 4:Δ​NLL\\Delta\\mathrm\{NLL\}by cluster on Llama\-3\.1\-8B for LRN \(top\) and MLN atP=1%P\{=\}1\\%\(bottom\) on WIKI \(left\) and FLORES \(right\)\. Each row corresponds to one of the top\-4 most frequent MLN clusters under ablation, columns are the seven evaluation languages\.
#### Multi\-language neurons\.

Figure[4](https://arxiv.org/html/2609.10993#S4.F4)contrasts LRN with our MLN on the four most\-populated cluster compositions \(Table[1](https://arxiv.org/html/2609.10993#S4.T1)\)\. We make three observations\. \(i\) Although our MLN uses about2\.86×2\.86\\timesfewer neurons than LRN \(465465vs\.13291329\), the within\-cluster damage is substantially stronger: averaged on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}per neuron over the four clusters, our MLN delivers6\.85×6\.85\\times/7\.12×7\.12\\times\(WIKI / FLORES\) the damage of LRN\. \(ii\) The most notable case is Chinese, which all three single\-language level identifiers miss: 93\.3% of the Chinese\-associated neurons identified by the proposed method are classified as MLNs \(Table[22](https://arxiv.org/html/2609.10993#A9.T22)\)\.

Table 1:Number of neurons that LRN and our MLN assign to each of the four most frequent multi\-language sets on Llama\-3\.1\-8B \(P=1%P\{=\}1\\%\)\. Summed over the four sets, LRN admits1,3291\{,\}329neurons against our465465\.It is the only language where a single MLN cluster accounts for the majority, with the\{\\\{zh,ja\}\\\}cluster comprising 57\.6% of its language neurons\. Consequently, for the\{\\\{zh,ja\}\\\}cluster, the Chinese damage registersΔ​NLLzh​\(𝒩\{zh,ja\}\)=1\.24/1\.12\\Delta\\mathrm\{NLL\}\_\{\\texttt\{zh\}\}\\left\(\\mathcal\{N\_\{\\\{\\texttt\{zh,ja\}\\\}\}\}\\right\)=1\.24/1\.12on WIKI / FLORES, with within\-cluster Japanese damage0\.92/0\.890\.92/0\.89supporting a Chinese–Japanese coalition, reflecting their shared character\-level structure\. Chinese language neurons are thus not absent from the model but rather are organized as a multi\-language cluster, not as a single\-language set\. \(iii\) The \{fr,es\} and \{id,vi\} clusters contribute only marginal MLN signal: for most languages, the proportion of SLNs is higher than that of MLNs\. Specifically, the\{\\\{fr,es\}\\\}cluster accounts for only 13\.2% of thefr\-SLNs and 15\.4% of thees\-SLNs\. The\{\\\{id,vi\}\\\}cluster also represents a small proportion compared to SLNs, comprising just 2\.6% of theid\-SLNs and 7\.5% of thevi\-SLNs\. Therefore, these four languages are already captured at the single\-language level, as their SLN diagonals reachΔ​NLL≥0\.6\\Delta\\mathrm\{NLL\}\\geq 0\.6on WIKI and≥0\.7\\geq 0\.7on FLORES\. Additional comparisons of SLN and MLN distributions are provided in Appendix[I\.1](https://arxiv.org/html/2609.10993#A9.SS1)\.

Figure 5:Selectivity of SLN across the seven languages over percentilePP\. Baselines are matched to our neuron count at everyPP\.
#### Selectivity robustness\.

Figure[5](https://arxiv.org/html/2609.10993#S4.F5)reports selectivity score across the seven languages depending onPP, for two models on both corpora\. The selected neurons at percentilePPcomprise the lowestPP% of clustering candidates, jointly retaining single\-language \(\|Aj\|=1\|A\_\{j\}\|=1\) and multi\-language \(\|Aj\|∈\{2,3\}\|A\_\{j\}\|\\in\\\{2,3\\\}\) neurons\. We match the baseline neuron budgets to ours at everyPP\.

Two patterns hold consistently\. First, our selectivity exceeds both LAPE and LSN at everyPPin the sweep on every \(model, corpus\) combination: the gap over LAPE reaches an order of magnitude on SmolLM3\-3B\-Base \(2020–32×32\\timeson WIKI,1818–26×26\\timeson FLORES\) and stays at44–8×8\\timeson Llama\-3\.1\-8B, while the gap over LSN is smaller but consistent \(1\.11\.1–1\.9×1\.9\\timeson Llama\-3\.1\-8B and2\.62\.6–3\.5×3\.5\\timeson SmolLM3\-3B\-Base\)\. Second, selectivity decreases monotonically withPP\. Since percentile thresholding admits top ranked candidates first, smallerPPretains only the most concentrated subset, which is the expected behavior of a well\-ordered ranking\. Section[5\.2](https://arxiv.org/html/2609.10993#S5.SS2)further verifies that the reported on\-target damage is not a mechanical consequence of the mean\-patch displacement magnitude\. In an extended sweep toP=5%P\{=\}5\\%, selectivity remains an order of magnitude above the budget\-matched LAPE throughout \(Appendix[D\.6](https://arxiv.org/html/2609.10993#A4.SS6)\)\.

### 4\.3Effect on Downstream Tasks

The evaluation so far measures the damage of an intervention as a change in token\-level likelihood,Δ​NLL\\Delta\\mathrm\{NLL\}\. We now ask whether the same intervention also changes accuracy on multilingual downstream tasks\. We evaluate on Belebele\([Bandarkar et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib1)\), a multiple\-choice reading comprehension task \(900900questions per language\), and MGSM\([Shi et al\., 2023](https://arxiv.org/html/2609.10993#bib.bib2)\), a multilingual grade\-school math benchmark \(250250problems per language\), with prompts and scoring detailed in Appendix[G\.1](https://arxiv.org/html/2609.10993#A7.SS1)\. For each language neuron identification method and each target languagekkcovered by the task, we repeat the mean\-patch intervention, replacing the activations of𝒩\{k\}\\mathcal\{N\}\_\{\\\{k\\\}\}with their off\-target means, and report the accuracy change on every evaluation language relative to the unintervened model\.

Table 2:Accuracy drop under the mean\-patch intervention, averaged over target languages\.k′=kk^\{\\prime\}\{=\}kis the on\-target drop andk′≠kk^\{\\prime\}\{\\neq\}kthe mean off\-target drop\. Belebele covers all seven languages, MGSM only five \(en,zh,fr,es,ja\)\.Bolddenotes the largest on\-target drop\.Table[2](https://arxiv.org/html/2609.10993#S4.T2)shows that the NLL result carries over to both tasks\. Notably, our SLNs induce the largest on\-target drop in every \(model, task\) combination, while off\-target accuracy remains essentially unchanged under all three identifiers\. The damage is concentrated rather than indiscriminate: on Belebele, whose chance accuracy is25\.025\.0and whose unintervened accuracy is47\.047\.0/45\.245\.2\(Llama\-3\.1\-8B / SmolLM3\-3B\-Base\), our intervention removes22%22\\%/27%27\\%of the model’s above\-chance margin on the target language, yet costs no off\-target language more than2\.402\.40points\. Our larger on\-target drop is not an artifact of the number of neurons patched\. Although LAPE patches up to1\.9×1\.9\\timesas many neurons as we do, our SLNs still remove more on\-target accuracy per neuron than both baselines, by up to4\.0×4\.0\\timesover LAPE and7\.0×7\.0\\timesover LSN\. Distributional separation therefore identifies neurons whose causal effect extends beyond token\-level likelihood to task behavior\.

Appendix[G\.2](https://arxiv.org/html/2609.10993#A7.SS2)reports the results for each target language, where the Chinese entry is near zero, as in theΔ​NLL\\Delta\\mathrm\{NLL\}evaluation\. Our identifier classifies almost all of the Chinese language neurons as members of the\{\\\{zh,ja\}\\\}MLN cluster, leaving the Chinese SLN pool nearly empty, so the Chinese effect surfaces under the MLN setting of Section[4\.2](https://arxiv.org/html/2609.10993#S4.SS2)\.

## 5Analysis

### 5\.1A Comparison between LAPE and SLN

Table 3:Mean selectivity across the seven languages, obtained by mean\-patching the intersection and set\-difference neurons of LAPE and SLN\. Thek′=kk^\{\\prime\}=kandk′≠kk^\{\\prime\}\\neq kcolumns report the averageΔ​NLLk​\(𝒩\{k′\}\)\\Delta\\mathrm\{NLL\}\_\{k\}\\left\(\\mathcal\{N\}\_\{\\\{k^\{\\prime\}\\\}\}\\right\), andSel⁡\(𝒩\)\\mathrm\{Sel\}\(\\mathcal\{N\}\)the resulting selectivity\.Boldandunderlinedenote the highest and second\-highest, respectively\.We first examine the following question: What are the key differences between our single\-language neuron and the neurons identified by LAPE? To analyze these differences, we define and investigate the corresponding neuron pools and their sub\-relations: the intersection \(LAPE∩SLN\\text\{LAPE\}\\cap\\text\{SLN\}\), and the set differences \(LAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\}andSLN∖LAPE\\text\{SLN\}\\setminus\\text\{LAPE\}\)\.

#### Effect on selectivity\.

We evaluate the functional importance of these distinct neuron pools by measuring their causal impact on language selectivity\. As shown in Table[3](https://arxiv.org/html/2609.10993#S5.T3), selectivity is highest for the intersection \(LAPE∩SLN\\text\{LAPE\}\\cap\\text\{SLN\}\), followed bySLN∖LAPE\\text\{SLN\}\\setminus\\text\{LAPE\}andLAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\}\. This implies that neurons satisfying both methods are the most effective language neurons\. Notably,SLN∖LAPE\\text\{SLN\}\\setminus\\text\{LAPE\}delivers a2\.86×2\.86\\times\(Llama\-3\.1\-8B\) and4\.71×4\.71\\times\(SmolLM3\-3B\-Base\) higher on\-targetΔ​NLLk​\(𝒩\{k\}\)\\Delta\\text\{NLL\}\_\{k\}\(\\mathcal\{N\}\_\{\\\{k\\\}\}\)thanLAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\}\(0\.2950\.295vs\.0\.1030\.103;0\.7070\.707vs\.0\.1500\.150\), despite using roughly half as many neurons\.

\(a\)Only target positive\(b\)Only target negative\(c\)All positive\(d\)All negative
Figure 6:Four representative activation patterns of single language neurons across seven languages\.
#### Distributional diversity of SLNs\.

We analyze the distributional characteristics of the SLNs identified by our method\. As illustrated in Figure[6](https://arxiv.org/html/2609.10993#S5.F6), the selected neurons can be categorized into four patterns: \([6\(a\)](https://arxiv.org/html/2609.10993#S5.F6.sf1)\) only target positive, where only the target language has positive activations; \([6\(b\)](https://arxiv.org/html/2609.10993#S5.F6.sf2)\) only target negative, where only the target language has negative activations; \([6\(c\)](https://arxiv.org/html/2609.10993#S5.F6.sf3)\) all positive, where all languages have positive activations but the target language shows a distinct magnitude shift; and \([6\(d\)](https://arxiv.org/html/2609.10993#S5.F6.sf4)\) all negative, where all activations are negative but the target language remains distributionally separated\. Unlike indicator\-based methods, which are limited to the target\-positive pattern, our distribution\-aware approach successfully uncovers this broader spectrum of language neurons\. For instance, Table[8](https://arxiv.org/html/2609.10993#A2.T8)shows that 63–72% of English SLNs have a negative target\-language mean, making them entirely undetectable to positive\-rate metrics despite their functional relevance\.

Table 4:Language accuracy of steered output\. LAPE uses its native6161English neurons\. DLN splits its identified English neurons by mean\-activation sign on English inputs into𝒩\{en\}\+\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\}^\{\+\}\}\(68 neurons\) and𝒩\{en\}−\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\}^\{\-\}\}\(196 neurons\), and𝒩\{en\}​,topk−\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\\text\{,topk\}\}^\{\-\}\}denotes the top\-6868subset of𝒩\{en\}−\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\}^\{\-\}\}under the same selection ranking\.Boldandunderlinedenote the highest and second\-highest accuracy, respectively\.
#### Functional role of negative\-region neurons\.

To investigate what role these negative\-region neurons serve in generation, we partition English neurons by the sign ofμtgt\\mu\_\{\\text\{tgt\}\}, the mean activation on target\-language inputs, and steer generation with mean\-patching in both translation directions\. Source\-language neurons are patched to the off\-source mean and target\-language neurons are patched to the target mean, thereby specifying the direction of the language switch\. We evaluate 200 samples per language from each of the WIKI and FLORES datasets, yielding 400 samples per language in total\. To verify whether the steered output is emitted in the intended target language, we measure language accuracy, the fraction of generations whose top\-1 fastText prediction[Joulin et al\. \(2016\)](https://arxiv.org/html/2609.10993#bib.bib25)matches the target, with LAPE included as a baseline\.

The results in Table[4](https://arxiv.org/html/2609.10993#S5.T4)reveal a clear functional split\. When the target output is English,𝒩\{en\}\+\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\}^\{\+\}\}and LAPE achieve near\-perfect switching accuracies of0\.960\.96and0\.970\.97, respectively, while the negative\-region neurons are markedly weaker at0\.710\.71\. This pattern reverses when the target output is a non\-English languagekk: only the negative subset is effective\. Specifically,𝒩\{en\}−\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\}^\{\-\}\}drives the switch with average language accuracy0\.750\.75, whereas𝒩\{en\}\+\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\}^\{\+\}\}and LAPE are almost ineffective with accuracies of0\.150\.15and0\.010\.01, respectively\. This effect is not merely an artifact of set size, since𝒩\{en\}​,topk−\\mathcal\{N\_\{\\\{\\texttt\{en\}\\\}\\text\{,topk\}\}^\{\-\}\}, which is matched in size to the positive subset, retains strong performance of0\.760\.76, indicating that the negative\-region neurons drive the switch functionally rather than through the difference in set size\.

In other words, the positive\-region neurons are responsible for producing English output, whereas the negative\-region neurons form a functionally distinct circuit that switches away from an English context into another language\. Further details are provided in Appendix[D\.5](https://arxiv.org/html/2609.10993#A4.SS5)\. Whether such interventions can also improve task performance is a separate research problem, and[Mondal et al\. \(2025\)](https://arxiv.org/html/2609.10993#bib.bib24)report negative results for language\-specific neurons facilitating cross\-lingual transfer\. Our scope is the faithful identification of language neurons, validated causally and on downstream tasks, rather than performance enhancement\.

### 5\.2Displacement\-matched Control

Because mean\-patching moves each neuron by the distributional separation it was selected for, one may ask whether the large on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}merely reflects a large displacement\. To test this, we apply the intervention to neurons randomly sampled from outside the DLN\-selected pools and count\-matched on every layer\. Each is shifted from its target\-language mean by the same number of standard deviations as its matched SLN \(1\.61\.6–2\.2​σ2\.2\\sigma; 3 seeds\)\. This matched displacement produces only∼1%\{\\sim\}1\\%of the SLN effect \(mean on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}of0\.0130\.013vs\.1\.1981\.198\) and no language selectivity\. Its on\-target and off\-target damage coincide \(0\.0130\.013each\), whereas the SLN intervention leaves off\-target damage at only0\.0060\.006\. The pattern replicates on SmolLM3\-3B\-Base \(0\.0100\.010vs\.1\.3951\.395\)\. The reportedΔ​NLL\\Delta\\mathrm\{NLL\}thus reflects which neurons are patched, not how far they are moved\. Per\-language results are provided in Appendix[F](https://arxiv.org/html/2609.10993#A6)\.

### 5\.3Identification across GLU Sites

A GLU feed\-forward block exposes two natural signals: the gate activationaj​\(x\)=\[σ⁡\(Wgateℓ​x\)\]na\_\{j\}\(x\)=\[\\sigma\(W\_\{\\mathrm\{gate\}\}^\{\\ell\}x\)\]\_\{n\}\(theaa\-site\) and the gate\-up productzj​\(x\)=aj​\(x\)⊗uj​\(x\)z\_\{j\}\(x\)=a\_\{j\}\(x\)\\otimes u\_\{j\}\(x\)\(thezz\-site\), whereuj​\(x\)=\[Wupℓ​x\]nu\_\{j\}\(x\)=\[W\_\{\\mathrm\{up\}\}^\{\\ell\}x\]\_\{n\}\. Prior neuron\-level interpretability has focused on theaa\-site for its clean above\-zero criterion\([Tang et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib3);[Dai et al\., 2022](https://arxiv.org/html/2609.10993#bib.bib5);[Geva et al\., 2021](https://arxiv.org/html/2609.10993#bib.bib18)\), yet thezz\-site is the signal that is causally operative\. We therefore ask whether our distribution\-aware identifier extends to thezz\-site\. The two sites differ sharply in sign: pooled across neurons and the seven languages,aais negative\-dominant \(ℙ⁡\(a\>0\)≈0\.25\\mathbb\{P\}\(a\>0\)\\approx 0\.25\) whilezzis sign\-balanced \(ℙ⁡\(z\>0\)≈0\.50\\mathbb\{P\}\(z\>0\)\\approx 0\.50; Table[9](https://arxiv.org/html/2609.10993#A3.T9)\)\. This matters because baselines rank neurons by the entropy of activation probabilities, which requires the activation rate to vary across languages\. At theaa\-site this rate spans an approximately3×3\\timesgap between the most\- and least\-active languages, so the entropy is informative; at thezz\-site the sign\-balanced product flattens every language to≈0\.5\\approx 0\.5, collapsing this gap and saturating the entropy\.

Table 5:Mean selectivity across the seven languages on WIKI\. Each row varies the neuron\-selection site and the patching site\.Bolddenotes the highest selectivity\.Our identifier demonstrates robustness when extending the selection fromaato thezz\-site, retaining most of its efficacy with a decrease of2929–37%37\\%\. Consequently, at thezz\-site, our method maintains a substantial advantage, outperforming LAPE by2525–44×44\\timesand LSN by1515–21×21\\times\. In contrast, the baselines lose7474–90%90\\%of their selectivity under this transition, confirming that the performance drop is driven by the difficulty of selecting at thezz\-site, not by the change of intervention site\.

In the\(z​\-site,z​\-site\)\(z\\text\{\-site\},z\\text\{\-site\}\)row, the cross\-language mean spread atzz\-site is about4×4\\timesnarrower than ataa\-site \(per\-neuron median0\.0180\.018vs\.0\.0770\.077on Llama\-3\.1\-8B\), so mean\-patching substitutes a smaller magnitude; this drop is uniform across methods and leaves the\(z,a\)\(z,a\)ordering intact\. That our criterion survives the move tozzshows distribution\-aware identification applies at the causally operative signal\. This locus remains underexplored under the gate\-only convention and is a natural target for mechanistic analysis\.

## 6Conclusion

In this paper, we propose adistribution\-aware language neuronselection method, which considers the full range of activation values and the pairwise relationship among language activation distributions, to better reflect the multilingual nature of mLLMs\. Experimental results demonstrate that our method effectively selects neurons that influence multilingual abilities\. In addition, the identified MLNs form coherent language groups whose members exhibit relatively large performance degradation when intervened on\. Further analysis shows that language neurons selected from the negative activation regime play functional roles in language switching during generation\. Finally, we demonstrate that our method can also be applied to other feasible locations in the model, such as the gate\-up product \(zz\-site\) in addition to the standard gate activation\. Overall, our work demonstrates that comprehensively modeling activation distributions provides an effective framework for identifying language neurons in mLLMs\.

## Limitations

#### Limited language coverage\.

The experiments and language neuron selection sample only seven languages \(English, Chinese, French, Spanish, Vietnamese, Indonesian, and Japanese\)\. While these cover multiple scripts and language families, they represent only a tiny fraction of the languages that modern multilingual LLMs are trained on\.

#### Architectural restriction to GLU\.

A key limitation of this work is its exclusive focus on autoregressive Transformer models equipped with Gated Linear Unit \(GLU\) feed\-forward blocks with the SiLU activation function\. Consequently, it remains unexplored whether this distribution\-aware neuron identification generalizes effectively to models employing standard MLP architectures that lack this bifurcated signal structure\.

#### Intervention penalty at thezz\-site\.

When shifting identification and intervention to thezz\-site \(the gate\-up product\), we observe an intervention\-side penalty\. The cross\-language mean spread at this site is approximately four times smaller than at theaa\-site\. As a result, the mean\-patching technique substitutes a proportionally smaller magnitude, causing a uniform drop in intervention effectiveness across all evaluated methods\.

## Use of AI Assistants

We used an AI assistant \(Gemini\) to polish the writing of this paper, limited to grammar correction and sentence\-level wording suggestions\. All suggestions were reviewed and revised by the authors\.

## Acknowledgements

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(No\.RS2024\-00456709, A Development of Self\-Evolving Deepfake Detection Technology to Prevent the Socially Malicious Use of Generative AI\), and by the InnoCORE program of the Ministry of Science and ICT\(N10260002\)\. This work utilized GPU resources from the “Advanced GPU Utilization Support Program” funded by MSIT, Republic of Korea \(awarded to KyungTae Lim\)\.

## References

- Aryabumiet al\.\(2024\)V\. Aryabumi, J\. Dang, D\. Talupuru, S\. Dash, D\. Cairuz, H\. Lin, B\. Venkitesh, M\. Smith, K\. Marchisio, S\. Ruder, A\. Locatelli, J\. Kreutzer, N\. Frosst, P\. Blunsom, M\. Fadaee, A\. Üstün, and S\. HookerAya 23: open weight releases to further multilingual progress\.External Links:2405\.15032Cited by:[§D\.4](https://arxiv.org/html/2609.10993#A4.SS4.p1.1)\.
- Bakouchet al\.\(2025\)E\. Bakouch, L\. Ben Allal, A\. Lozhkov, N\. Tazi, L\. Tunstall, C\. M\. Patiño, E\. Beeching, A\. Roucher, A\. J\. Reedi, Q\. Gallouédec, K\. Rasul, N\. Habib, C\. Fourrier, H\. Kydlicek, G\. Penedo, H\. Larcher, M\. Morlon, V\. Srivastav, J\. Lochner, X\. Nguyen, C\. Raffel, L\. von Werra, and T\. WolfSmolLM3: smol, multilingual, long\-context reasoner\.Note:[https://huggingface\.co/blog/smollm3](https://huggingface.co/blog/smollm3)Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p6.1),[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px2.p1.1)\.
- Bandarkaret al\.\(2024\)L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. KhabsaThe belebele benchmark: a parallel reading comprehension dataset in 122 language variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 749–775\.External Links:[Link](https://aclanthology.org/2024.acl-long.44/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.44)Cited by:[§4\.3](https://arxiv.org/html/2609.10993#S4.SS3.p1.1)\.
- Conneauet al\.\(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Link](https://aclanthology.org/2020.acl-main.747/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p1.1)\.
- Daiet al\.\(2022\)D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. WeiKnowledge neurons in pretrained transformers\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 8493–8502\.External Links:[Link](https://aclanthology.org/2022.acl-long.581/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.581)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p2.1),[§5\.3](https://arxiv.org/html/2609.10993#S5.SS3.p1.1)\.
- Denget al\.\(2025\)B\. Deng, Y\. Wan, B\. Yang, Y\. Zhang, and F\. FengUnveiling language\-specific features in large language models via sparse autoencoders\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 4563–4608\.External Links:[Link](https://aclanthology.org/2025.acl-long.229/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.229),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p2.1)\.
- Gevaet al\.\(2021\)M\. Geva, R\. Schuster, J\. Berant, and O\. LevyTransformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5484–5495\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.446/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p2.1),[§5\.3](https://arxiv.org/html/2609.10993#S5.SS3.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p6.1),[§2](https://arxiv.org/html/2609.10993#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px2.p1.1)\.
- Heimersheim and Nanda \(2024\)S\. Heimersheim and N\. NandaHow to use and interpret activation patching\.arXiv preprint arXiv:2404\.15255\.Cited by:[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px5.p1.1)\.
- Johnson \(1967\)S\. C\. JohnsonHierarchical clustering schemes\.Psychometrika32\(3\),pp\. 241–254\.Cited by:[§3\.2](https://arxiv.org/html/2609.10993#S3.SS2.p1.1)\.
- Joulinet al\.\(2016\)A\. Joulin, E\. Grave, P\. Bojanowski, M\. Douze, H\. Jégou, and T\. MikolovFasttext\.zip: compressing text classification models\.arXiv preprint arXiv:1612\.03651\.Cited by:[§D\.5](https://arxiv.org/html/2609.10993#A4.SS5.p2.1),[§5\.1](https://arxiv.org/html/2609.10993#S5.SS1.SSS0.Px3.p1.1)\.
- Kojimaet al\.\(2024\)T\. Kojima, I\. Okimura, Y\. Iwasawa, H\. Yanaka, and Y\. MatsuoOn the multilingual ability of decoder\-based pre\-trained language models: finding and controlling language\-specific neurons\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 6919–6971\.External Links:[Link](https://aclanthology.org/2024.naacl-long.384/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.384)Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p2.1),[§2](https://arxiv.org/html/2609.10993#S2.p2.1)\.
- Mondalet al\.\(2025\)S\. K\. Mondal, S\. Sen, A\. Singhania, and P\. JyothiLanguage\-specific neurons do not facilitate cross\-lingual transfer\.InThe Sixth Workshop on Insights from Negative Results in NLP,A\. Drozd, J\. Sedoc, S\. Tafreshi, A\. Akula, and R\. Shu \(Eds\.\),Albuquerque, New Mexico,pp\. 46–62\.External Links:[Link](https://aclanthology.org/2025.insights-1.6/),[Document](https://dx.doi.org/10.18653/v1/2025.insights-1.6),ISBN 979\-8\-89176\-240\-4Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.10993#S5.SS1.SSS0.Px3.p3.1)\.
- Pireset al\.\(2019\)T\. Pires, E\. Schlinger, and D\. GarretteHow multilingual is multilingual BERT?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4996–5001\.External Links:[Link](https://aclanthology.org/P19-1493/),[Document](https://dx.doi.org/10.18653/v1/P19-1493)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p1.1)\.
- Rahmanisaet al\.\(2025\)I\. Rahmanisa, L\. M\. Andrylie, M\. K\. Ihsani, A\. F\. Wicaksono, H\. A\. Wibowo, and A\. F\. AjiUnveiling the influence of amplifying language\-specific neurons\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,K\. Inui, S\. Sakti, H\. Wang, D\. F\. Wong, P\. Bhattacharyya, B\. Banerjee, A\. Ekbal, T\. Chakraborty, and D\. P\. Singh \(Eds\.\),Mumbai, India,pp\. 919–968\.External Links:[Link](https://aclanthology.org/2025.findings-ijcnlp.55/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-ijcnlp.55),ISBN 979\-8\-89176\-303\-6Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p2.1)\.
- Scaoet al\.\(2022\)T\. L\. Scao, A\. Fan, C\. Akiki, E\. Pavlick, S\. Ilić, D\. Hesslow, R\. Castagné, A\. S\. Luccioni, F\. Yvon,et al\.Bloom: a 176b\-parameter open\-access multilingual language model\.arXiv preprint arXiv:2211\.05100\.Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p1.1)\.
- Schutet al\.\(2025\)L\. Schut, Y\. Gal, and S\. FarquharDo multilingual LLMs think in english?\.InICLR 2025 Workshop on Building Trust in Language Models and Applications,External Links:[Link](https://openreview.net/forum?id=I8BOtOPcOv)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p1.1)\.
- Shazeer \(2020\)N\. ShazeerGlu variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[§3\.1](https://arxiv.org/html/2609.10993#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px2.p1.1)\.
- Shiet al\.\(2023\)F\. Shi, M\. Suzgun, M\. Freitag, X\. Wang, S\. Srivats, S\. Vosoughi, H\. W\. Chung, Y\. Tay, S\. Ruder, D\. Zhou, D\. Das, and J\. WeiLanguage models are multilingual chain\-of\-thought reasoners\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fR3wGCk-IXp)Cited by:[§4\.3](https://arxiv.org/html/2609.10993#S4.SS3.p1.1)\.
- Stanczaket al\.\(2022\)K\. Stanczak, E\. Ponti, L\. Torroba Hennigen, R\. Cotterell, and I\. AugensteinSame neurons, different languages: probing morphosyntax in multilingual pre\-trained models\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 1589–1598\.External Links:[Link](https://aclanthology.org/2022.naacl-main.114/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.114)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p2.1)\.
- Székely and Rizzo \(2013\)G\. J\. Székely and M\. L\. RizzoEnergy statistics: a class of statistics based on distances\.Journal of Statistical Planning and Inference143\(8\),pp\. 1249–1272\.External Links:ISSN 0378\-3758,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.jspi.2013.03.018),[Link](https://www.sciencedirect.com/science/article/pii/S0378375813000633)Cited by:[§3\.1](https://arxiv.org/html/2609.10993#S3.SS1.p3.1)\.
- Tanget al\.\(2024\)T\. Tang, W\. Luo, H\. Huang, D\. Zhang, X\. Wang, W\. X\. Zhao, F\. Wei, and J\. WenLanguage\-specific neurons: the key to multilingual capabilities in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5701–5715\.Cited by:[Appendix B](https://arxiv.org/html/2609.10993#A2.p1.1),[§1](https://arxiv.org/html/2609.10993#S1.p2.1),[§2](https://arxiv.org/html/2609.10993#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px3.p2.1),[§5\.3](https://arxiv.org/html/2609.10993#S5.SS3.p1.1),[footnote 2](https://arxiv.org/html/2609.10993#footnote2)\.
- Teamet al\.\(2024\)N\. Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard, A\. Sun, S\. Wang, G\. Wenzek, A\. Youngblood, B\. Akula, L\. Barrault, G\. M\. Gonzalez, P\. Hansanti, J\. Hoffman, S\. Jarrett, K\. R\. Sadagopan, D\. Rowe, S\. Spruit, C\. Tran, P\. Andrews, N\. F\. Ayan, S\. Bhosale, S\. Edunov, A\. Fan, C\. Gao, V\. Goswami, F\. Guzmán, P\. Koehn, A\. Mourachko, C\. Ropers, S\. Saleem, H\. Schwenk, and J\. WangScaling neural machine translation to 200 languages\.Nature630\(8018\),pp\. 841–846\.External Links:[Link](https://doi.org/10.1038/s41586-024-07335-x),[Document](https://dx.doi.org/10.1038/s41586-024-07335-x)Cited by:[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px3.p1.1)\.
- Wendleret al\.\(2024\)C\. Wendler, V\. Veselovsky, G\. Monea, and R\. WestDo llamas work in English? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15366–15394\.External Links:[Link](https://aclanthology.org/2024.acl-long.820/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.820)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p1.1)\.
- Wu and Dredze \(2019\)S\. Wu and M\. DredzeBeto, bentz, becas: the surprising cross\-lingual effectiveness of BERT\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 833–844\.External Links:[Link](https://aclanthology.org/D19-1077/),[Document](https://dx.doi.org/10.18653/v1/D19-1077)Cited by:[§2](https://arxiv.org/html/2609.10993#S2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2609.10993#S1.p1.1),[§2](https://arxiv.org/html/2609.10993#S2.p1.1)\.
- Zhanget al\.\(2026\)S\. Zhang, Z\. Lai, X\. Liu, S\. She, X\. Liu, Y\. Gong, S\. Huang, and J\. ChenHow does alignment enhance llms’ multilingual capabilities? a language neurons perspective\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 34799–34807\.Cited by:[Appendix A](https://arxiv.org/html/2609.10993#A1.p1.1),[Appendix B](https://arxiv.org/html/2609.10993#A2.p1.1),[§1](https://arxiv.org/html/2609.10993#S1.p2.1),[§2](https://arxiv.org/html/2609.10993#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px1.p2.1),[footnote 2](https://arxiv.org/html/2609.10993#footnote2)\.

## Appendix AImplementation Details for the Baselines

[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.10993#bib.bib4)use a balancing coefficientλ\\lambdain their score \(see Baselines in Section[4\.1](https://arxiv.org/html/2609.10993#S4.SS1.SSS0.Px1)\)\. It controls the trade\-off between the entropy term, which favors neurons concentrated on a few languages, and the max\-rate term, which favors neurons with at least one strongly\-activating language\. Table[6](https://arxiv.org/html/2609.10993#A1.T6)provides the auto\-determinedλauto\\lambda\_\{\\text\{auto\}\}obtained by following their procedure\.

Table 6:λauto\\lambda\_\{\\text\{auto\}\}for each model\.
## Appendix BComparison under Different Interventions

We employ mean\-patching for neuron intervention because it naturally aligns with the properties of our method, as discussed in Section[4\.1](https://arxiv.org/html/2609.10993#S4.SS1)\. Since LAPE and[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.10993#bib.bib4)were originally evaluated with zero\-out interventions, which align with their method designs, we additionally compare LAPE and SLN under zero\-out intervention for a fair comparison\. The SLN set is identical to that used in the main experiment in Figure[3](https://arxiv.org/html/2609.10993#S4.F3), whereP=1%P=1\\%\. For LAPE, we use the bottom1%1\\%of the entropy as the threshold, following[Tang et al\. \(2024\)](https://arxiv.org/html/2609.10993#bib.bib3)\.

Table[7](https://arxiv.org/html/2609.10993#A2.T7)shows theΔ\\DeltaNLL results under zero\-out intervention\. For the zero\-out intervention, our method identifies more effective language neurons than LAPE in most cases, despite selecting fewer neurons\. As discussed in MLNs results in Section[4\.2](https://arxiv.org/html/2609.10993#S4.SS2), SLNs are less effective than LAPE under both intervention settings for languages with high coherence to other languages, such aszhandja\. However, as shown in Figures[7](https://arxiv.org/html/2609.10993#A2.F7),[8](https://arxiv.org/html/2609.10993#A2.F8), this is because our method separates SLNs and MLNs effectively\. For example, zeroing out Chinese\-specific neurons identified by LAPE also affects the model’s Japanese ability, whereas zeroing out Chinese SLNs identified by our method has little effect on Japanese\.

One notable observation is the difference inΔ\\DeltaNLL between the two interventions\. For languages excepten, both methods show relatively similarΔ\\DeltaNLL\. However, foren, SLN yields a much larger increase inΔ\\DeltaNLL under mean\-patching than under zero\-out intervention, whereas LAPE shows little change\. This suggests that English SLNs identified by our method encode language\-specific information not only through positive activation but also through distributional structure in the negative activation regime\. Mean\-patching more directly disrupts this structure by replacing English\-specific activations with non\-English values, whereas zero\-out may be insufficient when the relevant signal lies below zero\.

Table 7:On\-targetΔ​NLL\\Delta\\mathrm\{NLL\}on WIKI after intervening on language neurons identified by LAPE and SLN under each intervention setting\.Boldindicates the best result within each intervention setting\.Table 8:Per\-language activation profile of the LAPE and SLN sets \(wiki calibration\)\. For each set,μtgt\\mu\_\{\\text\{tgt\}\}is the per\-neuron mean activation under target\-language inputs andμoff\\mu\_\{\\text\{off\}\}is the mean over off\-target inputs\. The/σ/\\sigmacolumns normalize\|μtgt\|\|\\mu\_\{\\text\{tgt\}\}\|and the language gap\|μoff−μtgt\|\|\\mu\_\{\\text\{off\}\}\-\\mu\_\{\\text\{tgt\}\}\|by the neuron’s pooled standard deviation; %neg is the fraction of SLN neurons withμtgt<0\\mu\_\{\\text\{tgt\}\}<0\. For SLN\-en \(underlined\), the target distribution lies within∼0\.5​σ\\sim\\\!0\.5\\sigmaof zero with the majority of neurons having a negative target mean, while the language gap remains∼1\.5​σ\\sim\\\!1\.5\\sigma\. This is precisely the configuration under which zero is approximately no operation on target inputs but the off\-target mean is not\.Table[8](https://arxiv.org/html/2609.10993#A2.T8)locates this difference in the activation distribution itself\. For English SLNs, the per\-neuron mean activation under target inputs sits within∼0\.5​σ\\sim\\\!0\.5\\sigmaof zero, compared with1\.71\.7–2\.0​σ2\.0\\sigmafor every other \(model, language\) cell\. Indeed,6363–72%72\\%of SLN neurons have a negative target\-language mean\. Under this configuration, setting the activation to zero leaves it within its natural target\-language manifold and produces no detectable damage on target inputs; setting it to the off\-target mean, which is∼1\.5​σ\\sim\\\!1\.5\\sigmaaway, places it on a language\-foreign value and yields the full on\-target effect\. This is the empirical signature of language\-specificity that is encoded in distributional shift rather than in the binary eventaj​\(x\)\>0a\_\{j\}\(x\)\>0, and it is the structure that the indicator\-based identifier family is structurally blind to\.

Taken together, the zero\-ablation results function as a direct empirical confirmation of the identifier–intervention coupling of Section[4\.1](https://arxiv.org/html/2609.10993#S4.SS1): when our SLN set is evaluated with the LAPE\-native counterfactual, the activation\-rate\-encoded subset of language\-specificity is preserved, while the distribution\-encoded subset becomes invisible\. The full advantage of the distributional identifier is realized only under the matched intervention, as a consequence of two distinct kinds of language\-specificity coexisting in the model\. Figures[7](https://arxiv.org/html/2609.10993#A2.F7)–[8](https://arxiv.org/html/2609.10993#A2.F8)provide the full per\-languageΔ​NLL\\Delta\\mathrm\{NLL\}heatmaps under zero\-ablation\.

![Refer to caption](https://arxiv.org/html/2609.10993v1/compare_lape_vs_sln_llama.png)Figure 7:Per\-languageΔ​NLL\\Delta\\mathrm\{NLL\}under zero\-ablation on Llama\-3\.1\-8B: LAPE \(top\) vs SLN \(bottom\); WIKI \(left\), FLORES \(right\)\.![Refer to caption](https://arxiv.org/html/2609.10993v1/compare_lape_vs_sln_smollm.png)Figure 8:Per\-languageΔ​NLL\\Delta\\mathrm\{NLL\}under zero\-ablation on SmolLM3\-3B\-Base: LAPE \(top\) vs SLN \(bottom\); WIKI \(left\), FLORES \(right\)\.
## Appendix CSite\-comparison: Supplementary Results

This collects the empirical material supporting Section[5\.3](https://arxiv.org/html/2609.10993#S5.SS3): per\-layer sign rates that quantify the structural sign\-balance ofzz\(Table[9](https://arxiv.org/html/2609.10993#A3.T9)\), and the full selectivity grid across both held\-out corpora \(Table[10](https://arxiv.org/html/2609.10993#A3.T10)\)\.

#### Per\-layer sign rates\.

Table[9](https://arxiv.org/html/2609.10993#A3.T9)lists the mean activation\-positive rateP\(⋅\>0\)P\(\\cdot\>0\)per layer, pooled across the seven calibration languages, for the post\-gate signalaaand the gate\-up productzz\. At the z\-site every layer in both models satisfies\|P⁡\(z\>0\)−0\.5\|≤0\.002\|P\(z\>0\)\-0\.5\|\\leq 0\.002, so the sign statistic carries no cross\-layer or cross\-language signal\. At the a\-site the rate is consistently below0\.50\.5and varies widely with depth \(range\[0\.16,0\.48\]\[0\.16,0\.48\]on Llama\-3\.1\-8B,\[0\.18,0\.46\]\[0\.18,0\.46\]on SmolLM3\-3B\-Base\), reflecting SiLU’s negative\-side asymmetry\.

Table 9:Mean activation\-positive rateP\(⋅\>0\)P\(\\cdot\>0\)per layer, pooled across the seven calibration languages, on50,00050\{,\}000tokens per language sampled fromWIKI\-20231101\. The z\-site rate stays within0\.0020\.002of0\.50\.5across all layers in both models, while the a\-site rate spans an order of magnitude in dynamic range\.
#### Full selectivity grid\.

Table[10](https://arxiv.org/html/2609.10993#A3.T10)extends Table[5](https://arxiv.org/html/2609.10993#S5.T5)with FLORES values for all three \(select\-site, patch\-site\) configurations and all three identifiers, on both held\-out corpora\. Moving the selection fromaatozz, indicator baselines lose7474–94%94\\%of their selectivity across \(model, corpus\) combinations \(LAPE7474–86%86\\%; LSN8989–94%94\\%\), while our identifier retains5050–71%71\\%\. At\(z,a\)\(z,a\)our identifier is2525–55×55\\timesstronger than LAPE and1515–29×29\\timesstronger than LSN across all \(model, corpus\) combinations, of the same order as the main\-result a\-site gap\. The drop from\(z,a\)\(z,a\)to\(z,z\)\(z,z\)within each method follows the intervention\-side mechanical penalty discussed in Section[5\.3](https://arxiv.org/html/2609.10993#S5.SS3)and applies uniformly across identifiers; it does not change the ordering established by the\(z,a\)\(z,a\)comparison\.

Table 10:Full selectivity grid for the three identifiers across \(select\-site, patch\-site\) configurations and both held\-out corpora\.\(a,a\)\(a,a\)is the main\-result protocol;\(z,a\)\(z,a\)varies only the selection site \(the controlled comparison\);\(z,z\)\(z,z\)varies both\. Each method is evaluated at its nativeP=1%P\{=\}1\\%operating point\. Bold entries are highest within each row\.

## Appendix DFurther Analysis of DLN

### D\.1Detailed Comparison with LAPE and SLN

![Refer to caption](https://arxiv.org/html/2609.10993v1/heatmap_lape_vs_sln_llama.png)Figure 9:Δ\\DeltaNLL by language on Llama\-3\.1\-8B forLAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\}\(top\) andLAPE∩SLN\\text\{LAPE\}\\cap\\text\{SLN\}\(middle\), andSLN∖LAPE\\text\{SLN\}\\setminus\\text\{LAPE\}single\-language neuron atP=1%P=1\\%\(bottom\) on WIKI and FLORES\.![Refer to caption](https://arxiv.org/html/2609.10993v1/heatmap_lape_vs_sln_smollm.png)Figure 10:Δ\\DeltaNLL by language on SmolLM3\-3B\-Base forLAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\}\(top\) andLAPE∩SLN\\text\{LAPE\}\\cap\\text\{SLN\}\(middle\), andSLN∖LAPE\\text\{SLN\}\\setminus\\text\{LAPE\}single\-language neuron atP=1%P=1\\%\(bottom\) on WIKI and FLORES\.Figures[9](https://arxiv.org/html/2609.10993#A4.F9)and[10](https://arxiv.org/html/2609.10993#A4.F10)present heatmap results that decompose the three subsets \(LAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\},LAPE∩SLN\\text\{LAPE\}\\cap\\text\{SLN\},SLN∖LAPE\\text\{SLN\}\\setminus\\text\{LAPE\}\), which were summarized by mean selectivity in Table[3](https://arxiv.org/html/2609.10993#S5.T3), into 7×\\times7 cells of \(target language, evaluation language\)\.

Three patterns are consistently observed across the two models\. First,LAPE∩SLN\\text\{LAPE\}\\cap\\text\{SLN\}\(middle\) exhibits the non\-zero diagonal cells and near\-zero off\-diagonal cells, indicating that the core neurons jointly recognized by both identifiers have the most concentrated on\-target effect\. Second, although the neurons inSLN∖LAPE\\text\{SLN\}\\setminus\\text\{LAPE\}\(bottom\) lie outside the LAPE pool, the magnitude of their diagonal cells exceeds that of the diagonal cells inLAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\}\(top\) in nearly all languages\. The gap is most pronounced in the English \(en\) cell, which is consistent with the distributional observation in Section[5\.1](https://arxiv.org/html/2609.10993#S5.SS1)and Appendix[B](https://arxiv.org/html/2609.10993#A2)that the target\-off separation of English SLNs occurs in the negative region\. Third,LAPE∖SLN\\text\{LAPE\}\\setminus\\text\{SLN\}\(top\) contains the largest number of neurons among the three subsets, yet exhibits the weakest diagonal cells and a relatively broader off\-diagonal spread, showing that the peripheral neurons of the LAPE pool, which are selected solely on the basis of activation\-rate statistics, possess on average both a weaker signal and a more diffuse off\-target influence\.

### D\.2Impact of MLNs on Language\-specific Performance

![Refer to caption](https://arxiv.org/html/2609.10993v1/mln_nll_compare.png)Figure 11:Comparison ofΔ\\DeltaNLL scores for the four most frequently occurring MLN clusters across Llama\-3\.1\-8B, SmolLM3\-3B\-Base, and Qwen3\-4B\-Base\. The results are evaluated on WIKI and FLORES\.Figure[11](https://arxiv.org/html/2609.10993#A4.F11)presents theΔ\\DeltaNLL heatmaps for four of the most frequent MLN clusters across the three models\. The performance of the languages included in these MLN clusters generally deteriorated in all three models\.

Furthermore, a larger performance drop tends to occur when an MLN cluster accounts for a higher proportion of the neurons associated with a specific language\. For instance, the\{\\\{zh,ja\}\\\}cluster accounts for a substantial proportion of the neurons for bothzhandja, which leads to significant performance degradation\. Conversely, for the\{\\\{fr,es\}\\\}and\{\\\{id,vi\}\\\}clusters, the performance degradation is marginal because the proportion of SLNs for each language remains higher than that of the corresponding MLNs\.

However, the SmolLM3\-3B\-Base model presents a notable exception\. Despite accounting for a relatively low proportion of the language\-specific neurons, the\{\\\{id,vi\}\\\}MLN cluster still caused a significant performance degradation\. This phenomenon can be attributed to the fact that these languages are not among the model’s officially supported languages \(English, French, Spanish, German, Italian, and Portuguese\), resulting in relatively unstable signals compared to the target languages\. In other words, for unsupported languages, crucial information may heavily rely on a small subset of neurons, suggesting that MLNs likely encompass the neurons essential for processing these languages\. Consequently, this demonstrates that the nature and importance of the information encoded within MLNs can vary depending on the model and its training data\. Therefore, to fully comprehend language\-specific mechanisms, an in\-depth analysis of MLNs, alongside SLNs, is essential\.

We scaled up the above experiment to a larger model, revealing a consistent trend\. Figure[13](https://arxiv.org/html/2609.10993#A4.F13)presents theΔ\\DeltaNLL heatmap for the four most frequently observed MLN clusters in Llama\-3\.1\-70B\. For the\{\\\{zh,ja\}\\\}cluster, which accounts for a high proportion of MLNs in its respective languages, a substantial performance drop is observed\. Conversely, the\{\\\{fr,es\}\\\}cluster, characterized by a lower MLN and higher SLN proportion, exhibits a relatively minor degradation\. A similar pattern is evident in the\{\\\{en,fr,es\}\\\}cluster, where the MLN share within each language is too minimal to cause performance drop\. These findings demonstrate that our proposed methodology and the observed MLN dynamics are not constrained by model scale, highlighting their broad applicability\.

### D\.3Scaling to Llama\-3\.1\-70B

![Refer to caption](https://arxiv.org/html/2609.10993v1/heatmap_sln_70b.png)Figure 12:Δ\\DeltaNLL by language on Llama\-3\.1\-70B for LAPE, LSN, and SLN atP=1%P=1\\%on WIKI and FLORES\.![Refer to caption](https://arxiv.org/html/2609.10993v1/heatmap_mln_70b.png)Figure 13:Comparison ofΔ\\DeltaNLL scores for the four most frequently occurring MLN clusters of Llama\-3\.1\-70B\. The results are evaluated on WIKI and FLORES\.Figure[12](https://arxiv.org/html/2609.10993#A4.F12)reports the SLN comparison on Llama\-3\.1\-70B atP=1%P\{=\}1\\%\. The findings from the smaller models carry over\. Our SLN concentrates far larger on\-target damage than LAPE and LSN while using less than half the neurons of LAPE, and its off\-target damage stays near zero\. English, where both baselines remain weak \(0\.230\.23and0\.150\.15on WIKI\), is again recovered only by SLN \(1\.491\.49on WIKI,1\.761\.76on FLORES\)\. The one weak diagonal, Chinese, mirrors the 8B models: its signal is organized in the\{\\\{zh,ja\}\\\}MLN cluster \(Figure[13](https://arxiv.org/html/2609.10993#A4.F13)\)\.

### D\.4Generalization to Aya\-23\-8B

We run the full pipeline on Aya\-23\-8B\([Aryabumi et al\., 2024](https://arxiv.org/html/2609.10993#bib.bib27)\), an instruction\-tuned model trained explicitly for multilinguality, under the identical protocol \(P=1%P\{=\}1\\%; LAPE4,5874\{,\}587/ LSN1,6181\{,\}618/ SLN3,8003\{,\}800neurons\)\. Table[11](https://arxiv.org/html/2609.10993#A4.T11)reports per\-language on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}\. All findings replicate: our SLN produces the largest on\-target damage on all seven languages and both corpora, while its mean off\-target damage remains the lowest \(0\.0060\.006vs\. LAPE’s0\.0120\.012on WIKI\), yielding mean selectivity254254against133133\(LSN\) and2424\(LAPE\) on WIKI, and691691against350350and3838on FLORES\.

The multi\-language structure is likewise stable\. The\{\\\{zh,ja\}\\\}coalition is again the largest MLN cluster \(355355neurons\), followed by\{\\\{fr,es\}\\\}and\{\\\{id,vi\}\\\}, and Chinese is the one language whose SLN diagonal is weak \(0\.250\.25\); the union pool recovers it \(2\.362\.36on WIKI\), with the extra damage concentrated on its cluster\-mate Japanese, exactly as the framework predicts\. Since no base checkpoint of Aya\-23 is released, these results show that the method transfers to post\-trained models\.

Table 11:Per\-language on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}on Aya\-23\-8B atP=1%P\{=\}1\\%\.Bolddenotes the largest damage per \(language, corpus\)\.
### D\.5Generation Validation via Cross\-lingual Steering

To analyze the causal impact of the identified language\-specific neurons on text generation, we employ activation steering during inference\. We apply this steering to the neurons extracted by each respective methodology, utilizing a paired intervention technique that simultaneously manipulates two distinct sets of neurons: source SLNs and target SLNs\. For the source SLNs, we suppress their language\-specific signals by applying mean\-patching \(Section[4\.1](https://arxiv.org/html/2609.10993#S4.SS1)\), which replaces their activations with a language\-neutral baseline\. However, to ensure a fair comparison with LAPE, zero\-patching, which deactivates the neurons by setting their values to zero, is applied to LAPE instead of mean\-patching\. Concurrently, the target language neurons are activated by injecting their pre\-computed average activation values specific to the target language\. When applying this technique to MLNs, any overlap between source and target neurons is resolved by prioritizing the target neuron activation\. Ultimately, this dual intervention effectively spoofs the model’s internal state, mimicking the natural activation patterns in the target language\.

Table 12:Performance comparison of LAPE, SLN, and SLN\+MLN on the X\-to\-English \(X2en\) transfer task\. The table reports accuracy and target language probability \(pt​g​tp\_\{tgt\}\) across three models\. Zero\-patching is applied to LAPE, whereas mean\-patching is utilized for our proposed SLN and SLN\+MLN interventions\.Boldindicates the highest accuracy\.For the experimental setup, we extract 200 samples per language from the FLORES dataset\. To ensure rigorous language verification, we utilize the fastText language identification model[Joulin et al\. \(2016\)](https://arxiv.org/html/2609.10993#bib.bib25), selecting only those samples where the predicted probability for the target language is 0\.98 or higher\.

Table[12](https://arxiv.org/html/2609.10993#A4.T12)presents the results of transferring text from a non\-English source languageXXto English as the target language\. The performance discrepancies are marginal, with an average accuracy difference of 0\.027 and a maximum per\-language difference of 0\.066\. This robust performance stems from the models being predominantly pre\-trained on English\. The models frequently default to generating English responses even when provided with non\-English inputs, naturally leading to consistently high baseline scores\.

To evaluate cross\-lingual transfer capabilities toward a non\-primary language, we conduct experiments transferring from a source languageXXto Chinese \(zh\) as the target language\. As demonstrated in Table[13](https://arxiv.org/html/2609.10993#A4.T13), DLN consistently outperforms LAPE across all scenarios\. Notably, a substantial performance gap is observed between SLNs and MLNs\. As analyzed in Appendix[I\.1](https://arxiv.org/html/2609.10993#A9.SS1)and[D\.2](https://arxiv.org/html/2609.10993#A4.SS2), this gap arises because Chinese\-specific textual features are predominantly processed within MLNs\.

Table 13:Performance comparison of LAPE, SLN, and SLN\+MLN on the X\-to\-Chinese \(X2zh\) transfer task\. The table reports accuracy and target language probability \(pt​g​tp\_\{tgt\}\) across three models\. Zero\-patching is applied to LAPE, whereas mean\-patching is utilized for our proposed SLN and SLN\+MLN interventions\.Boldindicates the highest accuracy\.Table[14](https://arxiv.org/html/2609.10993#A4.T14)demonstrates the performance when transferring from English as the source language to all other target languages\. In stark contrast to the X\-to\-English transfer task, the performance of LAPE approaches zero\. Conversely, DLN maintains solid performance, with accuracy ranging from 0\.599 to 0\.952 depending on the architecture\. This phenomenon is primarily driven by the negative regions analyzed in Section[5\.1](https://arxiv.org/html/2609.10993#S5.SS1.SSS0.Px3)\. However, our method exhibits lower performance for Vietnamese \(vi\) and Indonesian \(id\) within the SmolLM3\-3B architecture, which stems from unstable internal signals provided by the model, as discussed in Appendix[D\.2](https://arxiv.org/html/2609.10993#A4.SS2)\. This explanation is supported by the results of Llama\-3\.1\-8B, which underwent extensive multilingual pre\-training and consistently delivers stable, high\-tier performance\.

Table 14:Performance comparison of LAPE, SLN, and SLN\+MLN on the English\-to\-X \(en2X\) transfer task\. The table reports accuracy and target language probability \(pt​g​tp\_\{tgt\}\) across three models\. Zero\-patching is applied to LAPE, whereas mean\-patching is utilized for our proposed SLN and SLN\+MLN interventions\.Boldindicates the highest accuracy\.
### D\.6Extended Percentile Sweep

Table 15:Extended percentile sweep of the DLN pool on SmolLM3\-3B\-Base \(WIKI\)\. LAPE is matched to our neuron budget\|𝒩\|\|\\mathcal\{N\}\|at everyPP\.Figure[5](https://arxiv.org/html/2609.10993#S4.F5)sweeps the percentile threshold overP∈\[0\.2,1\.0\]P\\in\[0\.2,1\.0\]\. Here we extend the sweep toP=5%P\{=\}5\\%on SmolLM3\-3B\-Base \(WIKI\) under the identical protocol, evaluating the DLN pool with the LAPE budget matched to ours at everyPP\. As Table[15](https://arxiv.org/html/2609.10993#A4.T15)shows, our selectivity remains an order of magnitude above LAPE \(99–14×14\\times\) at every extended operating point\.

The mild decline of the raw on\-target damage \(0\.40→0\.290\.40\\rightarrow 0\.29\) is a composition effect rather than a weakening of the method: higherPPadmits additional three\-language MLN clusters with small per\-cluster damage \(13→3313\\rightarrow 33clusters\), which dilutes the unweighted cluster mean in the selectivity numerator\. Within the SLN stratum, whose cluster count is fixed at seven, on\-target damage instead rises monotonically \(1\.47→1\.641\.47\\rightarrow 1\.64\) while off\-target damage stays below0\.020\.02nats\. The percentile choice is therefore not load\-bearing anywhere in the extended range\.

### D\.7Histogram Bin\-count Ablation

Table 16:Bin\-count ablation on Llama\-3\.1\-8B \(P=1%P\{=\}1\\%\)\. Jaccard is measured against the paper’s400400\-bin selection; time is the CPU cost of the overlap computation and selection\. Target\-language assignments agree at100%100\\%for every bin count\.Our identifier approximates each per\-language density with a400400\-bin adaptive histogram \(Section[4\.1](https://arxiv.org/html/2609.10993#S4.SS1)\)\. To probe this choice, we rebuild the histogram cache at800800bins on Llama\-3\.1\-8B and derive every coarser resolution by aggregating adjacent bins, which is exact because the per\-neuron bin edges nest; each resolution is then passed through the identical selection atP=1%P\{=\}1\\%\. As Table[16](https://arxiv.org/html/2609.10993#A4.T16)shows, the selection is essentially invariant from5050to800800bins, a16×16\\timesrange\. The thresholdτ\\taumoves by less than0\.0020\.002, the SLN pool stays within Jaccard0\.9960\.996of the paper’s, and every selected neuron keeps the same target\-language assignment at every resolution, including1010bins\. Only at1010bins does the pool itself drift \(Jaccard0\.920\.92\), as coarse bins begin to blur the near\-zero negative\-region structure that distinguishes English neurons\. Since the bin count affects only the post\-forward overlap computation, which takes seconds on CPU, the resolution can be reduced substantially at no cost to the selection\.

## Appendix ERobustness of the Identifier Design

Table 17:Effect of the pairwise statistic and linkage rule on the selected pool \(Llama\-3\.1\-8B,P=1%P=1\\%\)\. Jac\. is the Jaccard overlap with our overlap\+\+single set;Δ​NLL\\Delta\\mathrm\{NLL\}is the on\-target damage and Sel\. the selectivity, each on WIKI \(W\) and FLORES \(F\)\. The last two rows truncate the scale\-bearing pools to our pool size by top\-ranked score \(neuron\-matched control\)\.We examine three design choices of our identifier on Llama\-3\.1\-8B atP=1%P\{=\}1\\%\. These are the pairwise overlap coefficient𝐒j\\mathbf\{S\}\_\{j\}\(Section[3\.2](https://arxiv.org/html/2609.10993#S3.SS2)\), the single\-linkage bipartitioning, and the fixed cluster countK=2K\{=\}2\. For a general pairwise distance, we take the most\-separated bipartition and threshold at the top\-PPpercentile\. Table[17](https://arxiv.org/html/2609.10993#A5.T17)reports the pool size\|𝒩\|\|\\mathcal\{N\}\|, its Jaccard overlap with our set, and the on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}and selectivity on WIKI / FLORES\.

#### Bounded statistics and linkage agree\.

Every bounded, shape\-sensitive statistic selects essentially the same neuron set\. Jensen–Shannon, squared Hellinger, and Kolmogorov–Smirnov stay within3%3\\%of our pool size, with Jaccard0\.920\.92–0\.990\.99and selectivity within∼10%\\sim\\\!10\\%on both corpora\. They also preserve the multi\-language structure, with Jensen–Shannon recovering650650MLNs against our654654under the same leading coalitions\. The linkage rule is equally immaterial \(Jaccard0\.960\.96–0\.990\.99across single/average/complete\), and we keep single linkage because it is exactly the stated max\-overlap criterion\.

#### Scale\-bearing metrics reward magnitude, not discriminability\.

Wasserstein\-1 and the energy distance instead inflate the pool \(1\.61\.6–1\.9×1\.9\\times\), fall to Jaccard0\.470\.47–0\.580\.58, roughly halve selectivity, and distort the MLN structure\. This is not merely over\-selection but a property of the ranking itself\. Truncating each pool to our size by top\-ranked score makes them worse rather than better \(0\.620\.62vs\.0\.960\.96on\-targetΔ​NLL\\Delta\\mathrm\{NLL\}for W1 on WIKI\)\. Their ranking places the largest\-variance neurons first\.

#### Why the overlap coefficient\.

Among these interchangeable statistics, we adopt the overlap coefficient for interpretability and cost\. It equals one minus the total variation distance,𝐒j​\[k,k′\]=1−TV⁡\(pjk,pjk′\)\\mathbf\{S\}\_\{j\}\[k,k^\{\\prime\}\]=1\-\\mathrm\{TV\}\(p\_\{j\}^\{k\},p\_\{j\}^\{k^\{\\prime\}\}\), soτ\\taureads directly as a bound on shared probability mass, and it is the cheapest to compute over all458458k neurons \(1010s on CPU, against9696s for Jensen–Shannon\)\.

#### TheK=2K\{=\}2bipartition is a selection decision, not a bimodality assumption\.

For every neuron we recompute the selection statistic inside the larger reference clusterAjcA\_\{j\}^\{c\}of its most\-separated bipartition, so a genuinely three\-group neuron clearsτ\\tauon this residual split as well\. The empirical structure is overwhelmingly two\-group\. The median residual\-split overlap is0\.780\.78among selected neurons and0\.920\.92among rejected ones, nearly twice the thresholdτ=0\.40\\tau\{=\}0\.40, and only5050of the3,6263\{,\}626selected neurons \(1\.4%1\.4\\%\) admit a threshold\-clearing residual split, with linguistically coherent nestings such as\{\\\{zh,ja\}\\\}then\{\\\{en\}\\\}\. Moreover, none of the455,126455\{,\}126rejected neurons admits a three\-group structure, which follows from the criterion itself, since three mutually separated groups already provide a bipartition whose cross\-cluster overlap clearsτ\\tau\. Such neurons are selected through their most\-separated split and labeled by its minority side\. A natural refinement is to choose the number of clusters by within\-cluster cohesion at additional computational cost, which we leave to future work\.

## Appendix FDisplacement\-matched Random Control

Table 18:Displacement\-matched random control on Llama\-3\.1\-8B \(WIKI,P=1%P\{=\}1\\%\)\.ddis the normalized displacement of the matched SLN set; Rand patches random layer\- and count\-matched neurons by the samedd, Rand×12\\,\{\\times\}\\frac\{1\}\{2\}by half\. Ratio is SLN over Rand on\-target damage\. The mean ratio is the ratio of the mean damages\.For each SLN identified for target languagekk, we measure its normalized displacement under mean\-patching, i\.e\., the gap between its mean activation on the target language and its mean over the non\-target languages, in units of its pooled standard deviation\. We then draw random neurons outside every selected pool and the LAPE pool, matched to the SLN set in layer and count, and shift each away from its own target\-language mean by the same normalized amount and direction \(3 seeds; we report the mean\)\. A*half\-displacement*condition repeats this at half the magnitude as a dose–response probe\. Table[18](https://arxiv.org/html/2609.10993#A6.T18)reports per\-language results on Llama\-3\.1\-8B \(WIKI,P=1%P\{=\}1\\%\)\.

Three observations follow\. \(i\) The same displacement applied at random locations yields∼1%\{\\sim\}1\\%of the SLN effect, so displacement magnitude does not mechanically produce the reported damage\. \(ii\) The control has no language selectivity: its on\-target and off\-target damage are identical \(0\.0130\.013vs\.0\.0130\.013\), i\.e\., random displacement acts as diffuse noise, while the SLN intervention concentrates its effect on the target language with off\-target damage \(0\.0060\.006\) below the control’s noise floor\. \(iii\) Halving the displacement leaves the control equally inert \(0\.0060\.006\), ruling out a dose–response account\. The smallest ratio, Chinese, is small for both interventions \(only2727Chinese SLNs atP=1%P\{=\}1\\%\); as in Section[4\.2](https://arxiv.org/html/2609.10993#S4.SS2), the Chinese signal resides in the\{\\\{zh,ja\}\\\}MLN cluster\. All three observations replicate on SmolLM3\-3B\-Base: control0\.0100\.010vs\. SLN1\.3951\.395on\-target, per\-language ratios2323–378×378\\times, and no dose response \(0\.0050\.005at half displacement\)\.

## Appendix GDownstream Evaluation

### G\.1Experimental Settings

#### Belebele\.

Each question is scored zero\-shot\. We build one prompt per answer option,

> \{passage\}Q: \{question\}A: \{option\}

and take the summed log\-probability of the\{option\}tokens as its score, predicting the highest\-scoring option\.

#### MGSM\.

Each problem is prompted with the88chain\-of\-thought exemplars from the dataset’s own training split for that language, so the exemplars, their reasoning, and the question prefixes are all in the evaluation language\. The prompt is

where each question sits on the line above its answer and the blocks are separated by a blank line, so the trailing answer prefix leaves the model to produce the reasoning\. Table[19](https://arxiv.org/html/2609.10993#A7.T19)lists the prefixes per language\. We decode greedily for at most512512new tokens, stopping at a blank line or at a new question prefix, and take the last number in the generation as the predicted answer, matched exactly against the gold value\.

Table 19:MGSM prompt prefixes for the five languages MGSM covers\.

### G\.2Per\-language Downstream Results

MethodenzhfresviidjaAvgLlama\-3\.1\-8B\|𝒩\{k\}\|\|\\mathcal\{N\}\_\{\\\{k\\\}\}\|LAPE617158226194421057871–LSN28107481318105540235–SLN18827780616199915247–BelebeleLAPE0\.333\.330\.671\.444\.562\.676\.222\.75LSN0\.001\.560\.560\.563\.440\.891\.561\.22SLN6\.560\.115\.007\.116\.782\.446\.564\.94MGSMLAPE0\.408\.000\.802\.00––23\.606\.96LSN−\-0\.803\.200\.804\.80––14\.004\.40SLN12\.00−\-0\.4011\.608\.80––15\.209\.44SmolLM3\-3B\-Base\|𝒩\{k\}\|\|\\mathcal\{N\}\_\{\\\{k\\\}\}\|LAPE45657539431547916827–LSN16119296207201375229–SLN15244475425357566235–BelebeleLAPE0\.111\.781\.561\.006\.892\.003\.892\.46LSN0\.00−\-0\.330\.780\.565\.441\.331\.001\.25SLN3\.22−\-0\.227\.899\.008\.334\.785\.895\.56MGSMLAPE3\.600\.000\.001\.20––13\.203\.60LSN0\.401\.60−\-0\.403\.60––−\-1\.600\.72SLN10\.000\.405\.209\.20––14\.007\.76

Table 20:Per\-language on\-target accuracy drop under the mean\-patch intervention, expanding Table[2](https://arxiv.org/html/2609.10993#S4.T2)\. The\|𝒩\{k\}\|\|\\mathcal\{N\}\_\{\\\{k\\\}\}\|block gives the number of neurons each identifier assigns to target languagekk; these pools are shared between the two tasks, so the MGSM totals in Table[2](https://arxiv.org/html/2609.10993#S4.T2)sum only over the five languages MGSM covers\.Bolddenotes the largest drop per column\.Table[20](https://arxiv.org/html/2609.10993#A7.T20)breaks Table[2](https://arxiv.org/html/2609.10993#S4.T2)down by target language\. English separates the methods most sharply\. Both baselines assign English a small subset,6161and2828neurons on Llama\-3\.1\-8B and4545and1616on SmolLM3\-3B\-Base, and patching it costs at most0\.330\.33points on Belebele and3\.603\.60on MGSM, whereas our English SLNs remove6\.566\.56and12\.0012\.00points on Llama\-3\.1\-8B and3\.223\.22and10\.0010\.00on SmolLM3\-3B\-Base\. This reproduces on task accuracy the English gap reported forΔ​NLL\\Delta\\mathrm\{NLL\}in Section[4\.2](https://arxiv.org/html/2609.10993#S4.SS2), and it is the downstream counterpart of the negative\-region analysis in Section[5\.1](https://arxiv.org/html/2609.10993#S5.SS1):6363–72%72\\%of the English SLNs patched here have a negative target\-language mean and therefore lie in the region that positive\-rate metrics discard by construction\.

Chinese is the one target where our SLNs fall below LAPE, at0\.110\.11versus3\.333\.33on Belebele and−0\.40\-0\.40versus8\.008\.00on MGSM for Llama\-3\.1\-8B\. This mirrors theΔ​NLL\\Delta\\mathrm\{NLL\}exception in Section[4\.2](https://arxiv.org/html/2609.10993#S4.SS2)and follows from the same property of our identifier: almost all Chinese language neurons are assigned to the\{\\\{zh,ja\}\\\}multi\-language cluster, leaving2727neurons on Llama\-3\.1\-8B and4444on SmolLM3\-3B\-Base as single\-language, so MLNs account for93\.3%93\.3\\%and88\.1%88\.1\\%of the Chinese language neurons \(Table[22](https://arxiv.org/html/2609.10993#A9.T22)\)\. The Chinese SLN column therefore patches these2727and4444neurons, not the pool that carries the Chinese effect; that pool is the MLN cluster, where mean\-patching𝒩\{zh,ja\}\\mathcal\{N\}\_\{\\\{\\texttt\{zh\},\\texttt\{ja\}\\\}\}yieldsΔ​NLLzh=1\.24\\Delta\\mathrm\{NLL\}\_\{\\texttt\{zh\}\}=1\.24on WIKI and1\.121\.12on FLORES\.

Outside Chinese, a baseline exceeds our raw on\-target drop in22of the2020remaining \(model, task, language\) cells: Indonesian on Belebele with Llama\-3\.1\-8B \(2\.672\.67versus2\.442\.44\) and Japanese on MGSM with Llama\-3\.1\-8B \(23\.6023\.60versus15\.2015\.20\)\. In both, LAPE patches a larger pool,10571057against915915neurons and871871against247247, and our per\-neuron drop is still larger, by1\.06×1\.06\\timesand2\.27×2\.27\\times\. The averages in Table[2](https://arxiv.org/html/2609.10993#S4.T2)therefore do not rest on a few target languages\.

on\-target\(k′=kk^\{\\prime\}\{=\}k\)off\-target\(k′≠kk^\{\\prime\}\{\\neq\}k\)MethodbbccppbbccppLlama\-3\.1\-8BBelebeleLAPE392219×10−122\\\!\\times\\\!10^\{\-12\}4093810\.34LSN282205×10−46\\\!\\times\\\!10^\{\-4\}1091180\.60SLN612301×10−254\\\!\\times\\\!10^\{\-25\}1531610\.69MGSMLAPE16275×10−82\\\!\\times\\\!10^\{\-8\}2972530\.07LSN11358×10−53\\\!\\times\\\!10^\{\-5\}90750\.28SLN21294×10−111\\\!\\times\\\!10^\{\-11\}1241150\.61SmolLM3\-3B\-BaseBelebeleLAPE402247×10−91\\\!\\times\\\!10^\{\-9\}5104580\.10LSN282203×10−44\\\!\\times\\\!10^\{\-4\}1131100\.89SLN685335×10−283\\\!\\times\\\!10^\{\-28\}2281730\.007MGSMLAPE12782×10−32\\\!\\times\\\!10^\{\-3\}2632600\.93LSN79700\.51701020\.02SLN17679×10−91\\\!\\times\\\!10^\{\-9\}1111310\.22

Table 21:Two\-sided exact McNemar tests on the item\-level outcomes behind Table[2](https://arxiv.org/html/2609.10993#S4.T2), pooled over target languages\.bbcounts items answered correctly by the unintervened model and incorrectly after the intervention, andcccounts the reverse, sob\>cb\>cindicates a drop\.Underlinemarks the three entries discussed in the text\.Because the accuracy differences in Table[2](https://arxiv.org/html/2609.10993#S4.T2)are small in absolute terms, we also test them at the item level\. Table[21](https://arxiv.org/html/2609.10993#A7.T21)reports two\-sided exact McNemar tests on the discordant pairs between the unintervened and intervened runs, pooled over target languages\. Every on\-target drop reachesp<0\.01p<0\.01except LSN on MGSM with SmolLM3\-3B\-Base \(p=0\.51p=0\.51\), whose0\.720\.72\-point entry in Table[2](https://arxiv.org/html/2609.10993#S4.T2)is therefore not distinguishable from no effect\. Our SLNs give the smallestpp\-values in all four settings, atp≤×10−25p\\leq 4\\\!\\times\\\!10^\{\-25\}on Belebele andp≤×10−9p\\leq 1\\\!\\times\\\!10^\{\-9\}on MGSM\.

Ten of the twelve off\-target entries are not significant at the0\.050\.05level\. The two exceptions are our SLNs on Belebele with SmolLM3\-3B\-Base \(p=0\.007p=0\.007\), whose magnitude is0\.150\.15accuracy points, and LSN on MGSM with SmolLM3\-3B\-Base \(p=0\.02p=0\.02withc\>bc\>b, that is, off\-target accuracy rises\)\. Both are likely overstated: unlike the on\-target test, where each target language contributes a disjoint item set, the off\-target test reuses the same evaluation items under the six or four other target\-language interventions, so the pooled discordant pairs are not independent and the test is anti\-conservative\.

## Appendix HConcrete Examples of Activation Distribution

In this section, we illustrate the four representative activation patterns presented in Figure[6](https://arxiv.org/html/2609.10993#S5.F6)\. Figures[14](https://arxiv.org/html/2609.10993#A8.F14)–[17](https://arxiv.org/html/2609.10993#A8.F17)display four SLN samples corresponding to each case as violin plots across the seven calibration languages, with each inset indicating the neuron’s location\(L,N\)\(L,N\), the target language, and whether it is identified as SLN and/or LAPE\.

Case 1 \(Figure[14](https://arxiv.org/html/2609.10993#A8.F14)\) is the canonical form in which only the target language activates in the positive region while the remaining languages are distributed in the negative region, and both identifiers identify the same neuron as a target\-language SLN\. Cases 2–4\(Figure[15](https://arxiv.org/html/2609.10993#A8.F15)–[17](https://arxiv.org/html/2609.10993#A8.F17)\) are all SLN\-only samples that LAPE fails to identify: case 2 is a sign\-inverted form in which the target language lies in the negative region and the remaining languages lie in the positive region; case 3 is a form in which all languages reside in the positive region but the target language is separated only by magnitude; and case 4 is a form in which all languages are distributed in the negative region but only the target language is separated by magnitude\.

Figure 14:Four SLN samples for case 1 \(rest negative / target positive\)\. Both identifiers agree on the target\-language SLN; inset box reports\(L,N,target language,Ours / LAPE verdict\)\(L,N,\\text\{target language\},\\text\{Ours / LAPE verdict\}\)\.Figure 15:Four SLN samples for case 2 \(rest positive / target negative\)\. SLN assigns the target\-language SLN while LAPE misses — the sign reversal leaves the normalized activation\-rate entropy near\-uniform; inset box reports\(L,N,target language,Ours / LAPE verdict\)\(L,N,\\text\{target language\},\\text\{Ours / LAPE verdict\}\)Figure 16:Four SLN samples for case 3 \(rest positive / target positive, different shape\)\. SLN assigns the target\-language SLN while LAPE misses — all languages cleara\>0a\>0, so the activation\-rate statistic carries no separating signal; inset box reports\(L,N,target language,Ours / LAPE verdict\)\(L,N,\\text\{target language\},\\text\{Ours / LAPE verdict\}\)\.Figure 17:Four SLN samples for case 4 \(rest negative / target negative, different shape\)\. SLN assigns the target\-language SLN while LAPE excludes these neurons from its pool — no language clears the max\-rate thresholdτ\\tau; inset box reports\(L,N,target language,Ours / LAPE verdict\)\(L,N,\\text\{target language\},\\text\{Ours / LAPE verdict\}\)
## Appendix IDetailed Neuron Population

### I\.1Language Population in SLN, MLN

Table 22:Per\-language counts of SLNs and MLNs, on Llama\-3\.1\-8B, SmolLM3\-3B\-Base, and Qwen3\-4B\-Base\.\|⋅\|\|\\cdot\|denote the number of neurons associated with each language\. Ratio=\|MLN\|\|SLN\|\+\|MLN\|=\\frac\{\|\\mathrm\{MLN\}\|\}\{\|\\mathrm\{SLN\}\|\+\|\\mathrm\{MLN\}\|\}gives the share of MLNs among all language neurons\.Figure 18:Language distribution of SLN and MLN neurons across different languages in three language models\. The top\-nnMLN denotes the MLN cluster with thenn\-th highest frequency among all MLN clusters that include the corresponding language\.Figure[18](https://arxiv.org/html/2609.10993#A9.F18)illustrates the number of SLNs and MLNs for each language across three models: Llama\-3\.1\-8B, SmolLM3\-3B\-Base, and Qwen3\-4B\-Base\. The “top\-nnMLN” denotes thenn\-th most frequent cluster among the MLNs associated with a given language\. The overall distributions of SLNs and MLNs exhibit similar patterns across all three models\. For SLNs,idaccounts for the highest proportion, whereaszhexhibits the lowest distribution in all models\. Notably, for Chinese \(zh\), the proportion of MLNs is higher than that of SLNs across all models, with the\{\\\{zh,ja\}\\\}cluster accounting for the largest share among the MLN populations\. This indicates that Chinese and Japanese share the most information within these models\.

The distributions of the most frequent MLN clusters, such as\{\\\{fr,es\}\\\},\{\\\{vi,zh,ja\}\\\}, and\{\\\{id,vi\}\\\}, are also consistent across the models\. These results demonstrate that the proposed methodology is model\-agnostic\. Furthermore, this demonstrates that our proposed methodology enables the analysis of linguistic similarities and relational dynamics, such as those between Chinese and Japanese\.

While the\{\\\{en,fr,es\}\\\}cluster is the most frequent among English MLNs in the other two models, the\{\\\{en,zh\}\\\}cluster is the most frequent in the Qwen3\-4B\-Base model\. This discrepancy can be attributed to the Qwen3\-4B\-Base model being heavily specialized in Chinese compared to the others, treating it as a major language with a representation hierarchy comparable to English\.

### I\.2Layer\-wise Neuron Population

In this section, we examine the layer\-wise distributions and differences of the neurons identified by the previous works and our proposed approach\. Similar to LAPE, both methodologies exhibit a U\-shaped distribution, where the identified neurons are concentrated in the early and late layers\. A key difference is that the neurons selected by our proposed methodology are more densely clustered in the extreme early and late layers compared to LAPE\. Furthermore, MLNs are rarely observed in the initial layers but become increasingly prevalent in deeper layers\. Detailed layer\-wise statistics are provided in Tables[23](https://arxiv.org/html/2609.10993#A9.T23)and[24](https://arxiv.org/html/2609.10993#A9.T24)\.

Table 23:Per\-layer neuron\-population counts for Llama\-3\.1\-8B atP=1%P\\\!=\\\!1\\%\. Each cell is the number of neurons of the given population in that layer; set\-difference and intersection columns are over \(layer, neuron\) pairs\.Table 24:Per\-layer neuron\-population counts for SmolLM3\-3B\-Base atP=1%P\\\!=\\\!1\\%\. Each cell is the number of neurons of the given population in that layer; set\-difference and intersection columns are over \(layer, neuron\) pairs\.