Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models

# Training Data Granularity Determines Parametric Modularity in Large Language Models
Source: [https://arxiv.org/html/2608.10214](https://arxiv.org/html/2608.10214)
## Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models

Marcus Armstrong Navid Ayoobi Arjun Mukherjee Department of Computer Science University of Houston Houston, TX 77204 \{miarmstr, nyoobi\}@cougarnet\.uh\.edu, amukher6@central\.uh\.edu

###### Abstract

Do large language models contain domain\-specific parametric shells: concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others? We apply a uniform causal methodology across two domain granularities, three model families \(1\.5B to 7B parameters\), and eight domains\. At the academic subject level, zero neurons exceed 60% domain selectivity across 939,008 combined FFN neurons and causal damage matrices are flat, despite domain identity being linearly decodable above 85% accuracy\. At the language and modality level, 0\.65–1\.14% of neurons exceed 60% selectivity, damage matrices are near\-perfectly diagonal \(ratios up to 595:1\), and shell neuron sets are essentially disjoint \(IoU<0\.003<0\.003\)\. Masking code\-selective neurons reduces mathematical reasoning accuracy by 16–24 percentage points across all models; masking Spanish or Chinese neurons leaves it at or below random\. Shell strength increases monotonically with scale and shells are spatially interleaved in a pattern that precludes group\-level selective quantization\. Parametric shells form where and only where training data was modular at the token level\.

## 1Introduction

Large language models \(LLMs\) process dozens of languages and thousands of subject domains within a single set of parameters, yet how this capacity is parametrically organized remains poorly understood\. Prior work has attempted to identify language\-specific neurons in multilingual LLMs:Tanget al\.\([2024](https://arxiv.org/html/2608.10214#bib.bib1)\)propose LAPE to locate neurons by differential activation likelihood, butLe and Li \([2026](https://arxiv.org/html/2608.10214#bib.bib2)\)subsequently show that activation preference does not imply functional necessity — LAPE\-identified neurons achieve LangSpec\-F1 near zero under direct causal intervention\.Le and Li \([2026](https://arxiv.org/html/2608.10214#bib.bib2)\)address this with CRANE, a relevance\-attribution framework with substantially stronger causal effects, but evaluate a single model family across three languages without a null condition against which to assess when parametric shells exist and when they do not\.

This paper addresses that open question\. We apply a uniform causal methodology across two domain granularities, three model families \(1\.5B to 7B parameters\), and eight domains: four academic subject categories and four language and modality domains \(English, code, Spanish, Chinese\)\. For each domain, we capture mean pre\-activation magnitudes across the FFN intermediate dimension, identify domain\-selective neurons via a share\-based metric with backbone exclusion, and measure causal damage matrices net of a count\-matched random baseline\. The results reveal a sharp granularity boundary: zero neurons exceed 60% domain selectivity at the subject level across 939,008 combined FFN neurons, while 0\.65–1\.14% exceed this threshold at the language level with near\-perfectly diagonal damage matrices and shell IoU below 0\.003\. Parametric shells form where and only where training data was modular at the token level\.

We make the following contributions:

1. 1\.A granularity boundary for parametric modularity\.We establish that language and modality domains produce functional parametric shells while academic subject domains do not, despite equivalent linear decodability, contextualizing prior work by identifying the conditions under which shells form\.
2. 2\.Shell concentration and functional load\.We provide the first systematic Pareto analysis of language shell structure, showing that 2% of neurons carries approximately 94% of maximum domain\-specific capability loss across all four language domains and all three model families\.
3. 3\.Scale analysis of shell effects\.Diagonal dNLL increases monotonically with model scale \(1\.82 to 3\.37 nats across 1\.5B to 7B parameters\), indicating that larger models develop more functionally load\-bearing shells independent of the proportion of selective neurons\.
4. 4\.Functional and spatial characterization\.Code\-selective neurons carry mathematical reasoning capability across all tested models; language\-selective neurons concentrate in final network layers for non\-English domains; and functional shells are spatially interleaved throughout the weight matrices, precluding group\-level selective quantization\.

## 2Related Work

#### Neuron specialization in language models\.

Prior work has established that individual FFN neurons encode interpretable features\(Bauet al\.,[2018](https://arxiv.org/html/2608.10214#bib.bib30)\), that specific neurons correlate with factual associations and their suppression degrades factual recall\(Daiet al\.,[2022](https://arxiv.org/html/2608.10214#bib.bib3)\), and that the FFN intermediate dimension functions as a population of sparse feature detectors rather than a monolithic block\(Voitaet al\.,[2024](https://arxiv.org/html/2608.10214#bib.bib4)\)\. Subsequent work localizes task\-specific\(Wanget al\.,[2022](https://arxiv.org/html/2608.10214#bib.bib5)\), culture\-specific\(Songet al\.,[2024](https://arxiv.org/html/2608.10214#bib.bib6)\), and syntactic\(Mueller and others,[2022](https://arxiv.org/html/2608.10214#bib.bib7)\)neurons within this population\. Our work applies the same causal methodology to ask not which domains have dedicated neurons but under what conditions any domain acquires them\.

#### Language\-specific neurons in multilingual LLMs\.

Tanget al\.\([2024](https://arxiv.org/html/2608.10214#bib.bib1)\)identify language\-specific neurons via activation entropy \(LAPE\) and show that deactivating them degrades target\-language performance\. However,Le and Li \([2026](https://arxiv.org/html/2608.10214#bib.bib2)\)demonstrate that activation preference does not imply functional necessity: LAPE\-identified neurons achieve LangSpec\-F1 near zero under direct causal intervention\.Le and Li \([2026](https://arxiv.org/html/2608.10214#bib.bib2)\)address this with CRANE, a relevance\-attribution framework with substantially stronger causal effects, though limited to a single model family and three languages without a null condition\.Kojima and others \([2024](https://arxiv.org/html/2608.10214#bib.bib8)\)andGurgurov and others \([2025](https://arxiv.org/html/2608.10214#bib.bib9)\)report complementary findings, noting that language\-selective neuron sets are nearly disjoint and sufficient to steer output language\. Our work differs in three respects: we apply a uniform methodology across two domain granularities, producing the controlled null result that contextualizes the language\-positive finding; we provide Pareto and scale analyses quantifying shell concentration and growth; and our causal effect magnitudes are substantially larger, with diagonal dNLL of 1\.82–3\.37 nats versus the sub\-unit perplexity changes reported byTanget al\.\([2024](https://arxiv.org/html/2608.10214#bib.bib1)\)\.

#### Multilingual representation structure\.

Wendleret al\.\([2024](https://arxiv.org/html/2608.10214#bib.bib12)\)show that LLMs process non\-English inputs by mapping them toward English representational space in intermediate layers before projecting back to the target language at output, providing a mechanistic account of the U\-shaped layer distribution of language\-specific neurons reported byTanget al\.\([2024](https://arxiv.org/html/2608.10214#bib.bib1)\)\. Our layer distribution findings complicate this picture: in instruction\-tuned models, language\-selective neurons concentrate primarily in final layers rather than at both ends, consistent with instruction tuning absorbing input\-side language mapping into the shared representation\.

## 3Methodology

### 3\.1Models and Domains

We evaluate three instruction\-tuned decoder\-only language models spanning distinct architectures and scales: Qwen2\.5\-1\.5B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2608.10214#bib.bib16)\), Llama\-3\.2\-3B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2608.10214#bib.bib17)\), and Mistral\-7B\-Instruct\-v0\.3\(Jianget al\.,[2023](https://arxiv.org/html/2608.10214#bib.bib18)\)\. These models containN=250,880N=250\{,\}880,229,376229\{,\}376, and458,752458\{,\}752FFN intermediate neurons respectively, for a combined total of939,008939\{,\}008neurons across all three model families\.

We study two domain granularities\.Subject domainsconsist of four labeled categories derived from MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.10214#bib.bib19)\), ARC\-Challenge\(Clarket al\.,[2018](https://arxiv.org/html/2608.10214#bib.bib20)\), and OpenBookQA\(Mihaylovet al\.,[2018](https://arxiv.org/html/2608.10214#bib.bib21)\): quantitative reasoning \(abstract algebra, mathematics, physics, formal logic, and related subjects\), biomedical \(anatomy, biology, chemistry, clinical knowledge, and related subjects\), humanities and social sciences \(history, law, philosophy, economics, and related subjects\), and elementary science \(all ARC\-Challenge and OpenBookQA items\)\. Evaluation uses letter\-logprob multiple\-choice scoring\.Language and modality domainsconsist of four corpora: English \(WikiText\-2\(Merityet al\.,[2016](https://arxiv.org/html/2608.10214#bib.bib22)\)\), code \(MBPP\(Austinet al\.,[2021](https://arxiv.org/html/2608.10214#bib.bib23)\)\), Spanish, and Chinese \(both from OPUS\-100\(Tiedemann and Thottingal,[2020](https://arxiv.org/html/2608.10214#bib.bib24)\)\)\. Evaluation uses mean negative log\-likelihood \(NLL\) over 512\-token segments\. We additionally evaluate mathematical reasoning using GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.10214#bib.bib25)\)to test whether domain\-selective neurons carry generalizable capability\.

### 3\.2Neuron Activation Capture

For each domainddand each model layerl∈\{1,…,L\}l\\in\\\{1,\\ldots,L\\\}, we capture the mean absolute pre\-activation of each FFN intermediate neuron\. Concretely, for a forward pass on inputxx, let𝐚\(l\)​\(x\)∈ℝdf\\mathbf\{a\}^\{\(l\)\}\(x\)\\in\\mathbb\{R\}^\{d\_\{f\}\}denote the input to the down\-projection matrix𝐖down\(l\)∈ℝdh×df\\mathbf\{W\}^\{\(l\)\}\_\{\\text\{down\}\}\\in\\mathbb\{R\}^\{d\_\{h\}\\times d\_\{f\}\}at layerll, averaged over all token positions\. We define the mean activation of neuronn=\(l,j\)n=\(l,j\)for domainddas:

Md,n=𝔼x∼𝒟d​\[1T​∑t=1T\|aj\(l\)​\(x\)t\|\]M\_\{d,n\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{d\}\}\\left\[\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\left\|a^\{\(l\)\}\_\{j\}\(x\)\_\{t\}\\right\|\\right\]\(1\)
where𝒟d\\mathcal\{D\}\_\{d\}is the set of inputs for domainddandTTis the sequence length\. For subject domains, inputs consist of the question text alone without answer choices, computed via a mean\-pooled forward pass\. Omitting answer choices is essential: a pilot experiment using full multiple\-choice prompts revealed that final\-token representations cluster by the model’s predicted answer letter rather than question content, confounding domain identification\. For language domains, inputs are 512\-token text segments\. All captures use a maximum input length of 1024 tokens with item\-level norm logging; items in the bottom 2% byℓ2\\ell\_\{2\}norm of the resulting representation are excluded from subspace fitting to prevent degenerate near\-zero vectors from distorting domain statistics\(c\.f\. Armstrong and others,[2026](https://arxiv.org/html/2608.10214#bib.bib26)\)\.

### 3\.3Domain Selectivity

LetKKdenote the number of domains andN=L⋅dfN=L\\cdot d\_\{f\}the total number of FFN neurons\. We define thedomain shareof neuronnnfor domainddas:

σd,n=Md,n∑d′=1KMd′,n\+ϵ\\sigma\_\{d,n\}=\\frac\{M\_\{d,n\}\}\{\\sum\_\{d^\{\\prime\}=1\}^\{K\}M\_\{d^\{\\prime\},n\}\+\\epsilon\}\(2\)
Under uniform domain activity,σd,n=1/K\\sigma\_\{d,n\}=1/Kfor alldd\. Themax\-shareof neuronnnissn=maxd⁡σd,ns\_\{n\}=\\max\_\{d\}\\sigma\_\{d,n\}, which equals1/K1/Kfor a neuron with no domain preference and approaches 1 for a neuron active exclusively for one domain\. We report the distribution ofsns\_\{n\}across all neurons as the selectivity histogram, usingsn\>0\.60s\_\{n\}\>0\.60as our primary threshold for identifying neurons with strong domain preference\.

We exclude a backbone setℬ\\mathcal\{B\}consisting of the top 0\.1% of neurons by global mean activationM¯n=1K​∑dMd,n\\bar\{M\}\_\{n\}=\\frac\{1\}\{K\}\\sum\_\{d\}M\_\{d,n\}from all subsequent analyses\. These neurons exhibit high activity across all domains and correspond to the massive\-activation or super\-weight phenomenon documented in prior work\(Sunet al\.,[2024](https://arxiv.org/html/2608.10214#bib.bib27)\); including them inflates apparent domain damage in masking experiments because their removal degrades all domains uniformly rather than selectively\.

For each domaindd, we define the domain shell𝒮d\\mathcal\{S\}\_\{d\}as the top\-mmneurons ranked by excess shareσd,n−1/K\\sigma\_\{d,n\}\-1/K, excludingℬ\\mathcal\{B\}, wheremmis determined by a fixed mask fractionρ∈\[0,1\]\\rho\\in\[0,1\]applied to the total neuron count:

𝒮d=top\-​mn∉ℬ​\(σd,n−1K\),m=⌊ρ​N⌋\\mathcal\{S\}\_\{d\}=\\underset\{n\\notin\\mathcal\{B\}\}\{\\text\{top\-\}m\}\\left\(\\sigma\_\{d,n\}\-\\frac\{1\}\{K\}\\right\),\\quad m=\\lfloor\\rho N\\rfloor\(3\)
We useρ=0\.02\\rho=0\.02as our primary mask fraction and report Pareto curves overρ∈\{0\.005,0\.01,0\.02,0\.05,0\.10,0\.20\}\\rho\\in\\\{0\.005,0\.01,0\.02,0\.05,0\.10,0\.20\\\}\. A count\-matched random baseline𝒮rand\\mathcal\{S\}\_\{\\text\{rand\}\}is sampled uniformly from\{1,…,N\}∖ℬ\\\{1,\\ldots,N\\\}\\setminus\\mathcal\{B\}at each fraction\.

### 3\.4Causal Masking and Damage Measurement

To assess functional necessity, we intervene on the model by zeroing the intermediate activations at positions in a target shell during inference\. Specifically, for a shell𝒮d′\\mathcal\{S\}\_\{d^\{\\prime\}\}and an inputxx, the masked forward pass sets:

a~j\(l\)​\(x\)=\{0if​\(l,j\)∈𝒮d′aj\(l\)​\(x\)otherwise\\tilde\{a\}^\{\(l\)\}\_\{j\}\(x\)=\\begin\{cases\}0&\\text\{if \}\(l,j\)\\in\\mathcal\{S\}\_\{d^\{\\prime\}\}\\\\ a^\{\(l\)\}\_\{j\}\(x\)&\\text\{otherwise\}\\end\{cases\}\(4\)
for all layerslland all token positions\. This is implemented via registered forward hooks on the down\-projection pre\-activation without modifying model weights\.

For language domains, we measure the NLL change under masking\. The damage of masking domaind′d^\{\\prime\}’s shell on languageddis reported net of the random baseline to isolate shell\-specific effects from general disruption:

Δd′→d=NLLd​\(𝒮d′\)⏟shell mask−NLLd​\(𝒮rand\)⏟random mask\\Delta\_\{d^\{\\prime\}\\to d\}=\\underbrace\{\\text\{NLL\}\_\{d\}\(\\mathcal\{S\}\_\{d^\{\\prime\}\}\)\}\_\{\\text\{shell mask\}\}\-\\underbrace\{\\text\{NLL\}\_\{d\}\(\\mathcal\{S\}\_\{\\text\{rand\}\}\)\}\_\{\\text\{random mask\}\}\(5\)
This yields aK×KK\\times Kdamage matrix where diagonal entries indicate how much masking domaindd’s shell degrades domainddbeyond random, and off\-diagonal entries indicate cross\-domain interference\. For subject domains, accuracy replaces NLL as the evaluation metric and the same net\-of\-random normalization applies\.

We summarize the damage matrix by its diagonal meanΔ¯diag=1K​∑dΔd→d\\bar\{\\Delta\}\_\{\\text\{diag\}\}=\\frac\{1\}\{K\}\\sum\_\{d\}\\Delta\_\{d\\to d\}and off\-diagonal meanΔ¯off=1K​\(K−1\)​∑d≠d′Δd′→d\\bar\{\\Delta\}\_\{\\text\{off\}\}=\\frac\{1\}\{K\(K\-1\)\}\\sum\_\{d\\neq d^\{\\prime\}\}\\Delta\_\{d^\{\\prime\}\\to d\}, and report their ratio as a single scalar measure of shell selectivity\.

### 3\.5Shell Overlap and Spatial Organization

To assess the disjointness of domain shells, we compute the Jaccard similarity \(IoU\) between the top\-mmshell sets for each domain pair:

IoU​\(d,d′\)=\|𝒮d∩𝒮d′\|\|𝒮d∪𝒮d′\|\\text\{IoU\}\(d,d^\{\\prime\}\)=\\frac\{\|\\mathcal\{S\}\_\{d\}\\cap\\mathcal\{S\}\_\{d^\{\\prime\}\}\|\}\{\|\\mathcal\{S\}\_\{d\}\\cup\\mathcal\{S\}\_\{d^\{\\prime\}\}\|\}\(6\)
To assess spatial organization, we record the layer\-wise distribution of shell neurons by counting, for each layerlland domaindd, the number of neurons\(l,j\)∈𝒮d\(l,j\)\\in\\mathcal\{S\}\_\{d\}\.

To assess whether functional shell structure permits selective quantization, we apply per\-group asymmetric INT4 fake quantization\(Frantaret al\.,[2023](https://arxiv.org/html/2608.10214#bib.bib28)\)to the down\-projection weight matrix𝐖down\(l\)\\mathbf\{W\}^\{\(l\)\}\_\{\\text\{down\}\}under three conditions: uniform quantization of all weight groups, quantization of groups whose constituent neurons are entirely in⋃d𝒮dc\\bigcup\_\{d\}\\mathcal\{S\}\_\{d\}^\{c\}\(the core\), and the inverse\. Group size isg=128g=128following standard deployment practice\(Linet al\.,[2025](https://arxiv.org/html/2608.10214#bib.bib29)\)\. NLL is evaluated after weight replacement without any calibration data, isolating the native quantization compatibility of the weight configuration\.

## 4Results

### 4\.1Subject\-Level Domains: Zero Parametric Modularity

Figure[1](https://arxiv.org/html/2608.10214#S4.F1)\(top row\) shows the selectivity histogram for subject\-level domains across all three models\. The distribution is unimodal and rapidly decaying in all cases: no neuron exceeds 0\.60 max\-share across a combined 939,008 FFN neurons, and the 99th percentile reaches at most 0\.388\.

![Refer to caption](https://arxiv.org/html/2608.10214v1/x1.png)Figure 1:Selectivity histograms for subject\-level domains \(top row\) and language/modality domains \(bottom row\) across three model families\. The y\-axis is log\-scaled percentage of total FFN neurons\. The dashed vertical line marks the 0\.60 max\-share threshold\. At the subject level, no neurons exceed 0\.60 across 939,008 combined neurons\. At the language level, 0\.65–1\.14% exceed 0\.60 in all three models\.Causal masking confirms functional indifference\. The subject\-level damage matrix has a diagonal mean of 0\.121–0\.181 accuracy points and an off\-diagonal mean of 0\.031–0\.076, with no model exceeding a diagonal\-to\-off\-diagonal ratio of 2\.4\. The elem\_sci row is an exception whose off\-diagonal entries reveal general disruption rather than a selective shell\. These results hold despite domain identity being linearly decodable above 85% accuracy via a logistic regression probe, providing a clean dissociation: domain information is present in activation geometry but not supported by a dedicated parametric substrate\.

### 4\.2Language\-Level Domains: Concentrated Parametric Shells

Figure[1](https://arxiv.org/html/2608.10214#S4.F1)\(bottom row\) shows the selectivity histogram for language and modality domains\. The fraction of neurons exceeding 0\.60 max\-share is 1\.14%, 0\.65%, and 0\.84% for Qwen2\.5\-1\.5B, Llama\-3\.2\-3B, and Mistral\-7B respectively, compared to 0\.00% in all three models at the subject level\.

Figure[2](https://arxiv.org/html/2608.10214#S4.F2)shows the language\-level damage matrices\. The matrices are strongly diagonal across all model families\. Diagonal means areΔ¯diag=1\.82\\bar\{\\Delta\}\_\{\\text\{diag\}\}=1\.82,2\.192\.19, and3\.373\.37nats; off\-diagonal means are−0\.012\-0\.012,−0\.060\-0\.060, and\+0\.006\+0\.006nats\. The diagonal\-to\-off\-diagonal ratio ranges from 37:1 to above 500:1\. The largest single entry is the Chinese diagonal in Mistral\-7B at\+8\.58\+8\.58nats, against an English off\-diagonal of\+0\.013\+0\.013nats under the same mask\.

![Refer to caption](https://arxiv.org/html/2608.10214v1/x2.png)Figure 2:Language\-level damage matrices \(dNLL net of count\-matched random baseline\) for all three models atρ=0\.02\\rho=0\.02\. Rows: ablated shell\. Columns: evaluated domain\. Diagonal means are 1\.82, 2\.19, and 3\.37 nats; off\-diagonal means are−0\.012\-0\.012,−0\.060\-0\.060, and\+0\.006\+0\.006nats\.Shell disjointness is near\-complete: the maximum IoU between any two domain shell sets is 0\.0029, with most pairs below 0\.001 \(Appendix[C](https://arxiv.org/html/2608.10214#A3)\)\. Layer\-wise shell distributions \(Appendix[A](https://arxiv.org/html/2608.10214#A1)\) show Spanish and Chinese neurons concentrating in final network layers, while English and code neurons distribute uniformly, consistent with instruction tuning consolidating non\-English language\-specific computation into the vocabulary projection stage\.

### 4\.3Pareto Analysis: Shell Concentration and Functional Load

![Refer to caption](https://arxiv.org/html/2608.10214v1/x3.png)Figure 3:Diagonal dNLL net of random as a function of mask fractionρ\\rhofor each language domain across all three models \(log\-scaled x\-axis\)\. Damage accumulates rapidly betweenρ=0\.005\\rho=0\.005andρ=0\.02\\rho=0\.02and saturates byρ=0\.05\\rho=0\.05for all domains\. The Chinese curve in Mistral\-7B reaches\+6\.09\+6\.09nats atρ=0\.005\\rho=0\.005\(2,293 neurons\) and\+8\.58\+8\.58nats atρ=0\.02\\rho=0\.02, representing 91\.6% of the total damage observed atρ=0\.20\\rho=0\.20\.Figure[3](https://arxiv.org/html/2608.10214#S4.F3)shows damage accumulating rapidly betweenρ=0\.005\\rho=0\.005andρ=0\.02\\rho=0\.02and saturating byρ=0\.05\\rho=0\.05for all language domains\. Atρ=0\.02\\rho=0\.02, damage represents on average 72% of the maximum observed; for Chinese in Mistral\-7B,ρ=0\.005\\rho=0\.005\(2,293 neurons\) already produces\+6\.09\+6\.09nats while the curve saturates at\+9\.03\+9\.03nats byρ=0\.05\\rho=0\.05\. At the subject level, damage curves are flat and undifferentiated from random throughout theρ\\rhorange\.

### 4\.4Code Shells Carry Mathematical Reasoning

Table[1](https://arxiv.org/html/2608.10214#S4.T1)reports GSM8K accuracy under each masking condition net of the count\-matched random baseline\. Masking the code shell produces the largest and most consistent effect:−17\.3\-17\.3,−24\.0\-24\.0, and−16\.0\-16\.0percentage points for Qwen2\.5\-1\.5B, Llama\-3\.2\-3B, and Mistral\-7B respectively\. Mathematical reasoning, despite being expressed in English\-language text, relies on the code\-selective neuron population rather than the English\-selective population\.

Masking Spanish produces effects at or below random in all three models\. The Chinese result varies in a pattern consistent with pretraining composition: Mistral\-7B shows near\-zero Chinese effect \(\+0\.7\+0\.7pp\), Qwen2\.5\-1\.5B shows a small negative effect \(−1\.3\-1\.3pp\), and Llama\-3\.2\-3B shows−10\.0\-10\.0pp, tracing a gradient that directly maps to presumed Chinese mathematical content during pretraining\. Shell purity is not a fixed property of a language but a property of how that language was represented in the training data\.111Qwen2\.5\-1\.5B’s random mask condition nominally exceeds its unmasked baseline \(36\.0% vs\. 30\.7%\); this 1\.4 standard error gap is consistent with sampling variability\. All Qwen effects are reported net of this elevated baseline and are therefore conservative\.

Table 1:GSM8K accuracy \(%\) under each masking condition\. Net\-of\-random effects in parentheses\. Bold: largest net effect per model\.
### 4\.5Scale Analysis

![Refer to caption](https://arxiv.org/html/2608.10214v1/x4.png)Figure 4:Language shell properties as a function of model scale\. Left: fraction of neurons exceeding 0\.60 max\-share shows a non\-monotonic relationship with parameter count\. Right: diagonal mean dNLL atρ=0\.02\\rho=0\.02increases monotonically with scale \(1\.82, 2\.19, 3\.37 nats\), indicating that larger models develop more functionally load\-bearing language shells\.Figure[4](https://arxiv.org/html/2608.10214#S4.F4)shows language shell strength as a function of model scale\. The fraction of neurons exceeding 0\.60 max\-share is non\-monotonic \(1\.14%, 0\.65%, 0\.84%\), reflecting differences in architecture and pretraining composition\. In contrast,Δ¯diag\\bar\{\\Delta\}\_\{\\text\{diag\}\}increases monotonically: 1\.82, 2\.19, and 3\.37 nats\. Atρ=0\.005\\rho=0\.005, Mistral\-7B’s Chinese shell produces 6\.09 nats of targeted damage versus 0\.57 nats in Qwen2\.5\-1\.5B\. Larger models develop not more selective neurons proportionally, but more functionally load\-bearing ones\.

### 4\.6Spatial Interleaving: Functional Shells Are Parametrically Dispersed

Under per\-group asymmetric INT4 quantization with group sizeg=128g=128, a group qualifies for core\-only quantization only if all 128 neurons fall below the selectivity threshold\. At shell densities of 12–21%, the probability that any group of 128 consecutive neurons is entirely core evaluates to2\.2×10−82\.2\\times 10^\{\-8\}for the highest\-density model\. In practice, zero qualifying groups are found in any of the three models\.

The empirical consequence: int4\_core and int4\_inv produce NLL values identical to BF16 to four decimal places\. Uniform INT4 produces language\-uniform increases of\+0\.028\+0\.028–\+0\.081\+0\.081nats, with no language showing systematically larger degradation, consistent with quantization affecting only shared\-core representations\. Functional modularity does not imply spatial locality: language\-selective neurons are interleaved throughout the intermediate dimension at every spatial scale accessible to standard group quantization\.

## 5The Granularity Principle

The results converge on a single organizing principle: parametric shells form where and only where training data was itself modular at the token level\.

#### Why language domains produce shells\.

English, Spanish, Chinese, and code occupy largely non\-overlapping regions of the token distribution during pretraining\. Gradient updates for Chinese inputs systematically push a subset of parameters toward configurations useful for Chinese processing while leaving parameters used for English largely undisturbed\. Repeated across billions of tokens, this pressure produces a population of neurons concentrated on one partition of the training distribution\. The shell IoU values below 0\.003 are the parametric residue of this partitioned training signal\.

#### Why subject domains do not\.

A question about cellular respiration and a question about differential equations are both written in English with similar syntactic structures and overlapping academic vocabulary\. The gradient signal for biology questions and physics questions updates largely overlapping parameter sets, producing no systematic pressure toward domain\-selective neurons\. The 88% probe accuracy establishes that domain information is present in activation geometry, but representational geometry and parametric organization are separable properties of a trained network and need not co\-occur\.

#### Shell purity tracks training data composition\.

Mistral\-7B’s Chinese shell carries essentially no mathematical reasoning capability \(\+0\.7\+0\.7pp on GSM8K net of random\), while Llama\-3\.2\-3B’s shows−10\.0\-10\.0pp, and Qwen2\.5\-1\.5B’s shows−1\.3\-1\.3pp\. This gradient maps directly to the presumed gradient in Chinese mathematical content during pretraining\. A shell learned on purely linguistic content carries only linguistic function; a shell learned on thematically diverse content carries a corresponding mixture\. Shell purity is a property of the training data, not of the domain\.

## 6Discussion

#### Implications for multilingual model design\.

Parametric separability across languages is not an emergent property of scale: it is a direct consequence of whether training data partitions by token distribution\. Languages whose training data overlaps substantially with English at the token level may develop weaker or less pure shells than typologically distant languages with distinct scripts\. Deliberate curation of partition sharpness across target languages, rather than simply maximizing token counts, may produce more predictable behavior under language\-targeted interventions\.

#### Implications for editing and compression\.

For language domains, the concentrated and disjoint shell structure makes neuron\-level editing feasible: the Chinese shell in Mistral\-7B is identifiable, functionally load\-bearing, and nearly orthogonal to all other domain shells\. For subject domains, no analogous structure exists\. For compression, spatial interleaving means functional modularity cannot be directly converted into memory efficiency under standard group\-level quantization\. Two directions remain open: per\-neuron precision assignment, and weight\-matrix reordering that clusters shell neurons contiguously before applying group quantization\. Both are weight\-equivalent and constitute concrete targets for future work\.

#### Relationship to prior work\.

Our results extendTanget al\.\([2024](https://arxiv.org/html/2608.10214#bib.bib1)\)andLe and Li \([2026](https://arxiv.org/html/2608.10214#bib.bib2)\)in three respects\. First, causal effect magnitudes are substantially larger: LAPE produces sub\-unit perplexity changes and CRANE produces moderate LangSpec\-F1 values, while our approach identifies neurons whose removal is functionally catastrophic, establishing that a concentrated population carries the functional load rather than merely correlating with language use\. Second, the subject\-level null condition, absent from prior work, is what makes the language\-level positive result interpretable as a boundary condition rather than a data point\. Third, the final\-layer concentration we observe in instruction\-tuned models differs from the U\-shaped distribution reported for base models\(Tanget al\.,[2024](https://arxiv.org/html/2608.10214#bib.bib1)\), consistent with instruction tuning consolidating language\-specific computation into the vocabulary projection stage\.

#### Limitations\.

Our analysis covers three instruction\-tuned model families in the 1\.5B–7B range; whether the granularity principle holds for base models, larger models, or substantially different architectures is not established\. The layer distribution comparison withTanget al\.\([2024](https://arxiv.org/html/2608.10214#bib.bib1)\)is indirect, as we evaluate different families under different training regimes\. Our activation\-magnitude identification method may not surface all functionally relevant neurons; gradient or relevance\-based approaches may identify complementary populations\. The subject\-level null covers four coarse categories; finer\-grained partitions or corpus\-based taxonomies might reveal structure at intermediate granularities\.

## 7Conclusion

We investigated whether large language models contain domain\-specific parametric shells: concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others\. Across two domain granularities, three model families, and eight domains, the answer depends entirely on granularity\. At the subject level, zero neurons exceed 60% domain selectivity across 939,008 combined FFN neurons and causal damage matrices are flat, despite domain identity being linearly decodable above 85% accuracy\. At the language and modality level, 0\.65–1\.14% of neurons exceed 60% selectivity, damage matrices are near\-perfectly diagonal with ratios up to 595:1, shell neuron sets are essentially disjoint \(IoU<0\.003<0\.003\), and masking code\-selective neurons reduces mathematical reasoning accuracy by 16–24 percentage points while masking Spanish or Chinese neurons leaves it at or below random\.

These results are unified by the granularity principle: parametric shells form where and only where training data was modular at the token level\. The model carves its parameters the same way its training data was carved by script, vocabulary, and syntactic register\.

## AI Use Statement

This manuscript was prepared with the assistance of large language model tools for drafting and editing\. All experimental design, data collection, analysis, and scientific claims are the work of the authors\. All AI\-assisted text was reviewed, edited, and verified by the authors\.

## Ethics Statement

This work analyzes publicly available open\-weight language models \(Qwen2\.5\-1\.5B\-Instruct, Llama\-3\.2\-3B\-Instruct, Mistral\-7B\-Instruct\-v0\.3\) using publicly available benchmark datasets \(MMLU, ARC\-Challenge, OpenBookQA, GSM8K, WikiText\-2, MBPP, OPUS\-100\)\. No human subjects were involved\. No personal data was collected or used\. Code will be released upon acceptance\.

## Reproducibility Statement

All hyperparameters are documented in Table[4](https://arxiv.org/html/2608.10214#A5.T4)\(Appendix[E](https://arxiv.org/html/2608.10214#A5)\)\. Activation capture and causal masking implementations are described in detail in Appendix[E](https://arxiv.org/html/2608.10214#A5)\. All experiments use a fixed random seed \(42\)\. Full model identifiers, dataset splits, and sampling procedures are specified in SectionLABEL:sec:methodand Appendix[E](https://arxiv.org/html/2608.10214#A5)\. Code will be released as supplementary material\.

## References

- TITLE to be filled in\.arXiv preprint\.Cited by:[§3\.2](https://arxiv.org/html/2608.10214#S3.SS2.p3.4)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton \(2021\)Program synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p2.1)\.
- A\. Bau, Y\. Belinkov, H\. Sajjad, N\. Durrani, F\. Dalvi, and J\. Glass \(2018\)Identifying and controlling important neurons in neural machine translation\.External Links:1811\.01157,[Link](https://arxiv.org/abs/1811.01157)Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p2.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p2.1)\.
- D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. Wei \(2022\)Knowledge neurons in pretrained transformers\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 8493–8502\.External Links:[Link](https://aclanthology.org/2022.acl-long.581/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.581)Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2023\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.External Links:2210\.17323,[Link](https://arxiv.org/abs/2210.17323)Cited by:[§3\.5](https://arxiv.org/html/2608.10214#S3.SS5.p4.3)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p1.4)\.
- D\. Gurgurovet al\.\(2025\)Targeted fine\-tuning of language\-specific neurons in multilingual language models\.arXiv preprint\.Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.External Links:2009\.03300,[Link](https://arxiv.org/abs/2009.03300)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p2.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p1.4)\.
- T\. Kojimaet al\.\(2024\)Language\-specific neurons in multilingual models\.arXiv preprint\.Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Le and Y\. Li \(2026\)CRANE: causal relevance analysis of language\-specific neurons in multilingual large language models\.External Links:2601\.04664,[Link](https://arxiv.org/abs/2601.04664)Cited by:[§1](https://arxiv.org/html/2608.10214#S1.p1.1),[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2608.10214#S6.SS0.SSS0.Px3.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, G\. Xiao, and S\. Han \(2025\)AWQ: activation\-aware weight quantization for on\-device llm compression and acceleration\.GetMobile: Mobile Comp\. and Comm\.28\(4\),pp\. 12–17\.External Links:ISSN 2375\-0529,[Link](https://doi.org/10.1145/3714983.3714987),[Document](https://dx.doi.org/10.1145/3714983.3714987)Cited by:[§3\.5](https://arxiv.org/html/2608.10214#S3.SS5.p4.3)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.External Links:1609\.07843,[Link](https://arxiv.org/abs/1609.07843)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p2.1)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2381–2391\.External Links:[Link](https://aclanthology.org/D18-1260/),[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p2.1)\.
- A\. Muelleret al\.\(2022\)Coloring the black box: what syntactic probes learn\.arXiv preprint\.Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px1.p1.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p1.4)\.
- R\. Song, S\. He, S\. Jiang, Y\. Xian, S\. Gao, K\. Liu, and Z\. Yu \(2024\)Does large language model contain task\-specific neurons?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7101–7113\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.403/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.403)Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. Liu \(2024\)Massive activations in large language models\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=F7aAhfitX6)Cited by:[§3\.3](https://arxiv.org/html/2608.10214#S3.SS3.p4.2)\.
- T\. Tang, W\. Luo, H\. Huang, D\. Zhang, X\. Wang, X\. Zhao, F\. Wei, and J\. Wen \(2024\)Language\-specific neurons: the key to multilingual capabilities in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 5701–5715\.External Links:[Link](https://aclanthology.org/2024.acl-long.309/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.309)Cited by:[Figure 5](https://arxiv.org/html/2608.10214#A1.F5),[§1](https://arxiv.org/html/2608.10214#S1.p1.1),[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.10214#S6.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.10214#S6.SS0.SSS0.Px4.p1.1)\.
- J\. Tiedemann and S\. Thottingal \(2020\)OPUS\-MT – building open translation services for the world\.InProceedings of the 22nd Annual Conference of the European Association for Machine Translation,A\. Martins, H\. Moniz, S\. Fumega, B\. Martins, F\. Batista, L\. Coheur, C\. Parra, I\. Trancoso, M\. Turchi, A\. Bisazza, J\. Moorkens, A\. Guerberof, M\. Nurminen, L\. Marg, and M\. L\. Forcada \(Eds\.\),Lisboa, Portugal,pp\. 479–480\.External Links:[Link](https://aclanthology.org/2020.eamt-1.61/)Cited by:[§3\.1](https://arxiv.org/html/2608.10214#S3.SS1.p2.1)\.
- E\. Voita, J\. Ferrando, and C\. Nalmpantis \(2024\)Neurons in large language models: dead, n\-gram, positional\.pp\. 1288–1301\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.75)Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Wang, K\. Wen, Z\. Zhang, L\. Hou, Z\. Liu, and J\. Li \(2022\)Finding skill neurons in pre\-trained transformer\-based language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 11132–11152\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.765/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.765)Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Wendler, V\. Veselovsky, G\. Monea, and R\. West \(2024\)Do llamas work in English? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15366–15394\.External Links:[Link](https://aclanthology.org/2024.acl-long.820/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.820)Cited by:[§2](https://arxiv.org/html/2608.10214#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. Rush \(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Q\. Liu and D\. Schlangen \(Eds\.\),Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.emnlp-demos.6/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[Appendix E](https://arxiv.org/html/2608.10214#A5.SS0.SSS0.Px1.p1.1)\.

## Appendix ALayer\-wise Shell Distributions

![Refer to caption](https://arxiv.org/html/2608.10214v1/x5.png)Figure 5:Layer\-wise distribution of language shell neurons \(normalized by the maximum count across layers per domain\) for each model\. Spanish and Chinese neurons concentrate strongly in final layers across all three model families\. English and code neurons are distributed more uniformly\. This final\-layer concentration for non\-English domains differs from the U\-shaped distribution reported for base models\[Tanget al\.,[2024](https://arxiv.org/html/2608.10214#bib.bib1)\], consistent with instruction tuning absorbing input\-side language mapping into the shared representation\.
## Appendix BSubject\-Level Damage Matrices

Figure[6](https://arxiv.org/html/2608.10214#A2.F6)shows the subject\-level damage matrices \(net of count\-matched random baseline, measured in accuracy points\) for all three models\. In contrast to the language\-level matrices in Figure[2](https://arxiv.org/html/2608.10214#S4.F2), no model produces a consistently diagonal structure\. For the quant, biomed, and humsoc rows, diagonal and off\-diagonal entries are of comparable magnitude across all three models, consistent with the interpretation that masking any of these subject shells disrupts general question\-answering capability rather than a domain\-specific parametric substrate\.

The elem\_sci row is an exception, showing larger values across all columns rather than a concentrated diagonal entry\. This pattern reflects the composition of the elementary science evaluation set \(ARC\-Challenge and OpenBookQA\), which draws on general English vocabulary and commonsense reasoning that overlaps substantially with all other subject domains\. Masking the elem\_sci shell therefore degrades all domains, a signature of disrupting shared linguistic processing rather than a domain\-specific shell\. This interpretation is supported by the subject\-level selectivity histogram \(Figure[1](https://arxiv.org/html/2608.10214#S4.F1), top row\): the elem\_sci shell contains no neurons with max\-share above 0\.60 and therefore cannot constitute a parametrically selective substrate for elementary science in particular\.

![Refer to caption](https://arxiv.org/html/2608.10214v1/x6.png)Figure 6:Subject\-level damage matrices \(accuracy points net of random baseline\) for all three models\. Rows correspond to the ablated shell; columns to the evaluated subject domain\. Unlike the language\-level matrices, no model produces a consistently diagonal structure\. The elem\_sci row shows large off\-diagonal values consistent with disruption of shared general question\-answering capability rather than a domain\-selective shell\. Domain labels: Q = quantitative, B = biomedical, H = humanities and social sciences, E = elementary science\.
## Appendix CShell Overlap Matrices

Tables[2](https://arxiv.org/html/2608.10214#A3.T2)and[3](https://arxiv.org/html/2608.10214#A3.T3)report the Jaccard similarity \(IoU\) between top\-2% shell sets for all domain pairs across all three models\.

Language shell overlap is near\-zero across all pairs and all models \(Table[2](https://arxiv.org/html/2608.10214#A3.T2)\)\. The maximum observed IoU is 0\.0029 \(Mistral\-7B, Spanish and Chinese\), and the majority of pairs report IoU below 0\.001\. The slight elevation in the Spanish–Chinese pair relative to other pairs is consistent with the observation that Spanish and Chinese both require final\-layer vocabulary mapping neurons that partially overlap in the intermediate dimension, though the overlap remains negligible in absolute terms\.

Subject shell overlap is similarly near\-zero across most pairs \(Table[3](https://arxiv.org/html/2608.10214#A3.T3)\), with the exception of the biomed–elem\_sci pair, which reaches 0\.0115–0\.0162 across models\. This modest elevation reflects the shared biological science content between biomedical MMLU questions and ARC elementary science questions\. Even at its maximum, the overlap across all pairs and all models remains below 1\.7%, confirming that the absence of functional shells at the subject level is not attributable to excessive overlap between shell definitions\.

Table 2:Jaccard similarity \(IoU\) between top\-2% language shell sets for each domain pair and model\. Values below 0\.003 across all pairs and all models confirm near\-complete shell disjointness\.Table 3:Jaccard similarity \(IoU\) between top\-2% subject shell sets for each domain pair and model\. The biomed–elem\_sci elevation reflects shared biological science content between MMLU biomedical questions and ARC elementary science items\.
## Appendix DPer\-Model Pareto Curves

Figure[7](https://arxiv.org/html/2608.10214#A4.F7)shows the Pareto curves reorganized by model rather than by language domain, allowing direct comparison of shell concentration across domains within each model\. Within each model, Chinese and Spanish consistently show the steepest initial curves, saturating earliest\. English shows the shallowest curve in all three models, consistent with English capability being distributed throughout the network rather than concentrated in a dedicated shell\. The ordering of domain steepness is consistent across all three models despite differences in absolute dNLL magnitude, supporting the interpretation that shell concentration is a property of the domain partition rather than of any particular model architecture\.

![Refer to caption](https://arxiv.org/html/2608.10214v1/x7.png)Figure 7:Diagonal dNLL net of random as a function of mask fractionρ\\rho, organized by model\. Each panel shows all four language domains for one model\. Chinese and Spanish curves are consistently steeper and saturate earlier than English and code curves within each model, regardless of architecture or scale\.
## Appendix EExperimental Details

#### Models\.

All models are loaded in bfloat16 precision via HuggingFace Transformers\[Wolfet al\.,[2020](https://arxiv.org/html/2608.10214#bib.bib31)\]withdevice\_map=auto\. Full model identifiers areQwen/Qwen2\.5\-1\.5B\-Instruct,meta\-llama/Llama\-3\.2\-3B\-Instruct, andmistralai/Mistral\-7B\-Instruct\-v0\.3\.

#### Hardware\.

All experiments were conducted on NVIDIA A100 80GB GPUs via Google Colab Pro\. Qwen2\.5\-1\.5B and Llama\-3\.2\-3B experiments are compatible with a single T4 16GB GPU\. Mistral\-7B requires at minimum an A100 40GB GPU in bfloat16 precision\.

#### Dataset construction\.

Subject domain evaluation pools are constructed from MMLU validation splits \(600 items total, stratified across all available subjects with a minimum of 1 item per subject\), ARC\-Challenge test split \(225 items, randomly sampled\), and OpenBookQA test split \(225 items, randomly sampled\)\. Subject assignment follows the taxonomy in SectionLABEL:sec:method, with all sampling using a fixed random seed\. Language domain corpora use 96 segments of 512 tokens each, drawn from WikiText\-2 test split \(English\), MBPP test split \(code\), and OPUS\-100 training splits \(Spanish, Chinese\) with streaming access\. GSM8K evaluation uses 150 randomly sampled test items with greedy decoding and a maximum of 256 new tokens\.

Table[4](https://arxiv.org/html/2608.10214#A5.T4)summarizes all primary hyperparameters used across experiments\.

Table 4:Experimental hyperparameters\.
#### Activation capture implementation\.

Neuron activations are captured via PyTorch forward hooks registered on thedown\_projpre\-activation at each FFN layer\. For subject domains, hooks are applied to question\-only forwards without answer choices or theAnswer:scaffold token, and activations are mean\-pooled across all non\-padding token positions\. For language domains, hooks are applied to teacher\-forced forwards over fixed 512\-token segments\. Hook registration and removal are performed immediately before and after each forward pass to prevent cross\-example contamination\. All captures accumulate into float64 accumulators to prevent precision loss during averaging over large item counts\.

#### Causal masking implementation\.

Neuron masking is implemented via the same forward hook mechanism, zeroing the pre\-activation of selected neurons at all token positions across all forward passes in the evaluation set\. Masks are defined once per condition and reused across all items; no per\-item mask computation occurs at evaluation time\. The random baseline mask is sampled once per experiment with a fixed random seed and held constant across all model and domain comparisons to ensure that random baseline variation does not confound cross\-condition comparisons\.