Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders

# Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders
Source: [https://arxiv.org/html/2608.23809](https://arxiv.org/html/2608.23809)
Igor BogdanovAffiliation:Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, CanadaCorrespondence to:[igorbogdanov@cmail\.carleton\.ca](mailto:[email protected])Changcheng HuangAffiliation:Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, Canada

###### Abstract

Multilingual language models can solve the same mathematical reasoning problem in different languages, but it remains unclear whether they rely on shared internal features or on language\-specific computations that only produce similar outputs\. We study this question in five models from four architecture families using the Multilingual Grade School Math \(MGSM\) dataset, with problems solved independently in English, German, French, Spanish, Russian, and Chinese, retaining only problems with valid reasoning traces in all six languages and replaying those traces through the model to record internal representations at multiple layers\. For each model, we first use Centered Kernel Alignment \(CKA\) to identify layers with strong cross\-language alignment\. At each selected layer, we train two sparse autoencoders: a baseline reconstruction\-only model and a contrastive variant introduced in this work, the Geometry\-Invariant Sparse Autoencoder \(GI\-SAE\)\. GI\-SAE supplements the reconstruction loss with an Information Noise\-Contrastive Estimation \(InfoNCE\) loss that trains the encoder to produce similar feature activations for traces of the same problem, regardless of language or token position\. We then test whether the resulting shared features are functionally interchangeable by swapping shared feature values between languages during the model’s forward pass and measuring the resulting change in output \(causal patching\), quantified by Kullback\-Leibler \(KL\) divergence per shared feature\. Although GI\-SAE yields higher CKA and Jaccard similarity at nearly every layer, higher geometric similarity does not consistently imply greater functional interchangeability across languages\. We find that cross\-language feature sharing is strongly model\- and architecture\-dependent in this sample and appears at different depths in different models\. GI\-SAE primarily amplifies cross\-language structure already present in each model: the pattern is model\-specific, with progressive strengthening in Qwen, no functional benefit in already\-saturated Gemma, and mixed layer\-dependent effects in Llama and Phi\.

###### Keywords:

Mechanistic Interpretability, Multilingual Large Language Models, Sparse Autoencoders, Cross\-Lingual Representations, Multilingual Reasoning, Causal Patching

## 1Introduction

Multilingual large language models solve mathematical reasoning problems across languages, but whether they develop*shared*internal representations for reasoning, or merely produce correct answers via language\-specific computations, remains an open question\. The distinction has practical consequences: shared representations would allow a single interpretability analysis to cover all languages, while language\-specific computations would require separateanalysisfor each language\.

Three gaps remain in this literature: to our knowledge, no study has tested whether cross\-language shared features are*functionally interchangeable*\(substituting values preserves behavior\); sharing has not been compared across model families under a common protocol; and it is unknown whether geometric metrics \(CKA, Jaccard\) reliably predict functional interchangeability\.

We address these gaps by training two top\-KKSAEs\([Gao et al\., 2025](https://arxiv.org/html/2608.23809#bib.bib10)\)at each layer of each model: a*baseline*SAE \(reconstruction only\) and a*geometry\-invariant*SAE \(GI\-SAE\)\([van den Oord et al\., 2019](https://arxiv.org/html/2608.23809#bib.bib12)\)that adds an InfoNCE contrastive loss encouraging similar feature activations across languages for the same problem\. We test functional interchangeability via causal patching: swapping shared feature values between languages and measuring KL divergence per shared feature\.

Across five models from four families \(78 layer observations, two SAEs each\), we find that GI\-SAE improves geometric similarity nearly everywhere but improves functional interchangeability only where baseline sharing is moderate\. Geometric similarity alone does not guarantee functional interchangeability\.

We contribute a causal measurement framework for testing whether cross\-language SAE features are functionally interchangeable; GI\-SAE, a top\-KKSAE with an InfoNCE objective; and a five\-model, six\-language empirical study showing that baseline shared fraction stratifies GI\-SAE outcomes: 83% win rate in the convergent profile \(95% CI \[64, 93\]%\), no systematic benefit in the low\-sharing profile, and 6% in the saturated profile\.

## 2Related Work

#### Multilingual representations in LLMs\.

Language\-specific neurons can steer output language\([Tang et al\., 2024](https://arxiv.org/html/2608.23809#bib.bib1);[Kojima et al\., 2024](https://arxiv.org/html/2608.23809#bib.bib2)\), and multilingual transformers appear to route non\-English inputs through a shared latent space\([Wendler et al\., 2024](https://arxiv.org/html/2608.23809#bib.bib3);[Tezuka and Inoue, 2025](https://arxiv.org/html/2608.23809#bib.bib5)\)\. Cross\-lingual reasoning ability has been linked to neuron overlap\([Hu et al\., 2025](https://arxiv.org/html/2608.23809#bib.bib4)\), though high behavioral consistency does not necessarily imply shared internal representations\([Ifergan et al\., 2025](https://arxiv.org/html/2608.23809#bib.bib6)\)\. Our work studies*reasoning traces*rather than factual recall, and evaluates sharing at the level of individual SAE features with causal interventions across multiple checkpoints\.

#### Sparse autoencoders for interpretability\.

SAEs decompose activations into sparse, interpretable features\([Bricken et al\., 2023](https://arxiv.org/html/2608.23809#bib.bib8);[Huben et al\., 2024](https://arxiv.org/html/2608.23809#bib.bib9)\);[Gao et al\. \(2025\)](https://arxiv.org/html/2608.23809#bib.bib10)characterize top\-KKSAE scaling\. In multilingual settings,[Deng et al\. \(2025\)](https://arxiv.org/html/2608.23809#bib.bib11)identify language\-specific features and[Brinkmann et al\. \(2025\)](https://arxiv.org/html/2608.23809#bib.bib7)show that SAE features encode shared grammatical concepts with causal validation\. We ask a different question: whether cross\-language shared features are functionally interchangeable, not just geometrically similar\.

#### Causal methods for evaluating representations\.

Causal intervention tests functional necessity\([Meng et al\., 2022](https://arxiv.org/html/2608.23809#bib.bib19);[Conmy et al\., 2023](https://arxiv.org/html/2608.23809#bib.bib20)\)\. Our protocol operates at individual SAE features\. This matters because CKA can assign high similarity to functionally different representations\([Davari et al\., 2023](https://arxiv.org/html/2608.23809#bib.bib16)\); we show that geometric convergence\([Huh et al\., 2024](https://arxiv.org/html/2608.23809#bib.bib17)\)does not imply functional interchangeability\.

## 3Methodology and Measurement Framework

Each model solves the same math problems independently in six languages, producing a reasoning trace per \(problem, language\) pair\. We record the model’s internal state at selected layers during each trace, then train two SAEs on these recorded activations: a baseline SAE trained only to reconstruct them, whose learned features we compare across languages to measure naturally occurring sharing, and GI\-SAE, which adds a contrastive loss that trains the encoder to produce similar feature activations across languages for the same problem\. Comparing the two under causal patching reveals where cross\-language shared features are functionally interchangeable and where they are not\.

### 3\.1Layer Selection via CKA Survey

In a transformer, each layer adds its output to a running sum called the*residual stream*\([Elhage et al\., 2021](https://arxiv.org/html/2608.23809#bib.bib18)\)\. At layerlland token positiontt, the residual\-stream vectorxt\(l\)∈ℝdmodelx^\{\(l\)\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}is the cumulative representation that all subsequent layers read from and write to\. All SAE training and causal interventions in this work operate on residual\-stream vectors\. Patching the residual stream at layerllmodifies the state read by all downstream layersl\+1,…,Ll\{\+\}1,\\ldots,L\.

Not all layers are equally relevant\. To identify informative layers, we compute linear CKA\([Kornblith et al\., 2019](https://arxiv.org/html/2608.23809#bib.bib15)\)between pre\-answer residual\-stream matrices for each language pair at each layer, with rows matched by problem ID \(both matrices always from the same layer\)\. We define thepre\-answer tokenas the final token of the extracted reasoning span, immediately before the JSON answer block; the final JSON block and its markup are excluded\. This token is used only for the CKA survey; later evaluations use multiple backward offsets\. We select the contiguous depth range with highest mean pairwise CKA; this range differs across models \(Table[1](https://arxiv.org/html/2608.23809#S4.T1)\) because cross\-language alignment peaks at different depths\.

### 3\.2Activation Extraction and Normalization

At identified layers, we replay each reasoning trace through the model using TransformerLens\([Nanda and Bloom, 2022](https://arxiv.org/html/2608.23809#bib.bib21)\)and record the residual\-stream vector at every token position, producing oneT×dmodelT\\times d\_\{\\mathrm\{model\}\}matrix per \(problem, language, layer\) triple\.

#### Normalization and patching convention\.

SAE training and geometric evaluation use z\-scored activations \(per\-coordinate statistics from the training split\)\. The main causal\-patching sweep uses native\-scale encoding; a normalized\-space sensitivity check on representative layers shows the Qwen/convergent results are stable \(Appendix[J](https://arxiv.org/html/2608.23809#A10)\)\.

### 3\.3Sparse Autoencoder Architecture

We use a top\-KKsparse autoencoder\([Gao et al\., 2025](https://arxiv.org/html/2608.23809#bib.bib10)\)with encoderWenc∈ℝdsae×dmodelW\_\{\\text\{enc\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{sae\}\}\\times d\_\{\\mathrm\{model\}\}\}, decoderWdec∈ℝdmodel×dsaeW\_\{\\text\{dec\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\\times d\_\{\\mathrm\{sae\}\}\}, and biases\. Given a single input activationx∈ℝdmodelx\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}\(one residual\-stream vector at one token position and one layer\):

z\\displaystyle z=Wencx\+benc,f=ReLU\(TopK\(z,K\)\),\\displaystyle=W\_\{\\text\{enc\}\}\\,x\+b\_\{\\text\{enc\}\},\\;f=\\operatorname\{ReLU\}\\\!\\bigl\(\\operatorname\{TopK\}\(z,K\)\\bigr\),\(1\)x^\\displaystyle\\hat\{x\}=Wdec​f\+bdec\.\\displaystyle=W\_\{\\text\{dec\}\}\\,f\+b\_\{\\text\{dec\}\}\.Sparsity is enforced by the top\-KKbottleneck \(K=128K\{=\}128\); noℓ1\\ell\_\{1\}penalty is used\. The expansion factor4×4\{\\times\}\(dsae=4​dmodeld\_\{\\mathrm\{sae\}\}=4d\_\{\\mathrm\{model\}\}\) controls dictionary size \(8,192–12,288 features; Table[1](https://arxiv.org/html/2608.23809#S4.T1)\), whileKKcontrols per\-input sparsity\. Decoder columns are unit\-ℓ2\\ell\_\{2\}normalized after each step\. Thebaseline SAEis trained with MSE reconstruction lossℒrecon\\mathcal\{L\}\_\{\\text\{recon\}\}; by comparing which features activate across languages under this objective, we observe naturally occurring cross\-language sharing\.

### 3\.4Geometry\-Invariant SAE \(GI\-SAE\)

GI\-SAE uses the same encoder\-decoder architecture \(Equation[1](https://arxiv.org/html/2608.23809#S3.E1)\) but adds an Information Noise\-Contrastive Estimation \(InfoNCE\)\([van den Oord et al\., 2019](https://arxiv.org/html/2608.23809#bib.bib12)\)contrastive term to the reconstruction loss\. Each training activation is labeled with its source problem IDyi∈\{1,…,Q\}y\_\{i\}\\in\\\{1,\\ldots,Q\\\}, whereQQis the number of training problems\. Activations sharing the same problem ID, regardless of language or token position, form*positive pairs*; activations from different problems are*negatives*:

ℒcontr=−1\|B′\|∑i∈B′log\(∑j∈Piexp⁡\(si​j\)∑k≠iexp⁡\(si​k\)\+ϵ\),\\mathcal\{L\}\_\{\\text\{contr\}\}=\-\\frac\{1\}\{\|B^\{\\prime\}\|\}\\sum\_\{i\\in B^\{\\prime\}\}\\log\\\!\\left\(\\frac\{\\sum\_\{j\\in P\_\{i\}\}\\exp\(s\_\{ij\}\)\}\{\\sum\_\{k\\neq i\}\\exp\(s\_\{ik\}\)\}\+\\epsilon\\right\)\\\!,\(2\)wheresi​j=f~i⋅f~j/τs\_\{ij\}=\\tilde\{f\}\_\{i\}\\cdot\\tilde\{f\}\_\{j\}/\\tauis the cosine similarity ofℓ2\\ell\_\{2\}\-normalized features divided by temperatureτ=0\.1\\tau\{=\}0\.1,Pi=\{j:yj=yi,j≠i\}P\_\{i\}=\\\{j:y\_\{j\}=y\_\{i\},\\,j\\neq i\\\}is the positive set,B′=\{i:\|Pi\|\>0\}B^\{\\prime\}=\\\{i:\|P\_\{i\}\|\>0\\\}excludes items without positives, andϵ=10−8\\epsilon=10^\{\-8\}preventslog⁡0\\log 0when no positive pairs exist in the batch\. Self\-pairs are excluded from the denominator\.

The joint objective is:

ℒGI=ℒrecon\+w⋅ℒcontr\.\\mathcal\{L\}\_\{\\text\{GI\}\}=\\mathcal\{L\}\_\{\\text\{recon\}\}\+w\\cdot\\mathcal\{L\}\_\{\\text\{contr\}\}\.\(3\)We usew=1\.0w\{=\}1\.0, selected on Qwen3\-1\.7B L20 and fixed across all models \(Appendix[F](https://arxiv.org/html/2608.23809#A6)\)\.

### 3\.5Backward Alignment of Reasoning Trajectories

The same reasoning step may occur at different absolute token positions across languages\. We therefore anchor at the last reasoning token and measurebackward offsets\(−1\-1to−500\-500\)\. The contrastive dataset labels all tokens by problem ID; positives include same\-language activations from other positions, so cross\-language invariance is tested explicitly by Jaccard similarity and causal patching rather than imposed solely by the objective\. This targets the answer\-proximal phase of the trace, where discourse and answer\-transition structure are most comparable across languages\.

### 3\.6Causal Validation via Feature Patching

Geometric metrics measure whether the same features activate but not whether swapping their values preserves behavior\. We validate withcausal patching\(Algorithm[1](https://arxiv.org/html/2608.23809#alg1)\)\.

Algorithm 1Causal feature patching0:Target residual

rtgtr\_\{\\text\{tgt\}\}, donor residual

rsrcr\_\{\\text\{src\}\}, trained SAE

0:KL divergence, autoregressive flip indicator

1:

ftgt←encode⁡\(rtgt\)f\_\{\\text\{tgt\}\}\\leftarrow\\operatorname\{encode\}\(r\_\{\\text\{tgt\}\}\);

fsrc←encode⁡\(rsrc\)f\_\{\\text\{src\}\}\\leftarrow\\operatorname\{encode\}\(r\_\{\\text\{src\}\}\)
2:

S←\{j:ftgt,j\>0∧fsrc,j\>0\}S\\leftarrow\\\{j:f\_\{\\text\{tgt\},j\}\>0\\;\\wedge\\;f\_\{\\text\{src\},j\}\>0\\\}\{shared active features\}

3:

fpatched←ftgtf\_\{\\text\{patched\}\}\\leftarrow f\_\{\\text\{tgt\}\}
4:

fpatched​\[S\]←fsrc​\[S\]f\_\{\\text\{patched\}\}\[S\]\\leftarrow f\_\{\\text\{src\}\}\[S\]\{swap shared values from donor\}

5:

rpatched←decode⁡\(fpatched\)\+\(rtgt−decode⁡\(ftgt\)\)r\_\{\\text\{patched\}\}\\leftarrow\\operatorname\{decode\}\(f\_\{\\text\{patched\}\}\)\+\\bigl\(r\_\{\\text\{tgt\}\}\-\\operatorname\{decode\}\(f\_\{\\text\{tgt\}\}\)\\bigr\)\{preserve reconstruction error\}

6:Run forward pass with

rpatchedr\_\{\\text\{patched\}\}replacing

rtgtr\_\{\\text\{tgt\}\}
7:return

KL\(pclean∥ppatched\)\\mathrm\{KL\}\(p\_\{\\text\{clean\}\}\\\|p\_\{\\text\{patched\}\}\),

𝟏\[argmaxppatched≠argmaxpclean\]\\mathbf\{1\}\[\\arg\\max p\_\{\\text\{patched\}\}\\neq\\arg\\max p\_\{\\text\{clean\}\}\]\{disruption, flip\}

Each trace is replayed using its original token sequence; the main sweep uses native\-scale patching \(Appendix[J](https://arxiv.org/html/2608.23809#A10)compares to normalized\-space patching\)\. The reconstruction error is preserved \(line 5\), so the intervention modifies only the SAE feature subspace\. Output disruption is measured by KL divergence\([Kullback and Leibler, 1951](https://arxiv.org/html/2608.23809#bib.bib13)\)between the clean and patched next\-token distributions at the final reasoning\-token position\. We also record an*autoregressive flip*\(top\-1 prediction change\)\. Same\-language pairs serve as near\-zero disruption controls\. Full implementation details are in Appendix[D](https://arxiv.org/html/2608.23809#A4)\.

### 3\.7Evaluation Metrics and Decision Rule

#### Geometric metrics \(representation\-level\)\.

Linear CKA\([Kornblith et al\., 2019](https://arxiv.org/html/2608.23809#bib.bib15)\)measures geometric similarity between SAE\-encoded activation matrices of two languages for the same problem\.Jaccard similarity\([Manning et al\., 2008](https://arxiv.org/html/2608.23809#bib.bib14)\)at backward offsetδ\\delta: the set overlap of active feature indices, defined asJ=\|SA∩SB\|/\|SA∪SB\|J=\|S\_\{A\}\\cap S\_\{B\}\|/\|S\_\{A\}\\cup S\_\{B\}\|\.

#### Causal metric \(function\-level\)\.

We define theshared fractionas the proportion ofKKfeatures active in both languages:ϕ=\|S\|¯cross/K\\phi=\\overline\{\|S\|\}\_\{\\text\{cross\}\}/K\. Theautoregressive flip rate\(AFR\) is the fraction of patched traces where the model’s top\-1 next\-token prediction at the pre\-answer position changes:AFR=1N∑i𝟏\[argmaxppatched,i≠argmaxpclean,i\]\\text\{AFR\}=\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbf\{1\}\[\\arg\\max p\_\{\\text\{patched\},i\}\\neq\\arg\\max p\_\{\\text\{clean\},i\}\]\. AFR measures local disruption to the reasoning\-to\-answer transition; it does not require continuing generation or parsing the final numerical answer\. The primary metric,KL/shared feature, normalizes the cross\-language disruption by the number of features intervened on, corrected for the same\-language baseline:

KL/feat=KL¯cross−KL¯same\|S\|¯cross\.\\text\{KL/feat\}=\\frac\{\\overline\{\\text\{KL\}\}\_\{\\text\{cross\}\}\-\\overline\{\\text\{KL\}\}\_\{\\text\{same\}\}\}\{\\overline\{\|S\|\}\_\{\\text\{cross\}\}\}\.\(4\)whereKL¯cross\\overline\{\\text\{KL\}\}\_\{\\text\{cross\}\},KL¯same\\overline\{\\text\{KL\}\}\_\{\\text\{same\}\}, and\|S\|¯cross\\overline\{\|S\|\}\_\{\\text\{cross\}\}denote means over all \(problem, language pair, backward offset\) evaluation instances at that layer\. Lower values indicate greater functional interchangeability\.

#### Decision rule\.

GI\-SAE*wins*at a layer when KL/featGI<KL/featbaseline\\text\{KL/feat\}\_\{\\text\{GI\}\}<\\text\{KL/feat\}\_\{\\text\{baseline\}\}\.

#### Key distinction\.

CKA compares matrix geometry \(rotation\-invariant\); Jaccard and shared fraction compare active dictionary indices; causal patching tests whether swapping shared\-index values preserves behavior\. These three levels need not agree\.

## 4Experimental Setup

#### Models, families, and layers

We evaluate five models from four architecture families, spanning 1\.7B to 4B parameters: Qwen3\-1\.7B and Qwen3\-4B\([Yang et al\., 2025](https://arxiv.org/html/2608.23809#bib.bib24)\), Llama\-3\.2\-3B\([Grattafiori et al\., 2024](https://arxiv.org/html/2608.23809#bib.bib25)\), Gemma\-3\-4B\([Gemma Team, 2025](https://arxiv.org/html/2608.23809#bib.bib26)\), and Phi\-3\-mini\-4k\([Abdin et al\., 2024](https://arxiv.org/html/2608.23809#bib.bib27)\)\. Table[1](https://arxiv.org/html/2608.23809#S4.T1)summarizes each model’s architecture and sweep range\. The two Qwen models constitute a within\-family, same\-generation, different\-scale comparison; Gemma, Llama, and Phi each represent a distinct family\.

Table 1:Models, layer sweeps, and dataset sizes\.dsae=4×dmodeld\_\{\\mathrm\{sae\}\}=4\\times d\_\{\\mathrm\{model\}\};K=128K\{=\}128for all models\. Sweep ranges are determined by the CKA survey\. Valid problems are those solved correctly in all six languages\. Activation vectors are the total number ofdmodeld\_\{\\mathrm\{model\}\}\-dimensional residual\-stream vectors extracted across all tokens, languages, and swept layers\.ModelFamilyParamsdmodeld\_\{\\mathrm\{model\}\}SweepValidActivation vectorsproblemsTrainValQwen3\-1\.7BQwen1\.7B2048L16–L26 \(11\)1512\.77M0\.65MQwen3\-4BQwen4B2560L21–L34 \(14\)2025\.42M1\.06MLlama\-3\.2\-3BLlama3B3072L11–L27 \(17\)600\.79M0\.20MGemma\-3\-4BGemma4B2560L15–L30 \(16\)1371\.95M0\.39MPhi\-3\-miniPhi3\.8B3072L1–L20\(20\)631\.18M0\.24MTotal61312\.10M2\.54MSweep ranges differ across models because they target the region of high cross\-language CKA identified by the preliminary survey\. Phi\-3’s sweep starts at L1 because its CKA peaks at L9 \(∼31%\{\\sim\}31\\%depth\), unlike all other models which peak at 88–100% depth\.

#### Common 6\-language reasoning protocol

We use Multilingual Grade School Math \(MGSM\)\([Shi et al\., 2023](https://arxiv.org/html/2608.23809#bib.bib23)\)as the evaluation corpus: 250 grade\-school math problems, each translated into multiple languages\. Each model solves every problem independently in six languages \(en, de, fr, es, ru, zh\), producing complete chain\-of\-thought reasoning traces via nucleus sampling\([Holtzman et al\., 2020](https://arxiv.org/html/2608.23809#bib.bib22)\)\(temperature 0\.6, top\-pp0\.95, top\-kk20, max 2048 new tokens, seed42\+problem index42\+\\text\{problem index\}for reproducibility\)\.

A problem is retained as “valid” only if the model solves it correctly*and*produces an extractable reasoning trace in all six languages\. All results in this paper are therefore conditional on successful multilingual reasoning; they should not be interpreted as estimating feature sharing over the full MGSM distribution\. This yields 15 cross\-language pairs and 6 same\-language control pairs per problem\. Valid problem counts differ substantially across models because multilingual mathematical competence varies \(Table[1](https://arxiv.org/html/2608.23809#S4.T1)\)\.

#### Corpus, training, and pipeline

Each problem is presented with a language\-specific system prompt instructing step\-by\-step reasoning and a JSON\-formatted answer\. Reasoning traces are extracted from think tags \(Qwen\) or from text preceding the final JSON answer block \(Gemma, Llama, Phi\); the extracted text contains no JSON block or markup, though natural\-language answer statements may remain in the reasoning span \(full prompts and extraction rules in Appendix[E](https://arxiv.org/html/2608.23809#A5)\)\. Valid problems are split into train/validation/test \(70/15/15%, seed 42\)\. Both SAE variants use Adam with learning rate10−410^\{\-4\}, batch size 256, maximum 200 epochs, and early stopping \(patience 30, monitoring validation MSE\)\. All experiments run on a single NVIDIA RTX 5090 GPU\. The per\-model pipeline has six stages \(activation extraction, SAE training, geometric evaluation, causal patching, aggregation, analysis\); each model takes 7–10 hours, with the full sweep requiring∼\{\\sim\}60 GPU\-hours \(Appendix[D](https://arxiv.org/html/2608.23809#A4)\)\.

## 5Results

#### Cross\-language feature sharing: baseline and GI\-SAE

The baseline SAE is trained only to reconstruct activations; it has no cross\-language objective\. By comparing which of its learned features activate for the same problem across languages, we observe three baseline\-sharing profiles and several model\-specific depth trajectories \(Figure[1](https://arxiv.org/html/2608.23809#S5.F1), left\)\.

![Refer to caption](https://arxiv.org/html/2608.23809v1/img/colorblind/fig0b_combined_profiles.png)Figure 1:Cross\-language shared fraction \(fraction of 128 active features shared between two languages for the same problem\) vs\. relative depth\.Left: baseline SAE \(reconstruction only\)\.Right: GI\-SAE \(reconstruction \+ contrastive\)\. Colored bands mark the three sharing profiles: low\-sharing \(<<15%, orange\), convergent \(15–60%, green\), saturated \(\>\>60%, red\)\.Gemma shares 87–96% of features across all layers; Qwen rises from 11–15% to 50–62% with depth \(both Qwen models track closely\); Phi and Llama remain at 1–17% and 4–6% respectively\. Models of similar parameter count occupy different profiles, suggesting architecture matters more than scale\. GI\-SAE \(Figure[1](https://arxiv.org/html/2608.23809#S5.F1), right\) increases Qwen’s sharing to 16–69% and Phi’s to 9–39%, but leaves Gemma unchanged and Llama only marginally improved\.

### 5\.1Geometric Similarity Does Not Imply Functional Interchangeability

![Refer to caption](https://arxiv.org/html/2608.23809v1/img/colorblind/fig1_dissociation_scatter.png)Figure 2:Each point is one \(model, layer\) observation\.x\-axis: CKA improvement from GI\-SAE over baseline \(geometric similarity gain\)\.y\-axis: KL/feature improvement \(positive = GI\-SAE has lower KL/feature\)\. Marker size scales with baseline shared fraction\. The upper\-right quadrant \(green\) indicates both geometric and functional benefit; the lower\-right \(orange\) indicates geometric improvement*without*functional benefit\.GI\-SAE improves CKA \(Δ\>0\\Delta\>0\) at nearly every observation across all five models \(Figure[2](https://arxiv.org/html/2608.23809#S5.F2)\)\. However, functional benefit does not follow uniformly:

- •Qwenoccupies the upper\-right quadrant: geometric similarity*and*functional interchangeability improve together\. CKAΔ\\Delta\+0\.14–0\.23 \(1\.7B\) and \+0\.04–0\.32 \(4B\); GI\-SAE wins on 11/11 and 14/14 test layers, respectively\.
- •Gemmasits in the lower\-right quadrant: CKA improves by \+0\.25–0\.41, yet GI\-SAE wins on 0/16 test layers\.
- •Llamaachieves the largest CKAΔ\\Deltain the study \(up to \+0\.67\) but wins on only 7/17 test layers; sustained functional gains appear in the deepest layers \(L23–L27\), with two isolated early wins \(L12–L13\)\.
- •Phiis mixed: early layers \(L1–L6\) in the upper\-right quadrant, mid\-to\-late layers in the lower\-right\.

This demonstrates that CKA improvement is*not sufficient*for functional interchangeability\. The gap between geometric similarity and functional interchangeability is widest for models with low baseline sharing \(Llama, Phi mid\-layers\) and for the saturated model \(Gemma\)\.

Causal controls support the functional\-interchangeability interpretation: same\-problem cross\-language patches are less disruptive than both different\-problem donors \(86% pass rate\) and random\-value donors \(93% pass rate\), yielding an overall 89% pass rate across 28 condition–variant pairs \(Appendix[C](https://arxiv.org/html/2608.23809#A3)\)\. GI\-SAE features show stronger problem\-specificity than baseline features \(100% vs\. 71% on the different\-problem control\)\.

### 5\.2A Phase Diagram for GI\-SAE Success

![Refer to caption](https://arxiv.org/html/2608.23809v1/img/colorblind/fig2_regime_map.png)Figure 3:Phase diagram: GI\-SAE KL/feature benefit vs\. baseline shared fraction \(test split\)\. Points above the dashed line indicate GI\-SAE wins\. Three profiles emerge with distinct win rates\.When all 78 observations are stratified by baseline shared fraction regardless of model identity \(Figure[3](https://arxiv.org/html/2608.23809#S5.F3)\), a useful empirical profile structure emerges:

Table 2:GI\-SAE win rates by baseline sharing profile \(test split\)\. 95% Wilson confidence intervals in brackets\.GI\-SAE reliably helps in the*convergent*profile \(enough baseline sharing to build on\), is counterproductive in the*saturated*profile \(Gemma; swapping heavily\-used shared features increases disruption\), and shows no systematic benefit in the*low\-sharing*profile\. Baseline shared fraction thus acts as a useful diagnostic in this sample\.

### 5\.3Family Profiles of Cross\-Language Feature Sharing

![Refer to caption](https://arxiv.org/html/2608.23809v1/img/colorblind/fig3_taxonomy_panel.png)Figure 4:Three representative family profiles\.Top: baseline shared fraction vs\. layer\.Middle: CKA \(baseline blue, GI\-SAE orange\)\.Bottom: KL/feature \(lower = greater functional interchangeability; shading marks GI\-SAE advantage in green, disadvantage in orange\)\. Left: Gemma \(saturated\)\. Center: Qwen3\-4B \(convergent\)\. Right: Phi\-3\-mini \(early\-extraction\)\.The five models reveal three recurring multilingual processing profiles \(Figure[4](https://arxiv.org/html/2608.23809#S5.F4); Llama and Qwen3\-1\.7B patterns are shown in the appendix\)\.

Qwenshows progressive sharing: baseline shared fraction rises monotonically with depth, and GI\-SAE wins on all layers for both models, which closely track one another\.Gemmaalready shares∼\{\\sim\}90% of features at baseline; GI\-SAE increases KL/feature at all 16 layers, supporting the interpretation that the baseline already captures available cross\-language structure\.Llamaachieves the largest CKA gains \(up to\+0\.67\+0\.67\) but functional gains appear only in the deepest layers \(L23–L27, with two isolated early wins at L12–L13\); mid\-depth layers \(L14–L22\) show no improvement\.Phipeaks in CKA at L9 \(∼\{\\sim\}31% depth\); GI\-SAE wins in early layers \(L1–L6\) and loses from L7 onward, mirroring Llama in reverse\.

A qualitative audit of the most consistently shared features \(Appendix[I](https://arxiv.org/html/2608.23809#A9)\) shows that they encode interpretable cross\-language discourse structure: answer\-boundary markers that fire on sentence\-final punctuation across 4–6 languages, answer\-transition phrases that fire on the verb introducing the answer regardless of surface form, and reasoning\-step markers active even at early layers \(L3 in Phi\)\. These categories appear across the observed model profiles, though the strongest shared features are concentrated near the reasoning\-to\-answer transition rather than in mid\-reasoning computation\.

### 5\.4Robustness and Training Health

Val and test KL/feature benefits are strongly aligned across the 78 layer observations \(descriptive Pearsonr=0\.894r=0\.894; Appendix[H](https://arxiv.org/html/2608.23809#A8)\)\. A normalized\-space patching sensitivity check preserves the Qwen/convergent conclusion while showing that low/mixed layers are more convention\-sensitive \(Appendix[J](https://arxiv.org/html/2608.23809#A10)\)\. SAE training diagnostics are in Appendix[F](https://arxiv.org/html/2608.23809#A6): baseline SAEs maintain\>\>99% alive features; GI\-SAE remains mostly high but drops to 84–88% for Llama and late Qwen\-1\.7B layers\. Table[3](https://arxiv.org/html/2608.23809#S5.T3)summarizes the five\-model sweep\.

Table 3:Five\-model summary \(test split\)\. 95% Wilson CIs for win rates\. Full per\-layer results in Appendix[B](https://arxiv.org/html/2608.23809#A2)\.

## 6Discussion

#### Interpretive scope of GI\-SAE

GI\-SAE is informative even when it does not improve functional interchangeability\. In the*saturated*profile \(Gemma\), GI\-SAE’s inability to improve sharing supports the interpretation that the baseline SAE already captures the cross\-language structure present in the model; the contrastive loss has nothing to add\. In the*low\-sharing*profile \(Llama early layers\), GI\-SAE achieves large geometric similarity gains \(CKAΔ\\Deltaup to \+0\.67\) without improving functional interchangeability, showing that CKA and Jaccard measure representation similarity, not whether shared features can be swapped without disrupting output\.

This gap is a general methodological insight: wherever CKA is used to infer functional equivalence, the same gap may apply\. Baseline shared fraction \(Table[2](https://arxiv.org/html/2608.23809#S5.T2)\) stratifies whether contrastive training helps in this sample\.

### 6\.1Limitations and Future Directions

#### Limitations\.

The taxonomy is descriptive: each non\-Qwen family is one checkpoint; Qwen provides only a within\-generation scale check\. \(i\) MGSM only; may not generalize to harder math, code, or non\-mathematical domains\. \(ii\) Llama and Phi have∼\{\\sim\}60 valid problems vs\. 151–202 for Qwen, limiting statistical power\. \(iii\) Models up to 4B; scaling to 8B\+ reduces the active dictionary fraction\. \(iv\) Fixed hyperparameters \(K=128K\{=\}128,4×4\{\\times\},τ=0\.1\\tau\{=\}0\.1,w=1\.0w\{=\}1\.0\) across all models;w=1\.0w\{=\}1\.0was selected from a single Qwen3\-1\.7B L20 comparison and held fixed as a common intervention strength \(Appendix[F](https://arxiv.org/html/2608.23809#A6)\)\. \(v\) Contrastive positives are problem\-level, not offset\-matched; this may align answer\-level rather than step\-level information\. \(vi\) One sampled trace per problem\-language pair; multiple traces would separate representation effects from sampling variance\.

#### Future directions\.

\(i\) Within\-family replications \(e\.g\., Gemma\-2\-2B, Llama\-3\.2\-1B\) to test whether sharing profiles are family\-stable\. \(ii\) CKA\-only lightweight surveys across additional models to extend the phase diagram without full pipeline cost\. \(iii\) Extension beyond mathematical reasoning to code, logical inference, and general instruction following\. \(iv\) Adaptive contrastive weight or temperature scheduling conditioned on baseline shared fraction\. \(v\) Investigation of what architectural or training choices produce high default feature sharing \(Gemma\) vs\. progressive building \(Qwen\) vs\. persistent language\-specificity \(Llama\)\.

## 7Conclusion

We introduced GI\-SAE, a contrastive sparse autoencoder, and applied it alongside a baseline SAE to five models from four families under a common 6\-language mathematical reasoning protocol\. Cross\-language feature sharing varies strongly across models \(progressive in Qwen, saturated in Gemma, mixed in Llama and Phi\) and geometric similarity does not reliably predict functional interchangeability: GI\-SAE improves CKA at nearly every layer, yet KL/feature improves only in the convergent profile \(83% win rate\) and not in the saturated profile \(6%\)\.

These findings show that geometric similarity should not be used as a proxy for functional interchangeability, and that cross\-language feature sharing must be analyzed on a per\-model and per\-architecture basis\.

## Impact Statement

This work is a foundational mechanistic\-interpretability study of internal representations in existing open\-source language models\. It does not introduce new model capabilities, generate content, or release new pretrained models or datasets with direct deployment implications\. Its potential positive impact is improved understanding of multilingual model behavior, which may support future interpretability and auditing research\. The study uses public data and involves no human subjects or private information\. We do not foresee specific negative societal impacts beyond those generally associated with research on existing large language models\.

## References

- M\. Abdin J\. Anejaet al\.Phi\-3 technical report: a highly capable language model locally on your phone\.arXiv preprint arXiv:2404\.14219\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2404.14219),[Link](https://arxiv.org/abs/2404.14219)Cited by:[§4](https://arxiv.org/html/2608.23809#S4.SS0.SSS0.Px1.p1.1)\.
- Brickenet al\.\(2023\)T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. L\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. OlahTowards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2023/monosemantic-features/index.html)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px2.p1.1)\.
- Brinkmannet al\.\(2025\)J\. Brinkmann, C\. Wendler, C\. Bartelt, and A\. MuellerLarge language models share representations of latent grammatical concepts across typologically diverse languages\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 6131–6150\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.312),[Link](https://aclanthology.org/2025.naacl-long.312/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px2.p1.1)\.
- Conmyet al\.\(2023\)A\. Conmy, A\. N\. Mavor\-Parker, A\. Lynch, S\. Heimersheim, and A\. Garriga\-AlonsoTowards automated circuit discovery for mechanistic interpretability\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://openreview.net/forum?id=89ia77nZ8u)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px3.p1.1)\.
- Davariet al\.\(2023\)M\. Davari, S\. Horoi, A\. Natik, G\. Lajoie, G\. Wolf, and E\. BelilovskyReliability of CKA as a similarity measure in deep learning\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8HRvyxc606)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px3.p1.1)\.
- Denget al\.\(2025\)B\. Deng, Y\. Wan, B\. Yang, Y\. Zhang, and F\. FengUnveiling language\-specific features in large language models via sparse autoencoders\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 4563–4608\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.229),[Link](https://aclanthology.org/2025.acl-long.229/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px2.p1.1)\.
- Elhageet al\.\(2021\)N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, N\. DasSarma, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. OlahA mathematical framework for transformer circuits\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§3\.1](https://arxiv.org/html/2608.23809#S3.SS1.p1.1)\.
- Gaoet al\.\(2025\)L\. Gao, T\. Dupré la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. WuScaling and evaluating sparse autoencoders\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tcsZt9ZNKD)Cited by:[§1](https://arxiv.org/html/2608.23809#S1.p3.1),[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2608.23809#S3.SS3.p1.2)\.
- Gemma Team \(2025\)Gemma TeamGemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2503.19786),[Link](https://arxiv.org/abs/2503.19786)Cited by:[§4](https://arxiv.org/html/2608.23809#S4.SS0.SSS0.Px1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori A\. Dubeyet al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2407.21783),[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4](https://arxiv.org/html/2608.23809#S4.SS0.SSS0.Px1.p1.1)\.
- Holtzmanet al\.\(2020\)A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. ChoiThe curious case of neural text degeneration\.InThe Eighth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rygGQyrFvH)Cited by:[§4](https://arxiv.org/html/2608.23809#S4.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2025\)P\. Hu, S\. Liu, C\. Gao, X\. Huang, X\. Han, J\. Feng, C\. Deng, and S\. HuangLarge language models are cross\-lingual knowledge\-free reasoners\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 1525–1542\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.72),[Link](https://aclanthology.org/2025.naacl-long.72/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px1.p1.1)\.
- Hubenet al\.\(2024\)R\. Huben, H\. Cunningham, L\. R\. Smith, A\. Ewart, and L\. SharkeySparse autoencoders find highly interpretable features in language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=F76bwRSLeK)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px2.p1.1)\.
- Huhet al\.\(2024\)M\. Huh, B\. Cheung, T\. Wang, and P\. IsolaPosition: the platonic representation hypothesis\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 20617–20642\.External Links:[Link](https://proceedings.mlr.press/v235/huh24a.html)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px3.p1.1)\.
- Iferganet al\.\(2025\)M\. Ifergan, L\. Choshen, R\. Aharoni, I\. Szpektor, and O\. AbendBeneath the surface of consistency: exploring cross\-lingual knowledge representation sharing in LLMs\.InFindings of the Association for Computational Linguistics: NAACL 2025,Albuquerque, New Mexico,pp\. 4630–4644\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.475/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px1.p1.1)\.
- Kojimaet al\.\(2024\)T\. Kojima, I\. Okimura, Y\. Iwasawa, H\. Yanaka, and Y\. MatsuoOn the multilingual ability of decoder\-based pre\-trained language models: finding and controlling language\-specific neurons\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 6919–6971\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.384),[Link](https://aclanthology.org/2024.naacl-long.384/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px1.p1.1)\.
- Kornblithet al\.\(2019\)S\. Kornblith, M\. Norouzi, H\. Lee, and G\. HintonSimilarity of neural network representations revisited\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 3519–3529\.External Links:[Link](https://proceedings.mlr.press/v97/kornblith19a.html)Cited by:[§3\.1](https://arxiv.org/html/2608.23809#S3.SS1.p2.1),[§3\.7](https://arxiv.org/html/2608.23809#S3.SS7.SSS0.Px1.p1.1)\.
- Kullback and Leibler \(1951\)S\. Kullback and R\. A\. LeiblerOn information and sufficiency\.The Annals of Mathematical Statistics22\(1\),pp\. 79–86\.External Links:[Document](https://dx.doi.org/10.1214/aoms/1177729694)Cited by:[§3\.6](https://arxiv.org/html/2608.23809#S3.SS6.p2.1)\.
- Manninget al\.\(2008\)C\. D\. Manning, P\. Raghavan, and H\. SchützeIntroduction to information retrieval\.Cambridge University Press,Cambridge, UK\.External Links:[Document](https://dx.doi.org/10.1017/CBO9780511809071)Cited by:[§3\.7](https://arxiv.org/html/2608.23809#S3.SS7.SSS0.Px1.p1.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 17359–17372\.External Links:[Link](https://openreview.net/forum?id=-h6WAS6eE4)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px3.p1.1)\.
- Nanda and Bloom \(2022\)N\. Nanda and J\. BloomTransformerLens\.Note:[https://github\.com/TransformerLensOrg/TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)Cited by:[§D\.2](https://arxiv.org/html/2608.23809#A4.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.23809#S3.SS2.p1.1)\.
- Shiet al\.\(2023\)F\. Shi, M\. Suzgun, M\. Freitag, X\. Wang, S\. Srivats, S\. Vosoughi, H\. W\. Chung, Y\. Tay, S\. Ruder, D\. Zhou, D\. Das, and J\. WeiLanguage models are multilingual chain\-of\-thought reasoners\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fR3wGCk-IXp)Cited by:[§4](https://arxiv.org/html/2608.23809#S4.SS0.SSS0.Px2.p1.1)\.
- Tanget al\.\(2024\)T\. Tang, W\. Luo, H\. Huang, D\. Zhang, X\. Wang, X\. Zhao, F\. Wei, and J\. WenLanguage\-specific neurons: the key to multilingual capabilities in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 5701–5715\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.309),[Link](https://aclanthology.org/2024.acl-long.309/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px1.p1.1)\.
- Tezuka and Inoue \(2025\)H\. Tezuka and N\. InoueThe transfer neurons hypothesis: an underlying mechanism for language latent space transitions in multilingual LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 31742–31792\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1618),[Link](https://aclanthology.org/2025.emnlp-main.1618/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px1.p1.1)\.
- van den Oordet al\.\(2019\)A\. van den Oord, Y\. Li, and O\. VinyalsRepresentation learning with contrastive predictive coding\.External Links:1807\.03748,[Document](https://dx.doi.org/10.48550/arXiv.1807.03748),[Link](https://arxiv.org/abs/1807.03748)Cited by:[§1](https://arxiv.org/html/2608.23809#S1.p3.1),[§3\.4](https://arxiv.org/html/2608.23809#S3.SS4.p1.1)\.
- Wendleret al\.\(2024\)C\. Wendler, V\. Veselovsky, G\. Monea, and R\. WestDo llamas work in English? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 15366–15394\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.820),[Link](https://aclanthology.org/2024.acl-long.820/)Cited by:[§2](https://arxiv.org/html/2608.23809#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4](https://arxiv.org/html/2608.23809#S4.SS0.SSS0.Px1.p1.1)\.

## Appendix AFull Per\-Model Layer Profiles

Figure[4](https://arxiv.org/html/2608.23809#S5.F4)in the main text shows depth profiles for Gemma, Qwen3\-4B, and Phi\-3\-mini\. Here we present the remaining two models: Llama\-3\.2\-3B and Qwen3\-1\.7B\.

#### Llama\-3\.2\-3B \(low\-sharing profile\)\.

Baseline shared fraction remains at 4–6% across all 17 swept layers \(L11–L27\)\. GI\-SAE increases sharing to 1–17%, with gradual growth toward deeper layers\. CKA improvements are the largest in the study \(up to\+0\.67\+0\.67\), yet sustained KL/feature improvements appear only in the deepest layers \(L23–L27\), with two isolated early wins \(L12–L13\)\. Mid\-depth layers \(L14–L22\) show geometric gain without functional benefit, indicating that the model processes languages through largely separate pathways at those depths\.

#### Qwen3\-1\.7B \(convergent profile\)\.

Shared fraction increases monotonically from 11% at L16 to 62% at L26, closely tracking Qwen3\-4B and confirming within\-family consistency under scale change\. GI\-SAE wins on all 11 test layers \(100% win rate, 95% CI \[74%, 100%\]\)\. Peak shared fraction reaches 67% under GI\-SAE\. The 1\.7B model achieves slightly lower peak CKA \(0\.69 vs\. 0\.79 for 4B\) but follows the same progressive\-sharing trajectory\.

## Appendix BFull Layer\-by\-Layer Results

Tables[4](https://arxiv.org/html/2608.23809#A2.T4)–[8](https://arxiv.org/html/2608.23809#A2.T8)report all 78 \(model, layer\) observations\. Shared fraction is\|S\|/K\|S\|/KwhereK=128K\{=\}128\. KL/feat \(×103\\times 10^\{3\}\) is the test\-split normalized metric \(Equation[4](https://arxiv.org/html/2608.23809#S3.E4)\)\. “Win” \(✓\) indicatesKL/featGI<KL/featBL\\text\{KL/feat\}\_\{\\text\{GI\}\}<\\text\{KL/feat\}\_\{\\text\{BL\}\}\.

Table 4:Qwen3\-1\.7B layer\-by\-layer results \(test split, L16–L26\)\. Shared% =\|S\|/128\|S\|/128\. KL/feat×103\\times 10^\{3\}\. GI\-SAE wins 11/11 layers\.Table 5:Qwen3\-4B layer\-by\-layer results \(test split, L21–L34\)\. GI\-SAE wins 14/14 layers\.Table 6:Llama\-3\.2\-3B layer\-by\-layer results \(test split, L11–L27\)\. GI\-SAE wins 7/17 layers: L12–L13 and L23–L27\.Table 7:Gemma\-3\-4B layer\-by\-layer results \(test split, L15–L30\)\. GI\-SAE wins 0/16 layers\.Table 8:Phi\-3\-mini layer\-by\-layer results \(test split, L1–L20\)\. GI\-SAE wins 7/20 layers, all in L1–L6 and L9\.
## Appendix CCausal Patching Controls

To test whether low cross\-language KL divergence reflects genuine functional interchangeability, we add two control conditions to the standard patching protocol, using the same trained SAEs \(no retraining\):different\-problem donor\(features from a different math problem, same source language\) andrandom\-value donor\(same shared feature indices, but activation values permuted across those indices\)\. If same\-problem cross\-language patches are less disruptive than both controls, the shared features encode problem\-specific reasoning, not generic structure\.

Table[9](https://arxiv.org/html/2608.23809#A3.T9)reports KL/shared feature for 7 representative layers across all 5 models and both SAE variants\.

Table 9:Causal patching controls \(test split\)\. SP = same\-problem cross\-language, DP = different\-problem, RV = random\-value\. Pass \(✓\\checkmark\) = control KL/feat\>\>SP KL/feat\.ModelLayerProfileVariantSPDPRVDP\>\>SPRV\>\>SPQwen\-4BL29ConvBL0\.00880\.01140\.0127✓✓GI0\.00440\.00710\.0111✓✓Qwen\-1\.7BL24ConvBL0\.00810\.00920\.0194✓✓GI0\.00260\.00320\.0064✓✓LlamaL17LowBL0\.01440\.02060\.0164✓✓GI0\.04950\.07410\.0519✓✓LlamaL25LowBL0\.00940\.01060\.0125✓✓GI0\.00510\.00800\.0056✓✓GemmaL21SatBL0\.00880\.00810\.0645—✓GI0\.01600\.01910\.0984✓✓PhiL3LowBL0\.00610\.00190\.0063—✓GI0\.00100\.00110\.0011✓✓PhiL15LowBL0\.00340\.00420\.0041✓✓GI0\.01950\.03330\.0192✓—Pass rate \(BL\)5/77/771%100%Pass rate \(GI\)7/76/7100%86%Overall12/1413/1486%93%Overall pass rate is 89% \(25/28\)\. The three failures are explainable edge cases: \(1\) Gemma BL different\-problem \(0\.0081 vs\. 0\.0088\): saturated profile shares∼\{\\sim\}90% of features regardless of problem, so different\-problem donors are nearly indistinguishable; \(2\) Phi L3 BL different\-problem \(0\.0019 vs\. 0\.0061\): very low baseline sharing \(∼\{\\sim\}13 features\) yields too few patched features for problem\-specificity to manifest; \(3\) Phi L15 GI random\-value \(0\.0192 vs\. 0\.0195\): statistical tie \(ratio 0\.98×\\times\)\.

GI\-SAE features show stronger problem\-specificity than baseline features: GI passes the different\-problem control 100% \(7/7\) vs\. baseline 71% \(5/7\)\.

## Appendix DPipeline and Implementation Details

### D\.1Per\-Model Experimental Pipeline

For each model, the pipeline proceeds through six stages: \(1\) activation extraction: replay all valid reasoning traces and record residual\-stream vectors at each target layer for every token; \(2\) SAE training: train baseline SAE and GI\-SAE on train\-split activations at each layer; \(3\) geometric evaluation: compute CKA and Jaccard similarity between language pairs on validation activations encoded through each SAE; \(4\) causal patching: run Algorithm 1 for all \(problem, language pair\) combinations on both validation and test splits; \(5\) aggregation: collect per\-layer results into a single results file; \(6\) analysis: generate per\-model comparison plots\. Each model’s pipeline takes approximately 7–10 hours; the complete five\-model sweep requires∼\{\\sim\}60 GPU\-hours on a single NVIDIA RTX 5090\.

### D\.2Causal Patching Protocol

For each valid reasoning trace, we replay the exact generated token sequence \(replaying exact generated tokens\) using TransformerLens\([Nanda and Bloom, 2022](https://arxiv.org/html/2608.23809#bib.bib21)\)\. At the target layer and a backward\-aligned token position, we record the target residual\-stream vectorrtgtr\_\{\\text\{tgt\}\}and the donor residual\-stream vectorrsrcr\_\{\\text\{src\}\}\(from a different language’s trace of the same problem\)\.

Both vectors are encoded through the trained SAE*without*z\-score normalization, because the patched vector is injected back into the model’s forward pass at the original activation scale\. Features active in both encodings \(ftgt,j\>0f\_\{\\text\{tgt\},j\}\>0andfsrc,j\>0f\_\{\\text\{src\},j\}\>0\) constitute the shared setSS\.

The patched residual is constructed as:

rpatched=decode⁡\(fpatched\)\+\(rtgt−decode⁡\(ftgt\)\),r\_\{\\text\{patched\}\}=\\operatorname\{decode\}\(f\_\{\\text\{patched\}\}\)\+\(r\_\{\\text\{tgt\}\}\-\\operatorname\{decode\}\(f\_\{\\text\{tgt\}\}\)\),wherefpatchedf\_\{\\text\{patched\}\}equalsftgtf\_\{\\text\{tgt\}\}except at indices inSS, which take donor values\. Adding the reconstruction error\(rtgt−decode⁡\(ftgt\)\)\(r\_\{\\text\{tgt\}\}\-\\operatorname\{decode\}\(f\_\{\\text\{tgt\}\}\)\)ensures the intervention modifies only the SAE feature subspace and preserves the residual component\.

The patched vector replacesrtgtr\_\{\\text\{tgt\}\}at the same layer and position via a TransformerLens hook, and the forward pass continues over all remaining layers\. KL divergence is measured between the clean and patched next\-token distributions at the*final reasoning\-token position*\(backward offset−1\-1\), after full downstream propagation\. Autoregressive flip rate \(AFR\) records whether the top\-1 next\-token prediction at that position changes\. AFR is a local prediction\-sensitivity metric, not a regenerated final\-answer comparison\.

When\|S\|=0\|S\|=0\(no shared active features\), the intervention is skipped for that \(problem, language pair, position\) triple\. Same\-language pairs serve as a near\-zero disruption control: because both encodings are identical \(same trace replayed\), patching produces no change\.

Backward offsets are processed independently; results at each offset are averaged across all valid \(problem, language pair\) combinations\. The primary metrics reported in the main text aggregate across all offsets\.

## Appendix EDataset Filtering and Prompts

### E\.1Prompts

Each problem is presented via a system prompt instructing the model to reason in the target language and a user prompt containing the problem\. Table[10](https://arxiv.org/html/2608.23809#A5.T10)shows the system prompts for all six languages\.111Russian and Chinese prompts are shown in romanized transliteration due to typesetting constraints\.

Table 10:System prompts by language\.User prompts follow the template: “Solve the problem\. Think step by step: \{problem\}\. Provide your final numerical answer strictly as JSON without units: \{key: VALUE\}\.” Equivalent translations are used for each language with localized JSON answer keys\.

### E\.2Reasoning Trace Extraction

Two strategies are used depending on model architecture:

1. 1\.Qwen models: explicit reasoning inside<think\>…\\ldots</think\>tags is extracted via regex\.
2. 2\.Gemma, Llama, Phi: reasoning is defined as all text preceding the final JSON answer block \(e\.g\.,\{‘‘answer’’: 42\}\)\. Trailing markdown code fences are stripped\.

In both cases, the extractedthink\_textcontains only reasoning with no JSON answer block or markup, though natural\-language answer statements may remain in the reasoning span\.

### E\.3Filtering and Validity

A problem is retained only if the model solves it correctly*and*produces an extractable reasoning trace in all six languages\. All results in this paper are therefore conditional on successful multilingual reasoning\. This design isolates the setting where cross\-language feature comparison is meaningful, but should not be interpreted as estimating feature sharing over the full MGSM distribution\.

Table[1](https://arxiv.org/html/2608.23809#S4.T1)in the main text reports valid problem counts per model: 202 \(Qwen\-4B\), 151 \(Qwen\-1\.7B\), 137 \(Gemma\), 63 \(Phi\), 60 \(Llama\)\. The large variance reflects differences in multilingual mathematical competence\.

### E\.4Inference Parameters

Table 11:Generation hyperparameters \(all models\)\.

## Appendix FSAE Training Diagnostics

Table[12](https://arxiv.org/html/2608.23809#A6.T12)summarizes training health across all models\. Baseline SAE achieves\>\>99% alive features everywhere\. GI\-SAE alive fraction remains above 90% for most layers but drops to 84–88% for Llama and late\-layer Qwen\-1\.7B, where the contrastive and reconstruction objectives are in strongest tension\. GI\-SAE validation loss is typically 1\.5–2\.0×\\timesbaseline, with spikes above 3×\\timesat select Qwen layers \(e\.g\., L16 and L19 for Qwen\-1\.7B\)\. These spikes do not affect downstream evaluation: affected layers still produce SAEs with\>\>90% active features\.

Table 12:SAE training diagnostics \(representative layers\)\. Val loss is reconstruction MSE on the validation split\. Alive% is the fraction ofdsaed\_\{\\text\{sae\}\}features firing on at least one validation sample\.ModelLayerBL ValGI ValRatioGI Alive%NoteQwen\-1\.7BL160\.2130\.6733\.2×\\times99\.7High ratioL190\.1530\.6494\.2×\\times99\.6High ratioL220\.1610\.3131\.9×\\times89\.4L260\.1930\.3431\.8×\\times88\.7Qwen\-4BL210\.2250\.6843\.0×\\times98\.7High ratioL220\.1970\.6753\.4×\\times98\.6High ratioL290\.1670\.3141\.9×\\times94\.3L340\.1750\.3231\.8×\\times94\.3LlamaL110\.3060\.3851\.3×\\times91\.7L170\.2530\.3561\.4×\\times86\.4L270\.2470\.3631\.5×\\times89\.9GemmaL150\.2590\.3571\.4×\\times96\.4L210\.1940\.3091\.6×\\times95\.9L300\.2020\.3641\.8×\\times95\.9PhiL10\.1280\.2512\.0×\\times99\.0L90\.2260\.3731\.7×\\times96\.8L200\.2100\.3091\.5×\\times96\.1#### Contrastive weight selection\.

We usew=1\.0w\{=\}1\.0as a fixed contrastive weight rather than tuningwwper model or layer\. This value was selected from a preliminary comparison on Qwen3\-1\.7B L20 using the merged single\-pass implementation\. Atw=1\.0w\{=\}1\.0, CKA improved from 0\.503 to 0\.673 and KL/feature improved from 0\.0070 to 0\.0003 relative to the baseline, while validation loss increased only from 31\.60 to 32\.29\. A smaller weight \(w=0\.1w\{=\}0\.1\) preserved reconstruction loss but produced weaker invariance \(CKA 0\.492, KL/feature 0\.0055\)\. We therefore fixedw=1\.0w\{=\}1\.0across the five\-model sweep; adaptive weighting remains future work\.

## Appendix GKL Decomposition

Table[13](https://arxiv.org/html/2608.23809#A7.T13)decomposes the KL/shared feature metric into its numerator \(raw cross\-language KL\) and denominator \(mean shared feature count\|S\|\|S\|\), aggregated by sharing profile\. This verifies that the metric is not an artifact of division by\|S\|\|S\|\.

Table 13:KL decomposition by sharing profile \(test split, all layers\)\. Raw KL = mean cross\-language KL divergence\.\|S\|\|S\|= mean shared feature count\.\|S\|/128\|S\|/128= shared fraction\. KL/feat = primary metric\. AFR = autoregressive flip rate\.In theconvergentprofile, GI\-SAE wins because both components improve: raw KL drops \(0\.43→0\.250\.43\\to 0\.25,−41%\-41\\%\) and sharing increases \(42%→60%42\\%\\to 60\\%,\+43%\+43\\%\)\. In thelow\-sharingprofile, GI\-SAE doubles sharing \(12%→24%12\\%\\to 24\\%\) but also increases raw KL \(0\.20→0\.300\.20\\to 0\.30\), yielding only marginal KL/feat improvement\. In thesaturatedprofile, sharing is ceiling\-ed \(89%≈89%89\\%\\approx 89\\%\) while raw KL more than doubles \(1\.58→3\.521\.58\\to 3\.52\), making the GI\-SAE intervention destructive\.

## Appendix HStatistical Uncertainty

![Refer to caption](https://arxiv.org/html/2608.23809v1/img/colorblind/fig7_val_test_consistency.png)Figure 5:Val vs\. test KL/feature benefit for all 78 observations \(descriptive Pearsonr=0\.894r=0\.894; layer observations are nested within models and should not be treated as independent samples\)\.Table[14](https://arxiv.org/html/2608.23809#A8.T14)reports 95% Wilson score confidence intervals for GI\-SAE win rates\. The Wilson intervals support qualitative separation between the convergent and saturated profiles: the convergent CI lower bound \(64%\) does not overlap with the saturated CI upper bound \(27%\)\.

Table 14:GI\-SAE win rates with 95% Wilson confidence intervals \(test split\)\.ScopeWinsNNRate95% Wilson CIQwen3\-1\.7B1111100%\[74%, 100%\]Qwen3\-4B1414100%\[78%, 100%\]Llama\-3\.2\-3B71741%\[22%, 64%\]Gemma\-3\-4B0160%\[0%, 19%\]Phi\-3\-mini72035%\[18%, 57%\]Low\-sharing183749%\[33%, 64%\]Convergent202483%\[64%, 93%\]Saturated1176%\[1%, 27%\]Overall397850%\[39%, 61%\]Under a layer\-level Bernoulli summary, the Qwen intervals lie above 50% \(lower bounds 74% and 78%\), while Gemma lies below 50% \(upper bound 19%\)\. Because layers within a model are correlated, we interpret these intervals descriptively\. The low\-sharing CI \[33%, 64%\] straddles 50%, consistent with the characterization that GI\-SAE helps only sometimes in this profile\.

## Appendix IQualitative Feature Audit

To verify that SAE features encode meaningful cross\-language discourse and answer\-boundary structure rather than arbitrary sparse coordinates, we audit the most consistently shared features from four representative \(model, layer\) pairs, one per sharing profile\. For each pair, we encode test\-split activations through the SAE, identify features active in≥\\geq4 of 6 languages at the pre\-answer token, rank by consistency \(nproblems×mean activationn\_\{\\text\{problems\}\}\\times\\text\{mean activation\}\), and map peak\-activation positions back to text snippets\.

Table 15:Cross\-language shared feature audit\. Bracketed tokens mark peak activation positions\. All features fire on semantically equivalent content across 4–6 languages\.ModelLayerFeatureLangsCategoryRepresentative snippetsQwen\-4BL29F96625/6BoundaryEN: “should be 29\[\.\]” / DE: “$29\.00\[\.\]” / ZH: “29\(yuan\)\[\.\]”F85195/6BoundaryEN: “answer should be 29\[\.\]” / DE: “correct total\[\.\]”F35965/6BoundaryEN: “that seems right\[\.\]” / FR: “that’s correct\[\.\]”Gemma\-4BL21F34116/6BoundaryEN: “final answer is 30\[\.\]” / DE: “gegeben\[\.\]” / ZH: “30\(yuan\)\[\.\]”F52994/6BoundaryEN: “Seth is 16\[\.\]” / FR: “16 ans\[\.\]”Llama\-3BL17F38824/6TransitionDE: “Antwort\[ist\]” / ES: “respuesta\[es\]” / ZH: “huida\[:\]’F64935/6BoundaryEN: “23 jewels\[\.\]” / FR: “23 bijoux\[\.\]” / ZH: “da’an\[:\]”Phi\-miniL3F21626/6Step/formatEN: “First,\[let\]’s find” / ES: “Prim\[ero\], calculamos” / DE: “finden\[wir\]heraus”F64456/6Format trans\.DE: “JSON\-Antwort\[:\]’ / FR: “JSON est donc\[:\]” / RU: “JSON\[:\]”Three feature categories emerge:

- •Answer\-boundary markers\(F9662, F8519, F3596, F3411, F5299, F6493\): fire on punctuation \(periods, colons\) at the reasoning→\\toanswer transition\. Language\-invariant because final answers are always numeric\.
- •Answer\-transition phrases\(F3882\): fire on the verb or copula introducing the answer \(“is”/“ist”/“es”/“:” \[zh\]\)\. Captures syntactic role, not surface form\.
- •Step/format markers\(F2162, F6445\): fire on tokens initiating reasoning steps or format transitions\. Active even at L3, suggesting early detection of discourse structure\.

## Appendix JPatching Convention Sensitivity

The main sweep uses native\-scale patching, while SAE training and geometric evaluation use z\-scored activations\. To test whether the causal conclusions depend on this convention, we rerun patching on representative layers using normalized\-space patching: residuals are normalized before SAE encoding, patched in normalized feature space, decoded, and unnormalized before injection into the residual stream\.

Table 16:KL/feature under native\-scale and normalized\-space patching on representative layers\. Lower is better\. Gemma L21 is omitted because this diagnostic script produced no valid normalized\-space interventions for that layer; we do not interpret this as evidence about Gemma’s sharing profile\. The full main sweep reports Gemma under the native\-scale convention\.The two Qwen convergent layers remain GI\-SAE wins under both conventions, and the aggregate win rate across valid layers is unchanged at 4/6 under both conventions\. However, normalized\-space patching changes absolute shared\-feature counts and flips several individual low/mixed\-profile layers\. We therefore interpret the Qwen/convergent conclusion as robust, but treat exact layer\-level verdicts in low/mixed profiles as convention\-sensitive\.