Rethinking the Multilingual Reasoning Gap with Layer Swap

# Rethinking the Multilingual Reasoning Gap with Layer Swap
Source: [https://arxiv.org/html/2605.26735](https://arxiv.org/html/2605.26735)
Maxence Lasbordes1,2Amélie Chatelain1Djamé Seddah2 1LightOn, Paris2Inria, Paris \{maxence\.lasbordes, amelie\}@lighton\.ai djame\.seddah@inria\.fr

###### Abstract

Recent reasoning Large Language Models produce a chain\-of\-thought \(CoT\) predominantly in English, even when prompted in non\-English languages\. Prior work suggests that forcing the CoT to remain in the input language \(*native reasoning*\) substantially degrades performance relative to allowing the model to reason in English before answering in the input language \(*English\-pivoted reasoning*\)\. However, most studies of this native reasoning gap rely on inference\-time interventions or limited native\-language training data\. We revisit this comparison at a larger scale and under comparable supervision\. We construct long multilingual reasoning datasets across six languages \(English, French, German, Spanish, Chinese and Swahili\); fine\-tune specialists in both native and English\-pivoted regimes on top ofQwen/Qwen3\-8B\-Base, and evaluate across mathematics, science, general knowledge, and code\. In this setting, the average native reasoning gap shrinks to 1\.9–3\.5% across the five non\-English languages, considerably smaller than previously reported\. Weight\-space analysis of the native specialists reveals aligned fine\-tuning updates in the middle layers and divergence in the outer layers\. This points to a largely language\-agnostic reasoning core surrounded by language\-specific layers\. Exploiting this structure, we introduce a Layer Swap: transferring the English specialist’s stronger reasoning mid\-layers into each native specialist, closing most of the native reasoning gap across the five non\-English languages while preserving CoT in the target language\. We release all models and datasets\.111[https://huggingface\.co/collections/lightonai/multilingual\-reasoning](https://huggingface.co/collections/lightonai/multilingual-reasoning)

Rethinking the Multilingual Reasoning Gap with Layer Swap

![Refer to caption](https://arxiv.org/html/2605.26735v1/x1.png)Figure 1:\(*left*\)Layer Swap: transferring a mid\-stack window of the English specialist into the native specialist keeps the CoT in the input language while reducing the remaining native reasoning gap\. \(*right*\) The two baselines compared in this work:Native Reasoning\(CoT in the input language, here French\) andEnglish\-pivoted Reasoning\(CoT in English regardless of input\), the default of open multilingual reasoning models\.## 1Introduction

Reasoning models such as OpenAI o1\(Jaechet al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib34)\), DeepSeek\-R1\(Guoet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib7)\), and Qwen3\(Yanget al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib6)\)rely on long chain\-of\-thought \(CoT\) to tackle complex tasks in mathematics, code, and science\. In current open reasoning models, the CoT is overwhelmingly produced in English, including on non\-English inputs: the model switches back to the input language only for the final response\(Sajiet al\.,[2026](https://arxiv.org/html/2605.26735#bib.bib24); Parket al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib16)\), a regime we refer to as*English\-pivoted reasoning*\(Figure[1](https://arxiv.org/html/2605.26735#S0.F1)\)\. This default has practical costs\. English\-only reasoning limits interpretability for non\-English users, reduces the linguistic and cultural nuance captured by native reasoning traces, and accumulates translation\-style errors that compound with task complexity\(Sajiet al\.,[2026](https://arxiv.org/html/2605.26735#bib.bib24)\)\. Constraining the CoT to remain in the input language,*native reasoning*\(Figure[1](https://arxiv.org/html/2605.26735#S0.F1)\), is therefore desirable, but most attempts substantially degrade accuracy on key benchmarks\(Baruaet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib23); Zhanget al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib18)\)\. We call models trained for native reasoning in a specific non\-English language*native specialists*\. However, these measurements are typically obtained either through prompting alone\(Sajiet al\.,[2026](https://arxiv.org/html/2605.26735#bib.bib24); Kanget al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib28)\)or through fine\-tuning on small amounts of native reasoning data\(Baruaet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib23); Qiet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib21)\), using post\-training budgets far below those required for meaningful native reasoning supervision\. Whether the reported gap persists once native post\-training is brought to a scale comparable to English\-pivoted post\-training remains, to our knowledge, understudied\.

In this work, we revisit the comparison under strictly comparable training conditions\. We construct a large multilingual reasoning dataset spanning six languages \(French, German, Spanish, Swahili, Chinese, and English\), with approximately 500k samples per language up to 32k tokens, and perform supervised fine\-tuning \(SFT\) onQwen/Qwen3\-8B\-Basewith roughly 10B tokens per language in both regimes \(native and English\-pivoted\)\. Across general knowledge, mathematics, code and science benchmarks, the native reasoning gap shrinks to 1\.9–3\.5% on average across the five non\-English languages \(Figure[2](https://arxiv.org/html/2605.26735#S1.F2)\), substantially smaller than previous reports suggest\(Baruaet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib23); Qiet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib21)\); the gap concentrates on complex reasoning benchmarks, with much smaller gaps on the other benchmarks\. Weight\-space analysis of the per\-language SFT updates further shows that cross\-language updates align tightly in the middle layers but diverge at the edges\. We exploit this structure with a*Layer Swap*\(Bandarkaret al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib15)\)\(Figure[1](https://arxiv.org/html/2605.26735#S0.F1)\), a training\-free method that transfers the English specialist’s middle layers into each native specialist\. To our knowledge, this technique has not previously been applied to long\-CoT reasoning models, nor to pairs of experts that share the same reasoning skills but differ in their training language, the setting we study here\. Layer Swap closes8383–8989% of the gap on French and German,6060% on Swahili,2727% on Chinese, and matches the English\-pivoted ceiling on Spanish, all while keeping the CoT in the target language \(Figure[2](https://arxiv.org/html/2605.26735#S1.F2)\)\.

Our contributions are:\(i\)a publicly released long\-CoT reasoning corpus across six languages at 32k context, with CoTs in the target language, covering both European and non\-European languages, including Swahili and Chinese;\(ii\)a large\-scale, strictly controlled measurement of the native\-vs\-English\-pivoted reasoning gap under controlled SFT token budgets;\(iii\)a weight\-space analysis revealing a largely language\-agnostic reasoning core in the middle layers, and a training\-free Layer Swap that exploits this structure to close most of the remaining gap while preserving the target\-language CoT; and\(iv\)an input\-language ablation showing that input understanding remains a primary bottleneck under matched native SFT\.

![Refer to caption](https://arxiv.org/html/2605.26735v1/x2.png)Figure 2:Average accuracy across MGSM\-Rev2, Global\-MMLU\-Lite, GPQA\-Diamond, AIME 24/25, and HumanEvalPlus in the target language, forxx∈\\in\{fr,de,es,zh,sw\}\. Three settings are compared per language:Native Reasoning\(Qwen3\-8B\-xx, CoT inxx\),Layer Swap\(Qwen3\-8B\-xx\-Swap, a mid\-stack window of the English specialist transferred into the native specialist, CoT inxx; see §[5](https://arxiv.org/html/2605.26735#S5)\), andEnglish\-Pivoted Reasoning\(Qwen3\-8B\-xx\-Pivot\-EN, CoT in English\)\.
## 2Related Work

#### Native multilingual reasoning training

Recent work has approached the native reasoning gap through both data and post\-training objectives with various degrees of success\. Publicly available native long CoT corpora remain scarce\(Ghoshet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib1)\): existing releases provide only a few hundred to a few thousand samples per language\(Baruaet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib23); Qiet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib21)\), while broader multilingual corpora either retain English CoT or do not target long\-form reasoning\. Within this limited regime,Baruaet al\.\([2025](https://arxiv.org/html/2605.26735#bib.bib23)\)show that translated long\-CoT supervision can effectively train non\-English reasoners, motivating our translated\-data pipeline at substantially larger scale across six languages\.

Post\-training approaches similarly reveal a substantial native reasoning gap\. Under pure SFT onQwen/Qwen3\-8B\-Base,Baruaet al\.\([2025](https://arxiv.org/html/2605.26735#bib.bib23)\)find that target\-language CoT underperforms English\-pivoted reasoning, with AIME 24/25 gaps averaging∼\\sim19% across nine languages \(∼\\sim17% for French\)\.Sonet al\.\([2025](https://arxiv.org/html/2605.26735#bib.bib12)\)report a similar gap in Korean and mitigate it by inserting English anchor segments into the reasoning trace\. Reinforcement learning\-based \(RL\) methods offer mixed evidence:Huanget al\.\([2025](https://arxiv.org/html/2605.26735#bib.bib17)\)show that RL on non\-English data can improve cross\-lingual transfer, whereasCross\-lingual Collapse\(Parket al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib16)\)identifies a recurrent failure mode in which CoT drifts back to English as accuracy improves under GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib19)\)\. Hybrid approaches combine both stages:Think Natively\(Zhanget al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib18)\)applies SFT followed by GRPO with language\-consistency and cross\-lingual alignment rewards, while concurrent independent work,ReasonXL\(Gurgurovet al\.,[2026](https://arxiv.org/html/2605.26735#bib.bib20)\), combines SFT and RLVR on SmolLM3\-3B across five European languages at 16k context\. Like several prior works,ReasonXLevaluates trained native specialists against the original base model, which can make it challenging to isolate the effect of native reasoning from the effect of specialization itself\. In this work, we instead compare native and English\-pivoted specialists trained under identical conditions, differing only in the language used for CoT reasoning\. Our experimental design enables this comparison at broader scope on an 8B model: six languages, including Chinese and Swahili; an extended 32k training context; and, for each language, matched native and English\-pivoted specialists trained on similar Q&A data\.

#### English as a latent reasoning language

Another line of work argues that multilingual LLMs internally route reasoning through English\-aligned representations\. Using logit\-lens probing on Llama\-2,Wendleret al\.\([2024](https://arxiv.org/html/2605.26735#bib.bib25)\)show that intermediate states traverse an English\-aligned region before resolving to the target language;Schutet al\.\([2025](https://arxiv.org/html/2605.26735#bib.bib26)\)confirm this with activation steering across several languages, finding stronger transfer from English\-derived steering vectors than from native\-language ones\. The same bias appears behaviourally:Etxanizet al\.\([2024](https://arxiv.org/html/2605.26735#bib.bib27)\)show that explicit self\-translation into English can outperform direct non\-English inference, whileSajiet al\.\([2026](https://arxiv.org/html/2605.26735#bib.bib24)\)argues that English\-pivoted reasoning introduces “Lost in Translation” errors that compound with task complexity\. Closest to our setting,Kanget al\.\([2025](https://arxiv.org/html/2605.26735#bib.bib28)\)attributes most of the multilingual reasoning gap to input understanding rather than the reasoning process itself\. We revisit this decomposition through an additional ablation that varies only the input language across our native specialists, which naturally continue reasoning in their training language without any constraint, isolating the contribution of reasoning language from input understanding\.

## 3Data and Benchmarks

### 3\.1Dataset Creation

Training open\-reasoning models in non\-English languages requires large native corpora with long CoT reasoning traces, which remain scarce\. We address this gap by constructing such a corpus through automatic translation from an English source, with samples up to32,76832,768tokens across five target languages\.

#### Source corpus

We start from*allenai/Dolci\-Think\-SFT\-32B*\(Olmoet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib4)\), a decontaminated English post\-training dataset covering mathematics, code, instruction following, science, safety, general chat, and structured data\. We sample∼\\sim500k examples uniformly, preserving the category distribution \(Table[3](https://arxiv.org/html/2605.26735#A1.T3), Appendix\)\.

#### Languages

We target five languages spanning diverse typological and resource settings: French, German, and Spanish \(high\-resource European, Latin script\), Chinese \(high\-resource, non\-alphabetic, typologically distant\), and Swahili \(low\-resource\)\. Together with the original English subset, this yields six per\-language corpora of∼\\sim500k samples each \(Table[A\.1](https://arxiv.org/html/2605.26735#A1.SS1), Appendix\)\.

#### Translation

Motivated by prior evidence that translated long\-CoT data outperforms direct native distillation\(Baruaet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib23)\), we translate the English corpus into the five target languages withgoogle/gemma\-3\-27b\-it, chosen for its strong multilingual coverage\. Single\-pass translation exhibited two failure modes: on the longest samples \(input plus output up to 64K tokens\), the model entered a degraded long\-context regime with frequent looping; and, independently of length, on a small but consistent fraction of samples it silently dropped the reasoning trace while still translating the question and final answer, a failure that persisted under alternative`<think\>`delimiters\. We therefore translate each sample component\-wise, splitting question, reasoning trace, and final answer into∼\\sim2k\-token chunks at sentence or paragraph boundaries, translating each chunk independently, and recomposing, at some cost in global coherence\. We manually inspected a subset of translations to verify quality and reasoning\-trace preservation\.

#### Filtering

We apply two filtering passes\. Before translation, we remove English samples that explicitly reference translating into or answering in a specific language, since they become self\-contradictory after translation\. After translation, we discard \(i\) empty outputs, \(ii\) samples whose zlib\(Gailly and Adler,[2012](https://arxiv.org/html/2605.26735#bib.bib35)\)compression ratio against the source deviates anomalously from the dataset mean, which catches degenerate or repeated outputs, \(iii\) samples whose translated\-to\-source length ratio similarly deviates, flagging truncations or over\-generation, and \(iv\) samples whose total length exceeds the 32K\-token training context, which disproportionately affects languages with less efficient tokenization \(e\.g\. Swahili\) or higher verbosity \(e\.g\. French, Spanish\)\. Chunk\-level translation limits per\-call context and keeps translation error rates low\.

### 3\.2Benchmarks

We evaluate across four different domains:

- •Mathematics:*MGSM\-Rev2*\(Peteret al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib2)\), a revised multilingual version of the grade\-school mathematics benchmark*GSM8K*\(Shiet al\.,[2022](https://arxiv.org/html/2605.26735#bib.bib5)\)that corrects translation errors; and multilingual versions of*AIME24*and*AIME25*\(Qiet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib21)\)for competition\-level hard reasoning problems\.
- •Science:A multilingual version\(Qiet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib21)\)of*GPQA\-Diamond*\(Reinet al\.,[2023](https://arxiv.org/html/2605.26735#bib.bib8)\), a PhD\-level science QA benchmark\.
- •Knowledge:*Global\-MMLU\-Lite*\(Singhet al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib3)\), a curated multilingual MMLU benchmark that addresses cultural and translation biases in the original version\.
- •Code:We translate*HumanEvalPlus*\(Liuet al\.,[2023](https://arxiv.org/html/2605.26735#bib.bib11)\)withgoogle/gemma\-3\-27b\-itinto our five target languages\.

#### Quality control

The translated reasoning benchmarks \(AIME24, AIME25, and GPQA\-Diamond\)222[https://huggingface\.co/collections/lightonai/multilingual\-reasoning](https://huggingface.co/collections/lightonai/multilingual-reasoning)initially contained a small number of translation artifacts that biased evaluation against non\-English languages\. To mitigate this, we performed an LLM\-as\-a\-judge review using Claude Opus 4\.7, which identified and rewrote a few malformed samples in each target language\. Evaluation was conducted using a forked version oflm\-eval\-harness\(Gaoet al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib22)\); the per\-language prompts are listed in Appendix[A\.3](https://arxiv.org/html/2605.26735#A1.SS3)\.

#### Evaluation protocol

We evaluate models using a temperature of11, as lower values caused frequent decoding loops in our models, top\-p0\.950\.95, top\-k2020, and min\-p0across all benchmarks\. We report mean accuracy over multiple random seeds, using1010sampled runs by default, except for AIME24 and AIME25, which we average over3030runs to reduce sampling variance on these smaller test sets\.

## 4The Native Reasoning Gap Under Matched Supervision

### 4\.1Experimental Setup

We run distributed training on multiple H100 nodes with TRL\(von Werraet al\.,[2020](https://arxiv.org/html/2605.26735#bib.bib13)\)\. At a 32K sequence length, full fine\-tuning of an 8B model exceeds single\-H100 memory\. To mitigate the issue, we combine DeepSpeed ZeRO\-3\(Rajbhandariet al\.,[2020](https://arxiv.org/html/2605.26735#bib.bib31)\)parameter sharding with Ulysses sequence parallelism\(Jacobset al\.,[2023](https://arxiv.org/html/2605.26735#bib.bib10)\)that splits attention heads across GPUs, together with FlashAttention\-3\(Shahet al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib9)\)and sequence packing for efficiency\. All specialists are fully fine\-tuned fromQwen/Qwen3\-8B\-Base\(Table[11](https://arxiv.org/html/2605.26735#A1.T11)\)\.

### 4\.2Scaling Across Languages

![Refer to caption](https://arxiv.org/html/2605.26735v1/media/scaling_law_avg_trend.png)Figure 3:Scaling curves of native reasoning performance across six languages, fine\-tuned fromQwen/Qwen3\-8B\-Base: average accuracy on MGSM\-Rev2, Global\-MMLU\-Lite, GPQA\-Diamond, AIME 24/25, and HumanEvalPlus in the training language, as a function of the SFT\-token budget\.Before measuring the native reasoning gap under matched supervision, we verify that our translated corpus produces useful training signal in every language\. We fine\-tune each of the six per\-language corpora at training\-data budgets ranging from∼\\sim100M to∼\\sim10B tokens, holding all other hyperparameters fixed, and evaluate each resulting specialist on our five benchmarks in its training language\. Figure[3](https://arxiv.org/html/2605.26735#S4.F3)shows that average accuracy increases monotonically with the budget in every language\. This confirms that the translated corpus carries usable training signal in every language\. We adopt the∼\\sim10B\-token budget for the matched\-supervision experiment that follows \(Table[12](https://arxiv.org/html/2605.26735#A1.T12), Appendix\)\.

The curves separate into three resource tiers \(Figure[3](https://arxiv.org/html/2605.26735#S4.F3)\)\. English forms the upper envelope at every budget\. The high\-resource cluster of French, German, Spanish, and Chinese tracks it closely, with a gap that stays in a narrow4\.54\.5–6%6\\%band across budgets\. Swahili sits below this cluster but shows the largest absolute gains, closing its gap to the high\-resource tier from roughly18%18\\%at the∼\\sim100M\-token budget to7%7\\%at∼\\sim10B\.

Table 1:Detailed per\-language evaluation, forxx∈\\in\{fr,de,es,zh,sw\}:Qwen3\-8B\-XX\(native reasoning inxx\),Qwen3\-8B\-XX\-Pivot\-EN\(English\-pivoted reasoning, samexxQ&A pairs asQwen3\-8B\-XX\),Qwen3\-8B\-XX\-Swap\(Layer Swap fromQwen3\-8B\-ENintoQwen3\-8B\-XX; see §[5](https://arxiv.org/html/2605.26735#S5)\), andQwen3\-8B\-EN\(English specialist, reference\)\. Bold marks the best score per column within each language group\.±\\pmdenotes the sample standard deviation across runs\.![Refer to caption](https://arxiv.org/html/2605.26735v1/x3.png)Figure 4:\(a\)Cross\-language alignment of the per\-language SFT updatesΔ​θL\(i\)\\Delta\\theta\_\{L\}^\{\(i\)\}as a function of transformer layer indexLL: mean pairwise cosinec¯L\\bar\{c\}\_\{L\}and top\-1 SVD variance sharesLs\_\{L\}across the six per\-language specialists\.\(b\)Mean accuracy on the five French benchmarks for the seven Swap variants, each transferring a contiguous 10\-layer window fromQwen3\-8B\-ENintoQwen3\-8B\-FR\.\(c\)Language fidelity, the fraction of generated reasoning traces classified as French, for each Swap variant\.
### 4\.3Matched\-Supervision Experiments

To compare native and English\-pivoted reasoning under matched supervision, we train, for each non\-English languagexx∈\{fr,de,es,zh,sw\}\\in\\\{\\textsc\{fr\},\\textsc\{de\},\\textsc\{es\},\\textsc\{zh\},\\textsc\{sw\}\\\}, two specialists fromQwen/Qwen3\-8B\-Baseon the same per\-language question–answer pairs, differing only in the CoT language: a native specialistQwen3\-8B\-XXwhose reasoning trace is also inxx, and an English\-pivoted specialistQwen3\-8B\-XX\-Pivot\-ENwhose questions and answers remain inxxbut whose reasoning trace is generated in English\. Both are trained with comparable per\-language token budgets of1010–1111B and identical hyperparameters \(Table[12](https://arxiv.org/html/2605.26735#A1.T12), Appendix\)\. Note that sample counts differ slightly because samples exceeding the 32K\-token training context are dropped, which disproportionately affects non\-English versions due to less efficient tokenization \(e\.g\. Swahili\) or higher verbosity \(e\.g\. French, Spanish\)\. We also include the English specialistQwen3\-8B\-ENfor reference, but the controlled comparison remainsQwen3\-8B\-XXvs\.Qwen3\-8B\-XX\-Pivot\-EN: comparingQwen3\-8B\-XXdirectly toQwen3\-8B\-ENwould conflate the CoT language with the fact thatQwen3\-8B\-ENhas seen noxx\-language supervision\.

Under this setting, the native reasoning gap is substantially smaller than prior reports suggest\. On the five\-benchmark average,Qwen3\-8B\-XXtrailsQwen3\-8B\-XX\-Pivot\-ENby2\.7%2\.7\\%,3\.1%3\.1\\%,1\.9%1\.9\\%,3\.1%3\.1\\%, and3\.5%3\.5\\%on French, German, Spanish, Chinese, and Swahili respectively \(Table[1](https://arxiv.org/html/2605.26735#S4.T1)\)\. The gap broadly tracks the resource tier: smallest on the high\-resource languages \(1\.91\.9–3\.1%3\.1\\%\) and largest on low\-resource Swahili \(3\.5%3\.5\\%\)\. Per benchmark, AIME 24/25 dominates the gap, with native specialists trailing their English\-pivoted counterparts by55–7%7\\%on French, German, Spanish, and Chinese and by12%12\\%on Swahili; the other benchmarks contribute much smaller differences \(around11–4%4\\%on average\), and HumanEvalPlus is essentially tied across languages\. The lower\-resource end is also where native SFT contributes the most in absolute terms:Qwen3\-8B\-ENscores only37\.96%37\.96\\%on the Swahili five\-benchmark average, and the matched native specialist nearly doubles this to66\.98%66\.98\\%, whereas the corresponding gain overQwen3\-8B\-ENis marginal for the high\-resource Latin\-script trio\.

## 5Layer Swap

### 5\.1Method and Layer Selection

Layer Swap\(Bandarkaret al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib15); Bandarkar and Peng,[2025](https://arxiv.org/html/2605.26735#bib.bib14)\)is a training\-free model\-composition technique\(Ilharcoet al\.,[2023](https://arxiv.org/html/2605.26735#bib.bib33)\)\. Starting from a shared base model, two*experts*are independently fine\-tuned and recomposed into a single hybrid model by replacing a contiguous range of transformer layers in one expert with the corresponding layers from the other, while the remaining layers are kept unchanged\.Bandarkaret al\.\([2024](https://arxiv.org/html/2605.26735#bib.bib15)\)apply this to short\-form math instruction tuning, combining a math\-knowledge expert trained in English with a target\-language fluency expert trained on generic instructions\. To our knowledge, the technique has not been applied to long\-CoT reasoning models, nor to a pair of experts that share the same reasoning skills but differ in their training language, the setting we study here\. The two experts are the native specialist \(e\.g\.Qwen3\-8B\-FR\) and the English specialist \(Qwen3\-8B\-EN\); we recompose them by transferring a contiguous range of the English specialist’s transformer layers into the native specialist\.

#### Weight\-space analysis

We measure how theNNper\-language specialists’ weight updatesΔ​θL\(xx\)=θL\(xx\)−θbase\\Delta\\theta\_\{L\}^\{\(\\textsc\{xx\}\)\}=\\theta\_\{L\}^\{\(\\textsc\{xx\}\)\}\-\\theta\_\{\\mathrm\{base\}\}\(for languagexx∈\{en,fr,de,es,zh,sw\}\\textsc\{xx\}\\in\\\{\\textsc\{en\},\\textsc\{fr\},\\textsc\{de\},\\textsc\{es\},\\textsc\{zh\},\\textsc\{sw\}\\\}at layerLL\) agree across languages, layer by layer, using two complementary statistics; in our setupN=6N=6\(English, French, German, Spanish, Chinese, and Swahili\)\. The first is the mean pairwise cosine of theNNdeltas,

c¯L=\(N2\)−1​∑xx<yy⟨Δ​θL\(xx\),Δ​θL\(yy\)⟩‖Δ​θL\(xx\)‖​‖Δ​θL\(yy\)‖,\\bar\{c\}\_\{L\}=\\binom\{N\}\{2\}^\{\-1\}\\\!\\\!\\sum\_\{\\textsc\{xx\}<\\textsc\{yy\}\}\\frac\{\\langle\\Delta\\theta\_\{L\}^\{\(\\textsc\{xx\}\)\},\\,\\Delta\\theta\_\{L\}^\{\(\\textsc\{yy\}\)\}\\rangle\}\{\\\|\\Delta\\theta\_\{L\}^\{\(\\textsc\{xx\}\)\}\\\|\\,\\\|\\Delta\\theta\_\{L\}^\{\(\\textsc\{yy\}\)\}\\\|\},\(1\)which captures local alignment between pairs of language\-specific deltas\. The second is the top\-1 SVD variance share: stacking theNNdeltas as rows of a matrixDL∈ℝN×PLD\_\{L\}\\in\\mathbb\{R\}^\{N\\times P\_\{L\}\}\(wherePLP\_\{L\}is the number of parameters in layerLL\) whose singular values areσ1≥⋯≥σN\\sigma\_\{1\}\\geq\\dots\\geq\\sigma\_\{N\},

sL=σ12∑k=1Nσk2,s\_\{L\}=\\frac\{\\sigma\_\{1\}^\{2\}\}\{\\sum\_\{k=1\}^\{N\}\\sigma\_\{k\}^\{2\}\},\(2\)capturing the fraction of cross\-language variance concentrated along a single direction in weight space\. Both statistics are low at the two ends of the stack, rise sharply between L9 and L13, and reach a high range across L13–L22, wherec¯L≈0\.6\\bar\{c\}\_\{L\}\\approx 0\.6andsLs\_\{L\}captures6565–74%74\\%of the cross\-language variance \(Figure[4](https://arxiv.org/html/2605.26735#S4.F4); \(a\)\), with per\-layer delta norms remaining comparable across the stack \(Appendix[A](https://arxiv.org/html/2605.26735#A1), Figure[6](https://arxiv.org/html/2605.26735#A1.F6)\)\. In this window, the six per\-language SFT updates align along a shared cross\-language direction \(a largely language\-agnostic component of the SFT update\) while at the early and late ends of the stack they diverge along language\-specific directions\. This depth structure is consistent with prior weight\-space analyses of multilingual layer specialization\(Bandarkaret al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib15); Tanget al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib30)\); we verify it here for long\-CoT native reasoning specialists across six languages\.

#### Motivation

This mid\-stack cross\-lingual alignment has a natural interpretation\. Current multilingual LLMs appear to route reasoning through English\-aligned intermediate representations\(Wendleret al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib25); Schutet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib26); Etxanizet al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib27); Zhaoet al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib32)\)\. In an English specialist, the middle layers may therefore develop reasoning circuits directly on top of representations already aligned with the pretraining distribution, which is not the case for a native specialist\. This asymmetry may limit how effectively the native specialist’s central layers specialize for reasoning during post\-training, yielding a stronger reasoning core in the English specialist’s middle layers\. Transferring only the English middle layers into the native specialist should therefore install part of the English reasoning advantage while leaving the native edge layers, and with them the target\-language CoT, intact\. We test whether this swap improves reasoning while preserving target\-language CoT generation\.

#### Layer\-range ablation

We conduct the layer\-range ablation on the French specialist as a case study\. Transferring layers from the English specialist \(Qwen3\-8B\-EN\) into the French specialist \(Qwen3\-8B\-FR\), we sweep the transferred layer range from L0 to L35 in contiguous windows and report two quantities per configuration: average accuracy on the French benchmarks \(Figure[4](https://arxiv.org/html/2605.26735#S4.F4); \(b\)\), and*language fidelity*\(Figure[4](https://arxiv.org/html/2605.26735#S4.F4); \(c\)\), the fraction of generated reasoning traces classified as French by a FastText language identifier\(Graveet al\.,[2018](https://arxiv.org/html/2605.26735#bib.bib29)\)\. Fidelity stays at∼\\sim100% for any window confined to L0–L22 and collapses for windows extending further, placing the CoT\-language gate around L22\. Accuracy improves materially only when the English specialist’s mid\-stack is included; early\-only swaps preserve fidelity but transfer little of its reasoning capability\. The L13–L22 window satisfies both criteria, preserving native CoT generation while capturing the English specialist’s reasoning advantage\. Replacing the English source specialist with either the Chinese or the German specialist eliminates the improvement \(Table[15](https://arxiv.org/html/2605.26735#A1.T15), Appendix[A\.6](https://arxiv.org/html/2605.26735#A1.SS6)\), confirming that the effect originates specifically from the English specialist’s middle\-layer representations\.

### 5\.2Results

In Table[1](https://arxiv.org/html/2605.26735#S4.T1), we compareQwen3\-8B\-XX\-Swap, obtained by transplanting a mid\-stack window from the English specialist into the native specialist, against the three models introduced in §[4\.3](https://arxiv.org/html/2605.26735#S4.SS3)\. For French, German, Spanish, and Swahili, we use the L13–L22 window identified above\. For Chinese, however, this window causes roughly60%60\\%of traces to revert to English mid\-generation, suggesting that the target\-language gate is located slightly earlier in the stack\. We therefore use L13–L20 instead, which fully restores Chinese CoT fidelity \(Appendix[A\.7](https://arxiv.org/html/2605.26735#A1.SS7)\)\.

In particular, the swapped models retain the target reasoning language while closing most of the native reasoning gap relative to English pivoting\. Measured as the fraction of the native\-vs\-pivoted gap closed on the five\-benchmark average, Swap closes89%89\\%of the gap on French,83%83\\%on German,27%27\\%on Chinese,60%60\\%on Swahili, and matches the English\-pivoted ceiling on Spanish\. In absolute terms, the remaining gap toQwen3\-8B\-XX\-Pivot\-ENshrinks to between0\.0%0\.0\\%and2\.3%2\.3\\%across languages\. The gap is essentially closed for the Latin\-script European trio, while the remaining differences on Chinese and Swahili reflect, respectively, typological distance from English and lower\-resource status\.

Across benchmarks, Layer Swap leads on HumanEvalPlus in all five languages and on MGSM\-Rev2 in most of them, while the remaining gap concentrates on the more reasoning\-intensive tasks \(Global\-MMLU\-Lite, GPQA\-Diamond, and AIME\)\. On AIME 24/25 specifically, Layer Swap narrows the gap to just∼\\sim3% on French, German, Spanish, and Chinese, with Swahili remaining the outlier at∼\\sim9%\. Language fidelity remains at∼\\sim100% across all five Swap variants\.

## 6The Understanding Gap

As an additional ablation enabled by our per\-language specialists, we isolate the input\-understanding component of the multilingual reasoning gap, a questionKanget al\.\([2025](https://arxiv.org/html/2605.26735#bib.bib28)\)address with inference\-time interventions onQwen/Qwen3\-4Bbut that remains open under large\-scale native SFT\. We evaluate each specialist on the native and English versions of every benchmark; each model continues to reason in its training language in both conditions without any prompting or constraint, so only the input language varies\.

Every non\-English specialist scores higher on English inputs than on native inputs despite never seeing English in SFT \(Table[2](https://arxiv.org/html/2605.26735#S6.T2); per\-benchmark numbers in Table[14](https://arxiv.org/html/2605.26735#A1.T14), Appendix\)\. The improvement grows sharply with the language’s distance from English in the base model, a∼\\sim4×\\timeseffect on Swahili relative to the European trio\. Two effects drive this\.\(i\)Pretraining data imbalance: the Qwen3 base most likely saw considerably more English and Latin\-script European tokens than Swahili during pretraining \(the exact mixture is not disclosed\), and our 10B\-token SFT only retargets the reasoning policy on top of the input\-side representations inherited from that pretraining\.\(ii\)Intermediate representations in English\-centric base models are more strongly aligned with the English representation space for high\-resource Latin\-script languages than for typologically distant or low\-resource ones\(Wendleret al\.,[2024](https://arxiv.org/html/2605.26735#bib.bib25); Schutet al\.,[2025](https://arxiv.org/html/2605.26735#bib.bib26)\), so the representational gap between input\-side embeddings and the English\-aligned reasoning mid\-stack is largest precisely for Swahili\.

Table 2:Average accuracy of each per\-language specialist, averaged over MGSM\-Rev2, Global\-MMLU\-Lite, GPQA\-Diamond, AIME 24/25, and HumanEvalPlus, on the English and native versions of each benchmark\.
## 7Conclusion

We revisited the native reasoning gap in multilingual LLMs under matched supervision\. We trained native and English\-pivoted specialists fromQwen/Qwen3\-8B\-Baseon the same per\-language Q&A pairs using∼\\sim10B\-token budgets and 32k context windows across French, German, Spanish, Chinese, and Swahili\. Across five evaluation benchmarks, the performance gap shrank to1\.91\.9–3\.5%3\.5\\%on average, substantially smaller than prior reports suggested, with the remaining differences concentrated in complex reasoning\-intensive mathematics tasks\. A weight\-space analysis revealed a largely language\-agnostic reasoning core in the mid\-stack, motivating a training\-free Layer Swap that transferred these English layers into each native specialist while preserving∼\\sim100% language fidelity\. On the five\-benchmark average, the gap was fully closed on Spanish, reduced by8383–8989% on French and German, and by2727–6060% on Chinese and Swahili\. Since English\-pivoted reasoning is the default in essentially all open reasoning models, the required English specialist is available off the shelf, making the recipe directly applicable to other model families\.

## 8Limitations

While this work establishes a controlled study of the native reasoning gap, several limitations remain\. First, our analysis was limited to a single model family and parameter scale\. Extending the study to a wider range of architectures, scales, and languages would strengthen the findings\. The recipe also presupposed a multilingually pretrained base and covered only six languages, with Swahili as the sole lower\-resource representative\. In addition, tokenization efficiency varied across languages, meaning that an equivalent token budget may have corresponded to less effective supervision for languages such as Swahili\. Our primary control was a matched token\-budget comparison rather than a matched\-example comparison\. A stricter design would train both regimes on the intersection of examples retained in both native and English\-pivoted form; we leave this paired control to future work\.

Second, we did not perform an extensive hyperparameter sweep for our∼\\sim10B\-token SFT runs, nor did we exhaustively explore the Layer Swap window, and finer\-grained sweeps over its position and width could identify a better configuration\. While our training budget was substantial for native long\-CoT supervision, it remained below production\-scale post\-training settings, where larger token budgets and multi\-stage curricula may have led to different outcomes\.

Finally, as is common in multilingual long\-CoT post\-training, our corpus has been machine\-translated, and the reported native\-vs\-English\-pivoted gap should therefore be interpreted within this translated supervision setting\. Our evaluation focused on mathematics, science, general knowledge, and code generation; extending it to tasks with stronger cultural or sociopragmatic components would provide a broader assessment of multilingual reasoning\. Moreover, our specialist models were trained solely with SFT, and combining the Swap models with RL or preference tuning remains a natural direction for future work\.

## 9Ethical Considerations

In terms of broader impact, our specialists maintain CoT in the input language across six languages, including the lower\-resource Swahili, making reasoning models more interpretable for non\-English users and reducing reliance on English as a mandatory intermediate language\. On the risk side, the released specialists inherit biases from bothQwen/Qwen3\-8B\-Baseand the translation model used to construct our corpus\. They have not undergone additional safety tuning, and may therefore exhibit unsafe or undesirable behaviors outside the scope of our evaluation setting\.

## 10Acknowledgments

We thank Adrien Cavaillès, Théo Lasnier, and Armel Randy Zebaze for helpful comments and discussions\. We also gratefully acknowledge the EuroHPC Joint Undertaking for awarding us Fast Lane allocations on MareNostrum 5, hosted by the Barcelona Supercomputing Center, under applications EHPC\-AIF\-2025FL01\-571 and EHPC\-AIF\-2026FL01\-212, which provided the computational resources used in this work\. This project is also supported by the OpenEuroLLM project, co\-funded by the Digital Europe Programme under GA no\. 101195233\. For more information see[https://openeurollm\.eu](https://openeurollm.eu/)\.

## References

- Layer swapping for zero\-shot cross\-lingual transfer in large language models\.arXiv preprint arXiv:2410\.01335\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p2.4),[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.SSS0.Px1.p1.15),[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.p1.1)\.
- L\. Bandarkar and N\. Peng \(2025\)The unreasonable effectiveness of model merging for cross\-lingual transfer in llms\.InProceedings of the 5th Workshop on Multilingual Representation Learning \(MRL 2025\),pp\. 131–148\.Cited by:[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.p1.1)\.
- J\. Barua, S\. Eisape, K\. Yin, and A\. Suhr \(2025\)Long chain\-of\-thought reasoning across languages\.arXiv preprint arXiv:2508\.14828\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1),[§1](https://arxiv.org/html/2605.26735#S1.p2.4),[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p2.2),[§3\.1](https://arxiv.org/html/2605.26735#S3.SS1.SSS0.Px3.p1.1)\.
- J\. Etxaniz, G\. Azkune, A\. Soroa, O\. L\. de Lacalle, and M\. Artetxe \(2024\)Do multilingual language models think better in english?\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 550–564\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.SSS0.Px2.p1.1)\.
- J\. Gailly and M\. Adler \(2012\)Zlib compression library\.External Links:[Link](https://api.semanticscholar.org/CorpusID:60948258)Cited by:[§3\.1](https://arxiv.org/html/2605.26735#S3.SS1.SSS0.Px4.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§3\.2](https://arxiv.org/html/2605.26735#S3.SS2.SSS0.Px1.p1.1)\.
- A\. Ghosh, D\. Datta, S\. Saha, and C\. Agarwal \(2025\)A survey of multilingual reasoning in language models\.arXiv preprint arXiv:2502\.09457\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Grave, P\. Bojanowski, P\. Gupta, A\. Joulin, and T\. Mikolov \(2018\)Learning word vectors for 157 languages\.InProceedings of the International Conference on Language Resources and Evaluation \(LREC 2018\),Cited by:[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.SSS0.Px3.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1)\.
- D\. Gurgurov, T\. Röhr, S\. von Rohrscheidt, J\. van Genabith, A\. Löser, and S\. Ostermann \(2026\)ReasonXL: shifting llm reasoning language without sacrificing performance\.arXiv preprint arXiv:2604\.12378\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p2.2)\.
- S\. Huang, Y\. Ding, J\. Pan, and Y\. Zhang \(2025\)Beyond english\-centric training: how reinforcement learning improves cross\-lingual reasoning in llms\.arXiv preprint arXiv:2509\.23657\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p2.2)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2023\)Editing models with task arithmetic\.InThe Eleventh International Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.p1.1)\.
- S\. A\. Jacobs, M\. Tanaka, C\. Zhang, M\. Zhang, S\. L\. Song, S\. Rajbhandari, and Y\. He \(2023\)Deepspeed ulysses: system optimizations for enabling training of extreme long sequence transformer models\.arXiv preprint arXiv:2309\.14509\.Cited by:[§4\.1](https://arxiv.org/html/2605.26735#S4.SS1.p1.1)\.
- A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.\(2024\)Openai o1 system card\.arXiv preprint arXiv:2412\.16720\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1)\.
- D\. Kang, S\. Hwang, D\. Kim, H\. Kim, and G\. G\. Lee \(2025\)Why do multilingual reasoning gaps emerge in reasoning language models?\.arXiv preprint arXiv:2510\.27269\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1),[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2605.26735#S6.p1.1)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.Advances in neural information processing systems36,pp\. 21558–21572\.Cited by:[4th item](https://arxiv.org/html/2605.26735#S3.I1.i4.p1.1)\.
- T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2025\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[§3\.1](https://arxiv.org/html/2605.26735#S3.SS1.SSS0.Px1.p1.1)\.
- C\. Park, J\. Kim, J\. Lee, S\. Bae, J\. Choo, and K\. M\. Yoo \(2025\)Cross\-lingual collapse: how language\-centric foundation models shape reasoning in large language models\.arXiv preprint arXiv:2506\.05850\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1),[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p2.2)\.
- J\. Peter, D\. Vilar, T\. Domhan, D\. Malkin, and M\. Freitag \(2025\)Mind the gap… or not? how translation errors and evaluation details skew multilingual results\.arXiv preprint arXiv:2511\.05162\.Cited by:[1st item](https://arxiv.org/html/2605.26735#S3.I1.i1.p1.1)\.
- J\. Qi, S\. Chen, Z\. Xiong, R\. Fernández, D\. S\. Bitterman, and A\. Bisazza \(2025\)When models reason in your language: controlling thinking trace language comes at the cost of accuracy\.External Links:2505\.22888,[Link](https://arxiv.org/abs/2505.22888)Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1),[§1](https://arxiv.org/html/2605.26735#S1.p2.4),[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2605.26735#S3.I1.i1.p1.1),[2nd item](https://arxiv.org/html/2605.26735#S3.I1.i2.p1.1)\.
- S\. Rajbhandari, J\. Rasley, O\. Ruwase, and Y\. He \(2020\)Zero: memory optimizations toward training trillion parameter models\.InSC20: international conference for high performance computing, networking, storage and analysis,pp\. 1–16\.Cited by:[§4\.1](https://arxiv.org/html/2605.26735#S4.SS1.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)Gpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[2nd item](https://arxiv.org/html/2605.26735#S3.I1.i2.p1.1)\.
- A\. Saji, R\. Dabre, A\. Kunchukuttan, and R\. Puduppully \(2026\)The reasoning lingua franca: a double\-edged sword for multilingual ai\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 329–344\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1),[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Schut, Y\. Gal, and S\. Farquhar \(2025\)Do multilingual llms think in english?\.arXiv preprint arXiv:2502\.15603\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2605.26735#S6.p2.2)\.
- J\. Shah, G\. Bikshandi, Y\. Zhang, V\. Thakkar, P\. Ramani, and T\. Dao \(2024\)Flashattention\-3: fast and accurate attention with asynchrony and low\-precision\.Advances in Neural Information Processing Systems37,pp\. 68658–68685\.Cited by:[§4\.1](https://arxiv.org/html/2605.26735#S4.SS1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p2.2)\.
- F\. Shi, M\. Suzgun, M\. Freitag, X\. Wang, S\. Srivats, S\. Vosoughi, H\. W\. Chung, Y\. Tay, S\. Ruder, D\. Zhou,et al\.\(2022\)Language models are multilingual chain\-of\-thought reasoners\.arXiv preprint arXiv:2210\.03057\.Cited by:[1st item](https://arxiv.org/html/2605.26735#S3.I1.i1.p1.1)\.
- S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, W\. Ko, M\. Smith, A\. Bosselut, A\. Oh, A\. F\. T\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. Hooker \(2024\)Global mmlu: understanding and addressing cultural and linguistic biases in multilingual evaluation\.External Links:2412\.03304,[Link](https://arxiv.org/abs/2412.03304)Cited by:[3rd item](https://arxiv.org/html/2605.26735#S3.I1.i3.p1.1)\.
- G\. Son, D\. Yang, H\. L\. Patel, A\. Agarwal, H\. Ko, C\. Lim, S\. Panda, M\. Kim, N\. Drolia, D\. Choi,et al\.\(2025\)Pushing on multilingual reasoning models with language\-mixed chain\-of\-thought\.arXiv preprint arXiv:2510\.04230\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p2.2)\.
- T\. Tang, W\. Luo, H\. Huang, D\. Zhang, X\. Wang, X\. Zhao, F\. Wei, and J\. Wen \(2024\)Language\-specific neurons: the key to multilingual capabilities in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5701–5715\.Cited by:[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.SSS0.Px1.p1.15)\.
- L\. von Werra, Y\. Belkada, L\. Tunstall, E\. Beeching, T\. Thrush, N\. Lambert, S\. Huang, K\. Rasul, and Q\. Gallouédec \(2020\)TRL: Transformers Reinforcement Learning\.Note:[https://github\.com/huggingface/trl](https://github.com/huggingface/trl)Cited by:[§4\.1](https://arxiv.org/html/2605.26735#S4.SS1.p1.1)\.
- C\. Wendler, V\. Veselovsky, G\. Monea, and R\. West \(2024\)Do llamas work in english? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15366–15394\.Cited by:[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2605.26735#S6.p2.2)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1)\.
- X\. Zhang, Y\. Liang, F\. Meng, S\. Zhang, K\. Huang, Y\. Chen, J\. Xu, and J\. Zhou \(2025\)Think natively: unlocking multilingual reasoning with consistency\-enhanced reinforcement learning\.arXiv preprint arXiv:2510\.07300\.Cited by:[§1](https://arxiv.org/html/2605.26735#S1.p1.1),[§2](https://arxiv.org/html/2605.26735#S2.SS0.SSS0.Px1.p2.2)\.
- Y\. Zhao, W\. Zhang, G\. Chen, K\. Kawaguchi, and L\. Bing \(2024\)How do large language models handle multilingualism?\.InAdvances in Neural Information Processing Systems,Cited by:[§5\.1](https://arxiv.org/html/2605.26735#S5.SS1.SSS0.Px2.p1.1)\.

## Appendix AAppendix

### A\.1Dataset Statistics

Table[3](https://arxiv.org/html/2605.26735#A1.T3)reports the category mix of the English source corpus, dominated by mathematics and code with long mean trace lengths\. Table[A\.1](https://arxiv.org/html/2605.26735#A1.SS1)gives the per\-language sample counts and total trained tokens after filtering using the Qwen3 tokenizer\.

Table 3:Category distribution and mean token length of the English subset of*Dolci\-Think\-SFT\-32B*\.Table 4:Per\-language statistics of the native reasoning SFT dataset: number of samples, mean sample length, and total token count\.### A\.2Translation

Table[5](https://arxiv.org/html/2605.26735#A1.T5)gives the chunk\-level translation prompt applied independently to question, reasoning trace, and final answer at∼\\sim2k\-token boundaries\. We perform translation withgoogle/gemma\-3\-27b\-itat a temperature of0\.150\.15\.

\\cellcolorgray\!10You are a translation engine\. Your ONLY task is to translate the text inside <source\> into <LANGUAGE\>\.HARD RULES \(must follow\): \- Output ONLY the translation, and nothing else\. \- Do NOT answer the message or continue the conversation\. \- Do NOT add advice, role labels, extra lines, or formatting\. \- Translate EVERYTHING inside <source\> completely\. \- Use informal address unless formality or plurality is clearly required\. \- Preserve meaning, intent, tone, and punctuation as naturally as possible\.SPECIAL CASES: \- For code, do NOT translate identifiers\. Translate only comments and human\-language strings, preserving whitespace and indentation\. \- For units, convert to metric while keeping calculations consistent\.INPUT: <source\> \{message\_text\} </source\>OUTPUT: <LANGUAGE\> translation only\.

Table 5:Prompt template used for our translation pipeline\.### A\.3Evaluation Prompts

We list the language\-specific instruction templates passed tolm\-eval\-harnessfor each benchmark family\. Prompts were translated and adapted from the English originals while preserving the answer\-extraction format used by the harness\. Table[6](https://arxiv.org/html/2605.26735#A1.T6)covers AIME 24/25, Table[7](https://arxiv.org/html/2605.26735#A1.T7)MGSM\-Rev2, Table[8](https://arxiv.org/html/2605.26735#A1.T8)Global\-MMLU\-Lite, Table[9](https://arxiv.org/html/2605.26735#A1.T9)GPQA\-Diamond, and Table[10](https://arxiv.org/html/2605.26735#A1.T10)HumanEvalPlus\.

Table 6:Prompt template used to evaluate the multilingual AIME24 and AIME25 benchmarks inlm\-eval\-harness\.Table 7:Prompt template used to evaluate the MGSM\-Rev2 benchmark inlm\-eval\-harness\.Table 8:Prompt prefix used to evaluate the Global\-MMLU benchmark inlm\-eval\-harness; the question and four options A/B/C/D follow on subsequent lines\.Table 9:Prompt template used to evaluate the GPQA\-Diamond benchmark inlm\-eval\-harness\. The dataset’sproblemfield already contains the question and four options A/B/C/D\.Table 10:Prompt template used to evaluate the HumanEvalPlus benchmark inlm\-eval\-harness\.### A\.4Training Settings

Table[11](https://arxiv.org/html/2605.26735#A1.T11)lists the SFT hyperparameters used to train every per\-language specialist, held identical across the native and English\-pivoted regimes\.

Table 11:SFT hyperparameters, identical across all per\-language specialists and across the native and English\-pivoted regimes\.### A\.5Detailed Training Results

Table[12](https://arxiv.org/html/2605.26735#A1.T12)reports the unique sample count and total training\-token count for each specialist\. Table[13](https://arxiv.org/html/2605.26735#A1.T13)expands the scaling curves of Figure[3](https://arxiv.org/html/2605.26735#S4.F3)with per\-budget mean accuracies in every language\. Table[14](https://arxiv.org/html/2605.26735#A1.T14)extends the cross\-lingual evaluation of Table[2](https://arxiv.org/html/2605.26735#S6.T2)with per\-benchmark scores on both the English and native versions of every benchmark\.

Table 12:Training\-data composition per specialist: number of unique samples and total trained tokens, after filtering and the 32K\-token context cap\.Table 13:Mean accuracy of the six per\-language specialists at each SFT\-token budget, averaged across MGSM\-Rev2, Global\-MMLU\-Lite, GPQA\-Diamond, AIME 24/25, and HumanEvalPlus in the training language\. Token counts are approximate\.±\\pmdenotes the sample standard deviation across runs\.Table 14:Detailed evaluation of each per\-language specialist on the English and native versions of MGSM\-Rev2, Global\-MMLU\-Lite, GPQA\-Diamond, AIME 24/25, and HumanEvalPlus\. Bold marks the best score per column within each model\.±\\pmdenotes the sample standard deviation across runs\.### A\.6Layer\-Swap Source Language

Table[15](https://arxiv.org/html/2605.26735#A1.T15)verifies that the Layer Swap gain originates specifically from the English specialist’s mid\-layers: replacingQwen3\-8B\-ENwith the Chinese specialistQwen3\-8B\-ZHas the swap source erases the improvement on French\.

Table 15:Layer\-Swap source\-language ablation on French: layers L13–L22 transferred intoQwen3\-8B\-FRfrom either the English specialistQwen3\-8B\-EN\(sourceen\), the Chinese specialistQwen3\-8B\-ZH\(sourcezh\), or the German specialistQwen3\-8B\-DE\(sourcede\)\. Bold marks the best score per column\.±\\pmdenotes the sample standard deviation across runs\.### A\.7Chinese Layer\-Swap Window

Figure[5](https://arxiv.org/html/2605.26735#A1.F5)repeats the layer\-range ablation of Figure[4](https://arxiv.org/html/2605.26735#S4.F4)onQwen3\-8B\-ZHand motivates the L13–L20 window adopted forQwen3\-8B\-ZH\-Swapsince the L13–L22 window leaks∼\\sim60% of Chinese traces back into English\.

![Refer to caption](https://arxiv.org/html/2605.26735v1/x4.png)Figure 5:Same ablation as Figure[4](https://arxiv.org/html/2605.26735#S4.F4)\(b,c\) applied toQwen3\-8B\-ZH: mean accuracy across the five Chinese benchmarks and Chinese language fidelity, as a function of the transferred layer window\.### A\.8Per\-Layer Update Magnitudes

Figure[6](https://arxiv.org/html/2605.26735#A1.F6)reports the per\-layer L2 norm of the per\-language SFT updates\. Norms remain comparable across the stack, confirming that the mid\-stack agreement of Figure[4](https://arxiv.org/html/2605.26735#S4.F4)\(a\) reflects directional alignment rather than vanishing updates\.

![Refer to caption](https://arxiv.org/html/2605.26735v1/x5.png)Figure 6:Per\-layer L2 norm of the language\-specific SFT updateΔ​θL\(xx\)=θL\(xx\)−θbase\\Delta\\theta\_\{L\}^\{\(\\textsc\{xx\}\)\}=\\theta\_\{L\}^\{\(\\textsc\{xx\}\)\}\-\\theta\_\{\\mathrm\{base\}\}across the six per\-language specialistsxx∈\\in\{en,fr,de,es,zh,sw\}; mean with±\\pmstd\.