Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch

# Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch

Source: https://arxiv.org/html/2604.15490

###### Abstract

Recent developments in reasoning capabilities have enabled large language models to solve increasingly complex mathematical, symbolic, and logical tasks. Interestingly, while reasoning models are often trained to generate monolingual text, these models have also been observed to code-switch (i.e., mix languages). Prior works have either viewed code-switching as an undesirable error, attempted to control code-switching through modifications to input prompts or the output decoding process, or focus on narrow subsets of languages, domains, tasks, and models. We address these gaps by introducing the first linguistically and behaviorally motivated fine-tuning framework for identifying beneficial code-switched reasoning behaviors in large language models and teaching these models to code-switch more effectively for reasoning. First, we create and systematically analyze a dataset of reasoning traces from diverse models, languages, tasks, and domains to understand the types of code-switching behaviors found in existing reasoning models. Then, we develop fine-tuning interventions that teach reasoning models to code-switch based on our observations of helpful behaviors in existing models. We find that our framework can significantly increase beneficial code-switched reasoning behaviors in a data-efficient manner. Interestingly, we also find that code-switching behaviors in reasoning models can be modified by fine-tuning for tasks that do not directly demonstrate code-switching in reasoning (e.g., machine translation). Our work suggests that data-efficient interventions can instill helpful forms of code-switching behavior in reasoning models.

## 1 Introduction

When human multilinguals mix multiple languages during communication, they are said to *code-switch* (Myers-Scotton, 2017). Recently, works including Guo et al. (2025) have observed the parallel phenomenon of code-switching in reasoning—the long chain-of-thought texts produced by large language models (LLMs) when solving complex problems. While mechanisms of human code-switching have been studied extensively (e.g., see Myers-Scotton (2017) and Bullock & Toribio (2009) for an overview), in comparison, the emergent behavior of code-switching in LLM reasoning remains relatively poorly understood. Some works (e.g., Guo et al., 2025; Marchisio et al., 2024) treat code-switching as an error, while others attempt to use code-switching strategically for reasoning tasks in particular (Li et al., 2025a). In this work, we are particularly interested in code-switched LLM reasoning as a route to improve reasoning capabilities for lower-resource languages. This issue is of critical importance given that currently available reasoning models are mostly limited to high-resource languages like English (Ghosh et al., 2025).

To understand how state-of-the-art reasoning models code-switch and design interventions that improve code-switching quality, we seek answers to the following questions:

1. **RQ 1:** How do LLMs code-switch during reasoning?
2. **RQ 2:** What types of code-switching help performance on tasks requiring reasoning?
3. **RQ 3:** What types of fine-tuning interventions can teach reasoning models beneficial forms of code-switching for reasoning?

To address the gaps in our understanding of models' code-switched reasoning behaviors, first we create a dataset to study code-switching in reasoning, sourced from diverse models, languages, tasks, and domains (Section 3). Then, based on the findings of our first study, we design supervised fine-tuning approaches (including training and evaluation data) to develop beneficial code-switched reasoning behaviors. Finally, we analyze the reasoning traces produced by our fine-tuned models to identify significant effects on beneficial code-switched reasoning behaviors (Section 4).

Our contributions are as follows. First, we introduce the Code-Switched Reasoning (CoRe) corpus, which includes ~7,000 reasoning traces from diverse models, languages, tasks, and domains and identify the characteristics of code-switching behaviors that benefit reasoning performance. Second, we introduce a linguistically and behaviorally motivated fine-tuning framework for teaching reasoning models to code-switch for better reasoning performance. This framework extends CoRe with 1M tokens of training data per task per language, across 6 tasks and 7 languages. Finally, we perform a systematic analysis of changes in code-switching behaviors produced through our framework and the impact of these behavioral changes on reasoning performance. Overall, we hope that this work will serve as the foundation for further future work on code-switching as a means to improving multilingual LLM reasoning in low-resource settings. We will release our models, data, and code under an MIT License upon publication.

## 2 Related Work

We incorporate prior insights from research on human code-switching to contribute to a growing body of work on controlling the languages that models use during reasoning (e.g., Tam et al., 2025; Gao et al., 2025). Broadly defined, code-switching occurs when a multilingual person mixes multiple languages during communication (Myers-Scotton, 2017). Human code-switching is driven by a complex array of social and cognitive factors, e.g., ethnolinguistic identity and language proficiency (Myers-Scotton, 2017; Gumperz, 1977; Bullock & Toribio, 2009). While language models—and particularly reasoning models—have likewise been observed to code-switch, the mechanisms behind this behavior remain poorly understood. Some works treat code-switching by language models as an undesirable behavior (Marchisio et al., 2024; Guo et al., 2025). However, our work more closely aligns with that of Li et al., who instead attempt to harness code-switching as a useful reasoning behavior. The main difference between this work and Li et al. (2025a) is that while they attempt to control reasoning language at the decoding stage, we modify model behavior through supervised fine-tuning.

Prior work on code-switched reasoning lacks our coverage of diverse models, languages, domains, tasks, and reasoning behaviors. Yong et al. (2025) study the s1 model family using commonsense, factual, cultural, and causal reasoning benchmarks, with prompts in four languages other than English. Li et al. (2025b; 2025a) study Chinese/English math reasoning from the Chinese-English QwQ-32B-Preview model. Wang et al. (2025b; 2025a) study the Chinese-English DeepSeek-R1 model family and multilingual QwQ-32B, Qwen3, and Gemini 2.0 Flash Thinking models on commonsense, factual, and logical reasoning in 15 languages. In contrast, we cover 17 models from 10 families with diverse multilingual capabilities, 21 languages, and reasoning domains/tasks beyond those in prior work, e.g., moral reasoning.

When characterizing code-switching behaviors, Yong et al. identify the "quote-and-think" pattern and apply the pre-existing linguistic concepts of a matrix language and intra-/inter-sentential switching. Wang et al. quantify relative proportions of reasoning languages. Li et al. identify four main patterns. In contrast, we introduce a structured taxonomy of code-switching patterns across three dimensions and quantify these patterns through 9 code-switching metrics validated by linguistics and NLP literatures. Our work combines top-down, theory-driven and bottom-up, data-driven approaches for classifying reasoning behavior. Gandhi et al. (2025) demonstrated the value of theory-driven approaches by identifying reasoning model behaviors that both drive performance and parallel human behaviors. In contrast, Lee et al. (2025) characterize reasoning using bottom-up LLM-assisted brainstorming and clustering of the resulting text. While we base our approach on Lee et al., we focus specifically on code-switching. We introduce manual curation as an extra step to ground the resulting taxonomy of code-switching behaviors in prior real-world observations and theories of code-switching. Also in contrast with Lee et al., we include smaller multilingual models in our study and domains beyond math, science, and general knowledge.

In contrast with prior work, we conduct supervised fine-tuning (SFT) of reasoning models to modify model code-switching behavior. Yong et al. (2025) and Zhao et al. (2026) both attempt to control reasoning language using prompt engineering and test-time interventions. Their approaches include instructing the model to reason in a different language through the prompt and appending tokens in the target language to the model's generation, which are then conditioned upon to guide the language of subsequent text generation. Alternatively, Li et al. (2025a) train a probe to guide language choice during the decoding process. Similarly, Wang et al. (2025a) guide the decoding process by masking out the logits of scripts of non-target reasoning languages during decoding. However, these prior approaches are fundamentally limited in the extent to which they can control model behavior, as they do not modify the underlying model weights. Our work overcomes this limitation by directly applying supervised fine-tuning to modify reasoning model behavior during multilingual reasoning. Our SFT approach is partially inspired by Muennighoff et al. (2025a; 2025b), which demonstrated that reasoning capabilities can be developed using SFT on a small but carefully curated training corpus.

## 3 Understanding Code-Switched Reasoning Behaviors

To answer RQ 1 on how LLMs code-switch, we create a new corpus, CoRe, of 7,000 reasoning traces from 15 models, described as follows.

### 3.1 Approach

Both top-down and bottom-up approaches to characterizing reasoning behaviors (cf. Section 2) face limitations. Theory-grounded, top-down approaches which look for pre-defined, human-aligned behaviors in model reasoning (e.g., Gandhi et al., 2025) may miss novel reasoning behaviors that do not align with humans. On the other hand, data-driven, bottom-up approaches which derive categories of behaviors from observations of model reasoning (e.g., Lee et al., 2025) may fail to surface misalignment with categories of human behaviors. In particular, these data-driven, bottom-up approaches may fail to observe behaviors that only appear in humans, and not in models. To address these limitations, we fuse top-down theory-driven and bottom-up data-driven approaches in our framework.

**Data and models.** To create our CoRe corpus, we generate examples of reasoning from diverse models, languages, tasks, and domains for generalizability. We generate reasoning examples from 15 models of diverse sizes, reasoning capabilities, and multilingual capabilities (see appendices A.2 and A.4 for full list). We select prompts for generating reasoning examples from seven datasets, from s1K (covering STEM, law, logic, puzzles, and humanities) to UniMoral (covering moral reasoning; see appendix A.4 for full list). Prompts cover 18 languages (Arabic, Bengali, Burmese, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Mandarin Chinese, Portuguese, Russian, Spanish, Swahili, Thai, and Yoruba), 10 scripts, and 8 language families (Hammarström et al., 2025; Unicode, 2025). We include approximately 50 instances from each model/language/dataset combination, for about 7,000 instances total. (See appendix A.5 for the prompting configurations used to generate the reasoning examples.)

**Code-switched reasoning behavior taxonomy.** Next, we develop a taxonomy of code-switched reasoning behaviors by fusing top-down theory-driven and bottom-up data-driven approaches. We rely on the reasoning examples described above as input context to Gemini 2.5 Flash for brainstorming criteria that differentiate code-switching strategies (Google, 2025a; Gemini Team et al., 2025). At this stage, we introduce helpful inductive biases to guide the taxonomy development by providing Gemini 2.5 Flash with a minimal and broad definition of code-switching ("use of multiple scripts or languages"). We apply topic modeling to Gemini's brainstormed criteria, using the BERTopic pipeline (Grootendorst, 2022). Finally, we introduce further helpful structure in a top-down manner. In particular, we manually consolidate redundant topics, dropping those unlikely to generalize (e.g., "Interpretation of Yoruba Terms"), and group topics into dimensions, categories, and subcategories. Manual curation of topics into the final hierarchy is guided by the authors' knowledge of the relevant literature on code-switching and reasoning in both humans and LLMs. See appendices A.6 and A.7 for details.

Our taxonomy of code-switched reasoning behaviors features three dimensions, grounded both in theories from prior literature and our dataset:

1. **Function** describes the purpose served by a particular code-switch during reasoning. Common functions of code-switching during reasoning include *translating* from one language into another and *quoting* material from the initial user prompt in its original language, potentially while reasoning primarily in a different language. Humans also code-switch to quote material in its original language, provide context in another language, and repeat phrases in another language for emphasis and clarity.