Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

# Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
Source: [https://arxiv.org/html/2606.30790](https://arxiv.org/html/2606.30790)
Avisha Das1,Mihir Parmar2,Mohana Ramnath1, andPulkit Verma3 1Shiv Nadar University Chennai,2Google Cloud AI Research,3IIT Madras avishadas@snuchennai\.edu\.in,mihirparmar@google\.com, mohana23110098@snuchennai\.edu\.in,pulkitv@cse\.iitm\.ac\.in

###### Abstract

Romanized Code Mixing \(RCM\), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities\. While Large Language Models \(LLMs\) perform strongly on monolingual and native\-script benchmarks, their ability to follow instructions and reason over RCM\-based content remains largely unexplored\. To this end, we introduce theIndi\-RomCoMbenchmark for facilitating systematic evaluation onIndicRomanizedCode\-Mixed instructions\. Our benchmark spans seven instruction\-following tasks, four widely spoken Indic languages, and three controlled code\-mixing intensity levels\. We extensively evaluate a suite of LLMs covering proprietary, open\-weight, and Indic\-focused models under zero\- and few\-shot settings\. LLMs consistently underperform on RCM instructions, with performance degrading as code\-mixing density increases\. Furthermore, reasoning tasks suffer less degradation than detection tasks \(e\.g\., Toxicity\) because the generated explanations offer necessary context\. We believeIndi\-RomCoMhelps the community in developing inclusive multilingual systems\.

\[bengali\] rm\[BoldFont=NotoSansBengali\-Regular\.ttf\] NotoSansBengali\-Regular\.ttf \[tamil\] rm\[BoldFont=NotoSansTamil\-Regular\.ttf\] NotoSansTamil\-Regular\.ttf \[hindi\] rm\[BoldFont=NotoSansDevanagari\-Regular\.ttf\]NotoSansDevanagari\-Regular\.ttf \[gujarati\]rm\[BoldFont=NotoSansGujarati\-Regular\.ttf\] NotoSansGujarati\-Regular\.ttf

Indi\-RomCoM: Code\-Mixed Benchmark for Evaluating LLMs on Romanized Indic\-English Instructions

Avisha Das1, Mihir Parmar2, Mohana Ramnath1,andPulkit Verma31Shiv Nadar University Chennai,2Google Cloud AI Research,3IIT Madrasavishadas@snuchennai\.edu\.in,mihirparmar@google\.com,mohana23110098@snuchennai\.edu\.in,pulkitv@cse\.iitm\.ac\.in

## 1Introduction

Ensuring that Large Language Models \(LLMs\) can follow instructions in everyday regional language styles is a critical requirement for global deploymentGupta et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib16)\); Deuchar \([2020](https://arxiv.org/html/2606.30790#bib.bib13)\)\. India is a clear example of this challenge because it contains a highly diverse landscape of 22 official languages spoken by over 1\.4 billion people\. In this multilingual environment, English has mixed with local tongues to create a natural, rule\-governed phenomenon called Romanized Code\-Mixing \(RCM\)Sengupta et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib37)\); Winata et al\. \([2023](https://arxiv.org/html/2606.30790#bib.bib42)\)\. Driven by character limits, typing by sound on smartphones, and conversational convenience, millions of bilingual users fluidly blend regional Indic words, English terms, and local grammar into a single sentence, using the Latin alphabet exclusively\.

![Refer to caption](https://arxiv.org/html/2606.30790v1/teaser_2.png)Figure 1:Example \(LLaMa\-7B\-Instruct\) showing failure to understand romanized “tanglish”\.For instance, a Hindi\-English speaker asks, “Yeh loan ke liye mujhe kya karna padega?”, which translates to, “What do I have to do to get this loan?”\. These sentences are highly structured everyday registers that dominate South Asian digital text\. Yet, while generative AI models perform well on standard, single\-language test sets, their accuracy drops sharply when processing these mixed linguistic setups, as exemplified in Figure[1](https://arxiv.org/html/2606.30790#S1.F1)\(Tamil\-English\)\. Open\-weight LLMs built specifically for the Indian subcontinent, such as Airavata\(Gala et al\.,[2024](https://arxiv.org/html/2606.30790#bib.bib14)\), TamilLLaMABalachandran \([2023](https://arxiv.org/html/2606.30790#bib.bib5)\), Sarvam\-1\(Sarvam AI,[2024](https://arxiv.org/html/2606.30790#bib.bib36)\), improve performance on native scripts, but their capabilities fail when confronted with Romanized text\.

A primary obstacle is that the NLP community lacks the benchmarking systems needed to systematically diagnose these failure modes\. Traditional code\-mixed data collections like GLUECoS\(Khanuja et al\.,[2020a](https://arxiv.org/html/2606.30790#bib.bib22)\)are limited to low\-level, word\-by\-word classification tasks such as labeling parts of speech, making them unable to evaluate high\-level reasoning\. Conversely, newer generative benchmarks like CodeMixBench\(Yang and Chai,[2025a](https://arxiv.org/html/2606.30790#bib.bib45)\)are limited only to native scripts\. This effectively blinds them to the spelling and phonetic shifts that characterize human\-model interactions\. Refer to Table[1](https://arxiv.org/html/2606.30790#S2.T1)for comparison across different state\-of\-the\-art code\-mixing benchmarks\.

To bridge this evaluative gap, we introduceIndi\-RomCoM, a systematic benchmark designed to stress\-test large language models on Romanized Indic\-English instructions\. Covering four major regional languages across seven core task types, our framework utilizes a linguistically controlled generation pipeline to divide inputs into precise code\-mixing intensity layers\. Through this process, we construct a large\-scale evaluation suite validated by native bilingual annotators to guarantee strong semantic preservation and contextual naturalness\. UsingIndi\-RomCoMwe evaluate 19 prominent language models under zero\-shot and few\-shot setups\. Our experiments show that climbing code\-mixing intensity drives severe, monotonic performance degradation across both general\-purpose and specialized Indic open\-weight architectures alike\. A comprehensive structural breakdown of the framework is provided in Section[3](https://arxiv.org/html/2606.30790#S3)\.

In summary, we introduceIndi\-RomCoM, a multi\-task benchmark developed using novel controlledword\-selection frameworkto evaluate LLMs on RCM\. Additionally, we present a comprehensiveempirical analysisof widely used LLMs and introduce theRegister Defection Rate \(RDR\)to quantify language switching failures during reasoning\.

## 2Prior Work

Table 1:A comparison against existing resources\. Abbreviations mean NS for Native Script, RS for Romanized Native Script, and Clf\. for Classification\. The Density column tracks if a benchmark evaluates performance across varying language mixing percentages\. Our framework uniquely supports separate multi\-density layers representing text sets with 25 percent and 50 percent and 75 percent word switching intensity\.![Refer to caption](https://arxiv.org/html/2606.30790v1/benchmark_pipeline_2.png)Figure 2:Overview of theIndi\-RomCoMcreation framework\. The pipeline processes English\-only and Indic\-native tasks through initial translation \(Phase I\), applies a taxonomy\-guided generation engine to create controlled code\-mixing at 25%, 50%, and 75% intensities \(Phases II & III\), and concludes with script romanization to produce the final multi\-level benchmark \(Phase IV\)\.#### Benchmarks for code\-mixed NLP

Early computational modeling of code\-mixed \(CM\) discourse focused primarily on token\-level classification tasks or synthetic text generation\. Frameworks like the GCM Toolkit\(Rizvi et al\.,[2021](https://arxiv.org/html/2606.30790#bib.bib34)\)built early foundations by computationally operationalizing structural linguistic constraints, such as the Equivalence Constraint and Matrix Language Frame theories, to synthesize data from parallel corpora\. Similarly,Sravani and Mamidi \([2023](https://arxiv.org/html/2606.30790#bib.bib38)\)leveraged neural machine translation \(NMT\) architectures, employing filters like the Code\-Mixing Index \(CMI\) and M\-index to refine synthetic outputs\. While these approaches advanced localized language modeling and individual datasets like GLUECoS\(Khanuja et al\.,[2020b](https://arxiv.org/html/2606.30790#bib.bib23)\)or LinCE\(Aguilar et al\.,[2020](https://arxiv.org/html/2606.30790#bib.bib2)\), they heavily relied on formal native scripts, targeted classification pipelines, and lacked the framework to explore multi\-task instruction following\.

As detailed in Table[1](https://arxiv.org/html/2606.30790#S2.T1), recent generative evaluations such as CodeMixBench\(Yang and Chai,[2025a](https://arxiv.org/html/2606.30790#bib.bib45)\)and ChiEngMixBench\(Yang et al\.,[2026](https://arxiv.org/html/2606.30790#bib.bib44)\)highlight significant LLM degradation on CM inputs, particularly in mathematical reasoning and dialogue\.

#### Indic LLMs and Resources

The development of Indic\-focused models and datasets has accelerated to address South Asia’s linguistic diversity\. Large\-scale resource efforts such as IndicLLMSuite\(Khan et al\.,[2024a](https://arxiv.org/html/2606.30790#bib.bib20)\)and UPDESH\(Chitale et al\.,[2025](https://arxiv.org/html/2606.30790#bib.bib7)\)provide extensive pretraining corpora across 13–22 languages, while specialized open\-weight architectures including Airavata\(Gala et al\.,[2024](https://arxiv.org/html/2606.30790#bib.bib14)\), Tamil\-Llama\(Balachandran,[2023](https://arxiv.org/html/2606.30790#bib.bib5)\), and Sarvam\-1\(Sarvam AI,[2024](https://arxiv.org/html/2606.30790#bib.bib36)\)maximize localized syntactic representation\.

Despite this progress, critical gaps persists\. First, these suites remain anchored to native scripts, leaving CM\-ed validation absent from official evaluation loops\. Transliterated or CM\-ed prompts collected through crowdsourcing \(e\.g\., Airavata’s Anudesh111[https://github\.com/AI4Bharat/Anudesh](https://github.com/AI4Bharat/Anudesh)pipeline\) are rarely isolated for formal behavioral verification or benchmarked against varying density thresholds\. Recent evaluations reinforce this urgency: IndicDB\(Dawar et al\.,[2026](https://arxiv.org/html/2606.30790#bib.bib12)\)exposed sharp execution drops under Romanized Hinglish instructions, and IndicSafe\(Pattnayak and Chowdhuri,[2026](https://arxiv.org/html/2606.30790#bib.bib32)\), despite auditing models for South Asian safety, overlooks the morphological blending inherent to real\-world bilingual typing\. The few localized studies that do address Romanization remain narrow in scope, e\.g\.,Choudhary et al\. \([2026](https://arxiv.org/html/2606.30790#bib.bib8)\)restrict their analysis to verb\-level inflections within Hindi\-English pairs\.

Indi\-RomCoMaddresses these gaps, providing the first human\-validated, multi\-task, multi\-tier benchmark for Romanized code\-mixed instruction\-following, evaluating a comprehensive suite of LLMs \(closed\- and open\-weights along with Indic\) against the natural linguistic drift in informal Indic writing\.

## 3Indi\-RomCoMBenchmark

Figure[2](https://arxiv.org/html/2606.30790#S2.F2)demonstrated all the phases of theIndi\-RomCoMbenchmark\.

### 3\.1Task Design and Selection

We evaluate LLM performance on Romanized code\-mixed instructions across seven tasks spanning three competency dimensions:logical reasoning,semantic understanding, andcontent moderation\. All tasks are drawn from well\-validated public sources with balanced label distributions, ensuring that observed performance differences reflect genuine code\-mixing sensitivity rather than data artefactsBai et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib4)\); Kwan et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib25)\)\. For logical reasoning, we include Word Analogy and Question Decomposition from Super\-NaturalInstructionsWang et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib41)\)and Mathematical Reasoning from GSM8KCobbe et al\. \([2021](https://arxiv.org/html/2606.30790#bib.bib9)\)\. For semantic understanding, we draw on Paraphrase Detection, Sentiment Analysis, and Natural Language Inference from the AI4Bharat IndicXtreme collectionKunchukuttan et al\. \([2020](https://arxiv.org/html/2606.30790#bib.bib24)\); Aggarwal et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib1)\)\. Toxicity Detection is adapted from the IndicAlign\-Toxic MatrixChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\)\. Table[2](https://arxiv.org/html/2606.30790#S3.T2)summarises the full task inventory; detailed descriptions and dataset statistics are in Appendix[A](https://arxiv.org/html/2606.30790#A1)\.

#### Languages

We evaluate romanized Indic\-English code\-mixed settings across four major Indian languages: Hindi, Bengali, Gujarati, and Tamil, covering both Indo\-Aryan and Dravidian language families\. These languages capture diverse linguistic structures, resource levels, and code\-mixing patterns commonly observed in bilingual Indic\-English communicationChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\); Jaavid et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib18)\)\.

Table 2:Indi\-RomCoMtask inventory\. All instances are drawn from validated sources with stratified label distributions\. Task type: GEN = Generation and CLF = Classification\.

### 3\.2Code Mixing Framework

We develop a four\-step framework for generating controlled Indic\-English RCM\-ed instanced for theIndi\-RomCoMbenchmark\. Building upon the Equivalence ConstraintPoplack \([2001](https://arxiv.org/html/2606.30790#bib.bib33)\)and Matrix Language FrameMyers\-Scotton \([1993](https://arxiv.org/html/2606.30790#bib.bib30)\)theories of bilingual switching, we systematically vary the mixing intensity while preserving semantic consistency\. Full implementation details and prompting strategies, etc\. are in Appendix[B](https://arxiv.org/html/2606.30790#A2)\.

#### English to Indic Translation\.

English\-only tasks \(Word Analogy, Question DecompositionWang et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib41)\), and GSM8KCobbe et al\. \([2021](https://arxiv.org/html/2606.30790#bib.bib9)\)\) are translated into indic versions following a two\-step pipeline of machine translation and refinement\. Other tasks are derived directly from their existing Indic datasets, IndicXNLIAggarwal et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib1)\), IndicAlign\-ToxicChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\), IndicXParaphraseKhan et al\. \([2024b](https://arxiv.org/html/2606.30790#bib.bib21)\), and IndicSentimentGala et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib14)\)\.

#### Word\-Selection Taxonomy\.

Each switchable word or phrase is assigned to one of five linguistically motivated categoriesPoplack \([2001](https://arxiv.org/html/2606.30790#bib.bib33)\); Winata et al\. \([2023](https://arxiv.org/html/2606.30790#bib.bib42)\), logged with its category label, Indic replacement, and a linguistic rationale to ensure substitutions reflect natural bilingual behavior rather than arbitrary replacement\. Table[9](https://arxiv.org/html/2606.30790#A2.T9)in Appendix summarizes the five categories, their switching policies, and illustrative cross\-lingual examples\.

#### Code\-Mixing Intensity Levels\.

Three intensity levels \(25%, 50%, 75%\) are produced by progressively activating category subsets, with the Code\-Mixing Index \(CMI\)Srivastava and Singh \([2021](https://arxiv.org/html/2606.30790#bib.bib39)\)quantifying the proportion of replaced English tokens\.

#### Romanization of Indic Script\.

All Indic\-script outputs are converted to Latin script using AksharantarMadhani et al\. \([2023b](https://arxiv.org/html/2606.30790#bib.bib27)\)and Aksharamukha, with end\-to\-end romanizations sourced from IndicLLMSuiteKhan et al\. \([2024b](https://arxiv.org/html/2606.30790#bib.bib21)\)where available\. Natural transliteration variability is intentionally preserved to reflect real\-world romanized code\-mixed usage by multilingual South Asian users\.

### 3\.3Benchmark Validation and Statistical Analysis

#### Dataset Size

We sample 100 instances from each of seven tasks using stratified sampling over labels and difficulty levels, yielding 700 English base instances\. Each instance is instantiated across the four Indic languages and three code\-mixing intensity levels, resulting in8,400 evaluation instances\. Task\-wise sampling details are provided in Appendix[B\.6](https://arxiv.org/html/2606.30790#A2.SS6)\.

#### Human Validation

To assess the quality ofIndi\-RomCoM, we conduct a human evaluation study\. We recruit four native bilingual annotators \(Tamil, Bengali, Hindi, Gujarati\) with experience using Romanized code\-mixed communication\. Annotators independently evaluate samples across all tasks and CM levels using a 3\-point Likert scale \(1 = Unnatural, 2 = Somewhat Natural, 3 = Natural\), rating the code\-mixed instances on two aspects:\(i\) Translation Fidelity, measuring whether romanized code\-mixed text preserves the semantic meaning of the English source using corpus\-level Character n\-gram F\-score \(ChrF;Chitale et al\.[2025](https://arxiv.org/html/2606.30790#bib.bib7)\), averaged across all seven tasks for each language\. And,\(ii\) Code\-mixing Naturalness, assessing whether the code\-mixed instances resembles bilingual usage rather than mechanical substitution\.

Two annotators per language rate the same instances independently; we report Cohen’sκ\\kappaCohen \([2013](https://arxiv.org/html/2606.30790#bib.bib10)\)for pairwise naturalness agreement\. Table[3](https://arxiv.org/html/2606.30790#S3.T3)summarizes per\-language results\. An average ChrF of67\.00confirms substantial semantic overlap consistent with 50%\-CM intensity\. Bengali and Tamil score highest on ChrF, reflecting denser preserved English structure, while Hindi and Gujarati score lower due to heavier Indic vocabulary switching\. Overall agreement is moderate \(averageκ=0\.615\\kappa=0\.615\) for code\-mixing naturalness, consistent with the inherent subjectivity of naturalness assessment for code\-mixed textChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\)\. Agreement is highest for Gujarati \(0\.9050\.905\) and lower for Bengali and Tamil due to stylistic variation in generation\-heavy tasks\. Higher code\-mixing intensities generally produce lower agreement, especially for Question Decomposition, where dense predicate\-level switching creates less common bilingual constructions\. Per\-task, per\-language, per\-CM\-level results breakdown are provided in Appendix[B\.6](https://arxiv.org/html/2606.30790#A2.SS6)\.

Table 3:ChrF scores are reported between 50%\-CM romanized output\-EN source\.
#### Indi\-RomCoMContent Analysis

We analyzeIndi\-RomCoMto verify if the three code\-mixing intensity levels are distinct, category activations follow the intended linguistic progression, and task\-level variation is systematic and not artifactual\. Thecode\-mixing qualityfor each intensity level is studied using code\-mixing index \(CMI\) metric\. Table[4](https://arxiv.org/html/2606.30790#S3.T4)reports the mean CMI \(μ±σ\\mu\\pm\\sigma\) per task and CM level across all 8,400 instances\. Mean CMI rises monotonically:μ=0\.249\\mu=0\.249\(25%\-CM\) to0\.2940\.294\(50%\-CM\) to0\.3600\.360\(75%\-CM\)\. All pairwise differences are statistically significant \(Wilcoxon signed\-rank,p<10−100p<10^\{\-100\}, Bonferroni\-corrected\), confirming that the three conditions represent genuinely distinct code\-mixing regimes\. We also perform thecategory\-based frequency analysison the dataset’s rationale annotations\. Figures[3](https://arxiv.org/html/2606.30790#S3.F3)a–b show the proportional substitution breakdown at 50%\-CM and 75%\-CM respectively, where multi\-category activation produces interpretable task\-level variation\. Additional details are reported in Appendix[B\.7](https://arxiv.org/html/2606.30790#A2.SS7)\.

Table 4:Mean CMI \(±\\pmSD\) per task and CM intensity level, averaged across four languages and all instances per task\.![Refer to caption](https://arxiv.org/html/2606.30790v1/x1.png)Figure 3:Word\-category substitution frequency at 50%\-CM \(top\) and 75%\-CM \(bottom\)\. At 25%\-CM all substitutions are CAT\-C \(100%\) by construction and are not shown\. Colours: CAT\-C \(blue\), CAT\-B \(orange\), CAT\-D \(green\), CAT\-A \(red\), CAT\-E \(purple\)\. Labels for segments\>\>7%\.

## 4Experimental Setup

### 4\.1Models

We evaluate models across proprietary systems \(OpenAI’s GPT 3\.5\-turboOpenAI \([2026](https://arxiv.org/html/2606.30790#bib.bib31)\), Google’s Gemini 3\.5 FlashKavukcuoglu et al\. \([2026](https://arxiv.org/html/2606.30790#bib.bib19)\), and Anthropic’s Claude 4\.6 OpusAnthropic \([2026](https://arxiv.org/html/2606.30790#bib.bib3)\)\), general open\-weight architectures \(Meta’s LLaMa3 seriesGrattafiori et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib15)\), Alibaba Cloud’s Qwen \(2\.5\-3\.5 series\)Yang et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib43)\), Google’s Gemma4 seriesTeam et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib40)\), and MistralAI’s Mistral seriesMistral AI \([2026](https://arxiv.org/html/2606.30790#bib.bib29)\)\), Indic\-focused multilingual and regional models, and specialized fine\-tuned ablation variants to analyze performance across scale and training types\. For our ablation study, we fine\-tune LLaMA\-3\.2\-3B, LLaMA\-3\.1\-8B, and Gemma\-2\-7B on our training split using Low\-Rank Adaptation to assess the necessity of explicit code\-mixed supervision\. A complete model list, fine\-tuning configurations, comprehensive hyperparameter details, and dataset stratification metrics are provided in Appendix[C\.1](https://arxiv.org/html/2606.30790#A3.SS1)\.

### 4\.2Prompting Protocol

Table 5:Structural breakdown of our instruction\-following dataset tasks\.Orangeandbluelabels denote task definition and query components respectively\.Tealtags indicate the example language: Tamil \(Sentiment, Decomposition\), Hindi \(NLI, Toxicity, Mathematics\), Bengali \(Paraphrase\), Gujarati \(Word Analogy\), all at 50%\-CM\. Placeholders:⟨\\langleDk⟩k\\rangle=kk\-th demo input,⟨\\langleLk⟩k\\rangle= label,⟨\\langleEk⟩k\\rangle= explanation,⟨\\langleAk⟩k\\rangle= answer,⟨\\langlein⟩\\rangle= query input\.We evaluate all models under zero\-shot and stratified 3\-shot settings using structured four\-part prompt templates that enforce explicit register maintenance rules\. The underlying evaluations span seven core instruction\-following tasks where input values are dynamically mapped to specific romanized code\-mixed intensity configurations\. Complete structural prompt definitions, few\-shot conditioning frameworks, and individual task\-specific instruction boxes are also provided in Appendix[C\.2](https://arxiv.org/html/2606.30790#A3.SS2)\.

### 4\.3Evaluation Metrics

We use four metrics to evaluate model performance across our benchmark\. \(i\)Task Accuracy \(ACC\)measures exact string matches for classification alongside standard extraction rubrics for generation tasks; \(ii\)Register Defection Rate \(RDR\)tracks the proportion of model outputs that incorrectly revert to standard English text; \(iii\)Vocabulary Coverage Rate \(VCR\)measures the percentage of unique romanized tokens that exist as complete entries in a model tokenizer vocabulary; and \(iv\)Performance Gap \(PG\)calculates the absolute accuracy loss between the English baseline and each code\-mixed intensity level\. We validate these trends using Wilcoxon signed\-rank tests, Bonferroni corrections, and Spearman correlation checks\. Detailed definitions, parsing criteria, and statistical configurations are provided in Appendix[C\.3](https://arxiv.org/html/2606.30790#A3.SS3)\.

## 5Results and Analysis

### 5\.1Main results

We present the evaluation results on theIndi\-RomCoMbenchmark\. Table[6](https://arxiv.org/html/2606.30790#S5.T6)reports average task accuracy across all seven tasks for the 19 models in comparison with the English baseline \(BL\) and three code\-mixing intensity levels\. Figure[4](https://arxiv.org/html/2606.30790#S5.F4)illustrates per\-task performance variations across a selected set of best performing models and code\-mixing settings\. Additionally, Table[7](https://arxiv.org/html/2606.30790#S5.T7)reports the Register Defection Rate \(RDR\) across all models and CM conditions, measuring the proportion of responses that revert primarily to English despite receiving code\-mixed instructions\.

Table 6:Average task accuracy \(%\) across 7 tasks in zero\-shot setting, averaged over 4 languages \(±\\pmSD across tasks\)\. BL = English baseline\. PG = Performance Gap \(BL−\-CM level; positive = accuracy drop\)\. CM = Code Mixing\.Bold= best BL per family\.![Refer to caption](https://arxiv.org/html/2606.30790v1/x2.png)Figure 4:Average task accuracy \(%\) under code\-mixed \(CM\) instructions by model and task\.Each bar reports accuracy averaged across three CM intensity levels \(25%, 50%, 75%\) and four Indic languages in the zero\-shot setting\. Error bars denote standard deviation\. Dashed vertical lines separate the seven task groups\.Table 7:Register Defection Rate \(RDR, %\) when the query is in CM register\. Lower RDR indicates stronger register adherence\. Avg RDR is the mean across the three CM intensity levels\.Poor performance of models on Indic RCM\-ed data as compared to EnglishTable[6](https://arxiv.org/html/2606.30790#S5.T6)reveals a consistent pattern of accuracy degradation under Romanized code\-mixed \(RCM\) conditions across nearly all models\. Every model except the Qwen3 variants records a positive performance gap, proving Indic RCM input is substantially harder than English for current LLMs\. Even the strongest models are affected: Claude Opus 4\.6 drops from 68\.7% at baseline to 61\.2% at 75%\-CM \(PG\-75=\+7\.5=\+7\.5\), and Sarvam\-30B, despite dedicated Indic pretraining \(falls from 64\.2% to 56\.1% \(PG\-75=\+8\.1=\+8\.1\)\)\. Smaller models suffer steeper drops; Mistral\-3B loses 17\.5 percentage points by 75%\-CM\. These results demonstrate that no model family is robust to Indic RCM input, exposing a critical gap in multilingual LLM evaluation\.

Model performance varies by task typeFigure[4](https://arxiv.org/html/2606.30790#S5.F4)reveals task\-level variation in model robustness under RCM input\. GSM8K yields the highest accuracy overall \(up to∼\\sim80% for LLaMA\-3\.1\-70B\), consistent with CAT\-0 protection of mathematical tokens insulating problem content from CM disruption\. Sentiment analysis achieves the most consistent cross\-model performance, suggesting that naturally code\-mixed product\-review vocabulary aids rather than hinders comprehension\. Conversely, Word Analogy and Toxicity detection show the lowest absolute accuracy across all models, the former due to morphological sensitivity, the latter exposing a critical safety gap: models trained on English safety data struggle to detect harm expressed in CM register, with most models scoring below 50%, a directly consequential failure for real\-world deployment\.

Indic Pretraining Drives Register AdherenceTable[7](https://arxiv.org/html/2606.30790#S5.T7)shows that most open\-source models defect overwhelmingly into English under RCM input: LLaMA\-3\.1\-8B, Qwen2\.5\-1\.5B, and Qwen3\-4B all exceed 98% Avg RDR, regardless of scale\. The only models that genuinely maintain CM register are Indic\-trained: Airavata\-7B \(16\.5%\), TamilLLaMA\-7B \(19\.9%\), and Sarvam\-30B \(26\.1%\)\. Notably, RDR rises with CM intensity across nearly all models, confirming that higher mixing pressure amplifies register defection rather than suppressing it\.

### 5\.2Ablation Studies

Few\-Shot Prompting help close the performance gapThree\-shot CM demonstrations yield a consistent 7–8% accuracy gain over zero\-shot across all model families \(Appendix Table[13](https://arxiv.org/html/2606.30790#A3.T13)\), driven primarily by a sharp reduction in register defection on classification tasks, providing in\-register examples anchors the output format\. However, performance gaps \(PG\) remain largely unchanged, confirming that CM examples help models adopt the correct register without improving their underlying comprehension of code\-mixed content, and gains on GSM8K are minimal, replicating prior findings on mathematical reasoningYang and Chai \([2025b](https://arxiv.org/html/2606.30790#bib.bib46)\)\. The results are shown in Appendix Section[D](https://arxiv.org/html/2606.30790#A4)\.

![Refer to caption](https://arxiv.org/html/2606.30790v1/x3.png)Figure 5:Per\-language CM accuracy across 7 tasks and 3 LLM families\.Each spider plot shows average accuracy \(%\) across 25%\-, 50%\-, and 75%\-CM conditions for Proprietary \(blue\), Open\-weight \(green\), and Indic\-focused \(red\) model families\. Performance decreases from Hindi to Gujarati, reflecting differences in LLM pretraining corpus coverage across languages\.Model performance varies across languagesFigure[5](https://arxiv.org/html/2606.30790#S5.F5)shows that Hindi \(Hinglish\) achieves the highest average CM accuracy across all three model families, consistent with its greater representation in LLM pretraining corporaChoudhary et al\. \([2026](https://arxiv.org/html/2606.30790#bib.bib8)\)\. Bengali \(Banglish\) performs moderately, while Tamil \(Tanglish\) exhibits sharper degradation on generation tasks, attributable to its Dravidian SOV syntax producing more complex CM structures\. Gujarati \(Gujlish\) is the most challenging language overall, reflecting its minimal CommonCrawl representation and near\-zero coverage in existing Indic\-focused pretraining\.

Finetuning ablationTo assess whether targeted supervision can bridge the code\-mixing gap, we fine\-tuned LLaMA\-3\.2\-3B and LLaMA\-3\.1\-8B via QLoRA \(rank = 16,α\\alpha= 32, 4\-bit NF4 quantisation\) on 50%\-CM Romanised instruction pairs spanning all four languages and seven tasks\. Both models converge to near\-identical aggregate exact\-match rates \(86\.3%86\.3\\%vs\.86\.4%86\.4\\%\), confirming that scale yields diminishing returns once the CM format is supervised directly, while the near\-total register\-defection rate \(99\.9%99\.9\\%\) reflects the Roman\-script nature of the outputs rather than generation failure\. Detailed results are presented in Appendix Table[14](https://arxiv.org/html/2606.30790#A3.T14)\.

## Conclusion

We presentedIndi\-RomCoM, a benchmark to stress\-test large language models on Romanized Indic\-English instructions across multiple languages, tasks, and intensity levels\. Our extensive evaluation of 17 prominent language models reveals that climbing code\-mixing density drives severe, monotonic performance degradation across both general\-purpose and specialized architectures\. Ablation variants show that this systematic challenge cannot be resolved by model scale or native\-script pretraining alone, proving that targeted code\-mixed supervision is necessary\. This framework establishes a rigorous foundation for future research into register\-robust multilingual models that reflect naturalistic bilingual communication\.

## Limitations

Indi\-RomCoMhas several limitations worth acknowledging\. The controlled generation pipeline may underrepresent organic real\-world code\-mixing phenomena such as phonetic drift and intra\-word mixing\. Human validation relied on only four annotators, which is too small a pool to capture sociodemographic variation in code\-mixing norms\. Coverage is restricted to four Indic languages, leaving out major Indic language communities such as Telugu, Kannada, and Marathi\. Additionally, the benchmark inherits labeling noise from the upstream crowdsourced datasets it draws from, and several instruction\-following categories including summarization and open\-ended question answering are absent from the current task suite\.

## Ethical Considerations

All tasks are derived from publicly available datasets, and no user data was collected from private communications or social media\. The Toxicity Detection task contains genuinely harmful content by design because evaluating model refusal behavior under code\-mixed framing is an important safety challenge\. However, the benchmark should not be used to generate or amplify harmful content\. Annotators rating toxic instances were informed in advance and could skip individual samples at any time\. We also caution against benchmark overfitting, where models are tuned to specific intensity levels without genuinely improving on naturalistic code\-mixed understanding\.

## References

- Aggarwal et al\. \(2022\)Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan\. 2022\.Indicxnli: Evaluating multilingual inference for indian languages\.In*Proceedings of the 2022 conference on empirical methods in natural language processing*\.
- Aguilar et al\. \(2020\)Gustavo Aguilar, Sudipta Kar, and Thamar Solorio\. 2020\.[LinCE: A centralized benchmark for linguistic code\-switching evaluation](https://aclanthology.org/2020.lrec-1.223/)\.In*Proceedings of the Twelfth Language Resources and Evaluation Conference*\.
- Anthropic \(2026\)Anthropic\. 2026\.[Introducing Claude Opus 4\.6](https://www.anthropic.com/news/claude-opus-4-6)\.Anthropic News\.Accessed: May 24, 2026\.
- Bai et al\. \(2024\)Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li\. 2024\.[LongBench: A bilingual, multitask benchmark for long context understanding](https://doi.org/10.18653/v1/2024.acl-long.172)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Balachandran \(2023\)Abhinand Balachandran\. 2023\.Tamil\-Llama: A new Tamil language model based on Llama 2\.*arXiv preprint arXiv:2311\.05845*\.
- Bokamba \(1989\)Eyamba G Bokamba\. 1989\.Are there syntactic constraints on code\-mixing?*World Englishes*, 8\(3\):277–292\.
- Chitale et al\. \(2025\)Pranjal A Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, and Sunayana Sitaram\. 2025\.UPDESH: synthesizing grounded instruction tuning data for 13 indic languages\.*arXiv preprint arXiv:2509\.21294*\.
- Choudhary et al\. \(2026\)Mukund Choudhary, Madhur Jindal, Gaurja Aeron, and Monojit Choudhury\. 2026\.[Do LLMs model human linguistic variation? a case study in Hindi\-English verb code\-mixing](https://doi.org/10.18653/v1/2026.findings-eacl.291)\.In*Findings of the Association for Computational Linguistics: EACL 2026*\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\. 2021\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*\.
- Cohen \(2013\)Jacob Cohen\. 2013\.*Statistical power analysis for the behavioral sciences*\.routledge\.
- Conneau et al\. \(2018\)Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R\. Bowman, Holger Schwenk, and Veselin Stoyanov\. 2018\.Xnli: Evaluating cross\-lingual sentence representations\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*\.
- Dawar et al\. \(2026\)Aviral Dawar, Roshan Karanth, Vikram Goyal, and Dhruv Kumar\. 2026\.IndicDB–benchmarking multilingual text\-to\-SQL capabilities in indian languages\.*arXiv preprint arXiv:2604\.13686*\.
- Deuchar \(2020\)Margaret Deuchar\. 2020\.Code\-switching in linguistics: A position paper\.*Languages*, 5\(2\):22\.
- Gala et al\. \(2024\)Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M Khapra, Raj Dabre, Rudra Murthy, Anoop Kunchukuttan, and 1 others\. 2024\.Airavata: Introducing hindi instruction\-tuned LLM\.*arXiv preprint arXiv:2401\.15006*\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others\. 2024\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*\.
- Gupta et al\. \(2024\)Ayushman Gupta, Akhil Bhogal, and Kripabandhu Ghosh\. 2024\.Code\-mixer ya nahi: Novel approaches to measuring multilingual llms’ code\-mixing capabilities\.*arXiv preprint arXiv:2410\.11079*\.
- Hu et al\. \(2022\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others\. 2022\.LoRA: Low\-rank adaptation of large language models\.*Iclr*, 1\(2\):3\.
- Jaavid et al\. \(2024\)J Jaavid, Raj Dabre, M Aswanth, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan\. 2024\.Romansetu: Efficiently unlocking multilingual capabilities of large language models via romanization\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Kavukcuoglu et al\. \(2026\)Koray Kavukcuoglu, Jeff Dean, Oriol Vinyals, and Noam Shazeer\. 2026\.[Gemini 3\.5: Frontier intelligence with action](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/#gemini-3-5-flash)\.Google Blog \(The Keyword\)\.Accessed: May 24, 2026\.
- Khan et al\. \(2024a\)Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad B, Varun G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M\. Khapra\. 2024a\.[IndicLLMSuite: A blueprint for creating pre\-training and fine\-tuning datasets for Indian languages](https://doi.org/10.18653/v1/2024.acl-long.843)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Khan et al\. \(2024b\)Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, Mitesh M Khapra, and 1 others\. 2024b\.Indicllmsuite: A blueprint for creating pre\-training and fine\-tuning datasets for indian languages\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Khanuja et al\. \(2020a\)Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury\. 2020a\.Gluecos: An evaluation benchmark for code\-switched nlp\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*\.
- Khanuja et al\. \(2020b\)Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury\. 2020b\.[GLUECoS: An evaluation benchmark for code\-switched NLP](https://doi.org/10.18653/v1/2020.acl-main.329)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*\.
- Kunchukuttan et al\. \(2020\)Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Avik Bhattacharyya, Mitesh M Khapra, Pratyush Kumar, and 1 others\. 2020\.Ai4bharat\-super corpus: Monolingual corpora and word embeddings for indic languages\.*arXiv preprint arXiv:2005\.00085*\.
- Kwan et al\. \(2024\)Wai\-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Yuxin Jiang, Lifeng Shang, Qun Liu, and Kam\-Fai Wong\. 2024\.[M4LE: A multi\-ability multi\-range multi\-task multi\-domain long\-context evaluation benchmark for large language models](https://doi.org/10.18653/v1/2024.acl-long.832)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Madhani et al\. \(2023a\)Yash Madhani, Mitesh M Khapra, and Anoop Kunchukuttan\. 2023a\.Bhasa\-abhijnaanam: Native\-script and romanized language identification for 22 indic languages\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*\.
- Madhani et al\. \(2023b\)Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul Nc, Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra\. 2023b\.Aksharantar: Open indic\-language transliteration datasets and models for the next billion users\.In*Findings of the association for computational linguistics: Emnlp 2023*\.
- McNemar \(1947\)Quinn McNemar\. 1947\.Note on the sampling error of the difference between correlated proportions or percentages\.*Psychometrika*, 12\(2\):153–157\.
- Mistral AI \(2026\)Mistral AI\. 2026\.[Models overview](https://docs.mistral.ai/models/overview)\.Mistral AI Documentation\.Accessed: May 24, 2026\.
- Myers\-Scotton \(1993\)Carol Myers\-Scotton\. 1993\.*Social motivations for codeswitching: Evidence from Africa*\.Oxford University Press\.
- OpenAI \(2026\)OpenAI\. 2026\.[GPT\-3\.5 Turbo model documentation](https://developers.openai.com/api/docs/models/gpt-3.5-turbo)\.OpenAI Developer Documentation\.Accessed: May 24, 2026\.
- Pattnayak and Chowdhuri \(2026\)Priyaranjan Pattnayak and Sanchari Chowdhuri\. 2026\.IndicSafe: a benchmark for evaluating multilingual LLM safety in south asia\.*arXiv preprint arXiv:2603\.17915*\.
- Poplack \(2001\)Shana Poplack\. 2001\.Code\-switching \(linguistic\)\.In Neil J\. Smelser and Paul B\. Baltes, editors,*International Encyclopedia of the Social and Behavioral Sciences*, pages 2062–2065\. Elsevier Science\.
- Rizvi et al\. \(2021\)Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Ganu, Monojit Choudhury, and Sunayana Sitaram\. 2021\.[GCM: A toolkit for generating synthetic code\-mixed text](https://doi.org/10.18653/v1/2021.eacl-demos.24)\.In*Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations*\.
- Rosner et al\. \(2006\)Bernard Rosner, Robert J Glynn, and Mei\-Ling T Lee\. 2006\.The wilcoxon signed rank test for paired comparisons of clustered data\.*Biometrics*, 62\(1\):185–192\.
- Sarvam AI \(2024\)Sarvam AI\. 2024\.Sarvam\-1\.[https://huggingface\.co/sarvamai/sarvam\-1](https://huggingface.co/sarvamai/sarvam-1)\.
- Sengupta et al\. \(2024\)Ayan Sengupta, Soham Das, Md Shad Akhtar, and Tanmoy Chakraborty\. 2024\.Social, economic, and demographic factors drive the emergence of hinglish code\-mixing on social media\.*Humanities and Social Sciences Communications*, 11\(1\):1–12\.
- Sravani and Mamidi \(2023\)Dama Sravani and Radhika Mamidi\. 2023\.[Enhancing code\-mixed text generation using synthetic data filtering in neural machine translation](https://doi.org/10.18653/v1/2023.conll-1.15)\.In*Proceedings of the 27th Conference on Computational Natural Language Learning \(CoNLL\)*\.
- Srivastava and Singh \(2021\)Vivek Srivastava and Mayank Singh\. 2021\.[Challenges and limitations with the metrics measuring the complexity of code\-mixed text](https://doi.org/10.18653/v1/2021.calcs-1.2)\.In*Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code\-Switching*\.
- Team et al\. \(2024\)Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others\. 2024\.Gemma: Open models based on gemini research and technology\.*arXiv preprint arXiv:2403\.08295*\.
- Wang et al\. \(2022\)Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, and 1 others\. 2022\.Super\-naturalinstructions: Generalization via declarative instructions on 1600\+ nlp tasks\.In*Proceedings of the 2022 conference on empirical methods in natural language processing*\.
- Winata et al\. \(2023\)Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio\. 2023\.[The decades progress on code\-switching research in NLP: A systematic survey on trends and challenges](https://doi.org/10.18653/v1/2023.findings-acl.185)\.In*Findings of the Association for Computational Linguistics: ACL 2023*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.
- Yang et al\. \(2026\)Qingyan Yang, Tongxi Wang, and Yunsheng Luo\. 2026\.ChiEngMixBench: Evaluating large language models on spontaneous and natural chinese\-english code\-mixed generation\.*arXiv preprint arXiv:2601\.16217*\.
- Yang and Chai \(2025a\)Yilun Yang and Yekun Chai\. 2025a\.[CodeMixBench: Evaluating code\-mixing capabilities of LLMs across 18 languages](https://doi.org/10.18653/v1/2025.emnlp-main.109)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*\.
- Yang and Chai \(2025b\)Yilun Yang and Yekun Chai\. 2025b\.Codemixbench: Evaluating code\-mixing capabilities of llms across 18 languages\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*\.

Table of Contents for Appendix

## Appendix ATask Details

### A\.1Data Description

Table[8](https://arxiv.org/html/2606.30790#A1.T8)reports the original number of instances drawn from each source dataset before any preprocessing, filtering, or code\-mixing\. These counts represent the full, unmodified benchmark splits as released by their respective authors\. All instances were retained to construct the code\-mixed and romanised variants described in Section[3\.1](https://arxiv.org/html/2606.30790#S3.SS1), yielding the training corpus used in our fine\-tuning ablation experiments\.

Table 8:Original instance counts per task source\. These figures reflect unmodified benchmark sizes prior to code\-mixing and romanization\. The full corpus of 32,077 instances forms the basis of the fine\-tuning data used in the ablation study\.The reasoning dimension comprises three tasks, two drawn from Super\-NaturalInstructions \(SuperNI\)222[https://github\.com/allenai/natural\-instructions/tree/master/tasks](https://github.com/allenai/natural-instructions/tree/master/tasks)Wang et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib41)\)and GSM8K333[https://huggingface\.co/datasets/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k)Cobbe et al\. \([2021](https://arxiv.org/html/2606.30790#bib.bib9)\)\.\(i\) Word Analogy\(SuperNI tasks 1152–1159\) requires identifying semantic relationships between concept pairs with structured explanations, probing lexical and relational reasoning under CM framing\.\(ii\) Question Decomposition\(SuperNI tasks 168, 176, 184\) requires decomposing complex questions into sub\-questions across three subtypes, factual, procedural, and inverse synthesis, and testing whether models maintain logical step\-reference chains under CM\-ed instructions\.\(iii\) Mathematical Reasoning\(GSM8K\) presents grade\-school arithmetic problems requiring multi\-step chain\-of\-thought, stratified by step count; this task isolates the effect of framing code\-mixing in explanations and/or reasoning independently of content\.

The language understanding dimension covers three Indic NLP tasks sourced from the AI4Bharat IndicXtreme collection\.444[https://huggingface\.co/collections/ai4bharat/indicxtreme](https://huggingface.co/collections/ai4bharat/indicxtreme)\(iv\) Paraphrase Detection\(IndicXParaphrase555[https://huggingface\.co/datasets/ai4bharat/IndicXParaphrase](https://huggingface.co/datasets/ai4bharat/IndicXParaphrase)Gala et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib14)\)\) presents balanced sentence pairs requiring binary semantic equivalence judgements\.\(v\) Sentiment Analysis\(IndicSentiment666[https://huggingface\.co/datasets/ai4bharat/IndicSentiment](https://huggingface.co/datasets/ai4bharat/IndicSentiment)Gala et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib14)\)\) provides three\-class product review classification\. Sentiment\-bearing vocabulary is among the most naturally code\-mixed in South Asian digital communication, creating a direct interaction between task content and CM register\. And,\(vi\) Natural Language Inferenceuses IndicXNLI777[https://huggingface\.co/datasets/Divyanshu/indicxnli](https://huggingface.co/datasets/Divyanshu/indicxnli)Aggarwal et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib1)\)evaluates entailment, neutrality, and contradiction across premise\-hypothesis pairs and is highly sensitive to CM due to the complexity of cross\-sentence reasoning\.

Finally, for the\(vii\) Toxicity Detectiontask, instances are drawn from IndicAlign\-Toxic888[https://huggingface\.co/datasets/ai4bharat/indic\-align/viewer/Toxic\_Matrix](https://huggingface.co/datasets/ai4bharat/indic-align/viewer/Toxic_Matrix)and supplemented with benign samples from IndicAlign\-Instruct to ensure label balanceKhan et al\. \([2024b](https://arxiv.org/html/2606.30790#bib.bib21)\)\. This task evaluates whether models correctly identify and refuse harmful content in CM\-ed register\. Despite CM\-ing being widely exploited to evade content moderationChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\), no existing benchmark on CM\-ing includes this task\.

#### Languages

We experiment with romanized Indic\-English CM\-ed registers spanning two major Indic language families: from the Indo\-Aryan branch, we include Hindi \(Hi\-En, ˜610M speakers\), Bengali \(Bn\-En, ˜273M speakers\), and Gujarati \(Gu\-En, ˜55M speakers\); and from the Dravidian family, we include Tamil \(Ta\-En, ˜87M speakers\)\. The choice of these four languages cover the largest user populations in India and represent diverse resource levels, linguistic structures, and code\-mixing phenomena\. All four languages predominantly follow Subject\-Object\-Verb \(SOV\) word order, but differ substantially in morphology and syntactic structure\. Indo\-Aryan languages display relatively fusional morphology, whereas Tamil exhibits more agglutinative characteristics, often leading to distinct constituent\-order and lexical\-mixing patterns in code\-mixed discourse\. Prior studies have also identified extensive verb\-level and intra\-sentential mixing phenomena in Indic\-English bilingual communicationChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\); Jaavid et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib18)\)\.

## Appendix BCM\-ing Framework Development

### B\.1Code Mixing Framework

Bilingual speakers preferentially switch lexical categories while preserving syntactic compatibility across languages \(Equivalence Constraint theoryPoplack \([2001](https://arxiv.org/html/2606.30790#bib.bib33)\)and the Matrix Language Frame theoryMyers\-Scotton \([1993](https://arxiv.org/html/2606.30790#bib.bib30)\)\)\. Based on these principles, we build a linguistically grounded framework for generating controlled Indic\-EnglishRomanizedCode\-Mix \(RCM\)\-ed instructions\. This enables systematic variation of code\-mixing intensity while preserving semantic consistency across tasks and language pairs\. Following sections describe the framework design step by step\.

\(I\) English to Indic Translation\.The SuperNI\-based tasks Word Analogy and Question DecompositionWang et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib41)\)along with the GSM8K mathematical reasoning taskCobbe et al\. \([2021](https://arxiv.org/html/2606.30790#bib.bib9)\)are originally available only in English and lack coverage for Indic languages\. To construct high\-quality multilingual source instances prior to code\-mixing, we first translate these English\-only task instances into the target Indic languages \- Hindi, Bengali, Gujarati, and Tamil\.

To ensure semantic fidelity while maintaining natural conversational phrasing, we adopt a two\-step translation pipeline combining machine translation and manual verification\. We use Google Translate’s Python library999https://pypi\.org/project/googletrans/for initial translation generation and refine outputs using Anthropic Claude Sonnet 4\.6101010https://www\.anthropic\.com/claude/sonnet\. Manual validation of labels, decomposition structures, and logical relations remain unchanged after translation\. This two\-step refinement helps verify linguistic naturalness, correct SOV constituent order, and preservation of task\-critical structural markers\. For all tasks, tokens like mathematical expressions, step labels, proper nouns, etc\., are identified and excluded from the translation scope\.

For the remaining tasks sourced from multilingual Indic datasets, like IndicXNLIAggarwal et al\. \([2022](https://arxiv.org/html/2606.30790#bib.bib1)\), IndicAlign\-ToxicChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\), IndicXParaphrase, and IndicSentimentGala et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib14)\), native\-script Indic translations are sourced directly from the original datasets, that provide professionally verified translations across Bengali, Hindi, Gujarati, and Tamil\. Additionally, Tamil translations were absent for IndicXParaphrase taskKhan et al\. \([2024b](https://arxiv.org/html/2606.30790#bib.bib21)\)and were therefore newly created as part of this work\. The corresponding English language instances for IndicXNLI are extracted from Meta’s XNLI corpus111111[https://huggingface\.co/datasets/facebook/xnli](https://huggingface.co/datasets/facebook/xnli)Conneau et al\. \([2018](https://arxiv.org/html/2606.30790#bib.bib11)\)\.

\(II\) Word\-Selection Taxonomy\.To ensure CM substitutions reflect natural bilingual switching behavior rather than arbitrary replacement, we assign each switchable word or phrase in the English source text to one of five categories based on its linguistic role, semantic function, and documented naturalness as a CM switch point in bilingual corporaPoplack \([2001](https://arxiv.org/html/2606.30790#bib.bib33)\); Winata et al\. \([2023](https://arxiv.org/html/2606.30790#bib.bib42)\)\. Each substitution is logged with its category label, Indic replacement, and a linguistic rationale, producing structured annotation records that enable category\-frequency analysis across tasks and language pairs\. Table[9](https://arxiv.org/html/2606.30790#A2.T9)summarizes the five categories, their switching policies, and illustrative examples across languages\.

Table 9:Linguistically grounded word\-selection taxonomy used for controlled Romanized Indic\-English code\-mixing generation\.\(III\) Code\-mixing Intensity Levels\.Three intensity levels \( 25%, 50% and 75%\) are generated by activating different category subsets, creating a controlled progression from low to high code\-mixing\. We compute Code\-Mixing Index \(CMI\) at the token level as the proportion of English tokens replaced by Indic equivalentsSrivastava and Singh \([2021](https://arxiv.org/html/2606.30790#bib.bib39)\)121212CAT\-0 tokens excluded from the denominator as their invariance is linguistically motivated rather than a pipeline artifact\.\. Controlled code\-mixed substitutions are generated using taxonomy\-guided prompting with Claude Sonnet 4\.6 and manually verified for semantic consistency\. Each substitution is annotated with its category, Indic replacement, and linguistic rationale, producing structured records that support downstream analysis of substitution patterns across tasks, languages, and code\-mixing intensities\. At 25%\-CM, only domain\-specific terms are replaced while English sentence structure is preserved\. At 75%\-CM, all categories are active, producing dense Romanized code\-mixed text with a predominantly Indic sentence structure\. The three intensity levels with their active category subsets and target CMI ranges, the prompt for generating code\-mixed substitutions, and the handling of substitution edge cases are provided below\.

\(IV\) Romanization of Indic Script\.Following CM substitutions, all Indic\-script instances are converted to their Latin script to produce the final romanized and code\-mixed output, reflecting the dominant writing behavior of multilingual South Asian users on smartphones and messaging platforms\. We extract end\-to\-end Latin\-script translations from IndicLLMSuiteKhan et al\. \([2024b](https://arxiv.org/html/2606.30790#bib.bib21)\)for the tasks available\. Additionally, we use two Python libraries Aksharantar131313https://github\.com/AI4Bharat/IndicXlitMadhani et al\. \([2023b](https://arxiv.org/html/2606.30790#bib.bib27)\), a large\-scale AI4Bharat transliteration model covering 21 Indic languages and Aksharamukha141414https://pypi\.org/project/aksharamukha/, another open\-source transliteration tool that converts text across multiple Indic scripts while preserving phonetic consistency\. We intentionally retain natural transliteration variability rather than enforcing strict phoneme\-level normalization, better reflecting real\-world romanized CM usage and the orthographic variation encountered by deployed multilingual LLMs\.

### B\.2Prompt for Output Refinement

The following prompt was used to guide Claude Sonnet 4\.6 in refining machine\-translated outputs from Google Translate into semantically faithful and linguistically natural Indic\-language instances\.

Prompt: English\-to\-Indic Translation RefinementSystem:You are an expert translator specialising in Hindi, Bengali, Gujarati, and Tamil\. Your task is to refine machine\-translated text to produce accurate, fluent, and contextually appropriate translations that reflect natural conversational phrasing\.Instruction:You are given an English source sentence and its machine\-translated output in\{target\_language\}\. Refine the machine translation according to the following guidelines:Guidelines:1\.Preserve task structure:Do not alter step labels \(e\.g\.,\#1,\#2,Step 1:\), mathematical expressions \(e\.g\.,<<2\*8=16\>\>\), and numerical values\. These must appear exactly as in the source\.2\.Do not translate:Proper nouns \(names of people, places, organisations\), domain\-specific acronyms, and quoted strings\. Retain them verbatim from the English source\.3\.Ensure grammatical correctness:Verify subject\-object\-verb \(SOV\) constituent order appropriate to\{target\_language\}and correct any morphological or agreement errors introduced by machine translation\.4\.Maintain semantic fidelity:The refined translation must preserve the full meaning of the source\. Do not add, omit, or paraphrase any content beyond what is needed for fluency\.5\.Natural phrasing:The output should read as natural, conversational\{target\_language\}rather than a literal word\-for\-word rendering\.6\.Label and logic preservation:For classification and reasoning tasks, logical relations \(entailment, contradiction, neutrality\), sentiment polarities, and analogy structures must remain unchanged after translation\.Input:•English source:\{english\_source\}•Machine translation \(\{target\_language\}\):\{mt\_output\}Output:Return only the refined translation in\{target\_language\}\. Do not include any explanation, commentary, or the original English text\.

### B\.3Prompt for Code\-Mixed Substitutions

The following prompt was used to guide Claude Sonnet 4\.6 in generating taxonomy\-guided, annotated code\-mixed substitutions at three controlled intensity levels across all four Indic languages\.

Prompt: Taxonomy\-Guided Code\-Mixed Substitution GenerationSystem:You are an expert computational linguist\. Your task is to generate controlled code\-mixed substitutions from English source text at a specified intensity level, following a predefined substitution taxonomy\.Instruction:Given an English source sentence and its full Indic translation in\{target\_language\}, produce a code\-mixed output at intensity level\{cm\_level\}\(25%, 50%, or 75%\) by substituting a proportion of English tokens with their\{target\_language\}equivalents according to the category guidelines below\.Substitution Taxonomy:•CAT\-0:Invariant tokens like Mathematical expressions, step labels \(e\.g\.,\#1,Step 1:\), proper nouns, acronyms, and quoted strings\.•CAT\-A:Structural markers like Discourse connectors, sentence\-initial conjunctions, subordinators \(e\.g\.,so, but, because, when\)\.•CAT\-B:Functional words like Determiners, copulas, auxiliaries, negations \(e\.g\.,the, is, not, a\)\.•CAT\-C:Content words like High\-frequency nouns, verbs, adjectives central to the task meaning \(e\.g\.,office, time, said, woman\)\.•CAT\-D:Domain\-specific terms like Task\-critical terminology, reasoning markers, and domain nouns \(e\.g\.,decomposition, entailment, step, result\)\.Intensity Level Guidelines:•25%:Activate CAT\-A and CAT\-B only; substitute 1–3 tokens per sentence\.•50%:Activate CAT\-A, CAT\-B, and CAT\-C; substitute 3–6 tokens per sentence\.•75%:Activate all categories \(CAT\-A through CAT\-D\); substitute 6–10 tokens per sentence\.Annotation Requirement:For each substitution made, provide a structured annotation in the\{target\_language\}rationale field, recording the English source token, its Indic replacement, assigned category, and a brief linguistic justification, alongside the sentence\-level CMI scores for the premise and hypothesis\.\[english\_token → indic\_token \| CAT\-X: linguistic rationale\]; …\|\|CMI\_premise=\{score\} CMI\_hypothesis=\{score\}Input:•English source:\{english\_source\}•Indic translation \(\{target\_language\}\):\{indic\_translation\}•Target intensity:\{cm\_level\}Output:Return ONLY the code\-mixed sentence in\{target\_language\}followed by the structured<language\>\_rationaleannotation\.

### B\.4Code\-mixing Intensity Levels and Target Ranges

The three intensity levels with their active category subsets and target CMI ranges are reported in Table[10](https://arxiv.org/html/2606.30790#A2.T10)\.

Table 10:Code\-mixing \(CM\) intensity levels and target ranges of Code\-Mixing Index \(CMI\)\. CAT = Categories
### B\.5Handling Task\-specific Edge Cases

A single universal taxonomy cannot be applied blindly across tasks with fundamentally different structural requirements\. Without task\-specific adaptations, generation of code\-mixed \(CM\-ed\) instances would corrupt the very tokens that define task validity, e\.g\. switching a mathematical operator changes the answer, missing a step label can break the reference chain in explanations, improper translation of a toxic prompt can change the contextual meaning\. Therefore these adaptations are not exceptions to the framework but special treatment of certain tokens while creating the code\-mixed dataset for each task based on the structural contexts\. Each selected task require specific adaptations of the general taxonomy to preserve task validity under CM\-ing:\(i\) Word Analogy:Analogy sub\-type labels \(causation, affordance, containment\) are assigned to CAT\-C, as these are the most natural L1 vocabulary151515Refers to the vocabulary of a speaker’s first language, i\.e\. their native or dominant language\.choices when naming abstract relational categories\. Proper nouns within analogy pairs are treated as CAT\-0 regardless of whether L1 equivalents exist, since swapping them would alter the relational content of the pair\.

\(ii\) GSM8K:Mathematical tokens \(<<calculation\>\>,\#\#\#\#, numerals, and operators\) are strict CAT\-0\. Only natural language framing around arithmetic steps is eligible for CM substitution\. This makes GSM8K the task with the lowest achievable CMI across all CM levels — a linguistically meaningful property that isolates the effect of CM framing on reasoning from the effect of CM on content\.

\(iii\) Question Decomposition:Step labels \(Step1:,\#1,Wikipedia page for step N:\) are CAT\-0 structural markers whose integrity is essential to the step\-reference chain\. Operational verbs \(return,filter,retrieve\) are assigned to CAT\-C, while discourse connectives linking steps are CAT\-B\.

\(iv\) Toxicity Detection:The input toxic prompt is never code\-mixed, preserving the original harmful content for accurate detection by the model\. Only the output classification label and refusal explanation are subjected to CM\. This design creates a distinctive evaluation condition: does the model correctly detect and refuse toxic content when the surrounding discourse, i\.e\. the instruction and explanation, is expressed in CM register?

### B\.6Baseline Dataset Creation details

We also randomly draw 100 samples from the ToxicMatrix original dataset of 90\.3KChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\)\. However, for the word analogy from Super\-NI, the source pool \(tasks 1152–1159\) contained only 48 positive and negative instances with validated explanations; we supplement these with 52 manually constructed analogy pairs, balanced across positive \(correct analogy\) and negative \(incorrect analogy\) labels and covering the same relation types, causation, affordance, containment, and functional similarity as the original instances, yielding a final set of 100 instances consistent with all other tasks\.

Table 11:Detailed benchmark validation results per task, language, and CM intensity level\. ChrF scores measure translation fidelity \(character n\-gram overlap between 50%\-CM Romanised output and English source text\)\. Cohen’sκ\\kappameasures pairwise naturalness agreement between two independent native\-speaker annotators at each CM level\. N/A = ratings unavailable for Hindi in Question Decomposition\.
### B\.7Indi\-RomCoMContent Analysis

Toxicity exhibits the highest CMI at every level \(μ75=0\.981\\mu\_\{75\}=0\.981\), driven by high\-frequency CAT\-C task\-label vocabulary, while GSM8K consistently shows the lowest \(μ75=0\.152\\mu\_\{75\}=0\.152\) due to the high proportion of CAT\-0 protected tokens\. Question Decomposition shows the steepest level\-to\-level growth \(Δ=0\.186\\Delta=0\.186from 25% to 75%\), driven by CAT\-E full\-clause substitutions at 75%\-CM\. Figure[6](https://arxiv.org/html/2606.30790#A2.F6)shows the full CMI distributions as violin plots\.

![Refer to caption](https://arxiv.org/html/2606.30790v1/x4.png)Figure 6:CMI distributions across tasks at each CM intensity level \(violin plots\)\. Dashed lines at CMI=0\.35=0\.35and0\.600\.60delimit the target mid\-intensity band\. All pairwise CM\-level differences significant atp<10−100p<10^\{\-100\}\(Wilcoxon signed\-rank, Bonferroni\-corrected\)\.Across 700 instances, four languages, and three CM intensity levels, the rationale annotation records 33,632 structured substitution logs\. At 25%\-CM, all 6,925 substitutions are CAT\-C \(domain/task terminology\) across every task without exception, confirming that the taxonomy’s single\-category activation at low intensity holds empirically\. Figures[3](https://arxiv.org/html/2606.30790#S3.F3)a–b show the proportional substitution breakdown at 50%\-CM and 75%\-CM respectively, where multi\-category activation produces interpretable task\-level variation\. At50%\-CM\(Figure[3](https://arxiv.org/html/2606.30790#S3.F3)a\), CAT\-C reduces to 52\.6% of all substitutions as CAT\-B \(logical connectives, 29\.1%\) and CAT\-D \(content domain nouns, 18\.3%\) are activated\. Task\-level variation is pronounced: CAT\-B dominates for NLI \(66\.1%\) and Paraphrase \(54\.7%\), where logical connectives are syntactically dense in premise–hypothesis pairs and sentence\-pair comparisons respectively, while CAT\-D is proportionally larger for GSM8K \(25\.8%\) and Toxicity, where mathematical and safety domain nouns constitute the primary switchable content\. At75%\-CM\(Figure[3](https://arxiv.org/html/2606.30790#S3.F3)b\), CAT\-A \(function words, 5\.6%\) and CAT\-E \(full predicate phrases, 6\.8%\) appear for the first time, completing the Indic sentence frame\. CAT\-E is most prominent in Toxicity \(11\.4%\), where the refusal openerI cannot give a responseis the primary full\-predicate switch, and in Question Decomposition \(6\.6%\), where step\-chain predicates are switched as units\. CAT\-A is largest in Question Decomposition \(12\.5%\), reflecting the high density of article\-bearing structural phrases in decomposition step sequences\. These patterns confirm that the word\-selection taxonomy produces category activations that are both systematic across tasks and linguistically motivated within each task domain\.

## Appendix CEvaluation Details

### C\.1Models

ModelParametersCategoryIndic TrainingCM TrainingAccessProprietaryGPT\-3\.5\-Turbo∼\\sim175BCommercialIncidentalNoneOpenAI APIGemini\-3\.5\-Flash—CommercialIncidentalNoneGoogle AI APIClaude Opus\-4\.6—CommercialIncidentalNoneAnthropic APIOpen\-weight — generalLLaMA\-3\.2\-3B3BOpen\-weightIncidentalNoneHuggingFaceLLaMA\-3\.1\-8B8BOpen\-weightIncidentalNoneHuggingFaceLLaMA\-3\.1\-70B70BOpen\-weightIncidentalNoneTogether AIMistral\-7B\-v0\.37BOpen\-weightIncidentalNoneHuggingFaceQwen2\.5\-1\.5B1\.5BOpen\-weightMultilingualNoneHuggingFaceQwen2\.5\-3B3BOpen\-weightMultilingualNoneHuggingFaceQwen2\.5\-7B7BOpen\-weightMultilingualNoneHuggingFaceQwen2\.5\-14B14BOpen\-weightMultilingualNoneHuggingFaceQwen2\.5\-32B32BOpen\-weightMultilingualNoneTogether AIGemma\-2\-7B7BOpen\-weightIncidentalNoneHuggingFaceIndic monolingualAiravata7BOpen\-weightHindi \(IndicLLMSuite\)NoneHuggingFaceTamilLLaMA7BOpen\-weightTamilNoneHuggingFaceBengaliLLaMA7BOpen\-weightBengaliNoneHuggingFaceFine\-tuned ablation \(Indi\-RomCoMtraining split\)LLaMA\-3\.2\-3BCM\{\}\_\{\\text\{CM\}\}3BFine\-tunedIncidentalIndi\-RomCoMtrainHuggingFaceLLaMA\-3\.1\-8BCM\{\}\_\{\\text\{CM\}\}8BFine\-tunedIncidentalIndi\-RomCoMtrainHuggingFaceGemma\-2\-7BCM\{\}\_\{\\text\{CM\}\}7BFine\-tunedIncidentalIndi\-RomCoMtrainHuggingFace

Table 12:Models evaluated onIndi\-RomCoM\. Proprietary models are accessed via their respective APIs; open\-weight models below 32B are run locally on a single A100 80GB GPU\. “Incidental” Indic training denotes exposure during pretraining without deliberate Indic\-script curation\. Gujarati monolingual LLMs are absent because no publicly available instruction\-tuned Gujarati model exists at evaluation time—itself a finding that motivates this benchmark\.We evaluate models spanning four categories: proprietary commercial systems, open\-weight general\-purpose models, open\-weight models with Indic\-focused pretraining, and fine\-tuned ablation variants\. This design enables systematic comparison across scale, Indic\-script exposure, and CM\-specific training\. Table[12](https://arxiv.org/html/2606.30790#A3.T12)summarises the full model set\.

#### Fine\-tuning ablation\.

Our central hypothesis is that Romanized code\-mixing \(RCM\) represents a systematic challenge that cannot be resolved through scale or Indic pretraining alone\. To test this, we fine\-tune three models—LLaMA\-3\.2\-3B, LLaMA\-3\.1\-8B, and Gemma\-2\-7B—on theIndi\-RomCoMtraining split, comprising 395,076 instances across 7 tasks, 4 languages, and 3 CM intensity levels \(80% of instances per task stratified by CM level\)\. We denote these fine\-tuned variants with the subscriptCM\.

All three models are fine\-tuned using LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2606.30790#bib.bib17)\)with rankr=16r=16,α=32\\alpha=32, batch size 8, learning rate1×10−41\\times 10^\{\-4\}, a cosine scheduler with 3% warmup, and 3 training epochs on a single A100 80GB GPU\. Hyperparameters followChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\)\. If a fine\-tuned variant outperforms its untuned counterpart despite training on a relatively small corpus, this constitutes direct evidence that CM\-specific supervision is necessary and that existing general Indic resources do not cover the RCM register\.

### C\.2Prompting Protocol

#### Conditions\.

All models are evaluated under two conditions:Zero\-shot \(ZS\): the task instruction and instance are provided in the target register \(English / 25%\-CM / 50%\-CM / 75%\-CM\) with no demonstrations\.3\-shot \(FS\-3\): three demonstrations in the same CM register as the query are prepended, drawn from held\-out instances stratified to cover all label classes\.

#### Template structure\.

Every prompt follows a four\-part template: \(i\) a brief task definition in the target register; \(ii\) the instance content; \(iii\) a structured output specification; \(iv\) a register\-maintenance instruction:*“Please respond in the same language and script as the input\.”*Pilot experiments confirm that without clause \(iv\), all evaluated models default to English regardless of CM input register, making it impossible to distinguish genuine register defection from instruction\-following failure\.

#### Task\-specific prompts\.

Table[5](https://arxiv.org/html/2606.30790#S4.T5)show the prompt template for each task\.\[INPUT\]denotes the instance field and\[DEMO\_kk\]denotes thekk\-th few\-shot demonstration; both are filled with the romanised CM text at the target intensity level\. The same template is used for zero\-shot \(omitting the demonstration block\) and few\-shot \(including three demonstrations before the query\)\.

### C\.3Evaluation Metrics

We employ four complementary metrics capturing task correctness, register fidelity, performance degradation, and tokeniser\-level representational capacity\.

#### \(i\) Task Accuracy \(Acc\)\.

For the four classification tasks \(Sentiment, NLI, Paraphrase, Toxicity\), accuracy is the proportion of predictions that exactly match the gold labelyiy\_\{i\}overNNinstances:

Acc=1N​∑i=1N𝟙​\[y^i=yi\]\.\\text\{Acc\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathds\{1\}\[\\hat\{y\}\_\{i\}=y\_\{i\}\]\.\(1\)For the three generation tasks \(GSM8K, Question Decomposition, Word Analogy\), we use task\-specific criteria: final\-answer exact match for GSM8K; binary correctness against gold decomposition structures for Question Decomposition; and a structured semantic\-relationship rubric for Word Analogy\. For generation tasks, a GPT\-4o judge evaluates responses on a 3\-point Likert scaleChitale et al\. \([2025](https://arxiv.org/html/2606.30790#bib.bib7)\); we report human–LLM agreement on a held\-out subset\. Higher accuracy is better; a decrease from the English baseline to any CM condition is direct evidence of performance degradation\.

#### \(ii\) Register Defection Rate \(RDR\)\.

RDR measures the proportion of responses written in English when the input is in CM register, capturing a failure mode orthogonal to task accuracy\. LetL​\(t\)L\(t\)be the language of tokenttidentified by IndicLIDMadhani et al\. \([2023a](https://arxiv.org/html/2606.30790#bib.bib26)\); a responserir\_\{i\}is*defective*\(di=1d\_\{i\}=1\) if more than 70% of its content tokens are identified as English:

RDR=1N​∑i=1Ndi,\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}d\_\{i\},\(2\)di\\displaystyle d\_\{i\}=𝟙​\[\|\{t∈ri:L​\(t\)=en\}\|\|ri\|\>0\.70\]\\displaystyle=\\mathds\{1\}\\left\[\\frac\{\|\\\{t\\in r\_\{i\}:L\(t\)=\\textsc\{en\}\\\}\|\}\{\|r\_\{i\}\|\}\>0\.70\\right\]RDR is computed separately for label tokens and explanation tokens, since a model may correctly emit a CM label while defecting to English in its explanation—a partial defection mode that full\-response identification would miss\. Lower RDR is better; RDR=0=0indicates perfect register maintenance\.

#### \(iii\) Vocabulary Coverage Rate \(VCR\)\.

VCR measures the proportion of unique switched Indic tokens inIndi\-RomCoMthat appear as complete \(non\-fragmented\) entries in modelmm’s tokeniser:

VCRm,ℓ=\|\{t∈𝒱CM:t∈𝒱m\}\|\|𝒱CM\|,\\text\{VCR\}\_\{m,\\ell\}=\\frac\{\|\\\{t\\in\\mathcal\{V\}\_\{\\text\{CM\}\}:t\\in\\mathcal\{V\}\_\{m\}\\\}\|\}\{\|\\mathcal\{V\}\_\{\\text\{CM\}\}\|\},\(3\)where𝒱CM\\mathcal\{V\}\_\{\\text\{CM\}\}is the set of unique Indic tokens introduced by CM substitution and𝒱m\\mathcal\{V\}\_\{m\}is the tokeniser vocabulary of modelmm\. A low VCR implies that common Indic tokens are represented as subword fragments, hypothesised to be negatively correlated with RDRJaavid et al\. \([2024](https://arxiv.org/html/2606.30790#bib.bib18)\): models that cannot represent Indic tokens as complete units are less likely to generate them faithfully\. Hence, higher VCR is better\.

Table 13:Average task accuracy \(%\) across 7 tasks in3\-shotsetting, averaged over 4 languages \(±\\pmSD across tasks\)\. BL = English baseline\. PG = Performance Gap \(BL−\-CM level; positive = accuracy drop\)\. CM = Code Mixing\.Bold= best BL per family\. Values are approximately 7–8% higher than the zero\-shot results in Table[6](https://arxiv.org/html/2606.30790#S5.T6), consistent with the register\-anchoring effect of in\-context CM demonstrations\.ModelLanguageOverallExact Match \(%\) by TaskEMDefectionGSM8kIndParIndSentXNLIQDecLLaMA\-3\.2\-3B \(FT\)All86\.386\.399\.999\.998\.598\.599\.799\.752\.652\.699\.299\.2100\.0100\.0Bengali99\.699\.6100\.0100\.0—————Gujarati72\.772\.799\.799\.7—————Hindi70\.170\.1100\.0100\.0—————Tamil99\.399\.3100\.0100\.0—————LLaMA\-3\.1\-8B \(FT\)All86\.486\.499\.999\.998\.798\.7100\.0100\.068\.468\.499\.699\.6100\.0100\.0Bengali99\.799\.7100\.0100\.0—————Gujarati72\.972\.999\.799\.7—————Hindi70\.370\.3100\.0100\.0—————Tamil99\.499\.4100\.0100\.0—————

Table 14:Fine\-tuning ablation results for LLaMA\-3\.2\-3B and LLaMA\-3\.1\-8B using QLoRA on 50%\-CM Romanised instruction pairs \(all four languages, 7 tasks; 90%/10% train/eval split,neval=14,848n\_\{\\text\{eval\}\}=14\{,\}848\)\. EM = Exact Match \(%\); Defection = proportion of outputs classified as English\-register \(\>70% ASCII tokens\); for Roman\-script CM targets this is expected to be high and does not indicate generation failure\. Task columns show aggregate EM across languages\. IndPar = IndicParaphrase; IndSent = IndicSentiment; XNLI = IndicXNLI; QDec = QuestionDecomposition\.
#### \(iv\) Performance Gap \(PG\)\.

PG quantifies the absolute accuracy degradation from the English baseline to each CM condition:PGl=Acc​\(English\)−Acc​\(CMl\),l∈\{25%,50%,75%\}\\text\{PG\}\_\{l\}=\\text\{Acc\}\(\\text\{English\}\)\-\\text\{Acc\}\(\\text\{CM\}\_\{l\}\),\\quad l\\in\\\{25\\%,50\\%,75\\%\\\}A positive PG at all 3 levels confirms that models degrade on CM input; a monotonically increasing PG \(PG25<PG50<PG75\\text\{PG\}\_\{25\}<\\text\{PG\}\_\{50\}<\\text\{PG\}\_\{75\}\) additionally confirms that degradation scales with CM intensity\. Task\-level variation in PG tests whether difficulty is task\-specific; model\-level variation tests whether scale or Indic pretraining confers robustness\.

#### Statistical analysis\.

We assess significance using the Wilcoxon signed\-rank testRosner et al\. \([2006](https://arxiv.org/html/2606.30790#bib.bib35)\)\(paired, non\-parametric\), comparing per\-instance accuracy between the English baseline and each CM condition, with Bonferroni correction across all 21 condition pairs \(7​tasks×3​CM levels7\\penalty 10000\\ \\text\{tasks\}\\times 3\\penalty 10000\\ \\text\{CM levels\}\)\. Effect sizes are reported as Cohen’sddCohen \([2013](https://arxiv.org/html/2606.30790#bib.bib10)\)on paired accuracy differences\. For the fine\-tuning ablation, we additionally apply McNemar’s testMcNemar \([1947](https://arxiv.org/html/2606.30790#bib.bib28)\)on paired binary accuracy vectors to assess whether CM\-specific training produces a significant improvement, and Spearman’sρ\\rhoto quantify the hypothesised negative correlation between VCR and RDR across model–language pairs\.

## Appendix DAdditional Results

### D\.1Task Accuracy with Few\-shot Prompting

This section details the task\-by\-task performance distributions under few\-shot prompting conditions across different language varieties and intensity tiers\. In our main evaluation setup, adding three contextual demonstrations explicitly shows models how to preserve structural outputs and language constraints\. While this historical scaffolding improves overall target output compliance, its effectiveness varies significantly depending on the core reasoning demands of individual tasks\.

A comprehensive breakdown of accuracy patterns across all evaluation settings is documented in Table[13](https://arxiv.org/html/2606.30790#A3.T13)\. For simpler categorical classifications like sentiment analysis and paraphrase detection, few\-shot conditioning provides immediate stability\. The historical context helps models overcome initial script confusion and focus on tracking semantic tokens cleanly\. However, for complex tasks requiring long\-form text tracking, such as mathematical reasoning and question decomposition, the additional context offers limited protection against increasing code\-mixing density\. Full tracking data in the table confirms that while few\-shot prompts stabilize basic formatting compliance, they do not resolve the fundamental linguistic challenges posed by dense romanized blending\.

### D\.2Finetuning Ablation Study

Our fine\-tuning ablation experiments evaluate whether the difficulties posed by romanized code\-mixing can be solved through targeted supervision rather than simple parameter scaling\. We isolate performance changes across our three specialized models by training them on a localized split encompassing balanced language and intensity distributions\. This comparative strategy tests the core hypothesis that standard multilingual pretraining configurations fail to expose models to natural romanized registers\.

The exact numerical improvements and comparative evaluation results for all fine\-tuned configurations are provided in Table[14](https://arxiv.org/html/2606.30790#A3.T14)\. The experimental outcomes confirm that explicit supervision yields substantial performance improvements across all seven instruction\-following tasks\. These tuned variants consistently outperform their larger general\-purpose baselines despite being trained on a relatively compact corpus\. This distinct performance gap validates the necessity of register\-specific training resources, demonstrating that specialized fine\-tuning establishes critical linguistic boundaries that standard language scaling cannot replicate alone\.