Multilingual GSM-Symbolic: What determines capability transfer across languages?

Hugging Face Daily Papers Papers
cross-lingual-transfer multilingual-evaluation mathematical-reasoning llm-benchmark model-scaling low-resource-languages symbolic-templates

Summary

The paper introduces Multilingual GSM-Symbolic, a symbolic-template dataset covering 15 languages and 30k item-matched QA pairs, to quantify what determines cross-lingual capability transfer in LLMs. It finds model size, language resource level, reasoning, and typological distance are key drivers, and shows a framework explaining 92% of between-language variation while predicting unseen-language performance within 6.0pp.

We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datasets and rarely examine its determinants jointly. Identifying what predicts transfer would let us avoid exhaustive evaluation across all language pairs and let developers target the factors that limit performance in low-resource languages. To evaluate cross-lingual capability transfer, we introduce Multilingual GSM-Symbolic, an extensible multilingual mathematical dataset covering 30,000 item-matched question-answer pairs and spanning 15 languages. It utilises symbolic templates to prevent overfitting and ensure generalisation by allowing generation of millions of high-quality variations from a single sample. Using Multilingual GSM-Symbolic, we quantify the largest determinants of capability as model size (β= 1.77), language resource level (β= 0.77), reasoning (β= 0.67) and typological distance (β= -0.25). This joint estimation allows these determinants to be expressed in terms of one another: a 32B model evaluated in Marathi performs like a 10B model in English. Our findings have important implications for model developers, showing that model size and reasoning narrow the performance gap between low- and high-resource languages (β= -0.27 and β= -0.20, respectively), while similar levers have little or no effect on typologically distant languages. Overall, our analysis framework explains 92% of between-language variation, but only 23% of the model-by-language variation, and predicts a model's performance on an unseen language within 6.0pp (r=.96). Incorporating measurements from just 10 templates in the target language reduces this to 4.19pp, enabling reasonable estimates of performance with little or no downstream dataset.
Original Article
View Cached Full Text

Cached at: 10/05/26, 12:47 PM

Paper page - Multilingual GSM-Symbolic: What determines capability transfer across languages?

Source: https://huggingface.co/papers/2610.03367 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Weunderstandlittleabouthowcapabilitiesacquiredinonelanguagecarryovertoanother,orwhatgovernsthistransfer:evaluationsrelyonincomparable,saturation-pronedatasetsandrarelyexamineitsdeterminantsjointly.Identifyingwhatpredictstransferwouldletusavoidexhaustiveevaluationacrossalllanguagepairsandletdeveloperstargetthefactorsthatlimitperformanceinlow-resourcelanguages.Toevaluatecross-lingualcapabilitytransfer,weintroduceMultilingualGSM-Symbolic,anextensiblemultilingualmathematicaldatasetcovering30,000item-matchedquestion-answerpairsandspanning15languages.Itutilisessymbolictemplatestopreventoverfittingandensuregeneralisationbyallowinggenerationofmillionsofhigh-qualityvariationsfromasinglesample.UsingMultilingualGSM-Symbolic,wequantifythelargestdeterminantsofcapabilityasmodelsize(β=1.77),languageresourcelevel(β=0.77),reasoning(β=0.67)andtypologicaldistance(β=-0.25).Thisjointestimationallowsthesedeterminantstobeexpressedintermsofoneanother:a32BmodelevaluatedinMarathiperformslikea10BmodelinEnglish.Ourfindingshaveimportantimplicationsformodeldevelopers,showingthatmodelsizeandreasoningnarrowtheperformancegapbetweenlow-andhigh-resourcelanguages(β=-0.27andβ=-0.20,respectively),whilesimilarlevershavelittleornoeffectontypologicallydistantlanguages.Overall,ouranalysisframeworkexplains92%ofbetween-languagevariation,butonly23%ofthemodel-by-languagevariation,andpredictsamodel’sperformanceonanunseenlanguagewithin6.0pp(r=.96).Incorporatingmeasurementsfromjust10templatesinthetargetlanguagereducesthisto4.19pp,enablingreasonableestimatesofperformancewithlittleornodownstreamdataset.

View arXiv pageView PDFProject pageGitHub4Add to collection

Get this paper in your agent:

hf papers read 2610\.03367

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.03367 in a model README.md to link it from this page.

Datasets citing this paper1

#### danish-foundation-models/multilingual-gsm-symbolic Viewer• Updated43 minutes ago • 184k • 1.14k • 3

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.03367 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

An In-Vitro Study on Cross-Lingual Generalization in Language Models

arXiv cs.CL

This paper introduces an in-vitro framework with two procedurally generated languages to study cross-lingual generalization in language models, finding that tokenization's preservation of reusable substructure is more critical than lexical similarity or data balance for transferring capabilities across languages.

Skill Issue: Are Skills Language-Invariant in LLMs?

Hugging Face Daily Papers

This paper quantifies cross-lingual skill inconsistencies in large language models through multilingual self-play in text-based games, revealing significant variations in performance across languages that can be partially mitigated by altering intermediate reasoning language.

Cross-Lingual Exploration for Parametric Knowledge

arXiv cs.CL

This paper explores cross-lingual prompting strategies to improve access to parametric knowledge in large language models, demonstrating significant gains in knowledge transfer and factual recall across 17 languages on multilingual benchmarks.