Skill Issue: Are Skills Language-Invariant in LLMs?
Summary
This paper quantifies cross-lingual skill inconsistencies in large language models through multilingual self-play in text-based games, revealing significant variations in performance across languages that can be partially mitigated by altering intermediate reasoning language.
View Cached Full Text
Cached at: 08/27/26, 03:20 PM
Paper page - Skill Issue: Are Skills Language-Invariant in LLMs?
Source: https://huggingface.co/papers/2608.25832
Abstract
Multilingual self-play reveals that large language models exhibit significant cross-lingual skill inconsistencies in reasoning and strategy, partly recoverable by altering intermediate reasoning language.
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifiescross-lingual skill inconsistencyorthogonally from knowledge and general benchmark performance. We do this viamultilingual self-play: two instances of the same model compete in atext-based game, each interacting through a different language interface. Since the model, opponent, rules, state space, and available actions remain fixed, this setting isolates the effect of language on the model’s realized behavior. We build a multilingual extension to TextArena and evaluate three open-weight models across eight languages and six games coveringspatial reasoning,imperfect information,resource allocation, and repeated interaction. We find that the same model can exhibit markedly different playing strength across languages, with systematic variation in win--loss margins, invalid actions, and strategic tendencies. Detailed analyses reveal language-specific failures inspatial reasoning, card-conditioned decisions, and optimal move selection. In some settings, changing only theintermediate reasoning languagerecovers much of the lost performance, suggesting that language can affect different stages of the decision process. These results show that skill discrepancies are a measurable major roadblock in the development of truly multilingual models. Better understanding these discrepancies can help us design models that perform more equitably across languages.
View arXiv pageView PDFGitHub421Add to collection
Get this paper in your agent:
hf papers read 2608\.25832
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.25832 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.25832 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.25832 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Hidden Language Consistency Phenomena in Reasoning LLMs
This paper studies multilingual reasoning models and reveals that language consistency in outputs can degrade or collapse with task difficulty, especially for less represented languages. It argues that evaluating multilingual capability requires jointly considering accuracy, language consistency, and task difficulty.
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
This paper systematically investigates cross-modal skill injection, where a domain-expert LLM is merged into a VLM to induce emergent multimodal capabilities. It evaluates different scenarios (instruction-following, cross-lingual, mathematical reasoning), merging methods (TA, DARE, etc.), and hyperparameters, finding that TA and DARE perform well except in mathematical reasoning.
Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
The paper introduces Skill Self-Play (Skill-SP), a co-evolutionary framework that uses a proposer, solver, and skill controller to bridge structured verification and open-ended exploration, improving LLM performance on tool-use and reasoning benchmarks.
XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks
XLGoBench introduces a synthetic benchmark of algorithmic tasks to detect cross-lingual skill gaps in LLMs, demonstrating persistent gaps across multiple state-of-the-art models.
Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents
This paper proposes MASA, a framework that adapts skills to each LLM backbone without modifying weights, using hierarchical evolution and a model-conditioned rewriter, achieving gains of up to 25.8 points over baselines.