Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Summary
Multi-LCB extends the LiveCodeBench benchmark to evaluate LLMs across twelve programming languages while preserving contamination controls, revealing Python overfitting and language-specific contamination issues.
View Cached Full Text
Cached at: 06/20/26, 02:27 PM
Paper page - Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Source: https://huggingface.co/papers/2606.20517
Abstract
Multi-LCB addresses the limitation of LiveCodeBench by providing a multi-language benchmark for evaluating LLMs across twelve programming languages while maintaining contamination controls and evaluation protocols.
LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluatinglarge language models(LLMs) oncode-generation tasks. By curatingcompetitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB providescontamination-aware evaluationand offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB’s contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment ofcross-language code generationcompetence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence ofPython overfitting,language-specific contamination, and substantial disparities inmultilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB’s primary limitation and exposing critical gaps in current LLM capabilities.
View arXiv pageView PDFProject pageGitHub22Add to collection
Get this paper in your agent:
hf papers read 2606\.20517
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.20517 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.20517 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.20517 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Open-source LLM benchmark runs 147 coding tasks every 4 hours, 5-trial median with 95% CI, and uses CUSUM for change-point detection. Curious what people think of the methodology
An open-source LLM benchmark with 147 coding tasks runs every 4 hours, using 5-trial median with 95% confidence intervals and CUSUM for change-point detection, sparking discussion on its methodology.
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
XL-SafetyBench is a benchmark of 5,500 test cases across 10 country-language pairs to evaluate LLM safety and cultural sensitivity, distinguishing jailbreak robustness from cultural awareness.
XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks
XLGoBench introduces a synthetic benchmark of algorithmic tasks to detect cross-lingual skill gaps in LLMs, demonstrating persistent gaps across multiple state-of-the-art models.
mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
Introduces mmPISA-bench, a compact multilingual reasoning benchmark derived from PISA, and evaluates proprietary LLMs across 43 languages, finding that they reason effectively with some performance variations, and that machine-translated questions do not degrade accuracy.
LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis
Introduces LingxiDiagBench, a large-scale multi-agent benchmark for evaluating LLMs on Chinese psychiatric consultation and diagnosis. Key findings show high accuracy on binary classification but poor performance on multi-way differential diagnosis, highlighting a decoupling between conversational quality and diagnostic accuracy.