Tag
The paper introduces sk-bench, a native-first benchmark for evaluating large language models in Slovak, covering 30 datasets (33 task variants) across ten skill categories to address the lack of native Slovak coverage in existing multilingual benchmarks.
The paper introduces Multilingual GSM-Symbolic, a symbolic-template dataset covering 15 languages and 30k item-matched QA pairs, to quantify what determines cross-lingual capability transfer in LLMs. It finds model size, language resource level, reasoning, and typological distance are key drivers, and shows a framework explaining 92% of between-language variation while predicting unseen-language performance within 6.0pp.
SWORD introduces a Wikidata-based benchmark to evaluate cross-lingual inconsistencies in LLMs' rejection of factual errors, revealing that models rely on distributional familiarity and show performance degradation in Asian languages.
This paper quantifies cross-lingual skill inconsistencies in large language models through multilingual self-play in text-based games, revealing significant variations in performance across languages that can be partially mitigated by altering intermediate reasoning language.
HealMed is an expert-reviewed benchmark for evaluating large language models in medicine across nine languages, developed by medical experts to assess multilingual performance in clinical tasks.
This paper evaluates retrieval quality in RAG systems for Bengali agricultural advisory, finding performance varies by query type and language conditions, and introduces a benchmark dataset to highlight the need for disaggregated evaluation in low-resource settings.
Introduces UA-Legal-Bench, a five-task benchmark for evaluating large language models on Ukrainian legal reasoning, built from the Unified State Register of Court Decisions. Evaluates 11 LLMs, revealing task-dependent few-shot effects and the misleading nature of accuracy on imbalanced legal tasks.
GaoYao introduces a 182k-sample benchmark across 26 languages and 51 regions to systematically evaluate LLMs’ multilingual and multicultural capabilities, revealing large geographical performance gaps.