ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions
Summary
Introduces ChLogic, an English-Chinese aligned benchmark that tests whether large language models preserve logical reasoning performance across languages, revealing persistent gaps influenced by surface realization and translation artifacts.
View Cached Full Text
Cached at: 06/17/26, 03:35 AM
Paper page - ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions
Source: https://huggingface.co/papers/2606.17905
Abstract
ChLogic benchmark reveals persistent performance gaps between English and Chinese logical reasoning in large language models, influenced by surface realization differences and translation artifacts.
Large language modelsperform increasingly well on standardizedlogical reasoning benchmarks, but whether this ability remains robust beyond English is unclear. We introduce ChLogic, an English--Chinese aligned benchmark that tests whether models preserve logical reasoning performance when the same latent logical structure is expressed in English and diverse Chinesesurface realizations. Built from formal logical templates, the benchmark contains three data sets: (i) the General aligned set, derived from 60 General Propositions across nine template families; (ii) the Difficult aligned set, derived from 40 Difficult Problems; and (iii) the Chinese-only set, covering 15 language-specific phenomenon types. Each aligned item pairs one English reference expression with five Chinese realizations. Experiments on Qwen3, Ministral, and GLM models reveal a persistent English--Chinese performance gap.Back-translationfrom standard Chinese into English often improves performance on the General aligned set, but produces mixed effects on the Difficult aligned set, where Qwen3-32B and GLM-5.1 perform worse after translation. These results indicate that Chinesesurface realization, translation artifacts, and model-specific behavior jointly affect multilingual logical reasoning. Overall, ChLogic provides a useful stress test for the robustness ofmultilingual reasoning.
View arXiv pageView PDFGitHubAdd to collection
Get this paper in your agent:
hf papers read 2606\.17905
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.17905 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.17905 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.17905 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening
LLMEval-Logic is a new Chinese benchmark for evaluating logical reasoning in LLMs, featuring solver-verified answers and adversarial hardening. The benchmark reveals significant gaps in current models, with the best reaching only 37.5% accuracy on hard items.
ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
ChaosBench-Logic v2 is a large-scale benchmark of 40,886 questions over 165 dynamical systems that evaluates LLMs' logical reasoning abilities, revealing near-random performance on regime transition reasoning and systematic failure modes even in frontier models.
Hidden Language Consistency Phenomena in Reasoning LLMs
This paper studies multilingual reasoning models and reveals that language consistency in outputs can degrade or collapse with task difficulty, especially for less represented languages. It argues that evaluating multilingual capability requires jointly considering accuracy, language consistency, and task difficulty.
Exploring the Capability Boundaries of LLMs in Mastering Chinese Chouxiang Language
This paper introduces Mouse, a specialized benchmark for evaluating LLMs on Chinese Chouxiang Language tasks across six NLP domains, revealing that current state-of-the-art models have significant limitations with this subcultural internet language despite performing well on contextual understanding tasks.
Evaluating Chinese Ambiguity Understanding in Large Language Models
This paper introduces CHA-Gen, a Chinese ambiguity dataset grounded in Potential Ambiguity theory, and evaluates several LLMs on ambiguity detection, finding that models struggle but benefit from chain-of-thought prompting, and that instruction tuning induces overconfidence.