A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
Summary
Introduces A2RBench, an automated pipeline for generating formally verifiable abstract reasoning benchmarks for LLMs, using cycle consistency to ensure unique solutions, and reveals that current LLMs underperform humans significantly on 3D reasoning tasks.
View Cached Full Text
Cached at: 05/19/26, 10:31 AM
Paper page - A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
Source: https://huggingface.co/papers/2605.17278
Abstract
Large language models demonstrate significant limitations in abstract reasoning abilities compared to human performance, particularly in complex 3D task understanding, while an automated benchmark pipeline shows that programmatic verification ensures solution uniqueness through cycle consistency.
Abstract reasoning abilityreflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce anautomated pipelinenamed A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove thatprogrammatic verification--testing whether theinverse operationperfectly reverses theforward operation(cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higherinformation complexitycan simplify the reasoning process.
View arXiv pageView PDFGitHub2Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.17278 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.17278 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.17278 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
AdvancedMathBench is a new benchmark suite for evaluating LLMs on advanced mathematical proof generation and verification. It includes ProverBench for generation and VerifierBench for verification, demonstrating that current models like GPT-5.5-xhigh achieve only modest performance.
FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games
FalsifyBench is a new evaluation framework for assessing inductive reasoning in LLMs, inspired by the Wason 2-4-6 task, where agents discover hidden semantic rules by proposing examples and receiving feedback. Evaluation of 12 LLMs shows reasoning models outperform instruction-tuned models, with negative testing (hypothesis falsification) being the key driver of success.
LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning
Introduces LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier LLMs on structured linear algebra computation across matrix dimensions, revealing that LLM mathematical failure is structurally constrained and transitions from execution errors to computational abandonment at 4x4 scale.
ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
ChaosBench-Logic v2 is a large-scale benchmark of 40,886 questions over 165 dynamical systems that evaluates LLMs' logical reasoning abilities, revealing near-random performance on regime transition reasoning and systematic failure modes even in frontier models.
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
SciR is a new controllable benchmark for evaluating LLMs on scientific reasoning including deduction, induction, and causal abduction, with parametric control over extraction and inference difficulty. Tests show both axes degrade performance across models, with reasoning models like DeepSeek-R1 outperforming instruct models on inference.