AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification

Hugging Face Daily Papers Papers
benchmark mathematical-reasoning proof-verification llm-evaluation advanced-mathematics

Summary

AdvancedMathBench is a new benchmark suite for evaluating LLMs on advanced mathematical proof generation and verification. It includes ProverBench for generation and VerifierBench for verification, demonstrating that current models like GPT-5.5-xhigh achieve only modest performance.

Large language models (LLMs) have achieved remarkable performance on high-school and olympiad-style mathematics, yet their capabilities on advanced mathematics remain poorly understood. Existing benchmarks, however, fall short in both scope and evaluation granularity: they provide limited disciplinary coverage and often rely on final-answer correctness or coarse judgments, leaving the validity of the reasoning process inadequately assessed. To bridge this gap, we introduce AdvancedMathBench, a benchmark suite designed to evaluate advanced mathematical reasoning capabilities. Its core proof-generation benchmark, ProverBench, contains 296 problems spanning undergraduate and doctoral qualifying-exam levels. To provide reliable evaluation of the proofs, we develop a dedicated automatic verification pipeline trained on large-scale expert annotations to produce both correctness verdicts and fine-grained assessments of proof errors, which exhibits strong agreement with human experts on held-out proof trajectories. We further introduce VerifierBench, consisting of 888 model-generated proof trajectories paired with expert ground truth, to evaluate whether models can correctly judge proof validity and provide sound verification rationales. Experiments show that AdvancedMathBench remains challenging for frontier models. On proof generation, the best-performing model, GPT-5.5-xhigh, achieves only 75.8 and 66.1 on the UGD and QE splits, respectively, indicating substantial room for improvement on advanced mathematical proof construction. On proof verification, the best model attains a Balanced F1 of only 65.1, and models generally exhibit low true negative rates, suggesting that critical error detection remains a major bottleneck.
Original Article
View Cached Full Text

Cached at: 07/14/26, 04:13 AM

Paper page - AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification

Source: https://huggingface.co/papers/2607.11849 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Largelanguagemodels(LLMs)haveachievedremarkableperformanceonhigh-schoolandolympiad-stylemathematics,yettheircapabilitiesonadvancedmathematicsremainpoorlyunderstood.Existingbenchmarks,however,fallshortinbothscopeandevaluationgranularity:theyprovidelimiteddisciplinarycoverageandoftenrelyonfinal-answercorrectnessorcoarsejudgments,leavingthevalidityofthereasoningprocessinadequatelyassessed.Tobridgethisgap,weintroduceAdvancedMathBench,abenchmarksuitedesignedtoevaluateadvancedmathematicalreasoningcapabilities.Itscoreproof-generationbenchmark,ProverBench,contains296problemsspanningundergraduateanddoctoralqualifying-examlevels.Toprovidereliableevaluationoftheproofs,wedevelopadedicatedautomaticverificationpipelinetrainedonlarge-scaleexpertannotationstoproducebothcorrectnessverdictsandfine-grainedassessmentsofprooferrors,whichexhibitsstrongagreementwithhumanexpertsonheld-outprooftrajectories.WefurtherintroduceVerifierBench,consistingof888model-generatedprooftrajectoriespairedwithexpertgroundtruth,toevaluatewhethermodelscancorrectlyjudgeproofvalidityandprovidesoundverificationrationales.ExperimentsshowthatAdvancedMathBenchremainschallengingforfrontiermodels.Onproofgeneration,thebest-performingmodel,GPT-5.5-xhigh,achievesonly75.8and66.1ontheUGDandQEsplits,respectively,indicatingsubstantialroomforimprovementonadvancedmathematicalproofconstruction.Onproofverification,thebestmodelattainsaBalancedF1ofonly65.1,andmodelsgenerallyexhibitlowtruenegativerates,suggestingthatcriticalerrordetectionremainsamajorbottleneck.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.11849 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.11849 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.11849 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory

arXiv cs.AI

The paper introduces GTBench, a curriculum-grounded benchmark for evaluating LLMs as mathematical research assistants in graph theory, containing 63 problems across three difficulty levels. It evaluates five frontier models and finds that performance degrades with difficulty, with GPT-5 achieving near-perfect results on basic problems but only 82% on graduate-level proofs.