Tag
Introduces TREAT, a benchmark for evaluating whether large language models can recover known theorem identities from equivalence-preserving transformations of mathematical formulas. The best tested model achieves only 60.73% accuracy, showing that theorem knowledge is fragile under representation changes.
This paper audits five widely used Lean theorem-proving benchmarks, uncovering 398 mechanically certified issues such as counterexamples, vacuous theorems, and unsound axioms. It proposes a fault taxonomy, automated checkers, and release standards to improve evaluation reliability and trustworthiness.
A new Google paper introduces LEAP, an agentic framework that enables general LLMs to solve formal math problems by planning proofs and checking each step, raising performance from under 10% to 70% on the Lean IMO benchmark and solving all 2025 Putnam problems.