NL2AGBench:面向 AlphaGeometry 的大语言模型自动形式化基准评测
摘要
该论文介绍了 NL2AGBench,这是一个基于执行的基准,用于评估大语言模型将自然语言几何问题转换为 AlphaGeometry 形式化领域专用语言(DSL)的能力。研究发现,前沿闭源模型的可执行翻译准确率超过 80%,而开源模型则明显落后。
查看缓存全文
缓存时间: 2026/11/20 09:13
# NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry
Source: [https://arxiv.org/html/2608.28481](https://arxiv.org/html/2608.28481)
Judy SongAffiliation:Vandegrift High School Austin, TXRory HuAffiliation:Groton School Cupertino, CAZiliang ZongAffiliation:Computer Science Department Texas State University
###### Abstract
Recent advances in large language models \(LLMs\) have demonstrated strong performance in natural language understanding and mathematical reasoning\. However, their ability to translate informal mathematical problem statements into formal representations remains underexplored\. This limitation is particularly important for neuro\-symbolic geometry systems such as AlphaGeometry, whose theorem\-proving engine requires inputs written in a specialized domain\-specific language \(DSL\)\. Although AlphaGeometry achieved near\-IMO gold\-medalist performance on Olympiad geometry problems, the manual conversion of natural\-language problems into its formal syntax remains a significant usability bottleneck\. To address this gap, we introduce Natural Language to AlphaGeometry Benchmark \(NL2AGBench\), which aims to evaluate the ability of LLMs to translate English geometry problems into AlphaGeometry\-compatible formal representations\. NL2AGBench evaluates translation quality using execution\-based verification within the AlphaGeometry framework rather than relying solely on textual similarity metrics\. We evaluate ten state\-of\-the\-art open\- and closed\-source LLMs spanning multiple parameter scales and analyze their performance in terms of executable translation accuracy, syntactic correctness, and error characteristics\. Our experiments reveal a substantial performance gap between frontier closed\-source models and open\-source alternatives\. While leading closed\-source models achieve executable translation rates exceeding 80%, even the largest open\-source models struggle to consistently preserve geometric constraints and produce valid formalizations\. To better understand these failures, we introduce an error taxonomy that categorizes AlphaGeometry execution failures into syntax and logic errors\. We further investigate mitigation strategies, including few\-shot prompting, fine\-tuning, and human\-guided hinting, demonstrating measurable improvements across multiple model families\.
###### Index Terms:
AlphaGeometry, benchmark, translation, neuro\-symbolic learning, prompt engineering\.
Fig\. 1:Translation gap between natural\-language geometry problems and AlphaGeometry’s formal DSL\. While AlphaGeometry has demonstrated strong theorem\-proving capabilities, the manual formalization step remains a major barrier to usability and large\-scale deployment\.## IIntroduction
Recent progress in large language models \(LLMs\) has significantly advanced natural language understanding, code generation, and mathematical reasoning\. Despite these advances, translating informal mathematical descriptions into machine\-verifiable formal representations remains a challenging problem\. This process, commonly referred to as auto\-formalization, is a critical prerequisite for many theorem\-proving and neuro\-symbolic reasoning systems because formal solvers require precise symbolic inputs rather than natural\-language descriptions\.
Geometry theorem proving presents a particularly challenging auto\-formalization setting\. Unlike algebraic problems, geometry problems often contain implicit spatial relationships, diagram\-dependent information, and complex geometric constraints that must be preserved exactly during translation\. Even small translation errors can fundamentally alter a problem’s meaning and prevent downstream reasoning systems from producing valid proofs\.
AlphaGeometry represents one of the most successful neuro\-symbolic systems for Olympiad\-level geometry reasoning\[[1](https://arxiv.org/html/2608.28481#bib.bib14)\]\. By combining neural\-guided auxiliary construction generation with symbolic deductive reasoning, AlphaGeometry solved 25 of 30 historical International Mathematical Olympiad \(IMO\) geometry problems and demonstrated performance comparable to elite human competitors\. However, AlphaGeometry requires all problems to be expressed in a specialized domain\-specific language \(DSL\)\. As a result, geometry problems written in standard mathematical English must first be manually translated into formal syntax before theorem proving can begin\. This translation requirement remains a major barrier to accessibility, scalability, and practical deployment\. Figure 1 illustrates this challenge\. On the left is a geometry problem written in natural mathematical language, similar to the format used in Olympiad competitions and textbooks\. On the right is the corresponding AlphaGeometry DSL representation required by the theorem prover\. While the English description is concise and intuitive for human readers, the formal representation must explicitly encode geometric objects, constraints, incidences, and proof targets using a strict symbolic syntax\. Minor omissions or incorrect constructions can fundamentally change the problem semantics and prevent the theorem prover from generating a valid proof\.
The challenge of converting natural\-language geometry problems into formal representations is closely related to semantic parsing and mathematical auto\-formalization\. Prior work has explored translating informal mathematics into proof\-assistant languages such as Lean, Isabelle, and Metamath, while recent studies have investigated the use of LLMs to assist formal theorem proving\[[2](https://arxiv.org/html/2608.28481#bib.bib1),[3](https://arxiv.org/html/2608.28481#bib.bib4)\]\. However, existing benchmarks primarily evaluate proof generation, mathematical reasoning, or general auto\-formalization\. To the best of our knowledge, no benchmark specifically evaluates the translation of natural\-language geometry problems into AlphaGeometry\-compatible formal representations\.
To address this gap, we introduce NL2AGBench, a benchmark designed to evaluate the ability of LLMs to translate English geometry problems into AlphaGeometry’s DSL\. NL2AGBench consists of geometry problems paired with verified formal representations and evaluates model outputs through direct execution within AlphaGeometry\. This execution\-based evaluation allows us to assess not only syntactic correctness but also whether generated translations preserve the geometric semantics required for theorem proving\.
Using NL2AGBench, we evaluate a diverse collection of open\- and closed\-source LLMs and analyze their ability to bridge the gap between natural\-language problem descriptions and formal geometric reasoning systems\. Our results reveal a substantial performance disparity between frontier closed\-source models and open\-source alternatives, highlighting the difficulty of geometry\-specific auto\-formalization\. We further characterize common failure modes through a systematic error taxonomy and analyze mitigation strategies including few\-shot prompting, fine\-tuning, and human\-guided hints\.
Our main contributions are summarized as follows:
- •We introduce NL2AGBench, a benchmark for evaluating the translation of natural\-language geometry problems into AlphaGeometry\-compatible domain\-specific language representations\. The benchmark provides a standardized framework for studying geometry auto\-formalization and LLM\-assisted theorem\-proving pipelines\.
- •We conduct a comprehensive evaluation of ten state\-of\-the\-art LLMs, including both open\-source and closed\-source models spanning multiple parameter scales\. Our results reveal a substantial performance gap between frontier proprietary models and currently available open\-source alternatives\. The largest open\-source model achieves a score of under 46% with an optimized prompting technique, while closed\-source large frontier models have reached 85% and above\.
- •We introduce a systematic error taxonomy for AlphaGeometry translation failures, categorizing execution failures into syntax and logic errors\. This taxonomy provides practical insights into common failure modes and facilitates future research on geometry auto\-formalization\.
- •We investigate multiple translation\-improvement strategies, including few\-shot prompting, fine\-tuning, and human\-guided hinting\. Our experiments demonstrate that targeted prompting and guidance can significantly improve pass rates for Llama3\.1:70b by 23%, GPT\-4o\-mini:8b by 8%, and Qwen3:235b by 33\.3%\.
## IIRelated Work
### II\-AAlphaGeometry and its Applications
AlphaGeometry is a neuro\-symbolic AI system designed to solve Euclidean geometry proof problems\[[1](https://arxiv.org/html/2608.28481#bib.bib14)\]\. It combines a symbolic deduction engine with a neural model that proposes auxiliary constructions when direct deduction is insufficient\. This builds on earlier symbolic geometry theorem\-proving methods, such as automated geometry proof systems\[[4](https://arxiv.org/html/2608.28481#bib.bib2)\]\. Beyond pure mathematics, AlphaGeometry has potential applications in scientific reasoning, synthetic\-data generation, and operating systems verification\. Other recent systems, such as HAGeo and TongGeometry, have also explored improved auxiliary construction strategies and guided search for Olympiad geometry\[[5](https://arxiv.org/html/2608.28481#bib.bib5),[6](https://arxiv.org/html/2608.28481#bib.bib19)\]\. However, these systems still depended on formal or semi\-formal problem representations\. Newclid addressed this usability issue by proposing a more user\-friendly alternative to AlphaGeometry\[[7](https://arxiv.org/html/2608.28481#bib.bib12)\]\. These works showed that while geometry theorem proving has advanced significantly, translating natural\-language problems into solver\-compatible syntax is still a key bottleneck\.
### II\-BChallenges in AlphaGeometry Translation
The task of converting English geometry problems into AlphaGeometry syntax is closely related to auto\-formalization\. Wu et al\. showed that large language models can perform part of this translation process for mathematical statements\[[8](https://arxiv.org/html/2608.28481#bib.bib17)\], while Jiang et al\. demonstrated that informal proof sketches can help guide formal theorem proving\[[9](https://arxiv.org/html/2608.28481#bib.bib10)\]\. Benchmarks such as miniF2F have also evaluated IMO Olympiad mathematical reasoning in formal systems such as Lean, Isabelle, and Metamath\[[10](https://arxiv.org/html/2608.28481#bib.bib20)\]\. More recent work on formal mathematical reasoning, such as research related to LeanDojo, ProofBridge, and AlphaProof, further showed the growing importance of connecting natural\-language reasoning with formal theorem\-proving environments\[[11](https://arxiv.org/html/2608.28481#bib.bib18),[12](https://arxiv.org/html/2608.28481#bib.bib9),[13](https://arxiv.org/html/2608.28481#bib.bib8)\]\. However, most of these works focused on general mathematical formalization or proof generation\. In contrast, we focus specifically on translating natural\-language geometry problems into AlphaGeometry\-compatible syntax, where readability, object definitions, givens, and target conclusions must all be correct\.
### II\-CLLM Benchmarks and Prompting
Existing LLM benchmarks evaluated mathematical reasoning or language translation, but not mathematical translation\. The MATH dataset and GSM8K benchmark evaluated general mathematical problem solving, while OlympiadBench evaluated Olympiad\-level scientific and mathematical reasoning\[[14](https://arxiv.org/html/2608.28481#bib.bib7),[15](https://arxiv.org/html/2608.28481#bib.bib3),[16](https://arxiv.org/html/2608.28481#bib.bib6)\]\. Model\-specific work such as DeepSeekMath showed that training on mathematical data can improve competition\-math performance, but stronger mathematical reasoning did not necessarily guarantee correct formal translation\[[17](https://arxiv.org/html/2608.28481#bib.bib11)\]\. For translation prompting, chain\-of\-thought prompting and self\-consistency prompting have shown that LLMs can improve reasoning performance when encouraged to generate intermediate steps or sample multiple reasoning paths\[[18](https://arxiv.org/html/2608.28481#bib.bib16),[19](https://arxiv.org/html/2608.28481#bib.bib15)\]\. For AlphaGeometry translation, similar prompting strategies can assist models with improving their translation\. We address a specific gap in mathematical reasoning and translation by evaluating LLMs on natural\-language\-to\-AlphaGeometry translation rather than only mathematical problem solving\.
## IIINL2AGBench
### III\-ABenchmark Construction
#### III\-A1Sourcing Dataset
NL2AGBench is constructed from geometry problems originally formalized for AlphaGeometry\. We begin with the Java Geometry Expert \(JGEX\) repository, an open\-source geometry theorem\-proving framework containing hundreds of Olympiad\-style geometry problems\. Among these problems, the AlphaGeometry project manually formalized 231 instances into its domain\-specific language\. These examples form the foundation of our benchmark because they enable direct evaluation of automatic translation from English geometry statements into AlphaGeometry\-compatible syntax\. Unlike general mathematical reasoning datasets, every problem in NL2AGBench contains verified formal specifications that can be executed directly by AlphaGeometry\.
#### III\-A2Benchmark Selection
From the 231 formalized problems, we select 48 representative problems to construct NL2AGBench\. Problems were chosen based on two criteria:
- •Translation diversity: Problems cover a broad range of geometric concepts including circles, cyclic quadrilaterals, angle bisectors, orthocenters, circumcenters, midpoints, reflections, and perpendicular constructions\.
- •Execution efficiency: Problems requiring excessive runtime or computational resources within AlphaGeometry are excluded to ensure scalable evaluation across LLMs\.
NL2AGBench contains problems \(see sample problems 1 and 2 below\) spanning varying levels of syntactic complexity and geometric reasoning difficulty while maintaining practical evaluation costs\. A visual diagram of 8 sample NL2AGBench problems is shown in Figure 2\.
Problem1:
LetABCbeatrianglewithcircumcenterO\.LetHbe
themidpointofBC,andletlinesOHandAB
intersectatD\.ThroughC,drawtheline
perpendiculartoCO,andthroughA,drawtheline
perpendiculartoAO;letthesetwolinesintersect
atE\.ProvethatpointsA,O,E,andDlieona
circle\.
Translation:
abc=triangleabc;o=circleoabc;h=
midpointhcb;d=on\_linedoh,on\_linedab;e
=on\_tlineecco,on\_tlineeaao
?cyclicaoed
Problem2:
LetABCbeatriangleinscribedinacirclewith
centerO\.LetMbethemidpointofsideBA\.LetNbe
apointonlineOMextendedthatalsoliesonthe
circlewithcenterO,soNisthesecond
intersectionoflineOMwiththecircumscribed
circleoftriangleABC\.DrawlineNC\.Provethatthe
angleACNisequaltotheangleNCB,thatis,prove
thatCNbisectstheangleACB\.
Translation:
abc=triangleabc;m=midpointmba;o=
circleoabc;n=on\_linenom,on\_circlenoa?
eqanglecacncncb
Fig\. 2:Sample NL2AGBench problems\. These selected problems encompass a wide range of geometry question types, while still being optimized for runtime and usability efficiency\.
#### III\-A3Evaluation Objective
Given an English geometry problem \(PNLP\_\{NL\}\), the goal is to generate a corresponding AlphaGeometry formalization \(PAGP\_\{AG\}\):
fθ\(PNL\)→PAGf\_\{\\theta\}\(P\_\{NL\}\)\\rightarrow P\_\{AG\}
wherefθf\_\{\\theta\}denotes an LLM\.
The generated translation must satisfy two requirements:
- •Syntactic correctness: the DSL must conform to AlphaGeometry grammar\.
- •Semantic correctness: the translation must preserve all geometric objects, constraints, and target conclusions contained in the original problem statement\.
A translation is considered successful if and only if it can be executed by AlphaGeometry without error and produces a valid problem specification\.
### III\-BPrompting Strategy
To provide models with sufficient knowledge of AlphaGeometry syntax, we employ a prompting framework, which consists of:
- •official AlphaGeometry syntax definitions
- •handwritten rules describing clause usage and formatting constraints
- •the target geometry problem
The LLM is instructed to generate a single two\-line output containing the problem identifier and its corresponding DSL representation\. No explanations or intermediate reasoning are requested\. This setup mirrors a realistic auto\-formalization scenario in which the model must map natural\-language geometry descriptions directly into executable symbolic representations\.
Usingthefilesabove,translatethefollowing
geometryproblemintotheAlphaGeometrydomain
languageformat\.Keeptheproblemtitleexactlyas:
\{problem\_id\}\.ProvideONLYONEtranslation\.Donot
includeexplanations,commentary,oralternatives
\-\-\-justthetwo\-lineproblemdefinition\(titleline
\+definitionline\)\.
### III\-CTranslation Verification
Evaluating translation quality using string similarity alone is insufficient because multiple formalizations may be syntactically different while remaining semantically equivalent\. Therefore, NL2AGBench employs an execution\-based evaluation protocol\. For each generated translation, we execute the resulting DSL specification using AlphaGeometry\. The execution outcome is categorized into one of three classes:
- •Successful Translation: The specification is accepted by AlphaGeometry and executes successfully\.
- •Syntax Error: The specification violates DSL grammar or construction rules\.
- •Logic Error: The specification is syntactically valid but semantically inconsistent with the intended geometry problem\.
This evaluation framework directly measures whether a generated translation can function as an effective interface between natural\-language geometry problems and AlphaGeometry’s symbolic reasoning engine\.
## IVError Classification
A central challenge in evaluating AlphaGeometry formalization is the ambiguity of execution failures\. AlphaGeometry frequently reports low\-level runtime exceptions that provide limited information regarding the underlying translation mistake\. To better understand model behavior, we introduce an execution\-based error taxonomy that maps AlphaGeometry failures to interpretable categories\.
### IV\-ASyntax Errors
Syntax errors occur when a generated translation violates the structural requirements of the AlphaGeometry domain\-specific language\. These errors prevent the symbolic engine from parsing or constructing the geometric configuration\. Below are the three cases for syntax errors\.
#### IV\-A1Assertion Error
The AssertionErrorsoccur when a construction receives an invalid number of arguments\. Such errors typically arise when the model correctly identifies the intended construction but fails to provide all required geometric entities\. Examples include incomplete reflection operations, malformed circle constructions, or incorrectly specified auxiliary points\. These errors indicate partial understanding of the problem but incomplete knowledge of DSL syntax\.
Exampleerroneoustranslation:
ab=segment;c=midpointba;d=reflectcb;e
=on\_circleca,r\_triangleecd;f=on\_tlineba
b,on\_lineae;g=on\_linebf,on\_linede?cong
eggf
In this example,AssertionErroris thrown atd=reflectcbbecause the reflect construction requires three points: the point that will be reflected and another two points to define the line being reflected upon\. Since only two points are provided, the computation results in an error\.
#### IV\-A2Key Error
AKeyErroroccurs when clauses, delimiters, or construction arguments are used improperly, similar toAssertionError\. Common causes include:
- •misuse of special characters \(e\.g\. semicolon, whitespace, proof\-operator\)
- •invalid output\-variable references
- •malformed construction declarations
- •hallucinated DSL clauses
Exampleerroneoustranslation:
bcda=quadrangle;o=circumcenterbcda;q
=midpointcb;s=midpointad;j=midpointsq;
m=reflectoj;i=on\_linead,on\_linebc?perp
smbc
AlphaGeometry’s DSL allows points to be repeated for readability\. For instance,a=midpointabcdefines a new pointathat is in the middle of segmentbc\. The pointacan be repeated to the right of the clause to show that it is an output point\. In the above example,KeyErroris thrown ato=circumcenterbcda\. The circumcenter clause normally takes three points to define the circle in which the circumcenter point is to be constructed\. The correct usage including repetition of the output point iso=circumcenterobcd\. There is a maximum of four points including the repeat to the right of the clause\. However, if the output point does not match \(odoes not matchbon the other side in the original example\) when the maximum of four points is used, aKeyErroris thrown\.
A second source ofKeyErroris incorrect use of special characters, particularly semicolons and question marks\. In the example below, aKeyErroroccurs before the proof statementfootade;?eqanglebecause semicolons may separate clauses but cannot immediately precede a proof clause marked by?\. Similar parsing issues, such as trailing whitespaces, can also triggerKeyError\.
Exampleerroneoustranslation:
ab=segment;c=midpointba;d=on\_circleca;
e=on\_lineab,on\_tlinedcd;f=
footade;?eqanglefadadab
The final observed case in which aKeyErroroccurs is when clauses are misplaced\.
Exampleerroneoustranslation:
abc=triangle;d=footbac;e=footabc;
f=footbde?
eqangleabaddbde=eqanglecbcffbfd
Here, an extraeqangleis hallucinated into the proof clause, causing aKeyError\.
#### IV\-A3Value Error
ValueErrorsoccur when geometric objects are defined inconsistently or recursively\. Examples include:
- •circular point definitions
- •self\-referential constructions
- •invalid proof predicates
- •incorrect parameter counts
These errors indicate that the model generated syntactically plausible structures that violate underlying geometric construction rules\. For example, theValueErroris triggered atd=on\_circumabc,on\_circumaod\. Although pointdis being defined, it is also used in the secondon\_circumclause\. Becausedhas not yet been constrained, it cannot be used to determine its own position, resulting in aValueError\.
Exampleerroneoustranslation:
abc=triangle;o=circumcenterabc;d=
on\_circumabc,on\_circumaod;p=on\_circuma
bc,on\_circumaop;f=footpad;g=foot
pab;h=footpbc;e=footpcd;i=on\_line
fg,on\_linehe?cyclicpgih
Another case is when the final proof clause has an incorrect amount of parameters\.
Exampleerroneoustranslation:
abc=r\_triangle;d=footcab?eqanglecacd
adabbcbd
Theeqangleclause only takes 8 arguments, the first four defining the first angle and the last four defining the second angle\. In this example,ValueErroris thrown because there are 12 arguments passed intoeqangle\.
Exampleerroneoustranslation:
abd=triangle;c=on\_tlinecab,on\_tlinecba
;d1=on\_linebc,angle\_bisectordbc;e=
midpointad1;f=on\_lineac,on\_linebd1?para
cebd1
Here,ValueErroris thrown at the definition of point c:c=on\_tlinecab,on\_tlinecba\. The constructionon\_tlinerequires three argument points to establish a point of tangency between a circle and a line\.
### IV\-BLogic Errors
Logic errors occur when the generated translation is syntactically valid but semantically incorrect\. Typical causes include:
- •omitted constraints
- •incorrect geometric relations
- •incorrect target predicates
- •misinterpretation of geometric terminology
Logic errors are particularly important because they reveal limitations in mathematical understanding, rather than deficiencies in syntax generation\.
Exampleerroneoustranslation:
abc=r\_triangleabc?perpabbc
In this example,r\_trianglecreates a right triangle with right angleCAB\. Therefore, linesCAandABare perpendicular, notABandBC\. Using the modified ag4masses\[[20](https://arxiv.org/html/2608.28481#bib.bib13)\]system, these logic errors are detected immediately and cause the terminal to freeze\. In contrast, the official system continues attempting to solve the problem before eventually freezing as well\. Therefore, we terminate any run that exceeds five minutes without producing new output and classify it as a logic error\.
I052912:26:07\.656355140703903308672graph\.py:498\]
test\_problem
I052912:26:07\.656621140703903308672graph\.py:499\]
abc=r\_triangleabc?perpabbc
\*terminalfreezesbeyondthispoint
### IV\-CSignificance of the Taxonomy
Our taxonomy provides a standardized framework for analyzing failures in AlphaGeometry auto\-formalization\. Rather than reporting only overall success rates, the taxonomy enables fine\-grained diagnosis of whether model failures arise from syntax generation, geometric interpretation, or semantic reasoning\. This distinction is critical for not only understanding the errors designing future mitigation strategies and improving LLM\-based mathematical formalization systems\.
### IV\-DBaseline Model Performance Results
TABLE I:Model Performance \(Zero\-Shot\)
## VImproving Translation Performance
Our baseline evaluation reveals a substantial performance gap between frontier closed\-source models and open\-source alternatives\. The hatched data in Figure 3 illustrates the performance of these models in zero\-shot circumstances and serve as a reference point\. While large proprietary models frequently generate executable AlphaGeometry translations, open\-source models often fail due to syntax violations, omitted constraints, or incorrect geometric interpretations\. To better understand how translation performance can be improved, we investigate three mitigation strategies: few\-shot prompting, human\-guided correction and supervised fine\-tuning\.
### V\-AFew\-Shot Prompting
Fig\. 3:Model performance for zero\-shot and few\-shot prompting\.Few\-shot prompting serves as the primary intervention strategy\. Each model receives 54 reference examples consisting of paired natural\-language geometry problems and AlphaGeometry translations\. These examples expose the model to common geometric constructions, DSL syntax patterns, and proof predicates\. Compared with zero\-shot prompting, few\-shot prompting consistently improves executable translation rates across all evaluated models\.
Below are two samples of the multi\-shot example translations that are supplied to LLMs\. The first section of the example depicts the English version of the problem, while the second is the manually\-verified DSL translation\.
\#Example1
LetABCbeatrianglewithAB<ACandcircumcenter
O\.TheanglebisectorofangleBACmeetsBCatD\.
ThelinethroughDperpendiculartoBCmeetsAOat
X\.YisthemidpointofAD\.ProvethatB,C,X,Y
areconcyclic\.
abc=triangle;o=circumcenterabc;d=
on\_linebc,angle\_bisectorbac;x=on\_lineao,
on\_tlinedbc;y=midpointad?cyclicbcxy
\#Example2
LetABCbeatrianglewithAB=ACandMthemidpoint
ofBC\.PisapointwithPB<PCandPAparalleltoBC\.
X,YareonlinesPB,PCwithBonPX,ConPY,and
anglePXM=anglePYM\.ProveAPXYiscyclic\.
abc=iso\_triangle;m=midpointbc;p=on\_pline
abc;x=on\_linepb;y=on\_linepc,eqangle3y
pmxmp?cyclicapxy
Figure 3 shows that few\-shot prompting improves translation accuracy across nearly all models\. Among closed\-source models, Gemini\-3\.1 achieves the best performance, increasing from 62\.5% accuracy in the zero\-shot setting to 83\.3% with few\-shot prompting, a gain of about 21 percentage points\. Grok\-3 improves from 52\.1% to 62\.5% \(\+10 points\), while Sonnet\-4\.6 rises from 37\.5% to 52\.1% \(\+14 points\)\. GPT\-5\.4 starts at a relatively high 75\.0% accuracy and reaches 79\.2% with few\-shot examples \(\+4 points\)\. In contrast, GPT\-4o:8B remains slightly improved, improving only from 4\.2% to 8\.3%\.
The largest relative gains are observed among open\-source models\. Qwen3:235B improves from 12\.5% to 45\.8% accuracy, a gain of around 33 percentage points and more than a threefold increase\. Llama3\.1:70B rises from 0% to 23%, while Mistral:7B, Nemotron\-3 Nano:4B, and Llama3\.2:3B fail to produce any correct translations in either setting\. The improvement is particularly pronounced for syntax\-related errors, suggesting that a substantial fraction of translation failures arise from unfamiliarity with AlphaGeometry’s DSL rather than a complete lack of geometric understanding\.
Overall, few\-shot prompting is the most effective and scalable approach, consistently reducing error rates and improving accuracy across models without requiring retraining\. However, a substantial performance gap remained between the strongest closed\-source models and most open\-source alternatives\.
### V\-BHuman\-Guided Hints
Our next strategy investigates whether targeted human guidance can help LLMs recover from translation failures\. Unlike few\-shot prompting and fine\-tuning, which provide general knowledge about AlphaGeometry syntax, this approach introduces problem\-specific information after an initial translation has been generated\.
As illustrated in Figure[4](https://arxiv.org/html/2608.28481#S5.F4), we employ a two\-stage human\-guided correction framework\.
#### V\-B1Stage 1: Diagram\-Based Hints
In the first stage, the model is provided with a geometric diagram corresponding to the target problem and is asked to re\-evaluate its translation\. The diagram supplies spatial information that may be difficult to infer reliably from text alone, including collinearity relationships, circle memberships, and auxiliary constructions\.
Many Olympiad geometry problems implicitly assume information that is visually obvious from the accompanying figure but only partially specified in the textual description\. By incorporating the diagram shown in Figure 4, the model gains access to additional geometric context that can reduce ambiguity during formalization\. This stage is effective when the original translation omits geometric constraints or incorrectly interprets relationships between points, lines, and circles\.
#### V\-B2Stage 2: Error\-Specific Hints
After examining the revised translation, a second round of guidance is provided when necessary\. Rather than supplying additional geometric context, these hints directly target observed translation errors\. As shown in the lower portion of Figure 3, the hint may identify a specific issue such as an incorrect number of clause parameters, misuse of a geometric predicate, or an invalid construction\. The model is then asked to revise its translation while preserving the remaining structure\. This stage functions as a targeted correction mechanism\. Instead of requiring the model to rediscover the source of failure, the hint narrows the search space and focuses attention on the problematic portion of the formalization\.
Fig\. 4:Two\-stage human\-guided correction framework\. In Stage 1, the model is provided with a geometry diagram to supply additional structural context and reduce ambiguity in the natural\-language problem statement\. In Stage 2, targeted error\-specific hints are provided to correct localized translation mistakes identified in the generated AlphaGeometry formalization\.
#### V\-B3Effectiveness and Limitations
Human\-guided hints improve translation quality for strong frontier models, particularly when the initial translation is close to correct\. In many cases, the models demonstrate sufficient geometric understanding to generate an accurate formalization once a specific ambiguity or error has been identified\. Figure[4](https://arxiv.org/html/2608.28481#S5.F4)highlights the complementary roles of the two stages\. The first stage addresses missing geometric information through diagram\-based reasoning, while the second stage addresses localized translation mistakes through targeted corrective feedback\. Together, they form an refinement process that improves executable translation rates without modifying model parameters\. Specifcially, the hint\-based methodology improves translation performance across all evaluated models: GPT\-5\.4 by 14\.29%, Gemini 3\.1 by 14\.63%, Claude Sonnet 4\.6 by 26\.47%, and Grok\-3 by 32\.14%\. These gains show that targeted human guidance can substantially improve translation rates, especially for cases involving recoverable formalization errors\.
Despite their effectiveness, human\-guided hints have limited scalability\. Diagram interpretation requires additional problem resources, and error\-specific hints require manual inspection of model outputs\. Consequently, this approach is best viewed as a diagnostic and correction framework rather than a fully automated solution for large\-scale benchmark evaluation\.
### V\-CSupervised Fine\-Tuning
We additionally investigate whether fine\-tuning can improve translation performance for smaller open\-source models\. For this experiment, we utilize HAGeo\-409, a human\-verified collection of geometry problems paired with AlphaGeometry formalizations\. Models are fine\-tuned to generate DSL representations directly from natural\-language inputs\. Fine\-tuning improves adherence to AlphaGeometry syntax and reduces several categories of grammar\-related errors\. However, improvements in executable translation accuracy remain limited\. Qualitative inspection reveals that many fine\-tuned models learn surface\-level DSL patterns without reliably capturing the geometric semantics required for correct formalization\. In several cases, models generate syntactically plausible but semantically incorrect constructions, indicating that formalization requires deeper geometric reasoning than simple syntax imitation\. These findings suggest that the primary bottleneck for smaller models is not solely DSL familiarity but also insufficient reasoning capacity\.
## VIConclusion and Future Work
This paper investigates a key bottleneck in automated geometry theorem proving: translating natural\-language geometry problems into the formal language required by AlphaGeometry\. To address this challenge, we introduce NL2AGBench, a benchmark specifically designed to evaluate the ability of large language models to translate English geometry problems into AlphaGeometry\-compatible formal representations\.
We systematically assess both open\-source and closed\-source LLMs across a diverse collection of geometry problems\. Our results reveal a substantial gap between frontier proprietary models and current open\-source alternatives, suggesting that successful formalization requires not only syntactic mastery of the target language but also accurate geometric interpretation and constraint preservation\. Through detailed error analysis, we construct a taxonomy of AlphaGeometry translation failures, categorizing them into syntax errors and logic errors\. This taxonomy provides a practical framework for understanding model weaknesses and diagnosing failure modes in future auto\-formalization systems\.
We further explore multiple mitigation strategies, including few\-shot prompting, supervised fine\-tuning, and human\-guided hints\. Among these approaches, few\-shot prompting is the most scalable and consistently effective strategy, while human\-guided correction demonstrates that some translation failures arise from localized ambiguities rather than a complete lack of geometric reasoning capability\. These findings suggest that improvements in mathematical formalization may require tighter integration between language understanding, geometric reasoning, and symbolic verification\.
Future work could extend NL2AGBench to additional theorem\-proving systems, incorporate geometric diagrams through multimodal formalization, develop specialized auto\-formalization models, and integrate symbolic feedback for iterative self\-correction\.
## References
- \[1\]T\. H\. Trinh, Y\. Wu, Q\. V\. Le, H\. He, and T\. Luong\(2024\)Solving olympiad geometry without human demonstrations\.Nature625,pp\. 476–482\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06747-5)Cited by:[§I](https://arxiv.org/html/2608.28481#S1.p3.1),[§II\-A](https://arxiv.org/html/2608.28481#S2.SS1.p1.1)\.
- \[2\]J\. Berant, A\. Chou, R\. Frostig, and P\. Liang\(2013\)Semantic parsing on freebase from question\-answer pairs\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,pp\. 1533–1544\.Cited by:[§I](https://arxiv.org/html/2608.28481#S1.p4.1)\.
- \[3\]L\. Dong and M\. Lapata\(2016\)Language to logical form with neural attention\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics,pp\. 33–43\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1004)Cited by:[§I](https://arxiv.org/html/2608.28481#S1.p4.1)\.
- \[4\]S\. Chou, X\. Gao, and J\. Zhang\(2000\)A deductive database approach to automated geometry theorem proving and discovering\.Journal of Automated Reasoning25,pp\. 219–246\.Cited by:[§II\-A](https://arxiv.org/html/2608.28481#S2.SS1.p1.1)\.
- \[5\]e\. al\. Duan\(2025\)HAGeo\.Note:Geometry theorem proving and auxiliary\-construction researchCited by:[§II\-A](https://arxiv.org/html/2608.28481#S2.SS1.p1.1)\.
- \[6\]e\. al\. Zhang\(2024\)TongGeometry\.Note:Geometry theorem proving researchCited by:[§II\-A](https://arxiv.org/html/2608.28481#S2.SS1.p1.1)\.
- \[7\]e\. al\. Sicca\(2024\)Newclid\.Note:Open\-source geometry theorem proving systemCited by:[§II\-A](https://arxiv.org/html/2608.28481#S2.SS1.p1.1)\.
- \[8\]Y\. Wu, A\. Q\. Jiang, W\. Li, M\. N\. Rabe, C\. Staats, M\. Jamnik, and C\. Szegedy\(2022\)Autoformalization with large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§II\-B](https://arxiv.org/html/2608.28481#S2.SS2.p1.1)\.
- \[9\]A\. Q\. Jiang, S\. Welleck, J\. P\. Zhou, W\. Li, J\. Liu, M\. Jamnik, T\. Lacroix, Y\. Wu, and G\. Lample\(2023\)Draft, sketch, and prove: guiding formal theorem provers with informal proofs\.InInternational Conference on Learning Representations,Cited by:[§II\-B](https://arxiv.org/html/2608.28481#S2.SS2.p1.1)\.
- \[10\]K\. Zheng, J\. M\. Han, and S\. Polu\(2022\)MiniF2F: a cross\-system benchmark for formal mathematical statement proving\.InInternational Conference on Learning Representations,Cited by:[§II\-B](https://arxiv.org/html/2608.28481#S2.SS2.p1.1)\.
- \[11\]K\. Yang, A\. Swope, A\. Gu, R\. Chalamala, P\. Song, S\. Yu, S\. Godil, R\. Prenger, and A\. Anandkumar\(2023\)LeanDojo: theorem proving with retrieval\-augmented language models\.InAdvances in Neural Information Processing Systems,Cited by:[§II\-B](https://arxiv.org/html/2608.28481#S2.SS2.p1.1)\.
- \[12\]e\. al\. Jana\(2025\)ProofBridge\.Note:Formal mathematical reasoning researchCited by:[§II\-B](https://arxiv.org/html/2608.28481#S2.SS2.p1.1)\.
- \[13\]e\. al\. Hubert\(2025\)AlphaProof\.Note:Formal mathematical theorem proving with reinforcement learningCited by:[§II\-B](https://arxiv.org/html/2608.28481#S2.SS2.p1.1)\.
- \[14\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the math dataset\.InAdvances in Neural Information Processing Systems,Cited by:[§II\-C](https://arxiv.org/html/2608.28481#S2.SS3.p1.1)\.
- \[15\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§II\-C](https://arxiv.org/html/2608.28481#S2.SS3.p1.1)\.
- \[16\]C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.\(2024\)OlympiadBench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.External Links:2402\.14008Cited by:[§II\-C](https://arxiv.org/html/2608.28481#S2.SS3.p1.1)\.
- \[17\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§II\-C](https://arxiv.org/html/2608.28481#S2.SS3.p1.1)\.
- \[18\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.Cited by:[§II\-C](https://arxiv.org/html/2608.28481#S2.SS3.p1.1)\.
- \[19\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2022\)Self\-consistency improves chain of thought reasoning in language models\.External Links:2203\.11171Cited by:[§II\-C](https://arxiv.org/html/2608.28481#S2.SS3.p1.1)\.
- \[20\]T\. P\. G\. H\.\(2024\)AG4MASSes\.Note:Modified AlphaGeometry systemCited by:[§IV\-B](https://arxiv.org/html/2608.28481#S4.SS2.p5.1)\.相似文章
NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation
Introduces NL2SHACL-Bench, a benchmark suite for translating natural language requirements into SHACL shapes, evaluating four state-of-the-art LLMs and showing they struggle with semantic equivalence for complex patterns.
A2RBench:一种自动化的可形式化验证抽象推理基准生成范式
本文介绍了A2RBench,一个用于为LLM生成可形式化验证的抽象推理基准的自动化流水线,它利用循环一致性来确保唯一解,并揭示当前LLM在3D推理任务上显著落后于人类。
LinAlg-Bench:揭示大语言模型数学推理中结构性失败模式的诊断性基准
介绍了LinAlg-Bench,这是一个诊断性基准,用于评估10个前沿大语言模型在矩阵维度上的结构化线性代数计算,揭示了大语言模型的数学失败在结构上受到约束,并在4x4规模下从执行错误过渡到计算放弃。
自然语言数学证明的高性价比自动评判
本文研究廉价的开放权重大语言模型能否以远低于前沿模型的成本,同样可靠地评判自然语言数学证明。在 IMO-GradingBench 上,三个廉价评判模型在通过/不通过的一致性上与前沿模型相当,作者推荐“三者全部通过”的一致性规则以实现高性价比部署。
LEAP:利用代理框架增强LLMs在形式数学中的能力
LEAP是一种代理框架,使通用LLMs能够在Lean中实现形式定理证明的最新性能,解决了2025年普特南竞赛的全部12个问题,并在新基准(Lean-IMO-Bench)上将形式化证明率从低于10%提升至70%,超越了专门系统。