RAS: Reflection-Augmented Scaling with In-Context Learning for Executable Cypher Query Generation

# RAS: Reflection-Augmented Scaling with In-Context Learning for Executable Cypher Query Generation
Source: [https://arxiv.org/html/2605.22937](https://arxiv.org/html/2605.22937)
Minseok Jung1,∗, Abhas Ricky1, Muhammad Rameez Chatni1 1Cloudera 6220 America Center Dr\. San Jose, CA 95002 \{minseok\.jung, abhas, mchatni\}@cloudera\.com

###### Abstract

Inference\-time scaling can reduce errors in structured query generation, but methods to allocate the compute for query code generation remains underexplored\. We study Text2Cypher, where language models generate Cypher queries that execute against property graph databases\. Non\-executable queries constitute a distinct syntactic failure separate from semantic inaccuracy: a syntax error triggers a system\-generated error message from the database\. These error messages are typically discarded at inference time rather than leveraged through in\-context learning \(ICL\)\. We compare two inference methods: Independent Scaling \(IS\), which performs memoryless resampling, and Reflection\-Augmented Scaling \(RAS\), which conditions each new attempt on prior execution feedback via ICL\. Across three Neo4j datasets and five code\-specialized language models, RAS reduces the Query Execution Error Rate by 41–50% atn=5n\{=\}5, outperforming IS at 32–38%\. Execution errors are not merely failures to discard but actionable feedback, and structuring inference\-time compute around them is a more efficient path to executability than scaling independent samples\.

RAS: Reflection\-Augmented Scaling with In\-Context Learning for Executable Cypher Query Generation

Keywords:Graph Databases, Coder Models, Cypher, Query Generation, Knowledge Graphs, Neo4j, Inference Scaling

## 1Introduction

Recent work has shown that allocating additional compute at inference time can substantially improve language model performance on complex reasoning and generation tasksSnellet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib16); Brownet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib12)\. Techniques such as scaling self\-consistency\(Wanget al\.,[2023](https://arxiv.org/html/2605.22937#bib.bib14)\)and repeated sampling\(Brownet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib12)\)increase the probability of producing better outputs without modifying model parameters\(Wu and others,[2024](https://arxiv.org/html/2605.22937#bib.bib13)\)\.

For an efficient scaling, a complementary line of work studies feedback\-driven refinement, where models improve outputs using signals from prior attempts\. Methods such as Self\-Refine\(Madaanet al\.,[2023](https://arxiv.org/html/2605.22937#bib.bib17)\), Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2605.22937#bib.bib18)\), and self\-debugging\(Chenet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib15)\)show that self\-critique or execution feedback can guide models toward better generation\.

![Refer to caption](https://arxiv.org/html/2605.22937v1/latex/figures/fig1.png)Figure 1:Our method iteratively generates queries and executes them against the database\. When execution fails, the system incorporates execution feedback through reflection\-based in\-context learning \(ICL\) to refine subsequent generations\. Increasing the inference scale expands the reflection context and improves the probability of producing executable queries![Refer to caption](https://arxiv.org/html/2605.22937v1/latex/figures/fig2.png)Figure 2:Reflection\-augmented inference\-time scaling for Text2Cypher\.Syntax and schema errors surface to users as execution failures that break interactive query workflows, making them a distinct and user\-visible failure mode\.Left:under single\-pass generation, a Cypher query is produced and executed against the graph database; execution failures trigger execution\-aware reflection via in\-context learning\.Right:as the inference scale increases, the framework iteratively refines query code by conditioning on feedback from prior failures to reduce syntax errors\.We study this question in structured generation, where outputs must satisfy strict syntactic and schema constraints\. In query code generation, models frequently produce invalid code due to schema mismatches and compositional complexity\(Gaoet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib6); Lianget al\.,[2021](https://arxiv.org/html/2605.22937#bib.bib20); Hainset al\.,[2023](https://arxiv.org/html/2605.22937#bib.bib21)\)\. Scaling methods have been invited\(Shinnet al\.,[2023](https://arxiv.org/html/2605.22937#bib.bib18); Chenet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib15)\)to overcome this limitation but they did not focus on the executability, which is prerequisite for the accuracy\. We introduceReflection\-Augmented Scaling \(RAS\), which conditions each new attempt on execution feedback from prior failures, and compare two inference\-time strategies for Text2Cypher:Independent Scaling \(IS\), which repeatedly samples without memory, andRAS\.

Across three Neo4j datasets and five code\-specialized language models, RAS consistently outperforms IS under the same compute budget\. Atn=5n=5, RAS achieves an 41–50% in QER, compared to 32–38% for IS\. Our results show that the effectiveness of inference\-time scaling depends not only on how much compute is used, but on how inference strategy is integrated\.

![Refer to caption](https://arxiv.org/html/2605.22937v1/latex/figures/error_band.png)Figure 3:Query Error Rate \(QER\) across three graph datasets under inference\-time scaling\. Q@1 denotes baseline single\-pass error \(n=1n\{=\}1\), while IS@5 and RAS@5 report error after scaling ton=5n\{=\}5using simple re\-run and reflection\-augmented strategies\. Reflection consistently achieves larger absolute reductions in QER\. Values are averaged across three query complexities\.
## 2Related Work

#### Semantic parsing for SQL\.

Translating natural language into executable database queries has long been studied under the framework of semantic parsing\(Berantet al\.,[2013](https://arxiv.org/html/2605.22937#bib.bib1); Yihet al\.,[2016](https://arxiv.org/html/2605.22937#bib.bib2)\)\. In relational settings, the Text2SQL task has benefited from standardized benchmarks such as SpiderYuet al\.\([2018](https://arxiv.org/html/2605.22937#bib.bib3)\), enabling systematic evaluation and rapid progress with neural and large language model–based approachesWanget al\.\([2020](https://arxiv.org/html/2605.22937#bib.bib4)\); Scholaket al\.\([2021](https://arxiv.org/html/2605.22937#bib.bib5)\); Gaoet al\.\([2024](https://arxiv.org/html/2605.22937#bib.bib6)\); Pourreza and Rafiei \([2023](https://arxiv.org/html/2605.22937#bib.bib7)\)\. Recent work further explores the use of language models as general interfaces to structured databases through in\-context learning and agentic workflowsDonget al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib8)\)\. Despite strong performance on relational schemas, these approaches do not transfer directly to graph query languages, which impose fundamentally different structural constraints\.

#### Semantic parsing for Cypher\.

Semantic parsing for graph databases remains comparatively underexplored\. Graph query languages such as Cypher require explicit reasoning over heterogeneous node and edge properties, direction\-sensitive relationships, and compositional multi\-hop traversal patterns\. Compared with SQL, graph queries introduce additional structural constraints including path pattern matching and schema sparsity, which increase the likelihood of syntax and execution errors\. Prior studies investigate natural\-language access to knowledge graphs via SPARQL generationLanet al\.\([2021](https://arxiv.org/html/2605.22937#bib.bib9)\), and more recent work explores graph\-aware retrieval and fine\-tuning approaches for property graph databasesClemedtson and others \([2025](https://arxiv.org/html/2605.22937#bib.bib10)\); Sivasubramaniam and others \([2024](https://arxiv.org/html/2605.22937#bib.bib11)\)\. Nevertheless, the dominant inference paradigm remains single forward\-pass generation, without leveraging additional inference\-time computation to recover from execution failures\.

#### Inference\-time scaling via independent resampling\.

An alternative to improving single\-pass generation is to allocate additional inference\-time computation through repeated sampling\. Test\-time compute scaling increases the effective reasoning budget without modifying model parameters, and recent studies show that such strategies can yield substantial gains on complex reasoning tasks\(Snellet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib16); Brownet al\.,[2024](https://arxiv.org/html/2605.22937#bib.bib12); Wu and others,[2024](https://arxiv.org/html/2605.22937#bib.bib13)\)\. Self\-consistency\(Wanget al\.,[2023](https://arxiv.org/html/2605.22937#bib.bib14)\)demonstrates that sampling multiple reasoning paths and selecting by majority vote consistently outperforms greedy decoding\. In structured query generation, repeated sampling reduces error rates by exploiting stochastic variation across independent outputs\. However, independent resampling maintains a fixed conditioning distribution over the input question and database schema, without incorporating evidence from prior execution failures\. Consequently, repeated samples tend to reproduce similar structural error modes — such as schema misalignment, incorrect relationship directionality, or invalid aggregation constructs — limiting the gains achievable through scaling alone\.

#### Execution\-grounded reflection and iterative refinement\.

A more targeted strategy is to condition subsequent generations on feedback from earlier failures, enabling iterative correction rather than independent exploration\. Self\-RefineMadaanet al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib17)\)and ReflexionShinnet al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib18)\)propose self\-refinement loops in which models critique and revise their own outputs, or maintain episodic memory of unsuccessful attempts in agentic reasoning settings\. Self\-DebuggingChenet al\.\([2024](https://arxiv.org/html/2605.22937#bib.bib15)\)demonstrates that program execution feedback can guide iterative repair on Text2SQL tasks, improving performance particularly on harder queries\. While these methods highlight the value of feedback\-driven reasoning, they have not been systematically studied in the context of schema\-constrained Cypher generation against live property graph databases\. Furthermore, prior work does not directly compare independent resampling and feedback\-driven refinement under controlled inference\-time compute budgets\. Our work addresses these gaps by formalizing execution\-aware reflection for Text2Cypher generation and providing a systematic empirical comparison of both strategies across diverse property graph datasets and query complexity levels\.

## 3Problem Formulation and Modeling

### 3\.1Text2Cypher as Structured Graph Query Generation

We formulate Text2Cypher as a structured semantic parsing task that maps a natural language question to an executable Cypher query\. Letx∈𝒳x\\in\\mathcal\{X\}denote a natural language input and𝒮\\mathcal\{S\}denote the database schema\. The objective is to generate a queryq∈𝒬q\\in\\mathcal\{Q\}, where𝒬\\mathcal\{Q\}denotes the space of syntactically valid Cypher queries over a database instanceGGconsistent with𝒮\\mathcal\{S\}\.

We model query generation as conditional structured generationScholaket al\.\([2021](https://arxiv.org/html/2605.22937#bib.bib5)\); Wanget al\.\([2020](https://arxiv.org/html/2605.22937#bib.bib4)\):

q∼pθ​\(q∣x,𝒮\),q\\sim p\_\{\\theta\}\(q\\mid x,\\mathcal\{S\}\),\(1\)wherepθp\_\{\\theta\}denotes the conditional distribution induced by an autoregressive language model with parametersθ\\theta, defining a probability measure over token sequences representing candidate Cypher programs\. Syntactic validity and executability are evaluated only after full decoding\. Unlike unconstrained text generation, Cypher queries must satisfy strict structural constraints imposed by the graph schema; small deviations — such as incorrect relationship directionality or references to invalid node properties — result in immediate execution failure\.

### 3\.2Executability and Accuracy

Executability and accuracy measure two conceptually distinct aspects of a generated query\.*Executability*refers to whether a query can be successfully processed by the database engine without raising syntax or schema errors\.*Accuracy*, in contrast, measures semantic correctness — whether the execution result matches the ground\-truth answer\.

A query may be executable yet semantically incorrect \(e\.g\., due to a missing filter or an incorrect aggregation\)\. However, a non\-executable query cannot be accurate, since it fails before producing any result\. This structural asymmetry motivates treating execution failure as a distinct and separable source of error\.

We formalize this distinction via two binary random variables defined for a generated query:

E∈\{0,1\}\(syntax\),A∈\{0,1\}\(semantic\)E\\in\\\{0,1\\\}\\quad\\text\{\(syntax\)\},\\quad A\\in\\\{0,1\\\}\\quad\\text\{\(semantic\)\}whereE=1E=1denotes a successfully executed query andE=0E=0denotes execution failure due to syntactic or schema violations \(e\.g\., undefined node labels, invalid relationship types, or malformed path syntax\), whileA=1A=1denotes a result matching the ground\-truth answer andA=0A=0denotes a semantic error\. Crucially,EEcaptures*structural validity*andAAcaptures*semantic fidelity*: a query may satisfyE=1E=1while failingA=1A=1, butE=0E=0guaranteesA=0A=0since no result is produced\.

We define the joint probabilities as

Pi​j=ℙ​\(E=i,A=j\),P\_\{ij\}=\\mathbb\{P\}\(E=i,\\,A=j\),summarized in Table[1](https://arxiv.org/html/2605.22937#S3.T1)\. Since a non\-executable query cannot be accurate,P01P\_\{01\}is undefined\. The execution failure probability is thereforeℙ​\(E=0\)=P00\\mathbb\{P\}\(E=0\)=P\_\{00\}, and this is the signal used to trigger the reflection loop in Algorithm[2](https://arxiv.org/html/2605.22937#alg2)\. The objective of the RAS algorithm is to decreaseP00P\_\{00\}, thereby increasing the total executable massP​\(E=1\)=P11\+P10P\(E=1\)=P\_\{11\}\+P\_\{10\}\. This paper focuses on reducing execution failure rather than directly optimizing the allocation between semantically correct executions \(P11P\_\{11\}\) and semantically incorrect but executable queries \(P10P\_\{10\}\)\.

Table 1:Joint distribution of executability \(EE\) and semantic accuracy \(AA\)\. The cellP01P\_\{01\}is undefined since a non\-executable query cannot produce a correct result\.
### 3\.3Query Execution Error

Given a generated queryqqand a fixed database instanceGG, execution is deterministic\. We write

\(r,m\)=ℰ​\(q,G\),\(r,\\,m\)=\\mathcal\{E\}\(q,G\),\(2\)whererris the query result returned by the database engine andmmis the accompanying execution status message \(e\.g\., a success confirmation or a structured error diagnostic\)\. AccuracyAAis evaluated againstrr, while executabilityEEis determined bymm: a query fails \(E=0E=0\) if and only ifmmindicates an error, in which caserris undefined\.

We define the*Query Execution Error*\(QEE\) indicator as

QEE​\(q,G\):=1​\[m∈ℳerror\],\\mathrm\{QEE\}\(q,G\)\\;:=\\;\\mathbf\{1\}\\\!\\left\[m\\in\\mathcal\{M\}\_\{\\mathrm\{error\}\}\\right\],\(3\)whereℳerror\\mathcal\{M\}\_\{\\mathrm\{error\}\}denotes the set of error\-class status messages\. Given an input distribution𝒟\\mathcal\{D\}overxx, the*Query Execution Error Rate*\(QER\) is

QER:=𝔼x∼𝒟​𝔼q∼pθ\(⋅∣x,𝒮\)​\[QEE​\(q,G\)\]\.\\mathrm\{QER\}\\;:=\\;\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\;\\mathbb\{E\}\_\{q\\sim p\_\{\\theta\}\(\\cdot\\mid x,\\mathcal\{S\}\)\}\\\!\\left\[\\mathrm\{QEE\}\(q,G\)\\right\]\.\(4\)ThusQER\\mathrm\{QER\}equalsℙ​\(E=0\)\\mathbb\{P\}\(E=0\)and is estimated empirically as the fraction of queries that fail execution across all inputs\.

From a system perspective, executability acts as a hard constraint on downstream correctness: when E = 0, no result is produced, and the retrieval stage in RAG pipelines fails entirely\. Consequently, reducing P\(E = 0\) is not only a syntactic objective but a prerequisite for enabling semantically grounded responses\.

### 3\.4Inference\-Time Scaling Strategies

We study two inference\-time strategies under a fixed compute budgetTT: \(i\)*Independent Scaling*\(IS\) and \(ii\)*Reflection\-Augmented Scaling*\(RAS\)\. Both allocate additional test\-time computation without updating model parameters\.

#### Independent Scaling \(IS\)\.

IS repeatedly samples from the same conditional distributionpθ\(⋅∣x,𝒮\)p\_\{\\theta\}\(\\cdot\\mid x,\\mathcal\{S\}\)until an executable query is obtained or the budgetTTis exhausted, as formalized in Algorithm[1](https://arxiv.org/html/2605.22937#alg1)\. Each sample is generated independently, without incorporating information from prior execution failuresWanget al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib14)\); Brownet al\.\([2024](https://arxiv.org/html/2605.22937#bib.bib12)\)\. If no executable query is found withinTTattempts, the final sampled query is returned as a fallback\.

Algorithm 1Independent Scaling \(IS\)1:Input

xx, schema

𝒮\\mathcal\{S\}, graph

GG, budget

TT
2:

qlast←∅q\_\{\\text\{last\}\}\\leftarrow\\varnothing
3:for

t=1t=1to

TTdo

4:

qt∼pθ\(⋅∣x,𝒮\)q\_\{t\}\\sim p\_\{\\theta\}\(\\cdot\\mid x,\\mathcal\{S\}\)
5:

qlast←qtq\_\{\\text\{last\}\}\\leftarrow q\_\{t\}
6:

mt←ℰ​\(qt,G\)m\_\{t\}\\leftarrow\\mathcal\{E\}\(q\_\{t\},G\)
7:if

mt∉ℳerrorm\_\{t\}\\notin\\mathcal\{M\}\_\{\\mathrm\{error\}\}then

8:return

qtq\_\{t\}
9:endif

10:endfor

11:return

qlastq\_\{\\text\{last\}\}

#### Reflection\-Augmented Scaling \(RAS\)\.

RAS augments the generation context with execution feedback from prior failed attempts, enabling iterative targeted correctionMadaanet al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib17)\); Shinnet al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib18)\)\. After each execution failure, the failed query and its error message are appended to the conditioning context𝒞\\mathcal\{C\}\. Subsequent generations condition on this expanded context, allowing the model to identify and correct specific structural errors\. Algorithm[2](https://arxiv.org/html/2605.22937#alg2)formalizes this procedure\. As with IS, if no executable query is produced withinTTattempts, the final generated query is returned as a fallback\.

Algorithm 2Reflection\-Augmented Scaling \(RAS\)1:Input

xx, schema

𝒮\\mathcal\{S\}, graph

GG, budget

TT
2:

𝒞←\(x,𝒮\)\\mathcal\{C\}\\leftarrow\(x,\\mathcal\{S\}\)
3:

qlast←∅q\_\{\\text\{last\}\}\\leftarrow\\varnothing
4:for

t=1t=1to

TTdo

5:

qt∼pθ\(⋅∣𝒞\)q\_\{t\}\\sim p\_\{\\theta\}\(\\cdot\\mid\\mathcal\{C\}\)
6:

qlast←qtq\_\{\\text\{last\}\}\\leftarrow q\_\{t\}
7:

mt←ℰ​\(qt,G\)m\_\{t\}\\leftarrow\\mathcal\{E\}\(q\_\{t\},G\)
8:if

mt∉ℳerrorm\_\{t\}\\notin\\mathcal\{M\}\_\{\\mathrm\{error\}\}then

9:return

qtq\_\{t\}
10:endif

11:

𝒞←𝒞⋅\(qt,mt\)\\mathcal\{C\}\\leftarrow\\mathcal\{C\}\\cdot\(q\_\{t\},m\_\{t\}\)⊳\\trianglerightAppend failure to context

12:endfor

13:return

qlastq\_\{\\text\{last\}\}

## 4Experimental Setup

### 4\.1Datasets

Table 2:Statistics of the Neo4j property\-graph datasets used in our Text2Cypher evaluation\. The datasets vary substantially in scale and structural density, enabling experiments across heterogeneous schema structures\.We evaluate our approach on three publicly available Neo4j property graph databases spanning diverse scales and structural characteristics:Healthcare,Fraud, andCrime\(Table[2](https://arxiv.org/html/2605.22937#S4.T2)\)\. These datasets vary substantially in graph size, schema complexity, and query topology, enabling robustness assessment across heterogeneous graph structures\.

Healthcareis derived from FDA adverse event reporting data and contains heterogeneous biomedical entities including drugs, patients, and reported reactions\. Queries over this graph frequently require multi\-entity joins and attribute\-based filtering, stressing schema alignment and type consistency during generation\.Fraudrepresents a large\-scale synthetic financial transaction network characterized by high connectivity and long relational paths\. The dense topology increases the likelihood of structural errors in generated Cypher queries, particularly in aggregation and multi\-hop traversal patterns\.Crimefollows the investigative POLE \(Person–Object–Location–Event\) data model used in law\-enforcement analytics, exhibiting moderate scale but complex relationship directionality and path constraints\.

### 4\.2Scope of Evaluation

This experiment evaluates execution reliability rather than full semantic correctness\. Our metric, QER, measures whether a generated Cypher query can be executed successfully against the target GDB\. This choice is deliberate: in graph\-based RAG and agentic database systems, non\-executable queries represent a hard failure mode because no evidence can be retrieved for downstream reasoning before measuring accuracy\. To put it another way, executability is a necessary condition for accuracy despite it is not a sufficient condition\. Although query code executability is not a substitute for semantic accuracy, it is a required precondition\.

### 4\.3Question Generation

For each dataset, natural language questions are generated at three complexity levels \(Easy,Medium, andHard\), conditioned on the graph schema and representative subgraph samples\. Full prompt templates and representative examples are provided in the Appendix\.

### 4\.4Text2Cypher

We implement Text2Cypher using LangChain’sGraphCypherQAChain, which constructs a structured generation prompt by injecting the database schema𝒮\\mathcal\{S\}retrieved directly from the Neo4j instance\. To isolate the query generation stage and focus directly on QEE, we do not evaluate downstream answer for accuracy\. Each generated query is executed against the live database instance to determine executability as defined in §[3\.2](https://arxiv.org/html/2605.22937#S3.SS2)\.

### 4\.5Models

We evaluate five open\-weight, code\-specialized language models:CodeLlama\-7BandCodeLlama\-13BRozièreet al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib26)\),DeepSeek\-Coder\-6\.7BGuoet al\.\([2024](https://arxiv.org/html/2605.22937#bib.bib27)\),Qwen2\.5\-Coder\-7BHuiet al\.\([2024](https://arxiv.org/html/2605.22937#bib.bib28)\), andStarCoder2\-7BLozhkovet al\.\([2024](https://arxiv.org/html/2605.22937#bib.bib29)\)\. These models are pretrained on large\-scale source code corpora, inducing a structural prior toward syntactically well\-formed token sequences\. We deliberately focus on small language models \(SLMs\) to reduce the latency of the inference stage\. All models are evaluated under identical schema injection, decoding configuration, and execution\-validation settings within the sameGraphCypherQAChainpipeline\.

### 4\.6Scaled Context Construction via In\-Context Learning

We instantiate the two inference methods \(Algorithms[1](https://arxiv.org/html/2605.22937#alg1)and[2](https://arxiv.org/html/2605.22937#alg2)\) at scale\. For our method, Alg\.[2](https://arxiv.org/html/2605.22937#alg2), the initial context𝒞0=\(x,𝒮\)\\mathcal\{C\}\_\{0\}=\(x,\\mathcal\{S\}\)is constructed by injecting the schema𝒮\\mathcal\{S\}directly from the Neo4j instance into a structured Cypher generation prompt\. For the RAS, each failed attempt appends the generated queryqtq\_\{t\}and its execution errormtm\_\{t\}to the prompt, along with a corrective instruction, forming𝒞t=𝒞t−1∪\{qt,mt\}\\mathcal\{C\}\_\{t\}=\\mathcal\{C\}\_\{t\-1\}\\cup\\\{q\_\{t\},m\_\{t\}\\\}\. This accumulated context is embedded into the prompt, preserving compatibility with the system architecture\. Alg\.[1](https://arxiv.org/html/2605.22937#alg1)keeps generating answers until it succeeds\. AtT=1T\{=\}1, both strategies are based on the identical single\-pass baseline, but the scaling with RAS keeps adding query code with system errors\.

### 4\.7Hyperparameters

Each experimental configuration — defined by a dataset, model, complexity level, and inference strategy — is evaluated over 128 independent runs\. We report the mean QER along with its standard deviationσQER\\sigma\_\{\\mathrm\{QER\}\}across runs\. All decoding is performed at temperatureτ=0\.9\\tau=0\.9to introduce sufficient stochastic diversity across trials while maintaining generation qualityWanget al\.\([2023](https://arxiv.org/html/2605.22937#bib.bib14)\)\. The inference scale is fixed atT=5T=5, selected via multi\-objective optimization \(MOO\) on the trade\-off between error reduction, latency and compute cost with the optimality identified at the knee point of Pareto frontiers\(Junget al\.,[2025](https://arxiv.org/html/2605.22937#bib.bib35)\)\.

Evaluation is based on the query execution that has been introduced in Table[3\.2](https://arxiv.org/html/2605.22937#S3.SS2)\. Additional information about the prompts by complexity and results by complexity are provided in Appendix[A\.1](https://arxiv.org/html/2605.22937#A1.SS1)and Appendix[A\.2](https://arxiv.org/html/2605.22937#A1.SS2)\.

Table 3:Query Execution Error Rate \(QER\) across three graph datasets and five coder models\. Q@1 denotes baseline error at a single generation \(n=1n\{=\}1\), computed as the average of IS@1 and RAS@1 across Easy, Medium, and Hard complexity levels\. IS@5 and RAS@5 denote error after test\-time scaling ton=5n\{=\}5using IS \(Simple Retrials\) and RAS \(Reflection\-Augmented Scaling\)\.Δ\\DeltaIS andΔ\\DeltaRAS represent reductions in QER relative to Q@1\. The average values are rounded up into two decimal points\. Detailed experimenta result is in Appendix[4](https://arxiv.org/html/2605.22937#A1.T4)\.

## 5Results

### 5\.1Inference Scaling Consistently Reduces QER

Table[3](https://arxiv.org/html/2605.22937#S4.T3)reports QER under single\-shot generation \(n=1n\{=\}1\) and scaled inference \(n=5n\{=\}5\) for both IS and RAS\. Across all datasets and models, increasing the inference scale reduces QER regardless of the inference methods\. The Q@1 is the average of the two methods, and both showed a similar error rate at the initial stage because the ICL of RAS is not applicable at RAS@1\.

### 5\.2RAS Outperforms Independent Scaling

At scalen=1n\{=\}1, both strategies shows similar QER, as no execution feedback has yet been incorporated\. However, atn=5n\{=\}5, RAS achieves QER of 3–6% across most configurations, representing a reduction of more than 10% over IS\. This gap demonstrates that execution feedback constitutes a qualitatively stronger inference\-time signal than independent resampling\. Rather than simply retrying query generations, RAS actively incorporates information from prior execution failures to steer subsequent generation toward executable queries\. The advantage of RAS over IS is consistent across all datasets, model architectures, and input complexities\.

### 5\.3QER Scaling Trajectories Differ Between Strategies

Beyond endpoint comparisons, Figure[3](https://arxiv.org/html/2605.22937#S1.F3)reveals a difference in how the two strategies reduce QER across the inference budget\. IS exhibits an approximately linear decrease in QER as the scale grows from 1 to 5, suggesting that each additional independent trial contributes a roughly constant marginal gain in executability\. RAS, by contrast, follows a concave trajectory: the sharpest drop occurs atn=2n\{=\}2, the first point at which execution feedback from a failed attempt becomes available, after which further improvements taper gradually\. This suggests that the primary benefit of execution\-grounded reflection is concentrated in the earliest feedback step, while subsequent iterations yield diminishing returns\.

### 5\.4RAS is Robust Across Datasets, Models, and Query Complexities

A critical requirement for any inference\-time strategy is that its gains generalize beyond a narrow set of conditions\. Table[3](https://arxiv.org/html/2605.22937#S4.T3)demonstrates that RAS show a greater QER decrease consistently across datasets and models\. Across all three datasets — Healthcare, Fraud, and Crime — RAS reduces QER to 0\.03–0\.06 atn=5n\{=\}5, despite differences are observed by graph topology, schema complexity, and domain vocabulary\. Robustness also holds across model architectures\. RAS achieves consistent gains over all five code\-specialized models spanning\. This uniformity suggests that the feedback mechanism is not dependent on any particular model’s prior capacity for schema\-aligned generation\. Further, the standard deviation of QER \(σQER\\sigma\_\{\\mathrm\{QER\}\}\) and consistency across input complexities reported in Table[4](https://arxiv.org/html/2605.22937#A1.T4)provides further evidence of the robustness\. Crucially, these values are consistent across model architectures and datasets with no outlying configurations, indicating that RAS produces reliably low error rates across independent runs rather than occasional successes masking high variance\. The concrete failure signal incorporated at each reflection step guides generation toward a stable, executable region of the query space, constraining the stochastic variation inherent in autoregressive decoding\.

## 6Conclusion

This paper studies inference\-time scaling for Text2Cypher generation by formalizing and comparing two strategies: Independent Scaling \(IS\) and Reflection\-Augmented Scaling \(RAS\)\. We evaluate both approaches under a fixed compute budget across three Neo4j property graph datasets, five code\-specialized language models, and multiple query complexity levels\.

Our results reveal that IS and RAS exhibit fundamentally different scaling behaviors\. IS reduces the Query Execution Error Rate \(QER\) in an approximately linear manner, where each additional sample provides a consistent marginal improvement through stochastic diversity\. In contrast, RAS follows a concave trajectory, achieving its largest reduction atn=2n\{=\}2, when execution feedback first becomes available, followed by diminishing returns\. This two\-phase behavior highlights a key asymmetry: the primary benefit of reflection is concentrated in the earliest feedback\-driven correction step rather than in prolonged iterative refinement\.

Empirically, RAS consistently outperforms IS under identical compute budgets, achieving substantially lower QER across all datasets and models\. Notably, no IS configuration matches the performance of RAS at the same inference scale\. These findings demonstrate that the structure of inference\-time computation—specifically, whether it incorporates feedback—can be more critical than increasing model size or the number of independent samples\.

Overall, our results suggest that effective inference\-time scaling for structured generation requires moving beyond independent resampling toward feedback\-driven strategies\. Execution feedback provides a structured and informative signal that transforms the correction process from stochastic exploration into targeted refinement\. Understanding how to optimally allocate compute across such feedback mechanisms has direct implications for deploying reliable and cost\-efficient language model systems in graph query generation\.

Although our experiments focus on Text2Cypher, the proposed framework is not limited to graph query languages\. The same execution\-grounded reflection mechanism can be extended to other structured semantic parsing tasks, such as Text2SQL, where outputs must satisfy strict syntactic and schema constraints\. More broadly, our findings point toward a general paradigm in which verifiable feedback signals can improve inference\-time scaling in program synthesis and agentic systems\.

## 7Limitations

This work has several limitations\. First, our evaluation is restricted to Text2Cypher on Neo4j property graphs; inference scaling dynamics may differ for other semantic parsing tasks \(e\.g\., Text2SQL\) or database engines with distinct execution semantics\.

Second, we cap the inference budget atn=5n\{=\}5, reflecting a practical performance–latency trade\-off; larger budgets may yield diminishing returns, higher latency, or over\-correction from compounding feedback\.

Third, we focus on open\-weight, code\-specialized models for reproducibility, leaving the scaling behavior of larger proprietary models as an open question\.

Fourth, we evaluate performance primarily through query executability\. Although executability does not guarantee semantic correctness, we treat it as a prerequisite reliability condition: in graph\-based RAG and agentic systems, a non\-executable query halts evidence retrieval entirely, causing the system to fail before any answer can be grounded in data\. Reducing execution errors is therefore a necessary first step toward reliable downstream reasoning, and future work should extend this framework to jointly optimize executability and semantic fidelity\.

Finally, our reliance on in\-context learning to incorporate execution feedback introduces token and latency overhead as the reflection context grows\. Execution traces are also used transiently rather than persisted; aggregating these logs could enable system\-level learning across queries beyond single\-session inference\.

## Acknowledgment

Thanks to Nashua Springberry, Michael Schuler, Sreenath Somarajapuram, and Neelabh Pant for constructive comments\.

## References

- Semantic parsing on Freebase from question\-answer pairs\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,pp\. 1533–1544\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. Mirhoseini \(2024\)Large language monkeys: scaling inference compute with repeated sampling\.arXiv preprint arXiv:2407\.21787\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p1.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2605.22937#S3.SS4.SSS0.Px1.p1.3)\.
- X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou \(2024\)Teaching large language models to self\-debug\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p2.1),[§1](https://arxiv.org/html/2605.22937#S1.p3.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px4.p1.1)\.
- G\. Clemedtsonet al\.\(2025\)GraphRAFT: retrieval augmented fine\-tuning for knowledge graphs on graph databases\.arXiv preprint arXiv:2504\.05478\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Dong, C\. Zhang, Y\. Ge, Y\. Mao, Y\. Gao, J\. Lin, D\. Lou,et al\.\(2023\)C3: zero\-shot text\-to\-sql with chatgpt\.arXiv preprint arXiv:2307\.07306\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Gao, H\. Wang, Y\. Li, X\. Sun, Y\. Qian, B\. Ding, and J\. Zhou \(2024\)Text\-to\-sql empowered by large language models: a benchmark evaluation\.Proceedings of the VLDB Endowment17\(5\),pp\. 1132–1145\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p3.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Guo, Q\. Zhu, D\. Yang, Z\. Xie, K\. Dong, W\. Zhang, G\. Chen, X\. Bi, Y\. Wu, Y\. K\. Li, F\. Luo, Y\. Xiong, and W\. Liang \(2024\)DeepSeek\-coder: when the large language model meets programming – the rise of code intelligence\.arXiv preprint arXiv:2401\.14196\.Cited by:[§4\.5](https://arxiv.org/html/2605.22937#S4.SS5.p1.1)\.
- G\. J\. D\. R\. Hains, Y\. Khmelevsky, and T\. Tachon \(2023\)From natural language to graph queries\.In2023 IEEE 19th International Conference on Software Engineering Research, Management and Applications \(SERA\),pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p3.1)\.
- B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Dang,et al\.\(2024\)Qwen2\.5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§4\.5](https://arxiv.org/html/2605.22937#S4.SS5.p1.1)\.
- M\. Jung, A\. Ricky, and M\. R\. Chatni \(2025\)3D optimization for ai inference scaling: balancing accuracy, cost, and latency\.arXiv preprint arXiv:2510\.18905\.Cited by:[§4\.7](https://arxiv.org/html/2605.22937#S4.SS7.p1.3)\.
- Y\. Lan, G\. He, J\. Jiang, J\. Jiang, W\. X\. Zhao, and J\. Wen \(2021\)A survey on complex knowledge base question answering: methods, challenges and solutions\.arXiv preprint arXiv:2105\.11644\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Liang, K\. Stockinger, T\. M\. de Farias, M\. Anisimova, and M\. Gil \(2021\)Querying knowledge graphs in natural language\.Journal of Big Data8\(1\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p3.1)\.
- A\. Lozhkov, R\. Li, L\. B\. Allal, F\. Cassano, J\. Lamy\-Poirier, N\. Tazi, A\. Tang, V\. Pyatkin,et al\.\(2024\)StarCoder 2 and the stack v2: the next generation\.InThe Twelfth International Conference on Learning Representations,Cited by:[§4\.5](https://arxiv.org/html/2605.22937#S4.SS5.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p2.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px4.p1.1),[§3\.4](https://arxiv.org/html/2605.22937#S3.SS4.SSS0.Px2.p1.2)\.
- M\. Pourreza and D\. Rafiei \(2023\)DIN\-sql: decomposed in\-context learning of text\-to\-sql with self\-correction\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Rozière, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez, J\. Rapin, A\. Kozhevnikov, I\. Evtimov, J\. Bitton, M\. Bhatt, C\. C\. Ferrer, A\. Grattafiori, W\. Xiong, A\. Défossez, J\. Copet, F\. Azhar, H\. Touvron, L\. Martin, N\. Usunier, T\. Scialom, and G\. Synnaeve \(2023\)Code Llama: open foundation models for code\.arXiv preprint arXiv:2308\.12950\.Cited by:[§4\.5](https://arxiv.org/html/2605.22937#S4.SS5.p1.1)\.
- T\. Scholak, N\. Schucher, and D\. Bahdanau \(2021\)PICARD: parsing incrementally for constrained auto\-regressive decoding from language models\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 9895–9901\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2605.22937#S3.SS1.p2.3)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. R\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p2.1),[§1](https://arxiv.org/html/2605.22937#S1.p3.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px4.p1.1),[§3\.4](https://arxiv.org/html/2605.22937#S3.SS4.SSS0.Px2.p1.2)\.
- M\. Sivasubramaniamet al\.\(2024\)SM3\-Text\-to\-Query: supervising, scaling and synthesizing data for few\-shot text\-to\-query\.arXiv preprint arXiv:2411\.05521\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling llm test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p1.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Wang, R\. Shin, X\. Liu, O\. Polozov, and M\. Richardson \(2020\)RAT\-SQL: relation\-aware schema encoding and linking for text\-to\-sql parsers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 7567–7578\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2605.22937#S3.SS1.p2.3)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p1.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2605.22937#S3.SS4.SSS0.Px1.p1.3),[§4\.7](https://arxiv.org/html/2605.22937#S4.SS7.p1.3)\.
- Y\. Wuet al\.\(2024\)Inference scaling laws: an empirical analysis of compute\-optimal inference for problem\-solving with language models\.arXiv preprint arXiv:2408\.00724\.Cited by:[§1](https://arxiv.org/html/2605.22937#S1.p1.1),[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Yih, M\. Richardson, C\. Meek, M\. Chang, and J\. Suh \(2016\)The value of semantic parse labeling for knowledge base question answering\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics,pp\. 201–206\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Yu, R\. Zhang, K\. Yang, M\. Yasunaga, D\. Wang, Z\. Li, J\. Ma, I\. Li, Q\. Yao, S\. Roman, Z\. Zhang, and D\. Radev \(2018\)Spider: a large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-sql task\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 3911–3921\.Cited by:[§2](https://arxiv.org/html/2605.22937#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAppendix

### A\.1Experimental Results

Table[4](https://arxiv.org/html/2605.22937#A1.T4)provides a granular breakdown of the Query Execution Error Rate \(QER\) across all evaluated datasets, models, and query complexities\. As expected, single\-pass generation \(QER@1\) degrades significantly as query difficulty increases, with execution failures climbing as high as 0\.97 on Hard queries\. While scaling ton=5n\{=\}5via Independent Scaling \(IS@5\) yields moderate improvements, Reflection\-Augmented Scaling \(RAS@5\) consistently suppresses the error rate to below 0\.10, even on the most topologically complex queries\. Furthermore, the reported standard deviations \(σIS\\sigma\_\{\\text\{IS\}\}andσRAS\\sigma\_\{\\text\{RAS\}\}\) reveal that RAS not only reduces the mean error but also tightens the variance across independent runs—particularly at the Hard complexity level\. This indicates that execution\-grounded reflection effectively constrains the generation space, yielding vastly more stable and predictable behavior than simple independent resampling\.

Table 4:QER per dataset atn=1n\{=\}1\(QER@1, averaged over IS and RAS baselines\) andn=5n\{=\}5\(IS@5, RAS@5\)\.σIS\\sigma\_\{\\text\{IS\}\}andσRAS\\sigma\_\{\\text\{RAS\}\}denote the standard deviation of QER atn=5n\{=\}5under IS and RAS, respectively\.
### A\.2Input text complexities

Table[5](https://arxiv.org/html/2605.22937#A1.T5)provides a representative sample of the natural language queries used in our evaluation, categorized by dataset and complexity tier\. To provide a concrete understanding of the structural challenges introduced at each level, we detail the specific input prompts alongside their corresponding evaluation identifiers\. As illustrated in the table,Easyqueries \(e\.g\.,crime\_easy\_1\) focus on straightforward entity retrieval and direct attribute filtering\.Mediumqueries \(e\.g\.,health\_medium\_1\) introduce relational constraints, requiring the model to successfully join multiple entity types\. Finally,Hardqueries \(e\.g\.,fraud\_hard\_1\) demand advanced structural reasoning, requiring the generated Cypher code to handle cyclic multi\-hop paths, mutual relationships, and conditional aggregations simultaneously\. These examples highlight the escalating topological complexity that the inference strategies must navigate\.

Query IDComplexityNatural Language QuestionCrime Datasetcrime\_easyEasyFind all crime incidents and return their incident IDs and crime types\.crime\_mediumMediumFind people involved in crime incidents at a specific location and return the person name and incident ID\.crime\_hardHardIdentify repeat offenders who are connected to more than one crime incident and return the person name and the number of incidents\.Fraud Datasetfraud\_easyEasyList all transactions that are marked as fraudulent and return their transaction IDs\.fraud\_mediumMediumFind customers who have more than two fraudulent transactions and return the customer name and the number of fraudulent transactions\.fraud\_hardHardFind pairs of accounts that have transferred money between each other and are both involved in at least one fraudulent transaction, and return the account IDs and the number of shared fraudulent transactions\.Healthcare Datasethealth\_easyEasyFind all patients in the database and return their names\.health\_mediumMediumList all medical records for a specific patient and the physicians associated with those records\.health\_hardHardFind patients who have been treated by more than one physician for the same medical condition and return the condition and physician count\.Table 5:Detailed natural language questions across the Crime, Fraud, and Healthcare datasets, including their assigned evaluation identifiers and complexity tiers\.

## Appendix BBroader Applications: Reflection as a Generalized Reasoning Engine

While this study focuses on Text2Cypher as a rigorous testbed for execution\-grounded scaling, the underlying reflection\-augmented in\-context learning \(ICL\) mechanism is fundamentally domain\-agnostic\. Beyond structured query generation, this iterative feedback loop can serve as the core component of a generalized advanced reasoning engine\.

In standard language model deployments, complex reasoning is often bottlenecked by a single\-forward pass\. By treating environmental feedback—such as database execution errors, compiler outputs, or external API responses—as structured diagnostic signals, the Reflection\-Augmented Scaling \(RAS\) methodology effectively transforms a static generation task into an agentic, self\-correcting workflow\.

As a generalized reasoning engine, this architecture can be extended to multi\-turn analytical tasks where a model must autonomously navigate knowledge graphs, verify its own intermediate logical steps, and recover from hallucinations before returning a final answer to the user\. Consequently, understanding the scaling trajectory of reflection \(e\.g\., the sharp performance gains atn=2n\{=\}2\) informs the compute\-optimal design of these broader agentic systems, ensuring that inference\-time compute is allocated efficiently during complex, multi\-step reasoning tasks\.

The full production implementation is not publicly released because it is part of the corporation’s internal reasoning\-engine\. To support reproducibility, we describe the inference procedure, prompt construction, execution\-validation protocol, datasets, model choices, decoding settings, and evaluation metrics in detail\. We also provide pseudocode for both Independent Scaling and Reflection\-Augmented Scaling, enabling independent reimplementation of the experimental logic without exposing proprietary system components\.