ExecuGraph: A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models

# A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models1footnote 11footnote 1Source code, configurations, benchmark definitions, and per-trial result logs that produce every numeric claim in this paper are available at https://github.com/rohithreddybc/multi-agent-dsa-backend-logic-synthesis.
Source: [https://arxiv.org/html/2607.20499](https://arxiv.org/html/2607.20499)
###### Abstract

Large Language Models generate plausible backend code but a single\-pass paradigm provides no guarantee of correctness or runtime reliability\. We present*ExecuGraph*, a multi\-agent framework that places execution\-based validation at the centre of backend code synthesis\. Six specialized agents—Planner, Code Generator, Logical Reviewer, Evaluator, Optimizer, and Explainer—are coordinated by a typed directed workflow with a bounded retry budget, implemented on LangGraph with locally hosted models \(Ollama\) and an optional retrieval layer for algorithmic technique recall\. A subprocess\-isolated sandbox with a wall\-clock timeout guards every evaluation\.

We evaluate on a curated 30\-problem DSA suite \(internal\-30\), HumanEval \(n=64n\{=\}64\), and an APPS\-introductory subset, contrasting ExecuGraph against a single\-agent one\-shot baseline and a single\-agent execution\-retry baseline \(a Reflexion\-style ablation that isolates the contribution of multi\-agent decomposition\)\. On internal\-30, the three conditions are statistically indistinguishable \(n=30n\{=\}30; paired Wilcoxonp=0\.59p\{=\}0\.59MF vs\. SO,p=0\.08p\{=\}0\.08SR vs\. SO\); 95% bootstrap confidence intervals on all pairwise mean differences include zero\. On HumanEval, multi\-full edges ahead by\+3\.1\+3\.1pp\. The strongest signal is cross\-model: with DeepSeek\-Coder\-V2\-Lite, graph\-category accuracy improves from 57\.5% \(one\-shot\) to 80\.0% \(multi\-full\), a\+22\.5\+22\.5pp jump that supports a scaling hypothesis—the value of multi\-agent decomposition grows with base\-model capability\.

The framework’s primary contribution is methodological: a single codebase that collapses by configuration into one\-shot, execution\-retry, and per\-agent ablation conditions, enabling controlled measurement of each lever’s marginal contribution\. A per\-agent ablation, retry\-budget sweep, error\-class taxonomy, and test\-source audit are reported\. All configurations, per\-trial JSON logs, and table\-generation scripts are released so every numeric claim is regenerable\.

###### keywords:

Code generation , large language models , multi\-agent systems , execution\-grounded validation , retrieval\-augmented generation , software reliability , workflow orchestration

††journal:Information Sciences\\affiliation

\[kitsaiml\]organization=Department of Computer Science and Engineering \(AI & ML\), Kakatiya Institute of Technology and Science, city=Warangal, state=Telangana, country=India\\affiliation\[kits\]organization=Department of Computer Science and Engineering, Kakatiya Institute of Technology and Science, city=Warangal, state=Telangana, country=India\\affiliation\[indep\]organization=Independent Researcher, country=India\\affiliation\[bu\]organization=Boston University, city=Boston, state=MA, country=USA

## 1Introduction

Large Language Models \(LLMs\) have rapidly become a default tool for automated code generation, achieving competitive performance on a broad range of programming tasks including backend code synthesis and algorithmic problem solving\[[3](https://arxiv.org/html/2607.20499#bib.bib1),[4](https://arxiv.org/html/2607.20499#bib.bib2),[13](https://arxiv.org/html/2607.20499#bib.bib10)\]\. By learning statistical mappings between natural\-language descriptions and source code, LLMs produce syntactically valid implementations from minimal human input, and they are increasingly relied on for rapid prototyping, coding assistance, and education\[[4](https://arxiv.org/html/2607.20499#bib.bib2),[10](https://arxiv.org/html/2607.20499#bib.bib11),[31](https://arxiv.org/html/2607.20499#bib.bib12)\]\.

Despite these strengths, reliability remains a significant limitation\. Most production\-grade LLM coding systems still adopt a*single\-agent, single\-pass*paradigm in which one model is responsible for problem understanding, solution generation, and implicit validation\. While effective for short or routine code, this strategy frequently fails for backend code that depends on data\-structure invariants, algorithmic efficiency, or precise edge\-case handling\. Generated programs often look correct yet contain semantic flaws, inefficient algorithm choices, or missing boundary handling that surface only at runtime\[[4](https://arxiv.org/html/2607.20499#bib.bib2),[13](https://arxiv.org/html/2607.20499#bib.bib10)\]\.

A common root cause is the absence of execution\-level validation\. Many approaches rely on textual self\-review or chain\-of\-thought style reasoning to assess correctness\[[32](https://arxiv.org/html/2607.20499#bib.bib3),[30](https://arxiv.org/html/2607.20499#bib.bib6)\], but text\-only checks are insufficient to detect runtime errors and performance issues that only manifest under execution\[[6](https://arxiv.org/html/2607.20499#bib.bib15),[12](https://arxiv.org/html/2607.20499#bib.bib14)\]\. Iterative\-refinement systems such as Reflexion\[[29](https://arxiv.org/html/2607.20499#bib.bib4)\]apply verbal self\-critique with episode memory, and execution\-filtered systems such as AlphaCode\[[21](https://arxiv.org/html/2607.20499#bib.bib16)\]validate sampled candidates against tests, but these approaches do not embed execution feedback inside an explicitly structured multi\-agent workflow with typed shared state\.

### 1\.1Contributions

This paper presents*ExecuGraph*, a graph\-structured multi\-agent framework that places execution\-grounded validation at the centre of backend code synthesis\. Concretely, we contribute:

1. 1\.An execution\-grounded multi\-agent workflow that decouples role decomposition from execution feedback, formally specified as a typed transition system over a shared state object with explicit decision predicates and bounded retries \(Section[3](https://arxiv.org/html/2607.20499#S3)\)\. The framework collapses by configuration into single\-agent one\-shot, single\-agent execution\-retry \(Reflexion\-style\), and per\-agent ablation conditions inside a single codebase and evaluation harness, enabling the first controlled empirical measurement of each lever’s marginal contribution—rather than the typical conflation of execution feedback with multi\-agent decomposition\.
2. 2\.A hybrid evaluation protocolcombining deterministic invariant tests for canonical algorithmic problems with LLM\-generated edge\-case tests for previously unseen tasks, executed inside a subprocess\-isolated sandbox with a wall\-clock timeout and a restricted import policy\.
3. 3\.An empirical studyacross the curated 30\-problem suite, the HumanEval benchmark, and an APPS\-introductory subset, comparing ExecuGraph against a single\-agent one\-shot baseline and a single\-agent execution\-retry baseline\. We report pass\-rate, retries, execution\-failure rate, wall\-clock, total LLM calls, total tokens, and paired\-Wilcoxonpp\-values per category\.
4. 4\.A per\-agent ablationquantifying the marginal contribution of the Planner, Logical Reviewer, Optimizer, and retrieval layer, plus a retry\-budget sweep over\{0,2\}\\\{0,2\\\}\.
5. 5\.A reproducible artifact: every numeric claim in the result tables is generated from per\-trial JSON logs by the includedanalysis/build\_tables\.pyscript, and every configuration, model digest, and seed used in the experiments is committed to the repository\.

The remainder of the paper is organized as follows\. Section[2](https://arxiv.org/html/2607.20499#S2)reviews related work on LLM code generation, iterative refinement, multi\-agent coding pipelines, execution\-based validation, and retrieval\-augmented generation\. Section[3](https://arxiv.org/html/2607.20499#S3)formalizes the proposed framework\. Section[4](https://arxiv.org/html/2607.20499#S4)describes the evaluation protocol\. Section[5](https://arxiv.org/html/2607.20499#S5)reports quantitative results\. Section[6](https://arxiv.org/html/2607.20499#S6)discusses implications and limitations\. Section[7](https://arxiv.org/html/2607.20499#S7)concludes\. Section[8](https://arxiv.org/html/2607.20499#S8)provides full reproducibility details\.

## 2Related Work

We organize prior work into five strands and end each subsection with the design choice that distinguishes ExecuGraph\.

### 2\.1LLMs for Code Generation

Foundational LLM work showed that scale and large code corpora yield strong code\-generation ability\[[3](https://arxiv.org/html/2607.20499#bib.bib1),[4](https://arxiv.org/html/2607.20499#bib.bib2),[7](https://arxiv.org/html/2607.20499#bib.bib17)\]\. Specialized pretraining objectives improved code understanding and synthesis\[[10](https://arxiv.org/html/2607.20499#bib.bib11),[31](https://arxiv.org/html/2607.20499#bib.bib12)\], and open\-source code\-LLMs broadened access\[[23](https://arxiv.org/html/2607.20499#bib.bib18),[11](https://arxiv.org/html/2607.20499#bib.bib19)\]\. Functional benchmarks such as HumanEval\[[4](https://arxiv.org/html/2607.20499#bib.bib2)\], MBPP\[[1](https://arxiv.org/html/2607.20499#bib.bib20)\], and APPS\[[13](https://arxiv.org/html/2607.20499#bib.bib10)\]formalized correctness via test execution rather than surface similarity\. Empirically, however, these same studies also document persistent semantic errors and brittle edge\-case handling, motivating mechanisms beyond raw decoding\.

ExecuGraph differsby treating an off\-the\-shelf code\-LLM as a*component*in a workflow rather than a self\-contained solver, and by making execution outcomes the load\-bearing acceptance signal\.

### 2\.2Iterative Refinement and Self\-Debugging

Chain\-of\-thought prompting\[[32](https://arxiv.org/html/2607.20499#bib.bib3)\]and self\-consistency sampling\[[30](https://arxiv.org/html/2607.20499#bib.bib6)\]improve intermediate reasoning but rely on the same model for both generation and judgement\. Reflexion\[[29](https://arxiv.org/html/2607.20499#bib.bib4)\]introduces verbal self\-critique with episode memory; Self\-Debug\[[5](https://arxiv.org/html/2607.20499#bib.bib21)\]prompts the same model to inspect and fix its own program with execution traces\. Both demonstrate that a feedback loop helps, but conflate “model\-as\-critic” with “decomposed roles”: it is unclear whether the gains come from feedback per se or from any form of role decomposition\.

ExecuGraph differsby separating the critic role into two distinct agents—a static Logical Reviewer \(advisory\) and an Evaluator \(authoritative, execution\-based\)—and by treating the Reviewer’s output as a structured, advisory artifact rather than a binding judgement\. This design choice is empirically supported by recent work on LLM\-as\-a\-judge stability: the JudgeSense benchmark\[[2](https://arxiv.org/html/2607.20499#bib.bib30)\]reports judge\-decision flip rates of8\.58\.5% to61\.361\.3% on coherence under semantically equivalent prompt rephrasings across thirteen frontier and open\-weight judges, and finds that frontier scale is not a reliable proxy for consistency\. Letting an LLM critic veto otherwise\-correct code on the basis of a possibly paraphrase\-sensitive verdict therefore introduces a non\-trivial false\-negative risk; ExecuGraph avoids this by gating acceptance solely on execution outcomes\. Section[4](https://arxiv.org/html/2607.20499#S4)reports a single\-agent execution\-retry baseline \(a Reflexion\-style configuration of the same framework\) so that the gain attributable to the multi\-agent decomposition itself is isolated\.

### 2\.3Multi\-Agent Coding Pipelines

Several recent systems explicitly decompose code synthesis into multiple roles\. AgentCoder\[[15](https://arxiv.org/html/2607.20499#bib.bib22)\]introduces test\-design and programmer agents; MapCoder\[[17](https://arxiv.org/html/2607.20499#bib.bib23)\]uses retrieval\-planning\-coding\-debugging stages; MetaGPT\[[14](https://arxiv.org/html/2607.20499#bib.bib24)\]composes role\-playing agents around a software\-development meta\-process; AutoGen\[[33](https://arxiv.org/html/2607.20499#bib.bib25)\]provides a conversational multi\-agent runtime\. ReAct\[[34](https://arxiv.org/html/2607.20499#bib.bib5)\]and Auto\-GPT\[[27](https://arxiv.org/html/2607.20499#bib.bib8)\]explore agentic reasoning and autonomous tool use\. AgentBench\[[22](https://arxiv.org/html/2607.20499#bib.bib7)\]documents reliability degradation in multi\-step agent settings, particularly when success depends on precise state propagation\.

ExecuGraph differsalong three axes: \(i\) it uses a*compiled, deterministic*workflow graph \(LangGraph\) with a typed shared state record rather than a free\-form conversation; \(ii\) the decision predicate is defined exclusively over execution outcomes, not over textual self\-assessment; and \(iii\) the framework exposes ablation toggles so that each role’s marginal contribution can be measured directly\.

### 2\.4Execution\-Grounded Validation and Program\-Aided Methods

Execution\-guided neural program synthesis injects runtime signals into decoding to filter invalid candidates\[[6](https://arxiv.org/html/2607.20499#bib.bib15)\]\. Program\-aided language models \(PAL\) delegate symbolic computation to interpreters\[[12](https://arxiv.org/html/2607.20499#bib.bib14)\]\. Competition\-level systems such as AlphaCode use large\-scale sampling with execution\-based filtering\[[21](https://arxiv.org/html/2607.20499#bib.bib16)\]\. Toolformer\[[28](https://arxiv.org/html/2607.20499#bib.bib13)\]learns when to call external tools\.

ExecuGraph differsby elevating execution from a*filter*to the*decision predicate*of the workflow itself\. The graph cannot accept code that has not passed runtime evaluation, regardless of the static reviewer’s verdict\.

### 2\.5Retrieval\-Augmented Generation for Code

RAG\[[20](https://arxiv.org/html/2607.20499#bib.bib9)\]improves knowledge\-intensive tasks by grounding generation in retrieved documents\. Vector stores such as ChromaDB\[[8](https://arxiv.org/html/2607.20499#bib.bib33)\]and orchestrators such as LangChain\[[18](https://arxiv.org/html/2607.20499#bib.bib31)\]and LangGraph\[[19](https://arxiv.org/html/2607.20499#bib.bib32)\]enable practical RAG\-and\-tools pipelines, while local\-first inference platforms such as Ollama\[[24](https://arxiv.org/html/2607.20499#bib.bib34)\]support reproducible deployment\. None of these primitives, however, prescribes*when*retrieval should fire in a code\-synthesis workflow\.

ExecuGraph differsby treating retrieval as an*optional*, configuration\-toggled input to the Planner only, and by reporting an explicit on/off ablation of the retrieval layer \(Section[5](https://arxiv.org/html/2607.20499#S5)\) so that its contribution can be assessed rather than assumed\.

## 3The ExecuGraph Framework

We first present the high\-level architecture \(Section[3\.1](https://arxiv.org/html/2607.20499#S3.SS1)\), then specify the workflow as a typed transition system \(Section[3\.2](https://arxiv.org/html/2607.20499#S3.SS2)\), describe each agent \(Section[3\.3](https://arxiv.org/html/2607.20499#S3.SS3)\), and document the execution sandbox \(Section[3\.5](https://arxiv.org/html/2607.20499#S3.SS5)\)\.

### 3\.1High\-Level Architecture

ExecuGraph follows a layered design \(Fig\.[1](https://arxiv.org/html/2607.20499#S3.F1)\)\. A*User Interaction Layer*accepts a backend / DSA problem statement\. A*Workflow Orchestration Layer*\(LangGraph\) compiles a directed graph that schedules agent invocations and routes control based on the typed shared state\. An*Agent Layer*contains six specialized agents \(Planner, Code Generator, Logical Reviewer, Evaluator, Optimizer, Explainer\)\. An*Execution and Evaluation Layer*runs the generated code in a subprocess\-isolated sandbox and applies a hybrid testing strategy\. An*optional Knowledge Memory Layer*\(ChromaDB\) supplies retrieved algorithmic\-technique snippets to the Planner only; it is config\-toggled and disabled by default in the headline numbers, with a dedicated on/off ablation in Section[5](https://arxiv.org/html/2607.20499#S5)\.

User Interaction LayerInput:Backend / DSAProblem StatementSynthesize LogicWorkflow Orchestration LayerLangGraph OrchestratorManages Agent Execution, Conditional Transitions,Retry Mechanisms, State\-Aware FlowAgent LayerPlanningAgentCodeGenerationAgentLogicalReviewAgentEvaluationAgentOptimizationAgentExplanationAgentExecution FeedbackExecution and Evaluation LayerCode Interpreter/ SandboxTest SuiteRunnerOptional Knowledge Memory LayerChromaDB Vector Store\(Algorithmic Techniques & Patterns\)Output Dashboard Generated Code & Plan Execution Results \(Pass/Fail, Tests\) System Metrics & Confidence Score Severity Analysis Execution Trace LogsQuery/Retrieve

Figure 1:Five\-layer architecture of ExecuGraph\. User input flows top\-to\-bottom through the Workflow Orchestration \(LangGraph\), Agent, Execution/Evaluation, and optional Knowledge Memory \(ChromaDB\) layers\. The Output Dashboard \(right\) aggregates generated code, test results, metrics, and trace logs\.The framework is governed by four design principles: \(i\)*separation of concerns*—each agent has a single, well\-defined output type; \(ii\)*execution\-centric validation*—acceptance is determined by runtime outcomes, not by static review; \(iii\)*deterministic workflow control*—the orchestrator is a compiled DAG with typed state, conditional edges, and a hard retry cap; and \(iv\)*reproducibility*—fixed inference configurations, persisted seeds, and per\-trial JSON logs that regenerate every result table\.

### 3\.2Workflow Semantics

We formalize the framework as a typed transition system𝒲=\(𝒮,A,δ,D,T\)\\mathcal\{W\}=\(\\mathcal\{S\},A,\\delta,D,T\)over a shared state record𝒮\\mathcal\{S\}\(see Fig\.[2](https://arxiv.org/html/2607.20499#S3.F2)for the corresponding DAG\)\.

#### Shared state\.

𝒮\\mathcal\{S\}is a typed record \(Listing[1](https://arxiv.org/html/2607.20499#LST1)\) that persists for the lifetime of one synthesis episode\. Each agent reads a subset of fields and writes a disjoint subset; conflicting writes are forbidden by construction\.

Listing 1:Shared graph state \(TypedDict\)\.classGraphState\(TypedDict\):

problem:str

plan:str

code:str

review:dict

evaluation:dict

explanation:str

retries:int

retry\_budget:int

passed:bool

optimized:bool

enable\_planner:bool

enable\_reviewer:bool

enable\_optimizer:bool

enable\_rag:bool

retrieved\_techniques:list

cost:dict

logs:list\[str\]

#### Agent set\.

A=\{plan,gen,rev,eval,opt,exp\}A=\\\{\\text\{plan\},\\text\{gen\},\\text\{rev\},\\text\{eval\},\\text\{opt\},\\text\{exp\}\\\}\. Eacha∈Aa\\in Ais a partial functiona:𝒮→Δ​𝒮a:\\mathcal\{S\}\\rightarrow\\Delta\\mathcal\{S\}, whereΔ​𝒮\\Delta\\mathcal\{S\}is a state delta\. The Planner is invoked at most once per episode \(idempotent guard\); the other agents may be re\-invoked subject to the routing function\.

#### Decision predicate\.

The acceptance predicate is purely execution\-driven:

D​\(s\)≡s\.evaluation\.passed=𝑇𝑟𝑢𝑒\.D\(s\)\\equiv s\.\\text\{evaluation\.passed\}=\\mathit\{True\}\.\(1\)Static\-review verdicts do not appear inDD\.

#### Routing\.

The routing functionρ:𝒮→A∪\{end\}\\rho:\\mathcal\{S\}\\rightarrow A\\cup\\\{\\text\{end\}\\\}is

ρ​\(s\)=\{optif​D​\(s\)∧¬s\.optimized∧s\.enable\_opt,expif​D​\(s\)∨s\.retries≥s\.retry\_budget,genotherwise\.\\rho\(s\)=\\begin\{cases\}\\text\{opt\}&\\text\{if \}D\(s\)\\wedge\\neg s\.\\text\{optimized\}\\wedge s\.\\text\{enable\\\_opt\},\\\\ \\text\{exp\}&\\text\{if \}D\(s\)\\vee s\.\\text\{retries\}\\geq s\.\\text\{retry\\\_budget\},\\\\ \\text\{gen\}&\\text\{otherwise\}\.\\end\{cases\}\(2\)The Optimizer, when enabled, re\-invokes the Evaluator before its output is accepted, preventing optimization\-induced regressions \(this addresses peer\-review point T1\)\.

#### Termination\.

TTholds when control reachesendviaρ\\rho\. Termination is guaranteed because \(i\)s\.retriess\.\\text\{retries\}strictly increases on everygenre\-entry, \(ii\) the budget is finite, and \(iii\) the success branch is acyclic\.

#### Run loop\.

Algorithm[1](https://arxiv.org/html/2607.20499#alg1)states the workflow procedure compactly\. The notation matches the transition system above:ssis the shared state,δa\\delta\_\{a\}is the state delta produced by agentaa,Sandboxis the subprocess\-isolated executor of Section[3\.5](https://arxiv.org/html/2607.20499#S3.SS5), andBBis the retry budget configured per run\.

Algorithm 1ExecuGraph synthesis loop\.1:problem

qq, retry budget

BB, agent toggles

τ\\tau, LLM backends, sandbox timeout

θ\\theta
2:final state

sswith code

s\.codes\.\\text\{code\}, evaluation

s\.evaluations\.\\text\{evaluation\}, and cost log

3:

s←InitState​\(q,B,τ\)s\\leftarrow\\textsc\{InitState\}\(q,\\,B,\\,\\tau\)
4:if

τ\.planner\\tau\.\\text\{planner\}then

5:

s\.plan←Planner\(s\.problem,Retrieve\(s\)ifτ\.rag\)s\.\\text\{plan\}\\leftarrow\\textsc\{Planner\}\(s\.\\text\{problem\},\\,\\textsc\{Retrieve\}\(s\)~\\textbf\{if\}~\\tau\.\\text\{rag\}\)
6:endif

7:

r←0r\\leftarrow 0⊳\\trianglerightretries used

8:loop

9:

s\.code←Generator\(s\.problem,s\.plan,s\.stderr\)s\.\\text\{code\}\\leftarrow\\textsc\{Generator\}\(s\.\\text\{problem\},\\,s\.\\text\{plan\},\\,s\.\\text\{stderr\}\)
10:if

τ\.reviewer\\tau\.\\text\{reviewer\}then

11:

s\.review←Reviewer\(s\.code\)s\.\\text\{review\}\\leftarrow\\textsc\{Reviewer\}\(s\.\\text\{code\}\)⊳\\trianglerightadvisory only

12:endif

13:

s\.evaluation←Evaluator\(s\.code;θ\)s\.\\text\{evaluation\}\\leftarrow\\textsc\{Evaluator\}\(s\.\\text\{code\};\\,\\theta\)
14:if

s\.evaluation\.passeds\.\\text\{evaluation\}\.\\text\{passed\}then

15:break

16:endif

17:if

r≥Br\\geq Bthen

18:break

19:endif

20:

r←r\+1r\\leftarrow r\+1
21:

s\.stderr←s\.evaluation\.stderrs\.\\text\{stderr\}\\leftarrow s\.\\text\{evaluation\}\.\\text\{stderr\}
22:endloop

23:if

s\.evaluation\.passeds\.\\text\{evaluation\}\.\\text\{passed\}and

τ\.optimizer\\tau\.\\text\{optimizer\}then

24:

c′←Optimizer\(s\.code\)c^\{\\prime\}\\leftarrow\\textsc\{Optimizer\}\(s\.\\text\{code\}\)
25:

e′←Evaluator​\(c′;θ\)e^\{\\prime\}\\leftarrow\\textsc\{Evaluator\}\(c^\{\\prime\};\\,\\theta\)⊳\\trianglerightre\-validate \(peer\-review T1\)

26:if

e′\.passede^\{\\prime\}\.\\text\{passed\}then

27:

s\.code←c′;s\.evaluation←e′s\.\\text\{code\}\\leftarrow c^\{\\prime\};~~s\.\\text\{evaluation\}\\leftarrow e^\{\\prime\}
28:endif

29:endif

30:

s\.explanation←Explainer\(s\.problem,s\.code,s\.plan\)s\.\\text\{explanation\}\\leftarrow\\textsc\{Explainer\}\(s\.\\text\{problem\},\\,s\.\\text\{code\},\\,s\.\\text\{plan\}\)
31:return

ss

SPlannerAgentGeneratorAgentReviewerAgentEvaluatorAgentDecisionNodeOptimizerAgentExplainerAgentEOptionalKnowledgeMemory\(ChromaDB\)Shared State plan code review retries eval\_result optimized logsPassMax retriesFailed & retries<<MAXExecution\-Based Feedback

Figure 2:ExecuGraph workflow DAG\. Solid arrows show the main pipeline \(S→\\toPlanner→\\toGenerator→\\toReviewer→\\toEvaluator→\\toDecision→\\toOptimizer/Explainer→\\toE\)\. The red curved arrow is the retry loop, active whenretries < retry\_budget\. Dashed purple arrows indicate the optional RAG query/retrieve path from the Planner through ChromaDB and back to the Generator\. The SharedGraphState panel \(right\) lists all fields persisted across agent transitions\.
#### Reduction to baselines\.

The same framework collapses to two baselines via configuration:

- 1\.*Single\-agent one\-shot*:enable\_planner=enable\_reviewer=enable\_opt=𝐹𝑎𝑙𝑠𝑒\\text\{enable\\\_planner\}=\\text\{enable\\\_reviewer\}=\\text\{enable\\\_opt\}=\\mathit\{False\},retry\_budget=0\\text\{retry\\\_budget\}=0\. The Generator runs once, the Evaluator records pass/fail for measurement only, and routing terminates immediately\.
- 2\.*Single\-agent execution\-retry \(Reflexion\-style\)*: same as above butretry\_budget\>0\\text\{retry\\\_budget\}\>0\. The Generator regenerates from the original problem plus the prior failed candidate’s stderr\.

This reduction is what enables the per\-component ablations of Section[5](https://arxiv.org/html/2607.20499#S5)\.

### 3\.3Agents

#### Planner\.

Readsproblem\(and optionallyretrieved\_techniques\)\. Emits a structured plan: classification of problem type, chosen algorithmic strategy, step\-by\-step pseudocode, expected time and space complexity, and a candidate\-edge\-cases list\. Backed by an instruction\-tuned LLM at temperature0\.20\.2\.

#### Code Generator\.

Readsproblemandplan\. Emits a single Python function\. Backed by a code\-specialized LLM at temperature0\.00\.0for determinism\.

#### Logical Reviewer\.

Readscode\. Emits a structured JSON object\{invariants, potential\_failures, severity\}, marked*advisory*\. The framework explicitly does not gate on its output\. This is the key change from systems that let a textual critic block correct code; the design is motivated both by an early\-development failure mode in which a confidently\-wrong reviewer suppressed correct candidate code, and by independent benchmark evidence that LLM\-as\-judge verdicts are unstable under paraphrase\[[2](https://arxiv.org/html/2607.20499#bib.bib30)\]\.

#### Evaluator\.

The acceptance authority\. Readscodeandproblem\. Sanitizes the code, executes it in a subprocess sandbox \(Section[3\.5](https://arxiv.org/html/2607.20499#S3.SS5)\), and applies a hybrid testing strategy:

1. 1\.If the problem is in the curated benchmark, deterministic invariant tests are dispatched \(with multi\-name signature aliasing\)\.
2. 2\.Otherwise, a code\-specialized LLM is asked to emit a JSON test\-suite of the form\[\{call, expected, description\}\], parsed by a strict JSON parser, and each test is executed via a sandboxed expression evaluator\.
3. 3\.If neither yields tests, a smoke test \(executes\-without\-error\) is recorded as the only signal and the trial is flagged as test\-poor in the JSON log\.

Returns\{passed: bool, test\_results: list, error\_class: str, stderr: str\}\.

#### Optimizer\.

Runs only whenD​\(s\)D\(s\)holds\. Receives the passingcodeand emits an optimized variant\. The Evaluator is then re\-invoked on the optimized code; if it fails, the original is retained and a regression note is recorded\.

#### Explainer\.

Always runs as the terminal node\. Produces a human\-readable explanation of the algorithm intuition, complexity, and any failure modes that occurred during the episode\.

### 3\.4Worked Example: Topological Sort

To make the data flow concrete, Listing[2](https://arxiv.org/html/2607.20499#LST2)shows the actual artifacts produced by ExecuGraph on thetopo\_sortproblem from the curated suite \(one of the three APPS\-derived items\)\. This is a real trace; the artifacts were emitted by the runner and copied verbatim fromresults/example\_run/trials\.jsonl\. It illustrates the canonical happy path—plan, generate, review, evaluate, optimize, explain—without retries\.

Listing 2:Per\-agent trace for thetopo\_sortproblem \(truncated for space\)\. Acceptance is decided by the Evaluator’spassed:true; the Reviewer’s verdict is recorded but not consulted\.\[PLANNER\]\(qwen2\.5\-coder:7b\-instruct\)

Category:Graph

Algorithm:Kahn’salgorithm\(BFStopologicalsort\)

Steps:buildadjlist\-\>computeindegrees\-\>

BFSoverzero\-indegreenodes\-\>if

visited\!=Nreturn\[\]\(cycle\)

Complexity:O\(V\+E\)time,O\(V\)space

\[GENERATOR\]\(qwen2\.5\-coder:7b\-instruct\)

deffindOrder\(numCourses,prerequisites\):

fromcollectionsimportdefaultdict,deque

g=defaultdict\(list\);indeg=\[0\]\*numCourses

fora,binprerequisites:

g\[b\]\.append\(a\);indeg\[a\]\+=1

q=deque\(iforiinrange\(numCourses\)

ifindeg\[i\]==0\)

out=\[\]

whileq:

u=q\.popleft\(\);out\.append\(u\)

forving\[u\]:

indeg\[v\]\-=1

ifindeg\[v\]==0:q\.append\(v\)

returnoutiflen\(out\)==numCourseselse\[\]

\[REVIEWER\]\(advisory;notconsultedbydecision\)

invariants:\["acyclicimplieslen\(out\)==N",

"cycleimplies\[\]"\]

potential\_failures:\[\]

severity:low

\[EVALUATOR\]\(subprocesssandbox,5stimeout\)

test1linearchainpassed

test2cycle\(2\-node\)passed

test3singletaskpassed

test4branchingdepspassed\(judge:validorder\)

test5complexcyclepassed

result:passed=true,error\_class=none

\[OPTIMIZER\]nofunctionalchange;re\-evaluationpassed

\[EXPLAINER\]returnsKahn’s\-algorithmexplanation

### 3\.5Execution Sandbox

Because every agent decision rests on the Evaluator, the sandbox is load\-bearing\. Generated code runs in a freshly forked Python subprocess with: \(i\) a hard wall\-clock timeout \(default55s, configurable per benchmark\); \(ii\) a curated\_\_builtins\_\_that excludeseval,exec,compile,open\(write\), and\_\_import\_\_ofos,subprocess,socket,shutil,ctypes; \(iii\) no inheritance of the parent process’s environment variables \(in particular noHUGGINGFACEHUB\_API\_TOKEN\); and \(iv\) stdout / stderr captured for the per\-trial JSON log\. The sandbox is the same for the deterministic and the LLM\-generated test paths\. Sandbox limitations are discussed in Section[6\.7](https://arxiv.org/html/2607.20499#S6.SS7)\.

### 3\.6Implementation Environment

ExecuGraph is implemented in Python and orchestrated with LangGraph\[[19](https://arxiv.org/html/2607.20499#bib.bib32)\]and LangChain\[[18](https://arxiv.org/html/2607.20499#bib.bib31)\]\. LLM access is provider\-agnostic: the default backend uses Ollama\[[24](https://arxiv.org/html/2607.20499#bib.bib34)\]with locally hosted Qwen2\.5\-7B\-Instruct and Qwen2\.5\-Coder\-7B\-Instruct \(4\-bit quantized\); a HuggingFace Inference API backend is provided as an opt\-in fallback for users without a local GPU\. The optional retrieval layer uses ChromaDB\[[8](https://arxiv.org/html/2607.20499#bib.bib33)\]with a small set of seeded algorithmic\-technique entries\. A Streamlit interface is provided for interactive inspection but is not part of the experimental measurement loop\.

## 4Experimental Protocol

### 4\.1Datasets

We evaluate on three suites with deliberately overlapping but distinct purposes\.

#### Internal\-30\.

A curated suite of 30 problems spanning dynamic programming \(10\), graph algorithms \(10\), and data structures \(10\)\. 27 are internally designed; 3 are adapted from the APPS\-introductory partition\[[13](https://arxiv.org/html/2607.20499#bib.bib10)\], marked explicitly in the benchmark loader\. Selection criteria, recorded per problem in the loader source, are: \(a\) each problem admits a textbook solution that is either reachable by a 7B\-parameter model or known to be a frequent failure mode for one; \(b\) tests cover at least one boundary, one canonical, and one stress case; \(c\) no problem requires inputs above 1 MB so that sandbox memory pressure does not confound failure analysis\. The full list with selection rationale is shipped with the code \(execugraph/benchmarks/internal30\.py\)\. Table[1](https://arxiv.org/html/2607.20499#S4.T1)lists all 30 problems by category\.

Table 1:Internal\-30 benchmark problems by category\.†\\daggermarks the three problems adapted from APPS\-introductory; all others are internally designed\.
#### HumanEval\.

We sample 64 problems from HumanEval’s 164\-problem test set\[[4](https://arxiv.org/html/2607.20499#bib.bib2)\], used to anchor external validity\. We use the official tests verbatim through the sandbox\.

#### APPS\-introductory subset\.

A 50\-problem subset of the APPS\-introductory partition\[[13](https://arxiv.org/html/2607.20499#bib.bib10)\], drawn from the test split with a fixed seed for reproducibility\. Test cases are taken from the dataset\.

### 4\.2Conditions

Five conditions, all instantiated as configurations of the same workflow:

- 1\.single\-oneshot: Generator only, no retries\.
- 2\.single\-retry: Generator \+ Evaluator, retry budget swept; no Planner, no Reviewer, no Optimizer\.
- 3\.multi\-full: All agents enabled; retry budget=2=2; RAG off \(default\)\.
- 4\.multi\-no\-XforX∈\{planner,reviewer,optimizer,rag\-on\}X\\in\\\{\\text\{planner\},\\text\{reviewer\},\\text\{optimizer\},\\text\{rag\-on\}\\\}: ablations\.
- 5\.retry\-sweep:multi\-fullwith retry budget∈\{0,2\}\\in\\\{0,2\\\}\.

### 4\.3Models

We use three open\-weight LLMs spanning two vendor families, all locally hosted via Ollama\[[24](https://arxiv.org/html/2607.20499#bib.bib34)\]with44\-bitq4\_K\_Mquantization, accessed through the same provider\-agnostic backend so that no per\-condition prompting differences exist outside of the configuration files\. To eliminate the model\-swap overhead that would otherwise dominate wall\-clock on a 6 GB\-VRAM laptop GPU, all five agents in a given run share a single backbone:

- 1\.Primary backbone \(headline numbers\):qwen2\.5\-coder:7b\-instruct\-q4\_K\_M\[[16](https://arxiv.org/html/2607.20499#bib.bib27)\]for all five agents\. We selected a 7 B\-class dense code model for the headline grid because it fits entirely in 6 GB VRAM with comfortable KV\-cache headroom, requiring no RAM offload, and yields ~30–80 s per multi\-full trial on the reference hardware\. This makes the full E1\+E4\+E5\+E2\+E3\+E7 grid feasible inside a ~7\-hour wall\-clock budget\. The Planner / Reviewer / Explainer are run at temperature0\.20\.2; the Generator and Optimizer at temperature0\.00\.0\.
- 2\.Cross\-model condition \(independent vendor\):deepseek\-coder\-v2:16b\-lite\-instruct\-q4\_K\_M\[[9](https://arxiv.org/html/2607.20499#bib.bib26)\]from DeepSeek\-AI — a 16B Mixture\-of\-Experts model with 2\.4B active parameters\. We initially evaluated OpenAI’sgpt\-oss:20b\[[25](https://arxiv.org/html/2607.20499#bib.bib29)\]as the cross\-model candidate, but at 20B dense parameters its inference latency under 6 GB\-VRAM offload \(~600 s per multi\-full trial\) made the full cross\-model grid infeasible inside our compute budget; we therefore use DeepSeek\-Coder\-V2\-Lite, which gives equivalent vendor diversity at materially higher throughput\. This condition isolates the framework’s gains from any single\-vendor effect\.
- 3\.Latest\-2026 supplementary:qwen3\-coder:30b\-a3b\-q4\_K\_M\[[26](https://arxiv.org/html/2607.20499#bib.bib28)\]— the strongest agentic\-code\-tuned open\-weight model that runs on the reference hardware as of May 2026\. Mixture\-of\-Experts with 30 B total parameters but only 3\.3 B active per token; ~13 GB of weights spill to system RAM, yielding ~270 s per multi\-full trial\. Reported on a 10\-problem stratified subset of internal\-30 in the supplementary E9 row, demonstrating that the framework’s gains hold on the May 2026 SOTA local\-runnable code model without absorbing the compute cost of a full grid at that scale\.

Model digests, quantization tags, and the exactollama pullcommands are recorded inREPRODUCIBILITY\.md\. We deliberately do not use any closed\-weight or paid\-API model in any condition: the entire grid is reproducible at zero marginal API cost on the hardware specified in Section[4\.4](https://arxiv.org/html/2607.20499#S4.SS4)\.

### 4\.4Hardware and Software

All measurements were taken on a single machine: 13th Gen Intel Core i5\-13420H \(8C/12T, 2\.10 GHz base\), 16 GB DDR5, NVIDIA GeForce RTX 4050 Laptop GPU \(6 GB VRAM\), Windows 11 64\-bit\. Python 3\.11, LangGraph 1\.0\.5, LangChain 1\.2\.7, Ollama \(latest GA at time of run; pinned digest inREPRODUCIBILITY\.md\)\.

### 4\.5Trials and Seeds

Each \(problem, condition, model\) tuple is repeated forNNindependent trials with seeds\{0,1,…,N−1\}\\\{0,1,\\ldots,N\-1\\\}passed both to the LLM backend \(where supported\) and to the LLM\-generated test sampler\. We useN=5N=5for the headline E1 conditions on internal\-30,N=2N=2for the per\-agent ablations and the retry\-budget sweep, andN=1N=1for HumanEval and APPS\-introductory \(their problem counts of 64 and 50 respectively already provide sufficient statistical power\)\. The retry\-budget sweep is run at\{0,2\}\\\{0,2\\\}rather than\{0,1,2,3\}\\\{0,1,2,3\\\}for compute\-budget reasons; the budgets at\{0,2\}\\\{0,2\\\}already span the practically interesting range\. The seed flows through to: \(i\) the generator’s sampling RNG, \(ii\) the LLM\-test sampler, \(iii\) the trial\-runner’s shuffling order so retries are reproducible\.

### 4\.6Metrics

Per \(problem, condition, model, trial\):

- 1\.passed:11iff all tests pass\.
- 2\.tests\_passed / tests\_total: per\-test pass rate\.
- 3\.retries\_used: number of regenerations consumed\.
- 4\.error\_class: one of \{none,syntax,runtime,timeout,wrong\_answer,empty\_output,sandbox\_violation\}\.
- 5\.wallclock\_s: end\-to\-end seconds\.
- 6\.tokens\_in,tokens\_out,llm\_calls: cost\.

Aggregated metrics: pass\-rate \(mean over trials\), pass@kk\[[4](https://arxiv.org/html/2607.20499#bib.bib2)\], retries\-mean, execution\-failure\-rate \(any error\_class≠\\neqnone, wrong\_answer\), 95% bootstrap CI on totals, paired Wilcoxonpp\-value vs\. baseline, McNemar test on per\-problem outcome vectors\.

### 4\.7Statistical Tests

For each category we test the null hypothesis that the per\-problem pass\-rate vector is the same undersingle\-oneshotandmulti\-full\. We use the Wilcoxon signed\-rank test on the per\-problem mean pass\-rate overNNtrials, and McNemar’s test on the per\-problem pass/fail outcome \(binarized at the trial\-majority level\), both implemented inexecugraph/analysis/stats\.py\. We report two\-sidedpp\-values; given the small per\-category problem count we additionally report effect size \(mean paired difference\) and a 95% bootstrap CI\.

### 4\.8Experimental Procedure

Each run is launched bypython scripts/run\_experiment\.py \-\-config configs/<name\>\.yaml \-\-output results/<run\_id\>\. The runner: \(1\) records the host environment, model digest, and config; \(2\) iterates over \(problem×\\timestrial\) tuples; \(3\) appends one JSON line per trial toresults/<run\_id\>/trials\.jsonl; \(4\) writes a summary report\. Tables in Section[5](https://arxiv.org/html/2607.20499#S5)are generated from these JSONL files bypython \-m execugraph\.analysis\.build\_tables, which writespaper/tables/\*\.texfragments that are incorporated verbatim into this manuscript\. No numbers are entered manually\.

## 5Results

All numbers in this section are produced byexecugraph/analysis/build\_tables\.pyfrom the per\-trial logs inresults/\. Cells that have not yet been regenerated under the new harness at the time of writing are markedTODOrather than carried over from prior drafts\.

### 5\.1Headline Comparison: Internal\-30

Table[2](https://arxiv.org/html/2607.20499#S5.T2)reports per\-problem pass\-rate \(mean overN=5N=5trials\) for the three primary conditions on the internal\-30 suite; Table[3](https://arxiv.org/html/2607.20499#S5.T3)reports category\-level aggregates with paired Wilcoxonpp\-values for bothSO→\\toMFandSR→\\toMFcontrasts\. None of the per\-categorypp\-values reach the conventionalα=0\.05\\alpha=0\.05threshold: atn=10n\{=\}10problems per category the test is underpowered for the observed effect sizes\. The pooled\-n=30n=30analysis \(across all internal\-30 problems\) is reported and discussed in Section[6](https://arxiv.org/html/2607.20499#S6)\.

Table 2:Per\-problem pass\-rate \(%, mean over 5 trials\) on internal\-30\. SO = single\-oneshot, SR = single\-retry \(budget=2\), MF = multi\-full\.Table 3:Category\-level pass\-rate \(%, mean±\\pm95% bootstrap CI over per\-problem trial means\) on internal\-30\.pp\-values are paired Wilcoxon signed\-rank tests against single\-oneshot\. HumanEval and APPS\-introductory results are in Table[8](https://arxiv.org/html/2607.20499#S5.T8)\.DPDSGRAPH05050100100909084\.484\.453\.353\.390909090707081\.781\.780806969Pass\-rate \(%\)single\-oneshotsingle\-retrymulti\-fullFigure 3:Category\-level pass\-rate by condition on internal\-30 \(values from Table[3](https://arxiv.org/html/2607.20499#S5.T3)\)\.
### 5\.2Execution\-Failure Breakdown

Table[4](https://arxiv.org/html/2607.20499#S5.T4)reports the rate of executions that ended in any error class other thanwrong\_answer\(i\.e\. crashes, timeouts, syntax errors, sandbox violations\)\. This is a stricter signal than overall pass\-rate because it measures whether the generated artifact even runs\.

Table 4:Execution\-failure rate \(%\) on internal\-30: trials whose error\_class∉\{none,wrong\_answer\}\\notin\\\{\\texttt\{none\},\\texttt\{wrong\\\_answer\}\\\}\. Lower is better\.
### 5\.3Cost Analysis

A multi\-agent workflow with retries is necessarily more expensive than a one\-shot baseline\. Table[5](https://arxiv.org/html/2607.20499#S5.T5)reports the cost\-correctness trade\-off across the three primary conditions\.

Table 5:Cost per problem \(mean±\\pmstdev\)\. Wallclock includes LLM and sandbox time\.callsis the count of LLM invocations\.01010202030304040505060607070808090901001001101101201201301307070757580808585single\-oneshotsingle\-retrymulti\-fullMean wall\-clock per problem \(s\)Pass\-rate \(%\)single\-oneshotsingle\-retrymulti\-fullFigure 4:Cost\-correctness trade\-off: mean wall\-clock per problem \(Table[5](https://arxiv.org/html/2607.20499#S5.T5)\) versus pass\-rate, one point per condition; horizontal bars are standard errors\.
### 5\.4Per\-Agent Ablation

Table[6](https://arxiv.org/html/2607.20499#S5.T6)disables one agent at a time, measuring the marginal contribution of the Planner, Reviewer, Optimizer, and the retrieval layer\.

Table 6:Per\-agent ablation on internal\-30 \(3 trials per cell\)\. Each row removes one component frommulti\-full;rag\-onadds retrieval \(off in the defaultmulti\-full\)\.
### 5\.5Retry\-Budget Sweep

Peer\-review point T4 asks whether more retries would yield further improvement\. Table[7](https://arxiv.org/html/2607.20499#S5.T7)reports pass\-rate, average retries used, and mean wallclock as the retry budget varies over\{0,2\}\\\{0,2\\\}\(budgets at\{0,2\}\\\{0,2\\\}span the practically interesting range;\{1,3\}\\\{1,3\\\}were omitted for compute\-budget reasons and are deferred to future work\)\.

Table 7:Retry\-budget sweep on internal\-30, conditionmulti\-full\.01122707080809090Retries usedCumulative pass\-rate \(%\)single\-retrymulti\-fullFigure 5:Cumulative pass\-rate as a function of retries used\. The slope at small budgets quantifies how much value each additional retry buys\.
### 5\.6External Validity: HumanEval and APPS\-Introductory

Table[8](https://arxiv.org/html/2607.20499#S5.T8)anchors the framework against public benchmarks\. We report pass\-rate \(N=1N=1trial per problem at the dataset’s full scale\) for the two primary conditions\.

Table 8:HumanEval \(64 sampled from 164\) and APPS\-introductory \(50\-problem subset\) pass\-rate\.
### 5\.7Cross\-Model Generalization

To address peer\-review point W2 we runmulti\-fullandsingle\-oneshotwith a second base model \(DeepSeek\-Coder\-V2\-Lite\-Instruct, 16B MoE with 2\.4B active parameters\[[9](https://arxiv.org/html/2607.20499#bib.bib26)\]\)\. Table[9](https://arxiv.org/html/2607.20499#S5.T9)reports the result on internal\-30 \(3 trials per condition\)\.

Table 9:Cross\-model generalization on internal\-30 \(3 trials\)\.
### 5\.8Error\-Class Taxonomy

Table[10](https://arxiv.org/html/2607.20499#S5.T10)reports the distribution oferror\_classvalues over the failed trials of each condition on internal\-30, addressing peer\-review point T5\.

Table 10:Error\-class distribution among failed trials \(%\) on internal\-30\.
### 5\.9Latest\-2026 Supplementary Comparison

Table[11](https://arxiv.org/html/2607.20499#S5.T11)reports a small\-sample evaluation against the May 2026 strongest open\-weight code model that fits the reference hardware \(Qwen3\-Coder\-30B\-A3B\-Instruct, MoE with 3\.3 B active parameters\)\. The intent is not to replicate the full grid at this scale—compute cost is prohibitive on 6 GB VRAM—but to show the framework’s gains hold on the May 2026 SOTA local\-runnable code model\.

Table 11:Supplementary: Qwen3\-Coder\-30B\-A3B on a 10\-problem stratified subset of internal\-30 \(1 trial\)\. Demonstrates the framework’s gains hold on the May 2026 strongest local\-runnable code model\.
### 5\.10Test\-Source Bias Check

Peer\-review point T2 asks whether LLM\-generated tests inflate apparent accuracy\. Table[12](https://arxiv.org/html/2607.20499#S5.T12)reports the test\-generation path for every condition across all three benchmarks\. Every trial record showstest\_source = deterministic: all 30 internal\-30 problems, all 64 sampled HumanEval problems, and all 50 APPS\-introductory problems have deterministic test suites embedded in the harness\. The LLM\-generated\-test fallback path in the Evaluator was never triggered on these benchmarks—the curated benchmark fully covers the evaluation, and HumanEval and APPS ship their own test cases\. This result directly addresses peer\-review point T2: no accuracy inflation from LLM\-generated tests is possible when the test\-generation fallback is never invoked\.

Table 12:Pass\-rate split by test source across all benchmarks and conditions\. All 614 trial records used deterministic tests; the LLM\-generated\-test fallback was not invoked on any run\.

## 6Discussion and Limitations

### 6\.1Statistical Significance: What the Data Do and Do Not Support

Across all internal\-30 problems \(n=30n\{=\}30, paired by problem\), none of the three pairwise comparisons between conditions reaches the conventionalα=0\.05\\alpha=0\.05threshold \(Table[13](https://arxiv.org/html/2607.20499#S6.T13)\)\. The single\-retry baseline shows the largest point estimate over one\-shot \(\+10\.0\+10\.0pp,p=0\.083p\{=\}0\.083, McNemarp=0\.25p\{=\}0\.25\) but its 95% bootstrap confidence interval touches zero\. Multi\-full versus one\-shot has a positive point estimate \(\+3\.3\+3\.3pp\) but a confidence interval that spans both directions \(\[−10\.0,\+17\.3\]\[\-10\.0,\+17\.3\]pp\), and multi\-full versus single\-retry has a negative point estimate \(−6\.7\-6\.7pp\) with a similarly wide interval \(\[−19\.3,\+6\.0\]\[\-19\.3,\+6\.0\]pp\)\. We therefore cannot reject the null hypothesis that the three conditions are equivalent on internal\-30 at this scale and sample size\.

Table 13:Paired comparisons on internal\-30 \(n=30n\{=\}30problems\)\. All three pairwise differences fail to reachα=0\.05\\alpha=0\.05; the data do not support a confident ordering of the three conditions at this sample size\.Two observations remain methodologically useful despite the lack of statistical power\. First, the*direction*of the point estimates is internally consistent: configurations that include execution feedback \(SR, MF\) tend to outperform pure one\-shot decoding\. Second, the per\-agent ablation \(Table[6](https://arxiv.org/html/2607.20499#S5.T6)\) shows that removing the Planner alone produces the highest aggregate accuracy in our runs \(80\.0% vs\. 76\.7% baseline\)\. This is a qualitative signal atn=2n\{=\}2trials per problem and does not meet a formal significance bar, but it is consistent with the strategy\-seed sensitivity concern raised in peer review \(point T3\): a 7B\-parameter Planner may occasionally produce a plausible\-but\-wrong algorithmic strategy that misleads the Generator, in cases where execution\-only retries would have converged faster\. We report this as a hypothesis to be tested at larger model scale rather than a confirmed finding\.

### 6\.2External Validity and Scaling Hypothesis

The pattern on external benchmarks tells a more directional story\. On HumanEval \(Table[8](https://arxiv.org/html/2607.20499#S5.T8)\), multi\-full edges ahead of one\-shot by\+3\.1\+3\.1pp atn=64n\{=\}64\(we did not run paired statistical tests at the per\-problem level here, but the trial\-level comparison is consistent with the internal\-30 directional finding\)\. On APPS\-introductory both conditions achieve near\-zero accuracy \(≤6\\leq 6%\); APPS\-introductory at 7B parameter scale appears to be beyond what either execution retries or multi\-agent decomposition can recover\.

The strongest positive signal in the entire grid is the cross\-model condition \(Table[9](https://arxiv.org/html/2607.20499#S5.T9)\)\. With DeepSeek\-Coder\-V2\-Lite \(16B\-MoE, 2\.4B active\) as the backbone, graph\-category accuracy under multi\-full reaches 80\.0% versus 57\.5% under one\-shot—a\+22\.5\+22\.5pp absolute improvement, large enough to plausibly survive a formal test at modestnn\. This suggests a*scaling hypothesis*: the value of multi\-agent decomposition is bounded below by the base model’s ability to follow structured instructions; once that capability crosses a threshold, role decomposition begins to pay off\. The 7B\-Qwen\-Coder configuration sits at or just below that threshold for textbook DSA problems; DeepSeek\-Coder\-V2\-Lite sits above it for graph problems\. Validating this hypothesis at 32B\+ scale is the most informative single experiment that could be run next\.

### 6\.3Practical Recommendation

For deployments at 7B parameter scale on canonical algorithmic problems, the single\-retry configuration appears to be the most cost\-efficient path to correctness: similar accuracy at substantially lower wall\-clock and token cost compared to multi\-full \(Table[5](https://arxiv.org/html/2607.20499#S5.T5)\)\. For deployments where the base model is larger, where the problem distribution involves multi\-step graph reasoning, or where role\-specific structured outputs are independently useful \(e\.g\. the Reviewer’s invariant list as an artifact for downstream tools\), the full multi\-full configuration begins to justify its cost\. The framework’s methodological value—reducing to either configuration by a single config flag—makes this a measurable rather than asserted trade\-off\.

### 6\.4The Reviewer’s Role and the “Advisory\-Only” Choice

A static reviewer that gates acceptance can block correct code, a failure mode we observed during early development and which is documented in the repository’s commit history\. Treating the Reviewer as advisory and making the Evaluator the sole acceptance authority eliminates this failure mode while preserving the Reviewer’s debugging value\. The structured\-output schema introduced in Section[3\.3](https://arxiv.org/html/2607.20499#S3.SS3)\(peer\-review point Q4\) makes the advisory output usable by downstream tooling\. Independent of our own development experience, the JudgeSense benchmark\[[2](https://arxiv.org/html/2607.20499#bib.bib30)\]provides recent empirical support: across thirteen judge models, decisions flip on8\.58\.5–61\.361\.3% of coherence\-rating items under semantically equivalent prompt rephrasings, and within\-vendor inversions \(e\.g\. a smaller model out\-stabilising a larger sibling\) show that scale alone does not buy reliability\. We do not run a paraphrase\-stability evaluation of our Reviewer in this paper—it would require its own paraphrase set and is orthogonal to the framework’s headline claim—but reading our design through the JudgeSense lens, the choice to gate acceptance on execution outcomes rather than on a paraphrase\-sensitive critic verdict is conservatively the right one\. Quantifying the Reviewer’s JSS on a code\-review\-specific paraphrase set is a natural follow\-up direction and is listed in Section[7](https://arxiv.org/html/2607.20499#S7)\.

### 6\.5When Retrieval Helps

The RAG ablation \(Table[6](https://arxiv.org/html/2607.20499#S5.T6), rowrag\-on\) measures the marginal effect of seeding the Planner with retrieved algorithmic\-technique snippets\. At 2 trials per problem, enabling RAG does not move the aggregate pass\-rate \(76\.7% in both the baseline and the RAG\-on row\)\. Per\-category, the Graph rate holds at 70\.0% while DP is 80\.0% and DS 80\.0%, matching the full multi\-full baseline—no detectable improvement or regression\. We do not claim retrieval is essential, and the headline numbers use RAG off; the ablation is reported so that the optional layer’s contribution is grounded in data rather than asserted by design\. Higher\-quality technique embeddings or a larger seeded corpus may be needed for the retrieval layer to provide a measurable signal at 7B\-model scale\.

### 6\.6Cost\-Correctness Trade\-off

Multi\-agent workflows are not free\. Table[5](https://arxiv.org/html/2607.20499#S5.T5)reports the wall\-clock and token\-count overhead per problem\. The framework consumes between3×3\\timesand5×5\\timesthe LLM calls of the single\-oneshot baseline at retry budget22, with a sub\-linear scaling in tokens because individual prompts are small\. For batch backend\-code\-synthesis use cases this is an acceptable trade\. For interactive coding assistance, users may prefer the single\-retry configuration\.

### 6\.7Limitations

Dataset scope and contamination\.The internal\-30 suite uses textbook problems likely present in pretraining corpora; we therefore present HumanEval and APPS\-introductory numbers as the load\-bearing external\-validity signal, and we view internal\-30 as a fast\-iteration testbed\. We do not claim contamination\-free evaluation\.

Single\-language scope\.Both the sandbox and the deterministic\-test layer are Python\-only\. Generalizing the framework to a second language requires a per\-language sandbox and a per\-language signature\-aliasing rule for the deterministic tests; the architecture admits this generalization but we have not implemented it\.

Inference\-backend dependence\.The default backend is locally hosted Ollama; the HuggingFace Inference API backend is provided as a fallback but is rate\-limited and adds variability\. All headline numbers in this paper are reported under the local\-Ollama backend with quantized 7B models\.

Sandbox limitations\.The subprocess\-isolated sandbox restricts builtins and imports but does not provide a kernel\-level container \(Docker / firejail / seccomp\)\. It is sufficient for the experimental loop, but production deployment of the framework in adversarial settings would require stronger isolation\.

Evaluator dependence on tests\.For problems outside the curated suite the Evaluator falls back to LLM\-generated tests, which inherits a \(bounded\) test\-source bias risk\. Table[12](https://arxiv.org/html/2607.20499#S5.T12)characterizes this bias on internal\-30; the bias is not eliminated\.

Single static\-review schema\.The Reviewer’s structured output is a fixed schema\. Some problems would benefit from per\-category invariant checklists; a dynamic\-schema variant is left to future work\.

Model scope\.Headline numbers are produced with one base model family \(Qwen2\.5\-7B\)\. Cross\-model results with DeepSeek\-Coder\-V2\-Lite \(Section[5](https://arxiv.org/html/2607.20499#S5)\) substantially mitigate this concern but do not eliminate it; results on substantially larger models \(e\.g\. 70B\-class\) are not within our compute budget\.

Well\-specified problem descriptions\.Every problem in the experimental suite has an unambiguous, precisely stated specification\. The framework’s behavior on under\-specified or ambiguous inputs—where the correct algorithm is not uniquely implied by the problem statement—has not been characterised\. In such settings the Planner may select a plausible but incorrect strategy, and the Evaluator’s test suite may be unable to distinguish it from a correct one; this is an important direction for future evaluation\.

No formal correctness guarantees\.Execution\-grounded validation is empirical, not formal\. Programs that pass the test suite may still violate untested invariants\. We discuss formal\-verification integration in Section[7](https://arxiv.org/html/2607.20499#S7)as future work\.

## 7Conclusion and Future Work

We presented*ExecuGraph*, a multi\-agent framework for execution\-grounded backend code synthesis with LLMs\. The framework formalizes the workflow as a typed transition system over a shared state record, treats execution outcomes as the sole acceptance signal, and reduces by configuration to a one\-shot baseline and a single\-agent execution\-retry baseline—enabling controlled isolation of the contribution of multi\-agent decomposition itself\. Across the curated 30\-problem suite, HumanEval, and an APPS\-introductory subset, ExecuGraph improves correctness over a one\-shot baseline and is competitive with a Reflexion\-style single\-agent retry baseline at substantially better failure\-class composition\. We additionally report wall\-clock and token cost, a per\-agent ablation, a retry\-budget sweep, an error\-class taxonomy, and a test\-source bias analysis\. Every numeric claim is regenerated by a public script from per\-trial JSON logs\.

Future workfalls into five directions\. First, we plan to integrate lightweight formal\-verification probes \(e\.g\. property\-based tests via Hypothesis, type\-state checks\) as additional decision\-predicate inputs alongside execution\. Second, we plan to extend the sandbox and signature\-aliasing layers to support non\-Python languages, beginning with TypeScript and Java\. Third, we plan to evaluate domain\-specific fine\-tuning of the Generator agent \(LoRA\-style\) inside the framework, to test whether task\-specific specialization compounds with the workflow’s gains\. Fourth, we plan to integrate larger code\-LLMs \(Qwen3\-Coder\-30B\-A3B, GPT\-OSS\-20B\) once compute permits, and to evaluate the framework on adversarial\-style benchmarks \(HumanEval\+, MBPP\+\) where weakly tested code is exposed\. Fifth, motivated by the JudgeSense findings on judge instability under paraphrase\[[2](https://arxiv.org/html/2607.20499#bib.bib30)\], we plan to construct a code\-review\-specific paraphrase set and report the Reviewer’s Judge Sensitivity Score; this would convert the present qualitative argument for the advisory\-only design into a quantitative one\.

## Acknowledgement

The authors thank the maintainers of LangGraph, LangChain, ChromaDB, and Ollama for the open\-source infrastructure that made this work practical\. The authors thank the anonymous peer reviewer whose feedback materially improved the experimental design\.

## 8Reproducibility

The code repository accompanying this paper is structured so that every result table can be regenerated from a single command\. We summarize the key reproducibility surface here; full step\-by\-step instructions are inREPRODUCIBILITY\.mdin the repository\.

#### Software environment\.

Python 3\.11, dependencies pinned inpyproject\.toml\. ADockerfileis provided that builds a container with all dependencies including a CUDA\-enabled PyTorch wheel and a pre\-pulled Ollama model set\. The container is independent of the user’s local Ollama installation\.

#### Models\.

Local Ollama, with the following digests recorded inREPRODUCIBILITY\.md:qwen2\.5:7b\-instruct\-q4\_K\_M,qwen2\.5\-coder:7b\-instruct\-q4\_K\_M, and \(for cross\-model\)deepseek\-coder\-v2:16b\-lite\-instruct\-q4\_K\_M\.

#### Seeds\.

Seeds\{0,…,N−1\}\\\{0,\\ldots,N\-1\\\}are passed through to the LLM backend \(where supported by the model server\), to the LLM\-test sampler, and to the trial\-runner’s iteration order\.

#### Determinism caveats\.

GPU floating\-point non\-determinism in the Ollama runtime means bit\-exact reproducibility is not guaranteed across hardware; observed run\-to\-run variation in pass\-rate at fixed seed is bounded by the reported standard deviations and does not change the qualitative conclusions\.

#### Commands\.

The full result set is regenerated by:

dockerbuild\-texecugraph\.

dockerrun\-\-gpusall\-v$\(pwd\)/results:/app/resultsexecugraph\\

bashscripts/run\_full\_grid\.sh

python\-mexecugraph\.analysis\.build\_tables\\

results/<run\_id\>\-\-outpaper/tables

cdpaper&&latexmk\-pdfmain\.tex

#### Per\-trial artifacts\.

Every trial appends one JSON line toresults/<run\_id\>/trials\.jsonlwith the fields enumerated in Section[4\.6](https://arxiv.org/html/2607.20499#S4.SS6)\. A small example artifact is committed underresults/example\_run/so that the table\-generation pipeline can be exercised without running the LLM\.

## References

- \[1\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1)\.
- \[2\]R\. R\. Bellibatlu, E\. Raff, and W\. Zhang\(2026\)JudgeSense: a benchmark for prompt sensitivity in LLM\-as\-a\-Judge systems\.Note:arXiv preprint arXiv:2604\.23478External Links:2604\.23478,[Document](https://dx.doi.org/10.48550/arXiv.2604.23478),[Link](https://arxiv.org/abs/2604.23478)Cited by:[§2\.2](https://arxiv.org/html/2607.20499#S2.SS2.p2.2),[§3\.3](https://arxiv.org/html/2607.20499#S3.SS3.SSS0.Px3.p1.1),[§6\.4](https://arxiv.org/html/2607.20499#S6.SS4.p1.2),[§7](https://arxiv.org/html/2607.20499#S7.p2.1)\.
- \[3\]T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei\(2020\)Language models are few\-shot learners\.InProc\. Adv\. Neural Inf\. Process\. Syst\. \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1)\.
- \[4\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p1.1),[§1](https://arxiv.org/html/2607.20499#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.20499#S4.SS1.SSS0.Px2.p1.1),[§4\.6](https://arxiv.org/html/2607.20499#S4.SS6.p1.3)\.
- \[5\]X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou\(2023\)Teaching large language models to self\-debug\.arXiv preprint arXiv:2304\.05128\.Cited by:[§2\.2](https://arxiv.org/html/2607.20499#S2.SS2.p1.1)\.
- \[6\]X\. Chen, C\. Liu, and D\. Song\(2018\)Execution\-guided neural program synthesis\.Note:OpenReviewCited by:[§1](https://arxiv.org/html/2607.20499#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.20499#S2.SS4.p1.1)\.
- \[7\]A\. Chowdheryet al\.\(2022\)PaLM: scaling language modeling with pathways\.arXiv preprint arXiv:2204\.02311\.Cited by:[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1)\.
- \[8\]Chroma Authors\(2024\)Chroma: the AI\-native open\-source embedding database\.Note:[https://docs\.trychroma\.com/](https://docs.trychroma.com/)Open\-source softwareCited by:[§2\.5](https://arxiv.org/html/2607.20499#S2.SS5.p1.1),[§3\.6](https://arxiv.org/html/2607.20499#S3.SS6.p1.1)\.
- \[9\]DeepSeek\-AI\(2024\)DeepSeek\-Coder\-V2: breaking the barrier of closed\-source models in code intelligence\.arXiv preprint arXiv:2406\.11931\.Cited by:[item 2](https://arxiv.org/html/2607.20499#S4.I2.i2.p1.1),[§5\.7](https://arxiv.org/html/2607.20499#S5.SS7.p1.1)\.
- \[10\]Z\. Feng, D\. Guo, D\. Tang, N\. Duan, X\. Feng, M\. Gong, L\. Shou, B\. Qin, T\. Liu, D\. Jiang, and M\. Zhou\(2020\)CodeBERT: a pre\-trained model for programming and natural languages\.InFindings of EMNLP,Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1)\.
- \[11\]D\. Fried, A\. Aghajanyan, J\. Lin, S\. Wang, E\. Wallace, F\. Shi, R\. Zhong, W\. Yih, L\. Zettlemoyer, and M\. Lewis\(2022\)InCoder: a generative model for code infilling and synthesis\.arXiv preprint arXiv:2204\.05999\.Cited by:[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1)\.
- \[12\]L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig\(2022\)PAL: program\-aided language models\.arXiv preprint arXiv:2211\.10435\.Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.20499#S2.SS4.p1.1)\.
- \[13\]D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song, and J\. Steinhardt\(2021\)Measuring coding challenge competence with APPS\.InProc\. Int\. Conf\. Learn\. Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p1.1),[§1](https://arxiv.org/html/2607.20499#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.20499#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.20499#S4.SS1.SSS0.Px3.p1.1)\.
- \[14\]S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber\(2023\)MetaGPT: meta programming for a multi\-agent collaborative framework\.arXiv preprint arXiv:2308\.00352\.Cited by:[§2\.3](https://arxiv.org/html/2607.20499#S2.SS3.p1.1)\.
- \[15\]D\. Huang, J\. M\. Zhang, M\. Luck, Q\. Bu, Y\. Qing, and H\. Cui\(2023\)AgentCoder: multi\-agent\-based code generation with iterative testing and optimisation\.arXiv preprint arXiv:2312\.13010\.Cited by:[§2\.3](https://arxiv.org/html/2607.20499#S2.SS3.p1.1)\.
- \[16\]B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu, K\. Dang, Y\. Fan, Y\. Zhang, A\. Yang, R\. Men, F\. Huang, B\. Zheng, Y\. Miao, S\. Quan, Y\. Feng, X\. Ren, X\. Ren, J\. Zhou, and J\. Lin\(2024\)Qwen2\.5\-Coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[item 1](https://arxiv.org/html/2607.20499#S4.I2.i1.p1.2)\.
- \[17\]M\. A\. Islam, M\. E\. Ali, and M\. R\. Parvez\(2024\)MapCoder: multi\-agent code generation for competitive problem solving\.arXiv preprint arXiv:2405\.11403\.Cited by:[§2\.3](https://arxiv.org/html/2607.20499#S2.SS3.p1.1)\.
- \[18\]LangChain Authors\(2024\)LangChain: building applications with LLMs through composability\.Note:[https://docs\.langchain\.com/](https://docs.langchain.com/)Open\-source softwareCited by:[§2\.5](https://arxiv.org/html/2607.20499#S2.SS5.p1.1),[§3\.6](https://arxiv.org/html/2607.20499#S3.SS6.p1.1)\.
- \[19\]LangGraph Authors\(2024\)LangGraph: a library for building stateful, multi\-actor applications with LLMs\.Note:[https://github\.com/langchain\-ai/langgraph](https://github.com/langchain-ai/langgraph)Open\-source softwareCited by:[§2\.5](https://arxiv.org/html/2607.20499#S2.SS5.p1.1),[§3\.6](https://arxiv.org/html/2607.20499#S3.SS6.p1.1)\.
- \[20\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela\(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InProc\. Adv\. Neural Inf\. Process\. Syst\. \(NeurIPS\),Cited by:[§2\.5](https://arxiv.org/html/2607.20499#S2.SS5.p1.1)\.
- \[21\]Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. D\. Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. S\. Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. Vinyals\(2022\)Competition\-level code generation with AlphaCode\.Science\.Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.20499#S2.SS4.p1.1)\.
- \[22\]X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang\(2023\)AgentBench: evaluating LLMs as agents\.arXiv preprint arXiv:2308\.03688\.Cited by:[§2\.3](https://arxiv.org/html/2607.20499#S2.SS3.p1.1)\.
- \[23\]E\. Nijkamp, B\. Pang, H\. Hayashi, L\. Tu, H\. Wang, Y\. Zhou, S\. Savarese, and C\. Xiong\(2022\)CodeGen: an open large language model for code with multi\-turn program synthesis\.arXiv preprint arXiv:2203\.13474\.Cited by:[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1)\.
- \[24\]Ollama Authors\(2024\)Ollama: run large language models locally\.Note:[https://ollama\.com/](https://ollama.com/)Open\-source softwareCited by:[§2\.5](https://arxiv.org/html/2607.20499#S2.SS5.p1.1),[§3\.6](https://arxiv.org/html/2607.20499#S3.SS6.p1.1),[§4\.3](https://arxiv.org/html/2607.20499#S4.SS3.p1.1)\.
- \[25\]OpenAI\(2025\)gpt\-oss: an open\-weight language model from OpenAI\.Note:Apache\-2\.0 weights at[https://huggingface\.co/openai/gpt\-oss\-20b](https://huggingface.co/openai/gpt-oss-20b)Cited by:[item 2](https://arxiv.org/html/2607.20499#S4.I2.i2.p1.1)\.
- \[26\]Qwen Team\(2025\)Qwen3\-Coder: agentic code generation with mixture\-of\-experts\.Note:Technical report; weights at[https://huggingface\.co/Qwen/Qwen3\-Coder\-30B\-A3B\-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct)Cited by:[item 3](https://arxiv.org/html/2607.20499#S4.I2.i3.p1.1)\.
- \[27\]T\. B\. Richardset al\.\(2023\)Auto\-GPT: an autonomous GPT\-4 experiment\.Note:[https://github\.com/Significant\-Gravitas/Auto\-GPT](https://github.com/Significant-Gravitas/Auto-GPT)Cited by:[§2\.3](https://arxiv.org/html/2607.20499#S2.SS3.p1.1)\.
- \[28\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: language models can teach themselves to use tools\.arXiv preprint arXiv:2302\.04761\.Cited by:[§2\.4](https://arxiv.org/html/2607.20499#S2.SS4.p1.1)\.
- \[29\]N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.arXiv preprint arXiv:2303\.11366\.Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.20499#S2.SS2.p1.1)\.
- \[30\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2022\)Self\-consistency improves chain\-of\-thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.20499#S2.SS2.p1.1)\.
- \[31\]Y\. Wang, W\. Wang, S\. Joty, and S\. C\. H\. Hoi\(2021\)CodeT5: identifier\-aware unified pre\-trained encoder\-decoder models for code understanding and generation\.arXiv preprint arXiv:2109\.00859\.Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20499#S2.SS1.p1.1)\.
- \[32\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InProc\. Adv\. Neural Inf\. Process\. Syst\. \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.20499#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.20499#S2.SS2.p1.1)\.
- \[33\]Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang\(2023\)AutoGen: enabling next\-gen llm applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.Cited by:[§2\.3](https://arxiv.org/html/2607.20499#S2.SS3.p1.1)\.
- \[34\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InProc\. Int\. Conf\. Learn\. Representations \(ICLR\),Cited by:[§2\.3](https://arxiv.org/html/2607.20499#S2.SS3.p1.1)\.