From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

# From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning
Source: [https://arxiv.org/html/2608.21897](https://arxiv.org/html/2608.21897)
###### Abstract

Reliable planning requires converting natural\-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways\. We study how to learn faithful natural\-language\-to\-PDDL formalization using only solver feedback, without human\-written demonstrations\. We propose a solver\-grounded multi\-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair\. The Actor proposes PDDL specifications, the Judge provides a solver\-calibrated quality signal, and the Editor performs bounded diagnostic\-conditioned refinement\. On PlanBench, our method improves average success from 35\.5% for LLM\+P to 70\.8%, achieves 66\.3% faithful success, and reduces semantic drift to 6\.4%\. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation\-free symbolic planning\.

††footnotetext:1University of Pennsylvania\.††footnotetext:2Unaffiliated\.††footnotetext:3Georgia Institute of Technology\.††footnotetext:4Pace University\.## 1Introduction

Structured planning is a central capability for intelligent agents that must transform high\-level instructions into executable action models under constraints\. It is critical for robotics, workflow automation, logistics, and interactive decision\-making, where success depends not only on producing plausible language but also on satisfying explicit state\-transition semantics\. Large language models \(LLMs\) have shown impressive general\-purpose reasoning and instruction\-following abilities\([Brown et al\. 2020](https://arxiv.org/html/2608.21897#bib.bib1);[OpenAI 2024](https://arxiv.org/html/2608.21897#bib.bib14);[Touvron et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib24);[Lin et al\. 2026](https://arxiv.org/html/2608.21897#bib.bib10)\), yet reliable planning remains difficult because valid plans require logical consistency, long\-horizon state tracking, and faithful grounding in task constraints\.

Recent benchmarks have made this limitation increasingly clear\. PlanBench shows that LLMs often fail on classical planning domains, especially when predicate names are obfuscated and surface lexical cues are removed\([Valmeekam et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib25)\)\. This supports the broader view that LLMs often behave as approximate retrievers rather than systematic planners over state transitions\([Kambhampati et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib8)\)\. Even reasoning\-enhanced models and chain\-of\-thought prompting remain brittle under planning\-specific perturbations, suggesting that fluent intermediate reasoning is not equivalent to executable planning\([Stechly et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib22);[Valmeekam et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib26)\)\. NATURAL PLAN further shows that realistic natural\-language planning becomes sharply harder as constraint complexity increases, and self\-correction does not reliably recover valid solutions\([Zheng et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib33)\)\.

A promising response is to use LLMs not as direct planners, but as formalizers that translate natural\-language tasks into symbolic representations such as Planning Domain Definition Language \(PDDL\), after which an external solver performs the actual planning\. LLM\+P demonstrates the value of this neuro\-symbolic decomposition by delegating search to a classical planner\([Liu et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib11)\), and subsequent work explores PDDL goal translation, generalized planning programs, and environment\-aided PDDL construction\([Xie et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib29);[Silver et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib20);[Mahdavi et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib13)\)\. However, these approaches still face a fundamental supervision bottleneck: high\-quality natural\-language\-to\-PDDL formalization typically requires curated demonstrations, fixed domain assumptions, or supervised initialization\. Instruction\-tuning methods such as PDDL\-Instruct improve symbolic planning performance, but their reliance on annotated examples limits scalability to new domains and language distributions\([Verma et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib27)\)\.

![Refer to caption](https://arxiv.org/html/2608.21897v1/teaser.png)Figure 1:One Brain Three Roles framework\. A single LLM plays Actor, Judge, and Editor roles with shared parameters, grounded by deterministic PDDL solver feedback\.The deeper challenge is that solver feedback alone is not automatically a faithful learning signal\. A PDDL solver can verify whether a generated specification is syntactically valid and executable, but solver success is defined with respect to the generated formal specification rather than the original natural\-language intent\. Thus, optimizing only for solver success can reward underspecified or semantically shifted formalizations that are easy to solve but no longer faithful to the task\. This creates a high\-level conflict between*annotation\-free learning*and*semantic faithfulness*: the former demands replacing human labels with environment feedback, while the latter requires preventing the model from exploiting imperfections or ambiguities in that feedback\. Similar concerns arise in broader reinforcement learning settings, where optimizing a proxy reward can induce reward gaming when the reward does not fully capture the intended objective\([Skalse et al\. 2022](https://arxiv.org/html/2608.21897#bib.bib21)\)\.

This paper asks:*Can an LLM learn reliable symbolic planning from solver feedback alone, without any human\-annotated PDDL demonstrations?*We address this question by treating the symbolic solver as the only grounded oracle while organizing learning around generation, verification, and repair\. At a high level, the model learns not only to produce a candidate formal specification, but also to evaluate solver\-grounded correctness and refine failed specifications using executable diagnostics, thereby converting sparse solver outcomes into a more informative training process\.

We proposeSolver\-Grounded Multi\-Role Reinforcement Learning, a zero\-annotation framework for natural\-language\-to\-PDDL planning\. A single LLM is conditioned to play three coordinated roles: an Actor that proposes formal specifications, a Judge that learns a solver\-calibrated quality signal, and an Editor that performs bounded repair from solver diagnostics\. This design preserves the scalability of solver\-only supervision while reducing the risk that optimization collapses into purely solver\-facing shortcuts\.

Our contributions are as follows:

- •We identify a key obstacle in annotation\-free symbolic planning: raw solver success is a necessary but insufficient learning signal because it can diverge from semantic faithfulness to the original natural\-language task\. This reframes NL\-to\-PDDL learning as a problem of solver\-grounded optimization under potential specification drift\.
- •We introduce a multi\-role reinforcement learning framework in which generation, verification, and repair are learned jointly from deterministic solver feedback, without human\-annotated PDDL demonstrations\. The shared\-role design enables verifier co\-evolution and diagnostic\-conditioned refinement while keeping most model parameters coupled across roles\.
- •Across PlanBench domains, our method achieves strong in\-domain planning success, including substantial gains on obfuscated and structurally challenging settings\. It also transfers zero\-shot to ProntoQA and NATURAL PLAN, indicating that solver\-grounded multi\-role learning improves not only benchmark\-specific PDDL synthesis but also broader planning and constraint\-satisfaction behavior\.

## 2Related Work

##### LLMs as planners\.

A first line asks whether scaling and prompting alone are sufficient for planning\. Chain\-of\-thought and search\-based deliberation can improve intermediate reasoning\([Wei et al\. 2022](https://arxiv.org/html/2608.21897#bib.bib28);[Yao et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib30);[Hao et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib5)\), but planning benchmarks show that fluent rationales do not guarantee executable state\-transition reasoning\. PlanBench and follow\-up analyses reveal large gaps between textual plausibility and valid planning behavior, especially under predicate obfuscation, longer horizons, and reasoning\-model variants\([Valmeekam et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib25);[Kambhampati et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib8);[Stechly et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib22);[Valmeekam et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib26)\)\. NATURAL PLAN further shows that realistic language planning degrades sharply as constraints become more complex\([Zheng et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib33)\)\. These works motivate execution\-grounded evaluation, whereas we use solver feedback as the training signal, optimizing planning competence through repeated interaction with a symbolic verifier\.

##### LLM\-as\-formalizer and neuro\-symbolic planning\.

A second line shifts from direct plan generation to inducing formal representations that planners can solve\. LLM\+P demonstrates this paradigm by translating natural language into PDDL, invoking a classical planner, and verbalizing the resulting plan\([Liu et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib11)\); subsequent work studies goal translation, generalized planning programs, domain generation, and text\-to\-PDDL evaluation\([Xie et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib29);[Silver et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib20);[Oswald et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib15);[Zuo et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib35);[Zhang et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib31)\)\. Recent studies show that LLM\-as\-formalizer can outperform direct planning, but still suffers from semantic omissions, naturalness shift, and scaling failures in large formal structures\([Huang and Zhang 2025](https://arxiv.org/html/2608.21897#bib.bib6);[Jiang et al\. 2026](https://arxiv.org/html/2608.21897#bib.bib7)\)\. Supervised methods such as PDDL\-Instruct improve performance with annotated instruction tuning\([Verma et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib27)\), while environment\-interaction methods refine PDDL through feedback\([Mahdavi et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib13)\)\. In contrast, we formulate NL\-to\-PDDL induction as annotation\-free reinforcement learning, where an Actor generates specifications, a Judge learns solver\-calibrated scoring, and an Editor performs diagnostic\-conditioned repair\.

##### Feedback\-driven self\-improvement and verifiable rewards\.

A third line explores feedback\-driven LLM self\-improvement without dense human supervision\. Reflexion, Self\-Refine, ISR\-LLM, and AdaPlanner use critique–revise or environment\-feedback loops\([Shinn et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib19);[Madaan et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib12);[Zhou et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib34);[Sun et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib23)\), while LLM\-as\-a\-judge methods reduce annotation cost but risk evaluator bias and reward hacking without external grounding\([Gu et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib3);[Li et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib9)\)\. Recent RLVR and self\-play systems show that verifiable feedback can scale reasoning training\([Shao et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib18);[Guo et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib4);[Zhao et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib32);[Chen et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib2)\), but they mainly target curated tasks or code/math verifiers and do not address the gap between solver success and semantic faithfulness\. Our work brings verifiable\-reward learning to symbolic planning by grounding Actor, Judge, and Editor roles in a deterministic PDDL solver, converting sparse executability feedback into calibrated verification and repair signals\.

## 3Method

Figure 2:Solver\-Grounded Multi\-Role RL for NL\-to\-PDDL Planning\.A single shared LLM backbone is conditioned into Actor, Judge, and Editor roles\. The Actor generates an initial PDDL specification\. A deterministic PDDL Solver verifies executability, producing a success label and diagnostics on failure\. The Judge predicts a calibrated solver score, and the Editor uses diagnostics to perform bounded local edits, feeding back to the solver\.### 3\.1Task and Solver\-Grounded Environment

We study natural\-language\-to\-PDDL specification induction without human annotations\. Each task is a natural\-language planning descriptionx∼𝒫datax\\sim\\mathcal\{P\}\_\{\\mathrm\{data\}\}\. The model outputs a PDDL specificationy=\(ydom,yprob\),y=\(y^\{\\mathrm\{dom\}\},y^\{\\mathrm\{prob\}\}\),consisting of a domain file and a problem file\. A deterministic PDDL environmentℰ\\mathcal\{E\}, implemented with Fast Downward, verifies the generated specification and returns

\(b,d\)=ℰ⁡\(y\),b∈\{0,1\},\(b,d\)=\\mathcal\{E\}\(y\),\\qquad b\\in\\\{0,1\\\},\(1\)whereb=1b=1indicates that the specification is syntactically valid and solver\-executable within the time limit, andddcontains structured diagnostics such as parse errors, type mismatches, unsatisfied preconditions, unreachable goals, and failed action traces\.

The central challenge is that solver success is not identical to task\-level semantic faithfulness\. Because the model controls the generated specification, optimizing onlybbcan reward degenerate specifications that weaken goals, remove constraints, or alter predicates to make the instance easier than the original task\. Our method keeps the solver as the only external feedback source, but separates the learning problem into generation, calibration, and repair roles so that sparse solver outcomes become more useful for policy optimization\.

### 3\.2Role\-Conditioned Multi\-Role Policy

We use a single language model with shared backbone parameters and lightweight role\-specific parameters\. Letρ∈\{A,J,E\}\\rho\\in\\\{\\mathrm\{A\},\\mathrm\{J\},\\mathrm\{E\}\\\}denote the role identifier for Actor, Judge, and Editor\. Each role has a learned embeddingeρe\_\{\\rho\}, and the role\-conditioned model defines

πθ\(⋅∣c,ρ\)=LMθ\(⋅∣\[eρ;c\]\),\\pi\_\{\\theta\}\(\\cdot\\mid c,\\rho\)=\\mathrm\{LM\}\_\{\\theta\}\\\!\\left\(\\cdot\\mid\[e\_\{\\rho\};c\]\\right\),\(2\)whereccis the role\-specific context\. The Actor context isxx, the Judge context is\(x,y\)\(x,y\), and the Editor context is\(x,yt,dt\)\(x,y\_\{t\},d\_\{t\}\), whereyty\_\{t\}is the current specification anddtd\_\{t\}is the latest solver diagnostic\.

The parameters are partitioned as

θ=θsh∪θA∪θJ∪θE\.\\theta=\\theta\_\{\\mathrm\{sh\}\}\\cup\\theta\_\{\\mathrm\{A\}\}\\cup\\theta\_\{\\mathrm\{J\}\}\\cup\\theta\_\{\\mathrm\{E\}\}\.\(3\)The shared backboneθsh\\theta\_\{\\mathrm\{sh\}\}contains approximately95%95\\%of the parameters, while the remaining parameters are lightweight role\-specific heads\. This design allows each role to specialize while keeping most representation learning coupled across generation, verification, and repair\.

### 3\.3Solver\-Grounded Training Episode

Algorithm[1](https://arxiv.org/html/2608.21897#alg1)summarizes one training episode\. The Actor first samples an initial PDDL specification\. The solver verifies it and returns a binary success label with diagnostics\. The Judge predicts a scalar score for the generated specification\. If the initial specification fails, the Editor performs at mostHmaxH\_\{\\max\}repair steps, each conditioned on the task, the current specification, and the latest diagnostic\. The episode stops when the solver succeeds or the repair budget is exhausted\.

Algorithm 1One solver\-grounded multi\-role training episode0:Task

xx, solver

ℰ\\mathcal\{E\}, maximum repair horizon

HmaxH\_\{\\max\}
1:Sample initial specification

y0∼πθ\(⋅∣x,A\)y\_\{0\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,\\mathrm\{A\}\)
2:Query solver

\(b0,d0\)=ℰ⁡\(y0\)\(b\_\{0\},d\_\{0\}\)=\\mathcal\{E\}\(y\_\{0\}\)
3:Compute Judge score

sJ​\(x,y0\)s\_\{J\}\(x,y\_\{0\}\)
4:Set

T←0T\\leftarrow 0,

yT←y0y\_\{T\}\\leftarrow y\_\{0\},

bT←b0b\_\{T\}\\leftarrow b\_\{0\},

dT←d0d\_\{T\}\\leftarrow d\_\{0\}
5:for

t=0,…,Hmax−1t=0,\\ldots,H\_\{\\max\}\-1do

6:if

bT=1b\_\{T\}=1then

7:break

8:endif

9:Sample repaired specification

yt\+1∼πθ\(⋅∣x,yt,dt,E\)y\_\{t\+1\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{t\},d\_\{t\},\\mathrm\{E\}\)
10:Query solver

\(bt\+1,dt\+1\)=ℰ⁡\(yt\+1\)\(b\_\{t\+1\},d\_\{t\+1\}\)=\\mathcal\{E\}\(y\_\{t\+1\}\)
11:Set

T←t\+1T\\leftarrow t\+1,

yT←yt\+1y\_\{T\}\\leftarrow y\_\{t\+1\},

bT←bt\+1b\_\{T\}\\leftarrow b\_\{t\+1\},

dT←dt\+1d\_\{T\}\\leftarrow d\_\{t\+1\}
12:endfor

13:Update Actor, Judge, and Editor using Eq\. \([5](https://arxiv.org/html/2608.21897#S3.E5)\) and Eq\. \([6](https://arxiv.org/html/2608.21897#S3.E6)\)

At inference time, the same procedure is used without gradient updates\. The system returns the first solver\-executable specification found within the repair horizon; if no repair succeeds, it returns the final edited specification for evaluation\. Decoding and optimization details are given in Appendix[B\.2](https://arxiv.org/html/2608.21897#A2.SS2)\.

### 3\.4Role\-Specific Learning Signals

The Actor learns global specification generation, the Judge learns solver\-calibrated verification, and the Editor learns diagnostic\-conditioned repair\. The Judge score is produced by a scalar head on top of the shared representation:

sJ​\(x,y\)=σ⁡\(gθ​\(\[eJ;x;y\]\)\)∈\(0,1\),s\_\{J\}\(x,y\)=\\sigma\\\!\\left\(g\_\{\\theta\}\(\[e\_\{\\mathrm\{J\}\};x;y\]\)\\right\)\\in\(0,1\),\(4\)wheregθg\_\{\\theta\}is the Judge logit head andσ\\sigmais the sigmoid function\.

For an episode with initial outputy0y\_\{0\}, initial solver labelb0b\_\{0\}, final outputyTy\_\{T\}, final solver labelbTb\_\{T\}, and repair lengthTT, the role\-specific learning signals are

RA​\(x,y0\)\\displaystyle R\_\{\\mathrm\{A\}\}\(x,y\_\{0\}\)=b0\+λJ​sJ​\(x,y0\),\\displaystyle=b\_\{0\}\+\\lambda\_\{\\mathrm\{J\}\}\\,s\_\{J\}\(x,y\_\{0\}\),\(5\)ℒJ​\(x,y0,b0\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{J\}\}\(x,y\_\{0\},b\_\{0\}\)=BCE⁡\(sJ​\(x,y0\),b0\)\+λband​ℓband​\(sJ​\(x,y0\),b0\),\\displaystyle=\\mathrm\{BCE\}\\\!\\left\(s\_\{J\}\(x,y\_\{0\}\),b\_\{0\}\\right\)\+\\lambda\_\{\\mathrm\{band\}\}\\ell\_\{\\mathrm\{band\}\}\\\!\\left\(s\_\{J\}\(x,y\_\{0\}\),b\_\{0\}\\right\),RE\(x,y0:T\)\\displaystyle R\_\{\\mathrm\{E\}\}\(x,y\_\{0:T\}\)=bT−β​T\.\\displaystyle=b\_\{T\}\-\\beta T\.The Actor reward combines the initial solver label with Judge\-based shaping\. The Judge loss trainssJs\_\{J\}to predict solver success and penalizes over\-confident false positives\. The Editor reward favors successful repairs with fewer editing steps\.

Actor and Editor token policies are optimized with PPO, while the Judge is optimized as a calibrated binary predictor\. The total loss is

ℒtotal=ℒPPOA\+λE​ℒPPOE\+λJ​ℒJ\.\\mathcal\{L\}\_\{\\mathrm\{total\}\}=\\mathcal\{L\}\_\{\\mathrm\{PPO\}\}^\{\\mathrm\{A\}\}\+\\lambda\_\{\\mathrm\{E\}\}\\mathcal\{L\}\_\{\\mathrm\{PPO\}\}^\{\\mathrm\{E\}\}\+\\lambda\_\{\\mathrm\{J\}\}\\mathcal\{L\}\_\{\\mathrm\{J\}\}\.\(6\)Appendix[B\.2](https://arxiv.org/html/2608.21897#A2.SS2)specifies the PPO objective, banded Judge penalty, decoding policy, and hyperparameters\.

### 3\.5Judge Calibration and Correctness Connection

The Judge is trained against solver outcomes rather than human semantic labels\. Its direct target is therefore solver\-verified executability\. The connection to task\-level correctness depends on how often solver success agrees with a reference correctness notion on the policy\-induced distribution\. LetB∈\{0,1\}B\\in\\\{0,1\\\}denote solver success andC∈\{0,1\}C\\in\\\{0,1\\\}denote task\-level correctness for a generated pair\(X,Y\)\(X,Y\)\. If the on\-policy disagreement rate satisfiesPr⁡\(B≠C\)≤ε\\Pr\(B\\neq C\)\\leq\\varepsilon, and the learned Judge has calibration errorδJ\\delta\_\{J\}with respect to the Bayes\-optimal solver\-success predictor, then

𝔼\(X,Y\)​\[\|sJ​\(X,Y\)−C⁡\(X,Y\)\|\]≤ε\+δJ\.\\mathbb\{E\}\_\{\(X,Y\)\}\\left\[\\left\|s\_\{J\}\(X,Y\)\-C\(X,Y\)\\right\|\\right\]\\leq\\varepsilon\+\\delta\_\{J\}\.\(7\)Thus, the Judge provides a reliable shaping signal when solver success and task\-level correctness have limited disagreement on generated samples\. Appendix[C](https://arxiv.org/html/2608.21897#A3)proves Eq\. \([7](https://arxiv.org/html/2608.21897#S3.E7)\), gives the corresponding ranking guarantee, and describes how to estimate the disagreement rate\.

### 3\.6Why Separate Actor, Judge, and Editor Roles

The three roles address different sources of difficulty in annotation\-free symbolic planning\. The Actor performs a global mapping from natural language to a complete PDDL domain–problem pair\. The Judge converts sparse solver outcomes into a calibrated score that can guide learning\. The Editor uses solver diagnostics to make bounded local repairs after failure\. A single final\-outcome policy can leave the initial specification weakly constrained once local repair becomes strong, because final solver success may no longer distinguish good initial formalizations from poor but repairable ones\. Appendix[D](https://arxiv.org/html/2608.21897#A4)formalizes this failure mode\.

### 3\.7Parameter Sharing and Reward\-Hacking Directions

We now analyze why the shared\-backbone design is less permissive than three fully separate role models in a local reward\-hacking sense\. LetJA​\(θ\)J\_\{\\mathrm\{A\}\}\(\\theta\)denote the Actor objective and letC⁡\(θ\)C\(\\theta\)denote expected task\-level correctness\. Around a reference pointθ0\\theta\_\{0\}, define the local reward\-hacking gradient

ghack​\(θ0\)=∇θJA​\(θ0\)−a​∇θC​\(θ0\),a\>0\.g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)=\\nabla\_\{\\theta\}J\_\{\\mathrm\{A\}\}\(\\theta\_\{0\}\)\-a\\nabla\_\{\\theta\}C\(\\theta\_\{0\}\),\\qquad a\>0\.\(8\)For a unit\-norm perturbationΔ​θ\\Delta\\theta, the first\-order Actor gain not explained by correctness is

Δhack​\(θ0,Δ​θ\)=ghack​\(θ0\)⊤​Δ​θ\.\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)=g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)^\{\\top\}\\Delta\\theta\.The admissible perturbation set depends on the architecture\. With three separate models, the Actor can move in its ownPAP\_\{\\mathrm\{A\}\}\-dimensional parameter space without directly changing Judge or Editor behavior\. With a shared backbone, Judge\- and Editor\-neutral perturbations are restricted to the small role\-private subspace, provided the shared backbone directions are observed by Judge and Editor objectives\. Appendix[E](https://arxiv.org/html/2608.21897#A5)states the assumptions and proof\.

###### Theorem 3\.1\(Reward hacking scaling with parameter count\)\.

Under the local isotropic\-gradient model and the shared\-backbone constraint in Appendix[E](https://arxiv.org/html/2608.21897#A5), the expected worst\-case first\-order reward\-hacking gain satisfies:

1. 1\.Three fully separate models\.If the Actor hasPAP\_\{\\mathrm\{A\}\}parameters, then 𝔼⁡\[supΔ​θ∈ℋsepΔhack​\(θ0,Δ​θ\)\]≍σ​PA\.\\mathbb\{E\}\\left\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{sep\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\right\]\\asymp\\sigma\\sqrt\{P\_\{\\mathrm\{A\}\}\}\.\(9\)
2. 2\.Shared\-backbone architecture\.If the Judge\- and Editor\-neutral perturbations are contained in an Actor head of dimensionhh, then 𝔼⁡\[supΔ​θ∈ℋshrΔhack​\(θ0,Δ​θ\)\]≲σ​h\.\\mathbb\{E\}\\left\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\right\]\\lesssim\\sigma\\sqrt\{h\}\.\(10\)

The theorem shows that separate models allow reward\-hacking gains to grow with the Actor parameter count\. By contrast, a shared backbone exposes most directions to Judge and Editor objectives, leaving only the small Actor head as the main role\-private subspace\. Thus, parameter sharing does not eliminate reward hacking, but reduces the local degrees of freedom for improving Actor reward without corresponding verification and repair changes\.

Table 1:Planning success rate \(%\) on PlanBench and zero\-shot transfer benchmarks\. BW denotes BlocksWorld and MBW denotes Mystery BlocksWorld\. Avg\. is the mean over the four PlanBench domains\.

## 4Experiments

### 4\.1Experimental Setup

Benchmarks\.We evaluate in\-domain planning performance on PlanBench\([Valmeekam et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib25)\), using four PDDL\-derived domains: BlocksWorld, Mystery BlocksWorld, Logistics, and Gripper\. Mystery BlocksWorld obfuscates object and predicate names, making it a diagnostic test of planning beyond lexical memorization\. We further evaluate zero\-shot transfer on ProntoQA\([Saparov and He 2023](https://arxiv.org/html/2608.21897#bib.bib17)\)and the Trip Planning and Calendar Scheduling subsets of NATURAL PLAN\([Zheng et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib33)\)\. Detailed benchmark descriptions are provided in Appendix[G](https://arxiv.org/html/2608.21897#A7)\.

Evaluation\.For PlanBench, generated domain–problem pairs are evaluated with Fast Downward under a 60\-second timeout\. An instance is counted as solved only if the generated PDDL is syntactically valid and the returned plan satisfies the goal conditions\. For ProntoQA and NATURAL PLAN, we follow the original evaluation protocol and report exact\-match success\.

Model\.Our method is built on Qwen2\.5\-7B\([Qwen et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib16)\)initialized from pretrained weights\. We use no human\-annotated PDDL demonstrations and no supervised fine\-tuning on planning data; learning is driven only by solver\-grounded multi\-role reinforcement signals described in Section[3\.4](https://arxiv.org/html/2608.21897#S3.SS4)\.

Baselines\.We compare with three zero\-annotation baselines: Chain\-of\-Thought prompting\([Wei et al\. 2022](https://arxiv.org/html/2608.21897#bib.bib28)\), Tree\-of\-Thought search\([Yao et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib30)\), and LLM\+P\([Liu et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib11)\)\. Unless otherwise stated, these baselines use GPT\-4o as the underlying model\. Appendix[G](https://arxiv.org/html/2608.21897#A7)gives the full baseline configurations\.

### 4\.2Main results

Overall planning performance and zero\-shot transfer\.Table[1](https://arxiv.org/html/2608.21897#S3.T1)summarizes the main in\-domain and out\-of\-domain results\. Our method achieves the best performance on all PlanBench domains, with an average success rate of70\.8%70\.8\\%, compared with35\.5%35\.5\\%for LLM\+P and much lower averages for prompting\-only baselines\. The gains are largest on Mystery BlocksWorld, Logistics, and Gripper, where lexical shortcuts are less useful and small symbolic errors can invalidate the whole plan\. The same model also transfers zero\-shot to ProntoQA and NATURAL PLAN, suggesting that solver\-grounded multi\-role training improves general constraint satisfaction rather than only memorizing PlanBench\-specific formats\. Additional discussion is provided in Appendix[H\.2](https://arxiv.org/html/2608.21897#A8.SS2)\.

Controlled comparison under matched backbone and solver budget\.To rule out the possibility that the gains come from a stronger backbone or more solver access, Table[2](https://arxiv.org/html/2608.21897#S4.T2)compares all methods under a matchedK=6K=6solver\-call budget\. Our method still outperforms the strongest same\-backbone baseline by21\.021\.0points in average PlanBench success, while using fewer solver calls on average\. It also achieves the highest faithful success and the lowest semantic drift, indicating that trained role specialization is more effective than prompting\-based diagnostic revision alone\. The controlled protocol is detailed in Appendix[H\.3](https://arxiv.org/html/2608.21897#A8.SS3)\.

Table 2:Controlled comparison under matched solver\-call budget\. All methods are allowed at mostK=6K=6solver calls per test instance\. Faithful denotes solver\-successful and semantically faithful outputs\. Drift is the conditional fraction of solver\-successful outputs that fail semantic checking\.
### 4\.3Analysis

Semantic faithfulness\.Because a solver\-executable PDDL specification can still deviate from the original task semantics, Table[3](https://arxiv.org/html/2608.21897#S4.T3)reports both solvability and faithful success\. Our method achieves a small solvability–faithfulness gap, with70\.8%70\.8\\%solvability and66\.3%66\.3\\%faithful success, while reducing drift to6\.4%6\.4\\%\. The per\-domain faithful\-success results show that the improvement persists across all PlanBench domains, especially on Mystery BlocksWorld and the structurally brittle Logistics and Gripper domains\. The reference\-checking protocol and metric definitions are given in Appendix[H\.4](https://arxiv.org/html/2608.21897#A8.SS4)\.

Table 3:Semantic faithfulness evaluation on PlanBench\. Left: aggregate faithfulness metrics\. Right: per\-domain faithful success rate\. Solv\. denotes solver success\. Faithful requires both solver executability and semantic consistency with the reference task\.\(a\) Aggregate faithfulness

\(b\) Per\-domain faithful success

![Refer to caption](https://arxiv.org/html/2608.21897v1/repair_diagnostic_analysis.png)Figure 3:Repair and diagnostic analysis\.Left: cumulative success rate over Editor repair steps\. Step00denotes the initial Actor output, and steps11–55denote successive repairs\. Right: diagnostic heatmap over remaining unsolved cases after each step\.Table 4:Ablation and semantic drift analysis\. Left: role ablation on Mystery BlocksWorld\. Right: distribution of semantic drift types among solver\-successful but semantically unfaithful outputs\.\(a\) Role ablation

\(b\) Drift categories

Figure 4:Solvability–faithfulness gap analysis\.Left: paired bars compare solver success and faithful success under matched\-budget settings\. Right: domain\-level slope chart comparing solver success and faithful success\.Figure 5:Judge score distribution over post\-hoc semantic categories\.Left: aggregate violin plot over all PlanBench domains\. Right: domain\-level grouped violin plot\.Table 5:Post\-hoc Judge separation analysis\. Scores are reported as mean±\\pmstandard deviation\. AUC measures how well the Judge separates faithful outputs from solved\-but\-unfaithful outputs among solver\-successful candidates\.Ablation and semantic drift types\.Table[4](https://arxiv.org/html/2608.21897#S4.T4)shows two diagnostic analyses\. The ablation results on Mystery BlocksWorld indicate that Actor, Judge, and Editor are all necessary: removing any role causes a large drop, and using three separate77B models collapses despite larger total parameter count\. The drift\-type analysis shows that our method reduces direct shortcut\-like failures such as goal weakening and object/type drift; the remaining drift cases are dominated by harder action\-schema errors\. Detailed interpretation is deferred to Appendix[H\.6](https://arxiv.org/html/2608.21897#A8.SS6)\.

Repair dynamics and diagnostic evolution\.Figure[3](https://arxiv.org/html/2608.21897#S4.F3)shows how success accumulates across Editor repair steps\. The initial Actor output solves46\.2%46\.2\\%of instances on average, and bounded repair increases success to70\.8%70\.8\\%\. Most improvement occurs early: the first two repairs contribute17\.317\.3out of the total24\.624\.6point gain, indicating that the Editor effectively corrects many localized defects rather than relying on long iterative search\. The diagnostic heatmap further shows that syntax and typing errors are removed first, while remaining failures increasingly concentrate in harder schema and transition\-model errors\. This pattern suggests a natural repair hierarchy: shallow formalization errors are rapidly resolved, whereas residual failures require deeper corrections to action semantics and state transitions\. The full repair protocol is described in Appendix[H\.7](https://arxiv.org/html/2608.21897#A8.SS7)\.

Solvability–faithfulness gap\.Figure[4](https://arxiv.org/html/2608.21897#S4.F4)visualizes the gap between solver success and faithful success\. Our method has a4\.54\.5\-point gap, compared with12\.312\.3points for Qwen\-Self\-Refine\+Solver and11\.311\.3points for GPT\-4o\-Self\-Refine\+Solver\. Thus, stronger prompting or stronger base models can improve solvability, but they do not fully prevent semantic drift\. More details are provided in Appendix[H\.8](https://arxiv.org/html/2608.21897#A8.SS8)\.

Judge separation of faithful and unfaithful outputs\.Finally, Figure[5](https://arxiv.org/html/2608.21897#S4.F5)and Table[5](https://arxiv.org/html/2608.21897#S4.T5)test whether the Judge distinguishes solver\-successful but semantically unfaithful outputs from truly faithful ones\. The Judge assigns low scores to unsolved outputs, intermediate scores to solved\-but\-unfaithful outputs, and high scores to faithful outputs, with an overall AUC of0\.890\.89among solver\-successful candidates\. This suggests that the Judge captures structural quality signals beyond raw solver executability\. Candidate construction and category definitions are given in Appendix[H\.9](https://arxiv.org/html/2608.21897#A8.SS9)\.

## 5Conclusion

We studied annotation\-free natural\-language\-to\-PDDL planning, where a model must learn executable and faithful symbolic specifications from solver feedback alone\. We introduced a solver\-grounded multi\-role reinforcement learning framework that conditions a single language model as an Actor, Judge, and Editor, separating global generation, calibrated verification, and diagnostic\-conditioned repair\. Across PlanBench and zero\-shot transfer benchmarks, the results show that structuring solver feedback into complementary roles improves both planning success and semantic faithfulness, rather than merely increasing solver executability\. These findings suggest that verifiable environments can provide scalable supervision for neuro\-symbolic planning when feedback is converted into generation, verification, and repair signals\.

A natural next step is to extend this paradigm from classical PDDL domains to richer interactive environments where symbolic constraints, tool feedback, and natural\-language goals co\-evolve\.

## References

- Brown et al\. \[2020\]Tom B\. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al\.Language models are few\-shot learners\.*CoRR*, abs/2005\.14165, 2020\.URL[https://arxiv\.org/abs/2005\.14165](https://arxiv.org/abs/2005.14165)\.
- Chen et al\. \[2025\]Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, and Jiaxuan You\.Multi\-agent evolve: Llm self\-improve through co\-evolution, 2025\.URL[https://arxiv\.org/abs/2510\.23595](https://arxiv.org/abs/2510.23595)\.
- Gu et al\. \[2024\]Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo\.A survey on llm\-as\-a\-judge\.*CoRR*, abs/2411\.15594, 2024\.doi:10\.48550/ARXIV\.2411\.15594\.URL[https://doi\.org/10\.48550/arXiv\.2411\.15594](https://doi.org/10.48550/arXiv.2411.15594)\.
- Guo et al\. \[2025\]Daya Guo, Dejian Yang, Haowei Zhang, et al\.Deepseek\-r1 incentivizes reasoning in llms through reinforcement learning\.*Nature*, 645\(8081\):633–638, September 2025\.ISSN 1476\-4687\.doi:10\.1038/s41586\-025\-09422\-z\.URL[http://dx\.doi\.org/10\.1038/s41586\-025\-09422\-z](http://dx.doi.org/10.1038/s41586-025-09422-z)\.
- Hao et al\. \[2023\]Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu\.Reasoning with language model is planning with world model\.In Houda Bouamor, Juan Pino, and Kalika Bali, editors,*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 8154–8173, Singapore, December 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.emnlp\-main\.507\.URL[https://aclanthology\.org/2023\.emnlp\-main\.507/](https://aclanthology.org/2023.emnlp-main.507/)\.
- Huang and Zhang \[2025\]Cassie Huang and Li Zhang\.On the limit of language models as planning formalizers\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 4880–4904, Vienna, Austria, July 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-251\-0\.doi:10\.18653/v1/2025\.acl\-long\.242\.URL[https://aclanthology\.org/2025\.acl\-long\.242/](https://aclanthology.org/2025.acl-long.242/)\.
- Jiang et al\. \[2026\]Owen Jiang, Cassie Huang, Ashish Sabharwal, and Li Zhang\.Language model planners do not scale, but do formalizers?*arXiv preprint arXiv:2603\.23844*, 2026\.
- Kambhampati et al\. \[2024\]Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy\.Position: LLMs can’t plan, but can help planning in LLM\-modulo frameworks\.In*Forty\-first International Conference on Machine Learning*, 2024\.URL[https://openreview\.net/forum?id=Th8JPEmH4z](https://openreview.net/forum?id=Th8JPEmH4z)\.
- Li et al\. \[2025\]Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu\.From generation to judgment: Opportunities and challenges of LLM\-as\-a\-judge\.In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 2757–2791, Suzhou, China, November 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-332\-6\.doi:10\.18653/v1/2025\.emnlp\-main\.138\.URL[https://aclanthology\.org/2025\.emnlp\-main\.138/](https://aclanthology.org/2025.emnlp-main.138/)\.
- Lin et al\. \[2026\]Zhiming Lin, Kai Zhao, Sophie Zhang, Peilai Yu, and Canran Xiao\.Cec\-zero: Zero\-supervision character error correction with self\-generated rewards\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40\(28\), pages 23612–23620, 2026\.
- Liu et al\. \[2023\]Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone\.Llm\+p: Empowering large language models with optimal planning proficiency, 2023\.URL[https://arxiv\.org/abs/2304\.11477](https://arxiv.org/abs/2304.11477)\.
- Madaan et al\. \[2023\]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark\.Self\-refine: Iterative refinement with self\-feedback, 2023\.URL[https://arxiv\.org/abs/2303\.17651](https://arxiv.org/abs/2303.17651)\.
- Mahdavi et al\. \[2024\]Sadegh Mahdavi, Raquel Aoki, Keyi Tang, and Yanshuai Cao\.Leveraging environment interaction for automated PDDL translation and planning with large language models\.In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*, 2024\.URL[https://openreview\.net/forum?id=RzlCqnncQv](https://openreview.net/forum?id=RzlCqnncQv)\.
- OpenAI \[2024\]OpenAI\.Gpt\-4 technical report, 2024\.URL[https://arxiv\.org/abs/2303\.08774](https://arxiv.org/abs/2303.08774)\.
- Oswald et al\. \[2024\]James Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, and Shirin Sohrabi\.Large language models as planning domain generators\.In*Proceedings of the International Conference on Automated Planning and Scheduling*, volume 34, pages 423–431, 2024\.
- Qwen et al\. \[2025\]Qwen, An Yang, Baosong Yang, et al\.Qwen2\.5 technical report, 2025\.URL[https://arxiv\.org/abs/2412\.15115](https://arxiv.org/abs/2412.15115)\.
- Saparov and He \[2023\]Abulhair Saparov and He He\.Language models are greedy reasoners: A systematic formal analysis of chain\-of\-thought\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=qFVVBzXxR2V](https://openreview.net/forum?id=qFVVBzXxR2V)\.
- Shao et al\. \[2024\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024\.URL[https://arxiv\.org/abs/2402\.03300](https://arxiv.org/abs/2402.03300)\.
- Shinn et al\. \[2023\]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao\.Reflexion: language agents with verbal reinforcement learning\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=vAElhFcKW6](https://openreview.net/forum?id=vAElhFcKW6)\.
- Silver et al\. \[2023\]Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B\. Tenenbaum, Leslie Pack Kaelbling, and Michael Katz\.Generalized planning in pddl domains with pretrained large language models, 2023\.URL[https://arxiv\.org/abs/2305\.11014](https://arxiv.org/abs/2305.11014)\.
- Skalse et al\. \[2022\]Joar Max Viktor Skalse, Nikolaus H\. R\. Howe, Dmitrii Krasheninnikov, and David Krueger\.Defining and characterizing reward gaming\.In Alice H\. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,*Advances in Neural Information Processing Systems*, 2022\.URL[https://openreview\.net/forum?id=yb3HOXO3lX2](https://openreview.net/forum?id=yb3HOXO3lX2)\.
- Stechly et al\. \[2024\]Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati\.Chain of thoughtlessness? an analysis of cot in planning\.In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*, 2024\.URL[https://openreview\.net/forum?id=kPBEAZU5Nm](https://openreview.net/forum?id=kPBEAZU5Nm)\.
- Sun et al\. \[2023\]Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang\.Adaplanner: Adaptive planning from feedback with language models\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=rnKgbKmelt](https://openreview.net/forum?id=rnKgbKmelt)\.
- Touvron et al\. \[2023\]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample\.Llama: Open and efficient foundation language models, 2023\.URL[https://arxiv\.org/abs/2302\.13971](https://arxiv.org/abs/2302.13971)\.
- Valmeekam et al\. \[2023\]Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati\.Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change\.In*Thirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023\.URL[https://openreview\.net/forum?id=YXogl4uQUO](https://openreview.net/forum?id=YXogl4uQUO)\.
- Valmeekam et al\. \[2024\]Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati\.Llms still can’t plan; can lrms? a preliminary evaluation of openai’s o1 on planbench, 2024\.URL[https://arxiv\.org/abs/2409\.13373](https://arxiv.org/abs/2409.13373)\.
- Verma et al\. \[2025\]Pulkit Verma, Ngoc La, Anthony Favier, Swaroop Mishra, and Julie A\. Shah\.Teaching llms to plan: Logical chain\-of\-thought instruction tuning for symbolic planning, 2025\.URL[https://arxiv\.org/abs/2509\.13351](https://arxiv.org/abs/2509.13351)\.
- Wei et al\. \[2022\]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H\. Chi, Quoc V Le, and Denny Zhou\.Chain of thought prompting elicits reasoning in large language models\.In Alice H\. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,*Advances in Neural Information Processing Systems*, 2022\.URL[https://openreview\.net/forum?id=\_VjQlMeSB\_J](https://openreview.net/forum?id=_VjQlMeSB_J)\.
- Xie et al\. \[2023\]Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh\.Translating natural language to planning goals with large\-language models, 2023\.URL[https://arxiv\.org/abs/2302\.05128](https://arxiv.org/abs/2302.05128)\.
- Yao et al\. \[2023\]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L\. Griffiths, Yuan Cao, and Karthik R Narasimhan\.Tree of thoughts: Deliberate problem solving with large language models\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=5Xc1ecxO1h](https://openreview.net/forum?id=5Xc1ecxO1h)\.
- Zhang et al\. \[2024\]Tianyi Zhang, Li Zhang, Zhaoyi Hou, Ziyu Wang, Yuling Gu, Peter Clark, Chris Callison\-Burch, and Niket Tandon\.Proc2pddl: Open\-domain planning representations from texts\.In*Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations \(@ ACL 2024\)*, pages 13–24, 2024\.
- Zhao et al\. \[2025\]Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang\.Absolute zero: Reinforced self\-play reasoning with zero data, 2025\.URL[https://arxiv\.org/abs/2505\.03335](https://arxiv.org/abs/2505.03335)\.
- Zheng et al\. \[2024\]Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng\-Tze Cheng, Quoc V\. Le, Ed H\. Chi, and Denny Zhou\.Natural plan: Benchmarking llms on natural language planning, 2024\.URL[https://arxiv\.org/abs/2406\.04520](https://arxiv.org/abs/2406.04520)\.
- Zhou et al\. \[2023\]Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma\.Isr\-llm: Iterative self\-refined large language model for long\-horizon sequential task planning, 2023\.URL[https://arxiv\.org/abs/2308\.13724](https://arxiv.org/abs/2308.13724)\.
- Zuo et al\. \[2025\]Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael Littman, and Stephen Bach\.Planetarium: A rigorous benchmark for translating text to structured planning languages\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 11223–11240, 2025\.

## Appendix AReproducibility Details

##### Scope\.

This appendix consolidates all implementation\- and experiment\-related details required to reproduce the proposed solver\-grounded multi\-role reinforcement learning \(RL\) framework for NL\-to\-PDDL planning\.

### A\.1Task, Data, and Evaluation

##### Task definition\.

Given a natural\-language planning descriptionxx, the goal is to generate a PDDL specificationy=\(𝒟,𝒫\)y=\(\\mathcal\{D\},\\mathcal\{P\}\)consisting of a domain file𝒟\\mathcal\{D\}and a problem file𝒫\\mathcal\{P\}that can be verified by a PDDL solver \(i\.e\., achieves the goal with valid syntax\)\.

##### Metric\.

Primary metric is planning success rate \(%\), measured by the solver’s binary verification result\.

### A\.2Grounded Verifier \(PDDL Solver\)

##### Solver\.

The environment verifierEEis the Fast Downward planner with a 60\-second timeout\. Given a candidate specificationyy, the solver returns\(r,f\)=E⁡\(y\)\(r,f\)=E\(y\), wherer∈\{0,1\}r\\in\\\{0,1\\\}indicates goal achievement, andffcontains structured diagnostics such as: syntax errors, unsatisfied preconditions, unreachable goals, and execution traces\.

### A\.3Model, Roles, and Parameter Sharing

##### Base language model\.

Qwen2\.5\-7Bis used as the pretrained backbone \(initialized from pretrained weights only; no supervised fine\-tuning\)\.

##### One Brain, Three Roles\.

A single model plays three roles via learned role conditioning: Actor \(r=Ar=A\), Judge \(r=Jr=J\), and Editor \(r=Er=E\)\. Role conditioning is implemented with a learnable role embeddingere\_\{r\}:

pθ​\(y∣x,r\)=LMθ​\(y∣\[er;x\]\)\.p\_\{\\theta\}\(y\\mid x,r\)=\\mathrm\{LM\}\_\{\\theta\}\\\!\\left\(y\\mid\[e\_\{r\};x\]\\right\)\.

##### Parameter sharing\.

Parameters are partitioned asθ=θshared∪θA∪θJ∪θE\\theta=\\theta\_\{\\mathrm\{shared\}\}\\cup\\theta\_\{A\}\\cup\\theta\_\{J\}\\cup\\theta\_\{E\}, where\|θshared\|≈0\.95​\|θ\|\\lvert\\theta\_\{\\mathrm\{shared\}\}\\rvert\\approx 0\.95\\lvert\\theta\\rvert\. Role\-specific parameters \(about1\.67%1\.67\\%each\) are lightweight projection heads\.

### A\.4RL Objective, Rewards, and Decoding

##### MDP objective\.

The generation\-refinement process is modeled as an MDP with objective:

J⁡\(θ\)=𝔼τ∼πθ​\[∑t=0Tγt​R​\(st,at\)\],J\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\left\[\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}R\(s\_\{t\},a\_\{t\}\)\\right\],whereγ\\gammais the discount factor = 0\.9\.

##### Actor \(generation\)\.

The Actor generates an initial PDDL specificationy0y\_\{0\}from the natural\-language input\. During training, nucleus sampling with temperatureTsamp=0\.7T\_\{\\mathrm\{samp\}\}=0\.7is used, and switches to greedy decoding at inference\.

##### Actor reward\.

The Actor reward combines solver success and the Judge’s quality score:

RA=rsolver\+λJ⋅sJudge,R\_\{A\}=r\_\{\\text\{solver\}\}\+\\lambda\_\{J\}\\cdot s\_\{\\text\{Judge\}\},wherersolver∈\{0,1\}r\_\{\\text\{solver\}\}\\in\\\{0,1\\\}andsJudge∈\[0,1\]s\_\{\\text\{Judge\}\}\\in\[0,1\]\.λJ=0\.3\\lambda\_\{J\}=0\.3\.

##### Judge \(quality prediction\)\.

The Judge predicts a quality score:

sJudge=σ⁡\(fθ​\(x,y0,r=J\)\)\.s\_\{\\text\{Judge\}\}=\\sigma\(f\_\{\\theta\}\(x,y\_\{0\},r=J\)\)\.Judge reward enforces consistency with the solver outcome using thresholdτ\\tau:

RJ=\{\+1sJudge\>τ∧rsolver=1,−1sJudge\>τ∧rsolver=0,\+0\.5sJudge≤τ∧rsolver=0,−0\.5sJudge≤τ∧rsolver=1\.R\_\{J\}=\\begin\{cases\}\+1&s\_\{\\text\{Judge\}\}\>\\tau\\ \\wedge\\ r\_\{\\text\{solver\}\}=1,\\\\ \-1&s\_\{\\text\{Judge\}\}\>\\tau\\ \\wedge\\ r\_\{\\text\{solver\}\}=0,\\\\ \+0\.5&s\_\{\\text\{Judge\}\}\\leq\\tau\\ \\wedge\\ r\_\{\\text\{solver\}\}=0,\\\\ \-0\.5&s\_\{\\text\{Judge\}\}\\leq\\tau\\ \\wedge\\ r\_\{\\text\{solver\}\}=1\.\\end\{cases\}withτ=0\.5\\tau=0\.5\.

##### Editor \(bounded refinement\)\.

When the solver fails \(r0=0r\_\{0\}=0\), the Editor iteratively refines the specification using solver diagnosticsftf\_\{t\}:

yt\+1∼πE\(⋅∣x,yt,ft\),t=0,1,…,T\.y\_\{t\+1\}\\sim\\pi\_\{E\}\(\\cdot\\mid x,y\_\{t\},f\_\{t\}\),\\quad t=0,1,\\dots,T\.The maximum refinement steps isTmax=5T\_\{\\max\}=5\.

##### Editor reward\.

The Editor trades off final success and number of edits:

RE=rsolver​\(yT\)−β⋅T,R\_\{E\}=r\_\{\\text\{solver\}\}\(y\_\{T\}\)\-\\beta\\cdot T,whereβ=0\.1\\beta=0\.1andT≤TmaxT\\leq T\_\{\\max\}\.

### A\.5Training Protocol and Optimization

##### Training loop\.

Training is conducted in cycles\. Each cycle includes: \(1\) Actor generatesy0y\_\{0\}and queries the solver for\(r0,f0\)\(r\_\{0\},f\_\{0\}\); \(2\) Judge scoressJudges\_\{\\text\{Judge\}\}and receivesRJR\_\{J\}based on solver outcome; \(3\) ifr0=0r\_\{0\}=0, Editor refines up toTmaxT\_\{\\max\}steps and receivesRER\_\{E\}based on final outcome\.

##### Joint parameter update\.

All roles share the backbone and are updated with an aggregate gradient:

∇θJ=∇θJA\+λJ​∇θJJ\+λE​∇θJE,\\nabla\_\{\\theta\}J=\\nabla\_\{\\theta\}J\_\{A\}\+\\lambda\_\{J\}\\nabla\_\{\\theta\}J\_\{J\}\+\\lambda\_\{E\}\\nabla\_\{\\theta\}J\_\{E\},whereλJ=0\.3\\lambda\_\{J\}=0\.3andλE=0\.5\\lambda\_\{E\}=0\.5\.

##### RL algorithm \(PPO\)\.

Optimization uses Proximal Policy Optimization \(PPO\) with: learning rate3×10−53\\times 10^\{\-5\}, batch size3232, and clipping ratioϵ=0\.2\\epsilon=0\.2\.

### A\.6Hyperparameter Summary

Shown in Table[6](https://arxiv.org/html/2608.21897#A1.T6)

Table 6:Hyperparameters and settings\.
### A\.7Compute Resources

All experiments were conducted on a single multi\-GPU compute node with8×8\\timesNVIDIA A100 GPUs\. Each GPU has 80GB memory, and the node has 64 CPU cores and 512GB system memory\. The Qwen2\.5\-7B backbone was trained with mixed precision\. Fast Downward was executed on CPU with a fixed 60\-second timeout for each generated PDDL specification\.

The main solver\-grounded multi\-role RL run uses approximately 12,000 solver interactions across 8 training cycles\. A full training run takes approximately 12–16 hours on the above hardware\.

## Appendix BAdditional Method Details

### B\.1Solver Environment and Diagnostics

The solver environmentℰ\\mathcal\{E\}is implemented with Fast Downward using a 60\-second timeout\. Given a generated PDDL specificationy=\(ydom,yprob\)y=\(y^\{\\mathrm\{dom\}\},y^\{\\mathrm\{prob\}\}\), the solver first checks whether the domain and problem files can be parsed and grounded\. If parsing and grounding succeed, the planner searches for a plan that reaches the stated goal\. The returned labelb∈\{0,1\}b\\in\\\{0,1\\\}is defined asb=1b=1when the solver returns a valid plan within the timeout andb=0b=0otherwise\.

The diagnostic fieldddrecords the most informative failure mode available from the solver pipeline\. We use diagnostics from four categories: syntax and parse errors, type and object mismatches, invalid operator definitions such as missing preconditions or effects, and search\-level failures such as unreachable goals\. The Editor receives the raw diagnostic text together with the current specification\. During training, generated specifications and solver outputs are stored as on\-policy interaction data for Actor, Judge, and Editor updates\.

### B\.2Optimization and Hyperparameters

##### Decoding\.

During training, the Actor and Editor use stochastic decoding to maintain exploration\. We use nucleus sampling with temperatureτtemp=0\.7\\tau\_\{\\mathrm\{temp\}\}=0\.7\. During evaluation, both roles use greedy decoding for deterministic comparison\. The repair horizon is fixed toHmax=5H\_\{\\max\}=5\. If the initial Actor output succeeds, no Editor step is taken\.

##### Actor and Editor PPO objectives\.

For roleρ∈\{A,E\}\\rho\\in\\\{\\mathrm\{A\},\\mathrm\{E\}\\\}, letotρo\_\{t\}^\{\\rho\}denote the context given to the role policy,atρa\_\{t\}^\{\\rho\}the generated token sequence, andA^tρ\\hat\{A\}\_\{t\}^\{\\rho\}the advantage computed from the corresponding role reward\. PPO minimizes

ℒPPOρ=−𝔼t​\[min⁡\(qtρ​\(θ\)​A^tρ,clip⁡\(qtρ​\(θ\),1−ϵppo,1\+ϵppo\)​A^tρ\)\],\\mathcal\{L\}\_\{\\mathrm\{PPO\}\}^\{\\rho\}=\-\\mathbb\{E\}\_\{t\}\\left\[\\min\\left\(q\_\{t\}^\{\\rho\}\(\\theta\)\\hat\{A\}\_\{t\}^\{\\rho\},\\operatorname\{clip\}\\\!\\left\(q\_\{t\}^\{\\rho\}\(\\theta\),1\-\\epsilon\_\{\\mathrm\{ppo\}\},1\+\\epsilon\_\{\\mathrm\{ppo\}\}\\right\)\\hat\{A\}\_\{t\}^\{\\rho\}\\right\)\\right\],\(11\)where

qtρ​\(θ\)=πθ​\(atρ∣otρ,ρ\)πθold​\(atρ∣otρ,ρ\)\.q\_\{t\}^\{\\rho\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}^\{\\rho\}\\mid o\_\{t\}^\{\\rho\},\\rho\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}^\{\\rho\}\\mid o\_\{t\}^\{\\rho\},\\rho\)\}\.The Actor advantage is computed fromRAR\_\{\\mathrm\{A\}\}, and the Editor advantage is computed fromRER\_\{\\mathrm\{E\}\}\. We useϵppo=0\.2\\epsilon\_\{\\mathrm\{ppo\}\}=0\.2\.

##### Judge loss\.

The Judge is trained as a binary predictor of solver success\. The binary cross\-entropy term is

BCE⁡\(s,b\)=−b​log⁡s−\(1−b\)​log⁡\(1−s\)\.\\mathrm\{BCE\}\(s,b\)=\-b\\log s\-\(1\-b\)\\log\(1\-s\)\.The banded penalty is

ℓband​\(s,b\)=\(1−b\)​max⁡\(0,s−τJ\)2\+12​b​max⁡\(0,τJ−s\)2\.\\ell\_\{\\mathrm\{band\}\}\(s,b\)=\(1\-b\)\\max\(0,s\-\\tau\_\{\\mathrm\{J\}\}\)^\{2\}\+\\frac\{1\}\{2\}b\\max\(0,\\tau\_\{\\mathrm\{J\}\}\-s\)^\{2\}\.\(12\)The first term penalizes false positives above the decision threshold, while the second term penalizes under\-confident scores on solver\-successful specifications\. We useτJ=0\.5\\tau\_\{\\mathrm\{J\}\}=0\.5\.

##### Joint update\.

All trainable parameters are updated with

ℒtotal=ℒPPOA\+λE​ℒPPOE\+λJ​ℒJ\.\\mathcal\{L\}\_\{\\mathrm\{total\}\}=\\mathcal\{L\}\_\{\\mathrm\{PPO\}\}^\{\\mathrm\{A\}\}\+\\lambda\_\{\\mathrm\{E\}\}\\mathcal\{L\}\_\{\\mathrm\{PPO\}\}^\{\\mathrm\{E\}\}\+\\lambda\_\{\\mathrm\{J\}\}\\mathcal\{L\}\_\{\\mathrm\{J\}\}\.The shared backbone receives gradients from all three roles\. The Actor head receives Actor gradients, the Judge head receives Judge gradients, and the Editor head receives Editor gradients\.

Table 7:Hyperparameters used in the solver\-grounded multi\-role training procedure\.

## Appendix CJudge Calibration Analysis

This section proves the calibration statement used in Section[3\.5](https://arxiv.org/html/2608.21897#S3.SS5)\. The result is distributional and on\-policy: it concerns the generated specifications encountered under the current policy\.

##### Setup\.

LetX∼𝒫dataX\\sim\\mathcal\{P\}\_\{\\mathrm\{data\}\}be a task and letY∼πθ\(⋅∣X\)Y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid X\)be a generated specification\. The solver\-success label isB=b⁡\(X,Y\)∈\{0,1\}B=b\(X,Y\)\\in\\\{0,1\\\}, and the task\-level correctness label isC=c⁡\(X,Y\)∈\{0,1\}C=c\(X,Y\)\\in\\\{0,1\\\}\. The solver and the reference correctness checker are deterministic for a fixed pair\(x,y\)\(x,y\); randomness comes from task sampling and stochastic generation\.

Define the Bayes\-optimal solver\-success predictor

sJ∗​\(x,y\)=Pr⁡\(B=1∣X=x,Y=y\)\.s\_\{J\}^\{\*\}\(x,y\)=\\Pr\(B=1\\mid X=x,Y=y\)\.The learned Judge has calibration error

δJ=supx,y\|sJ​\(x,y\)−sJ∗​\(x,y\)\|\.\\delta\_\{J\}=\\sup\_\{x,y\}\\left\|s\_\{J\}\(x,y\)\-s\_\{J\}^\{\*\}\(x,y\)\\right\|\.\(13\)
###### Assumption C\.1\(Bounded solver–correctness disagreement\)\.

For the policy\-induced distribution over\(X,Y\)\(X,Y\), there existsε<1/2\\varepsilon<1/2such that

Pr⁡\(B≠C\)≤ε\.\\Pr\(B\\neq C\)\\leq\\varepsilon\.\(14\)

###### Lemma C\.1\(Bayes\-optimality of the Judge target\)\.

The minimizer of the conditional cross\-entropy loss for predictingBBis

sJ∗​\(x,y\)=Pr⁡\(B=1∣X=x,Y=y\)\.s\_\{J\}^\{\*\}\(x,y\)=\\Pr\(B=1\\mid X=x,Y=y\)\.

###### Proof\.

For a fixed pair\(x,y\)\(x,y\), letη=Pr⁡\(B=1∣X=x,Y=y\)\\eta=\\Pr\(B=1\\mid X=x,Y=y\)\. The conditional cross\-entropy is

ℓ⁡\(s,η\)=−η​log⁡s−\(1−η\)​log⁡\(1−s\),s∈\(0,1\)\.\\ell\(s;\\eta\)=\-\\eta\\log s\-\(1\-\\eta\)\\log\(1\-s\),\\qquad s\\in\(0,1\)\.Its derivative is−η/s\+\(1−η\)/\(1−s\)\-\\eta/s\+\(1\-\\eta\)/\(1\-s\), which vanishes only ats=ηs=\\eta\. The second derivative is positive on\(0,1\)\(0,1\), sos=ηs=\\etais the unique minimizer\. ∎

###### Theorem C\.2\(Judge–correctness connection\)\.

Under Assumption[C\.1](https://arxiv.org/html/2608.21897#A3.Thmassumption1), any Judge with calibration errorδJ\\delta\_\{J\}satisfies

𝔼\(X,Y\)​\[\|sJ​\(X,Y\)−C⁡\(X,Y\)\|\]≤ε\+δJ\.\\mathbb\{E\}\_\{\(X,Y\)\}\\left\[\\left\|s\_\{J\}\(X,Y\)\-C\(X,Y\)\\right\|\\right\]\\leq\\varepsilon\+\\delta\_\{J\}\.\(15\)Moreover,

Pr⁡\(\|sJ​\(X,Y\)−C⁡\(X,Y\)\|\>δJ\)≤ε\.\\Pr\\\!\\left\(\\left\|s\_\{J\}\(X,Y\)\-C\(X,Y\)\\right\|\>\\delta\_\{J\}\\right\)\\leq\\varepsilon\.\(16\)

###### Proof\.

By Lemma[C\.1](https://arxiv.org/html/2608.21897#A3.Thmtheorem1),sJ∗​\(X,Y\)=Pr⁡\(B=1∣X,Y\)s\_\{J\}^\{\*\}\(X,Y\)=\\Pr\(B=1\\mid X,Y\)\. SinceBBis deterministic given\(X,Y\)\(X,Y\), we havesJ∗​\(X,Y\)=B⁡\(X,Y\)s\_\{J\}^\{\*\}\(X,Y\)=B\(X,Y\)almost surely\. Therefore,

\|sJ​\(X,Y\)−C⁡\(X,Y\)\|≤\|sJ​\(X,Y\)−sJ∗​\(X,Y\)\|\+\|B⁡\(X,Y\)−C⁡\(X,Y\)\|\.\|s\_\{J\}\(X,Y\)\-C\(X,Y\)\|\\leq\|s\_\{J\}\(X,Y\)\-s\_\{J\}^\{\*\}\(X,Y\)\|\+\|B\(X,Y\)\-C\(X,Y\)\|\.Taking expectations yields Eq\. \([15](https://arxiv.org/html/2608.21897#A3.E15)\)\. For Eq\. \([16](https://arxiv.org/html/2608.21897#A3.E16)\), note that on the eventB=CB=C, the first term is at mostδJ\\delta\_\{J\}\. Thus the event\|sJ−C\|\>δJ\|s\_\{J\}\-C\|\>\\delta\_\{J\}can occur only whenB≠CB\\neq C, whose probability is at mostε\\varepsilon\. ∎

###### Theorem C\.3\(Margin ranking bound\)\.

LetY1,Y2∼i\.i\.d\.πθ\(⋅∣X\)Y\_\{1\},Y\_\{2\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\pi\_\{\\theta\}\(\\cdot\\mid X\)\. Under Assumption[C\.1](https://arxiv.org/html/2608.21897#A3.Thmassumption1),

Pr⁡\(CLOSE\\displaystyle\\Pr\\Big\(sJ​\(X,Y1\)≥sJ​\(X,Y2\)\+2​δJ\\displaystyle s\_\{J\}\(X,Y\_\{1\}\)\\geq s\_\{J\}\(X,Y\_\{2\}\)\+2\\delta\_\{J\}\(17\)OPEN∧C⁡\(X,Y1\)<C⁡\(X,Y2\)\)≤2​ε\.\\displaystyle\\wedge\\ C\(X,Y\_\{1\}\)<C\(X,Y\_\{2\}\)\\Big\)\\leq 2\\varepsilon\.

###### Proof\.

LetGGbe the event thatB⁡\(X,Yi\)=C⁡\(X,Yi\)B\(X,Y\_\{i\}\)=C\(X,Y\_\{i\}\)for bothi=1,2i=1,2\. By a union bound,Pr⁡\(Gc\)≤2​ε\\Pr\(G^\{c\}\)\\leq 2\\varepsilon\. OnGG, the learned score differs from the binary correctness value by at mostδJ\\delta\_\{J\}for each candidate\. Therefore, a score margin of2​δJ2\\delta\_\{J\}cannot rank an incorrect candidate above a correct candidate\. The mis\-ranking event in Eq\. \([17](https://arxiv.org/html/2608.21897#A3.E17)\) is contained inGcG^\{c\}, so its probability is at most2​ε2\\varepsilon\. ∎

### C\.1Estimating the Solver–Correctness Disagreement Rate

For a fixed checkpointθ\\theta, the disagreement rate can be estimated by sampling tasksXiX\_\{i\}, generating specificationsYiY\_\{i\}, recording solver labelsBiB\_\{i\}, and evaluating task\-level correctness labelsCiC\_\{i\}using a reference validator or audit protocol:

ε^θ=1n∑i=1n𝟏\{Bi≠Ci\}\.\\widehat\{\\varepsilon\}\_\{\\theta\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbf\{1\}\\\{B\_\{i\}\\neq C\_\{i\}\\\}\.\(18\)For anyδ∈\(0,1\)\\delta\\in\(0,1\), Hoeffding’s inequality gives

εθ≤ε^θ\+log⁡\(1/δ\)2​n\\varepsilon\_\{\\theta\}\\leq\\widehat\{\\varepsilon\}\_\{\\theta\}\+\\sqrt\{\\frac\{\\log\(1/\\delta\)\}\{2n\}\}\(19\)with probability at least1−δ1\-\\delta\. For a finite set of checkpointsΘckpt\\Theta\_\{\\mathrm\{ckpt\}\}, a uniform bound is obtained by replacingδ\\deltawithδ/\|Θckpt\|\\delta/\|\\Theta\_\{\\mathrm\{ckpt\}\}\|\.

## Appendix DWhy Separate Actor and Editor Roles

This section clarifies why the Actor and Editor are trained as separate roles rather than a single policy optimized only for final solver success\.

##### Initial correctness\.

For a policy that produces an initial specificationY0Y\_\{0\}, define

Cinit​\(π\)=𝔼⁡\[c⁡\(X,Y0\)\]\.C\_\{\\mathrm\{init\}\}\(\\pi\)=\\mathbb\{E\}\\left\[c\(X,Y\_\{0\}\)\\right\]\.Herec⁡\(X,Y0\)=1c\(X,Y\_\{0\}\)=1means that the initial specification is correct with respect to the task\-level reference semantics\.

##### Actor\-only alignment\.

If the Actor is trained with the initial solver labelb⁡\(X,Y0\)b\(X,Y\_\{0\}\), then under the bounded disagreement conditionPr⁡\(b⁡\(X,Y0\)≠c⁡\(X,Y0\)\)≤ε\\Pr\(b\(X,Y\_\{0\}\)\\neq c\(X,Y\_\{0\}\)\)\\leq\\varepsilon,

\|𝔼⁡\[b⁡\(X,Y0\)\]−Cinit​\(π\)\|≤ε\.\\left\|\\mathbb\{E\}\[b\(X,Y\_\{0\}\)\]\-C\_\{\\mathrm\{init\}\}\(\\pi\)\\right\|\\leq\\varepsilon\.Thus the initial solver label remains a distributional surrogate for initial correctness whenever solver success and task\-level correctness agree on most on\-policy samples\.

##### Final\-only Editor objective\.

Consider a monolithic repair policy that both initializes and edits, and receives only the final reward

Rmono=b⁡\(X,YT\)−β​T\.R\_\{\\mathrm\{mono\}\}=b\(X,Y\_\{T\}\)\-\\beta T\.Suppose there exists a set of repairable initial specifications𝒴rep​\(x\)\\mathcal\{Y\}\_\{\\mathrm\{rep\}\}\(x\)such that everyy0∈𝒴rep​\(x\)y\_\{0\}\\in\\mathcal\{Y\}\_\{\\mathrm\{rep\}\}\(x\)can be repaired to solver success in the same number of steps\. Then all initializers supported on𝒴rep​\(x\)\\mathcal\{Y\}\_\{\\mathrm\{rep\}\}\(x\)receive the same final return, even if their initial correctness differs\. The final\-only objective therefore does not identify the quality ofY0Y\_\{0\}\.

###### Proposition D\.1\(Initial specification is unidentifiable under saturated repair\)\.

Assume that for everyxx, ally0∈𝒴rep​\(x\)y\_\{0\}\\in\\mathcal\{Y\}\_\{\\mathrm\{rep\}\}\(x\)can be repaired tob=1b=1in exactlyT†T^\{\\dagger\}steps\. If𝒴rep​\(x\)\\mathcal\{Y\}\_\{\\mathrm\{rep\}\}\(x\)contains both correct and incorrect initial specifications, then for anyα∈\[0,1\]\\alpha\\in\[0,1\]there exists an initializer withCinit=αC\_\{\\mathrm\{init\}\}=\\alphaand final return1−β​T†1\-\\beta T^\{\\dagger\}\.

###### Proof\.

For eachxx, choosey\+​\(x\),y−​\(x\)∈𝒴rep​\(x\)y^\{\+\}\(x\),y^\{\-\}\(x\)\\in\\mathcal\{Y\}\_\{\\mathrm\{rep\}\}\(x\)such thatc⁡\(x,y\+​\(x\)\)=1c\(x,y^\{\+\}\(x\)\)=1andc⁡\(x,y−​\(x\)\)=0c\(x,y^\{\-\}\(x\)\)=0\. Define the initializer to outputy\+​\(x\)y^\{\+\}\(x\)with probabilityα\\alphaandy−​\(x\)y^\{\-\}\(x\)with probability1−α1\-\\alpha\. By the saturated repair assumption, both choices obtain final return1−β​T†1\-\\beta T^\{\\dagger\}\. The expected initial correctness isα\\alpha\. ∎

This proposition motivates the Actor–Editor decomposition\. The Actor receives an initial\-stage reward, which keeps global formalization tied to the initial specification\. The Editor receives a final\-stage repair reward, which specializes it for local diagnostic\-conditioned correction\.

## Appendix EProof of Reward\-Hacking Scaling

This section proves Theorem[3\.1](https://arxiv.org/html/2608.21897#S3.Thmtheorem1)\.

##### Local hacking gain\.

LetJA​\(θ\)J\_\{\\mathrm\{A\}\}\(\\theta\)denote the Actor objective andC⁡\(θ\)C\(\\theta\)denote expected task\-level correctness\. Around a reference pointθ0\\theta\_\{0\}, define

ghack​\(θ0\)=∇θJA​\(θ0\)−a​∇θC​\(θ0\),a\>0\.g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)=\\nabla\_\{\\theta\}J\_\{\\mathrm\{A\}\}\(\\theta\_\{0\}\)\-a\\nabla\_\{\\theta\}C\(\\theta\_\{0\}\),\\qquad a\>0\.For a unit\-norm perturbationΔ​θ\\Delta\\theta, the first\-order Actor gain not explained by correctness is

Δhack​\(θ0,Δ​θ\)=ghack​\(θ0\)⊤​Δ​θ\.\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)=g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)^\{\\top\}\\Delta\\theta\.

##### Admissible perturbation sets\.

For three separate models, the Actor has its own parameter vectorθA∈ℝPA\\theta\_\{\\mathrm\{A\}\}\\in\\mathbb\{R\}^\{P\_\{\\mathrm\{A\}\}\}\. The admissible perturbation set is

ℋsep=\{Δ​θA∈ℝPA:‖Δ​θA‖2≤1\}\.\\mathcal\{H\}\_\{\\mathrm\{sep\}\}=\\left\\\{\\Delta\\theta\_\{\\mathrm\{A\}\}\\in\\mathbb\{R\}^\{P\_\{\\mathrm\{A\}\}\}:\\\|\\Delta\\theta\_\{\\mathrm\{A\}\}\\\|\_\{2\}\\leq 1\\right\\\}\.\(20\)
For the shared\-backbone architecture, write

θ=\(θsh,θA,θJ,θE\),\\theta=\(\\theta\_\{\\mathrm\{sh\}\},\\theta\_\{\\mathrm\{A\}\},\\theta\_\{\\mathrm\{J\}\},\\theta\_\{\\mathrm\{E\}\}\),whereθsh∈ℝp\\theta\_\{\\mathrm\{sh\}\}\\in\\mathbb\{R\}^\{p\}is the shared backbone andθA∈ℝh\\theta\_\{\\mathrm\{A\}\}\\in\\mathbb\{R\}^\{h\}is the Actor head\. We consider perturbations that are neutral to Judge and Editor on the shared backbone and do not modify the Judge or Editor heads:

ℋshr=\{Δθ:∥Δθ∥2≤1,ΔθJ=0,ΔθE=0,Δθsh∈Null\(ΣJE\)\}\.\\mathcal\{H\}\_\{\\mathrm\{shr\}\}=\\left\\\{\\Delta\\theta:\\\|\\Delta\\theta\\\|\_\{2\}\\leq 1,\\,\\Delta\\theta\_\{\\mathrm\{J\}\}=0,\\,\\Delta\\theta\_\{\\mathrm\{E\}\}=0,\\,\\Delta\\theta\_\{\\mathrm\{sh\}\}\\in\\mathrm\{Null\}\(\\Sigma\_\{\\mathrm\{JE\}\}\)\\right\\\}\.\(21\)HereΣJE\\Sigma\_\{\\mathrm\{JE\}\}is the Judge–Editor backbone sensitivity matrix defined below\.

###### Assumption E\.1\(Judge and Editor constrain the shared backbone\)\.

LetgJ∈ℝpg\_\{\\mathrm\{J\}\}\\in\\mathbb\{R\}^\{p\}andgE∈ℝpg\_\{\\mathrm\{E\}\}\\in\\mathbb\{R\}^\{p\}denote stochastic backbone gradients from the Judge and Editor objectives\. Define

ΣJE=𝔼⁡\[gJ​gJ⊤\]\+𝔼⁡\[gE​gE⊤\]\.\\Sigma\_\{\\mathrm\{JE\}\}=\\mathbb\{E\}\[g\_\{\\mathrm\{J\}\}g\_\{\\mathrm\{J\}\}^\{\\top\}\]\+\\mathbb\{E\}\[g\_\{\\mathrm\{E\}\}g\_\{\\mathrm\{E\}\}^\{\\top\}\]\.\(22\)AssumeNull⁡\(ΣJE\)=\{0\}\\mathrm\{Null\}\(\\Sigma\_\{\\mathrm\{JE\}\}\)=\\\{0\\\}\. Therefore, any Judge\- and Editor\-neutral perturbation inℋshr\\mathcal\{H\}\_\{\\mathrm\{shr\}\}must lie in the Actor head subspace\.

###### Assumption E\.2\(Local isotropic hacking\-gradient model\)\.

There existsσ\>0\\sigma\>0such that the restriction ofghack​\(θ0\)g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)to the relevant admissible subspace is isotropic Gaussian\. In the separate setting,

ghack​\(θ0\)∼𝒩⁡\(0,σ2​IPA\)\.g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{P\_\{\\mathrm\{A\}\}\}\)\.In the shared setting,

ProjA​ghack​\(θ0\)∼𝒩⁡\(0,σ2​Ih\),\\mathrm\{Proj\}\_\{\\mathrm\{A\}\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{h\}\),whereProjA\\mathrm\{Proj\}\_\{\\mathrm\{A\}\}denotes projection onto the Actor head subspace\.

###### Lemma E\.1\(Gaussian supremum over a unit ball\)\.

Letg∼𝒩⁡\(0,σ2​Id\)g\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\)\. Then

𝔼⁡\[sup‖u‖2≤1g⊤​u\]=𝔼​‖g‖2=Θ⁡\(σ​d\)\.\\mathbb\{E\}\\left\[\\sup\_\{\\\|u\\\|\_\{2\}\\leq 1\}g^\{\\top\}u\\right\]=\\mathbb\{E\}\\\|g\\\|\_\{2\}=\\Theta\(\\sigma\\sqrt\{d\}\)\.\(23\)

###### Proof\.

For any fixedgg, Cauchy–Schwarz gives

sup‖u‖2≤1g⊤​u=‖g‖2,\\sup\_\{\\\|u\\\|\_\{2\}\\leq 1\}g^\{\\top\}u=\\\|g\\\|\_\{2\},with equality atu=g/‖g‖2u=g/\\\|g\\\|\_\{2\}wheng≠0g\\neq 0\. Sinceg∼𝒩⁡\(0,σ2​Id\)g\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\),‖g‖2/σ\\\|g\\\|\_\{2\}/\\sigmafollows aχd\\chi\_\{d\}distribution, whose expectation isΘ⁡\(d\)\\Theta\(\\sqrt\{d\}\)\. ∎

###### Proof of Theorem[3\.1](https://arxiv.org/html/2608.21897#S3.Thmtheorem1)\.

For the separate\-model setting,ℋsep\\mathcal\{H\}\_\{\\mathrm\{sep\}\}is the unit ball in the Actor parameter space\. Therefore,

supΔ​θ∈ℋsepΔhack​\(θ0,Δ​θ\)=sup‖Δ​θA‖2≤1ghack​\(θ0\)⊤​Δ​θA=‖ghack​\(θ0\)‖2\.\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{sep\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)=\\sup\_\{\\\|\\Delta\\theta\_\{\\mathrm\{A\}\}\\\|\_\{2\}\\leq 1\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)^\{\\top\}\\Delta\\theta\_\{\\mathrm\{A\}\}=\\\|g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\\|\_\{2\}\.By Assumption[E\.2](https://arxiv.org/html/2608.21897#A5.Thmassumption2)and Lemma[E\.1](https://arxiv.org/html/2608.21897#A5.Thmtheorem1)withd=PAd=P\_\{\\mathrm\{A\}\},

𝔼⁡\[supΔ​θ∈ℋsepΔhack​\(θ0,Δ​θ\)\]=Θ⁡\(σ​PA\)\.\\mathbb\{E\}\\left\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{sep\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\right\]=\\Theta\(\\sigma\\sqrt\{P\_\{\\mathrm\{A\}\}\}\)\.
For the shared\-backbone setting, Assumption[E\.1](https://arxiv.org/html/2608.21897#A5.Thmassumption1)givesΔ​θsh=0\\Delta\\theta\_\{\\mathrm\{sh\}\}=0for everyΔ​θ∈ℋshr\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}, and by constructionΔ​θJ=Δ​θE=0\\Delta\\theta\_\{\\mathrm\{J\}\}=\\Delta\\theta\_\{\\mathrm\{E\}\}=0\. Hence every admissible perturbation lies in the Actor head subspace:

Δ​θ=\(0,Δ​θA,0,0\),‖Δ​θA‖2≤1\.\\Delta\\theta=\(0,\\Delta\\theta\_\{\\mathrm\{A\}\},0,0\),\\qquad\\\|\\Delta\\theta\_\{\\mathrm\{A\}\}\\\|\_\{2\}\\leq 1\.Thus,

supΔ​θ∈ℋshrΔhack​\(θ0,Δ​θ\)=sup‖Δ​θA‖2≤1\(ProjA​ghack​\(θ0\)\)⊤​Δ​θA\.\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)=\\sup\_\{\\\|\\Delta\\theta\_\{\\mathrm\{A\}\}\\\|\_\{2\}\\leq 1\}\\left\(\\mathrm\{Proj\}\_\{\\mathrm\{A\}\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\right\)^\{\\top\}\\Delta\\theta\_\{\\mathrm\{A\}\}\.By Assumption[E\.2](https://arxiv.org/html/2608.21897#A5.Thmassumption2)and Lemma[E\.1](https://arxiv.org/html/2608.21897#A5.Thmtheorem1)withd=hd=h,

𝔼⁡\[supΔ​θ∈ℋshrΔhack​\(θ0,Δ​θ\)\]=𝔼⁡\[‖ProjA​ghack​\(θ0\)‖2\]≲σ​h\.\\mathbb\{E\}\\left\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\right\]=\\mathbb\{E\}\\left\[\\left\\\|\\mathrm\{Proj\}\_\{\\mathrm\{A\}\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\right\\\|\_\{2\}\\right\]\\lesssim\\sigma\\sqrt\{h\}\.This proves both claims\. ∎

##### Effective\-rank variant\.

IfNull⁡\(ΣJE\)\\mathrm\{Null\}\(\\Sigma\_\{\\mathrm\{JE\}\}\)has dimensionk\>0k\>0, the same proof gives a shared\-architecture scaling of order

Θ⁡\(σ​h\+k\),\\Theta\(\\sigma\\sqrt\{h\+k\}\),wherekkis the number of Judge\- and Editor\-neutral backbone directions\. Thus the relevant quantity is the dimension of the role\-private or role\-neutral subspace, rather than the total backbone parameter count\.

## Appendix FProofs for One Brain Three Roles

Throughout,XXdenotes a task sampled from𝒟\\mathcal\{D\},YYa PDDL specification produced by the policy, andR,C∈\{0,1\}R,C\\in\\\{0,1\\\}are respectively the*solver success indicator*\(“solvability”\) and the*ground\-truth plan correctness indicator*\(“correctness”\) of the final plan for\(X,Y\)\(X,Y\):R=1R=1iff the PDDL solver parsesYYand returns a plan that reaches its goal under the semantics encoded by\(X,Y\)\(X,Y\), andC=1C=1iff that final plan is correct under the true task semantics\. We writeC⁡\(θ\)=𝔼⁡\[C\]C\(\\theta\)=\\mathbb\{E\}\[C\]for the expected correctness under policyπθ\\pi\_\{\\theta\}\.

### F\.1Judge–Correctness Consistency

##### Deterministic oracles; randomness only from sampling\.

Although the planner and the validator are deterministic maps for any fixed natural\-language taskxxand PDDL specificationyy, the pair\(X,Y\)\(X,Y\)is random because \(i\) tasks are sampled asX∼𝒟X\\sim\\mathcal\{D\}and \(ii\) specifications are sampled asY∼πθ\(⋅∣X\)Y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid X\)\(e\.g\., via stochastic decoding\)\. Hence,RRandCCare random variables through\(X,Y\)\(X,Y\), even though their dependence on\(x,y\)\(x,y\)is deterministic\.

Formally, there exist measurable functionsr,c:𝒳×𝒴→\{0,1\}r,c:\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\\\{0,1\\\}such that

R=r⁡\(X,Y\)∈\{0,1\},C=c⁡\(X,Y\)∈\{0,1\}\.R\\;=\\;r\(X,Y\)\\in\\\{0,1\\\},\\qquad C\\;=\\;c\(X,Y\)\\in\\\{0,1\\\}\.\(24\)We define the conditional success and correctness probabilities

η\(x,y\)≔𝔼\[R∣X=x,Y=y\]=Pr\(R=1∣X=x,Y=y\),\\eta\(x,y\)\\coloneqq\\mathbb\{E\}\[R\\mid X=x,Y=y\]=\\Pr\(R=1\\mid X=x,Y=y\),\(25\)κ\(x,y\)≔𝔼\[C∣X=x,Y=y\]=Pr\(C=1∣X=x,Y=y\)\.\\kappa\(x,y\)\\coloneqq\\mathbb\{E\}\[C\\mid X=x,Y=y\]=\\Pr\(C=1\\mid X=x,Y=y\)\.\(26\)
Under determinism,η⁡\(x,y\)=r⁡\(x,y\)\\eta\(x,y\)=r\(x,y\)andκ⁡\(x,y\)=c⁡\(x,y\)\\kappa\(x,y\)=c\(x,y\)are\{0,1\}\\\{0,1\\\}\-valued\. We keep the probabilistic notation because all bounds below are taken with respect to the*distribution of*\(X,Y\)\(X,Y\)induced by\(𝒟,πθ\)\(\\mathcal\{D\},\\pi\_\{\\theta\}\)\.

##### Bounded solver–correctness drift \(on\-policy\)\.

We allow “specification drift” in which the planner declares success for a generated specification while the resulting behavior is not correct under the ground\-truth validator; we assume such drift is rare on the distribution induced by policies of interest\.

###### Assumption F\.1\(Bounded solver–correctness drift rate\.\)\.

There existsε∈\[0,1/2\)\\varepsilon\\in\[0,1/2\)such that for every policyπθ\\pi\_\{\\theta\}considered in the analysis,

εθ≔PrX∼𝒟,Y∼πθ\(⋅∣X\)\(R≠C\)≤ε\.\\varepsilon\_\{\\theta\}\\;\\coloneqq\\;\\Pr\_\{X\\sim\\mathcal\{D\},\\,Y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid X\)\}\\\!\\bigl\(R\\neq C\\bigr\)\\;\\leq\\;\\varepsilon\.\(27\)

Justification and empirical results in[F\.2](https://arxiv.org/html/2608.21897#A6.SS2)

##### Judge training and calibration\.

The Judge outputssJ:𝒳×𝒴→\(0,1\)s\_\{J\}:\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\(0,1\)and is trained by conditional cross\-entropy for predicting the planner outcomeRR:

ℒ⁡\(s\)=𝔼⁡\[−R​log⁡s⁡\(X,Y\)−\(1−R\)​log⁡\(1−s⁡\(X,Y\)\)\],\\mathcal\{L\}\(s\)\\;=\\;\\mathbb\{E\}\\\!\\left\[\-R\\log s\(X,Y\)\-\(1\-R\)\\log\\bigl\(1\-s\(X,Y\)\\bigr\)\\right\],\(28\)where the expectation is taken over the joint distribution of\(X,Y\)\(X,Y\)used to train the Judge \(typically the on\-policy distribution induced by the Actor\)\. LetsJ∗​\(x,y\)s\_\{J\}^\{\*\}\(x,y\)denote the Bayes pointwise minimizer of \([28](https://arxiv.org/html/2608.21897#A6.E28)\)\. We quantify approximation/calibration by

δJ≔supx,y\|sJ​\(x,y\)−sJ∗​\(x,y\)\|\.\\delta\_\{J\}\\;\\coloneqq\\;\\sup\_\{x,y\}\\bigl\|s\_\{J\}\(x,y\)\-s\_\{J\}^\{\*\}\(x,y\)\\bigr\|\.\(29\)
###### Lemma F\.1\(Bayes\-optimal Judge under cross\-entropy\)\.

For every\(x,y\)\(x,y\), the pointwise minimizer of \([28](https://arxiv.org/html/2608.21897#A6.E28)\) satisfies

sJ∗​\(x,y\)=η⁡\(x,y\)=Pr⁡\(R=1∣X=x,Y=y\)\.s\_\{J\}^\{\*\}\(x,y\)\\;=\\;\\eta\(x,y\)\\;=\\;\\Pr\(R=1\\mid X=x,Y=y\)\.\(30\)

###### Proof\.

Fix\(x,y\)\(x,y\)and writeη=η⁡\(x,y\)\\eta=\\eta\(x,y\)\. The conditional cross\-entropy for predictingRRfrom a scores∈\(0,1\)s\\in\(0,1\)isℓ⁡\(s,η\)=−η​log⁡s−\(1−η\)​log⁡\(1−s\)\\ell\(s;\\eta\)=\-\\eta\\log s\-\(1\-\\eta\)\\log\(1\-s\)\. Differentiating gives∂ℓ∂s\(s;η\)=−η/s\+\(1−η\)/\(1−s\)\\frac\{\\partial\\ell\}\{\\partial s\}\(s;\\eta\)=\-\\eta/s\+\(1\-\\eta\)/\(1\-s\), which vanishes only ats=ηs=\\eta\. Moreover,∂2ℓ∂s2​\(s,η\)=η/s2\+\(1−η\)/\(1−s\)2\>0\\frac\{\\partial^\{2\}\\ell\}\{\\partial s^\{2\}\}\(s;\\eta\)=\\eta/s^\{2\}\+\(1\-\\eta\)/\(1\-s\)^\{2\}\>0for alls∈\(0,1\)s\\in\(0,1\), henceℓ⁡\(⋅,η\)\\ell\(\\cdot;\\eta\)is strictly convex ands=ηs=\\etais the unique minimizer\. ∎

##### Bounding solvability vs\. correctness \(distributionally\)\.

The key deterministic fact is that, sinceR,C∈\{0,1\}R,C\\in\\\{0,1\\\}and both are deterministic functions of\(X,Y\)\(X,Y\), the pointwise gap\|η⁡\(X,Y\)−κ⁡\(X,Y\)\|\|\\eta\(X,Y\)\-\\kappa\(X,Y\)\|is exactly the drift indicator𝕀\{R≠C\}\\mathbb\{I\}\\\{R\\neq C\\\}\.

###### Lemma F\.2\(Distributional solvability–correctness gap\)\.

Under Assumption[F\.1](https://arxiv.org/html/2608.21897#A6.Thmassumption1),

𝔼⁡\[\|η⁡\(X,Y\)−κ⁡\(X,Y\)\|\]=Pr⁡\(R≠C\)≤ε\.\\mathbb\{E\}\\bigl\[\\,\|\\eta\(X,Y\)\-\\kappa\(X,Y\)\|\\,\\bigr\]\\;=\\;\\Pr\(R\\neq C\)\\;\\leq\\;\\varepsilon\.\(31\)Consequently,

\|𝔼⁡\[R\]−𝔼⁡\[C\]\|≤ε\.\\bigl\|\\mathbb\{E\}\[R\]\-\\mathbb\{E\}\[C\]\\bigr\|\\;\\leq\\;\\varepsilon\.\(32\)

###### Proof\.

BecauseR=r⁡\(X,Y\)R=r\(X,Y\)andC=c⁡\(X,Y\)C=c\(X,Y\)are\{0,1\}\\\{0,1\\\}\-valued, almost surely\|R−C\|=𝕀\{R≠C\}\|R\-C\|=\\mathbb\{I\}\\\{R\\neq C\\\}\. Under determinism,η\(X,Y\)=𝔼\[R∣X,Y\]=R\\eta\(X,Y\)=\\mathbb\{E\}\[R\\mid X,Y\]=Randκ\(X,Y\)=𝔼\[C∣X,Y\]=C\\kappa\(X,Y\)=\\mathbb\{E\}\[C\\mid X,Y\]=Calmost surely\. Thus\|η\(X,Y\)−κ\(X,Y\)\|=\|R−C\|=𝕀\{R≠C\}\|\\eta\(X,Y\)\-\\kappa\(X,Y\)\|=\|R\-C\|=\\mathbb\{I\}\\\{R\\neq C\\\}\. Taking expectations yields \([31](https://arxiv.org/html/2608.21897#A6.E31)\), and \([32](https://arxiv.org/html/2608.21897#A6.E32)\) follows from Jensen:\|𝔼⁡\[R\]−𝔼⁡\[C\]\|=\|𝔼⁡\[R−C\]\|≤𝔼⁡\[\|R−C\|\]≤ε\|\\mathbb\{E\}\[R\]\-\\mathbb\{E\}\[C\]\|=\|\\mathbb\{E\}\[R\-C\]\|\\leq\\mathbb\{E\}\[\|R\-C\|\]\\leq\\varepsilon\. ∎

###### Theorem F\.3\(Judge–correctness consistency\)\.

LetsJ∗s\_\{J\}^\{\*\}be the Bayes\-optimal Judge from Lemma[F\.1](https://arxiv.org/html/2608.21897#A6.Thmtheorem1), and letsJs\_\{J\}be any learned Judge with calibration errorδJ\\delta\_\{J\}defined in \([29](https://arxiv.org/html/2608.21897#A6.E29)\)\. Under Assumption[F\.1](https://arxiv.org/html/2608.21897#A6.Thmassumption1),

𝔼⁡\[\|sJ​\(X,Y\)−κ⁡\(X,Y\)\|\]\\displaystyle\\mathbb\{E\}\\bigl\[\\,\|s\_\{J\}\(X,Y\)\-\\kappa\(X,Y\)\|\\,\\bigr\]≤ε\+δJ,\\displaystyle\\;\\leq\\;\\varepsilon\+\\delta\_\{J\},\(33\)Pr⁡\(\|sJ​\(X,Y\)−κ⁡\(X,Y\)\|\>δJ\)\\displaystyle\\Pr\\bigl\(\|s\_\{J\}\(X,Y\)\-\\kappa\(X,Y\)\|\>\\delta\_\{J\}\\bigr\)≤ε\.\\displaystyle\\;\\leq\\;\\varepsilon\.\(34\)Moreover, ifY1,Y2∼i\.i\.d\.πθ\(⋅∣X\)Y\_\{1\},Y\_\{2\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\pi\_\{\\theta\}\(\\cdot\\mid X\), then the Judge’s*margin*controls correctness ranking up to drift:

Pr⁡\(C⁡\(X,Y1\)<C⁡\(X,Y2\)∧sJ​\(X,Y1\)≥sJ​\(X,Y2\)\+2​δJ\)≤2​ε\.\\Pr\\\!\\Bigl\(C\(X,Y\_\{1\}\)<C\(X,Y\_\{2\}\)\\ \\wedge\\ s\_\{J\}\(X,Y\_\{1\}\)\\geq s\_\{J\}\(X,Y\_\{2\}\)\+2\\delta\_\{J\}\\Bigr\)\\;\\leq\\;2\\varepsilon\.\(35\)

###### Proof\.

By Lemma[F\.1](https://arxiv.org/html/2608.21897#A6.Thmtheorem1),sJ∗​\(x,y\)=η⁡\(x,y\)s\_\{J\}^\{\*\}\(x,y\)=\\eta\(x,y\)\.

*\(i\) Expected consistency\.*By triangle inequality,\|sJ−κ\|≤\|sJ−sJ∗\|\+\|sJ∗−κ\|=\|sJ−sJ∗\|\+\|η−κ\|\|s\_\{J\}\-\\kappa\|\\leq\|s\_\{J\}\-s\_\{J\}^\{\*\}\|\+\|s\_\{J\}^\{\*\}\-\\kappa\|=\|s\_\{J\}\-s\_\{J\}^\{\*\}\|\+\|\\eta\-\\kappa\|\. Taking expectations and using𝔼⁡\[\|sJ−sJ∗\|\]≤δJ\\mathbb\{E\}\[\|s\_\{J\}\-s\_\{J\}^\{\*\}\|\]\\leq\\delta\_\{J\}and Lemma[F\.2](https://arxiv.org/html/2608.21897#A6.Thmtheorem2)gives \([33](https://arxiv.org/html/2608.21897#A6.E33)\)\.

*\(ii\) High\-probability consistency\.*On the event\{R=C\}\\\{R=C\\\}we haveκ=η=sJ∗\\kappa=\\eta=s\_\{J\}^\{\*\}, hence\|sJ−κ\|=\|sJ−sJ∗\|≤δJ\|s\_\{J\}\-\\kappa\|=\|s\_\{J\}\-s\_\{J\}^\{\*\}\|\\leq\\delta\_\{J\}\.Therefore\{\|sJ−κ\|\>δJ\}⊆\{R≠C\}\\\{\|s\_\{J\}\-\\kappa\|\>\\delta\_\{J\}\\\}\\subseteq\\\{R\\neq C\\\}andPr⁡\(\|sJ−κ\|\>δJ\)≤Pr⁡\(R≠C\)≤ε\\Pr\(\|s\_\{J\}\-\\kappa\|\>\\delta\_\{J\}\)\\leq\\Pr\(R\\neq C\)\\leq\\varepsilon, proving \([34](https://arxiv.org/html/2608.21897#A6.E34)\)\.

*\(iii\) Ranking with a margin\.*LetG≔\{R\(X,Y1\)=C\(X,Y1\)\}∩\{R\(X,Y2\)=C\(X,Y2\)\}G\\coloneqq\\\{R\(X,Y\_\{1\}\)=C\(X,Y\_\{1\}\)\\\}\\cap\\\{R\(X,Y\_\{2\}\)=C\(X,Y\_\{2\}\)\\\}\. By a union bound and Assumption[F\.1](https://arxiv.org/html/2608.21897#A6.Thmassumption1),Pr⁡\(Gc\)≤Pr⁡\(R≠C​for​\(X,Y1\)\)\+Pr⁡\(R≠C​for​\(X,Y2\)\)≤2​ε\\Pr\(G^\{c\}\)\\leq\\Pr\(R\\neq C\\text\{ for \}\(X,Y\_\{1\}\)\)\+\\Pr\(R\\neq C\\text\{ for \}\(X,Y\_\{2\}\)\)\\leq 2\\varepsilon\. OnGG, we haveC⁡\(X,Yi\)=R⁡\(X,Yi\)=η⁡\(X,Yi\)C\(X,Y\_\{i\}\)=R\(X,Y\_\{i\}\)=\\eta\(X,Y\_\{i\}\)fori=1,2i=1,2, and\|sJ​\(X,Yi\)−η⁡\(X,Yi\)\|≤δJ\|s\_\{J\}\(X,Y\_\{i\}\)\-\\eta\(X,Y\_\{i\}\)\|\\leq\\delta\_\{J\}\. Hence onGG,sJ​\(X,Y1\)≥sJ​\(X,Y2\)\+2​δJ⇒η⁡\(X,Y1\)≥η⁡\(X,Y2\)⇒C⁡\(X,Y1\)≥C⁡\(X,Y2\)s\_\{J\}\(X,Y\_\{1\}\)\\geq s\_\{J\}\(X,Y\_\{2\}\)\+2\\delta\_\{J\}\\Rightarrow\\eta\(X,Y\_\{1\}\)\\geq\\eta\(X,Y\_\{2\}\)\\Rightarrow C\(X,Y\_\{1\}\)\\geq C\(X,Y\_\{2\}\)\. Thus the event in \([35](https://arxiv.org/html/2608.21897#A6.E35)\) can only occur onGcG^\{c\}, so its probability is at most2​ε2\\varepsilon\. ∎

### F\.2Justifying the Drift\-Rate Parameterε\\varepsilon

This subsection justifies Assumption[F\.1](https://arxiv.org/html/2608.21897#A6.Thmassumption1)\(bounded solver–correctness drift rate\) and outlines how to setε\\varepsilonin a deterministic planning environment\.

##### Deterministic oracles; on\-policy drift rate\.

The planner and the \(ground\-truth\) validator are deterministic given a taskxxand a specificationyy:

R=r⁡\(x,y\)∈\{0,1\},C=c⁡\(x,y\)∈\{0,1\}\.R=r\(x,y\)\\in\\\{0,1\\\},\\qquad C=c\(x,y\)\\in\\\{0,1\\\}\.Randomness enters only through samplingX∼𝒟X\\sim\\mathcal\{D\}and stochastic generationY∼πθ\(⋅∣X\)Y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid X\)\. Define the drift indicator

D≔𝕀\{R≠C\}∈\{0,1\}\.D\\;\\coloneqq\\;\\mathbb\{I\}\\\{R\\neq C\\\}\\in\\\{0,1\\\}\.\(36\)For a fixed policyπθ\\pi\_\{\\theta\}, we define the induced*on\-policy drift rate*

εθ≔PrX∼𝒟,Y∼πθ\(⋅∣X\)\(R≠C\)=𝔼\[D\]=𝔼\[\|R−C\|\]\.\\varepsilon\_\{\\theta\}\\;\\coloneqq\\;\\Pr\_\{X\\sim\\mathcal\{D\},\\;Y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid X\)\}\(R\\neq C\)\\;=\\;\\mathbb\{E\}\[D\]\\;=\\;\\mathbb\{E\}\\bigl\[\|R\-C\|\\bigr\]\.\(37\)Assumption[F\.1](https://arxiv.org/html/2608.21897#A6.Thmassumption1)postulates that for the policy class of interest,supθεθ≤ε\\sup\_\{\\theta\}\\varepsilon\_\{\\theta\}\\leq\\varepsilonfor some finiteε<12\\varepsilon<\\tfrac\{1\}\{2\}\. Note that this assumption is*distributional*: it bounds the*frequency*of drift under the policy\-induced distribution, rather than imposing a pointwise \(conditional\) noise model\.

##### Empirical evidence from solvability vs\. correctness

[Huang and Zhang 2025](https://arxiv.org/html/2608.21897#bib.bib6)\([Huang and Zhang 2025](https://arxiv.org/html/2608.21897#bib.bib6)\) report two evaluation metrics closely aligned with our notation:*Solvability*\(whether a planner finds a plan\) and*Correctness*\(whether the plan is accepted under a higher\-fidelity reference semantics/validator\)\. In their experiments across domains and models \(e\.g\. GPT\-40, Llama\-8B\), the observed gap between the two metrics is typically small \(typically well below0\.10\.1<<1/21/2\), suggesting that solver–correctness drift is limited for realistic LLM policies\.

Formally, for any fixedπθ\\pi\_\{\\theta\}define the induced on\-policy rates

S⁡\(θ\)≔Pr⁡\(R=1\),K⁡\(θ\)≔Pr⁡\(C=1\),S\(\\theta\)\\coloneqq\\Pr\(R=1\),\\qquad K\(\\theta\)\\coloneqq\\Pr\(C=1\),where probabilities are over\(X,Y\)∼\(𝒟,πθ\)\(X,Y\)\\sim\(\\mathcal\{D\},\\pi\_\{\\theta\}\)\. Then the observable solvability–correctness gap is always controlled by the drift rate:

\|S⁡\(θ\)−K⁡\(θ\)\|=\|𝔼⁡\[R\]−𝔼⁡\[C\]\|≤𝔼⁡\[\|R−C\|\]=εθ\.\|S\(\\theta\)\-K\(\\theta\)\|\\;=\\;\\bigl\|\\mathbb\{E\}\[R\]\-\\mathbb\{E\}\[C\]\\bigr\|\\;\\leq\\;\\mathbb\{E\}\\bigl\[\|R\-C\|\\bigr\]\\;=\\;\\varepsilon\_\{\\theta\}\.\(38\)Moreover, under the common evaluation convention that correctness is defined only for solver\-produced plans \(soC≤RC\\leq Ralmost surely\), the gap equals the \(false\-positive\) drift probability:

εθ=Pr⁡\(R=1,C=0\)=Pr⁡\(R=1\)−Pr⁡\(C=1\)=S⁡\(θ\)−K⁡\(θ\)\.\\varepsilon\_\{\\theta\}\\;=\\;\\Pr\(R=1,C=0\)\\;=\\;\\Pr\(R=1\)\-\\Pr\(C=1\)\\;=\\;S\(\\theta\)\-K\(\\theta\)\.\(39\)Thus, the small empirical solvability–correctness gaps reported by[Huang and Zhang 2025](https://arxiv.org/html/2608.21897#bib.bib6)provide direct evidence thatεθ\\varepsilon\_\{\\theta\}is small in practice for realistic policies, and in particular that taking a uniform constantε<12\\varepsilon<\\tfrac\{1\}\{2\}is reasonable\.

##### Estimatingεθ\\varepsilon\_\{\\theta\}and choosing a uniformε\\varepsilon\.

For a fixed policyπθ\\pi\_\{\\theta\}, we can estimateεθ\\varepsilon\_\{\\theta\}by sampling\{Xi\}i=1n∼𝒟\\\{X\_\{i\}\\\}\_\{i=1\}^\{n\}\\sim\\mathcal\{D\}, generatingYi∼πθ\(⋅∣Xi\)Y\_\{i\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid X\_\{i\}\), and computingRi=r⁡\(Xi,Yi\)R\_\{i\}=r\(X\_\{i\},Y\_\{i\}\)andCi=c⁡\(Xi,Yi\)C\_\{i\}=c\(X\_\{i\},Y\_\{i\}\), yielding the empirical drift rate

ε^θ≔1n∑i=1n𝕀\{Ri≠Ci\}\.\\widehat\{\\varepsilon\}\_\{\\theta\}\\;\\coloneqq\\;\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{I\}\\\{R\_\{i\}\\neq C\_\{i\}\\\}\.\(40\)By Hoeffding’s inequality, for anyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta,

εθ≤ε^θ\+log⁡\(1/δ\)2​n\.\\varepsilon\_\{\\theta\}\\;\\leq\\;\\widehat\{\\varepsilon\}\_\{\\theta\}\+\\sqrt\{\\frac\{\\log\(1/\\delta\)\}\{2n\}\}\.\(41\)To obtain a uniform bound over a finite set of checkpointsΘckpt\\Theta\_\{\\mathrm\{ckpt\}\}, we may set

ε≔supθ∈Θckptε^θ\+log⁡\(\|Θckpt\|/δ\)2​n,\\varepsilon\\;\\coloneqq\\;\\sup\_\{\\theta\\in\\Theta\_\{\\mathrm\{ckpt\}\}\}\\widehat\{\\varepsilon\}\_\{\\theta\}\\;\+\\;\\sqrt\{\\frac\{\\log\(\|\\Theta\_\{\\mathrm\{ckpt\}\}\|/\\delta\)\}\{2n\}\},\(42\)so that \(by a union bound\)supθ∈Θckptεθ≤ε\\sup\_\{\\theta\\in\\Theta\_\{\\mathrm\{ckpt\}\}\}\\varepsilon\_\{\\theta\}\\leq\\varepsilonholds with probability at least1−δ1\-\\delta\. Our theory only requires thatε\\varepsilonis finite and satisfiesε<12\\varepsilon<\\tfrac\{1\}\{2\}, i\.e\., drift is not the majority behavior on\-policy\.

### F\.3Why Actor–Editor: Actor\-only vs\. Editor\-only vs\. Actor–Editor

We compare three training designs for PDDL formalization:*Actor\-only*,*Editor\-only*\(a single monolithic Editor\-like policy\), and*Actor–Editor*\. All results are stated for deterministic planner/validator semantics; all probabilities are over\(X,Y\)\(X,Y\)induced by task samplingX∼𝒟X\\sim\\mathcal\{D\}and stochastic generationY∼π\(⋅∣X\)Y\\sim\\pi\(\\cdot\\mid X\)\.

##### Setup\.

LetR=r⁡\(X,Y\)∈\{0,1\}R=r\(X,Y\)\\in\\\{0,1\\\}be solver success andC=c⁡\(X,Y\)∈\{0,1\}C=c\(X,Y\)\\in\\\{0,1\\\}be true correctness\. For any policy that produces an*initial*specificationY0Y\_\{0\}, define

Cinit​\(π\)≔𝔼⁡\[C⁡\(X,Y0\)\]∈\[0,1\]\.C\_\{\\mathrm\{init\}\}\(\\pi\)\\;\\coloneqq\\;\\mathbb\{E\}\\bigl\[C\(X,Y\_\{0\}\)\\bigr\]\\in\[0,1\]\.\(43\)
###### Assumption F\.2\(Bounded solver–correctness drift rate\)\.

There existsε∈\[0,1/2\)\\varepsilon\\in\[0,1/2\)such that for every policyπ\\piconsidered in this comparison,

PrX∼𝒟,Y∼π\(⋅∣X\)\(r\(X,Y\)≠c\(X,Y\)\)≤ε\.\\Pr\_\{X\\sim\\mathcal\{D\},\\;Y\\sim\\pi\(\\cdot\\mid X\)\}\\\!\\bigl\(r\(X,Y\)\\neq c\(X,Y\)\\bigr\)\\;\\leq\\;\\varepsilon\.\(44\)

##### A generic optimization\-to\-alignment lemma\.

###### Lemma F\.4\(Generic gap bound\)\.

LetC⁡\(θ\)∈\[0,1\]C\(\\theta\)\\in\[0,1\]andJ⁡\(θ\)∈ℝJ\(\\theta\)\\in\\mathbb\{R\}satisfy

\|J⁡\(θ\)−\(a​C​\(θ\)\+b\)\|≤Δ∀θ,\\bigl\|J\(\\theta\)\-\(a\\,C\(\\theta\)\+b\)\\bigr\|\\leq\\Delta\\qquad\\forall\\theta,\(45\)for somea\>0a\>0,b∈ℝb\\in\\mathbb\{R\},Δ≥0\\Delta\\geq 0\. IfθJ∈arg⁡maxθ⁡J⁡\(θ\)\\theta\_\{J\}\\in\\arg\\max\_\{\\theta\}J\(\\theta\)andθC∈arg⁡maxθ⁡C⁡\(θ\)\\theta\_\{C\}\\in\\arg\\max\_\{\\theta\}C\(\\theta\), then

C⁡\(θC\)−C⁡\(θJ\)≤2​Δa\.C\(\\theta\_\{C\}\)\-C\(\\theta\_\{J\}\)\\;\\leq\\;\\frac\{2\\Delta\}\{a\}\.\(46\)

###### Proof\.

From \([45](https://arxiv.org/html/2608.21897#A6.E45)\), for anyθ\\theta,J⁡\(θ\)≥a​C​\(θ\)\+b−ΔJ\(\\theta\)\\geq aC\(\\theta\)\+b\-\\DeltaandJ⁡\(θ\)≤a​C​\(θ\)\+b\+ΔJ\(\\theta\)\\leq aC\(\\theta\)\+b\+\\Delta\. Using optimality ofθJ\\theta\_\{J\}andθC\\theta\_\{C\}yieldsa​C​\(θJ\)\+b\+Δ≥J⁡\(θJ\)≥J⁡\(θC\)≥a​C​\(θC\)\+b−ΔaC\(\\theta\_\{J\}\)\+b\+\\Delta\\geq J\(\\theta\_\{J\}\)\\geq J\(\\theta\_\{C\}\)\\geq aC\(\\theta\_\{C\}\)\+b\-\\Delta, hencea⁡\(C⁡\(θC\)−C⁡\(θJ\)\)≤2​Δa\(C\(\\theta\_\{C\}\)\-C\(\\theta\_\{J\}\)\)\\leq 2\\Delta\. ∎

#### Actor\-only: solver reward aligns initial correctness on\-policy

##### Definition\.

Actor\-only samplesY0∼πA\(⋅∣X\)Y\_\{0\}\\sim\\pi\_\{A\}\(\\cdot\\mid X\)and maximizes

JA−only​\(πA\)≔𝔼⁡\[r⁡\(X,Y0\)\]\.J\_\{\\mathrm\{A\-only\}\}\(\\pi\_\{A\}\)\\;\\coloneqq\\;\\mathbb\{E\}\\bigl\[r\(X,Y\_\{0\}\)\\bigr\]\.\(47\)
###### Theorem F\.5\(Actor\-only initial\-alignment under drift rate\)\.

Under Assumption[F\.2](https://arxiv.org/html/2608.21897#A6.Thmassumption2), for allπA\\pi\_\{A\},

\|JA−only​\(πA\)−Cinit​\(πA\)\|≤ε\.\\bigl\|J\_\{\\mathrm\{A\-only\}\}\(\\pi\_\{A\}\)\-C\_\{\\mathrm\{init\}\}\(\\pi\_\{A\}\)\\bigr\|\\;\\leq\\;\\varepsilon\.\(48\)Consequently any maximizerπA⋆∈arg⁡maxπA​JA−only​\(πA\)\\pi\_\{A\}^\{\\star\}\\in\\arg\\max\_\{\\pi\_\{A\}\}J\_\{\\mathrm\{A\-only\}\}\(\\pi\_\{A\}\)satisfies

Cinit​\(πA⋆\)≥supπACinit​\(πA\)−2​ε\.C\_\{\\mathrm\{init\}\}\(\\pi\_\{A\}^\{\\star\}\)\\;\\geq\\;\\sup\_\{\\pi\_\{A\}\}C\_\{\\mathrm\{init\}\}\(\\pi\_\{A\}\)\-2\\varepsilon\.\(49\)

###### Proof\.

Sincer,c∈\{0,1\}r,c\\in\\\{0,1\\\},

\|JA−only​\(πA\)−Cinit​\(πA\)\|=\|𝔼⁡\[r⁡\(X,Y0\)−c⁡\(X,Y0\)\]\|≤𝔼⁡\[\|r−c\|\]=Pr⁡\(r≠c\)≤ε,\\bigl\|J\_\{\\mathrm\{A\-only\}\}\(\\pi\_\{A\}\)\-C\_\{\\mathrm\{init\}\}\(\\pi\_\{A\}\)\\bigr\|=\\bigl\|\\mathbb\{E\}\[r\(X,Y\_\{0\}\)\-c\(X,Y\_\{0\}\)\]\\bigr\|\\leq\\mathbb\{E\}\\bigl\[\|r\-c\|\\bigr\]=\\Pr\(r\\neq c\)\\leq\\varepsilon,proving \([48](https://arxiv.org/html/2608.21897#A6.E48)\)\. Apply Lemma[F\.4](https://arxiv.org/html/2608.21897#A6.Thmtheorem4)witha=1a=1,b=0b=0,Δ=ε\\Delta=\\varepsilonto obtain \([49](https://arxiv.org/html/2608.21897#A6.E49)\)\. ∎

#### Editor\-only: final\-outcome objective cannot constrain initial correctness

##### Monolithic Editor\-only baseline\.

A monolithic policy initializesY0∼q\(⋅∣X\)Y\_\{0\}\\sim q\(\\cdot\\mid X\)and iteratively repairs:

Yt\+1∼πEmono\(⋅∣X,Yt,Ft,t\),t=0,…,T−1\.Y\_\{t\+1\}\\sim\\pi\_\{E\}^\{\\mathrm\{mono\}\}\(\\cdot\\mid X,Y\_\{t\},F\_\{t\},t\),\\qquad t=0,\\dots,T\-1\.The objective depends only on the final solver outcome:

Jmono​\(q,πEmono\)≔𝔼⁡\[r⁡\(X,YT\)−β​T\]\.J\_\{\\mathrm\{mono\}\}\(q,\\pi\_\{E\}^\{\\mathrm\{mono\}\}\)\\;\\coloneqq\\;\\mathbb\{E\}\\bigl\[r\(X,Y\_\{T\}\)\-\\beta T\\bigr\]\.\(50\)
###### Assumption F\.3\(Editor\-saturated repair\)\.

There exist sets𝒴good​\(x\)⊆𝒴\\mathcal\{Y\}\_\{\\mathrm\{good\}\}\(x\)\\subseteq\\mathcal\{Y\}and constantsT†∈ℕT^\{\\dagger\}\\in\\mathbb\{N\},β≥0\\beta\\geq 0such that:

1. 1\.\(*Non\-catastrophic initializations*\)Pr⁡\(Y0∈𝒴good​\(X\)\)=1\\Pr\(Y\_\{0\}\\in\\mathcal\{Y\}\_\{\\mathrm\{good\}\}\(X\)\)=1\.
2. 2\.\(*Uniform repair success*\) There exists a repair policyπE†\\pi\_\{E\}^\{\\dagger\}such that for allxxand ally0∈𝒴good​\(x\)y\_\{0\}\\in\\mathcal\{Y\}\_\{\\mathrm\{good\}\}\(x\), PrπE†⁡\(r⁡\(x,YT†\)=1∣X=x,Y0=y0\)=1,andT=T†​a\.s\.\\Pr\_\{\\pi\_\{E\}^\{\\dagger\}\}\\bigl\(r\(x,Y\_\{T^\{\\dagger\}\}\)=1\\mid X=x,Y\_\{0\}=y\_\{0\}\\bigr\)=1,\\quad\\text\{and\}\\quad T=T^\{\\dagger\}\\ \\text\{a\.s\.\}

###### Assumption F\.4\(Nontrivial good\-set\)\.

For everyxxthere existy\+​\(x\),y−​\(x\)∈𝒴good​\(x\)y^\{\+\}\(x\),y^\{\-\}\(x\)\\in\\mathcal\{Y\}\_\{\\mathrm\{good\}\}\(x\)such thatc⁡\(x,y\+​\(x\)\)=1c\(x,y^\{\+\}\(x\)\)=1andc⁡\(x,y−​\(x\)\)=0c\(x,y^\{\-\}\(x\)\)=0\.

###### Theorem F\.6\(Editor\-only unidentifiability of initial correctness\)\.

Under Assumptions[F\.3](https://arxiv.org/html/2608.21897#A6.Thmassumption3)and[F\.4](https://arxiv.org/html/2608.21897#A6.Thmassumption4), for everyα∈\[0,1\]\\alpha\\in\[0,1\]there exists a monolithic policy\(qα,πE†\)\(q^\{\\alpha\},\\pi\_\{E\}^\{\\dagger\}\)such that

Jmono​\(qα,πE†\)=1−β​T†andCinit​\(qα\)=α\.J\_\{\\mathrm\{mono\}\}\(q^\{\\alpha\},\\pi\_\{E\}^\{\\dagger\}\)=1\-\\beta T^\{\\dagger\}\\quad\\text\{and\}\\quad C\_\{\\mathrm\{init\}\}\(q^\{\\alpha\}\)=\\alpha\.\(51\)In particular,JmonoJ\_\{\\mathrm\{mono\}\}admits globally optimal policies with arbitrarily lowCinitC\_\{\\mathrm\{init\}\}\.

###### Proof\.

Fixα∈\[0,1\]\\alpha\\in\[0,1\]and defineqα\(⋅∣x\)q^\{\\alpha\}\(\\cdot\\mid x\)byY0=y\+​\(x\)Y\_\{0\}=y^\{\+\}\(x\)with probabilityα\\alphaandY0=y−​\(x\)Y\_\{0\}=y^\{\-\}\(x\)with probability1−α1\-\\alpha\. By Assumption[F\.3](https://arxiv.org/html/2608.21897#A6.Thmassumption3), for anyy0∈𝒴good​\(x\)y\_\{0\}\\in\\mathcal\{Y\}\_\{\\mathrm\{good\}\}\(x\), repair byπE†\\pi\_\{E\}^\{\\dagger\}yieldsr⁡\(x,YT†\)=1r\(x,Y\_\{T^\{\\dagger\}\}\)=1a\.s\. andT=T†T=T^\{\\dagger\}a\.s\., hence𝔼\[r\(X,YT\)−βT∣X=x,Y0=y0\]=1−βT†\\mathbb\{E\}\[r\(X,Y\_\{T\}\)\-\\beta T\\mid X=x,Y\_\{0\}=y\_\{0\}\]=1\-\\beta T^\{\\dagger\}independent ofy0y\_\{0\}\. Taking expectation overXXandY0∼qα\(⋅∣X\)Y\_\{0\}\\sim q^\{\\alpha\}\(\\cdot\\mid X\)givesJmono​\(qα,πE†\)=1−β​T†J\_\{\\mathrm\{mono\}\}\(q^\{\\alpha\},\\pi\_\{E\}^\{\\dagger\}\)=1\-\\beta T^\{\\dagger\}\.

Moreover, by Assumption[F\.4](https://arxiv.org/html/2608.21897#A6.Thmassumption4),c⁡\(X,y\+​\(X\)\)=1c\(X,y^\{\+\}\(X\)\)=1andc⁡\(X,y−​\(X\)\)=0c\(X,y^\{\-\}\(X\)\)=0, so

Cinit​\(qα\)=𝔼⁡\[c⁡\(X,Y0\)\]=𝔼⁡\[α⋅1\+\(1−α\)⋅0\]=α\.C\_\{\\mathrm\{init\}\}\(q^\{\\alpha\}\)=\\mathbb\{E\}\[c\(X,Y\_\{0\}\)\]=\\mathbb\{E\}\[\\alpha\\cdot 1\+\(1\-\\alpha\)\\cdot 0\]=\\alpha\.∎

#### Actor–Editor: restores initial alignment and improves final solvability

##### Actor reward with a calibrated Judge\.

Actor–Editor samplesY0∼πA\(⋅∣X\)Y\_\{0\}\\sim\\pi\_\{A\}\(\\cdot\\mid X\)and uses an Actor reward

RA≔r⁡\(X,Y0\)\+λJ​sJ​\(X,Y0\),λJ≥0,R\_\{A\}\\;\\coloneqq\\;r\(X,Y\_\{0\}\)\+\\lambda\_\{J\}s\_\{J\}\(X,Y\_\{0\}\),\\qquad\\lambda\_\{J\}\\geq 0,\(52\)with objectiveJA​\(πA\)≔𝔼⁡\[RA\]J\_\{A\}\(\\pi\_\{A\}\)\\coloneqq\\mathbb\{E\}\[R\_\{A\}\]\. We assume the Judge is calibrated to the solver labelr⁡\(X,Y\)r\(X,Y\)\.

###### Assumption F\.5\(Judge calibration to the solver\)\.

There existsδJ≥0\\delta\_\{J\}\\geq 0such that

supx,y\|sJ​\(x,y\)−r⁡\(x,y\)\|≤δJ\.\\sup\_\{x,y\}\\bigl\|s\_\{J\}\(x,y\)\-r\(x,y\)\\bigr\|\\leq\\delta\_\{J\}\.\(53\)

###### Lemma F\.7\(Judge is correctness\-consistent on\-policy\)\.

Under Assumptions[F\.2](https://arxiv.org/html/2608.21897#A6.Thmassumption2)and[F\.5](https://arxiv.org/html/2608.21897#A6.Thmassumption5), for any policyπ\\pigenerating\(X,Y\)\(X,Y\),

𝔼⁡\[\|sJ​\(X,Y\)−c⁡\(X,Y\)\|\]≤δJ\+ε\.\\mathbb\{E\}\\bigl\[\\,\|s\_\{J\}\(X,Y\)\-c\(X,Y\)\|\\,\\bigr\]\\;\\leq\\;\\delta\_\{J\}\+\\varepsilon\.\(54\)Moreover,

Pr⁡\(\|sJ​\(X,Y\)−c⁡\(X,Y\)\|\>δJ\)≤ε\.\\Pr\\bigl\(\|s\_\{J\}\(X,Y\)\-c\(X,Y\)\|\>\\delta\_\{J\}\\bigr\)\\leq\\varepsilon\.\(55\)

###### Proof\.

By triangle inequality,\|sJ−c\|≤\|sJ−r\|\+\|r−c\|\|s\_\{J\}\-c\|\\leq\|s\_\{J\}\-r\|\+\|r\-c\|\. Taking expectation gives𝔼​\|sJ−c\|≤𝔼\|sJ−r\|\+𝔼​\|r−c\|≤δJ\+Pr⁡\(r≠c\)≤δJ\+ε\\mathbb\{E\}\|s\_\{J\}\-c\|\\leq\\mathbb\{E\}\|s\_\{J\}\-r\|\+\\mathbb\{E\}\|r\-c\|\\leq\\delta\_\{J\}\+\\Pr\(r\\neq c\)\\leq\\delta\_\{J\}\+\\varepsilon\. For the high\-probability statement, on the event\{r=c\}\\\{r=c\\\}we have\|sJ−c\|=\|sJ−r\|≤δJ\|s\_\{J\}\-c\|=\|s\_\{J\}\-r\|\\leq\\delta\_\{J\}, so\{\|sJ−c\|\>δJ\}⊆\{r≠c\}\\\{\|s\_\{J\}\-c\|\>\\delta\_\{J\}\\\}\\subseteq\\\{r\\neq c\\\}andPr⁡\(\|sJ−c\|\>δJ\)≤Pr⁡\(r≠c\)≤ε\\Pr\(\|s\_\{J\}\-c\|\>\\delta\_\{J\}\)\\leq\\Pr\(r\\neq c\)\\leq\\varepsilon\. ∎

###### Theorem F\.8\(Actor objective aligns initial correctness in Actor–Editor\)\.

Under Assumptions[F\.2](https://arxiv.org/html/2608.21897#A6.Thmassumption2)and[F\.5](https://arxiv.org/html/2608.21897#A6.Thmassumption5), for allπA\\pi\_\{A\},

\|JA​\(πA\)−\(1\+λJ\)​Cinit​\(πA\)\|≤\(1\+λJ\)​ε\+λJ​δJ\.\\Bigl\|J\_\{A\}\(\\pi\_\{A\}\)\-\(1\+\\lambda\_\{J\}\)\\,C\_\{\\mathrm\{init\}\}\(\\pi\_\{A\}\)\\Bigr\|\\;\\leq\\;\(1\+\\lambda\_\{J\}\)\\varepsilon\+\\lambda\_\{J\}\\delta\_\{J\}\.\(56\)Consequently, any maximizerπA⋆∈arg⁡maxπA​JA​\(πA\)\\pi\_\{A\}^\{\\star\}\\in\\arg\\max\_\{\\pi\_\{A\}\}J\_\{A\}\(\\pi\_\{A\}\)satisfies

Cinit​\(πA⋆\)≥supπACinit​\(πA\)−2​\(\(1\+λJ\)​ε\+λJ​δJ\)1\+λJ\.C\_\{\\mathrm\{init\}\}\(\\pi\_\{A\}^\{\\star\}\)\\;\\geq\\;\\sup\_\{\\pi\_\{A\}\}C\_\{\\mathrm\{init\}\}\(\\pi\_\{A\}\)\\;\-\\;\\frac\{2\\bigl\(\(1\+\\lambda\_\{J\}\)\\varepsilon\+\\lambda\_\{J\}\\delta\_\{J\}\\bigr\)\}\{1\+\\lambda\_\{J\}\}\.\(57\)

###### Proof\.

By definition,

JA​\(πA\)=𝔼⁡\[r⁡\(X,Y0\)\]\+λJ​𝔼​\[sJ​\(X,Y0\)\]\.J\_\{A\}\(\\pi\_\{A\}\)=\\mathbb\{E\}\[r\(X,Y\_\{0\}\)\]\+\\lambda\_\{J\}\\,\\mathbb\{E\}\[s\_\{J\}\(X,Y\_\{0\}\)\]\.Subtract\(1\+λJ\)​𝔼​\[c⁡\(X,Y0\)\]\(1\+\\lambda\_\{J\}\)\\mathbb\{E\}\[c\(X,Y\_\{0\}\)\]and apply triangle inequality:

\|JA−\(1\+λJ\)​Cinit\|≤\|𝔼⁡\[r−c\]\|\+λJ​\|𝔼⁡\[sJ−c\]\|≤𝔼\|r−c\|\+λJ​𝔼​\|sJ−c\|\.\\Bigl\|J\_\{A\}\-\(1\+\\lambda\_\{J\}\)C\_\{\\mathrm\{init\}\}\\Bigr\|\\leq\\bigl\|\\mathbb\{E\}\[r\-c\]\\bigr\|\+\\lambda\_\{J\}\\bigl\|\\mathbb\{E\}\[s\_\{J\}\-c\]\\bigr\|\\leq\\mathbb\{E\}\|r\-c\|\+\\lambda\_\{J\}\\,\\mathbb\{E\}\|s\_\{J\}\-c\|\.Assumption[F\.2](https://arxiv.org/html/2608.21897#A6.Thmassumption2)gives𝔼​\|r−c\|=Pr⁡\(r≠c\)≤ε\\mathbb\{E\}\|r\-c\|=\\Pr\(r\\neq c\)\\leq\\varepsilon, and Lemma[F\.7](https://arxiv.org/html/2608.21897#A6.Thmtheorem7)gives𝔼​\|sJ−c\|≤δJ\+ε\\mathbb\{E\}\|s\_\{J\}\-c\|\\leq\\delta\_\{J\}\+\\varepsilon, yielding \([56](https://arxiv.org/html/2608.21897#A6.E56)\)\. Apply Lemma[F\.4](https://arxiv.org/html/2608.21897#A6.Thmtheorem4)witha=1\+λJa=1\+\\lambda\_\{J\},b=0b=0, andΔ=\(1\+λJ\)​ε\+λJ​δJ\\Delta=\(1\+\\lambda\_\{J\}\)\\varepsilon\+\\lambda\_\{J\}\\delta\_\{J\}to obtain \([57](https://arxiv.org/html/2608.21897#A6.E57)\)\. ∎

##### Editor monotonicity for final solvability\.

Let the Editor iteratively refineYtY\_\{t\}fort≤Tmaxt\\leq T\_\{\\max\}\. Define

Rfinal≔𝕀⁡\{∃t≤Tmax:r⁡\(X,Yt\)=1\}\.R\_\{\\mathrm\{final\}\}\\;\\coloneqq\\;\\mathbb\{I\}\\\{\\exists t\\leq T\_\{\\max\}:r\(X,Y\_\{t\}\)=1\\\}\.\(58\)
###### Lemma F\.9\(Repair is weakly monotone in solver success\)\.

If the Editor action space includes a no\-op \(i\.e\., it can keepYt\+1=YtY\_\{t\+1\}=Y\_\{t\}\), then for any fixed Actor policyπA\\pi\_\{A\}and any Editor policyπE\\pi\_\{E\},

𝔼⁡\[Rfinal\]≥𝔼⁡\[r⁡\(X,Y0\)\]\.\\mathbb\{E\}\[R\_\{\\mathrm\{final\}\}\]\\;\\geq\\;\\mathbb\{E\}\[r\(X,Y\_\{0\}\)\]\.\(59\)

###### Proof\.

Pointwise, the event\{r\(X,Y0\)=1\}\\\{r\(X,Y\_\{0\}\)=1\\\}implies\{∃t≤Tmax:r⁡\(X,Yt\)=1\}\\\{\\exists t\\leq T\_\{\\max\}:r\(X,Y\_\{t\}\)=1\\\}, since the Editor can keepYt=Y0Y\_\{t\}=Y\_\{0\}\. ThusRfinal≥r⁡\(X,Y0\)R\_\{\\mathrm\{final\}\}\\geq r\(X,Y\_\{0\}\)almost surely, and taking expectations yields \([59](https://arxiv.org/html/2608.21897#A6.E59)\)\. ∎

###### Theorem F\.10\(Why Actor–Editor\)\.

Assume Assumption[F\.2](https://arxiv.org/html/2608.21897#A6.Thmassumption2)\.

1. 1\.\(Actor\-only\) MaximizingJA−onlyJ\_\{\\mathrm\{A\-only\}\}yields an initial\-correctness guarantee within2​ε2\\varepsilonof the optimum \(Theorem[F\.5](https://arxiv.org/html/2608.21897#A6.Thmtheorem5)\)\.
2. 2\.\(Editor\-only\) In the saturated\-repair regime \(Assumptions[F\.3](https://arxiv.org/html/2608.21897#A6.Thmassumption3)and[F\.4](https://arxiv.org/html/2608.21897#A6.Thmassumption4)\), the monolithic objective admits globally optimal policies with arbitrarily lowCinitC\_\{\\mathrm\{init\}\}\(Theorem[F\.6](https://arxiv.org/html/2608.21897#A6.Thmtheorem6)\)\.
3. 3\.\(Actor–Editor\) With a calibrated Judge \(Assumption[F\.5](https://arxiv.org/html/2608.21897#A6.Thmassumption5)\), maximizing the Actor objectiveJAJ\_\{A\}enforces initial\-correctness alignment up to𝒪⁡\(ε\+δJ\)\\mathcal\{O\}\(\\varepsilon\+\\delta\_\{J\}\)\(Theorem[F\.8](https://arxiv.org/html/2608.21897#A6.Thmtheorem8)\), while the Editor weakly improves final solvability \(Lemma[F\.9](https://arxiv.org/html/2608.21897#A6.Thmtheorem9)\)\.

### F\.4Scaling of Local Reward Hacking: Separate vs\. Shared Parameters

This subsection provides a proof of the scaling claim in Theorem 3\.2 with the*worst\-case first\-order reward\-hacking gain*under a local linearization\.

##### Local linearization and hacking gain\.

Fix a reference pointθ0\\theta\_\{0\}and a constanta\>0a\>0\. LetJA​\(θ\)J\_\{A\}\(\\theta\)denote the Actor objective andC⁡\(θ\)C\(\\theta\)the \(true\) plan\-correctness objective\. Define the*reward\-hacking gradient*

ghack​\(θ0\)≔∇θJA​\(θ0\)−a​∇θC​\(θ0\)\.g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\;\\coloneqq\\;\\nabla\_\{\\theta\}J\_\{A\}\(\\theta\_\{0\}\)\-a\\,\\nabla\_\{\\theta\}C\(\\theta\_\{0\}\)\.\(60\)For a perturbationΔ​θ\\Delta\\thetawith‖Δ​θ‖2≤1\\\|\\Delta\\theta\\\|\_\{2\}\\leq 1, the first\-order \(local\) hacking gain is

Δhack​\(θ0,Δ​θ\)≔ghack​\(θ0\)⊤​Δ​θ\.\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\;\\coloneqq\\;g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)^\{\\top\}\\Delta\\theta\.\(61\)Given an admissible perturbation setℋ\\mathcal\{H\}, the worst\-case local hacking gain is

supΔ​θ∈ℋΔhack​\(θ0,Δ​θ\)\.\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\.\(62\)

##### Architectures and admissible perturbation sets\.

- •Separate models\.The Actor has its own parametersθA∈ℝPA\\theta\_\{A\}\\in\\mathbb\{R\}^\{P\_\{A\}\}\. A unit\-norm perturbation can move freely in the Actor space: ℋsep≔\{Δ​θA∈ℝPA:‖Δ​θA‖2≤1\}\.\\mathcal\{H\}\_\{\\mathrm\{sep\}\}\\;\\coloneqq\\;\\bigl\\\{\\Delta\\theta\_\{A\}\\in\\mathbb\{R\}^\{P\_\{A\}\}:\\ \\\|\\Delta\\theta\_\{A\}\\\|\_\{2\}\\leq 1\\bigr\\\}\.\(63\)
- •Shared backbone\.Parameters decompose as θ=\(θsh,θA,θJ,θE\),\\theta=\(\\theta\_\{\\mathrm\{sh\}\},\\theta\_\{A\},\\theta\_\{J\},\\theta\_\{E\}\),whereθsh∈ℝp\\theta\_\{\\mathrm\{sh\}\}\\in\\mathbb\{R\}^\{p\}is the shared backbone, andθA∈ℝh\\theta\_\{A\}\\in\\mathbb\{R\}^\{h\}is the Actor head \(withh≪ph\\ll p\)\. We define admissible perturbations as those that are*\(Judge,Editor\)\-neutral on the backbone*and do not modify the Judge/Editor heads: ℋshr≔\{Δθ:∥Δθ∥2≤1,ΔθJ=0,ΔθE=0,Δθsh∈Null\(ΣJ​E\)\},\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\\;\\coloneqq\\;\\Bigl\\\{\\Delta\\theta:\\ \\\|\\Delta\\theta\\\|\_\{2\}\\leq 1,\\ \\Delta\\theta\_\{J\}=0,\\ \\Delta\\theta\_\{E\}=0,\\ \\Delta\\theta\_\{\\mathrm\{sh\}\}\\in\\mathrm\{Null\}\(\\Sigma\_\{JE\}\)\\Bigr\\\},\(64\)whereΣJ​E∈ℝp×p\\Sigma\_\{JE\}\\in\\mathbb\{R\}^\{p\\times p\}is defined in Assumption[F\.6](https://arxiv.org/html/2608.21897#A6.Thmassumption6)below\.

###### Assumption F\.6\(Judge and Editor constrain the backbone \(covariance form\)\)\.

Atθ0\\theta\_\{0\}, letgJ∈ℝpg\_\{J\}\\in\\mathbb\{R\}^\{p\}andgE∈ℝpg\_\{E\}\\in\\mathbb\{R\}^\{p\}denote the \(random\) backbone gradients associated with the Judge and Editor objectives under their respective training distributions \(e\.g\., per\-sample or per\-trajectory stochastic gradients\)\. Define the \(backbone\) second\-moment / Gram matrix

ΣJ​E≔𝔼⁡\[gJ​gJ⊤\]\+𝔼⁡\[gE​gE⊤\]∈ℝp×p\.\\Sigma\_\{JE\}\\;\\coloneqq\\;\\mathbb\{E\}\\bigl\[g\_\{J\}g\_\{J\}^\{\\top\}\\bigr\]\+\\mathbb\{E\}\\bigl\[g\_\{E\}g\_\{E\}^\{\\top\}\\bigr\]\\in\\mathbb\{R\}^\{p\\times p\}\.\(65\)AssumeΣJ​E\\Sigma\_\{JE\}is full rank on the backbone, equivalently

Null⁡\(ΣJ​E\)=\{0\}\.\\mathrm\{Null\}\(\\Sigma\_\{JE\}\)=\\\{0\\\}\.\(66\)

Assumption[F\.6](https://arxiv.org/html/2608.21897#A6.Thmassumption6)is a high\-rank*coverage*condition: Judge and Editor gradients collectively “see” all backbone directions \(in second moment\), so there is no nonzero backbone direction that is simultaneously neutral to both\.

###### Assumption F\.7\(Isotropic hacking gradient\)\.

There existsσ\>0\\sigma\>0such that the restriction ofghack​\(θ0\)g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)to the relevant parameter subspace is isotropic Gaussian\. Concretely:

1. 1\.In the*separate*setting,ghack​\(θ0\)∈ℝPAg\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\in\\mathbb\{R\}^\{P\_\{A\}\}satisfies ghack​\(θ0\)∼𝒩⁡\(0,σ2​IPA\)\.g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{P\_\{A\}\}\)\.\(67\)
2. 2\.In the*shared*setting, the projection ofghack​\(θ0\)g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)onto the Actor\-head subspaceℝh\\mathbb\{R\}^\{h\}satisfies ProjA​ghack​\(θ0\)∼𝒩⁡\(0,σ2​Ih\)\.\\mathrm\{Proj\}\_\{A\}\\,g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{h\}\)\.\(68\)

#### A Gaussian supremum lemma

###### Lemma F\.11\(Supremum of an isotropic Gaussian over a unit ball\)\.

Letg∼𝒩⁡\(0,σ2​Id\)g\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\)inℝd\\mathbb\{R\}^\{d\}and let𝔹d≔\{u∈ℝd:‖u‖2≤1\}\\mathbb\{B\}\_\{d\}\\coloneqq\\\{u\\in\\mathbb\{R\}^\{d\}:\\\|u\\\|\_\{2\}\\leq 1\\\}\. Then

𝔼⁡\[supu∈𝔹dg⊤​u\]=𝔼⁡\[‖g‖2\]=σ​𝔼​\[‖Z‖2\],Z∼𝒩⁡\(0,Id\)\.\\mathbb\{E\}\\Big\[\\sup\_\{u\\in\\mathbb\{B\}\_\{d\}\}g^\{\\top\}u\\Big\]=\\mathbb\{E\}\\big\[\\\|g\\\|\_\{2\}\\big\]=\\sigma\\,\\mathbb\{E\}\\big\[\\\|Z\\\|\_\{2\}\\big\],\\qquad Z\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\.\(69\)Moreover, for alld≥1d\\geq 1,

σ​d−1≤𝔼⁡\[‖g‖2\]≤σ​d,\\sigma\\sqrt\{d\-1\}\\;\\leq\\;\\mathbb\{E\}\\big\[\\\|g\\\|\_\{2\}\\big\]\\;\\leq\\;\\sigma\\sqrt\{d\},\(70\)and in particular𝔼⁡\[‖g‖2\]=Θ⁡\(σ​d\)\\mathbb\{E\}\[\\\|g\\\|\_\{2\}\]=\\Theta\(\\sigma\\sqrt\{d\}\)\.

###### Proof\.

For any fixedgg, the Cauchy–Schwarz inequality givessup‖u‖≤1g⊤​u=‖g‖2\\sup\_\{\\\|u\\\|\\leq 1\}g^\{\\top\}u=\\\|g\\\|\_\{2\}, achieved byu=g/‖g‖2u=g/\\\|g\\\|\_\{2\}ifg≠0g\\neq 0\. Taking expectation yields the first equality in \([69](https://arxiv.org/html/2608.21897#A6.E69)\)\.

The upper bound in \([70](https://arxiv.org/html/2608.21897#A6.E70)\) follows from Jensen:𝔼​‖g‖2≤𝔼​‖g‖22=𝔼⁡\[g⊤​g\]=σ​d\.\\mathbb\{E\}\\\|g\\\|\_\{2\}\\leq\\sqrt\{\\mathbb\{E\}\\\|g\\\|\_\{2\}^\{2\}\}=\\sqrt\{\\mathbb\{E\}\[g^\{\\top\}g\]\}=\\sigma\\sqrt\{d\}\.The lower bound𝔼​‖Z‖2≥d−1\\mathbb\{E\}\\\|Z\\\|\_\{2\}\\geq\\sqrt\{d\-1\}forZ∼𝒩⁡\(0,Id\)Z\\sim\\mathcal\{N\}\(0,I\_\{d\}\)is classical \(equivalently for aχd\\chi\_\{d\}random variable\), and follows for example from standard gamma\-function bounds for𝔼​χd=2​Γ​\(d\+12\)/Γ⁡\(d2\)\\mathbb\{E\}\\chi\_\{d\}=\\sqrt\{2\}\\,\\Gamma\\bigl\(\\frac\{d\+1\}\{2\}\\bigr\)/\\Gamma\\bigl\(\\frac\{d\}\{2\}\\bigr\)\. Multiplying byσ\\sigmayields \([70](https://arxiv.org/html/2608.21897#A6.E70)\)\. ∎

#### Main result

###### Theorem F\.12\(Scaling of local reward\-hacking gain: separate vs\. shared\)\.

Under Assumptions[F\.6](https://arxiv.org/html/2608.21897#A6.Thmassumption6)and[F\.7](https://arxiv.org/html/2608.21897#A6.Thmassumption7):

1. 1\.Three separate models\.Withℋsep\\mathcal\{H\}\_\{\\mathrm\{sep\}\}as in \([63](https://arxiv.org/html/2608.21897#A6.E63)\), 𝔼⁡\[supΔ​θ∈ℋsepΔhack​\(θ0,Δ​θ\)\]=Θ⁡\(σ​PA\)\.\\mathbb\{E\}\\Bigl\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{sep\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\Bigr\]\\;=\\;\\Theta\\\!\\bigl\(\\sigma\\sqrt\{P\_\{A\}\}\\bigr\)\.\(71\)
2. 2\.One shared backbone\.Withℋshr\\mathcal\{H\}\_\{\\mathrm\{shr\}\}as in \([64](https://arxiv.org/html/2608.21897#A6.E64)\), 𝔼⁡\[supΔ​θ∈ℋshrΔhack​\(θ0,Δ​θ\)\]≤σ​h,\\mathbb\{E\}\\Bigl\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\Bigr\]\\;\\leq\\;\\sigma\\sqrt\{h\},\(72\)hence the expected local reward\-hacking gain is𝒪⁡\(σ​h\)\\mathcal\{O\}\(\\sigma\\sqrt\{h\}\)and does not scale with the backbone dimensionpp\.

###### Proof\.

We prove the two items\.

##### \(1\) Separate models\.

In the separate setting,Δ​θ\\Delta\\thetaranges over the Actor parameter spaceℝPA\\mathbb\{R\}^\{P\_\{A\}\}, and by \([61](https://arxiv.org/html/2608.21897#A6.E61)\)–\([63](https://arxiv.org/html/2608.21897#A6.E63)\),

supΔ​θ∈ℋsepΔhack​\(θ0,Δ​θ\)=sup‖Δ​θ‖2≤1ghack​\(θ0\)⊤​Δ​θ=‖ghack​\(θ0\)‖2\.\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{sep\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)=\\sup\_\{\\\|\\Delta\\theta\\\|\_\{2\}\\leq 1\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)^\{\\top\}\\Delta\\theta=\\\|g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\\|\_\{2\}\.By Assumption[F\.7](https://arxiv.org/html/2608.21897#A6.Thmassumption7)\(separate case\) and Lemma[F\.11](https://arxiv.org/html/2608.21897#A6.Thmtheorem11)withd=PAd=P\_\{A\},

𝔼⁡\[supΔ​θ∈ℋsepΔhack​\(θ0,Δ​θ\)\]=𝔼​‖ghack​\(θ0\)‖2=Θ⁡\(σ​PA\),\\mathbb\{E\}\\Bigl\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{sep\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\Bigr\]=\\mathbb\{E\}\\\|g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\\|\_\{2\}=\\Theta\(\\sigma\\sqrt\{P\_\{A\}\}\),proving \([71](https://arxiv.org/html/2608.21897#A6.E71)\)\.

##### \(2\) Shared backbone\.

By Assumption[F\.6](https://arxiv.org/html/2608.21897#A6.Thmassumption6),Null⁡\(ΣJ​E\)=\{0\}\\mathrm\{Null\}\(\\Sigma\_\{JE\}\)=\\\{0\\\}, henceΔ​θsh=0\\Delta\\theta\_\{\\mathrm\{sh\}\}=0for allΔ​θ∈ℋshr\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}by \([64](https://arxiv.org/html/2608.21897#A6.E64)\)\. MoreoverΔ​θJ=Δ​θE=0\\Delta\\theta\_\{J\}=\\Delta\\theta\_\{E\}=0by definition ofℋshr\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\. Therefore anyΔ​θ∈ℋshr\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}lies entirely in the Actor\-head subspace, so we can writeΔ​θ=\(0,Δ​θA,0,0\)\\Delta\\theta=\(0,\\Delta\\theta\_\{A\},0,0\)with‖Δ​θA‖2≤1\\\|\\Delta\\theta\_\{A\}\\\|\_\{2\}\\leq 1\. Thus,

supΔ​θ∈ℋshrΔhack​\(θ0,Δ​θ\)=sup‖Δ​θA‖2≤1\(ProjA​ghack​\(θ0\)\)⊤​Δ​θA=‖ProjA​ghack​\(θ0\)‖2\.\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)=\\sup\_\{\\\|\\Delta\\theta\_\{A\}\\\|\_\{2\}\\leq 1\}\\bigl\(\\mathrm\{Proj\}\_\{A\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\bigr\)^\{\\top\}\\Delta\\theta\_\{A\}=\\bigl\\\|\\mathrm\{Proj\}\_\{A\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\bigr\\\|\_\{2\}\.By Assumption[F\.7](https://arxiv.org/html/2608.21897#A6.Thmassumption7)\(shared case\),ProjA​ghack​\(θ0\)∼𝒩⁡\(0,σ2​Ih\)\\mathrm\{Proj\}\_\{A\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{h\}\)\. Applying Lemma[F\.11](https://arxiv.org/html/2608.21897#A6.Thmtheorem11)withd=hd=hyields

𝔼⁡\[supΔ​θ∈ℋshrΔhack​\(θ0,Δ​θ\)\]=𝔼​‖ProjA​ghack​\(θ0\)‖2≤σ​h,\\mathbb\{E\}\\Bigl\[\\sup\_\{\\Delta\\theta\\in\\mathcal\{H\}\_\{\\mathrm\{shr\}\}\}\\Delta\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\};\\Delta\\theta\)\\Bigr\]=\\mathbb\{E\}\\bigl\\\|\\mathrm\{Proj\}\_\{A\}g\_\{\\mathrm\{hack\}\}\(\\theta\_\{0\}\)\\bigr\\\|\_\{2\}\\leq\\sigma\\sqrt\{h\},which is \([72](https://arxiv.org/html/2608.21897#A6.E72)\)\. The bound depends onhhbut not onpp\. ∎

##### Remark \(effective\-rank variant\)\.

IfNull⁡\(ΣJ​E\)\\mathrm\{Null\}\(\\Sigma\_\{JE\}\)has dimensionk\>0k\>0\(i\.e\., Judge\+Editor do not constrainkkbackbone directions\), then the same proof yields a shared\-architecture bound of orderσ​h\+k\\sigma\\sqrt\{h\+k\}, wherekkis the number of unconstrained backbone directions\. This makes explicit that the scaling depends on the*dimension of the \(Judge,Editor\)\-neutral subspace*, not directly on the backbone sizepp\.

## Appendix GDetailed Experimental Setup

### G\.1Benchmarks

##### PlanBench\.

PlanBench\[[Valmeekam et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib25)\]evaluates whether language models can solve classical planning problems described in natural language while preserving the underlying symbolic transition structure\. Each instance is derived from a PDDL planning problem and converted into a natural\-language prompt\. We report results on four domains: BlocksWorld, Mystery BlocksWorld, Logistics, and Gripper\. BlocksWorld tests basic object manipulation and state\-transition reasoning\. Mystery BlocksWorld preserves the same transition dynamics but systematically renames objects and predicates, thereby removing lexical cues and testing whether the model recovers the underlying symbolic structure\. Logistics and Gripper introduce more brittle object typing, movement constraints, and action\-schema dependencies\.

##### Zero\-shot transfer benchmarks\.

To evaluate whether solver\-grounded training improves planning behavior beyond the PlanBench domains, we additionally test zero\-shot transfer on ProntoQA\[[Saparov and He 2023](https://arxiv.org/html/2608.21897#bib.bib17)\]and NATURAL PLAN\[[Zheng et al\. 2024](https://arxiv.org/html/2608.21897#bib.bib33)\]\. ProntoQA requires multi\-hop logical inference from explicitly stated facts and rules\. NATURAL PLAN evaluates realistic natural\-language planning under constraints; we use the Trip Planning and Calendar Scheduling subsets\. In NATURAL PLAN, all external information needed for planning, such as flight connectivity or calendar availability, is provided in\-context, so performance reflects planning and constraint satisfaction rather than tool invocation\.

### G\.2Evaluation Protocol

##### PDDL\-based evaluation\.

For PlanBench, each model output is parsed as a PDDL domain–problem pair\. We use Fast Downward with a 60\-second timeout as the external planner\. An instance is counted as successful only when both generated files are syntactically valid, the planner returns a plan within the timeout, and the plan satisfies the stated goal conditions\. Outputs that fail parsing, grounding, type checking, or goal achievement are counted as failures\.

##### Non\-PDDL evaluation\.

For ProntoQA and NATURAL PLAN, we follow the original benchmark protocols and report exact\-match success against the gold label or gold plan\. We apply the same answer\-normalization rules across all methods\. No benchmark\-specific supervised examples are used for training our method\.

### G\.3Model and Training Setting

##### Backbone\.

Our framework uses Qwen2\.5\-7B\[[Qwen et al\. 2025](https://arxiv.org/html/2608.21897#bib.bib16)\]initialized from pretrained weights\. The model is conditioned into three roles: Actor, Judge, and Editor\. The Actor generates the initial PDDL specification, the Judge predicts a solver\-calibrated quality score, and the Editor repairs failed specifications using solver diagnostics\.

##### Annotation\-free training\.

No human\-written PDDL demonstrations, gold domain files, gold problem files, or supervised planning traces are used to train our method\. The only external feedback used during training is produced by the symbolic planner and its diagnostics\. This ensures that the reported performance reflects solver\-grounded learning rather than supervised imitation of annotated PDDL\.

##### Inference\.

At inference time, the Actor first generates an initial specification\. If the specification fails solver verification, the Editor performs bounded diagnostic\-conditioned repair\. Unless otherwise specified, we use the same maximum repair horizon as in training\. The system returns the first solver\-executable specification found within the repair budget; if no repair succeeds, the final edited specification is submitted for evaluation\.

### G\.4Baselines

##### Chain\-of\-Thought prompting\.

The CoT baseline uses chain\-of\-thought prompting\[[Wei et al\. 2022](https://arxiv.org/html/2608.21897#bib.bib28)\]to elicit intermediate reasoning before producing the final plan or formal specification\. We use an out\-of\-domain prompt and do not provide task\-specific PDDL annotations\.

##### Tree\-of\-Thought search\.

The ToT baseline follows the search\-based reasoning framework of[Yao et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib30)\. The language model expands and evaluates intermediate reasoning states using breadth\-first search\. The search process does not access symbolic execution feedback during thought expansion\.

##### LLM\+P\.

LLM\+P\[[Liu et al\. 2023](https://arxiv.org/html/2608.21897#bib.bib11)\]translates natural\-language planning descriptions into PDDL, invokes a classical planner, and then maps the resulting plan back to natural language when needed\. In our setup, the baseline uses fixed domain assumptions and in\-context PDDL examples, but no additional human\-annotated training data\.

##### Backbone choice\.

The main comparison uses GPT\-4o for CoT, ToT, and LLM\+P, providing strong off\-the\-shelf zero\-annotation baselines\. To separate algorithmic gains from backbone effects and solver\-call budgets, we additionally report matched\-backbone and matched\-budget comparisons in Appendix[H\.1](https://arxiv.org/html/2608.21897#A8.SS1), where all controlled baselines use Qwen2\.5\-7B and the same maximum number of solver calls as our method\.

## Appendix HAdditional Experimental Details and Discussion

### H\.1Fair Baseline Comparison under Matched Backbone and Solver Budget

The main comparison in Table[1](https://arxiv.org/html/2608.21897#S3.T1)follows prior planning baselines with strong prompting and neuro\-symbolic pipelines\. However, to rule out confounding factors from different backbone models and different verifier\-access budgets, we conduct an additional controlled comparison\. All methods in this subsection use the same Qwen2\.5\-7B backbone as our method and are allowed the same maximum number of solver calls at inference time\. Since our default inference procedure consists of one initial Actor generation followed by at most five Editor repairs, the maximum solver\-call budget is

K=Hmax\+1=6\.K=H\_\{\\max\}\+1=6\.For prompting baselines that do not use solver diagnostics, we allow up toKKindependently decoded candidates and select the first solver\-successful output if one exists\. For the self\-refine baseline, the model receives the raw solver diagnostic after each failed attempt and revises its previous PDDL specification for at most five rounds\. This gives all baselines comparable access to solver verification while isolating the effect of trained role specialization\.

##### Controlled baselines\.

We evaluate four same\-backbone baselines:

- •Qwen\-CoTuses chain\-of\-thought prompting with up to six sampled candidates\. The solver is used only for candidate selection, not for textual feedback\.
- •Qwen\-ToTuses tree\-of\-thought search with Qwen2\.5\-7B as both generator and evaluator\. We cap the final solver\-checked candidates at six\.
- •Qwen\-LLM\+Pprompts Qwen2\.5\-7B to translate the natural\-language task into PDDL and invokes the planner on each candidate\. We allow up to six independently sampled formalizations\.
- •Qwen\-Self\-Refine\+Solveruses the same backbone and the same solver\-call budget as our method, but has no trained Actor, Judge, or Editor\. It revises its previous PDDL output using the solver diagnostic through prompting alone\.

None of these baselines uses supervised PDDL annotations or task\-specific training\.

Table 8:Planning success rate \(%\) under matched backbone and matched solver\-call budget\. All Qwen baselines use Qwen2\.5\-7B and at mostK=6K=6solver calls per test instance\. Avg\. denotes the mean across the four PlanBench domains\.Table[8](https://arxiv.org/html/2608.21897#A8.T8)shows that the advantage of our method is not caused by using a stronger backbone or by having more access to the symbolic solver\. When all methods use Qwen2\.5\-7B and the same maximum solver\-call budget, CoT and ToT remain weak, especially on Mystery BlocksWorld and Logistics\. This indicates that sampling more reasoning traces or searching over textual thoughts does not reliably recover valid symbolic transition models\. Qwen\-LLM\+P performs well on BlocksWorld but degrades sharply on Mystery BlocksWorld, Logistics, and Gripper, suggesting that in\-context PDDL formalization remains sensitive to domain familiarity and brittle action\-schema construction\.

The strongest same\-backbone baseline is Qwen\-Self\-Refine\+Solver, which uses the same solver diagnostics available to our Editor at inference time\. It improves over Qwen\-LLM\+P by revising syntactic and type\-level mistakes, reaching 49\.8% average success\. However, it still remains 21\.0 percentage points behind our method\. This gap suggests that solver diagnostics alone are useful but insufficient: without a trained Judge and diagnostic\-conditioned Editor, prompting\-based repair often fixes local errors while leaving global predicate structure, goal preservation, or action semantics unstable\. The GPT\-4o self\-refine baseline provides an even stronger untrained reference, but it still underperforms our Qwen2\.5\-7B method\. Therefore, the improvement is better explained by the learned multi\-role training procedure rather than by backbone strength or additional solver calls\.

Table 9:Budget efficiency and semantic faithfulness under the matchedK=6K=6solver\-call setting\. Avg\. Calls is the average number of solver invocations actually used before success or budget exhaustion\. Faithful denotes the fraction of all tasks that are both solver\-successful and semantically faithful according to the reference checker\. Drift is the conditional fraction of solver\-successful outputs that fail semantic checking\.Table[9](https://arxiv.org/html/2608.21897#A8.T9)further shows that our method is not simply spending the solver budget more aggressively\. Although all methods are allowed up to six solver calls, our method uses only 3\.2 calls on average because many instances are solved by the initial Actor output or by early Editor repairs\. In contrast, CoT, ToT, and LLM\+P frequently exhaust the budget without producing executable specifications\. Self\-refine baselines use diagnostics more effectively, but their semantic drift remains high: many repaired specifications become solver\-executable without fully preserving the intended task semantics\. Our method has the smallest gap between solvability and faithful success, and its drift rate is substantially lower than all matched\-budget baselines\. This supports the default design choice: the learned Judge and Editor do not merely increase the chance of passing the solver, but also stabilize the symbolic specification so that solver success remains aligned with task\-level faithfulness\.

These controlled comparisons address two possible alternative explanations\. First, the gain is not due to backbone choice, since all same\-backbone Qwen2\.5\-7B baselines remain below our method\. Second, the gain is not due to a larger verifier budget, since every method is capped at the sameK=6K=6solver calls, and our method uses fewer calls on average\. The remaining performance gap is therefore attributable to the trained multi\-role decomposition: the Actor learns global PDDL construction, the Judge suppresses solver\-facing shortcuts, and the Editor learns targeted diagnostic\-conditioned repair\.

### H\.2Discussion of Main Planning and Transfer Results

Table[1](https://arxiv.org/html/2608.21897#S3.T1)shows that LLM\+P performs strongly on BlocksWorld but drops sharply on Logistics and Gripper\. This pattern is consistent with the fact that BlocksWorld is a canonical planning domain frequently appearing in PDDL tutorials and planning examples, whereas Logistics and Gripper require more careful object typing, action preconditions, and movement constraints\. Mystery BlocksWorld is especially diagnostic because it preserves the same transition dynamics as BlocksWorld while removing lexical cues through systematic renaming\. The large gain on Mystery BlocksWorld therefore suggests that our method is not simply relying on familiar predicate names, but is learning a solver\-grounded mapping from task descriptions to symbolic structure\.

For zero\-shot transfer, ProntoQA evaluates multi\-hop logical inference from explicit facts and rules, while NATURAL PLAN evaluates realistic planning tasks expressed in natural language\. The Trip Planning and Calendar Scheduling subsets require satisfying multiple constraints using information provided in\-context\. The improvement on these benchmarks indicates that the learned roles transfer beyond PDDL syntax: the Actor learns to propose structured solutions, the Judge learns to score consistency, and the Editor learns to refine outputs under constraint feedback\.

### H\.3Controlled Baseline Protocol

The controlled comparison in Table[2](https://arxiv.org/html/2608.21897#S4.T2)is designed to isolate the effect of trained role specialization\. All methods are allowed at most

solver calls per instance, matching our default inference procedure: one Actor generation followed by at most five Editor repairs\.

For CoT, ToT, and LLM\+P variants without diagnostic repair, we allow up toKKindependently decoded candidate outputs and select the first solver\-successful specification\. For self\-refine baselines, the model receives the raw solver diagnostic after each failed attempt and revises the previous PDDL specification for up to five rounds\. This gives prompting\-based repair access to the same type of solver feedback available to our Editor at inference time, but without any role\-specific training\.

Average solver calls are computed as the number of solver invocations used before either the first successful specification is found or the budget is exhausted\. The faithful and drift columns are computed using the same post\-hoc reference checker described in Appendix[H\.4](https://arxiv.org/html/2608.21897#A8.SS4)\. The result that our method uses3\.23\.2calls on average while achieving the highest faithful success indicates that the trained Actor and Editor reduce both search cost and semantic drift\.

### H\.4Semantic Faithfulness Protocol

The standard PlanBench metric evaluates whether the generated PDDL specification is solver\-executable\. However, solver success alone does not guarantee that the generated specification preserves the original natural\-language task\. A model may weaken goals, omit constraints, alter object types, or distort action schemas while still producing a solvable PDDL instance\. We therefore evaluate semantic faithfulness with a post\-hoc reference checker that is used only for evaluation and never for training\.

For each taskxix\_\{i\}, let

yi=\(yidom,yiprob\)y\_\{i\}=\(y\_\{i\}^\{\\mathrm\{dom\}\},y\_\{i\}^\{\\mathrm\{prob\}\}\)be the generated PDDL specification and letπ^i\\hat\{\\pi\}\_\{i\}be the plan returned by Fast Downward whenyiy\_\{i\}is solvable\. The solver\-success indicator is

Si=𝟏\{ℰ\(yi\)=1\}\.S\_\{i\}=\\mathbf\{1\}\\\{\\mathcal\{E\}\(y\_\{i\}\)=1\\\}\.\(73\)The reference checker𝒱ref\\mathcal\{V\}\_\{\\mathrm\{ref\}\}verifies three conditions:

1. 1\.Goal preservation:generated goal conditions are semantically equivalent to the reference goal after canonicalizing object and predicate names;
2. 2\.Object and type preservation:generated objects and type assignments preserve the entity structure of the original task;
3. 3\.Action\-schema consistency:generated action schemas preserve the reference transition semantics, measured by canonical precondition/effect matching and plan replay under the reference transition model\.

The faithful\-success indicator is then

Ci=𝟏\{Si=1∧𝒱ref\(xi,yi,π^i\)=1\}\.C\_\{i\}=\\mathbf\{1\}\\\{S\_\{i\}=1\\wedge\\mathcal\{V\}\_\{\\mathrm\{ref\}\}\(x\_\{i\},y\_\{i\},\\hat\{\\pi\}\_\{i\}\)=1\\\}\.\(74\)We report

Solvability=1N​∑i=1NSi,FaithfulSuccess=1N​∑i=1NCi,\\mathrm\{Solvability\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}S\_\{i\},\\qquad\\mathrm\{FaithfulSuccess\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}C\_\{i\},\(75\)and the conditional drift rate

Drift=∑i=1NSi​\(1−Ci\)∑i=1NSi\.\\mathrm\{Drift\}=\\frac\{\\sum\_\{i=1\}^\{N\}S\_\{i\}\(1\-C\_\{i\}\)\}\{\\sum\_\{i=1\}^\{N\}S\_\{i\}\}\.\(76\)A lower drift rate means that fewer solver\-successful outputs are achieved through semantic shortcuts\.

### H\.5Discussion of Faithfulness and Drift Results

The faithfulness results in Table[3](https://arxiv.org/html/2608.21897#S4.T3)show that solver\-only RL increases raw solvability but also creates substantial specification drift\. This confirms the central concern that raw solver success is an exploitable proxy when the model controls the generated formal specification\. In contrast, the full model improves solvability and faithful success simultaneously, while maintaining a much lower drift rate\.

The per\-domain faithful\-success results show that the advantage is not restricted to easier domains\. On Mystery BlocksWorld, where lexical cues are removed, our method retains67%67\\%faithful success\. On Logistics and Gripper, faithful success remains lower because small mistakes in typing, movement constraints, or precondition/effect structure can distort the intended transition system\. Nevertheless, our method remains substantially better than LLM\+P and solver\-only variants, suggesting that the multi\-role design improves specification quality rather than merely increasing solver executability\.

### H\.6Ablation and Drift\-Type Discussion

The ablation results in Table[4](https://arxiv.org/html/2608.21897#S4.T4)\(a\) isolate the contribution of each role on Mystery BlocksWorld\. The Actor\-only setting collapses because the policy must search a large symbolic specification space using sparse solver feedback\. Adding the Judge without the Editor improves success but remains limited because the system lacks a targeted repair mechanism\. Adding the Editor without the Judge also fails, suggesting that diagnostic repair alone can still drift toward solver\-facing shortcuts\. The full model succeeds because the Actor handles global formalization, the Judge supplies a calibrated quality signal, and the Editor performs bounded local correction\.

The separate\-model baseline with3×3\\times7B parameters also collapses\. This result is consistent with the analysis in Section[3\.7](https://arxiv.org/html/2608.21897#S3.SS7): fully separate role models expose more role\-private directions through which the Actor can improve solver\-facing reward without being constrained by Judge calibration or Editor repair behavior\. The shared\-backbone design reduces these degrees of freedom and stabilizes cross\-role credit assignment\.

Table[4](https://arxiv.org/html/2608.21897#S4.T4)\(b\) categorizes the remaining unfaithful but solver\-successful outputs\. LLM\+P and solver\-only RL contain many direct shortcut failures, especially goal weakening and object/type drift\. Our method reduces these shortcut\-like errors substantially\. Among the few remaining drift cases, action\-schema drift becomes the largest category, indicating that the residual failures are mostly subtle transition\-model mistakes rather than systematic goal removal or object manipulation\.

### H\.7Repair and Diagnostic Protocol

For each test instance, the Actor first generates an initial PDDL specificationy0y\_\{0\}, which is verified by the solver\. If verification fails, the Editor performs up toHmax=5H\_\{\\max\}=5sequential repairs, each conditioned on the task, the current specification, and the latest solver diagnostic\. LetSi\(h\)∈\{0,1\}S\_\{i\}^\{\(h\)\}\\in\\\{0,1\\\}denote whether instanceiihas been solved by stephh, whereh=0h=0corresponds to the initial Actor output andh∈\{1,…,5\}h\\in\\\{1,\\dots,5\\\}denotes the number of Editor repairs\. We report the cumulative success rate

CSR\(h\)=1N∑i=1NSi\(h\),h=0,1,…,5,\\mathrm\{CSR\}\(h\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}S\_\{i\}^\{\(h\)\},\\qquad h=0,1,\\dots,5,\(77\)and the marginal repair gain

Δ⁡\(h\)=CSR⁡\(h\)−CSR⁡\(h−1\),h≥1\.\\Delta\(h\)=\\mathrm\{CSR\}\(h\)\-\\mathrm\{CSR\}\(h\-1\),\\qquad h\\geq 1\.\(78\)
Diagnostics are grouped into six categories:syntax/parse error,object/type mismatch,predicate/schema mismatch,precondition/effect mismatch,goal/constraint drift, andunreachable planning search\. The diagnostic heatmap in Figure[3](https://arxiv.org/html/2608.21897#S4.F3)reports the dominant diagnostic type among remaining unsolved cases after each repair step\.

Averaged across domains, the initial Actor solves46\.2%46\.2\\%of instances\. Cumulative success rises to56\.5%56\.5\\%,63\.5%63\.5\\%,67\.5%67\.5\\%,69\.3%69\.3\\%, and70\.8%70\.8\\%after one to five repairs\. Thus, the Editor contributes24\.624\.6absolute points beyond the initial Actor output\. The first two repairs account for17\.317\.3points, or about70%70\\%of the total repair gain, which supports using a bounded repair horizon rather than an unbounded search loop\.

### H\.8Solvability–Faithfulness Gap Discussion

The solvability–faithfulness gap is defined as

Gap=Solvability−FaithfulSuccess\.\\mathrm\{Gap\}=\\mathrm\{Solvability\}\-\\mathrm\{FaithfulSuccess\}\.\(79\)This metric measures how tightly solver success aligns with task\-level semantic correctness\. A large gap means that many solver\-successful outputs are semantically unfaithful\.

Figure[4](https://arxiv.org/html/2608.21897#S4.F4)shows that our method has the smallest gap\. The Qwen\-Self\-Refine\+Solver baseline improves raw solvability by using solver diagnostics, but its faithful success remains much lower, indicating that prompting\-based repair often fixes local syntax or type errors without preserving the deeper transition semantics\. The GPT\-4o self\-refine baseline has a similar issue: stronger language modeling improves executable output generation, but does not fully suppress specification drift\. The smaller gap of our method suggests that the Actor, Judge, and Editor jointly maintain a closer connection between executable PDDL and the original natural\-language task\.

### H\.9Judge Candidate Construction and Interpretation

To evaluate whether the Judge distinguishes faithful outputs from solver\-successful but unfaithful outputs, we collect a candidate pool from model\-generated specifications during inference\. The pool includes initial Actor outputs, intermediate Editor repairs, and final returned specifications\. Each candidateyyis assigned two post\-hoc labels:

S⁡\(y\)∈\{0,1\},C⁡\(y\)∈\{0,1\},S\(y\)\\in\\\{0,1\\\},\\qquad C\(y\)\\in\\\{0,1\\\},whereS⁡\(y\)=1S\(y\)=1means the specification is solver\-executable, andC⁡\(y\)=1C\(y\)=1means the specification is both solver\-executable and semantically faithful under the reference checker\.

We partition candidates into three groups:

𝒰\\displaystyle\\mathcal\{U\}=\{y:S⁡\(y\)=0\},\\displaystyle=\\\{y:S\(y\)=0\\\},\(80\)𝒟\\displaystyle\\mathcal\{D\}=\{y:S\(y\)=1,C\(y\)=0\},\\displaystyle=\\\{y:S\(y\)=1,\\ C\(y\)=0\\\},ℱ\\displaystyle\\mathcal\{F\}=\{y:S\(y\)=1,C\(y\)=1\},\\displaystyle=\\\{y:S\(y\)=1,\\ C\(y\)=1\\\},corresponding to unsolved, solved\-but\-unfaithful, and solved\-and\-faithful outputs\. For each candidate, we record the Judge scoresJ​\(x,y\)∈\[0,1\]s\_\{J\}\(x,y\)\\in\[0,1\]\.

If the Judge only predicted raw solver acceptance, solved\-but\-unfaithful and faithful outputs would receive similar scores because both pass the solver\. Instead, Figure[5](https://arxiv.org/html/2608.21897#S4.F5)and Table[5](https://arxiv.org/html/2608.21897#S4.T5)show a clear ordering:

𝒰<𝒟<ℱ\.\\mathcal\{U\}<\\mathcal\{D\}<\\mathcal\{F\}\.This indicates that the Judge captures structural quality signals correlated with semantic faithfulness, even though it is not trained with human semantic labels\. The separation is strongest on BlocksWorld and weaker on Logistics and Gripper, where subtle action\-schema errors can still pass solver checks under distorted specifications\.