LLM-Only PDDL Domain Repair with Open-Weight Models

# LLM-Only PDDL Domain Repair with Open-Weight Models
Source: [https://arxiv.org/html/2608.17341](https://arxiv.org/html/2608.17341)
###### Abstract

AI planning is concerned with finding a sequence of actions that achieves a specified goal\. It relies on explicit models of the world, commonly represented in the Planning Domain Definition Language \(PDDL\)\. An active line of research investigates how errors in such models can be detected and repaired\. For example, users may provide positive test plans that are solutions, and negative test plans that fail during execution\. Automated repair methods then modify the PDDL model to satisfy these constraints\. In this paper, we evaluate the ability of recent open\-weight large language models to perform this repair task using an LLM\-only approach\. Our experiments show that the symbolic baseline achieves anF1F\_\{1\}score of\.49\.49, while the best\-performing LLM reaches\.87\.87with high reasoning effort, an absolute improvement of\.38\.38\. However, that setting has a mean test pass rate of only\.82\.82, falling to\.06\.06on the Thoughtful domain; even the best setting that includes the test traces reaches only\.92\.92\. Thus, current open\-weight models cannot guarantee satisfaction of the test constraints required for reliable automated model repair\.

## Introduction

Explainability is a central requirement for AI systems that interact with or support humans in decision making\. In AI planning, this requirement is naturally addressed by the explicit representation of actions, states, and goals: planners generate solutions by reasoning over structured models of the world\. Compared to black\-box machine learning techniques, this explicit reasoning process makes planning inherently transparent and interpretable\. However, one of the main challenges to deploying planning in practice lies in constructing the planning models themselves\([16](https://arxiv.org/html/2608.17341#bib.bib4)\)\.

The recent success of large language models \(LLMs\) has motivated extensive research into their application to AI planning tasks\([5](https://arxiv.org/html/2608.17341#bib.bib10);[15](https://arxiv.org/html/2608.17341#bib.bib9);[7](https://arxiv.org/html/2608.17341#bib.bib6);[9](https://arxiv.org/html/2608.17341#bib.bib7);[6](https://arxiv.org/html/2608.17341#bib.bib8)\)\. A recent survey\([16](https://arxiv.org/html/2608.17341#bib.bib4)\)highlights the potential of LLMs to support the construction and refinement of planning models\. While verifiable planning modules remain the backbone of reliability, robustness, and explainability, LLMs can act as assistants to reduce the manual burden of defining domain models\. We believe that one promising avenue is*domain repair*, where the goal is to identify a set of modifications to a domain such that the positive traces become executable and the negative traces become non\-executable\.

Repairs can themselves be understood as explanations\. Following[14](https://arxiv.org/html/2608.17341#bib.bib5)’s \([14](https://arxiv.org/html/2608.17341#bib.bib5)\) account of contrastive explanations in the social sciences, a repair answers the question of why a given trace fails in the current domain and provides a counterfactual justification of how the domain could have behaved differently\. Each repair is thus not only a technical fix, but also a form of interpretable feedback to the human domain modeler\.

Several symbolic approaches enforce executability constraints by automatically modifying PDDL models to satisfy them\([1](https://arxiv.org/html/2608.17341#bib.bib13);[4](https://arxiv.org/html/2608.17341#bib.bib14);[3](https://arxiv.org/html/2608.17341#bib.bib12);[2](https://arxiv.org/html/2608.17341#bib.bib15)\), while other work investigates the computational complexity of such repairs\([10](https://arxiv.org/html/2608.17341#bib.bib16)\)\. We build on the setting introduced by[11](https://arxiv.org/html/2608.17341#bib.bib2)and[13](https://arxiv.org/html/2608.17341#bib.bib3), in which planning domains are repaired using positive, or whitelist, traces that must be valid plans and negative, or blacklist, traces that must be rendered inapplicable\. They provide a symbolic algorithm that can find cardinality\-minimal solutions that satisfy the blacklist and whitelist constraints\. The algorithm was evaluated using precision and recall by comparing the predicted repairs with the ground\-truth corrections for PDDL models that had been corrupted through the random addition or removal of action preconditions and effects\.

In this paper, we investigate the ability of recent open\-weight large language models of varying sizes to solve the PDDL domain\-repair problem and compare their performance with the symbolic optimizer described above\. We evaluate all models on the benchmark suite introduced by[13](https://arxiv.org/html/2608.17341#bib.bib3), enabling a direct comparison with the symbolic approach\. Our experimental setting follows the LLM\-only repair formulation introduced under the same name in our earlier work\([8](https://arxiv.org/html/2608.17341#bib.bib11)\), but extends that study by evaluating a broader range of more recent open\-weight models, allowing us to assess the current state of the art in LLM\-based PDDL model repair\. In addition to the domain\-onlyNoTraceprompt used previously, we investigate whether theWithTraceprompt, which also provides the LLM with the whitelist and blacklist traces, yields further performance improvements\.

## Planning Formalism

Since our focus is on repairing lifted PDDL domains, we introduce the lifted planning formalism\. A lifted planning problem is defined as a tupleΠ=\(𝒫,𝒜,α,𝒪,sI,sg\)\\Pi=\(\\mathcal\{P\}\{\},\\mathcal\{A\}\{\},\\alpha,\\mathcal\{O\}\{\},s^\{I\},s^\{g\}\), where the domain is𝒟=\(𝒫,𝒜,α\)\\mathcal\{D\}\{\}=\(\\mathcal\{P\}\{\},\\mathcal\{A\}\{\},\\alpha\)and the task is𝒯=\(𝒪,sI,sg\)\\mathcal\{T\}\{\}=\(\\mathcal\{O\}\{\},s^\{I\},s^\{g\}\)\.

#### Objects, Types, and Variables\.

Let𝒪\\mathcal\{O\}\{\}be the set of objects in the planning task\. We consider a set of variables𝒱\\mathcal\{V\}, each acting as a placeholder for an object\. The type of a variablev∈𝒱v\\in\\mathcal\{V\}is written asv\|tv\|t, wheret∈Θt\\in\\Theta\{\}\. Every typet∈Θt\\in\\Theta\{\}is associated with the set of objects𝒪​⟦t⟧⊆𝒪\\mathcal\{O\}\{\}\\llbracket t\\rrbracket\\subseteq\\mathcal\{O\}\{\}\. We say thatt∈Θt\\in\\Theta\{\}is a*subtype*oft′∈Θt^\{\\prime\}\\in\\Theta\{\}iff𝒪⁡⟦t⟧⊆𝒪⁡⟦t′⟧\\mathcal\{O\}\{\}\\llbracket t\\rrbracket\\subseteq\\mathcal\{O\}\{\}\\llbracket t^\{\\prime\}\\rrbracket\.

#### Predicates\.

A predicate𝐩=P\(v1\|t1,…,vk\|tk\)\\mathbf\{p\}=P\(v\_\{1\}\|t\_\{1\},\\ldots,v\_\{k\}\|t\_\{k\}\)is defined by a unique namePPand a tuple ofk∈ℕ0k\\in\\mathbb\{N\}\_\{0\}typed variables, written aspar\(𝐩\):=\(v1\|t1,…,vk\|tk\)\\operatorname\{par\}\(\\mathbf\{p\}\):=\(v\_\{1\}\|t\_\{1\},\\ldots,v\_\{k\}\|t\_\{k\}\)\. The predicate has zero variables ifk=0k=0\. We denote by𝒫\\mathcal\{P\}\{\}the set of all predicates inΠ\\Pi\.

#### Variable Substitution\.

A variable substitution functionϱ:𝒱→𝒪\\varrho:\\mathcal\{V\}\\rightarrow\\mathcal\{O\}maps each typed variablev\|tv\|tto an objectϱ⁡\(v\|t\)∈𝒪​⟦t⟧\\varrho\(v\|t\)\\in\\mathcal\{O\}\{\}\\llbracket t\\rrbracketof the same typett\.

#### Facts\.

Given a predicate𝐩∈𝒫\\mathbf\{p\}\\in\\mathcal\{P\}and a substitution functionϱ\\varrho, a*fact*is obtained by grounding𝐩\\mathbf\{p\}, that is, by replacing each parameter\(v1,…,vk\)\(v\_\{1\},\\ldots,v\_\{k\}\)with the corresponding objects given byϱ\\varrho:f=ϱ⁡\(𝐩\)=P⁡\(ϱ⁡\(v1\),…,ϱ⁡\(vk\)\)f=\\varrho\(\\mathbf\{p\}\)=P\(\\varrho\(v\_\{1\}\),\\ldots,\\varrho\(v\_\{k\}\)\)\. The set of all grounded predicates is denoted byℱ\\mathcal\{F\}, and any set of facts constitutes a*state*\.

#### Action Schemas\.

Let𝒜\\mathcal\{A\}\{\}denote the set of action schemas\. An action schema𝐚=A\(v1\|t1,…,vk\|tk\)\\mathbf\{a\}=A\(v\_\{1\}\|t\_\{1\},\\ldots,v\_\{k\}\|t\_\{k\}\)is defined by a unique nameAAand a tuple ofkkvariables, written aspar\(𝐚\):=\(v1\|t1,…,vk\|tk\)\\operatorname\{par\}\(\\mathbf\{a\}\):=\(v\_\{1\}\|t\_\{1\},\\ldots,v\_\{k\}\|t\_\{k\}\)\. Each schema is associated with a mapping

α⁡\(𝐚\)=\(𝑝𝑟𝑒𝑐\+​\(𝐚\),𝑝𝑟𝑒𝑐−​\(𝐚\),𝑒𝑓𝑓\+​\(𝐚\),𝑒𝑓𝑓−​\(𝐚\)\)\\alpha\(\\mathbf\{a\}\{\}\)=\(\\mathit\{prec\}^\{\+\}\(\\mathbf\{a\}\{\}\),\\mathit\{prec\}^\{\-\}\(\\mathbf\{a\}\{\}\),\\mathit\{eff\}^\{\+\}\(\\mathbf\{a\}\{\}\),\\mathit\{eff\}^\{\-\}\(\\mathbf\{a\}\{\}\)\)representing a tuple of four sets of compatible predicates, as defined below\.

###### Definition 1\(Compatible Predicates\)\.

For an action schema𝐚\\mathbf\{a\}, the set of*compatible predicates*𝒫𝐚\\mathcal\{P\}^\{\\mathbf\{a\}\}contains all predicates whose set of parameters is a subset of those of𝐚\\mathbf\{a\}:

𝒫𝐚:=\{𝐩∈𝒫∣s​\-​par⁡\(𝐩\)⊆s​\-​par⁡\(𝐚\)\},\\mathcal\{P\}^\{\\mathbf\{a\}\}:=\\\{\\mathbf\{p\}\\in\\mathcal\{P\}\{\}\\mid\\operatorname\{s\\text\{\-\}par\}\(\\mathbf\{p\}\)\\subseteq\\operatorname\{s\\text\{\-\}par\}\(\\mathbf\{a\}\)\\\},wheres​\-​par\\operatorname\{s\\text\{\-\}par\}converts the parameters into a set\.

#### Actions\.

Given an action schema𝐚\\mathbf\{a\}and a substitution functionϱ\\varrho, the corresponding*action*is obtained by replacing each parameter of𝐚\\mathbf\{a\}according toϱ\\varrho, and is denoteda=𝐚⁡\[ϱ\]a=\\mathbf\{a\}\[\\varrho\]\. Actions describe transitions in the state space\. An actionaais*applicable*in a statessiff𝑝𝑟𝑒𝑐\+​\(a\)⊆s\\mathit\{prec\}^\{\+\}\(a\)\\subseteq sand𝑝𝑟𝑒𝑐−​\(a\)∩s=∅\\mathit\{prec\}^\{\-\}\(a\)\\cap s=\\emptyset\. Applying an applicable actionaainssproduces the successor state

s′=\(s∖𝑒𝑓𝑓−​\(a\)\)∪𝑒𝑓𝑓\+​\(a\),s^\{\\prime\}=\(s\\setminus\\mathit\{eff\}^\{\-\}\(a\)\)\\cup\\mathit\{eff\}^\{\+\}\(a\),which we denote bys→as′s\\to\_\{a\}s^\{\\prime\}\.

Throughout this paper, we use boldface \(e\.g\.,𝐩\\mathbf\{p\},𝐚\\mathbf\{a\}\) for predicates and action schemas, and regular typeface \(e\.g\.,ff,aa\) for facts and actions\.

#### Solutions\.

Letγ=⟨a1,…,ak⟩\\gamma=\\langle a\_\{1\},\\ldots,a\_\{k\}\\ranglebe an action sequence\. We writes→∗γs′s\\rightarrow^\{\*\}\_\{\\gamma\}s^\{\\prime\}to denote thats′s^\{\\prime\}results from applyingγ\\gammatossvia a state trajectory⟨s0,…,sk⟩\\langle s\_\{0\},\\ldots,s\_\{k\}\\ranglewheres0=ss\_\{0\}=s,sk=s′s\_\{k\}=s^\{\\prime\}, and each action is applicable in its preceding state\. A solution to a planning problem is an action sequenceγ=⟨a1,…,ak⟩\\gamma=\\langle a\_\{1\},\\ldots,a\_\{k\}\\ranglesuch thatsI→∗γs′s\_\{I\}\\to^\{\*\}\_\{\\gamma\}s^\{\\prime\}for somes′s^\{\\prime\}withsg⊆s′s^\{g\}\\subseteq s^\{\\prime\}, and eachaia\_\{i\}is a grounding of some action schema𝐚∈𝒜\\mathbf\{a\}\\in\\mathcal\{A\}\{\}\.

## The Repair Problem

We begin by introducing the notation and syntax used to define possible repair operations for a given planning domain\. Next, we describe how a set of such repairs can be applied to produce a modified domain\. Based on these concepts, we then formalize the domain repair problem in terms of the defined repair operations and a set of positive and negative plans\. Our formulation follows the setting introduced by[13](https://arxiv.org/html/2608.17341#bib.bib3)as closely as possible, while explicitly highlighting the modifications needed to incorporate LLMs for prioritizing semantically meaningful repair sets\.

Variation 1:NoTrace\(Domain Only\)\.1\.Infer the intended semantics of each action in the flawed PDDL domain\.2\.Propose plausible single\-edit repairs: add or remove a positive or negative precondition or effect\.3\.Return a concise action overview, the recommended repairs grouped by action, and a justification for each repair\.Constraints\.•Use semantic cues from action and predicate names\.•Introduce no new predicates, variables, or constants; added atoms use arguments available to the action\.Input:a flawed PDDL domain\.Variation 2:WithTrace\(Domain and Test Traces\)\.•Follow allNoTracesteps and constraints\.•Also use test traces as behavioral evidence\.•Positive \(whitelist\) plans must execute and reach the goal\.•For a negative \(blacklist\) plan with failure indexnn, its prefix must be executable, and actionnnmust be inapplicable\.Input:a flawed PDDL domain and positive/negative test traces\.

Figure 1:TheNoTraceandWithTraceLLM\-only prompt abstractions used in our evaluation\.In a planning domain𝒟=\(𝒫,𝒜,α\)\\mathcal\{D\}=\(\\mathcal\{P\},\\mathcal\{A\},\\alpha\), an*atomic repair*is a modification denoted byr⟦𝐚,𝐩,c,op⟧r\\llbracket\\mathbf\{\\mathbf\{a\}\},\\mathbf\{\\mathbf\{p\}\},c,op\\rrbracket\. Here,𝐚∈𝒜\\mathbf\{a\}\\in\\mathcal\{A\}is an action schema,𝐩∈𝒫\\mathbf\{p\}\\in\\mathcal\{P\}is a predicate compatible with𝐚\\mathbf\{a\},c∈\{prec\+,prec−,eff\+,eff−\}c\\in\\\{\\mathrm\{prec\}^\{\+\},\\mathrm\{prec\}^\{\-\},\\mathrm\{eff\}^\{\+\},\\mathrm\{eff\}^\{\-\}\\\}indicates whether the change concerns a positive or negative precondition or effect, ando​p∈\{\+,−\}op\\in\\\{\+,\-\\\}specifies whether the component is added or removed\. We write𝒟⇒r𝒟′\\mathcal\{D\}\\Rightarrow\_\{r\}\\mathcal\{D\}^\{\\prime\}to indicate that applyingrrto𝒟\\mathcal\{D\}yields𝒟′=\(𝒫,𝒜,α′\)\\mathcal\{D\}^\{\\prime\}=\(\\mathcal\{P\},\\mathcal\{A\},\\alpha^\{\\prime\}\), whereα′\\alpha^\{\\prime\}results from applyingrrtoα\\alpha\.

A*repair set*δ\\deltafor a domain is a finite collection of zero or more atomic repairs\. We say thatδ\\deltais*valid*if and only if it contains no two repairsr,r′∈δr,r^\{\\prime\}\\in\\deltasuch that one reverses the effect of the other\. Specifically, two repairsr=r⟦𝐚,𝐩,c,op⟧r=r\\llbracket\\mathbf\{\\mathbf\{a\}\},\\mathbf\{\\mathbf\{p\}\},c,op\\rrbracketandr′=r′⟦𝐚′,𝐩′,c′,op′⟧r^\{\\prime\}=r^\{\\prime\}\\llbracket\\mathbf\{a^\{\\prime\}\},\\mathbf\{p^\{\\prime\}\},c^\{\\prime\},op^\{\\prime\}\\rrbracketare considered to undo each other if𝐚=𝐚′\\mathbf\{a\}=\\mathbf\{a\}^\{\\prime\},𝐩=𝐩′\\mathbf\{p\}=\\mathbf\{p\}^\{\\prime\},c=c′c=c^\{\\prime\}, ando​p≠o​p′op\\neq op^\{\\prime\}\.

Let𝒟\\mathcal\{D\}be a domain andδ\\deltaa valid repair set for𝒟\\mathcal\{D\}\. Applying the repairs inδ\\deltain any order yields the same modified domain𝒟′\\mathcal\{D\}^\{\\prime\}\. We use𝒟⇒δ∗𝒟′\\mathcal\{D\}\\Rightarrow\_\{\\delta\}^\{\*\}\\mathcal\{D\}^\{\\prime\}to indicate that𝒟′\\mathcal\{D\}^\{\\prime\}is obtained from𝒟\\mathcal\{D\}by applying the valid repair setδ\\delta\.

###### Definition 2\(Domain Repair Problem\)\.

The domain repair problem is defined as a pairℛ=\(𝒟,𝕋\)\\mathcal\{R\}=\(\\mathcal\{D\}\{\},\\mathbb\{T\}\), where𝒟\\mathcal\{D\}\{\}denotes a planning domain and𝕋=\{𝐓1,…,𝐓n\}\\mathbb\{T\}=\\\{\\mathbf\{T\}\_\{1\},\\dots,\\mathbf\{T\}\_\{n\}\\\}for somen∈ℕn\\in\\mathbb\{N\}\. Each element𝐓i\\mathbf\{T\}\_\{i\}is a triple\(Πi,ℙi,𝔼i\)\(\\Pi\_\{i\},\\mathbb\{P\}\_\{i\},\\mathbb\{E\}\_\{i\}\)\. Here,Πi=\(𝒟,𝒯i\)\\Pi\_\{i\}=\(\\mathcal\{D\}\{\},\\mathcal\{T\}\_\{i\}\)denotes the planning problem;ℙi\\mathbb\{P\}\_\{i\}is a finite, nonempty set of positive plansπk\+\\pi^\{\+\}\_\{k\}forΠi\\Pi\_\{i\}; and𝔼i\\mathbb\{E\}\_\{i\}is a finite, nonempty set of pairs\(πk−,ik\)\(\\pi\_\{k\}^\{\-\},i\_\{k\}\)associated withΠi\\Pi\_\{i\}\. Eachπk−\\pi^\{\-\}\_\{k\}denotes a negative plan \(a sequence of actions considered undesirable\) forΠi\\Pi\_\{i\}, andiki\_\{k\}is at most the length of that plan\.

###### Definition 3\(Solution to the Repair Problem\)\.

A*solution*toℛ\\mathcal\{R\}is a valid repair setδ\\deltathat transforms the original domain𝒟\\mathcal\{D\}into a modified domain𝒟′\\mathcal\{D\}^\{\\prime\}through the sequence of repair operations𝒟⇒δ∗𝒟′\\mathcal\{D\}\\Rightarrow\_\{\\delta\}^\{\*\}\\mathcal\{D\}^\{\\prime\}\. This repair must satisfy the following conditions: for every indexiiwith1≤i≤n1\\leq i\\leq n, all positive plansπ\+∈ℙi\\pi^\{\+\}\\in\\mathbb\{P\}\_\{i\}must be valid solutions to the updated planning problemΠi′=\(𝒟′,𝒯i\)\\Pi\_\{i\}^\{\\prime\}=\(\\mathcal\{D\}^\{\\prime\},\\mathcal\{T\}\_\{i\}\), meaning that they are executable and achieve the goal; and for each pair\(πk−,ik\)∈𝔼i\(\\pi\_\{k\}^\{\-\},i\_\{k\}\)\\in\\mathbb\{E\}\_\{i\},πk−\\pi\_\{k\}^\{\-\}must not be a valid plan forΠi′\\Pi\_\{i\}^\{\\prime\}, with the action at positioniki\_\{k\}being the first that cannot be applied\.

## Solving the Repair Problem

In this section, we briefly review our symbolic baseline and then introduce our LLM\-only approach\.

### The Baseline Approach

The symbolic baseline\([11](https://arxiv.org/html/2608.17341#bib.bib2);[13](https://arxiv.org/html/2608.17341#bib.bib3)\)uses a sound algorithm based on conditional hitting sets to solve the domain repair problem\. The authors report runtime and evaluate precision and recall against known ground\-truth repairs\. To obtain the ground truth, they perturb IPC domains by randomly adding or removing preconditions and effects, which allows precision and recall to be computed directly\.

The algorithm executes all whitelist and blacklist test plans\. For each failing test, it identifies the possible repairs that could resolve the failure and encodes them as a conditional hitting\-set problem\. The solver then selects a minimal repair set, applies it to the domain, and re\-evaluates the tests under the modified model\. If additional tests fail, the newly identified repair requirements are incorporated and the process continues iteratively until a minimal repair set satisfying all tests is found\.

The baseline does not exploit semantic cues encoded by the modeler in predicate, action, or domain names within the PDDL file\. Its hitting\-set solver optimizes only the size of the repair set; when multiple diagnoses share the same cardinality, it returns an arbitrary one\. Moreover, the ground\-truth repair need not be cardinality\-minimal, so it can be missed under this objective\.

Table 1:The top table shows theNoTraceand the bottom one shows theWithTraceLLM\-only results\. Each row reports the metrics averaged over the 12 error\-injected IPC domains described in the text\. Parameter counts are given in billions\. Pr, Re, andF1F\_\{1\}denote precision, recall, and their harmonic mean, respectively; TR is the average test pass rate, i\.e\., the fraction of tests satisfied by the repairs\. The grayed\-out first row of the top table gives the symbolic\-only baseline, whose TR is 1 by construction and is marked1∗1^\{\\ast\}\.*Default Effort*uses the model’s default reasoning configuration, while*High Effort*uses the enhanced configuration specified in that column\.*Unspec\.*means default reasoning enablement is not advertised\. A “\-” in an effort column means no alternative configuration is supported; elsewhere, it denotes an inapplicable result\.
### The LLM\-Only Approach

In our previous work\([8](https://arxiv.org/html/2608.17341#bib.bib11)\), we introduced several ideas on how LLMs can be exploited to improve purely symbolic repair approaches, including the LLM\-only approach studied here\. In this approach, an LLM directly predicts a repair set without symbolic post\-processing\. In that study, we evaluated only a single LLM, limiting the scope of the study\. Here, we extend the results for that approach using more recent LLMs\. The present study evaluates whether LLM reasoning can usefully complement symbolic repair\. It also examines whether current LLMs are strong enough for the LLM\-only approach to produce repairs of sufficient overall quality\.

Specifically, we evaluate the two prompts summarized in Figure[1](https://arxiv.org/html/2608.17341#Sx3.F1)\. TheNoTraceprompt supplies only the corrupted domain and asks the LLM to infer intended preconditions and effects from the action and predicate names\. TheWithTraceprompt additionally supplies the positive and negative test traces and instructs the LLM to choose repairs that make every positive trace executable and goal\-achieving and every negative trace fail at its designated action\.

The LLM\-onlyWithTracesetting directly addresses the repair problem because the LLM has access to the test constraints, although its output may still be noisy or unsound\. In contrast,NoTracedoes not have access to the tests and therefore addresses a relaxed variant of the problem in which no test constraints must be satisfied\. Consequently, multiple semantically plausible repairs may be proposed for any action without regard to whether they solve the actual repair instance\.

## Experiments

We evaluate our LLM\-only approach on the benchmark suite introduced by[13](https://arxiv.org/html/2608.17341#bib.bib3)and later used in our previous study\([8](https://arxiv.org/html/2608.17341#bib.bib11)\), enabling comparison under the same precision, recall, andF1F\_\{1\}metrics\. For a ground\-truth repair setGGand predicted repair setPP, precision is\|P∩G\|/\|P\|\|P\\cap G\|/\|P\|, recall is\|P∩G\|/\|G\|\|P\\cap G\|/\|G\|, andF1F\_\{1\}is their harmonic mean, with all three scores defined as00whenPPis empty\. We additionally report the test pass rate \(TR\), the fraction of tests satisfied by the predicted repair\. TR is particularly important because a solution to the domain repair problem must satisfy every test: highF1F\_\{1\}without near\-perfect TR does not provide the required correctness and is therefore insufficient for a reliable stand\-alone solver\. Each domain represents one repair problem, for which we perform one experimental run; all LLM calls were made through OpenRouter111https://openrouter\.ai/\. If a call fails or its response cannot be parsed as a repair set, we repeat the same request up to three additional times, and an instance that still has no valid response is treated as an empty prediction\. These retries recover failed calls rather than constituting additional experimental runs\. The AVG and STD rows report, respectively, the unweighted mean and standard deviation over the per\-domain results, so the reported averages are macro averages\.

We omit the older versions of the Logistics and Woodworking domains \(LOGISTICS98 and WOODWORKING08\), retaining LOGISTICS00 and WOODWORKING11\. We also omit MPRIME because that domain is designed to use misleading names, making it unsuitable for evaluating an LLM’s ability to exploit semantic cues\. Because the symbolic method is stochastic, we repeat in Table[1](https://arxiv.org/html/2608.17341#Sx4.T1)the five\-run average published by[13](https://arxiv.org/html/2608.17341#bib.bib3)instead of rerunning it\.

Table[1](https://arxiv.org/html/2608.17341#Sx4.T1)presents our main results for the recent models\. We repeat the symbolic result in its first row only to provide a common point of reference\. Compared with theNoTraceprompt used for the earlier runs reported in our previous work\([8](https://arxiv.org/html/2608.17341#bib.bib11)\), theNoTraceprompt in these experiments is shorter: it neither requests an explicit reasoning trace nor provides a one\-shot example\. We made this change to keep the task and its context simpler for smaller models, whose performance is a central focus of this evaluation\. TheWithTraceprompt minimally extendsNoTraceby supplying the whitelist and blacklist test traces and asking the LLM to satisfy all tests\.

Table 2:AverageF1F\_\{1\}and TR for each trace\-inclusion and reasoning\-effort setting, with the corresponding best\-performing model\. These metrics are reported as in Table[1](https://arxiv.org/html/2608.17341#Sx4.T1); see its caption for their definitions\. DSV4\-P: DeepSeek V4 Pro; GLM\-5\.2: GLM 5\.2\. For High Effort withWithTrace, GLM\-5\.2 matches DSV4\-P’s meanF1F\_\{1\}, but DSV4\-P is shown because itsF1F\_\{1\}standard deviation is lower \(\.17 versus \.18\)\. Per\-domain breakdowns are reported in Tables[3](https://arxiv.org/html/2608.17341#Sx8.T3),[4](https://arxiv.org/html/2608.17341#Sx8.T4),[5](https://arxiv.org/html/2608.17341#Sx8.T5), and[6](https://arxiv.org/html/2608.17341#Sx8.T6)\.Our previous work reports a meanF1F\_\{1\}of\.46\.46for GPT\-4o with the originalNoTraceprompt\([8](https://arxiv.org/html/2608.17341#bib.bib11)\), compared with\.38\.38using the shorter prompt in Table[1](https://arxiv.org/html/2608.17341#Sx4.T1)\. With test traces excluded and default reasoning, GLM 5\.2 achieves the highest meanF1F\_\{1\}of\.85\.85\. The release dates in Table[1](https://arxiv.org/html/2608.17341#Sx4.T1)also show a pronounced generational improvement: the best 2025 model reachesF1=\.53F\_\{1\}=\.53, whereas the best 2026 model reachesF1=\.85F\_\{1\}=\.85\.

Table[2](https://arxiv.org/html/2608.17341#Sx5.T2)summarizes the interaction between trace inclusion and reasoning effort\. Supplying the test traces does not improve the best result at default effort: GLM 5\.2 remains the best model, but its meanF1F\_\{1\}falls from\.85\.85to\.78\.78\. Reasoning jointly over many potentially long plans is combinatorial and can also consume or exceed the LLM’s context window, which may explain why the additional evidence is not consistently useful\. The per\-domain results support this interpretation: Table[3](https://arxiv.org/html/2608.17341#Sx8.T3)contains a numeric result for every model\-domain pair, whereas Table[4](https://arxiv.org/html/2608.17341#Sx8.T4)contains both context\-limit errors and invalid or unparsable outputs\.

Higher reasoning effort helps several models, but not all of them\. Without test traces, Table[1](https://arxiv.org/html/2608.17341#Sx4.T1)shows that GLM 5\.2 improves fromF1=\.85F\_\{1\}=\.85to\.87\.87\. The increase is larger for Mistral Small 4, from\.13\.13to\.41\.41\. Nevertheless, Table[2](https://arxiv.org/html/2608.17341#Sx5.T2)shows that providing test traces at high effort yields a best meanF1F\_\{1\}of only\.82\.82, shared by DeepSeek V4 Pro and GLM 5\.2\. This is below the best default\-effortNoTraceresult of\.85\.85\.

Test satisfaction exposes a more consequential limitation\. The strongest LLM result, GLM 5\.2 with high effort andNoTrace, improves substantially on the symbolic baseline inF1F\_\{1\}\(\.87\.87versus\.49\.49\), but its mean TR is only\.82\.82\. Its TR falls to\.06\.06on Thoughtful, meaning that only6%6\\%of that domain’s tests are satisfied\. Thus, high repair\-set overlap does not provide the strict correctness guarantee required by the repair problem\.

Reasoning over the test traces does help with their combinatorial constraints in one important case: for DeepSeek V4 Pro withWithTrace, increasing the reasoning effort raises mean TR from\.82\.82to\.92\.92while also raisingF1F\_\{1\}from\.74\.74to\.82\.82\. Nevertheless,\.92\.92still falls short of complete test satisfaction, and TR again drops to\.06\.06on Thoughtful\. Recent open\-weight models therefore improve semantic repair quality without guaranteeing that the predicted repair is a solution\. These findings are supported only by our error\-injected IPC benchmark, which is publicly available online222https://zenodo\.org/records/14533200\([12](https://arxiv.org/html/2608.17341#bib.bib1)\)\. The models may therefore have encountered the underlying domains or some of their fixes during training and could be recalling them rather than deriving every repair from the supplied instance\. This possible benchmark contamination limits how strongly the observedF1F\_\{1\}and TR gains can be expected to generalize\.

We argue for hybrid methods that use an LLM to rank or filter semantically promising candidates while a symbolic component provides theoretical test\-satisfaction guarantees, as proposed in our previous work\([8](https://arxiv.org/html/2608.17341#bib.bib11)\); realizing such a method remains future work\.

## Conclusion & Future Work

Our results convey two main messages\. First, recent open\-weight LLMs can use semantic cues to identify repairs that agree much more closely with the ground truth than the symbolic baseline: the best meanF1F\_\{1\}is\.87\.87, compared with\.49\.49\. Reasoning effort can strengthen this ability, but its effect is model\-dependent, and supplying test traces does not improve the bestF1F\_\{1\}\. Second, repair\-set overlap is not a correctness guarantee\. The setting with the bestF1F\_\{1\}has a mean TR of\.82\.82and a TR of only\.06\.06on Thoughtful; even the strongest trace\-aware setting reaches a mean TR of\.92\.92and again only\.06\.06on Thoughtful\. Current LLM\-only methods therefore improve semantic repair quality but cannot reliably solve the repair problem, as every test must pass\.

A promising direction is consequently to combine the complementary strengths of both approaches: an LLM can express semantic repair preferences, while a symbolic reasoner preserves theoretical test\-satisfaction guarantees\([8](https://arxiv.org/html/2608.17341#bib.bib11)\)\. We will also isolate the effects of prompt length, explicit reasoning requests, and examples\. Finally, because the evaluated IPC\-derived benchmark is public, possible training\-data recall limits the generality of our results; evaluation on novel, unpublished domains is needed to establish whether these gains transfer to genuinely unseen repair problems\.

## Acknowledgments

Pascal Bercher is the recipient of an Australian Research Council \(ARC\) Discovery Early Career Researcher Award \(DECRA\), project number DE240101245, funded by the Australian Government\.

## References

- Ainetoet al\.\(2018\)D\. Aineto, S\. Jiménez, and E\. OnaindiaLearning STRIPS action models with classical planning\.InProc\. of the 28th ICAPS,pp\. 399–407\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p4.1)\.
- Bercheret al\.\(2025\)P\. Bercher, S\. Sreedharan, and M\. VallatiA survey on model repair in AI planning\.InProc\. of the 34th IJCAI,pp\. 10371–10380\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p4.1)\.
- Beriachvili and Bit\-Monnot \(2026\)N\. Beriachvili and A\. Bit\-MonnotA constraint formulation for domain repair with ground or lifted test plans\.InProc\. of the 36th ICAPS,pp\. 20–28\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p4.1)\.
- Grageraet al\.\(2025\)A\. Gragera, R\. Fuentetaja, Á\. G\. Olaya, and F\. FernándezOn the gains from using action observations in domain repair\.InProc\. of the 35th ICAPS,pp\. 343–347\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p4.1)\.
- Guanet al\.\(2023\)L\. Guan, K\. Valmeekam, S\. Sreedharan, and S\. KambhampatiLeveraging pre\-trained large language models to construct and utilize world models for model\-based task planning\.InProc\. of the 37th NeurIPS,Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p2.1)\.
- Huanget al\.\(2025a\)S\. Huang, T\. Cohn, and N\. LipovetzkyChasing progress, not perfection: revisiting strategies for end\-to\-end LLM plan generation\.InProc\. of the 35th ICAPS,pp\. 204–212\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p2.1)\.
- Huanget al\.\(2025b\)S\. Huang, N\. Lipovetzky, and T\. CohnPlanning in the dark: llm\-symbolic planning pipeline without experts\.InProc\. of the 39th AAAI,pp\. 26542–26550\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p2.1)\.
- Karimi Bavandpour and Bercher \(2025\)N\. Karimi Bavandpour and P\. BercherFinding semantically guided repairs in PDDL domains using LLMs\.InICAPS 2025 Workshop on Human–Aware and Explainable Planning \(HAXP\),External Links:[Link](https://openreview.net/forum?id=Zj8UqVxClT)Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p5.1),[The LLM\-Only Approach](https://arxiv.org/html/2608.17341#Sx4.SSx2.p1.1),[Experiments](https://arxiv.org/html/2608.17341#Sx5.p1.1),[Experiments](https://arxiv.org/html/2608.17341#Sx5.p3.1),[Experiments](https://arxiv.org/html/2608.17341#Sx5.p4.1),[Experiments](https://arxiv.org/html/2608.17341#Sx5.p9.1),[Conclusion & Future Work](https://arxiv.org/html/2608.17341#Sx6.p2.1)\.
- Katzet al\.\(2025\)M\. Katz, H\. Kokel, C\. Muise, S\. Sohrabi, and S\. SreedharanMake planning research rigorous again\!\.CoRRabs/2505\.21674\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p2.1)\.
- Lin and Bercher \(2021\)S\. Lin and P\. BercherChange the world – how hard can that be? on the computational complexity of fixing planning models\.InProc\. of the 30th IJCAI,pp\. 4152–4159\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p4.1)\.
- Linet al\.\(2023\)S\. Lin, A\. Grastien, and P\. BercherTowards automated modeling assistance: an efficient approach for repairing flawed planning domains\.InProc\. of the 37th AAAI,pp\. 12022–12031\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p4.1),[The Baseline Approach](https://arxiv.org/html/2608.17341#Sx4.SSx1.p1.1)\.
- Linet al\.\(2024\)S\. Lin, A\. Grastien, R\. Shome, and P\. BercherExperimental results for the aaai 2025 paper: “told you that will not work: optimal corrections to planning domains using counter\-example plans”\.Zenodo\.Note:https://zenodo\.org/records/14533200External Links:[Document](https://dx.doi.org/10.5281/ZENODO.14533200)Cited by:[footnote 2](https://arxiv.org/html/2608.17341#footnote2)\.
- Linet al\.\(2025\)S\. Lin, A\. Grastien, R\. Shome, and P\. BercherTold you that will not work: optimal corrections to planning domains using counter\-example plans\.InProc\. of the 39th AAAI,pp\. 26596–26604\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p4.1),[Introduction](https://arxiv.org/html/2608.17341#Sx1.p5.1),[The Repair Problem](https://arxiv.org/html/2608.17341#Sx3.p1.1),[The Baseline Approach](https://arxiv.org/html/2608.17341#Sx4.SSx1.p1.1),[Experiments](https://arxiv.org/html/2608.17341#Sx5.p1.1),[Experiments](https://arxiv.org/html/2608.17341#Sx5.p2.1)\.
- Miller \(2019\)T\. MillerExplanation in artificial intelligence: insights from the social sciences\.AIJ267,pp\. 1–38\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p3.1)\.
- Oswaldet al\.\(2024\)J\. T\. Oswald, K\. Srinivas, H\. Kokel, J\. Lee, M\. Katz, and S\. SohrabiLarge language models as planning domain generators\.InProc\. of the 34th ICAPS,pp\. 423–431\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p2.1)\.
- Tantakounet al\.\(2025\)M\. Tantakoun, C\. Muise, and X\. ZhuLLMs as planning formalizers: a survey for leveraging large language models to construct automated planning models\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 25167–25188\.Cited by:[Introduction](https://arxiv.org/html/2608.17341#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.17341#Sx1.p2.1)\.

## Appendix: Per\-Domain Results

Tables[3](https://arxiv.org/html/2608.17341#Sx8.T3),[4](https://arxiv.org/html/2608.17341#Sx8.T4),[5](https://arxiv.org/html/2608.17341#Sx8.T5), and[6](https://arxiv.org/html/2608.17341#Sx8.T6)provide the per\-domain results for open\-weight models: DSV4\-F \(DeepSeek V4 Flash\), DSV4\-P \(DeepSeek V4 Pro\), GLM\-5\.2 \(GLM 5\.2\), OSS20B \(gpt\-oss\-20B\), OSS120B \(gpt\-oss\-120B\), Minst\-14B \(Ministral 3 14B\), MistS\-4 \(Mistral Small 4\), MistM\-3\.5 \(Mistral Medium 3\.5\), MistL\-3 \(Mistral Large 3\), Nem3\-S \(NVIDIA Nemotron 3 Super\), Nem3\-U \(NVIDIA Nemotron 3 Ultra\), Phi4 \(Phi\-4\), Qwen3\-30B \(Qwen3\-30B\-A3B\), Qwen3\-32B\. Best values per row are bold; the final two rows report the mean and standard deviation across domains\.

Table 3:Per\-domain results using theNoTraceprompt in Figure[1](https://arxiv.org/html/2608.17341#Sx3.F1); test traces are not sent to the LLM\. The default effort setting is used for each LLM\. TR is the test pass rate, i\.e\., the fraction of tests satisfied by the repair; the AVG row averages this rate across domains\. For each domain and the AVG row, the best precision, recall,F1F\_\{1\}, and TR values across all models are shown in bold\. The upper and lower parts of the table contain models released in 2025 and 2026, respectively\.
Table 4:Per\-domain results using theWithTraceprompt in Figure[1](https://arxiv.org/html/2608.17341#Sx3.F1); test traces are sent to the LLM\. The default effort setting is used for each LLM\. TR is the test pass rate, i\.e\., the fraction of tests satisfied by the repair; the AVG row averages this rate across domains\. For each domain and the AVG row, the best precision, recall,F1F\_\{1\}, and TR values across all models are shown in bold\. The upper part of the table contains models released in 2025, and the lower part contains models released in 2026\. Missing values are marked ase1e\_\{1\}for context\-limit errors,e2e\_\{2\}for other API\-call errors, ande3e\_\{3\}for invalid or unparsable output\. Missing data are treated as 0 in our calculations\.Table 5:Per\-domain results using theNoTraceprompt in Figure[1](https://arxiv.org/html/2608.17341#Sx3.F1); test traces are not sent to the LLM\. The high\-effort setting is used for each LLM when available \(see Table[1](https://arxiv.org/html/2608.17341#Sx4.T1)\); models without this option are omitted\. TR is the test pass rate, i\.e\., the fraction of tests satisfied by the repair; the AVG row averages this rate across domains\. For each domain and the AVG row, the best precision, recall,F1F\_\{1\}, and TR values across all models are shown in bold\. The upper part of the table contains models released in 2025, and the lower part contains models released in 2026\. Missing values are marked ase1e\_\{1\}for context\-limit errors,e2e\_\{2\}for other API\-call errors, ande3e\_\{3\}for invalid or unparsable output\. Missing data are treated as 0 in our calculations\.Table 6:Per\-domain results using theWithTraceprompt in Figure[1](https://arxiv.org/html/2608.17341#Sx3.F1); test traces are sent to the LLM\. The high\-effort setting is used for each LLM when available \(see Table[1](https://arxiv.org/html/2608.17341#Sx4.T1)\); models without this option are omitted\. TR is the test pass rate, i\.e\., the fraction of tests satisfied by the repair; the AVG row averages this rate across domains\. For each domain and the AVG row, the best precision, recall,F1F\_\{1\}, and TR values across all models are shown in bold\. The upper part of the table contains models released in 2025, and the lower part contains models released in 2026\. Missing values are marked ase1e\_\{1\}for context\-limit errors,e2e\_\{2\}for other API\-call errors, ande3e\_\{3\}for invalid or unparsable output\. Missing data are treated as 0 in our calculations\.