Testing the Planning Claims of LLM-Based Agents against Formal Semantics
Summary
该论文通过隐藏的PDDL形式语义对LLM智能体的规划能力进行了严格验证，发现最佳配置离线可解78%的任务但在线仅40%，且复杂提示策略在无明显收益的情况下多消耗5至22倍token，质疑了现有LLM智能体规划能力的声明。
arXiv:2610.22469v1 Announce Type: new
Abstract: LLM-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain-of-thought, search-based prompting, self-refinement, reflection, critics and tool-using loops. Yet these claims are usually supported by end-to-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers. We test LLM planning against formal semantics. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all. Across five domains (two of them new, all released with the paper), four models, six prompting strategies and three agent architectures, the best configuration solves 78% of the solvable tasks offline but only 40% online, where agents mostly act without inspecting the state. Unsolvable tasks are recognised in under 10% of episodes, and heavier prompting strategies use 5 to 22 times as many tokens as direct prompting with no clear gain. In our experiments, planning performance varied more across models than across prompting strategies or agent architectures. Claims that agents can plan require evidence from validated action traces.
View Cached Full Text
Cached at: 10/19/26, 07:03 AM
# Testing the Planning Claims of LLM-Based Agents against Formal Semantics Source: [https://arxiv.org/html/2610.22469](https://arxiv.org/html/2610.22469) \\tl\_set:Ne\\pddlfile pddlfile\\tl\_set:Ne\\brieffilebrieffile\\tl\_set:Ne\\plainfileplainfile\\tl\_set:Ne\\toolcalltoolcall\\tl\_set:Ne\\promptfilepromptfile\\tl\_set:Ne\\msgfilemsgfile Oriol Miro\-Lopez\-Feliu,Oscar Miró López\-FeliuAffiliation:University of Amsterdam,Amsterdam,The NetherlandsAffiliation:Safe AI Netherlands,The Netherlandsemail:[oscar\.miro\.lopez\.feliu@student\.uva\.nl](mailto:[email protected]),Victor Gimenez\-AbalosAffiliation:Barcelona Supercomputing Center,Universitat Politècnica de Catalunya,Barcelona,Spainemail:[victor\.gimenez@bsc\.es](mailto:[email protected]),Javier Vazquez\-SalcedaAffiliation:Universitat Politècnica de Catalunya,Barcelona,Spainemail:[jvazquez@cs\.upc\.edu](mailto:[email protected]),Victor Carballo ArarunaAffiliation:Barcelona Supercomputing Center,Barcelona,Spainemail:[victor\.carballo@bsc\.es](mailto:[email protected])andSergio Alvarez\-NapagaoAffiliation:Barcelona Supercomputing Center,Universitat Politècnica de Catalunya,Barcelona,Spainemail:[sergio\.alvarez@bsc\.es](mailto:[email protected]) © none ###### Abstract\. LLM\-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions\. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain\-of\-thought, search\-based prompting, self\-refinement, reflection, critics and tool\-using loops\. Yet these claims are usually supported by end\-to\-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers\. We test LLM planning against formal semantics\. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all\. Across five domains \(two of them new, all released with the paper\), four models, six prompting strategies and three agent architectures, the best configuration solves 78% of the solvable tasks offline but only 40% online, where agents mostly act without inspecting the state\. Unsolvable tasks are recognised in under 10% of episodes, and heavier prompting strategies use 5 to 22 times as many tokens as direct prompting with no clear gain\. In our experiments, planning performance varied more across models than across prompting strategies or agent architectures\. Claims that agents can plan require evidence from validated action traces\. ###### Keywords: LLM\-based agents; practical reasoning; planning and acting; plan validation; PDDL; epistemic actions; infeasibility; agent evaluation ## 1\.Introduction Large language models \(LLMs\) are increasingly deployed as the core of*LLM agents*in real applications\. Without changing the model’s parameters, developers embed it in two kinds of structure\.*Prompting strategies*elicit intermediate reasoning\([Wei et al\., 2022](https://arxiv.org/html/2610.22469#bib.bib33)\), critique and revise drafts, or search over candidate continuations;*agent architectures*run control loops that call tools and act on an environment\. In the vocabulary of practical reasoning\([Bratman, 1987](https://arxiv.org/html/2610.22469#bib.bib18);[Wooldridge, 2009](https://arxiv.org/html/2610.22469#bib.bib19)\), these strategies and architectures are credited with deliberation, means–ends reasoning, and the monitoring and repair of the agent’s own course of action\. In ReAct, reasoning traces “help the model induce, track, and update action plans”\([Yao et al\., 2023b](https://arxiv.org/html/2610.22469#bib.bib39)\)\. Tree of Thoughts is described as enabling “deliberate decision making” on planning and search tasks\([Yao et al\., 2023a](https://arxiv.org/html/2610.22469#bib.bib38)\)\. Self\-refinement, reflection and tool\-based critics are presented as forms of self\-monitoring and correction\([Madaan et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib37);[Shinn et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib40);[Gou et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib41)\)\. LLM agent architectures for planning and agentic workflows are now also studied within the agents community\([Choi et al\., 2026](https://arxiv.org/html/2610.22469#bib.bib8);[Hsiao et al\., 2026](https://arxiv.org/html/2610.22469#bib.bib9)\), alongside calls to ground agentic AI in established agent concepts\([Dignum and Dignum, 2025](https://arxiv.org/html/2610.22469#bib.bib6)\)\. Such claims concern properties of behaviour: that an agent respects constraints at every step, reaches its goals over long horizons, corrects its own errors, and recognises when a goal cannot be achieved\. The evidence offered for them is usually an end\-to\-end success rate\([Liu et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib44);[Zhou et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib45);[Mialon et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib48)\), a judgement by another LLM\([Zheng et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib53)\), or the persuasiveness of a generated rationale, which need not reflect the computation behind the answer\([Turpin et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib55)\)\. None of these checks the properties being claimed\. A trajectory may reach the goal after an illegal intermediate action\. A run may fail because of a single violated precondition or because it never approached the goal\. An agent may exhaust its budget pursuing a goal that was unachievable from the outset\. Outcome\-only evaluation conflates these cases, and surveys and benchmark studies of agent evaluation note the lack of fine\-grained, diagnostic signals\([Yehudai et al\., 2026](https://arxiv.org/html/2610.22469#bib.bib51);[Ma et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib50)\)\. Automated planning provides the formal criteria for testing these claims\. A plan is valid only if every action is applicable in the state in which it is executed and the final state satisfies the goal\([Ghallab et al\., 2004](https://arxiv.org/html/2610.22469#bib.bib12);[Geffner and Bonet, 2013](https://arxiv.org/html/2610.22469#bib.bib13)\)\. Validity can be checked mechanically\([Howey et al\., 2004](https://arxiv.org/html/2610.22469#bib.bib16)\), and unsolvability can be proved and certified\([Eriksson et al\., 2017](https://arxiv.org/html/2610.22469#bib.bib17)\)\. Agent theory contributes a further distinction, which current usage of “agentic” blurs\. An agent is characterised by the mapping from its history of interaction with an environment to its next action\([Russell and Norvig, 2021](https://arxiv.org/html/2610.22469#bib.bib3);[Wooldridge, 2009](https://arxiv.org/html/2610.22469#bib.bib19)\)\. Many systems described as agentic never extend that history: they reason internally and then emit an answer\. We therefore use*LLM\-based system*as the umbrella term and reserve*agent*for systems that act on an environment, whether by sensing it or by changing it\.*Planning\-by\-acting*, in which a controller interleaves observation, planning and execution\([Ghallab et al\., 2016](https://arxiv.org/html/2610.22469#bib.bib11)\), constitutes a different regime\. Within it, actions that inspect the world \(epistemic actions\) differ from actions that change it \(ontic actions\)\([Bolander and Andersen, 2011](https://arxiv.org/html/2610.22469#bib.bib10)\)\. We test these planning claims against formal semantics\. Each task is posed in natural language but grounded in a hidden PDDL model\([McDermott et al\., 1998](https://arxiv.org/html/2610.22469#bib.bib14)\)\. In the*offline*regime, a system receives the task description and must return a complete plan or declare the goal unachievable\. In the*online*regime, an agent must achieve the goal through a tool interface that separates epistemic from ontic actions, and an inapplicable action ends the episode as a safety violation \(Figure[1](https://arxiv.org/html/2610.22469#S1.F1)\)\. Since an online agent can always retrieve the task description, plan as an offline system would and then execute the plan, the online regime is never harder than the offline one, given sufficient budget \(Proposition[3\.5](https://arxiv.org/html/2610.22469#S3.Thmtheorem5)\)\. Any shortfall of an online agent relative to offline planning with the same model therefore lies in the agent rather than in the environment\. Every plan and every executed action is validated against the hidden model\. This allows us to measure executability, goal achievement, efficiency, infeasibility recognition and sensing behaviour, to locate the first failure in each trace, and to relate these outcomes to token usage and monetary cost\. Validating LLM\-generated plans against PDDL is an established practice\([Valmeekam et al\., 2023a](https://arxiv.org/html/2610.22469#bib.bib22);[Stein et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib28)\)\. What is new here is its use to test the competences claimed for prompting strategies and agent architectures, with unsolvable tasks and an online regime that separates sensing from acting\. We compare six prompting strategies and three agent architectures across four models from two families, in 1,800 validated episodes\. The results provide little evidence that prompting strategies or agent architectures give LLM\-based systems planning competence\. Offline, the best configuration solves 78% of the solvable tasks\. Gains from prompting are model\-specific: chain\-of\-thought and Plan\-and\-Solve raise success by up to 20 points for three of the four models, but no strategy improves on direct prompting for Gemini 2\.0 Flash\-Lite\. No gain survives correction for multiple comparisons\. Tree of Thoughts consumes an order of magnitude more tokens than direct prompting and is Pareto\-dominated\. Online, success declines for every model, to at most 40%\. Agents change the state far more often than they inspect it, and precondition violations terminate most failed episodes; neither reflective memory nor a critic improves on ReAct\. In both regimes, systems rarely recognise that a goal is unachievable, yet they almost never abstain on solvable tasks\. Overall, success varies more across models than across prompting strategies or agent architectures\. ##### Contributions\. 1. \(1\)A formal evaluation framework that operationalises the planning claims made for LLM\-based agents as properties checked against an explicit transition model, separates offline planning from online planning\-by\-acting through the interaction history, and shows that the online regime weakly dominates the offline one \(Section[3](https://arxiv.org/html/2610.22469#S3)\)\. 2. \(2\)An open benchmark and harness: 50 natural\-language tasks backed by hidden PDDL models, in five domains \(two of them new\) and including ten unsolvable tasks; a validation oracle; an agent environment with epistemic and ontic actions under fatal\-violation semantics, which logs every trace with cost accounting \(Section[4](https://arxiv.org/html/2610.22469#S4)\)\. 3. \(3\)A controlled study of six prompting strategies and three agent architectures under identical semantics, with failure\-mode, sensing and cost–performance analyses \(Sections[5](https://arxiv.org/html/2610.22469#S5)and[6](https://arxiv.org/html/2610.22469#S6)\)\. BriefsB\(Π\)B\(\\Pi\)LLM\-Based SystemFull Planπ\\piorUNSOLVABLEEvaluatorSuccessFail / Invalidvalid orcorrect⊥\\botError \(a\) Offline planning LLM\-BasedAgentEnvironmentValidatorState /SimulatorssPrecond\.violationSubmitgoal checkxtx\_\{t\}OnticEpistemicValidObservationoto\_\{t\} \(b\) Online planning\-by\-acting Figure 1\.Evaluation protocols\. \(a\) Offline, the planner maps the natural\-language task to a complete plan orUNSOLVABLE, which is validated once\. \(b\) Online, the agent interacts through tools: epistemic calls return observations without changing the state, ontic calls are validated before the state changes, and an inapplicable action ends the episode\.Two diagrams side by side\. The first shows a pipeline from the natural\-language briefs to an LLM\-based system, which outputs a full plan or UNSOLVABLE; an evaluator then marks it as success or failure\. The second shows an LLM\-based agent issuing calls to an environment\. Epistemic actions go directly to the state and return observations\. Ontic actions go through a validator: valid ones update the state, invalid ones cause a precondition violation\. A submit action triggers the goal check\. ## 2\.Related Work Much of the evidence on whether LLMs can plan comes from classical planning, where every plan can be validated\. PlanBench\([Valmeekam et al\., 2023a](https://arxiv.org/html/2610.22469#bib.bib22)\)and the accompanying critical investigation\([Valmeekam et al\., 2023b](https://arxiv.org/html/2610.22469#bib.bib21)\)established that LLM\-generated plans often look plausible yet fail validation, especially once action and object names are obfuscated\. Follow\-up work found that chain\-of\-thought gains in Blocksworld depend on narrow, problem\-specific prompts\([Stechly et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib25)\), that LLM self\-critique degrades plans while sound external verifiers improve them\([Stechly et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib26)\), and that reasoning models improve markedly but remain unreliable on larger and obfuscated problems\([Valmeekam et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib23)\)\. These findings motivated LLM\-Modulo architectures, in which sound external verifiers carry correctness\([Kambhampati et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib24)\), and hybrids in which LLMs write models or programs for planners\([Liu et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib63);[Guan et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib62);[Silver et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib32)\)\. We study the complementary setting of general\-purpose LLM\-based systems with no planner in the loop, which is where the planning claims above are made\. Subsequent benchmarks keep this formal grounding but change what is measured\. ACPBench\([Kokel et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib29)\)probes atomic reasoning about applicability, progression and reachability rather than plan generation\. Closest to our setting, AutoPlanBench\([Stein et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib28)\)translates PDDL domains into natural language with an LLM, as we do, and compares non\-interactive planning with interactive ReAct\-style action choice, while Göbel et al\.\([Göbel et al\., 2026](https://arxiv.org/html/2610.22469#bib.bib30)\)expose a PDDL simulator to an LLM through tool calls on Blocksworld\. We share their use of formal validation but ask a different question\. In our setting, agents must choose between inspecting the state and changing it, and an inapplicable action is fatal, so that sensing and unsafe commitment become measurable\. Unsolvable tasks further test whether systems recognise that a goal is unachievable, and Proposition[3\.5](https://arxiv.org/html/2610.22469#S3.Thmtheorem5)attributes any online shortfall relative to offline planning to the agent rather than to the environment\. Finally, we compare six prompting strategies and three agent architectures under the same semantics, with cost accounting\. In contrast, interactive benchmarks place LLM agents in web, operating\-system, software and household environments\([Liu et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib44);[Zhou et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib45);[Xie et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib46);[Jimenez et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib47);[Shridhar et al\., 2021](https://arxiv.org/html/2610.22469#bib.bib49);[Mialon et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib48)\)\. Their realism comes with implicit dynamics: success is checked by scripts or end\-state predicates, so failures may stem from perception, interface brittleness or missing knowledge as much as from planning, and success does not certify that intermediate actions were admissible\. Attempts to sharpen the evaluation signal leave this gap open\. AgentBoard\([Ma et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib50)\)adds a progress\-rate metric to expose partial progress\. APB\([Sun et al\., 2026](https://arxiv.org/html/2610.22469#bib.bib31)\)provides a large diagnostic suite that separates planning failures from execution failures and includes unsolvable cases, but it has no transition model against which each action can be validated\. LLM judges, a common fallback, are sensitive to framing and prone to leniency\([Zheng et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib53);[Thakur et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib54)\)\. Our benchmark trades realism for an explicit transition model, which makes every action checkable\. An explicit transition model also lets us examine whether prompting strategies and agent architectures deliver what is claimed for them\. Kapoor et al\.\([Kapoor et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib52)\)observe that complex agents are often dominated by simple baselines and argue that agent evaluations must therefore report cost alongside accuracy; we follow that recommendation\. Studies of individual strategies raise similar doubts\. Intrinsic self\-correction often fails to improve reasoning and can degrade it\([Huang et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib42)\)\. The performance of ReAct owes more to the similarity between exemplars and queries than to interleaved reasoning\([Bhambri et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib27)\)\. LLM\-in\-the\-loop search such as Tree of Thoughts gives up soundness and completeness at a high cost in model calls\([Katz et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib61)\)\. These studies, however, typically examine a single strategy in a single regime\. We compare prompting strategies and agent architectures side by side, on the same tasks and with the same measurements, which separates their effects on executability, goal achievement and abstention\. The agents literature offers precise notions of agency\([Wooldridge and Jennings, 1995](https://arxiv.org/html/2610.22469#bib.bib1);[Franklin and Graesser, 1997](https://arxiv.org/html/2610.22469#bib.bib2)\), architectures for practical reasoning\([Rao and Georgeff, 1995](https://arxiv.org/html/2610.22469#bib.bib4)\), and work on integrating planning into agent reasoning\([Meneguzzi and de Silva, 2015](https://arxiv.org/html/2610.22469#bib.bib20)\)\. It also studies how planning, execution and monitoring are interleaved\([Brenner and Nebel, 2009](https://arxiv.org/html/2610.22469#bib.bib57);[Ghallab et al\., 2016](https://arxiv.org/html/2610.22469#bib.bib11)\)and how often an agent should reconsider its commitments\([Kinny and Georgeff, 1991](https://arxiv.org/html/2610.22469#bib.bib58)\)\. Recent work connects these concepts to LLMs, for instance by using language models for plan selection in BDI agents\([Ichida et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib7)\); Dignum and Dignum argue that agentic AI should build on the conceptual tools of the AAMAS community\([Dignum and Dignum, 2025](https://arxiv.org/html/2610.22469#bib.bib6)\), and Wooldridge revisits multi\-agent systems research in light of LLMs\([Wooldridge, 2026](https://arxiv.org/html/2610.22469#bib.bib5)\)\. We use two of these concepts for evaluation\. The agent function over interaction histories separates offline planning from planning\-by\-acting, and the distinction between epistemic \(sensing\) and ontic actions\([Scherl and Levesque, 2003](https://arxiv.org/html/2610.22469#bib.bib60);[Bolander and Andersen, 2011](https://arxiv.org/html/2610.22469#bib.bib10)\)allows us to measure whether agents sense before they act\. Dropping an intention that has become unachievable\([Cohen and Levesque, 1990](https://arxiv.org/html/2610.22469#bib.bib59)\)has a counterpart in planning, where recognising infeasibility is a classical problem with formal certificates\([Eriksson et al\., 2017](https://arxiv.org/html/2610.22469#bib.bib17)\)\. LLMs handle this poorly: abstention on unanswerable questions remains weak and has been reported to degrade with reasoning fine\-tuning\([Kirichenko et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib56)\), and in planning, o1\-preview recognised only 27% of unsolvable Blocksworld instances\([Valmeekam et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib23)\)\. Plancraft\([Dagan et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib64)\)adds intentionally unsolvable tasks to an interactive crafting benchmark for LLM agents\. We test abstention in every domain and in both regimes\. ## 3\.Evaluation Framework We treat planning as a set of behavioural properties defined by an explicit transition model\. Systems see tasks only in natural language, whereas correctness is decided by hidden formal semantics\. ### 3\.1\.Tasks behind natural\-language briefs We use the deterministic model of classical planning\([Ghallab et al\., 2004](https://arxiv.org/html/2610.22469#bib.bib12)\)\. ###### Definition 3\.1 \(Planning task\)\. A*planning task*is a tupleΠ=⟨S,A,γ,s0,G⟩\\Pi=\\langle S,A,\\gamma,s\_\{0\},G\\rangleof statesSS, grounded actionsAA, a partial transition functionγ:S×A⇀S\\gamma:S\\times A\\rightharpoonup S, an initial states0∈Ss\_\{0\}\\in Sand goal statesG⊆SG\\subseteq Swiths0∉Gs\_\{0\}\\notin G; an actionaais*applicable*inssifγ\(s,a\)\\gamma\(s,a\)is defined\. A planπ=⟨a1,…,aT⟩∈A∗\\pi=\\langle a\_\{1\},\\dots,a\_\{T\}\\rangle\\in A^\{\*\}of length\|π\|=T\|\\pi\|=Tis*executable*if eachata\_\{t\}is applicable inst−1s\_\{t\-1\}, wherest=γ\(st−1,at\)s\_\{t\}=\\gamma\(s\_\{t\-1\},a\_\{t\}\), and*valid*if, in addition,sT∈Gs\_\{T\}\\in G\.Π\\Piis*solvable*if it admits a valid plan, and we then writeL⋆\(Π\)L^\{\\star\}\(\\Pi\)for the length of a shortest valid plan\. All tasks are specified in PDDL\([McDermott et al\., 1998](https://arxiv.org/html/2610.22469#bib.bib14)\), with numeric fluents\([Fox and Long, 2003](https://arxiv.org/html/2610.22469#bib.bib15)\)for the capacities inLoadBalancing\. All actions have unit cost, so plan quality is plan length\. Systems never see the PDDL\. Each task is presented through two natural\-language briefs,B\(Π\)B\(\\Pi\)\. The*domain brief*describes in prose the object types, the world rules and, for every action, its parameters, requirements and effects\. The*instance brief*lists the objects and describes the initial situation, the goal and the optimisation metric\. For example, one requirement ofassign\-bucketin theLoadBalancingbrief reads “The server must have available space \(its current load must be strictly less than its capacity\)\.” Briefs keep the exact object and action names, so that plans can be parsed, but never expose predicates or PDDL syntax\. They were generated from the PDDL files with GPT\-5 \(gpt\-5\-2025\-08\-07, minimal reasoning effort\) in two stages: first the domain brief, then each instance brief conditioned on it\. The briefs were checked manually against the source files, and the generation prompts are provided in the supplementary material\. ### 3\.2\.Offline planning and planning\-by\-acting An agent is commonly modelled as a function from its*interaction history*, the calls it has made to its environment and the observations it has received, to its next action\([Russell and Norvig, 2021](https://arxiv.org/html/2610.22469#bib.bib3);[Wooldridge, 2009](https://arxiv.org/html/2610.22469#bib.bib19)\)\. Internal model calls, such as the intermediate steps of a prompting strategy, do not extend this history; only calls to the environment do\. This yields the classical distinction between offline problem solving, which computes a complete solution before acting, and online problem solving, which interleaves acting and observing\([Russell and Norvig, 2021](https://arxiv.org/html/2610.22469#bib.bib3), Sec\. 4\.5\)\(Figure[1](https://arxiv.org/html/2610.22469#S1.F1)\)\. ###### Definition 3\.2 \(Offline planner\)\. An*offline planner*maps the briefsB\(Π\)B\(\\Pi\), possibly at random, to an output that is parsed intoy∈A∗∪\{⊥\}y\\in A^\{\*\}\\cup\\\{\\bot\\\}, where⊥\\botdenotes the flagUNSOLVABLE, the claim thatΠ\\Piis unsolvable; an output that cannot be parsed in this way is*malformed*\. The planner may perform any number of internal model calls, but its only input from the environment isB\(Π\)B\(\\Pi\)\. ###### Definition 3\.3 \(Online agent\)\. The environment of an*online agent*is a simulator ofΠ\\Pithat maintains a current statess, initiallys0s\_\{0\}, and accepts three kinds of calls\. An*epistemic*call returns an observation and leavesssunchanged\. An*ontic*callact\(a\)\\texttt\{act\}\(a\), witha∈Aa\\in A, ends the episode with a*precondition violation*ifaais not applicable inss; otherwise, it setss←γ\(s,a\)s\\leftarrow\\gamma\(s,a\)and returns only an acknowledgement\. The*terminal*callsubmit\(\)\\texttt\{submit\}\(\)ends the episode with the claim thats∈Gs\\in G, andsubmit\(⊥\)\\texttt\{submit\}\(\\bot\)with the claim thatΠ\\Piis unsolvable\. The agent is a possibly randomised policy that maps each interaction historyht=⟨o0,x1,o1,…,xt,ot⟩h\_\{t\}=\\langle o\_\{0\},x\_\{1\},o\_\{1\},\\dots,x\_\{t\},o\_\{t\}\\rangleof callsxix\_\{i\}and observationsoio\_\{i\}\(whereo0o\_\{0\}is the instruction\) to the next call\. An ill\-formed call \(an unknown tool, action or object, a wrongly typed argument, or a wrong number of arguments\) returns an error observation and leavesssunchanged\. Every call, however many tokens it uses, consumes one unit of a budgetK\(Π\)K\(\\Pi\), which is not disclosed to the agent; exhausting it without a terminal call ends the episode with a*timeout*\. In our environment, the epistemic calls return a task overview \(scenario, a summary of the initial situation and the goal\), object listings and descriptions, the action catalogue with natural\-language requirements and effects, a rendering of the current state, and the history of executed actions \(Appendix C of the supplementary material shows them for an example task, with a complete episode\)\. Offline, a step that names a non\-existent or wrongly typed object makes the plan malformed\. The budget isK\(Π\)=max\(20,3L⋆\(Π\)\)K\(\\Pi\)=\\max\(20,3L^\{\\star\}\(\\Pi\)\)calls for solvable tasks and 40 for unsolvable ones\. This budget allows an agent to retrieve the task with four calls, query the state before every step of an optimal plan, and submit \(2L⋆\(Π\)\+52L^\{\\star\}\(\\Pi\)\+5calls in total\), with at least two calls to spare\. Treating an inapplicable action as fatal encodes a safety requirement: in a deployed system, such an action would be unsafe, illegal or physically impossible\. An ill\-formed call, by contrast, is a failure of communication that the environment can reject harmlessly\. ### 3\.3\.Planning properties Table[1](https://arxiv.org/html/2610.22469#S3.T1)maps each attributed competence to the property that operationalises it\. ###### Definition 3\.4 \(Planning properties\)\. Letπ\\pibe the plan that an episode onΠ\\Picommits to: the steps ofyyoffline \(the empty plan ify=⊥y=\\bot\), and the well\-formedactcalls online\. The episode is*executable*if its output is not malformed andπ\\piis executable\. It*succeeds*ifΠ\\Piis solvable andπ\\piis valid \(online, the episode must also end withsubmit\(\)\\texttt\{submit\}\(\)\), or ifΠ\\Piis unsolvable and the episode claims so \(y=⊥y=\\botoffline,submit\(⊥\)\\texttt\{submit\}\(\\bot\)online\)\. A correct claim of unsolvability is a*detection*; an incorrect one is a*false abstention*\. The*score*isL⋆\(Π\)/\|π\|L^\{\\star\}\(\\Pi\)/\|\\pi\|, the ratio of the optimal to the committed plan length, for a success on a solvable task,11for a detection and00otherwise\. For online episodes, the*inspection rate*is the fraction ofactcalls preceded by a query of the current state since the previousactcall\. Table 1\.Competences attributed to prompting strategies and agent architectures \(with the methods credited with them\), the property that operationalises each, and its measure\.All measures are computed from validated plans and trajectories, never from model\-generated rationales\. Every failed episode on a solvable task is assigned exactly one terminal category:*malformed output*\(offline\),*precondition violation*,*goal not reached*\(all actions applicable, final state not inGG\),*timeout*\(online\), or*false abstention*\. On an unsolvable task, every failure is a*missed infeasibility*, since only⊥\\botcan succeed, but we still record how the episode ended\. ### 3\.4\.Relating the two regimes The two regimes are not symmetric\. The following elementary observation makes comparisons between them interpretable\. ###### Proposition 3\.5 \(Acting weakly dominates offline planning\)\. Suppose that the observations of a fixed sequence ofccepistemic calls, issued at the start of an episode, determineB\(Π\)B\(\\Pi\)\. For every offline plannerPP, there is an online agent𝒜P\\mathcal\{A\}\_\{P\}such that, whenever a run ofPPon a taskΠ\\Pioutputsyyand commits to the planπ\\pi, withc\+\|π\|\+1≤K\(Π\)c\+\|\\pi\|\+1\\leq K\(\\Pi\), the episode of𝒜P\\mathcal\{A\}\_\{P\}that simulates this run achieves at least the success and score of that run\. ###### Proof\. 𝒜P\\mathcal\{A\}\_\{P\}first issues theccepistemic calls, reconstructsB\(Π\)B\(\\Pi\)from their observations and runsPPon it internally, with the same random choices, to obtainyy\. Ify=⊥y=\\bot, it callssubmit\(⊥\)\\texttt\{submit\}\(\\bot\); otherwise, it callsact\(at\)\\texttt\{act\}\(a\_\{t\}\)for each stepata\_\{t\}ofyyin turn and thensubmit\(\)\\texttt\{submit\}\(\)\. This uses at mostc\+\|π\|\+1≤K\(Π\)c\+\|\\pi\|\+1\\leq K\(\\Pi\)calls\. Ifyyfails, its success and score are00\. Ify=⊥y=\\botsucceeds,Π\\Piis unsolvable andsubmit\(⊥\)\\texttt\{submit\}\(\\bot\)is a detection\. If a planyysucceeds, it is valid\. Sinceγ\\gammais deterministic, and the simulator starts ins0s\_\{0\}and applies the sameγ\\gammaagainst whichyyis validated offline, everyactcall is applicable and the episode reaches the final state ofyy, which lies inGG, beforesubmit\(\)\\texttt\{submit\}\(\)\. The episode commits to the same planπ\\piand therefore obtains the same score\. ∎ In our benchmark the premise holds withc=4c=4: the overview, the object listing, the action catalogue and the current state give the objects, initial situation, goal and action rules, and so determineΠ\\Piand hence its briefsB\(Π\)B\(\\Pi\), which are fixed for each task\. Every successful offline plan we observed also satisfies the length condition\. If an agent performs worse than the offline planner built on the same model, the shortfall is therefore not imposed by the environment\. It lies in how the agent architecture presents the task to the model, in what the agent chooses to sense, and in when it commits to an action\. ## 4\.Benchmark The suite has five domains with ten tasks each: three standard planning domains and two new ones\.Blocksworld,GripperandLogisticscome from the planning literature\([Ghallab et al\., 2004](https://arxiv.org/html/2610.22469#bib.bib12);[McDermott, 2000](https://arxiv.org/html/2610.22469#bib.bib65)\), and each targets a different difficulty:Blocksworldstresses subgoal interaction under strict preconditions,Gripperbounded resources and batching with two single\-capacity grippers, andLogisticslong causal chains that alternate between trucks and an aeroplane\. Because these domains are widely published, we replace object identifiers with random strings \(e\.g\.,airplane\_1becomesair\_4f77ec\) to reduce the chance that plans are recalled rather than constructed\. Action names remain meaningful, since obfuscating them\([Valmeekam et al\., 2023a](https://arxiv.org/html/2610.22469#bib.bib22)\)would test symbol manipulation rather than the natural\-language planning to which the claims refer\. The two new domains,LoadBalancingandHospital, are unlikely to have been memorised, and they model applied settings in which an invalid action has real consequences\.LoadBalancingassigns traffic buckets to servers subject to region, service availability and numeric capacity\. Its actions power servers and start or stop services, and its goals can require reconfiguration, such as decommissioning a server\.Hospitalmanages patient placement under ward suitability and isolation requirements\. Beds can be dirty or blocked, and transfers and discharges leave beds dirty, so plans must account for side effects\. In the classification of Russell and Norvig\([Russell and Norvig, 2021](https://arxiv.org/html/2610.22469#bib.bib3)\), every task environment is fully observable, deterministic, static, discrete, known and single\-agent: the most benign class except that tasks are sequential\. Failures therefore cannot be attributed to uncertainty or to other agents\. An ideal agent could track the state from the task description alone; state queries allow an agent to correct its own state tracking\. Each domain has eight solvable and two unsolvable tasks \(Table[2](https://arxiv.org/html/2610.22469#S4.T2)\)\. Optimal plan lengths, computed by breadth\-first search over the reachable state space, range from 2 to 63 actions, but length is a poor proxy for difficulty: longGripperplans repeat a simple pattern, while shortBlocksworldplans require reasoning about interfering subgoals\. Unsolvable tasks are small modifications of solvable ones, for example removing the only truck in the destination city \(Logistics\), lowering a server’s capacity below the demand it must absorb \(LoadBalancing\), requiring a cyclic tower \(Blocksworld\), or starting with neither gripper free nor holding a ball, so that no ball can be picked up \(Gripper\)\. Table 2\.Optimal plan lengthL⋆L^\{\\star\}per task; a dash marks an unsolvable task\.Every offline plan is validated with VAL\([Howey et al\., 2004](https://arxiv.org/html/2610.22469#bib.bib16)\), which also reports the first failing step and its unsatisfied preconditions\. The agent environment is built on the same validator: each proposed action is checked by re\-validating the executed prefix extended with the candidate, and the state returned to the agent is reconstructed from the validator’s trace\. Offline planners and online agents are therefore judged by identical semantics\. Before validation, model outputs must be turned into actions\. An extractor parses PDDL\-style action lines, with a fallback that compiles short imperative steps using the action catalogue\. Agent controllers are implemented in DSPy\([Khattab et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib43)\), and every run logs its prompts, raw outputs, parsed plan, validator transcript, full tool\-call trajectory, token counts and cost\. The benchmark itself is stored as data \(PDDL files plus one JSON file per domain with the briefs and tool\-facing schemas\), so domains and problems can be added without code changes\. Code and data are provided in the supplementary material, whose Appendices A and B list the PDDL domains and two example tasks\. ## 5\.Experimental Setup Our experiments address four research questions\.RQ1\(offline planning\): do prompting strategies improve the executability, goal achievement and efficiency of offline plans?RQ2\(planning\-by\-acting\): do agents match or exceed the success rate of offline planning with the same model, which Proposition[3\.5](https://arxiv.org/html/2610.22469#S3.Thmtheorem5)shows to be possible, and does reflective memory or a critic improve on ReAct?RQ3\(abstention\): do systems recognise unsolvable tasks without abstaining on solvable ones?RQ4\(cost and scale\): are strategies and architectures worth their overhead, and do they narrow the gap between lower\- and higher\-tier models? We evaluate four models from two providers, one lower\-tier and one higher\-tier model from each \(Table[3](https://arxiv.org/html/2610.22469#S5.T3)\), which allows comparisons across tiers within each family, although the providers do not disclose parameter counts\. We use non\-reasoning models by design: the strategies and architectures under test were proposed for such models, and models trained for long reasoning internalise part of what these methods externalise\. All calls use temperature 0 except the proposal step of Tree of Thoughts, and offline calls are capped at 4,096 output tokens\. Table 3\.Evaluated models and list prices \(USD per million tokens, as of January 2026\) used for cost estimates\.Table 4\.Results per model and method\.*Succ\.*: success on the 40 solvable tasks \(%\)\.*Exec\.*: share of those episodes that are executable \(well formed, with every action applicable\), whether or not they reach the goal \(Definition[3\.4](https://arxiv.org/html/2610.22469#S3.Thmtheorem4); %\)\.*Det\.*: unsolvable tasks recognised \(out of 10\)\.*Tok\.*: mean tokens per episode in thousands, averaged over the four models \(per\-model values are in the supplementary material\)\. Bold marks the best success per model in each regime\. No difference from the within\-model baseline \(Direct offline, ReAct online\) survives the Holm correction;∗marksp<0\.05p<0\.05before correction \(exact McNemar test\)\.GPT\-4o\-miniGPT\-4oGemini 2\.0 Flash\-LiteGemini 2\.0 FlashMethodTok\.Succ\.Exec\.Det\.Succ\.Exec\.Det\.Succ\.Exec\.Det\.Succ\.Exec\.Det\.*Offline \(prompting strategies\)*Direct1\.512\.512\.5157\.562\.5125\.045\.0045\.047\.50Zero\-shot CoT4\.327\.5∗32\.5177\.5∗80\.0322\.527\.5062\.565\.02Plan\-and\-Solve2\.917\.522\.5075\.077\.5122\.525\.0065\.0∗65\.00Step\-Back3\.020\.025\.0157\.557\.5120\.030\.0057\.560\.03Self\-Refine7\.17\.510\.0157\.560\.0122\.542\.5050\.060\.04Tree of Thoughts32\.317\.520\.0042\.552\.5115\.017\.5035\.047\.51*Online \(agent architectures\)*ReAct25\.67\.520\.0040\.052\.5412\.517\.5032\.542\.51Reflexion28\.77\.512\.5035\.055\.0212\.512\.5127\.550\.02Critic45\.10\.030\.0035\.040\.0010\.020\.0032\.547\.50Offline, each model is prompted with six strategies\. All receive the same briefs, all prompts are zero\-shot, and the final call of every strategy must follow the same output contract: a single fenced block with one grounded action per line, or the flagUNSOLVABLE\. If several blocks are produced, we keep the last; intermediate calls are unconstrained\.*Direct*makes a single call\.*Zero\-shot CoT*\([Kojima et al\., 2022](https://arxiv.org/html/2610.22469#bib.bib34)\)first elicits a free\-form reasoning trace \(“Let’s think step by step”\) and then asks for the plan conditioned on it\.*Plan\-and\-Solve*\([Wang et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib35)\)first asks for a sketch of 4–10 subgoals without grounded actions, and*Step\-Back*\([Zheng et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib36)\)first abstracts the problem into 6–12 constraints and invariants\. *Self\-Refine*\([Madaan et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib37)\)alternates drafting and critique by the same model for up to three rounds\. Since LLM self\-critique of plan semantics is unreliable\([Stechly et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib26)\), the critique may report only mechanically checkable issues: unknown names, wrong arity or argument order, and formatting errors\. The loop stops early when the critique finds no such issue or the plan stops changing\.*Tree of Thoughts*\(ToT\)\([Yao et al\., 2023a](https://arxiv.org/html/2610.22469#bib.bib38)\)runs a beam search over plan prefixes\. At each of six depths, it samples three continuations of one to three actions for each prefix \(temperature 0\.7\), scores them in a separate call on a 0–10 scale, and keeps the two best prefixes\. The best prefix is finally completed into a full plan\. Online, the same models act as agents under three architectures, which give the acting model the same zero\-shot instruction: it warns that an inapplicable action ends the episode, advises querying the state when unsure and explains how to declare a task unsolvable\. All prompts are given in Appendix D of the supplementary material\.*ReAct*\([Yao et al\., 2023b](https://arxiv.org/html/2610.22469#bib.bib39)\)is a single policy that emits a reasoning step and one tool call per turn, conditioned on the full interaction history\. The other two architectures extend ReAct with the components named in RQ2: reflective memory and a critic\.*Reflexion*\([Shinn et al\., 2023](https://arxiv.org/html/2610.22469#bib.bib40)\)is adapted to a single episode, because a precondition violation ends the episode and a deployed agent cannot retry after an unsafe act\. An actor behaves as in ReAct; after every ill\-formed call, a reflector writes a short lesson \(e\.g\., the correct signature of an action\) into a memory that the actor sees for the rest of the episode\.*Critic\-guided ReAct*\(Critic\) adapts CRITIC\([Gou et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib41)\)\. Every proposedactorsubmitcall is first reviewed by a critic, a separate call to the same model that sees the same interaction history\. A rejection is returned to the actor, together with the critic’s reason, in place of an observation\. Unlike in CRITIC, the critic calls no tools, so it judges each proposal from the history alone\. Every proposal, approved or rejected, counts against the budget, which bounds the exchange between actor and critic\. In total, every model is run once with every strategy and architecture on all 50 tasks, giving4×6×50=1,2004\\times 6\\times 50=1\{,\}200offline and4×3×50=6004\\times 3\\times 50=600online episodes\. Closed APIs can retain some non\-determinism even at temperature 0, but single runs at temperature 0 keep the comparison as reproducible as these APIs allow; our intervals and tests therefore capture variation over tasks, not over repeated runs\. We report success on the 40 solvable tasks, detection on the 10 unsolvable ones, and the score over all 50, together with costs computed from logged token counts at the list prices in Table[3](https://arxiv.org/html/2610.22469#S5.T3); for the best configurations, we give 95% bootstrap confidence intervals over tasks\. Because all conditions share the same tasks, we compare each strategy and architecture with its within\-model baseline \(Direct offline, ReAct online\) using exact McNemar tests on paired outcomes, and we control the family\-wise error rate with the Holm–Bonferroni procedure within each regime \(20 comparisons offline, 8 online\)\. With 40 paired tasks, a difference must reach at least 25 points offline \(22\.5 online\) to survive the correction\. ## 6\.Results  \(a\) How episodes on solvable tasks end  \(b\) Score against cost Figure 2\.Outcomes and cost\. \(a\) Episodes on solvable tasks, pooled over the four models \(160 per method\); labels give the percentage of successes and of precondition violations, and*Other*covers malformed outputs and false abstentions\. \(b\) Score against mean cost per episode for the 36 configurations; colour marks the regime, marker shape the model, and the line the Pareto front\. PS: Plan\-and\-Solve; SB: Step\-Back; CoT: chain\-of\-thought; ToT: Tree of Thoughts\.Two panels\. Left: horizontal stacked bars, one per method\. Offline, Direct, zero\-shot CoT, Plan\-and\-Solve, Step\-Back, Self\-Refine and Tree of Thoughts succeed in 35, 48, 45, 39, 34 and 28 percent of episodes, and precondition violations end 58, 46, 53, 56, 56 and 64 percent\. Online, ReAct, Reflexion and Critic succeed in 23, 21 and 19 percent, and precondition violations end 67, 68 and 66 percent; Critic also has a 15 percent share of timeouts\. Right: scatter plot of score against cost on a logarithmic axis\. Offline configurations, shown in blue, form the upper\-left boundary; online configurations, shown in red, all lie below it\. GPT\-4o with chain\-of\-thought has the highest score, 0\.67, at about 0\.017 dollars per episode\. GPT\-4o with Tree of Thoughts and GPT\-4o with ReAct cost about 0\.1 dollars per episode and score 0\.34 and 0\.39\.Table[4](https://arxiv.org/html/2610.22469#S5.T4)gives the results of every configuration, and Figure[2](https://arxiv.org/html/2610.22469#S6.F2)shows how episodes end and what they cost\. ### 6\.1\.Offline planning \(RQ1\) The best offline configuration, GPT\-4o with zero\-shot chain\-of\-thought, solves 77\.5% of the solvable tasks \(31 of 40; 95% bootstrap interval 65–90%\)\. Gains from prompting are model\-specific\. Chain\-of\-thought and Plan\-and\-Solve raise the success rate of GPT\-4o from 57\.5% to 77\.5% and 75\.0%, and that of Gemini 2\.0 Flash from 45\.0% to 62\.5% and 65\.0%\. Chain\-of\-thought also raises GPT\-4o\-mini’s success rate from 12\.5% to 27\.5%\. In contrast, no strategy improves on direct prompting for Flash\-Lite\. Self\-Refine never exceeds direct prompting by more than five points, and Tree of Thoughts lowers success for three of the four models\. None of these differences survives the Holm correction\. Three are significant before correction: chain\-of\-thought for GPT\-4o \(9 tasks gained, 1 lost;p=0\.02p=0\.02\) and for GPT\-4o\-mini \(6 gained, none lost;p=0\.03p=0\.03\), and Plan\-and\-Solve for Flash \(10 gained, 2 lost;p=0\.04p=0\.04\)\. With 40 tasks, only large effects are detectable \(Section[5](https://arxiv.org/html/2610.22469#S5)\), so these gains may still be genuine\. Precondition violations are the dominant failure mode\. Under every strategy, 46–64% of the plans for solvable tasks contain an inapplicable action \(Figure[2](https://arxiv.org/html/2610.22469#S6.F2)a\), and such plans account for 89% of the failed ones\. Executable plans that miss the goal are less frequent \(2\.5–9% per strategy\), and malformed outputs are rarer still \(1\.3% overall\)\. Consequently, executability is close to success for the strongest configurations \(80\.0% against 77\.5% for GPT\-4o with chain\-of\-thought\)\. The gap is larger for some weaker configurations: the direct plans of Flash\-Lite are executable in 45\.0% of the solvable episodes but reach the goal in only 25\.0%\. Successful plans are usually optimal: 83% of them have optimal length, and the mean score of a success is 0\.95\. ### 6\.2\.Online planning\-by\-acting \(RQ2\) Every model performs worse online than offline\. The best online configuration, GPT\-4o with ReAct, solves 40\.0% of the solvable tasks \(16 of 40; 95% interval 25–55%\), against 77\.5% offline\. For GPT\-4o, Flash and Flash\-Lite, the best online result is about half of the best offline one; for GPT\-4o\-mini, it is about a quarter \(7\.5% against 27\.5%\)\. The drop already appears between the baselines: ReAct falls below direct prompting for all four models\. Pooled over models and methods, the gap appears in every domain \(9 to 23 points\), and the new domains are not systematically harder than the classical ones \(Appendix F of the supplementary material\)\. By Proposition[3\.5](https://arxiv.org/html/2610.22469#S3.Thmtheorem5), the gap is not caused by the environment, because every successful offline plan fits within the budget\. Agents rarely use the online regime’s queries before acting\. Only 7\.3% of their actions are preceded by a query of the current state \(the inspection rate; ReAct 6\.4%, Reflexion 8\.7%, Critic 6\.7%\), and in 58% of the episodes in which an agent acts at all, it never queries the state\. An average episode contains 5\.8 actions but only 0\.5 state queries, even though the instruction advises the agent to query the state when unsure\. Precondition violations terminate 66–68% of the solvable episodes under every architecture and account for 84% of the failed ones \(77–94% per model\)\. Neither reflective memory nor a critic improves performance\. Reflexion and Critic never outperform ReAct for any model, and no difference survives the correction \(smallest uncorrectedp=0\.25p=0\.25\)\. The architectures alter how agents fail rather than how often\. The critic leaves precondition violations almost unchanged \(66% of solvable episodes, against 67% for ReAct\) but adds timeouts: 15\.0% of its solvable episodes exhaust the budget, against 0\.6% for ReAct, because every rejected proposal consumes a call\. Reflexion improves executability over ReAct for the two higher\-tier models, but more of its episodes end without reaching the goal\. ### 6\.3\.Abstention on unsolvable tasks \(RQ3\) Systems rarely recognise that a task is unsolvable\. Only 32 of the 360 episodes on unsolvable tasks end with a detection \(8\.9%\): 22 of 240 offline and 10 of 120 online\. Online, about nine in ten missed infeasibilities end with a precondition violation\. The best configurations recognise 4 of the 10 tasks \(Flash with Self\-Refine offline, GPT\-4o with ReAct online\)\. Critic recognises none\. Detections are also concentrated on a few tasks: 13 of the 22 offline detections occur on a singleGrippertask in which no gripper can ever pick up a ball\. False abstentions, by contrast, are almost absent: only 1 of the 1,440 episodes on solvable tasks claims that no plan exists\. The deficit therefore lies in recall rather than precision\. Systems tend to attempt a plan rather than abstain, and no strategy or architecture reverses this tendency\. ### 6\.4\.Cost and scale \(RQ4\) The most expensive strategies and architectures do not produce gains that justify their cost\. Relative to direct prompting, Plan\-and\-Solve and Step\-Back use about twice as many tokens, chain\-of\-thought about three times, Self\-Refine about five times and Tree of Thoughts about 22 times as many\. Online, Reflexion uses 1\.1 times and Critic 1\.8 times as many tokens as ReAct, and an online episode costs $0\.031 on average, against $0\.007 offline\. Every configuration on the Pareto front of score against cost is offline \(Figure[2](https://arxiv.org/html/2610.22469#S6.F2)b\), and Tree of Thoughts, Self\-Refine and all online configurations are dominated\. Chain\-of\-thought and Plan\-and\-Solve widen the gap between model tiers rather than narrowing it\. GPT\-4o leads GPT\-4o\-mini by 45 points with direct prompting and by 50 points with chain\-of\-thought, and Flash leads Flash\-Lite by 20 points with direct prompting and by 40 points with chain\-of\-thought\. Averaged over strategies, offline success ranges from 17% to 61% across models; averaged over models, it ranges only from 28% to 48% across strategies\. Online, the corresponding ranges are 5–37% across models and 19–23% across architectures\. Success therefore varies more across models than across prompting strategies and architectures\. ## 7\.Discussion and Limitations The results support few of the competences attributed to the strategies and architectures in Table[1](https://arxiv.org/html/2610.22469#S3.T1)\. Chain\-of\-thought and Plan\-and\-Solve improve means–ends reasoning and constraint tracking for three of the four models, but by margins that are too small to confirm on 40 tasks\. Search\-based prompting does not improve planning: Tree of Thoughts reproduces the form of search without the systematic exploration of the state space on which the reliability of a planner rests\. Self\-monitoring architectures do not reduce failures, none induces sensing before acting, and simple strategies dominate expensive ones in cost\-effectiveness\. Acting should have worked in the agents’ favour: by Proposition[3\.5](https://arxiv.org/html/2610.22469#S3.Thmtheorem5), an agent in our benchmark can always plan first and then execute\. The agents we evaluated nevertheless perform worse online, and most of their failures result from inapplicable actions that a state query would have exposed\. Rare sensing explains why agents do not benefit from online queries, but not why they perform worse than offline planners, which do not sense at all\. A more likely cause is overconfident commitment: agents commit to one action at a time before they have a complete plan, based on beliefs about the state that they rarely check\. In this respect, they resemble the bold agents of Kinny and Georgeff\([Kinny and Georgeff, 1991](https://arxiv.org/html/2610.22469#bib.bib58)\), which execute their plans without stopping to reconsider\. Our critic does not remedy this overconfidence, since it reads the same history as the actor and cannot query the state itself\. These results suggest that the model should not decide whether to monitor execution: a sound validator should intercept inapplicable actions before execution, as LLM\-Modulo architectures propose\([Kambhampati et al\., 2024](https://arxiv.org/html/2610.22469#bib.bib24)\)\. In our environment, such a gate would turn every precondition violation into a recoverable error; whether agents would then recover remains an open question\. A rational agent drops an intention once it believes the goal to be unachievable\([Cohen and Levesque, 1990](https://arxiv.org/html/2610.22469#bib.bib59)\), but the systems we evaluated almost never do so\. GPT\-4o and Flash together account for 27 of the 32 detections, yet even these models miss most unsolvable tasks\. Planning ability therefore does not guarantee the recognition of infeasibility, which must be tested directly\. Our results suggest four practices for substantiating claims about agentic planning: report executability, failure modes and detection rather than success alone; state whether a system is offline or online in the agent\-theoretic sense, since the two regimes test different competences; include tasks that admit no valid plan, without which an inability to abstain remains invisible; and report cost, because complex designs are often dominated by simple ones\([Kapoor et al\., 2025](https://arxiv.org/html/2610.22469#bib.bib52)\)\. Where an explicit model of the task exists, planning formalisms make such checks inexpensive\. The benchmark is small \(50 tasks\), and each configuration was run once, at temperature 0\.Its domains are deterministic, fully observable and single\-agent, which excludes partial observability, stochastic effects and interaction with other agents\. Three of the five domains are classical, and randomised identifiers reduce, but do not eliminate, the advantage of familiarity\. The briefs were generated with an LLM and checked by hand, so a residual translation error cannot be ruled out\. Treating precondition violations as fatal is a deliberate safety choice; a regime that merely reports them would favour online agents\. Multi\-agent LLM systems add coordination and communication, which our semantics does not cover; multi\-agent planning domains are a natural next step\. The results are specific to the evaluated closed models, whose serving behaviour may change over time, and models trained for long reasoning may behave differently\. Finally, we re\-implemented all strategies and architectures and adapted some of them to our protocol \(Section[5](https://arxiv.org/html/2610.22469#S5)\): Reflexion reflects within one episode rather than across retries; Self\-Refine critiques only mechanically checkable issues, so semantic repair is tested only by Critic; and in Critic, the critic calls no tools\. Other prompts or hyperparameters could change individual results\. ## 8\.Conclusion We asked whether prompting strategies and agent architectures give LLM\-based systems the planning competences claimed for them\. For the four models we evaluated, and planning problems with a short length \(2 to 30 actions\) the evidence is largely negative\. LLM\-based planning ignores action preconditions and fails to reach the goal in the majority of cases, with performance differences across architectures and prompting strategies being model\-specific and statistically fragile: no one strategy solves this problem\. In the cases where the agent can choose to inspect the state, the agent foregoes this option to commit to potentially dangerous or impossible actions\. Moreover, despite being informed of the possibility of problems being impossible, these tasks are rarely recognised and the LLM wastes resources in trying to solve them\. The most expensive strategies increase cost by up to 20 to 30 times without improving average success\. Planning performance appears to depend mainly on the underlying model\. A strong advice to agent designers is to use validators to block inapplicable actions, as it is the most frequent mode of failure; and that evaluations should check planning properties rather than end\-to\-end success alone\. ## References - Bhambriet al\.\(2025\)S\. Bhambri, M\. Verma, and S\. KambhampatiDo think tags really help LLMs plan? A critical evaluation of ReAct\-style prompting\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=aFAMPSmNHR)Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p4.1)\. - Bolander and Andersen \(2011\)T\. Bolander and M\. B\. AndersenEpistemic planning for single\- and multi\-agent systems\.Journal of Applied Non\-Classical Logics21\(1\),pp\. 9–34\.External Links:[Document](https://dx.doi.org/10.3166/jancl.21.9-34)Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p3.1),[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Bratman \(1987\)M\. E\. BratmanIntention, plans, and practical reason\.Harvard University Press,Cambridge, MA\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1)\. - Brenner and Nebel \(2009\)M\. Brenner and B\. NebelContinual planning and acting in dynamic multiagent environments\.Autonomous Agents and Multi\-Agent Systems19\(3\),pp\. 297–331\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Choiet al\.\(2026\)J\. Choi, H\. Kim, H\. Ong, Y\. Yoon, M\. Jang, D\. Kim, and J\. KimReAcTree: hierarchical LLM agent trees with control flow for long\-horizon task planning\.InProceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),Richland, SC,pp\. 319–328\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1)\. - Cohen and Levesque \(1990\)P\. R\. Cohen and H\. J\. LevesqueIntention is choice with commitment\.Artificial Intelligence42\(2–3\),pp\. 213–261\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p6.1),[§7](https://arxiv.org/html/2610.22469#S7.p4.1)\. - Daganet al\.\(2025\)G\. Dagan, F\. Keller, and A\. LascaridesPlancraft: an evaluation dataset for planning with LLM agents\.InProceedings of the Second Conference on Language Modeling \(COLM\),Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p6.1)\. - Dignum and Dignum \(2025\)V\. Dignum and F\. DignumAgentifying agentic AI\.External Links:2511\.17332Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1),[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Erikssonet al\.\(2017\)S\. Eriksson, G\. Röger, and M\. HelmertUnsolvability certificates for classical planning\.InProceedings of the 27th International Conference on Automated Planning and Scheduling \(ICAPS\),External Links:[Document](https://dx.doi.org/10.1609/icaps.v27i1.13818)Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p3.1),[§2](https://arxiv.org/html/2610.22469#S2.p6.1)\. - Fox and Long \(2003\)M\. Fox and D\. LongPDDL2\.1: an extension to PDDL for expressing temporal planning domains\.Journal of Artificial Intelligence Research20,pp\. 61–124\.Cited by:[§3\.1](https://arxiv.org/html/2610.22469#S3.SS1.p2.1)\. - Franklin and Graesser \(1997\)S\. Franklin and A\. GraesserIs it an agent, or just a program?: a taxonomy for autonomous agents\.InIntelligent Agents III: Agent Theories, Architectures, and Languages \(ATAL 1996\),Lecture Notes in Computer Science, Vol\.1193\.External Links:[Document](https://dx.doi.org/10.1007/BFb0013570)Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Geffner and Bonet \(2013\)H\. Geffner and B\. BonetA concise introduction to models and methods for automated planning\.Synthesis Lectures on Artificial Intelligence and Machine Learning,Morgan & Claypool\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p3.1)\. - Ghallabet al\.\(2004\)M\. Ghallab, D\. Nau, and P\. TraversoAutomated planning: theory and practice\.Morgan Kaufmann\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p3.1),[§3\.1](https://arxiv.org/html/2610.22469#S3.SS1.p1.1),[§4](https://arxiv.org/html/2610.22469#S4.p1.1)\. - Ghallabet al\.\(2016\)M\. Ghallab, D\. Nau, and P\. TraversoAutomated planning and acting\.Cambridge University Press\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p3.1),[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Göbelet al\.\(2026\)K\. Göbel, P\. Lorang, P\. Zips, and T\. GlückAgentic LLM planning via step\-wise PDDL simulation: an empirical characterisation\.External Links:2603\.06064Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p2.1)\. - Gouet al\.\(2024\)Z\. Gou, Z\. Shao, Y\. Gong, Y\. Shen, Y\. Yang, N\. Duan, and W\. ChenCRITIC: large language models can self\-correct with tool\-interactive critiquing\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1),[§5](https://arxiv.org/html/2610.22469#S5.p6.1)\. - Guanet al\.\(2023\)L\. Guan, K\. Valmeekam, S\. Sreedharan, and S\. KambhampatiLeveraging pre\-trained large language models to construct and utilize world models for model\-based task planning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1)\. - Howeyet al\.\(2004\)R\. Howey, D\. Long, and M\. FoxVAL: automatic plan validation, continuous effects and mixed initiative planning using PDDL\.InProceedings of the 16th IEEE International Conference on Tools with Artificial Intelligence \(ICTAI\),pp\. 294–301\.External Links:[Document](https://dx.doi.org/10.1109/ICTAI.2004.120)Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p3.1),[§4](https://arxiv.org/html/2610.22469#S4.p5.1)\. - Hsiaoet al\.\(2026\)V\. Hsiao, M\. Roberts, and L\. SmithProcedural knowledge improves agentic LLM workflows\.InProceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),Richland, SC,pp\. 1425–1433\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1)\. - Huanget al\.\(2024\)J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. ZhouLarge language models cannot self\-correct reasoning yet\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p4.1)\. - Ichidaet al\.\(2024\)A\. Y\. Ichida, F\. Meneguzzi, and R\. C\. CardosoBDI agents in natural language environments\.InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),Richland, SC,pp\. 880–888\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Kambhampatiet al\.\(2024\)S\. Kambhampati, K\. Valmeekam, L\. Guan, M\. Verma, K\. Stechly, S\. Bhambri, L\. Saldyt, and A\. MurthyPosition: LLMs can’t plan, but can help planning in LLM\-Modulo frameworks\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1),[§7](https://arxiv.org/html/2610.22469#S7.p3.1)\. - Kapooret al\.\(2025\)S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, N\. Nadgir, and A\. NarayananAI agents that matter\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p4.1),[§7](https://arxiv.org/html/2610.22469#S7.p5.1)\. - Katzet al\.\(2024\)M\. Katz, H\. Kokel, K\. Srinivas, and S\. SohrabiThought of search: planning with language models through the lens of efficiency\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p4.1)\. - Khattabet al\.\(2024\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. PottsDSPy: compiling declarative language model calls into state\-of\-the\-art pipelines\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.22469#S4.p6.1)\. - Kinny and Georgeff \(1991\)D\. N\. Kinny and M\. P\. GeorgeffCommitment and effectiveness of situated agents\.InProceedings of the 12th International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 82–88\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1),[§7](https://arxiv.org/html/2610.22469#S7.p2.1)\. - Kirichenkoet al\.\(2025\)P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. BellAbstentionBench: reasoning LLMs fail on unanswerable questions\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p6.1)\. - Kojimaet al\.\(2022\)T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. IwasawaLarge language models are zero\-shot reasoners\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§5](https://arxiv.org/html/2610.22469#S5.p3.1)\. - Kokelet al\.\(2025\)H\. Kokel, M\. Katz, K\. Srinivas, and S\. SohrabiACPBench: reasoning about action, change, and planning\.InProceedings of the 39th AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 26559–26568\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i25.34857)Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p2.1)\. - Liuet al\.\(2023\)B\. Liu, Y\. Jiang, X\. Zhang, Q\. Liu, S\. Zhang, J\. Biswas, and P\. StoneLLM\+P: empowering large language models with optimal planning proficiency\.External Links:2304\.11477Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1)\. - Liuet al\.\(2024\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. TangAgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p2.1),[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Maet al\.\(2024\)C\. Ma, J\. Zhang, Z\. Zhu, C\. Yang, Y\. Yang, Y\. Jin, Z\. Lan, L\. Kong, and J\. HeAgentBoard: an analytical evaluation board of multi\-turn LLM agents\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Vol\.37\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p2.1),[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. ClarkSelf\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1),[§5](https://arxiv.org/html/2610.22469#S5.p4.1)\. - McDermottet al\.\(1998\)D\. McDermott, M\. Ghallab, A\. Howe, C\. Knoblock, A\. Ram, M\. Veloso, D\. Weld, and D\. WilkinsPDDL—the planning domain definition language, version 1\.2\.Technical reportTechnical ReportCVC TR\-98\-003,Yale Center for Computational Vision and Control\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.22469#S3.SS1.p2.1)\. - McDermott \(2000\)D\. M\. McDermottThe 1998 AI planning systems competition\.AI Magazine21\(2\)\.External Links:[Document](https://dx.doi.org/10.1609/aimag.v21i2.1506)Cited by:[§4](https://arxiv.org/html/2610.22469#S4.p1.1)\. - Meneguzzi and de Silva \(2015\)F\. Meneguzzi and L\. de SilvaPlanning in BDI agents: a survey of the integration of planning algorithms and agent reasoning\.The Knowledge Engineering Review30\(1\),pp\. 1–44\.External Links:[Document](https://dx.doi.org/10.1017/S0269888913000337)Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. ScialomGAIA: a benchmark for general AI assistants\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p2.1),[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Rao and Georgeff \(1995\)A\. S\. Rao and M\. P\. GeorgeffBDI agents: from theory to practice\.InProceedings of the First International Conference on Multi\-Agent Systems \(ICMAS\),pp\. 312–319\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Russell and Norvig \(2021\)S\. J\. Russell and P\. NorvigArtificial intelligence: a modern approach\.4th edition,Pearson\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p3.1),[§3\.2](https://arxiv.org/html/2610.22469#S3.SS2.p1.1),[§4](https://arxiv.org/html/2610.22469#S4.p3.1)\. - Scherl and Levesque \(2003\)R\. B\. Scherl and H\. J\. LevesqueKnowledge, action, and the frame problem\.Artificial Intelligence144\(1–2\),pp\. 1–39\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1),[§5](https://arxiv.org/html/2610.22469#S5.p6.1)\. - Shridharet al\.\(2021\)M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. HausknechtALFWorld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Silveret al\.\(2024\)T\. Silver, S\. Dan, K\. Srinivas, J\. B\. Tenenbaum, L\. Kaelbling, and M\. KatzGeneralized planning in PDDL domains with pretrained large language models\.InProceedings of the 38th AAAI Conference on Artificial Intelligence,Vol\.38\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i18.30006)Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1)\. - Stechlyet al\.\(2024\)K\. Stechly, K\. Valmeekam, and S\. KambhampatiChain of thoughtlessness? An analysis of CoT in planning\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1)\. - Stechlyet al\.\(2025\)K\. Stechly, K\. Valmeekam, and S\. KambhampatiOn the self\-verification limitations of large language models on reasoning and planning tasks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1),[§5](https://arxiv.org/html/2610.22469#S5.p4.1)\. - Steinet al\.\(2025\)K\. Stein, D\. Fišer, J\. Hoffmann, and A\. KollerAutomating the generation of prompts for LLM\-based action choice in PDDL planning\.InProceedings of the 35th International Conference on Automated Planning and Scheduling \(ICAPS\),Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p5.1),[§2](https://arxiv.org/html/2610.22469#S2.p2.1)\. - Sunet al\.\(2026\)H\. Sun, W\. Wang, M\. Song, J\. He, W\. Zhang, Y\. Liu, Y\. Yang, and Y\. ChengAgent planning benchmark: a diagnostic framework for planning capabilities in LLM agents\.External Links:2606\.04874Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Thakuret al\.\(2025\)A\. S\. Thakur, K\. Choudhary, V\. S\. Ramayapally, S\. Vaidyanathan, and D\. HupkesJudging the judges: evaluating alignment and vulnerabilities in LLMs\-as\-judges\.InProceedings of the Fourth Workshop on Generation, Evaluation and Metrics \(GEM\),pp\. 404–430\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Turpinet al\.\(2023\)M\. Turpin, J\. Michael, E\. Perez, and S\. R\. BowmanLanguage models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p2.1)\. - Valmeekamet al\.\(2023a\)K\. Valmeekam, M\. Marquez, A\. Olmo, S\. Sreedharan, and S\. KambhampatiPlanBench: an extensible benchmark for evaluating large language models on planning and reasoning about change\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Vol\.36\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p5.1),[§2](https://arxiv.org/html/2610.22469#S2.p1.1),[§4](https://arxiv.org/html/2610.22469#S4.p1.1)\. - Valmeekamet al\.\(2023b\)K\. Valmeekam, M\. Marquez, S\. Sreedharan, and S\. KambhampatiOn the planning abilities of large language models—a critical investigation\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1)\. - Valmeekamet al\.\(2024\)K\. Valmeekam, K\. Stechly, and S\. KambhampatiLLMs still can’t plan; can LRMs? A preliminary evaluation of OpenAI’s o1 on PlanBench\.External Links:2409\.13373Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p1.1),[§2](https://arxiv.org/html/2610.22469#S2.p6.1)\. - Wanget al\.\(2023\)L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\. Lee, and E\. LimPlan\-and\-solve prompting: improving zero\-shot chain\-of\-thought reasoning by large language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§5](https://arxiv.org/html/2610.22469#S5.p3.1)\. - Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1)\. - Wooldridge and Jennings \(1995\)M\. Wooldridge and N\. R\. JenningsIntelligent agents: theory and practice\.The Knowledge Engineering Review10\(2\),pp\. 115–152\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Wooldridge \(2009\)M\. WooldridgeAn introduction to multiagent systems\.2nd edition,John Wiley & Sons\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1),[§1](https://arxiv.org/html/2610.22469#S1.p3.1),[§3\.2](https://arxiv.org/html/2610.22469#S3.SS2.p1.1)\. - Wooldridge \(2026\)M\. WooldridgeRethinking multi\-agent systems in the era of LLMs\.InProceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),Richland, SC\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p5.1)\. - Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. YuOSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Vol\.37\.Cited by:[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Yaoet al\.\(2023a\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. NarasimhanTree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1),[§5](https://arxiv.org/html/2610.22469#S5.p4.1)\. - Yaoet al\.\(2023b\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p1.1),[§5](https://arxiv.org/html/2610.22469#S5.p5.1)\. - Yehudaiet al\.\(2026\)A\. Yehudai, L\. Eden, A\. Li, G\. Uziel, Y\. Zhao, R\. Bar\-Haim, A\. Cohan, and M\. Shmueli\-ScheuerA survey on evaluation of LLM\-based agents\.InFindings of the Association for Computational Linguistics: ACL 2026,San Diego, California, United States,pp\. 26690–26714\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1330)Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p2.1)\. - Zhenget al\.\(2024\)H\. S\. Zheng, S\. Mishra, X\. Chen, H\. Cheng, E\. H\. Chi, Q\. V\. Le, and D\. ZhouTake a step back: evoking reasoning via abstraction in large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§5](https://arxiv.org/html/2610.22469#S5.p3.1)\. - Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Vol\.36\.Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p2.1),[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. - Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.22469#S1.p2.1),[§2](https://arxiv.org/html/2610.22469#S2.p3.1)\. ## Appendices This document complements the paper\. Appendix[A](https://arxiv.org/html/2610.22469#A1)lists the five planning domains in PDDL\. Appendix[B](https://arxiv.org/html/2610.22469#A2)gives two example tasks from the new domains, with the natural\-language briefs that offline planners receive\. Appendix[C](https://arxiv.org/html/2610.22469#A3)shows the online interface, an offline run and an online episode on one of these tasks\. Appendix[D](https://arxiv.org/html/2610.22469#A4)gives every prompt: those that generated the briefs, those of the six prompting strategies and those of the three agent architectures\. Appendix[E](https://arxiv.org/html/2610.22469#A5)reports token use and cost per model, and Appendix[F](https://arxiv.org/html/2610.22469#A6)reports results per domain and task\. The accompanying archive contains the code and all 50 tasks\. ###### Contents 1. [1Introduction](https://arxiv.org/html/2610.22469#S1) 2. [2Related Work](https://arxiv.org/html/2610.22469#S2) 3. [3Evaluation Framework](https://arxiv.org/html/2610.22469#S3)1. [3\.1Tasks behind natural\-language briefs](https://arxiv.org/html/2610.22469#S3.SS1) 2. [3\.2Offline planning and planning\-by\-acting](https://arxiv.org/html/2610.22469#S3.SS2) 3. [3\.3Planning properties](https://arxiv.org/html/2610.22469#S3.SS3) 4. [3\.4Relating the two regimes](https://arxiv.org/html/2610.22469#S3.SS4) 4. [4Benchmark](https://arxiv.org/html/2610.22469#S4) 5. [5Experimental Setup](https://arxiv.org/html/2610.22469#S5) 6. [6Results](https://arxiv.org/html/2610.22469#S6)1. [6\.1Offline planning \(RQ1\)](https://arxiv.org/html/2610.22469#S6.SS1) 2. [6\.2Online planning\-by\-acting \(RQ2\)](https://arxiv.org/html/2610.22469#S6.SS2) 3. [6\.3Abstention on unsolvable tasks \(RQ3\)](https://arxiv.org/html/2610.22469#S6.SS3) 4. [6\.4Cost and scale \(RQ4\)](https://arxiv.org/html/2610.22469#S6.SS4) 7. [7Discussion and Limitations](https://arxiv.org/html/2610.22469#S7) 8. [8Conclusion](https://arxiv.org/html/2610.22469#S8) 9. [References](https://arxiv.org/html/2610.22469#bib) 10. [APlanning domains](https://arxiv.org/html/2610.22469#A1)1. [A\.1Blocksworld](https://arxiv.org/html/2610.22469#A1.SS1) 2. [A\.2Gripper](https://arxiv.org/html/2610.22469#A1.SS2) 3. [A\.3Logistics](https://arxiv.org/html/2610.22469#A1.SS3) 4. [A\.4LoadBalancing](https://arxiv.org/html/2610.22469#A1.SS4) 5. [A\.5Hospital](https://arxiv.org/html/2610.22469#A1.SS5) 11. [BExample tasks](https://arxiv.org/html/2610.22469#A2)1. [B\.1LoadBalancing, task 1](https://arxiv.org/html/2610.22469#A2.SS1) 2. [B\.2Hospital, task 1](https://arxiv.org/html/2610.22469#A2.SS2) 12. [CThe online environment and example runs](https://arxiv.org/html/2610.22469#A3)1. [C\.1The online interface](https://arxiv.org/html/2610.22469#A3.SS1) 2. [C\.2An offline run](https://arxiv.org/html/2610.22469#A3.SS2) 3. [C\.3An online episode](https://arxiv.org/html/2610.22469#A3.SS3) 13. [DPrompts](https://arxiv.org/html/2610.22469#A4)1. [D\.1Brief generation](https://arxiv.org/html/2610.22469#A4.SS1) 2. [D\.2Prompting strategies](https://arxiv.org/html/2610.22469#A4.SS2) 3. [D\.3Agent architectures](https://arxiv.org/html/2610.22469#A4.SS3) 14. [EToken use and cost per model](https://arxiv.org/html/2610.22469#A5) 15. [FResults per domain and task](https://arxiv.org/html/2610.22469#A6) 16. [GUse of AI tools](https://arxiv.org/html/2610.22469#A7) ## Appendix APlanning domains The domain files below are those used by the validator and by the agent environment, with comments removed\. Plan quality is plan length in every domain: each action in the four typed domains increasestotal\-costby one, andBlocksworldhas no cost function\.BlocksworldandLogisticsfollow the standard encodings;Gripperadds a single condition, that the robot cannot enter a locked room, which only Gripper task 9 uses\. In these three domains, object names in the problem files are random identifiers \(Section 4 of the paper\)\. The problem files of all 50 tasks are in the archive, underexamples/<domain\>/problems\_pddl/, together with their briefs \(examples/<domain\>/prompts\.json\)\. In the listings, PDDL keywords are shown inblueand variables invermilion\. ### A\.1\.Blocksworld Blocks are untyped\. A single arm picks up a clear block from the table, unstacks a clear block from another block, puts the block it holds down on the table or stacks it on a clear block; it holds at most one block at a time\.\\pddlfileBlocksworld domaingen/pddl\_domain\_blocks\.txt ### A\.2\.Gripper A robot with two grippers moves balls between rooms\. Each gripper holds at most one ball\. The robot can move from its room to any room that is not locked\.\\pddlfileGripper domaingen/pddl\_domain\_gripper\.txt ### A\.3\.Logistics Packages are carried by trucks between locations of the same city and by aeroplanes between airports\. Trucks are confined to their city; aeroplanes fly only between airports\.\\pddlfileLogistics domaingen/pddl\_domain\_logistics\.txt ### A\.4\.LoadBalancing Traffic buckets are assigned to servers\. A bucket can be assigned only to an online server in its region that runs the bucket’s service and has spare capacity; numeric fluents hold the load and the capacity of each server\. A server can be powered off only when its load is zero, and services can be started and stopped on online servers\. The capacity condition is written as𝑙𝑜𝑎𝑑\+1≤𝑐𝑎𝑝𝑎𝑐𝑖𝑡𝑦\\mathit\{load\}\+1\\leq\\mathit\{capacity\}, which the validator evaluates exactly; for the integer loads that arise, it is equivalent to𝑙𝑜𝑎𝑑<𝑐𝑎𝑝𝑎𝑐𝑖𝑡𝑦\\mathit\{load\}<\\mathit\{capacity\}\.\\pddlfileLoadBalancing domaingen/pddl\_domain\_balancer\.txt ### A\.5\.Hospital Waiting patients are admitted to free beds in the ward they require, and patients who require isolation need an isolation bed\. A bed must also be clean and unblocked to receive a patient\. Transfers and discharges leave the vacated bed dirty, and a dirty bed must be cleaned before it is free again\. Free beds can be blocked, for example for maintenance, and unblocked\.\\pddlfileHospital domaingen/pddl\_domain\_hospital\.txt ## Appendix BExample tasks We illustrate the two new domains with their first tasks\. For each, we give the problem file \(comments removed\), an optimal plan, which the validator accepts, and the two briefs that offline planners receive\. The briefs were generated from the PDDL files with the prompts of Appendix[D\.1](https://arxiv.org/html/2610.22469#A4.SS1)and checked by hand\. ### B\.1\.LoadBalancing, task 1 \\pddlfile Problem filegen/pddl\_problem\_balancer\_1\.txt Optimal plan \(4 actions\)[⬇](data:text/plain;base64,KHVuYXNzaWduLWJ1Y2tldCBiMiBzMl9ldSkKKGFzc2lnbi1idWNrZXQgYjIgczFfZXUgZXVfd2VzdCBzZWFyY2gpCihhc3NpZ24tYnVja2V0IGIzIHMxX2V1IGV1X3dlc3Qgc2VhcmNoKQoocG93ZXItb2ZmIHMyX2V1KQ==)\(unassign\-bucketb2s2\_eu\)\(assign\-bucketb2s1\_eueu\_westsearch\)\(assign\-bucketb3s1\_eueu\_westsearch\)\(power\-offs2\_eu\) \\brieffile Domain briefgen/brief\_domain\_balancer\.txt\\brieffileInstance briefgen/brief\_instance\_balancer\_1\.txt ### B\.2\.Hospital, task 1 \\pddlfile Problem filegen/pddl\_problem\_hospital\_1\.txt Optimal plan \(2 actions\)[⬇](data:text/plain;base64,KGFkbWl0IHAxIGIxIHdfbWVkKQooYWRtaXQgcDIgYjIgd19tZWQp)\(admitp1b1w\_med\)\(admitp2b2w\_med\) \\brieffile Domain briefgen/brief\_domain\_hospital\.txt\\brieffileInstance briefgen/brief\_instance\_hospital\_1\.txt ## Appendix CThe online environment and example runs ### C\.1\.The online interface An online agent receives the instruction in Appendix[D\.3](https://arxiv.org/html/2610.22469#A4.SS3)and acts through nine tools\. Seven are epistemic and leave the state unchanged:get\_task\_overview,list\_objects,describe\_object,list\_actions,describe\_action,get\_stateandget\_history\. The ontic toolactexecutes one grounded action, and the terminal toolsubmitends the episode, either with the claim that the goal is reached or, assubmit\(unsolvable=True\), with the claim that the task is unsolvable\. Every call consumes one unit of the budget \(Definition 3\.3 of the paper\)\. The ReAct and Reflexion actors also see thefinishtool that DSPy’s ReAct module adds \(Appendix[D\.3](https://arxiv.org/html/2610.22469#A4.SS3)\); the harness treats a call to it as a goal claim, likesubmit\(\)\. Below are the observations that the tools return on LoadBalancing task 1\. Box colours mark the kind of call: epistemicexecuted actionill\-formed callprecondition violationterminal #### C\.1\.1\.At the start of the episode #### C\.1\.2\.Acting The call with the unknown objectb4is ill\-formed: it is rejected, the state is unchanged and the episode continues\. The second call executes an applicable action\. A goal claim ends the episode, and the environment reports whether the executed plan reaches the goal; here, after one action, it does not\. #### C\.1\.3\.An inapplicable action In a new episode on the same task, powering offs2\_euis inapplicable, because the server still has load 1\. The call ends the episode with a precondition violation\. ### C\.2\.An offline run GPT\-4o with direct prompting on LoadBalancing task 1\. The prompt is the Direct prompt of Appendix[D\.2](https://arxiv.org/html/2610.22469#A4.SS2)with the briefs of Appendix[B\.1](https://arxiv.org/html/2610.22469#A2.SS1)\.\\offlineoutcome\\plainfileModel responsegen/example\_offline\_response\.txt\\pddlfileParsed plangen/example\_offline\_plan\.txt\\plainfileValidator report \(VAL, excerpt\)gen/example\_offline\_val\.txt ### C\.3\.An online episode GPT\-4o with ReAct on LoadBalancing task 1\. Each step shows the agent’s reasoning \(in italics\), its tool call and the observation it receives, coloured as in Appendix C\.1\.\\onlineoutcome ## Appendix DPrompts ### D\.1\.Brief generation The briefs were generated with GPT\-5 \(gpt\-5\-2025\-08\-07, minimal reasoning effort, temperature 1\) in two stages, with the system message “You are a helpful assistant\.” First, the domain brief is generated from the domain file; then each instance brief is generated from its problem file and the domain brief\. The placeholders in double braces are replaced by the file contents and by the generated domain brief\.\\plainfilePrompt for the domain briefgen/prompt\_domain\_prompt\_template\.txt\\plainfilePrompt for an instance briefgen/prompt\_instance\_prompt\_template\.txt ### D\.2\.Prompting strategies Every prompt begins with the same context, which presents the two briefs, and the final call of every strategy ends with the same output requirements\. In the prompts below,<CONTEXT\>and<OUTPUT REQUIREMENTS\>stand for these two blocks,<DOMAIN BRIEF\>and<INSTANCE BRIEF\>for the briefs, and other terms in angle brackets for outputs of earlier calls\. The final plan is taken from the last fenced block of the final call\. Tree of Thoughts keeps a beam of two prefixes; at each of six depths, it asks for three candidate continuations of every prefix in the beam and scores them in a separate call, adding each score to the score of its prefix, and finally completes the best prefix\. Self\-Refine stops early when the critique is exactlyNO\_ISSUESor the revised plan equals the previous one\.\\promptfileContextshared by every callgen/prompt\_offline\_context\.txt\\promptfileOutput requirementsend of every final callgen/prompt\_offline\_output\_requirements\.txt ### D\.3\.Agent architectures The agents are DSPy modules\. DSPy turns the signature of each module \(its instruction and its input and output fields\) into a system message, and the current values of the input fields into a user message; field markers such as\[\[ \#\# objective \#\# \]\]delimit the fields\. Below are the messages of the first call of each module on LoadBalancing task 1, rendered with DSPy 3\.0\.4\. The harness logs the exact messages of every call of every episode \(lm\_calls\.json\)\. The three architectures give the acting model the same instruction \(the numbered strategy in each actor’s system message\)\. The Reflexion actor additionally sees the lessons in its memory\. The Critic actor, which is not a DSPy ReAct module, receives the list of tools with their arguments in its instruction; DSPy’s ReAct module adds the same list automatically for the other two agents\. ReAct \\msgfile System messagegen/agent\_react\_system\.txt\\msgfileUser message of the first callgen/agent\_react\_user\.txt\\msgfileUser message of the second call: the trajectory so far is appended after every stepgen/agent\_react\_user\_second\.txt Reflexion \\msgfile Actor: system messagegen/agent\_reflexion\_actor\_system\.txt\\msgfileActor: user message of the first callgen/agent\_reflexion\_actor\_user\.txt\\msgfileReflector \(called after every ill\-formed call\): system messagegen/agent\_reflexion\_reflector\_system\.txt\\msgfileReflector: example user messagegen/agent\_reflexion\_reflector\_user\.txt Critic \\msgfile Actor: system messagegen/agent\_critic\_actor\_system\.txt\\msgfileActor: user message of the first callgen/agent\_critic\_actor\_user\.txt\\msgfileCritic \(reviews every proposed act or submit call\): system messagegen/agent\_critic\_critic\_system\.txt\\msgfileCritic: example user messagegen/agent\_critic\_critic\_user\.txt ## Appendix EToken use and cost per model Table[E\.1](https://arxiv.org/html/2610.22469#A5.T1)gives the mean number of tokens \(input and output\) and the mean cost per episode of every configuration, over its 50 episodes\. Costs use the list prices of Table 3 of the paper\. Table 4 of the paper reports the token column averaged over the four models\. Table E\.1\.Mean tokens and cost per episode, per model and method\. ## Appendix FResults per domain and task Table[F\.1](https://arxiv.org/html/2610.22469#A6.T1)gives success, executability and detection per domain, pooled over the four models and over the six prompting strategies \(offline\) or the three agent architectures \(online\)\. Figure[F\.1](https://arxiv.org/html/2610.22469#A6.F1)gives the success rate for each solvable task\. As in Table 4 of the paper, success and executability are computed on the solvable tasks and detection on the unsolvable ones\. Online success is lower than offline success in every domain\. For individual models, averaged over methods, it is lower in 17 of the 20 combinations of model and domain, and five of the 40 tasks are solved more often online\. Difficulty does not follow plan length\.Logisticshas the longest plans but the same offline success asBlocksworld, andGripper, whose long plans repeat a simple pattern, is the second easiest domain offline\. The new domains are not systematically harder than the classical ones: offline,LoadBalancingis the easiest domain andHospitalthe hardest\. No configuration solvesBlocksworldtask 10 orHospitaltasks 4 and 7 offline, and ten tasks are never solved online\. Offline detections are concentrated onGrippertask 1 \(13 of the 22\) and onHospital\(7\), and no offline episode recognises an unsolvableBlocksworldtask\. Table F\.1\.Results per domain, pooled over models and methods\. Success and executability are computed on the eight solvable tasks of each domain \(192 offline and 96 online episodes per domain\) and detection on its two unsolvable tasks \(48 and 24 episodes\)\.*Best configuration*: success of the best single configuration \(model and method\) on the solvable tasks of the domain\.L⋆L^\{\\star\}: range of optimal plan lengths\.Figure F\.1\.Success on each solvable task: the share of the 24 offline configurations \(four models and six strategies\) and of the 12 online configurations \(four models and three architectures\) that solve it\. The second row of labels gives the optimal plan lengthL⋆L^\{\\star\}; unsolvable tasks are omitted\. ## Appendix GUse of AI tools GPT\-5 \(gpt\-5\-2025\-08\-07\) generated the natural\-language briefs from the PDDL files, with the prompts of Appendix[D\.1](https://arxiv.org/html/2610.22469#A4.SS1); the authors checked the briefs by hand\. An AI assistant \(Claude Opus 5\.5, Anthropic\) was used to edit parts of the paper’s text from the authors’ own write\-up of this work\. The authors reviewed all such output and take full responsibility for the paper and the code\.
Similar Articles
When Planning Problems Become Hard for LLMs: Separating Structural Complexity from Semantic Interpretability
该论文研究LLM在经典规划问题上的失败，发现传统delete-relaxation启发式h_add在估计结构难度上远优于ARC等学习模型，而通过逐步重命名谓词的受控实验表明LLM规划成功率随语义可解释性下降而从67.5%降至40.0%，指出混合规划系统的关键在于建模LLM的语义行为而非改进结构难度预测器。
Competence-Aware Orchestration for LLM-Based Multi-Agent Systems
该论文提出了一种两阶段的 competence-aware 框架，用于 LLM 多智能体系统的自动编排，根据模型在当前查询上的能力动态调整角色激活与通信图，在六个基准上平均达到 93.81%，同时在高能力骨干上降低 36.26% 的 token 成本。
Planning Commitment of LLM under Partial Observability
该论文系统研究了LLM规划智能体在部分可观测环境中的计划承诺长度问题，比较了激进式与保守式策略在安全性与效率之间的权衡，并提出可调的混合策略。
From Natural Language Intent to Executable Control: Hierarchical Feedback-Guided LLM Planning for Multi-component Cyber-Physical Systems
该论文提出一种层次化反馈引导的LLM规划框架，将自然语言指令转换为多组件信息物理系统(CPS)的可执行任务计划与STL约束，并通过诊断反馈修正不一致，GPT-5在七个场景上达到87.9%的成功率。
M-Plan: Comprehensive Planning For LLM Agents with Memory Arrangement
M-Plan is a closed-loop multi-agent system framework that integrates long-term and short-term memory with global-local and past-current guidance to improve LLM agents' planning on long-horizon multi-turn tasks, outperforming baselines on ALFWorld, Webshop, and PDDL.