AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints

Hugging Face Daily Papers Papers
adaptive-planning benchmark llm-agents constraints interactive multi-turn household-tasks

Summary

AdaPlanBench is a dynamic benchmark for evaluating LLM agents' ability to adaptively plan under progressively revealed world and user constraints through multi-turn interactions, showing current models struggle especially with user constraints.

Planning for real-world problems by language models often involves both world and user constraints, which may not be fully specified upfront and are progressively disclosed through interaction. However, existing benchmarks still underexplore adaptive planning under such progressively revealed dual constraints. To address this gap, we introduce AdaPlanBench, a dynamic interactive benchmark for evaluating whether Large Language Model (LLM) agents can adaptively plan and re-plan under progressively revealed world and user constraints. AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task with dual constraints. At runtime, agents interact with the environment in a multi-turn protocol where hidden constraints are revealed only when the agent proposes a plan that violates them, requiring iterative plan revision under accumulating feedback. This makes planning challenging, as agents must infer and track constraints from feedback while re-planning effectively. Experiments on ten leading LLMs show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy. We further observe that performance degrades as more constraints accumulate, with user constraints posing a particularly large challenge and failures often stemming from weaker physical grounding and reduced effectiveness. These results establish AdaPlanBench as a testbed for dual-constrained interactive planning and highlight the challenge of reliable adaptation to dynamically revealed constraints in LLM agents.
Original Article
View Cached Full Text

Cached at: 06/05/26, 06:06 AM

Paper page - AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints

Source: https://huggingface.co/papers/2606.05622

Abstract

AdaPlanBench presents a dynamic interactive benchmark for evaluating LLM agents’ ability to adaptively plan under progressively revealed world and user constraints through multi-turn interactions.

Planning for real-world problems by language models often involves both world and user constraints, which may not be fully specified upfront and are progressively disclosed through interaction. However, existing benchmarks still underexploreadaptive planningunder such progressively revealeddual constraints. To address this gap, we introduce AdaPlanBench, a dynamicinteractive benchmarkfor evaluating whetherLarge Language Model(LLM) agents can adaptively plan and re-plan under progressively revealed world and user constraints. AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task withdual constraints. At runtime, agents interact with the environment in amulti-turn protocolwhere hidden constraints are revealed only when the agent proposes a plan that violates them, requiring iterativeplan revisionunder accumulating feedback. This makes planning challenging, as agents must infer and track constraints from feedback while re-planning effectively. Experiments on ten leading LLMs show thatadaptive planningunderdual constraintsremains challenging, with the best model reaching only 67.75% accuracy. We further observe that performance degrades as more constraints accumulate, with user constraints posing a particularly large challenge and failures often stemming from weakerphysical groundingand reduced effectiveness. These results establish AdaPlanBench as a testbed for dual-constrained interactive planning and highlight the challenge of reliable adaptation to dynamically revealed constraints in LLM agents.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2606\.05622

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.05622 in a model README.md to link it from this page.

Datasets citing this paper1

#### JiayuJeff/AdaPlanBench Updatedabout 3 hours ago • 6 • 1

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.05622 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models

arXiv cs.AI

PlanningBench is a framework for generating scalable, diverse, and verifiable planning data to evaluate and train large language models, featuring a constraint-driven synthesis pipeline with adaptive difficulty control and quality filtering. Experiments show that frontier LLMs struggle with coupled constraints, and reinforcement learning on PlanningBench data improves performance on unseen planning tasks.