PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models
Summary
PlanningBench is a framework for generating scalable, diverse, and verifiable planning data to evaluate and train large language models, featuring a constraint-driven synthesis pipeline with adaptive difficulty control and quality filtering. Experiments show that frontier LLMs struggle with coupled constraints, and reinforcement learning on PlanningBench data improves performance on unseen planning tasks.
View Cached Full Text
Cached at: 05/22/26, 08:49 AM
# PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models Source: [https://arxiv.org/abs/2605.20873](https://arxiv.org/abs/2605.20873) [View PDF](https://arxiv.org/pdf/2605.20873) > Abstract:Planning is a fundamental capability for large language models \(LLMs\) because such complex tasks require models to coordinate goals, constraints, resources, and long\-term consequences into executable and verifiable solutions\. Existing planning benchmarks, however, usually treat planning data as fixed collections of instances rather than controllable generation targets\. This limits scenario coverage, ties difficulty to surface\-level proxies rather than structural sources, and offers limited support for scalable generation, automatic verification, or planning\-oriented training\. We introduce PlanningBench, a framework for generating scalable, diverse, and verifiable planning data for both evaluation and training\. PlanningBench starts from real planning scenarios and abstracts practical workflows into a structured taxonomy of more than 30 task types, subtasks, constraint families, and difficulty factors\. Guided by this taxonomy, a constraint\-driven synthesis pipeline instantiates self\-contained planning problems with adaptive difficulty control, quality filtering, and instance\-level verification checklists\. This shifts planning data construction from fixed benchmark collection to controllable generation while preserving realistic task grounding\. We use PlanningBench to evaluate open\-source and closed\-source frontier LLMs, and find that current models still struggle to produce complete solutions under coupled constraints\. Beyond evaluation, reinforcement learning on verified PlanningBench data improves performance on unseen planning benchmarks and broader instruction\-following tasks\. Further analysis suggests that determinate or well\-specified optimal solutions provide clearer reward signals and more stable training dynamics\. Overall, PlanningBench provides a controllable source of planning data for diagnosing and improving generalizable planning abilities in LLMs\. ## Submission history From: Zhao Wang \[[view email](https://arxiv.org/show-email/ff3706ed/2605.20873)\] **\[v1\]**Wed, 20 May 2026 08:10:15 UTC \(1,614 KB\)
Similar Articles
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
AdaPlanBench is a dynamic benchmark for evaluating LLM agents' ability to adaptively plan under progressively revealed world and user constraints through multi-turn interactions, showing current models struggle especially with user constraints.
PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
This paper introduces PlanBench-V, the first comprehensive benchmark for evaluating Vision-Language Models on spatial planning map interpretation, including an expert-annotated dataset and a four-dimension evaluation framework. Experiments show significant progress but highlight persistent challenges in implementation-oriented tasks.
Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment
This paper introduces UPBench, a benchmark to evaluate large language models on urban planning knowledge across four knowledge pillars and five cognitive levels, finding that models perform better on higher-order analysis than factual recall, and identifying epistemic limitations such as regulatory hallucination and phronetic deficit.
EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation
This paper introduces EnvSimBench, a benchmark for evaluating Large Language Models' ability to simulate environments for agent training. It identifies a 'state change cliff' in current LLMs and proposes a constraint-driven pipeline to reduce hallucinations and costs.
Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly
Introduces Flat-Pack Bench, a benchmark for evaluating fine-grained spatio-temporal reasoning in large vision-language models using furniture assembly tasks. Experiments show current LVLMs struggle with tracking and spatial interactions.