MA-Bench: Measuring Operational Planning in Language Models

Hugging Face Daily Papers Papers
benchmark language-models operational-planning reasoning evaluation military-planning llm-judge

Summary

MA-Bench introduces the first benchmark for evaluating how well language models perform mission analysis (military operational planning) by turning incomplete, contradictory document packets into structured plans graded against 1,257 expert-written requirements.

How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss κ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts.
Original Article
View Cached Full Text

Cached at: 10/15/26, 02:55 AM

Paper page - MA-Bench: Measuring Operational Planning in Language Models

Source: https://huggingface.co/papers/2610.19138

Abstract

Howwellcanalanguagemodelturnalarge,incompleteandpartlycontradictorybodyofdocumentsintoaplanwhennosinglecorrectanswerexists?WeintroduceMissionAnalysis(MA)-Bench,builtonmissionanalysis,thefirststepofmilitaryoperationalplanning,inwhichastaffreadsthefullsituationandstatestheproblembeforeanyoneproposesasolution.Its10expert-builtscenariospairsyntheticpacketsoftentotwelvedocumentswith1,257expert-writtenpass/failrequirements,eachcontaininginformationontheirsource,questiontype,andimportancelevel.Areasoning-modeljudgegradeseachbriefagainsttherequirementswithoutseeingthedocuments;itgradeslikeafifthcredentialedplanner,leavingafour-plannerpanel’sagreementunchanged(Fleissκ0.42to0.41),andajudgefromasecondproviderreproducesthemodelranking(Spearman0.94).Acrosstwentyfrontierandopen-weightsmodelsthebestgets62%oftherubricitemsright.Modelsarestrongestonfactsstatedinthedocuments(58%)andweakestonjudgmentsthatmustbeinferred(33%),andsixfailuresintenareomissionsratherthanerrors.MA-Benchgradesthedeterminations,nottheprose.Wereleasethepackets,rubrics,prompts,scorerandeverygradedbriefwithitem-levelverdicts.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.19138

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.19138 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.19138 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.19138 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MA-Bench: Measuring Operational Planning in Language Models

arXiv cs.LG

Scale AI 与 Data Machines 推出 MA-Bench 基准，通过军事任务分析场景评估语言模型在不完整、矛盾信息下的操作规划判断能力；20 个前沿模型中最佳仅答对 62% 的评分项，最弱环节是需要推断的判断。

PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models

arXiv cs.AI

PlanningBench is a framework for generating scalable, diverse, and verifiable planning data to evaluate and train large language models, featuring a constraint-driven synthesis pipeline with adaptive difficulty control and quality filtering. Experiments show that frontier LLMs struggle with coupled constraints, and reinforcement learning on PlanningBench data improves performance on unseen planning tasks.

PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models

arXiv cs.CL

This paper introduces PlanBench-V, the first comprehensive benchmark for evaluating Vision-Language Models on spatial planning map interpretation, including an expert-annotated dataset and a four-dimension evaluation framework. Experiments show significant progress but highlight persistent challenges in implementation-oriented tasks.

Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

arXiv cs.CL

MissionBench is a new benchmark for evaluating multimodal large language models (MLLMs) on long-horizon embodied tasks in aerial 3D environments, revealing that even the best models succeed on fewer than 35% of missions compared to 84.4% human performance.