FDE-Bench：评估从描述不充分的真实世界业务需求到端到端交付的能力

# EvaluatingEnd-to-End Delivery from UnderspecifiedReal-World Business Requests
Source: [https://arxiv.org/html/2610.14539](https://arxiv.org/html/2610.14539)
\\paperstyle

simple\\papercolornavy 1\]FM Agent Team, Baidu AI Cloud 2\]Beijing University of Posts and Telecommunications 3\]The Chinese University of Hong Kong 4\]Tsinghua University 5\]Peking University 6\]University of the Chinese Academy of Sciences 7\]Institute of Automation, Chinese Academy of Sciences

## FDE\-Bench: Evaluating End\-to\-End Delivery from Underspecified Real\-World Business Requests

Can HuangChufan WuDanyu LiuXiaomin YuanQinze LiYui LoWenhui BaiQianlong WangZengle GeXiyu YanYuheng ChengJinhui RenGuokai ChenJiaqun LiuHaoran LiQianhui LiuAnnan LiJianmin WuDawei YinDou ShenAffiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[

###### Abstract

Real\-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience\. Forward\-deployed engineers \(FDEs\) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria\. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks\. We introduce FDE\-Bench, the first benchmark built from real customer projects to evaluate agents on end\-to\-end delivery tasks that require interactive requirement clarification and iterative optimization\. Its 49 cases span combinatorial optimization and machine learning\. Clarification targets are distilled from discussions between frontline practitioners and FDEs\. Agents elicit missing requirements from a simulated customer, and case\-specific evaluators assess final artifacts against acceptance criteria using customer\-accepted FDE solutions as quality references\. Across 18 agent configurations, the highest mean delivery score is only0\.5510, compared with an FDE reference of1\.0\. The highest delivery success rate \(DSR\), defined as the fraction of planned runs reaching this reference, is only6\.12%\. These results show that reliable FDE\-level end\-to\-end delivery remains out of reach for the evaluated agents on underspecified real\-world business tasks\. The benchmark project page is available at[baidubce\.github\.io/FM\-Agent/FDE\-Bench](https://baidubce.github.io/FM-Agent/FDE-Bench/)\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author\.## 1Introduction

In real customer projects, frontline practitioners often rely on business rules learned through experience but struggle to make them explicit in their requests\[[1](https://arxiv.org/html/2610.14539#bib.bib1)\]\. Forward\-deployed engineers \(FDEs\) must elicit these tacit requirements through dialogue and iteratively optimize solutions until they meet the customer’s acceptance criteria\[[2](https://arxiv.org/html/2610.14539#bib.bib2)\]\. Figure[1](https://arxiv.org/html/2610.14539#S1.F1)\([1\(a\)](https://arxiv.org/html/2610.14539#S1.F1.sf1)\) illustrates a warehouse request to keep each stack to “one product type” where possible while avoiding unnecessary stock moves\. The request leaves implicit that different batches of the same product \(SKU\) also count as mixed\. Grouping only by SKU can therefore leave mixed stacks under the customer’s criterion\. After clarifying this rule, the agent must still iteratively refine storage assignments and moves to meet the delivery requirements\.

\(a\)A warehouse case: clarify the mixing rule, then optimize\.\(b\)Overview of FDE\-Bench:49 real customer cases across task families and subtypes\.
Figure 1:FDE\-Bench: motivation and coverage\.\(a\) Clarifying mixing rules guides iterative warehouse optimization\. \(b\) 49 real customer cases span two task families\.Representative benchmarks do not yet fully capture this challenge\. Optimization benchmarks such as HeuriGym\[[3](https://arxiv.org/html/2610.14539#bib.bib3)\]and Frontier\-Eng\[[4](https://arxiv.org/html/2610.14539#bib.bib4)\]provide explicit objectives and constraints, leaving requirement acquisition outside their scope\. ClarEval\[[5](https://arxiv.org/html/2610.14539#bib.bib5)\]perturbs programming instructions to create omissions and ambiguities, while HIL\-Bench\[[6](https://arxiv.org/html/2610.14539#bib.bib6)\]introduces human\-validated blockers requiring interaction in coding and SQL tasks\. Their information gaps are constructed within predefined tasks; we instead ask whether agents can elicit requirements grounded in frontline practitioners’ experience and iteratively optimize solutions to meet real customers’ delivery standards\.

We introduce FDE\-Bench, a benchmark of 49 cases from real customer projects for evaluating agents from requirement clarification through iterative optimization to final delivery\. As summarized in Figure[1](https://arxiv.org/html/2610.14539#S1.F1)\([1\(b\)](https://arxiv.org/html/2610.14539#S1.F1.sf2)\), the cases cover 29 combinatorial\-optimization and 20 machine\-learning tasks across diverse business settings\. Each case links the original request and business data, requirements distilled from frontline practitioner–FDE discussions, and a customer\-accepted FDE solution\. Agents clarify requirements with a simulated customer, then iteratively execute and improve their solutions\. After submission, case\-specific evaluators assess final artifacts against the project’s acceptance criteria, using the accepted FDE solution as the quality reference\. Separate clarification scores measure how well agents’ questions cover the annotated requirements\. Since performance depends on both the model and its scaffold\[[7](https://arxiv.org/html/2610.14539#bib.bib7)\], we build an extensible evaluation harness to compare different configurations under a shared protocol\.

We evaluate 18 agent configurations spanning seven model families and six agent scaffolds\. Across configurations, the highest mean delivery score is only 0\.5510 against a customer\-accepted FDE reference of 1\.0\. The highest delivery success rate \(DSR\)—the fraction of planned runs reaching that reference—is only 6\.12%\. Broader requirement coverage in agents’ questions is associated with higher delivery quality\. Paired information comparisons show higher scores with required clarification than with no customer answers, but even complete business information leaves all compared configurations below the FDE reference on average\. These results expose gaps in both requirement acquisition and solution optimization\.

Our contributions are:

- •A benchmark for end\-to\-end FDE delivery\.FDE\-Bench is the first benchmark to couple interactive requirement clarification with iterative optimization on real customer projects, using customer\-accepted FDE solutions as delivery references\.
- •An extensible evaluation harness\.Backend adapters support new models and agent scaffolds; a shared protocol records interactions and evaluates submitted artifacts, with separate measures of requirement coverage and delivery quality\.
- •A systematic study of current agents\.Across 18 configurations, we analyze representative delivery failures and the association between requirement coverage and delivery quality\. Information ablations show that complete business information alone does not close the delivery gap\.

## 2Related Work

Optimization modeling and search\.NL4Opt and OptiMUS study natural\-language optimization modeling, while OptiBench and OptMATH provide data for evaluating or training modeling systems\[[8](https://arxiv.org/html/2610.14539#bib.bib8),[9](https://arxiv.org/html/2610.14539#bib.bib9),[10](https://arxiv.org/html/2610.14539#bib.bib10),[11](https://arxiv.org/html/2610.14539#bib.bib11)\]\. MIPLIB\-NL extends evaluation to industrial\-scale mixed\-integer problems\[[12](https://arxiv.org/html/2610.14539#bib.bib12)\]\. CO\-Bench, HeuriGym, and ALE\-Bench evaluate iterative algorithm search and optimization\[[13](https://arxiv.org/html/2610.14539#bib.bib13),[3](https://arxiv.org/html/2610.14539#bib.bib3),[14](https://arxiv.org/html/2610.14539#bib.bib14)\], while ORAgentBench and Opti\-Agent\-Bench extend evaluation to workflows involving problem formulation, file inspection, and execution\[[15](https://arxiv.org/html/2610.14539#bib.bib15),[16](https://arxiv.org/html/2610.14539#bib.bib16)\]\. These works primarily assess how agents formulate and solve given optimization tasks\. FDE\-Bench additionally evaluates whether agents can elicit unstated business requirements through customer interaction and use them to guide optimization\.

Clarification and interactive agents\.MINT evaluates multi\-turn tool use with language feedback, whileτ\\tau\-bench studies customer interaction and policy compliance in API\-based service tasks\[[17](https://arxiv.org/html/2610.14539#bib.bib17),[18](https://arxiv.org/html/2610.14539#bib.bib18)\]\. CLAMBER focuses on identifying ambiguous information needs and generating clarifying questions\[[19](https://arxiv.org/html/2610.14539#bib.bib19)\]\. HIL\-Bench introduces blockers into coding and SQL tasks, while ClarEval injects ambiguities into programming instructions\[[6](https://arxiv.org/html/2610.14539#bib.bib6),[5](https://arxiv.org/html/2610.14539#bib.bib5)\]\. Ambig\-SWE evaluates clarification in software tasks\[[20](https://arxiv.org/html/2610.14539#bib.bib20)\], while[Pan et al\. \[21\]](https://arxiv.org/html/2610.14539#bib.bib21)study code generation with interactive feedback\. SWE\-RPG diagnoses implicit\-requirement recovery from coding trajectories but does not evaluate interactive user feedback\[[22](https://arxiv.org/html/2610.14539#bib.bib22)\]\. FDE\-Bench grounds clarification targets in discussions between frontline practitioners and FDEs from real customer projects, then evaluates delivery quality after agents clarify requirements and iteratively optimize their solutions\.

Executable evaluation and expert references\.SWE\-bench and Terminal\-Bench use executable verification to assess software and command\-line tasks\[[23](https://arxiv.org/html/2610.14539#bib.bib23),[24](https://arxiv.org/html/2610.14539#bib.bib24)\]\. Frontier\-Eng and EngDesign evaluate engineering solutions through task\-specific scoring or simulation\[[4](https://arxiv.org/html/2610.14539#bib.bib4),[25](https://arxiv.org/html/2610.14539#bib.bib25)\]\. MLE\-bench measures machine\-learning engineering performance, while RE\-Bench compares agents with human experts on ML research engineering tasks\[[26](https://arxiv.org/html/2610.14539#bib.bib26),[27](https://arxiv.org/html/2610.14539#bib.bib27)\]\. FDE\-Bench also uses executable evaluation and reference solutions\. Its distinguishing feature is that the initial request, clarification targets, and customer\-accepted FDE reference originate from the same customer project\. Agents must elicit initially unstated requirements and optimize their deliverables against that project’s acceptance criteria\. Appendix[A\.1](https://arxiv.org/html/2610.14539#A1.SS1)provides a comparison with representative benchmarks\.

## 3FDE\-Bench: Tasks and Construction

### 3\.1FDE Delivery Workflow

Each case draws on the FDE delivery workflow of a real customer project \(Figure[2](https://arxiv.org/html/2610.14539#S3.F2)\)\. Frontline practitioners provide an underspecified business request and business data\. Through discussion, the FDE elicits business rules and acceptance criteria, then iteratively optimizes a solution until customer acceptance\. The initial request and data, project discussions, and accepted FDE solution from the same project supply task inputs, requirement annotations, and delivery references, respectively\. The original discussion transcripts remain hidden from evaluated agents\.

Figure 2:FDE\-Bench construction pipeline\.Real\-project materials become agent\-visible inputs, annotated requirements, case\-specific evaluation, and a reviewed case package\.
### 3\.2Benchmark Construction Pipeline

##### Prepare task inputs\.

The initial request and business data are de\-identified with agent assistance and reviewed by an FDE\. The process preserves the request’s original meaning and level of detail, applies replacements consistently across related files, and retains numerical relationships and constraints needed for execution and evaluation\. The visible inputs areinstruction\.mdanddata/; the de\-identification agent does not decide requirements or acceptance criteria\.

##### Distill requirements\.

The FDE responsible for the original delivery annotates requirements established in the customer–FDE discussions\. We store complete business context ininformation\.mdand clarification targets ingt\.json\. A target is retained only if it cannot be resolved from the initial request and visible data, and its resolution can affect feasibility, the objective or target value, or customer acceptance\. Examples include the meaning of a data field, the scope of a constraint, and the convention used to measure an objective\. Each annotation records the requirement and the information to clarify, without prescribing question wording or dialogue order\. The 49 cases contain 266 targets, or 5\.43 per case on average\.

##### Establish delivery evaluation\.

Complete requirements and project acceptance criteria are translated into a case\-specific evaluator that checks the artifact schema, hard constraints, and case\-defined quality\. The customer\-accepted FDE deliverable supplies the reference, normalized to 1\.0\. This is an achieved project standard rather than a claim of global optimality; different algorithms are therefore judged against a common set of requirements\.

##### Human review and selection\.

Reviewers run each FDE reference solution with its evaluator and check that the evaluation matches the documented business requirements and acceptance criteria\. They also review the FDE annotations using common criteria across cases, checking targets against the evaluator and reference artifact, removing duplicates, and confirming that each target has a documented customer answer\. Cases with incomplete materials or inconsistent checks are excluded\. To select projects involving sustained delivery work, we retain those that required more than one month of FDE effort on clarification and iterative optimization before customer acceptance\. Reviewers select cases across computational task types and business settings to ensure coverage\.

### 3\.3Task Coverage and Specification

The release contains 29 combinatorial\-optimization and 20 machine\-learning cases, spanning manufacturing, logistics, energy, retail, finance, and engineering settings\. Primary computational subtypes are listed in the case inventory; scenario tags can overlap and are not mutually exclusive counts\.

We represent each case as𝒞=\(u,D,I,G,E\)\\mathcal\{C\}=\(u,D,I,G,E\), comprising the initial requestuu, business dataDD, complete business\-information fileII, clarification targetsGG, and case\-specific evaluatorEE\. Each released case package also includes an FDE reference deliverable and its reference quality, an execution environment, and metadata\. Under the main condition,Interact\-Req, onlyuuandDDare initially exposed in the solving workspace;II,GG,EE, and the reference deliverable remain outside it\. Section[4](https://arxiv.org/html/2610.14539#S4)specifies how agents interact with the simulated customer and how their submitted artifacts and clarification questions are evaluated\.

## 4Evaluation Protocol

FDE\-Bench evaluates two outputs of each run: the submitted solution file \(the*artifact*\) and the interaction log\. Case\-specific evaluators score delivery, while a separate judge panel assesses clarification coverage \(Figure[3](https://arxiv.org/html/2610.14539#S4.F3)\)\. Section[5\.4](https://arxiv.org/html/2610.14539#S5.SS4)compares the main condition,Interact\-Req, with no\-answer and full\-information conditions\.

![Refer to caption](https://arxiv.org/html/2610.14539v1/pic4.png)Figure 3:FDE\-Bench evaluation framework\.Agents clarify requirements and iteratively optimize deliverables\. After submission, hidden evaluation extracts and scores artifacts against FDE references without online feedback; clarification coverage is assessed separately\.### 4\.1Evaluation Procedure

##### Customer interaction\.

Agents query a simulated customer for business information\. The customer uses the requestuu, complete business informationII, and dialogue history to answer, without access to clarification targetsGG\. We use GLM\-5\.3 with a fixed prompt and record exchanges\. Benchmark prompts and the native\-tool, callback, and text\-exchange interfaces are provided with the benchmark\.

##### Solving and submission\.

TheInteract\-Reqprompt instructs agents to elicit missing information before implementation, continuing until they consider the information sufficient\. Within the execution budget and interaction limits, agents choose their questions, implement a candidate solution, execute it locally, inspect the resulting files and diagnostics, and repeatedly revise and re\-execute the solution\. They submit the final artifact only after this local optimization loop\. The main campaign allows 43,200 seconds \(12 hours\) per solving run, including interaction, and at most 30 clarification rounds with no separate question\-count cap\. The hidden evaluator and artifact extractor run only after submission and provide no score or evaluator feedback during this loop\.

##### Extraction and case evaluation\.

After submission, extractors convert artifacts into the case evaluator’s input format while preserving the agent’s solution and decisions\. The evaluator checks the case’s hard constraints\. Violations receive a delivery score of zero; otherwise, it computes case\-defined quality and normalizes it to the customer\-accepted FDE reference, which scores 1\.0\. Up to five models—GLM\-5\.2, Qwen3\.8\-27B, DeepSeek\-V4\-Flash, Gemini\-3\.6\-Flash, and Grok\-4\.5—extract the artifact for scoring by the same case evaluator\. Remaining routes are skipped once three scores agree after rounding to four decimal places\. A score is recorded when at least three routes agree\. The validity outcome is taken from an evaluation supporting that score; this model\-panel design is related to prior work on LLM\-assisted evaluation\[[28](https://arxiv.org/html/2610.14539#bib.bib28),[29](https://arxiv.org/html/2610.14539#bib.bib29)\]\.

##### Requirement coverage\.

We split recorded questions into units and use the same five models to assess coverage ofGG\. For each requirement, judges independently assess whether answering at least one question would reveal it\. Judges receive the requirement and numbered questions, but not customer answers or requirement\-category metadata\. Coverage is accepted when at least three of five judgments agree \(≥3/5\\geq 3/5\); otherwise the requirement is uncovered\.

### 4\.2Evaluation Metrics

FDE tasks require agents to elicit implicit business requirements and turn them into a deliverable that meets the customer’s standard\. We measure clarification and delivery separately to examine whether effective requirement acquisition translates into successful delivery\.

##### Delivery metrics\.

In the main evaluation, each configuration runs three times on each of 49 cases\. Quality first averages the three delivery scores within each case, then averages across cases with equal weight\. Delivery Success Rate \(DSR\) is the fraction of the 147 runs reaching the FDE reference\. Letsi​rs\_\{ir\}denote the delivery score for repeatrrof caseii\. We define

Quality=149∑i=149\(13∑r=13si​r\),DSR=1147∑i=149∑r=13𝟏\[si​r≥1\]\.\\mathrm\{Quality\}=\\frac\{1\}\{49\}\\sum\_\{i=1\}^\{49\}\\left\(\\frac\{1\}\{3\}\\sum\_\{r=1\}^\{3\}s\_\{ir\}\\right\),\\qquad\\mathrm\{DSR\}=\\frac\{1\}\{147\}\\sum\_\{i=1\}^\{49\}\\sum\_\{r=1\}^\{3\}\\mathbf\{1\}\[s\_\{ir\}\\geq 1\]\.Following the pass@k framing in code\-generation evaluation\[[30](https://arxiv.org/html/2610.14539#bib.bib30)\], we also report pass@3 and pass^3: the fractions of cases with at least one and all three valid runs, respectively\. These measure validity across repeated runs; DSR measures attainment of the FDE reference\.

##### Clarification metrics\.

Missing business requirements can lead an agent to solve the wrong problem\. Recall \(RR\) measures how fully its questions cover the reference requirements, while Precision \(PP\) measures how focused the questions are on those requirements\. Their harmonic mean, Ask\-F1, rewards broad coverage with focused questioning, following the standard precision–recall terminology in information retrieval\[[31](https://arxiv.org/html/2610.14539#bib.bib31)\]\. For each run, letGcov⊆GG\_\{\\mathrm\{cov\}\}\\subseteq Gdenote covered requirements,QQthe set of question\-unit indices, andQrel⊆QQ\_\{\\mathrm\{rel\}\}\\subseteq Qthe indices matched to at least one covered requirement\. These three primary metrics are

R=\|Gcov\|\|G\|,P=\|Qrel\|\|Q\|,Ask​\-​F1=2​P​RP\+R\.R=\\frac\{\|G\_\{\\mathrm\{cov\}\}\|\}\{\|G\|\},\\qquad P=\\frac\{\|Q\_\{\\mathrm\{rel\}\}\|\}\{\|Q\|\},\\qquad\\mathrm\{Ask\\text\{\-\}F1\}=\\frac\{2PR\}\{P\+R\}\.Recall is question coverage: it measures whether a question could elicit a reference requirement, not whether the customer stated it or the agent used it; questions matching multiple requirements count once\.

Efficient clarification should elicit business requirements with few customer exchanges and limited question burden\. First\-turn coverage \(KQC1\\mathrm\{KQC\}\_\{1\}\) measures how much agents cover in the initial exchange, while average turns to clarify \(ATC\) measures how early they address the requirements eventually covered\. Unmatched\-question count \(Redund\.\) captures the absolute number of questions unmatched to covered requirements\. Interpreted alongside Recall, these diagnostics characterize the trade\-off between requirement coverage and interaction effort\. LetGcov,1⊆GG\_\{\\mathrm\{cov\},1\}\\subseteq Gdenote requirements covered in the first interaction turn, andGcov⋆⊆GcovG\_\{\\mathrm\{cov\}\}^\{\\star\}\\subseteq G\_\{\\mathrm\{cov\}\}those with a recorded matching question and turn\. For each requirementg∈Gcov⋆g\\in G\_\{\\mathrm\{cov\}\}^\{\\star\}, lettgt\_\{g\}be its first covering turn, counting from 1\. We define

KQC1=\|Gcov,1\|\|G\|,ATC=1\|Gcov⋆\|∑g∈Gcov⋆tg,Redund\.=\|Q\|−\|Qrel\|\.\\displaystyle\\mathrm\{KQC\}\_\{1\}=\\frac\{\|G\_\{\\mathrm\{cov\},1\}\|\}\{\|G\|\},\\qquad\\mathrm\{ATC\}=\\frac\{1\}\{\|G\_\{\\mathrm\{cov\}\}^\{\\star\}\|\}\\sum\_\{g\\in G\_\{\\mathrm\{cov\}\}^\{\\star\}\}t\_\{g\},\\qquad\\mathrm\{Redund\.\}=\|Q\|\-\|Q\_\{\\mathrm\{rel\}\}\|\.Clarification metrics are computed per run and averaged over runs for which each metric is defined\.

### 4\.3Evaluation Reliability

Two human experts independently audited stratified samples of retained traces, covering 1,303 simulated\-customer answers, 120 artifact extractions, and 120 clarification items across all 18 configurations\. Sampling procedures, confidence intervals, and full audit results are provided in Appendix[A\.8](https://arxiv.org/html/2610.14539#A1.SS8)\. For answer grounding, experts A and B marked 95\.93% and 90\.48% of answers as supported, respectively; their support labels agreed on 92\.10% of records \(Cohen’sκ=0\.385\\kappa=0\.385\[[32](https://arxiv.org/html/2610.14539#bib.bib32)\]\)\. For extraction fidelity, on a sample enriched for runs with extraction issues, they judged 93\.33% and 91\.00% of assessable extractions faithful to the submitted artifacts, respectively, with 91\.67% three\-category agreement \(Cohen’sκ=0\.7656\\kappa=0\.7656\)\. For clarification coverage, the model panel agreed with expert A on 94\.74% \(κ=0\.8947\\kappa=0\.8947\) and with B on 93\.33% \(κ=0\.8667\\kappa=0\.8667\), excluding uncertain labels\. These results support the reliability of the evaluation pipeline on the audited subsets; the extractionκ\\kappaspecifically measures expert\-label agreement, rather than extractor accuracy against an adjudicated gold standard\. These checks reduce but do not eliminate extraction\-related score risk\.

## 5Experiments

We evaluate 18 agent systems and ask three questions\.RQ1\(§[5\.2](https://arxiv.org/html/2610.14539#S5.SS2)\): how close do agents come to the accepted FDE deliverable?RQ2\(§[5\.3](https://arxiv.org/html/2610.14539#S5.SS3)\): which clarification behaviors distinguish stronger systems?RQ3\(§[5\.4](https://arxiv.org/html/2610.14539#S5.SS4)\): how does delivery quality change with access to business information?

### 5\.1Experimental Setup

We evaluate scaffold–model systems: general scaffoldsopencodeandopenhands, each with seven models \(GPT\-6\-Astra, GLM\-5\.3, Qwen3\.8\-Max, Grok\-4\.6, DeepSeek\-V4\-Pro, Kimi\-K3, Gemini\-3\.1\-Pro\), and four native pairs \(codex,deepseek\-harness,kimi,gemini\)\. All main runs useInteract\-Reqwith a fixed simulated customer, a 12\-hour budget, and at most 30 clarification rounds; each system solves each of 49 cases three times\. Paired contrasts use two\-sided Wilcoxon signed\-rank tests\[[33](https://arxiv.org/html/2610.14539#bib.bib33)\]on case means with Holm correction\[[34](https://arxiv.org/html/2610.14539#bib.bib34)\]\. Settings and bootstrap intervals are reported with the released results\.

### 5\.2RQ1: End\-to\-End Delivery

Table 1:End\-to\-end evaluation\.49 cases, three runs per case,Interact\-Req\. Quality is case\-macro; other metrics are percentages\. Bold marks blockwise maxima before rounding\.Delivery↑\\uparrowClarification↑\\uparrowAgent / modelQualityDSRpass@3pass^3RecallPrecisionAsk\-F1*General scaffold:opencode*GPT\-6\-Astra0\.54595\.4491\.8481\.6383\.6436\.5348\.82GLM\-5\.30\.54131\.3691\.8477\.5566\.8339\.7047\.46Qwen3\.8\-Max0\.53872\.7295\.9273\.4770\.5635\.4145\.11Grok\-4\.60\.52212\.7287\.7679\.5968\.1938\.3046\.42DeepSeek\-V4\-Pro0\.49861\.3691\.8471\.4355\.0837\.9342\.73Kimi\-K30\.41870\.0091\.8453\.0655\.0337\.6043\.27Gemini\-3\.1\-Pro0\.38730\.0089\.8059\.1847\.1039\.4840\.38*General scaffold:openhands*GPT\-6\-Astra0\.53024\.7693\.8867\.3584\.6736\.8049\.50GLM\-5\.30\.52742\.0491\.8467\.3567\.1236\.7945\.48Grok\-4\.60\.51590\.6891\.8471\.4369\.9136\.9545\.91DeepSeek\-V4\-Pro0\.51040\.6893\.8871\.4365\.2436\.4144\.92Qwen3\.8\-Max0\.50716\.1293\.8865\.3175\.3730\.8942\.24Gemini\-3\.1\-Pro0\.45710\.0087\.7665\.3145\.4537\.7739\.25Kimi\-K30\.41190\.6885\.7157\.1457\.9435\.3941\.34*Native scaffolds*codex\+ GPT\-6\-Astra0\.55104\.0893\.8875\.5184\.7037\.8750\.12deepseek\-harness\+ DeepSeek\-V4\-Pro0\.52711\.3691\.8475\.5161\.7939\.3245\.93kimi\+ Kimi\-K30\.42030\.6889\.8059\.1855\.6936\.6942\.90gemini\+ Gemini\-3\.1\-Pro0\.40370\.0087\.7657\.1444\.9438\.8239\.37##### FDE\-level delivery remains rare\.

Across the 18 systems, the highest mean Quality is only 0\.5510 \(codex\+ GPT\-6\-Astra\), and the highest DSR is 6\.12% \(Table[1](https://arxiv.org/html/2610.14539#S5.T1)\)\. Only 8 of 49 cases have any reference\-level run, and in 23 the best system averages below 0\.8\. Four systems, including all three Gemini\-3\.1\-Pro systems, reach the reference in none of their 147 runs, and no ML run reaches it in any system\.

##### Validity varies across repeated runs\.

Across systems, 85\.7–95\.9% of cases have at least one valid repeat among three attempts \(pass@3\)\. However, pass^3 trails pass@3 for every system by 8\.2–38\.8 percentage points \(mean 23\.0\): for a typical system, 23% of cases are valid in some runs but not in all three\. Of the 548 zero\-score runs, 78\.5% record a constraint or feasibility violation and 1\.5% a schema error \(Appendix[A\.7](https://arxiv.org/html/2610.14539#A1.SS7)\): agents submit well\-formed artifacts that break business rules\.

##### Runtime and case difficulty\.

Across systems, mean run time ranges from 0\.3 to 5\.5 hours, with no significant association with Quality \(Spearmanρ=0\.18\\rho=0\.18,p=0\.47p=0\.47\)\[[35](https://arxiv.org/html/2610.14539#bib.bib35)\]\. Case\-level Quality rankings are similar across systems \(mean pairwise Spearmanρ=0\.71\\rho=0\.71\)\.

##### The model matters more than the scaffold\.

With the scaffold fixed, Quality spans 0\.159 across models onopencodeand 0\.118 onopenhands; with the model fixed, the two general scaffolds differ by at most 0\.032 for six models \(Gemini\-3\.1\-Pro: 0\.387 vs\. 0\.457\)\. Native scaffolds add 0\.005 and 0\.017 for GPT\-6\-Astra and DeepSeek\-V4\-Pro, whilegeminitrailsopenhandsby 0\.053\.

### 5\.3RQ2: Clarification and Delivery

Figure[4](https://arxiv.org/html/2610.14539#S5.F4)correlates clarification and delivery metrics across the 18 systems\.

![Refer to caption](https://arxiv.org/html/2610.14539v1/pic5.png)Figure 4:Clarification–delivery correlations\.Cells report rank correlations across 18 configurations\. Stars mark Holm\-adjusted two\-sidedpp\-values over 24 comparisons; down arrows denote lower\-is\-better diagnostics\.##### Stronger systems cover more requirements; they do not ask more precisely\.

Recall ranges from 44\.9% to 84\.7%, while Precision stays at 30\.9–39\.7%; every system spends 60–69% of its question units on questions that match no covered requirement\. Recall and Ask\-F1 track Quality \(ρ=0\.83\\rho=0\.83and0\.880\.88, Holmp<0\.001p<0\.001\), and Recall tracks DSR \(ρ=0\.89\\rho=0\.89\), whereas Precision tracks no delivery metric \(Holmp=1\.0p=1\.0\)\. Within systems and cases, higher\-Recall repeats score higher in 45% and lower in 37% of 626 cells \(mean\+0\.031\+0\.031, Wilcoxonp=0\.012p=0\.012\)\. Coverage of each of the four requirement domains also correlates withQuality\(ρ=0\.81\\rho=0\.81–0\.830\.83\); domain\-level results and question\-type shares are detailed in Appendix[A\.5](https://arxiv.org/html/2610.14539#A1.SS5)\. These observational comparisons do not isolate the causal effect of clarification\.

##### Requirement coverage is concentrated in the first round\.

Average turns to clarify is 1\.01–1\.16, and first\-turn coverage is 91–99% of final Recall\. UnderInteract\-Req’s instruction to clarify before implementation, most covered requirements are addressed in the initial exchange\.

##### More questions do not consistently improve delivery\.

Unmatched\-question count correlates with DSR \(ρ=0\.75\\rho=0\.75, Holmp=0\.006p=0\.006\): systems that ask more cover more requirements and also ask more unmatched questions\. On the five most requirement\-dense cases, mandatory rather than optional clarification raises question count for all eight tested systems and Recall for seven, but Quality for only five; the changes range from−0\.125\-0\.125to\+0\.196\+0\.196and none is significant \(Holmp≥0\.50p\\geq 0\.50\)\.

##### Case study: asking about a rule does not ensure compliance\.

In case 10 \(retail replenishment\),opencode\+ GPT\-6\-Astra asks whether same\-city stock must be used before supplier purchases\. Its local validator reportspass, but the submitted plan purchases an item from a supplier while the local warehouse still holds 327 units\. The hidden evaluator identifies violations of this priority rule at eight stores, and the run receives Quality 0 \(Appendix[A\.7](https://arxiv.org/html/2610.14539#A1.SS7.SSS0.Px1)\)\. This run illustrates a gap between asking about a business rule and enforcing it in the final deliverable\.

### 5\.4RQ3: Information Ablation

Table[2](https://arxiv.org/html/2610.14539#S5.T2)comparesHidden\(request and data only\),Interact\-Req\(main runs reused\), andFull\(complete business information, no interaction\) on 20 cases and eight systems; it reports paired differences:Δreq=Interact\-Req−Hidden\\Delta\_\{\\rm req\}=\\texttt\{Interact\-Req\}\-\\texttt\{Hidden\}andΔremain=Full−Interact\-Req\\Delta\_\{\\rm remain\}=\\texttt\{Full\}\-\\texttt\{Interact\-Req\}\.

Table 2:Information ablation\.Case\-macro Quality on 20 paired cases \(three runs/condition\)\. Bold marks maxima\. Intervals and adjusted tests are in Appendix[A\.6](https://arxiv.org/html/2610.14539#A1.SS6.SSS0.Px2)\.Agent / modelHiddenInteract\-ReqFull𝚫𝐫𝐞𝐪\\bm\{\\Delta\_\{\\rm req\}\}𝚫𝐫𝐞𝐦𝐚𝐢𝐧\\bm\{\\Delta\_\{\\rm remain\}\}*General scaffold:opencode*opencode\+ GPT\-6\-Astra0\.18030\.58000\.7461\+0\.3997\+0\.1662opencode\+ DeepSeek\-V4\-Pro0\.14230\.50630\.6821\+0\.3640\+0\.1758opencode\+ Gemini\-3\.1\-Pro0\.12180\.29120\.4886\+0\.1694\+0\.1974opencode\+ Kimi\-K30\.12440\.36940\.4488\+0\.2450\+0\.0794*Native scaffolds*codex\+ GPT\-6\-Astra0\.19800\.59800\.7946\+0\.4000\+0\.1966deepseek\-harness\+ DeepSeek\-V4\-Pro0\.19980\.56420\.7339\+0\.3644\+0\.1697gemini\+ Gemini\-3\.1\-Pro0\.12310\.37630\.5184\+0\.2532\+0\.1420kimi\+ Kimi\-K30\.14980\.36930\.7184\+0\.2195\+0\.3490##### Interaction closes 39–76% of the score gap\.

UnderHidden, systems score 0\.12–0\.20;Interact\-Reqadds\+0\.17\+0\.17–\+0\.40\+0\.40\(significant for all eight, Holm over 24 tests\), recovering 64–76% of theHidden\-to\-FullQuality gap for six systems, 46% foropencode\+ Gemini\-3\.1\-Pro, and 39% forkimi\+ Kimi\-K3\.FullexceedsInteract\-Reqfor all eight \(\+0\.08\+0\.08–\+0\.35\+0\.35; significant for five\), although it removes interaction\.

##### Full information leaves a substantial delivery gap\.

UnderFull, mean Quality ranges from 0\.449 to 0\.795, leaving all eight systems below the FDE reference of 1\.0\. Providing complete business information improves delivery but is insufficient to reach the reference on average\.

Figure 5:Oracle information coverage\.Codex \+ GPT\-6\-Astra receives nested answer subsets; panels \(a\)–\(d\) show four cases and \(e\) their macro mean\.Oracle returns are nonlinear and case\-dependent\. We givecodex\+ GPT\-6\-Astra nested random subsets of the registered answers on four cases, without interaction\. At 100% coverage, agents receive all registered answers, not the complete business\-information file used byFull\. Mean Quality rises from 0\.059 at 0% coverage to 0\.352 at 50% and 0\.869 at 100%; the last quarter of answers yields half of the gain\. Case 38 scores zero in every run up to 50% coverage and 0\.965 at full coverage, and at intermediate coverage cases 09 and 38 obtain a nonzero score in one of three repeats and score zero in the others \(Figure[5](https://arxiv.org/html/2610.14539#S5.F5)\)\. A few unresolved requirements can void an otherwise reasonable solution, which is consistent with RQ1: the highest Recall is 84\.7%, yet no system exceeds 6\.12% DSR\.

## 6Conclusion

We introduced FDE\-Bench to evaluate agents’ end\-to\-end delivery from underspecified real\-world business requests\. Its 49 cases use customer\-accepted FDE solutions as delivery references\. Across 18 configurations, the highest mean Quality is only 0\.5510 against an FDE reference of 1\.0\. The highest delivery success rate is only 6\.12%\. Broader requirement coverage is associated with higher delivery quality, but even complete business information leaves all compared configurations below the FDE reference on average\. Reliable FDE\-level delivery therefore requires progress in both requirement clarification and solution optimization\. FDE\-Bench provides a shared testbed for comparing new agent systems and examining whether better requirement elicitation translates into better delivery\.

## 7Limitations and Future Work

FDE\-Bench evaluates delivery with stable business requirements and fixed datasets\. The simulated customer does not revise its requirements in response to intermediate solutions, whereas real customers may reveal or change expectations during development\[[2](https://arxiv.org/html/2610.14539#bib.bib2)\]\. The benchmark therefore does not yet assess agents’ ability to maintain delivery quality as production data and customer requirements evolve\. Coverage is limited to 49 real customer projects in combinatorial optimization and machine learning\. These cases do not cover all FDE settings\.

We plan to expand FDE\-Bench with additional real customer projects across more domains, including scientific and engineering applications\. Selection will prioritize task types and business settings that are underrepresented in the current release\. New cases will undergo the same requirement review and reference\-solution checks\. Future evaluations should incorporate continuously updated production data, evolving requirements, and human customers, testing both sustained delivery quality and the realism of simulated interactions\. Evaluation in these settings should assess successive deliverables against the customer’s current acceptance criteria and track how agents respond to requirement changes\.

## References

- \[1\]Bashar Nuseibeh and Steve Easterbrook\.Requirements engineering: a roadmap\.In*Proceedings of the Conference on the Future of Software Engineering*, pages 35–46, 2000\.
- \[2\]Didar Zowghi and Christine Coulin\.Requirements elicitation: A survey of techniques, approaches, and tools\.In*Engineering and managing software requirements*, pages 19–46\. Springer, 2005\.
- \[3\]Hongzheng Chen, Yingheng Wang, Yaohui Cai, Hins Hu, Jiajie Li, Shirley Huang, Chenhui Deng, Rongjian Liang, Shufeng Kong, Haoxing Ren, et al\.Heurigym: An agentic benchmark for llm\-crafted heuristics in combinatorial optimization\.In*International Conference on Learning Representations*, volume 2026, pages 27466–27520, 2026\.
- \[4\]Yizhe Chi, Deyao Hong, Dapeng Jiang, Tianwei Luo, Kaisen Yang, Boshi Zhang, Zhe Cao, Xiaoyan Fan, Bingxiang He, Han Hao, et al\.Frontier\-eng: Benchmarking self\-evolving agents on real\-world engineering tasks with generative optimization\.*arXiv preprint arXiv:2604\.12290*, 2026\.
- \[5\]Jialin Li, Yuan Wu, and Yi Chang\.Clareval: A benchmark for evaluating clarification skills of code agents under ambiguous instructions\.*arXiv preprint arXiv:2603\.00187*, 2026a\.
- \[6\]Tu Trinh, Mohamed Elfeki, Guangze Luo, Kelvin Luu, Nathan Hunt, Ernesto Hernandez, Nandan Marwaha, Yannis Yiming He, Charles Wang, Fernando Carabedo, et al\.Hil\-bench \(human\-in\-loop benchmark\): Do agents know when to ask for help?*arXiv preprint arXiv:2604\.09408*, 2026\.
- \[7\]Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, et al\.Holistic agent leaderboard: The missing infrastructure for ai agent evaluation\.In*International Conference on Learning Representations*, volume 2026, pages 98778–98849, 2026\.
- \[8\]Rindranirina Ramamonjison, Timothy Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghaddar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi\-Dehkordi, Zirui Zhou, et al\.Nl4opt competition: Formulating optimization problems based on their natural language descriptions\.In*NeurIPS 2022 competition track*, pages 189–203\. PMLR, 2023\.
- \[9\]Ali Ahmaditeshnizi, Wenzhi Gao, and Madeleine Udell\.OptiMUS: Scalable optimization modeling with \(MI\)LP solvers and large language models\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235, pages 577–596\. PMLR, 21–27 Jul 2024\.
- \[10\]Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Shi Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang\.Optibench meets resocratic: Measure and improve llms for optimization modeling\.In*International Conference on Learning Representations*, volume 2025, pages 24726–24759, 2025\.
- \[11\]Hongliang Lu, Zhonglin Xie, Yaoyu Wu, Can Ren, Yuxuan Chen, and Zaiwen Wen\.OptMATH: A scalable bidirectional data synthesis framework for optimization modeling\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267, pages 40769–40802\. PMLR, 13–19 Jul 2025\.
- \[12\]Zhong Li, Hongliang Lu, Tao Wei, Yuxuan Chen, Wenyu Liu, Yuan Lan, Fan Zhang, and Zaiwen Wen\.Constructing industrial\-scale optimization modeling benchmark\.In*Forty\-third International Conference on Machine Learning*, 2026b\.
- \[13\]Weiwei Sun, Shengyu Feng, Shanda Li, and Yiming Yang\.Co\-bench: Benchmarking language model agents in algorithm search for combinatorial optimization\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pages 33126–33134, 2026\.
- \[14\]Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba\.Ale\-bench: A benchmark for long\-horizon objective\-driven algorithm engineering\.*Advances in Neural Information Processing Systems*, 38, 2026\.
- \[15\]Jiajun Li, Mingshu Cai, Yixuan Li, Yu Ding, Ran Hou, Guanyu Nie, Xiongwei Han, and Wanyuan Wang\.Oragentbench: Can llm agents solve challenging operations research tasks end to end?*arXiv preprint arXiv:2606\.19787*, 2026c\.
- \[16\]Yongchang Fu, Xinjie Huang, Chengjun Dai, Chengzhe Feng, Junshao Zhang, and Hong Zhu\.Opti\-agent\-bench: Benchmarking end\-to\-end optimization r&d agents on real\-world business problems\.*arXiv preprint arXiv:2607\.10768*, 2026\.
- \[17\]Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji\.Mint: Evaluating llms in multi\-turn interaction with tools and language feedback\.In*International Conference on Learning Representations*, volume 2024, pages 32593–32627, 2024\.
- \[18\]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\.τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains\.In*13th International Conference on Learning Representations, ICLR 2025*, pages 74824–74876, 2025\.
- \[19\]Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat\-Seng Chua\.Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 10746–10766, 2024\.
- \[20\]Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig\.Ambig\-swe: Interactive agents to overcome underspecificity in software engineering\.In*International Conference on Learning Representations*, volume 2026, pages 146716–146737, 2026\.
- \[21\]Jane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, and Valerie Chen\.When benchmarks talk: Re\-evaluating code llms with interactive feedback\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 24672–24700, 2025\.
- \[22\]Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, and David Lo\.SWE\-RPG: A unified issue resolution benchmark for requirement clarification, planning, and code generation for coding agents\.*arXiv preprint arXiv:2608\.09072*, 2026\.
- \[23\]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan\.Swe\-bench: Can language models resolve real\-world github issues?In*International Conference on Learning Representations*, volume 2024, pages 54107–54157, 2024\.
- \[24\]Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al\.Terminal\-bench: Benchmarking agents on hard, realistic tasks in command line interfaces\.In*International Conference on Learning Representations*, volume 2026, pages 40903–40986, 2026\.
- \[25\]Xingang Guo, Yaxin Li, Xiangyi Kong, Yilan Jiang, Xiayu Zhao, Zhihua Gong, Yufan Zhang, Daixuan Li, Tianle Sang, Beixiao Zhu, et al\.Toward engineering agi: Benchmarking the engineering design capabilities of llms\.*Advances in Neural Information Processing Systems*, 38, 2026\.
- \[26\]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al\.Mle\-bench: Evaluating machine learning agents on machine learning engineering\.In*International Conference on Learning Representations*, volume 2025, pages 50466–50494, 2025\.
- \[27\]Hjalmar Wijk, Tao Roa Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Joshua M Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Jun Koba Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes\.RE\-bench: Evaluating frontier AI r&d capabilities of language model agents against human experts\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267, pages 66772–66832\. PMLR, 13–19 Jul 2025\.
- \[28\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.*Advances in neural information processing systems*, 36:46595–46623, 2023\.
- \[29\]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\.G\-eval: Nlg evaluation using gpt\-4 with better human alignment\.In*Proceedings of the 2023 conference on empirical methods in natural language processing*, pages 2511–2522, 2023\.
- \[30\]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- \[31\]Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan\.*Introduction to information retrieval*, volume 39\.Cambridge University Press Cambridge, 2008\.
- \[32\]Jacob Cohen\.A coefficient of agreement for nominal scales\.*Educational and psychological measurement*, 20\(1\):37–46, 1960\.
- \[33\]Frank Wilcoxon\.Individual comparisons by ranking methods\.*Biometrics bulletin*, 1\(6\):80–83, 1945\.
- \[34\]Sture Holm\.A simple sequentially rejective multiple test procedure\.*Scandinavian journal of statistics*, pages 65–70, 1979\.
- \[35\]Charles Spearman\.*The proof and measurement of association between two things\.*Appleton\-Century\-Crofts, 1961\.
- \[36\]Bradley Efron\.Bootstrap methods: another look at the jackknife\.In*Breakthroughs in statistics: Methodology and distribution*, pages 569–593\. Springer, 1992\.

## Appendix

## Appendix AAdditional Evaluation Details

PositioningBenchmark comparison \([A\.1](https://arxiv.org/html/2610.14539#A1.SS1)\)\.CasesCase map, scoring references, and clarification points \([A\.2](https://arxiv.org/html/2610.14539#A1.SS2)\)\.ProtocolDelivery and process metrics and runtime settings \([A\.3](https://arxiv.org/html/2610.14539#A1.SS3)\)\.ResultsFull results \([A\.4](https://arxiv.org/html/2610.14539#A1.SS4)\), clarification types \([A\.5](https://arxiv.org/html/2610.14539#A1.SS5)\), ablations \([A\.6](https://arxiv.org/html/2610.14539#A1.SS6)\), and zero scores with a case study \([A\.7](https://arxiv.org/html/2610.14539#A1.SS7)\)\.ReliabilityCustomer, judge, and extraction audits \([A\.8](https://arxiv.org/html/2610.14539#A1.SS8)\)\.PromptsAgent, customer, extractor, and judge templates \([A\.9](https://arxiv.org/html/2610.14539#A1.SS9)\)\.

### A\.1Benchmark comparison

Table[3](https://arxiv.org/html/2610.14539#A1.T3)compares FDE\-Bench with representative benchmarks and systems \(Section[2](https://arxiv.org/html/2610.14539#S2)\)\. Requirements denotes implicit business requirements annotated from customer–engineer discussions in real delivery projects\. Reference denotes a customer\-accepted project solution used as a delivery\-quality baseline\. These definitions concern the provenance of the requirements and reference; RE\-Bench, for example, provides human expert comparisons and reference solutions, but not customer\-accepted project deliverables\.

Table 3:Comparison with representative benchmarks and systems\.✓: primary design feature; ✗: not a primary design feature\. Requirements and Reference are defined above\. Clarification: evaluation of clarifying questions or the use of user feedback; Execution: task outcomes verified through code or tool execution\.Benchmarks / systemsTask typeRequirementsClarificationExecutionReferenceMIPLIB\-NL\[[12](https://arxiv.org/html/2610.14539#bib.bib12)\]Optimization modeling✗✗✓✗OptiMUS\[[9](https://arxiv.org/html/2610.14539#bib.bib9)\]Optimization modeling✗✗✓✗OptiBench\[[10](https://arxiv.org/html/2610.14539#bib.bib10)\]Optimization modeling✗✗✓✗CO\-Bench\[[13](https://arxiv.org/html/2610.14539#bib.bib13)\]CO search✗✗✓✗HeuriGym\[[3](https://arxiv.org/html/2610.14539#bib.bib3)\]CO search✗✗✓✗ALE\-Bench\[[14](https://arxiv.org/html/2610.14539#bib.bib14)\]CO search✗✗✓✗ORAgentBench\[[15](https://arxiv.org/html/2610.14539#bib.bib15)\]End\-to\-end OR✗✗✓✗Opti\-Agent\-Bench\[[16](https://arxiv.org/html/2610.14539#bib.bib16)\]End\-to\-end OR✗✗✓✗MINT\[[17](https://arxiv.org/html/2610.14539#bib.bib17)\]Interactive agents✗✓✓✗τ\\tau\-bench\[[18](https://arxiv.org/html/2610.14539#bib.bib18)\]Interactive agents✗✓✓✗CLAMBER\[[19](https://arxiv.org/html/2610.14539#bib.bib19)\]Clarification/HITL✗✓✗✗HIL\-Bench\[[6](https://arxiv.org/html/2610.14539#bib.bib6)\]Clarification/HITL✗✓✓✗ClarEval\[[5](https://arxiv.org/html/2610.14539#bib.bib5)\]Clarification/HITL✗✓✓✗SWE\-RPG\[[22](https://arxiv.org/html/2610.14539#bib.bib22)\]Requirement diagnosis✗✗✓✗SWE\-bench\[[23](https://arxiv.org/html/2610.14539#bib.bib23)\]Execution agents✗✗✓✗Terminal\-Bench\[[24](https://arxiv.org/html/2610.14539#bib.bib24)\]Execution agents✗✗✓✗Frontier\-Eng\[[4](https://arxiv.org/html/2610.14539#bib.bib4)\]Execution agents✗✗✓✗EngDesign\[[25](https://arxiv.org/html/2610.14539#bib.bib25)\]Execution agents✗✗✓✗MLE\-bench\[[26](https://arxiv.org/html/2610.14539#bib.bib26)\]ML/R&D✗✗✓✗RE\-Bench\[[27](https://arxiv.org/html/2610.14539#bib.bib27)\]ML/R&D✗✗✓✗FDE\-Bench \(Ours\)CO \+ ML✓✓✓✓
### A\.2Cases and clarification points

##### Case map\.

Figure 6:Case map\.The 49 cases by task family and subtype, numbered 01–49\.Each case has one primary subtype for reporting; Figure[6](https://arxiv.org/html/2610.14539#A1.F6)gives the counts\.

##### Scoring references\.

Table[4](https://arxiv.org/html/2610.14539#A1.T4)lists each case’s raw objective, hard constraints, normalization direction, and FDE reference valueRR\.

Table 4:Case\-level evaluator reference\.Numbering follows Figure[6](https://arxiv.org/html/2610.14539#A1.F6)\.No\.CaseFam\.Raw objectiveHard constraintsNorm\. / directionFDERR01Vehicle\-drag predictionMLmean relativeCdC\_\{d\}error \(Cd​\_​MRE\\mathrm\{Cd\\\_MRE\}\)coverage, unique finite rows;0\.05≤Cd≤1\.50\.05\\leq\\mathrm\{Cd\}\\leq 1\.5R/xR/x\[\-1pt\]\(min\)0\.0035752702City delivery routingCOcombined route cost \(distance/time/penalties\)all required stops, feasible routes and valid timestampsR/xR/x\[\-1pt\]\(min\)28149703Crop\-yield predictionMLyearly mean RMSErequired predictions, finite values and complete test coverageR/xR/x\[\-1pt\]\(min\)3\.7728104Electricity\-price forecastingMLweighted MAE/p95/p99/high\-price underprediction errorcomplete horizon, finite values and aligned timestampsR/xR/x\[\-1pt\]\(min\)0\.4800405Flight\-booking forecastingMLseat\-booking forecast scorecomplete horizon, finite nonnegative forecastsx/Rx/R\[\-1pt\]\(max\)0\.16065306Ion\-beam uniformityMLmean WAPE\-derived uniformity scoreall required rows, finite predictions and valid boundsx/Rx/R\[\-1pt\]\(max\)0\.6457207Lubricant formulationMLweighted formulation quality \(property/constraint terms\)valid formulation schema, bounds and required compositionx/Rx/R\[\-1pt\]\(max\)98\.878308Material\-property predictionMLweighted material\-property prediction qualitycomplete test coverage and finite predictionsx/Rx/R\[\-1pt\]\(max\)0\.94533409Nuclear\-site schedulingCOweighted construction schedule scoreschema, precedence/resource feasibility and horizon limitsx/Rx/R\[\-1pt\]\(max\)97\.204410Retail stock replenishmentCOfulfillment/lateness/cost absolute scorecomplete orders and feasible inventory/transfer recordsx/Rx/R\[\-1pt\]\(max\)0\.99747611Retail\-zone routingCOpartition plus routing composite qualityall zones/items covered and routing/partition checks passx/Rx/R\[\-1pt\]\(max\)0\.83733712Steel slitting and cuttingCOcut utilization and cut\-count scorevalid cut plan, demand coverage and capacity checksx/Rx/R\[\-1pt\]\(max\)1\.664513Vessel stowage planningCOweighted vessel\-stowage objective componentsschema, placement, capacity and stability checksx/Rx/R\[\-1pt\]\(max\)0\.50087814Warehouse stock levelsCOmean daily policy score minus variability penaltyvalid daily policy schema and simulator checksx/Rx/R\[\-1pt\]\(max\)105\.69715Truck\-load planningCO0\.600\.60coverage\+0\.25\+0\.25length utilization\+0\.15\+0\.15balanceall items assigned once and truck/capacity checksx/Rx/R\[\-1pt\]\(max\)116Multi\-table APS schedulingCOdelay/utilization absolute schedule scoreall jobs assigned and precedence/resource constraintsx/Rx/R\[\-1pt\]\(max\)0\.11444617Cold\-rolling sequencingCOcoil\-sequencing penaltycomplete sequence, no duplicates and transition feasibilityR/xR/x\[\-1pt\]\(min\)10718\.918Multi\-batch schedulingCOmulti\-batch scheduling composite scoreschema, assignment, precedence and capacity checksx/Rx/R\[\-1pt\]\(max\)0\.99843819Multi\-line schedulingCOtotal production penaltycomplete schedule and line/time feasibilityR/xR/x\[\-1pt\]\(min\)63760020Multi\-warehouse shippingCOlexicographic shipping score with tie\-break costall orders served and inventory/route constraintsx/Rx/R\[\-1pt\]\(max\)0\.67440421Soybean\-meal reschedulingCOweighted smoothness/safety/balance rescheduling scorevalid schedule and safety/quality checksx/Rx/R\[\-1pt\]\(max\)87\.06122Wheel\-hub schedulingCOtotal weighted delaytwo\-stage schedule complete and all hard constraintsR/xR/x\[\-1pt\]\(min\)92423Auction\-picking routesCOpicking\-route distanceall required picks served with valid routeR/xR/x\[\-1pt\]\(min\)23773824Quay\-crane schedulingCO100,000,000​ncranes100\{,\}000\{,\}000\\,n\_\{\\rm cranes\}\+\+total move secondsall hard quay\-crane constraints C1–C7R/xR/x\[\-1pt\]\(min\)1\.1e925Container loadingCOcontainer stability/loading qualityall items placed once; weight/geometry/stability checksx/Rx/R\[\-1pt\]\(max\)62\.014126Three\-SKU packingCOvalidated ITEM\_C count/valuevalid container schema and packing/weight checksx/Rx/R\[\-1pt\]\(max\)148\.00627Flight\-crew rosteringCOcrew\-roster combined scorecomplete roster and labor/rest/qualification rulesx/Rx/R\[\-1pt\]\(max\)4624\.9828Elevator\-call forecastingML100/\(1\+MAE\)100/\(1\+\\mathrm\{MAE\}\)minus skew penaltycomplete calls, finite nonnegative predictionsx/Rx/R\[\-1pt\]\(max\)21\.727729Hydropower dispatchCOtotal cascade\-hydropower energycomplete dispatch, bounds and cascade constraintsx/Rx/R\[\-1pt\]\(max\)16073230Syngas compositionMLmeanR2R^\{2\}across syngas targetscomplete target coverage and finite predictionsx/Rx/R\[\-1pt\]\(max\)0\.84680931Credit\-default scoringML0\.5​AUC\+0\.5​KS0\.5\\,\\mathrm\{AUC\}\+0\.5\\,\\mathrm\{KS\}complete labels/probabilities and valid score rangex/Rx/R\[\-1pt\]\(max\)0\.86555232Grid\-load forecastingMLMAPEcomplete time horizon and finite forecastsR/xR/x\[\-1pt\]\(min\)0\.37446533LLM\-traffic forecastingMLtraffic RMSEcomplete horizon and finite forecastsR/xR/x\[\-1pt\]\(min\)23222234Jewelry\-sales forecastingML0\.50\.5quantity score\+0\.5\+0\.5F1complete SKU coverage and valid nonnegative forecastsx/Rx/R\[\-1pt\]\(max\)0\.19319335Corn\-futures forecastingMLmean forecasting errorcomplete horizon and finite forecastsR/xR/x\[\-1pt\]\(min\)0\.29457936Fermentation outcomesMLfermentation MAEcomplete targets and finite predictionsR/xR/x\[\-1pt\]\(min\)0\.43507737MicroGC pillar designCO0\.6​snrmse\+0\.2​spress\+0\.2​sperim0\.6s\_\{\\rm nrmse\}\+0\.2s\_\{\\rm press\}\+0\.2s\_\{\\rm perim\}valid geometry/field outputs and CFD checksx/Rx/R\[\-1pt\]\(max\)0\.6269638Display\-fab schedulingCO60​pach\+30​oach\+10​qscore60p\_\{\\rm ach\}\+30o\_\{\\rm ach\}\+10q\_\{\\rm score\}complete schedule and machine/order constraintsx/Rx/R\[\-1pt\]\(max\)98\.620839Warehouse forklift schedulingCOnested warehouse objectiveassignment, stack/forklift and temporal constraintsR/xR/x\[\-1pt\]\(min\)39\.460540Frailty\-risk groupingMLWFIT​R2\+WRISK​\(c−0\.5\)W\_\{\\rm FIT\}R^\{2\}\+W\_\{\\rm RISK\}\(c\-0\.5\)complete cohort outputs and valid risk/fit fieldsx/Rx/R\[\-1pt\]\(max\)0\.84550441Casting thermal fieldsMLmean thermal\-field RMSEcomplete field output and finite valuesR/xR/x\[\-1pt\]\(min\)0\.1895242Superalloy strength predictionMLsuperalloy prediction errorcomplete target coverage and finite valuesR/xR/x\[\-1pt\]\(min\)13\.339843Coagulant dosingMLwater\-coagulant dosing errorcomplete horizon and finite valuesR/xR/x\[\-1pt\]\(min\)0\.2131944Bank\-branch siting scoresMLbranch\-siting score MAEall branches scored with valid predictionsR/xR/x\[\-1pt\]\(min\)7\.1299345Data\-center rack allocationCOrack\-allocation total costall racks assigned and capacity/power constraintsR/xR/x\[\-1pt\]\(min\)28\.719746Routing with 3D loadingCO100,000​nvehicles100\{,\}000\\,n\_\{\\rm vehicles\}\+\+routing distancepacking and route feasibilityR/xR/x\[\-1pt\]\(min\)1\.502e647Regional fleet dispatchCOregional dispatch total objectiveall orders assigned; violations penalized in objectiveR/xR/x\[\-1pt\]\(min\)1\.452e648Hybrid\-energy dispatchCO8,760\-hour total incomecomplete dispatch and operational constraintsx/Rx/R\[\-1pt\]\(max\)1\.854e849Fuel\-tanker routingCOmulti\-compartment tanker distanceall demands served and vehicle/compartment constraintsR/xR/x\[\-1pt\]\(min\)2\.827e6
##### Clarification points\.

A clarification point is retained only if \(i\) it cannot be resolved from the agent\-visible request and data, \(ii\) the customer’s answer can change a feasible decision, the objective, or the evaluator’s acceptance judgment, and \(iii\) the answer can be stated as a fact, rule, or convention that the evaluator applies to the artifact\. Implementation preferences, facts visible in the workspace, and questions about the agent’s method are excluded\.

##### Splitting and merging\.

A candidate is split when two answers change different decisions or evaluator checks, e\.g\., the error metric, which changes the score, and the valid value range, which changes whether the artifact is accepted\. Paraphrases and examples of the same decision are merged into one point\. Table[5](https://arxiv.org/html/2610.14539#A1.T5)lists boundary cases\.

Table 5:Boundary examples for clarification points\.CandidateDecisionReason“Is error MAE or relative error?”Keep as one valid pointChanges the evaluator’s scoring convention\.“Does same\-city supplier precedence override cost minimization?”Keep as one valid pointChanges a routing/fulfillment rule and can change feasibility\.“Are out\-of\-range predictions rejected?”Keep as one valid pointChanges a hard constraint\.“What Python library should I use?”ExcludeImplementation choice is not scored\.“How many rows are in the visible CSV?”ExcludeThe fact is already observable to the agent\.“Which metric and which value range are required?”Split into two pointsMetric changes the score; range changes acceptance\.“Should same\-city orders precede transfers, and should they be grouped?”Merge when the evaluator has one precedence rule; split otherwiseThe split is determined by independent counterfactual answers, not punctuation\.
##### Registry fields\.

Each point ingt\.jsonhas a key and four fields: the requirement description, issue type \(missing, ambiguous, invalidated, or conflicting\), content domain \(scoring convention, constraint rule, computation convention, or data semantics\), and rule source \(nullif none\)\.

##### Review protocol\.

The FDE responsible for each original delivery annotated the requirements established in the customer–FDE discussions\. A second review applied the same criteria against the evaluator and reference artifact, removed duplicates, and confirmed that each point had a customer answer; disagreements were resolved against the evaluator code\.

### A\.3Metrics and runtime settings

##### Score selection\.

Up to five extraction routes score each artifact, with remaining routes skipped once three scores agree after rounding to four decimal places\. Extractions without source provenance are rejected; majority agreement alone does not prove that every decision was preserved \(Appendix[A\.8](https://arxiv.org/html/2610.14539#A1.SS8.SSS0.Px4)\)\. Without a majority, the score is unavailable\.

##### Aggregation\.

A run is valid when its artifact satisfies the case’s hard constraints \(Table[4](https://arxiv.org/html/2610.14539#A1.T4)\); pass@3 and pass^3 count valid repeats\. Zero scores and invalid runs are included in Quality\. Each main configuration contributes three scored runs on each of 49 cases, with all 147 runs included in DSR \(Section[4\.2](https://arxiv.org/html/2610.14539#S4.SS2)\)\. All ablation runs also have numeric scores\. Clarification metrics are never imputed, and runtime, tool calls, and cost are not part of any delivery metric\.

##### Question units\.

Agent messages are split at question marks into question units; a message without a question mark counts as one unit\. Turn indices come from the interaction log\.

##### Coverage judgments\.

Judges also receive annotated trigger examples as matching hints\. Merely touching a requirement’s topic does not count as coverage\.

A covered point’s matched questions are those cited by the agreeing judges, and Precision counts distinct matched questions\.

##### Undefined values\.

With no recorded questions, Precision and Ask\-F1 are undefined and Recall is zero\. If every judgment in a run fails, all three metrics are unavailable; each metric is averaged only over runs where it is defined\.

##### Runtime\.

Each solving run has 43,200 seconds, including interaction, and at most 30 clarification rounds, with no cap on the number of questions or on cumulative tokens\. OpenHands runs at most 200 agent steps \(max\_iterations\); these steps are distinct from customer exchanges\. OpenCode passes a per\-call output limit of 131,072 tokens with a 1,000,000\-token context\. The native CLIs use SDK or provider defaults\. OpenCode sets neither temperature nor reasoning effort, and no run overrides the temperature\. Exact runner, scaffold, and dependency pins are provided with the artifact; the controls above are the settings relevant to interpreting the experiments\.

Table 6:Complete results\.Estimates with 95% case\-bootstrap intervals; bold marks blockwise best values\.\(a\)Delivery

Agent / modelQualityDSRpass@3pass^3*General scaffold:opencode*GPT\-6\-Astra0\.546\[0\.448, 0\.647\]5\.4\[0\.7, 12\.2\]91\.8 \[83\.7, 98\.0\]81\.6\[71\.4, 91\.8\]GLM\-5\.30\.541 \[0\.445, 0\.632\]1\.4 \[0\.0, 3\.4\]91\.8 \[83\.7, 98\.0\]77\.6 \[65\.3, 87\.8\]Qwen3\.8\-Max0\.539 \[0\.444, 0\.629\]2\.7 \[0\.0, 6\.1\]95\.9\[89\.8, 100\.0\]73\.5 \[61\.2, 85\.7\]Grok\-4\.60\.522 \[0\.423, 0\.621\]2\.7 \[0\.7, 5\.4\]87\.8 \[77\.6, 95\.9\]79\.6 \[67\.3, 89\.8\]DeepSeek\-V4\-Pro0\.499 \[0\.404, 0\.593\]1\.4 \[0\.0, 4\.1\]91\.8 \[83\.7, 98\.0\]71\.4 \[59\.2, 83\.7\]Kimi\-K30\.419 \[0\.336, 0\.503\]0\.0 \[0\.0, 0\.0\]91\.8 \[83\.7, 98\.0\]53\.1 \[38\.8, 67\.3\]Gemini\-3\.1\-Pro0\.387 \[0\.309, 0\.470\]0\.0 \[0\.0, 0\.0\]89\.8 \[81\.6, 98\.0\]59\.2 \[46\.9, 73\.5\]*General scaffold:openhands*GPT\-6\-Astra0\.530\[0\.427, 0\.624\]4\.8 \[0\.0, 10\.9\]93\.9\[87\.8, 100\.0\]67\.3 \[53\.1, 79\.6\]GLM\-5\.30\.527 \[0\.431, 0\.615\]2\.0 \[0\.0, 5\.4\]91\.8 \[83\.7, 98\.0\]67\.3 \[55\.1, 79\.6\]Grok\-4\.60\.516 \[0\.418, 0\.601\]0\.7 \[0\.0, 2\.0\]91\.8 \[83\.7, 98\.0\]71\.4\[59\.2, 83\.7\]DeepSeek\-V4\-Pro0\.510 \[0\.418, 0\.598\]0\.7 \[0\.0, 2\.0\]93\.9\[85\.7, 100\.0\]71\.4\[59\.2, 83\.7\]Qwen3\.8\-Max0\.507 \[0\.421, 0\.592\]6\.1\[1\.4, 11\.6\]93\.9\[87\.8, 100\.0\]65\.3 \[53\.0, 77\.6\]Gemini\-3\.1\-Pro0\.457 \[0\.360, 0\.548\]0\.0 \[0\.0, 0\.0\]87\.8 \[77\.6, 95\.9\]65\.3 \[51\.0, 77\.6\]Kimi\-K30\.412 \[0\.331, 0\.496\]0\.7 \[0\.0, 2\.0\]85\.7 \[75\.5, 93\.9\]57\.1 \[42\.9, 71\.4\]*Native scaffolds*codex\+ GPT\-6\-Astra0\.551\[0\.448, 0\.647\]4\.1\[0\.0, 9\.5\]93\.9\[85\.7, 100\.0\]75\.5\[63\.3, 87\.8\]deepseek\-harness\+ DeepSeek\-V4\-Pro0\.527 \[0\.427, 0\.619\]1\.4 \[0\.0, 3\.4\]91\.8 \[83\.7, 98\.0\]75\.5\[63\.3, 87\.8\]kimi\+ Kimi\-K30\.420 \[0\.333, 0\.502\]0\.7 \[0\.0, 2\.0\]89\.8 \[81\.6, 98\.0\]59\.2 \[44\.9, 73\.5\]gemini\+ Gemini\-3\.1\-Pro0\.404 \[0\.319, 0\.494\]0\.0 \[0\.0, 0\.0\]87\.8 \[77\.6, 95\.9\]57\.1 \[42\.9, 71\.4\]
\(b\)Clarification

Agent / modelRecallPrecisionAsk\-F1KQC1ATC↓\\downarrowRedund\.↓\\downarrow*General scaffold:opencode*GPT\-6\-Astra83\.6\[\-1pt\]\[79\.5, 87\.6\]36\.5\[\-1pt\]\[32\.1, 41\.1\]48\.8\[\-1pt\]\[44\.3, 53\.7\]78\.1\[\-1pt\]\[73\.3, 82\.6\]1\.07\[\-1pt\]\[1\.04, 1\.11\]15\.6\[\-1pt\]\[13\.7, 17\.8\]GLM\-5\.366\.8\[\-1pt\]\[60\.7, 72\.6\]39\.7\[\-1pt\]\[35\.3, 44\.2\]47\.5\[\-1pt\]\[43\.2, 51\.8\]62\.8\[\-1pt\]\[57\.2, 68\.7\]1\.05\[\-1pt\]\[1\.02, 1\.08\]10\.1\[\-1pt\]\[8\.7, 11\.8\]Qwen3\.8\-Max70\.6\[\-1pt\]\[65\.4, 76\.0\]35\.4\[\-1pt\]\[31\.5, 39\.6\]45\.1\[\-1pt\]\[40\.8, 49\.5\]68\.0\[\-1pt\]\[62\.8, 73\.3\]1\.03\[\-1pt\]\[1\.01, 1\.06\]16\.3\[\-1pt\]\[14\.0, 19\.1\]Grok\-4\.668\.2\[\-1pt\]\[62\.3, 74\.3\]38\.3\[\-1pt\]\[34\.0, 43\.0\]46\.4\[\-1pt\]\[42\.3, 50\.6\]65\.2\[\-1pt\]\[58\.9, 71\.8\]1\.04\[\-1pt\]\[1\.02, 1\.08\]8\.8\[\-1pt\]\[7\.4, 10\.5\]DeepSeek\-V4\-Pro55\.1\[\-1pt\]\[48\.7, 61\.9\]37\.9\[\-1pt\]\[33\.3, 42\.8\]42\.7\[\-1pt\]\[38\.1, 47\.3\]52\.9\[\-1pt\]\[46\.4, 59\.9\]1\.04\[\-1pt\]\[1\.01, 1\.07\]8\.3\[\-1pt\]\[7\.2, 9\.6\]Kimi\-K355\.0\[\-1pt\]\[49\.3, 60\.8\]37\.6\[\-1pt\]\[33\.8, 41\.6\]43\.3\[\-1pt\]\[39\.6, 47\.3\]53\.6\[\-1pt\]\[48\.3, 59\.5\]1\.02\[\-1pt\]\[1\.01, 1\.04\]10\.0\[\-1pt\]\[8\.4, 11\.7\]Gemini\-3\.1\-Pro47\.1\[\-1pt\]\[41\.3, 52\.9\]39\.5\[\-1pt\]\[34\.3, 44\.8\]40\.4\[\-1pt\]\[35\.8, 45\.4\]43\.0\[\-1pt\]\[37\.4, 48\.7\]1\.16\[\-1pt\]\[1\.06, 1\.29\]7\.5\[\-1pt\]\[6\.0, 9\.0\]*General scaffold:openhands*GPT\-6\-Astra84\.7\[\-1pt\]\[80\.5, 88\.5\]36\.8\[\-1pt\]\[32\.7, 41\.3\]49\.5\[\-1pt\]\[44\.8, 54\.0\]80\.2\[\-1pt\]\[75\.2, 85\.1\]1\.06\[\-1pt\]\[1\.03, 1\.10\]14\.4\[\-1pt\]\[13\.0, 15\.8\]GLM\-5\.367\.1\[\-1pt\]\[61\.1, 73\.0\]36\.8\[\-1pt\]\[32\.4, 41\.1\]45\.5\[\-1pt\]\[41\.1, 49\.9\]64\.3\[\-1pt\]\[58\.2, 70\.7\]1\.04\[\-1pt\]\[1\.02, 1\.06\]12\.6\[\-1pt\]\[10\.7, 14\.8\]Grok\-4\.669\.9\[\-1pt\]\[64\.0, 75\.9\]37\.0\[\-1pt\]\[32\.7, 41\.0\]45\.9\[\-1pt\]\[41\.9, 49\.8\]66\.7\[\-1pt\]\[60\.4, 73\.1\]1\.05\[\-1pt\]\[1\.02, 1\.08\]11\.0\[\-1pt\]\[9\.1, 13\.2\]DeepSeek\-V4\-Pro65\.2\[\-1pt\]\[58\.6, 72\.0\]36\.4\[\-1pt\]\[31\.6, 40\.9\]44\.9\[\-1pt\]\[40\.1, 49\.4\]64\.3\[\-1pt\]\[57\.6, 71\.1\]1\.01\[\-1pt\]\[1\.00, 1\.02\]11\.0\[\-1pt\]\[9\.6, 12\.6\]Qwen3\.8\-Max75\.4\[\-1pt\]\[69\.8, 80\.7\]30\.9\[\-1pt\]\[27\.0, 34\.5\]42\.2\[\-1pt\]\[38\.0, 46\.5\]74\.0\[\-1pt\]\[68\.1, 79\.7\]1\.02\[\-1pt\]\[1\.01, 1\.03\]22\.6\[\-1pt\]\[19\.9, 25\.6\]Gemini\-3\.1\-Pro45\.5\[\-1pt\]\[39\.0, 52\.1\]37\.8\[\-1pt\]\[32\.6, 42\.8\]39\.2\[\-1pt\]\[33\.9, 44\.5\]42\.2\[\-1pt\]\[36\.2, 48\.5\]1\.06\[\-1pt\]\[1\.03, 1\.09\]6\.3\[\-1pt\]\[5\.3, 7\.4\]Kimi\-K357\.9\[\-1pt\]\[52\.4, 63\.4\]35\.4\[\-1pt\]\[31\.7, 39\.1\]41\.3\[\-1pt\]\[37\.8, 44\.9\]57\.1\[\-1pt\]\[51\.6, 62\.9\]1\.02\[\-1pt\]\[1\.01, 1\.04\]11\.2\[\-1pt\]\[10\.0, 12\.5\]*Native scaffolds*codex\+ GPT\-6\-Astra84\.7\[\-1pt\]\[80\.8, 88\.7\]37\.9\[\-1pt\]\[33\.2, 42\.6\]50\.1\[\-1pt\]\[45\.6, 54\.6\]79\.4\[\-1pt\]\[74\.9, 83\.7\]1\.06\[\-1pt\]\[1\.03, 1\.09\]13\.9\[\-1pt\]\[12\.2, 15\.9\]deepseek\-harness\+ DeepSeek\-V4\-Pro61\.8\[\-1pt\]\[55\.3, 68\.1\]39\.3\[\-1pt\]\[34\.8, 43\.8\]45\.9\[\-1pt\]\[41\.0, 50\.2\]59\.9\[\-1pt\]\[53\.5, 66\.4\]1\.03\[\-1pt\]\[1\.01, 1\.06\]9\.6\[\-1pt\]\[8\.1, 11\.4\]kimi\+ Kimi\-K355\.7\[\-1pt\]\[49\.3, 62\.3\]36\.7\[\-1pt\]\[32\.4, 41\.1\]42\.9\[\-1pt\]\[38\.3, 47\.3\]54\.2\[\-1pt\]\[47\.9, 60\.8\]1\.03\[\-1pt\]\[1\.01, 1\.05\]9\.8\[\-1pt\]\[8\.3, 11\.6\]gemini\+ Gemini\-3\.1\-Pro44\.9\[\-1pt\]\[39\.3, 50\.6\]38\.8\[\-1pt\]\[33\.6, 44\.1\]39\.4\[\-1pt\]\[34\.5, 44\.2\]44\.2\[\-1pt\]\[38\.4, 50\.3\]1\.01\[\-1pt\]\[1\.00, 1\.03\]6\.0\[\-1pt\]\[5\.1, 6\.9\]

### A\.4Full results

Entries are means with 95% case\-bootstrap intervals \(2,000 resamples, seed 12345\), following the bootstrap framework of Efron\[[36](https://arxiv.org/html/2610.14539#bib.bib36)\]\.

##### Delivery metrics\.

Table[6](https://arxiv.org/html/2610.14539#A1.T6)\(a\) gives delivery results with intervals\. Quality is case\-macro and DSR is run\-level; pass@3 and pass^3 are case\-level \(Section[4\.2](https://arxiv.org/html/2610.14539#S4.SS2)\)\. Every pair of unpaired Quality intervals overlaps, because the intervals carry the variance between cases\. Paired by case, the systems form two groups\. The 12 systems built on GPT\-6\-Astra, GLM\-5\.3, Qwen3\.8\-Max, Grok\-4\.6, and DeepSeek\-V4\-Pro score 0\.499–0\.551; the three Kimi\-K3 systems and Gemini\-3\.1\-Pro underopencodeandgeminiscore 0\.387–0\.420\. All 60 differences between the groups exclude zero under a paired case bootstrap, and none of the 76 differences within a group does\. All differences above 0\.08 are resolved and none of 0\.04 or less, so 49 cases separate models but not scaffolds of the same model\. The exception isopenhands\+ Gemini\-3\.1\-Pro \(0\.457\), which exceeds the other two Gemini\-3\.1\-Pro systems by 0\.053 and 0\.070 \(both resolved\)\. DSR does not rank systems: 15 of the 18 DSR intervals include zero\.

##### Clarification metrics\.

Clarification metrics are run\-level \(Section[4\.2](https://arxiv.org/html/2610.14539#S4.SS2.SSS0.Px2); Table[6](https://arxiv.org/html/2610.14539#A1.T6)\(b\)\)\. Across the 18 systems, Spearmanρ\\rhobetween Recall and Quality is 0\.83; after subtracting each model’s mean over its scaffolds, it is−0\.31\-0\.31\(p=0\.22p=0\.22\)\. Moving a model fromopencodetoopenhandsraises Recall for six of seven models \(DeepSeek\-V4\-Pro: 55\.1% to 65\.2%\) and Redund\. for five \(Qwen3\.8\-Max: 16\.3 to 22\.6\), lowers Precision for six, and lowers Quality for five, by at most 0\.032\. Scaffold changes that increase coverage do not consistently improve Quality; the policy ablation also shows mixed Quality changes \(Appendix[A\.6](https://arxiv.org/html/2610.14539#A1.SS6.SSS0.Px3)\)\.

##### Compute\.

Provider cost was available for 15 of 18 configurations and showed no detectable association with Quality; detailed accounting is included with the artifact rather than repeated here\.

### A\.5Clarification types

Each clarification point ingt\.jsoncarries an issue type and a content domain \(Table[7](https://arxiv.org/html/2610.14539#A1.T7)\)\. These labels are hidden from agents and judges and are not used by the primary scorer\.

Table 7:Composition of the clarification\-point registry\.266 points in 49 cases\.Annotation familyTypePointsShareCasesIssue typeMissing20777\.8%48Ambiguous4516\.9%30Invalidated93\.4%8Conflicting51\.9%4Content domainScoring convention7829\.3%39Constraint rule6524\.4%33Computation convention6223\.3%27Data semantics6122\.9%35Rule\-source labels are too sparse \(246 unlabeled\) to analyze\. Type\-specific Recall is the share of a case’s points of that type that are covered, averaged over repeats and then over cases containing the type, and correlated with Quality across the 18 configurations \(Table[8](https://arxiv.org/html/2610.14539#A1.T8)\(a\)\)\.

Table 8:Issue types and delivery\.Spearmanρ\\rhowith case\-bootstrap 95% intervals\.\(a\)Type\-specific Recall and Quality across 18 configurations; Holm correction over four tests

Issue typeMean RecallQualityρ\\rho95% CIHolmppMissing66\.3%0\.808\[0\.467, 0\.882\]<0\.001<0\.001Ambiguous60\.5%0\.801\[0\.423, 0\.884\]<0\.001<0\.001Invalidated60\.5%0\.513\[0\.004, 0\.707\]0\.029Conflicting77\.5%0\.641\[0\.101, 0\.705\]0\.008
\(b\)Issue\-type share and delivery across 49 cases

Issue typeCases with typeQualityρ\\rho\[95% CI\]DSRρ\\rho\[95% CI\]Missing480\.032\[\-1pt\]\[\-0\.253, 0\.341\]−0\.092\-0\.092\[\-1pt\]\[\-0\.322, 0\.161\]Ambiguous30−0\.081\-0\.081\[\-1pt\]\[\-0\.358, 0\.209\]0\.018\[\-1pt\]\[\-0\.212, 0\.260\]Invalidated80\.010\[\-1pt\]\[\-0\.274, 0\.283\]0\.414\[\-1pt\]\[0\.021, 0\.747\]Conflicting4−0\.102\-0\.102\[\-1pt\]\[\-0\.357, 0\.216\]0\.085\[\-1pt\]\[\-0\.159, 0\.457\]

Table[9](https://arxiv.org/html/2610.14539#A1.T9)breaks these Recall values down by configuration\.

Table 9:Type\-specific Recall \(%\) by configuration\.Bold marks blockwise maxima\.Agent / modelOverallMissingAmbiguousInvalidatedConflicting*General scaffold:opencode*GPT\-6\-Astra83\.6487\.875\.764\.695\.8GLM\-5\.366\.8369\.063\.156\.275\.0Qwen3\.8\-Max70\.5670\.767\.881\.283\.3Grok\-4\.668\.1971\.266\.564\.670\.8DeepSeek\-V4\-Pro55\.0858\.046\.552\.170\.8Kimi\-K355\.0358\.148\.362\.575\.0Gemini\-3\.1\-Pro47\.1049\.943\.941\.754\.2*General scaffold:openhands*GPT\-6\-Astra84\.6787\.278\.375\.087\.5GLM\-5\.367\.1268\.958\.962\.575\.0Grok\-4\.669\.9171\.764\.472\.983\.3DeepSeek\-V4\-Pro65\.2465\.663\.760\.487\.5Qwen3\.8\-Max75\.3777\.268\.085\.4100\.0Gemini\-3\.1\-Pro45\.4545\.746\.533\.350\.0Kimi\-K357\.9459\.952\.068\.862\.5*Native scaffolds*codex\+ GPT\-6\-Astra84\.7087\.980\.070\.8100\.0deepseek\-harness\+ DeepSeek\-V4\-Pro61\.7962\.362\.456\.283\.3kimi\+ Kimi\-K355\.6956\.955\.752\.183\.3gemini\+ Gemini\-3\.1\-Pro44\.9445\.846\.529\.258\.3The ranking of systems is stable across issue types\. Missing and Ambiguous Recall have Spearmanρ=0\.95\\rho=0\.95over the 18 configurations, and GPT\-6\-Astra has the highest and Gemini\-3\.1\-Pro the lowest of both in every scaffold block\. Invalidated points are the exception: Qwen3\.8\-Max ranks first on them in both general scaffolds \(81\.2% and 85\.4%\) although its Missing Recall is 10–17 points below GPT\-6\-Astra; the Invalidated and Conflicting columns rest on 8 and 4 cases\. Within a case, the issue type does not decide which points are covered\. In the 29 cases with both Missing and Ambiguous points, 16 of 18 systems cover Ambiguous points less often \(mean gap 7\.2 points\), but the gap reverses in 12 of the 29 cases \(95% case\-bootstrap CI\[−3\.8,17\.7\]\[\-3\.8,17\.7\]\)\.

The content domain does\. Pooled over runs, agents cover 72\.6% of constraint\-rule points, 64\.8% of computation conventions, 56\.6% of scoring conventions, and 56\.6% of data\-semantics points, and constraint rules have the highest Recall for 16 of 18 systems\. Within the same case, constraint rules are covered 14\.6 points more often than data semantics \(22 cases; CI\[3\.9,25\.0\]\[3\.9,25\.0\]\) and 15\.9 points more than computation conventions \(22 cases;\[2\.8,28\.9\]\[2\.8,28\.9\]\); the gap to scoring conventions \(\+12\.5\+12\.5, 26 cases\) has an interval that includes zero\. Agents ask about stated business rules more than about how fields are read and how quantities are computed\. Covering a constraint rule also has the largest association with delivery: within the same system and case, runs that cover a constraint\-rule point score 0\.059 higher than runs that do not \(292 system–case–point cells with mixed coverage; Wilcoxonp=0\.006p=0\.006\), against 0\.027–0\.035 for the other domains\.

Across systems, coverage of every domain tracks Quality equally \(Spearmanρ=0\.81\\rho=0\.81–0\.830\.83\), and the two groups of Appendix[A\.4](https://arxiv.org/html/2610.14539#A1.SS4)differ by 16\.2–20\.7 points in every domain\. Table[10](https://arxiv.org/html/2610.14539#A1.T10)gives the domain mix of each model’s matched questions; a question matched to points of several domains is split equally among them\. GPT\-6\-Astra is the only model that directs more of its matched questions to scoring conventions \(28\.8%\) than to constraint rules \(26\.7%\); DeepSeek\-V4\-Pro, Gemini\-3\.1\-Pro, and Grok\-4\.6 direct 20\.1–21\.7% to scoring conventions and 31\.5–33\.0% to constraint rules\. Over the 18 systems, the scoring share correlates with Quality \(ρ=0\.61\\rho=0\.61\) and the constraint share with lower Quality \(ρ=−0\.60\\rho=\-0\.60; Holmp=0\.028p=0\.028for both over four domains\)\. Neither association is robust: the case\-bootstrap 95% intervals include zero \(\[−0\.07,0\.64\]\[\-0\.07,0\.64\]and\[−0\.69,0\.08\]\[\-0\.69,0\.08\]\), and without the three GPT\-6\-Astra systems the correlations fall to 0\.38 \(p=0\.16p=0\.16\) and−0\.32\-0\.32\(p=0\.24p=0\.24\)\.

Table 10:Domain mix of matched questions by model \(%\)\.Quality is the mean over the model’s systems\.ModelConstraintComputationScoringDataQualityGPT\-6\-Astra26\.725\.428\.819\.10\.542GLM\-5\.330\.724\.526\.618\.30\.534Qwen3\.8\-Max28\.127\.426\.318\.20\.523Grok\-4\.631\.529\.721\.717\.20\.519DeepSeek\-V4\-Pro33\.027\.920\.119\.00\.512Kimi\-K331\.126\.124\.618\.20\.417Gemini\-3\.1\-Pro33\.029\.620\.417\.00\.416Table[8](https://arxiv.org/html/2610.14539#A1.T8)\(b\) instead correlates each case’s share of points of a type with its delivery, averaged over configurations, across all 49 cases\.

No Quality interval excludes zero, so the type mix of a case does not predict its difficulty\. The positive Invalidated–DSR association rests on eight cases\.

### A\.6Ablations

##### Scope\.

The three\-condition ablation uses a 20\-case subset\. The policy ablation uses the five cases in that subset with the most clarification points; exact case IDs and registry counts are provided with the artifact\.

##### Information conditions\.

Table[11](https://arxiv.org/html/2610.14539#A1.T11)expands Table[2](https://arxiv.org/html/2610.14539#S5.T2)\. Each estimate averages 20 paired case differences\. Intervals are 95% case\-bootstrap intervals \(2,000 resamples, seed 12345\);∗marks two\-sided Wilcoxon signed\-rank tests that remain significant after Holm correction over the 24 comparisons\.

Table 11:Paired Quality differences between information conditions\.∗: Holm\-adjustedp<0\.05p<0\.05\.Agent / model𝚫𝐫𝐞𝐪\\bm\{\\Delta\_\{\\rm req\}\}𝚫𝐫𝐞𝐦𝐚𝐢𝐧\\bm\{\\Delta\_\{\\rm remain\}\}𝚫𝐭𝐨𝐭𝐚𝐥\\bm\{\\Delta\_\{\\rm total\}\}opencode\+ GPT\-6\-Astra\+0\.3997∗\[0\.2284, 0\.5983\]\+0\.1662\[0\.0369, 0\.3167\]\+0\.5658∗\[0\.3941, 0\.7367\]opencode\+ DeepSeek\-V4\-Pro\+0\.3640∗\[0\.2367, 0\.5046\]\+0\.1758∗\[0\.0893, 0\.2733\]\+0\.5398∗\[0\.4026, 0\.6741\]opencode\+ Gemini\-3\.1\-Pro\+0\.1694∗\[0\.0824, 0\.2684\]\+0\.1974∗\[0\.0810, 0\.3276\]\+0\.3668∗\[0\.2266, 0\.5130\]opencode\+ Kimi\-K3\+0\.2450∗\[0\.1326, 0\.3595\]\+0\.0794\[\-0\.0125, 0\.1740\]\+0\.3244∗\[0\.2137, 0\.4438\]codex\+ GPT\-6\-Astra\+0\.4000∗\[0\.2211, 0\.5751\]\+0\.1966∗\[0\.0853, 0\.3228\]\+0\.5966∗\[0\.4348, 0\.7473\]deepseek\-harness\+ DeepSeek\-V4\-Pro\+0\.3644∗\[0\.2344, 0\.5002\]\+0\.1697∗\[0\.0689, 0\.2949\]\+0\.5341∗\[0\.3960, 0\.6677\]gemini\+ Gemini\-3\.1\-Pro\+0\.2532∗\[0\.1422, 0\.3757\]\+0\.1420\[0\.0189, 0\.2758\]\+0\.3952∗\[0\.2680, 0\.5216\]kimi\+ Kimi\-K3\+0\.2195∗\[0\.1044, 0\.3488\]\+0\.3490∗\[0\.2226, 0\.4809\]\+0\.5685∗\[0\.4156, 0\.7132\]Δreq\\Delta\_\{\\rm req\}andΔtotal\\Delta\_\{\\rm total\}are significant for all eight systems;Δremain\\Delta\_\{\\rm remain\}is significant for five\. What interaction recovers depends on how many requirements the system asks about; what remains does not\. Over the eight systems,Δreq\\Delta\_\{\\rm req\}rises with Recall on the same 20 cases \(Spearmanρ=0\.86\\rho=0\.86,p=0\.007p=0\.007\), whileΔremain\\Delta\_\{\\rm remain\}hasρ=0\.00\\rho=0\.00\. The two Kimi\-K3 systems show the difference: with Recall of 60\.4% and 60\.7% and the sameInteract\-ReqQuality \(0\.369\), the nativekimiCLI reaches 0\.718 underFullandopencode0\.449, soΔremain\\Delta\_\{\\rm remain\}is\+0\.349\+0\.349and\+0\.079\+0\.079\.

##### Clarification policy\.

Both policies are run on the five policy cases with the eight ablation configurations \(15 runs per configuration and policy\)\.Interactlets the agent decide whether to ask;Interact\-Reqrequires clarification before implementation\. \#Q counts question units, and ask rate is the share of runs with at least one question\.

Table 12:Optional versus required clarificationon five cases\.\(a\)Metrics by condition

ConditionQualityAsk rate\#​𝑸\\bm\{\\\#Q\}RecallPrecisionAsk\-F1*opencode\+ GPT\-6\-Astra*Interact0\.3262100\.00%15\.0063\.61%47\.72%53\.84%Interact\-Req0\.3830100\.00%36\.2785\.24%49\.04%61\.19%*opencode\+ DeepSeek\-V4\-Pro*Interact0\.4061100\.00%10\.2037\.18%56\.92%43\.73%Interact\-Req0\.5552100\.00%21\.1349\.76%53\.51%50\.62%*opencode\+ Gemini\-3\.1\-Pro*Interact0\.330493\.33%10\.0739\.45%53\.15%46\.42%Interact\-Req0\.2742100\.00%14\.4738\.53%47\.24%41\.01%*opencode\+ Kimi\-K3*Interact0\.099280\.00%8\.0023\.67%46\.33%34\.53%Interact\-Req0\.2955100\.00%24\.2752\.59%43\.83%45\.28%*codex\+ GPT\-6\-Astra*Interact0\.4612100\.00%16\.2769\.97%51\.47%58\.61%Interact\-Req0\.3911100\.00%31\.2781\.03%48\.50%59\.43%*deepseek\-harness\+ DeepSeek\-V4\-Pro*Interact0\.426980\.00%8\.3333\.09%57\.05%46\.32%Interact\-Req0\.5579100\.00%18\.6053\.26%55\.03%53\.50%*gemini\+ Gemini\-3\.1\-Pro*Interact0\.253693\.33%8\.0030\.73%47\.06%38\.12%Interact\-Req0\.3424100\.00%13\.0741\.19%51\.12%44\.28%*kimi\+ Kimi\-K3*Interact0\.3603100\.00%8\.0738\.17%56\.13%43\.33%Interact\-Req0\.235186\.67%19\.4741\.54%44\.32%43\.57%
\(b\)Quality change from required clarification, with Holm\-adjustedpp\-values

Agent / modelReq−\-Interact95% CIHolmppopencode\+ GPT\-6\-Astra\+0\.0569\[\-0\.0727, 0\.2129\]1\.000opencode\+ DeepSeek\-V4\-Pro\+0\.1490\[0\.0535, 0\.2467\]0\.875opencode\+ Gemini\-3\.1\-Pro\-0\.0563\[\-0\.1866, 0\.0383\]1\.000opencode\+ Kimi\-K3\+0\.1963\[0\.0882, 0\.3217\]0\.500codex\+ GPT\-6\-Astra\-0\.0701\[\-0\.2065, 0\.0440\]1\.000deepseek\-harness\+ DeepSeek\-V4\-Pro\+0\.1310\[\-0\.1608, 0\.4518\]1\.000gemini\+ Gemini\-3\.1\-Pro\+0\.0888\[\-0\.1103, 0\.3339\]1\.000kimi\+ Kimi\-K3\-0\.1252\[\-0\.5651, 0\.2291\]1\.000

Without the requirement, agents still ask in 80–100% of runs, but they ask fewer questions: requiring clarification multiplies \#Q by 1\.4–3\.0 in all eight systems\. The added questions are worth less than the ones agents ask on their own\. Measured as the change in matched questions over the change in \#Q, they match a covered requirement at 13\.5–45\.8%, below theInteractPrecision of the same system in all eight\. Recall rises in seven systems, but by at most 2\.1 points per added question\. Agents ask their most useful questions first\. Quality changes are mixed \(Table[12](https://arxiv.org/html/2610.14539#A1.T12)\(b\)\): no Holm\-adjustedppis below 0\.5\.

##### Oracle answer coverage\.

We selected four cases \(09, 11, 17, and 38\) and rancodex\+ GPT\-6\-Astra without a clarification channel\. We evaluate five target coverage levels,r∈\{0,0\.25,0\.5,0\.75,1\}r\\in\\\{0,0\.25,0\.5,0\.75,1\\\}, using nested subsets of the registered answers and three repeats per case and level\. The 0% level is run separately fromHidden; the 100% level supplies the registered answers, not the complete information file used byFull\. Table[13](https://arxiv.org/html/2610.14539#A1.T13)gives the values plotted in Figure[5](https://arxiv.org/html/2610.14539#S5.F5); exact subset construction is provided with the artifact\.

Table 13:Oracle Quality by answer coverage\.Mean±\\pmSD over three repeats\.Case0%25%50%75%100%09 Nuclear\-site scheduling0\.0000±\\pm0\.00000\.0000±\\pm0\.00000\.3272±\\pm0\.56670\.3206±\\pm0\.55530\.9770±\\pm0\.010911 Retail\-zone routing0\.0000±\\pm0\.00000\.4661±\\pm0\.40370\.7442±\\pm0\.06290\.7935±\\pm0\.06700\.8558±\\pm0\.058217 Cold\-rolling sequencing0\.2347±\\pm0\.11890\.1637±\\pm0\.03990\.3357±\\pm0\.15950\.4260±\\pm0\.14180\.6782±\\pm0\.278738 Display\-fab scheduling0\.0000±\\pm0\.00000\.0000±\\pm0\.00000\.0000±\\pm0\.00000\.3190±\\pm0\.55250\.9650±\\pm0\.0093Four\-case mean0\.05870\.15750\.35180\.46480\.8690Answer coverage improves Quality overall, but the case\-level trends are non\-monotonic\. Specific answers matter more than the number of answers, which also explains the high variance at intermediate coverage levels\. The registered\-answer endpoint differs fromFull, and coverage of a blocking point is necessary but not sufficient for correct implementation; the table therefore provides an information\-sensitivity analysis rather than a claim that more answers always improve delivery\.

### A\.7Zero scores and a case study

Of the 2,646 main runs, 548 score zero\. Of these, 509 \(92\.9%\) are invalid and 39 \(7\.1%\) are valid but earn no Quality; all have a complete artifact and pipeline record\. Table[14](https://arxiv.org/html/2610.14539#A1.T14)groups the recordederror\_infointo mutually exclusive signatures \(precedence: constraint, fatal, schema, execution\)\. Constraint or feasibility violations dominate \(430, 78\.5%\), and 61 runs \(11\.1%\) carry no explicit message\.

Table 14:Evaluator signatures of 548 zero\-score runs\.Recorded signatureRunsShareInterpretationConstraint/feasibility violation43078\.5%Case constraints or feasibility checks are explicitly violated\.Fatal case\-specific check295\.3%A case\-defined fatal condition is recorded, such as incomplete coverage or an out\-of\-horizon event\.Schema/output validation error81\.5%The submitted representation fails a required schema or output check\.Evaluator/scoring execution exception40\.7%The evaluator records an exception or cannot form the required score\.Quality/threshold warning162\.9%A quality or threshold warning accompanies zero Quality for a valid artifact\.No explicit evaluator message6111\.1%The artifact is scored zero but the retained record does not identify a more specific reason\.Zero scores belong to cases more than to systems\. They occur in 33 of the 49 cases, and 7 cases account for half of them\. The five cases with the most zeros \(21, 10, 13, 18, and 38\) contribute 233 \(42\.5%\); in each, 17 or 18 of the 18 systems score zero at least once, and in case 21, 17 systems score zero in all three repeats\. Within a system and case, clarification separates failing from passing repeats only slightly: in the 202 cells with both zero and nonzero repeats, zero\-score repeats have Recall 58\.6% against 63\.6% \(lower in 89 cells, higher in 72; Wilcoxonp=0\.021p=0\.021\) and ask as many questions \(22\.8 vs\. 23\.2\)\. Seventy\-two zeros \(13\.1%\) reached the runtime limit but still produced a scored artifact; 54 of them also record a constraint violation, so a timeout does not by itself explain the zero\.

##### Case study: case 10\.

We trace one run of case 10 \(retail replenishment; 17 days, 73 stores, 3,055 items;opencode\+ GPT\-6\-Astra,Interact\-Req, first repeat\); Table[15](https://arxiv.org/html/2610.14539#A1.T15)\(a\) gives the evidence chain\.

Table 15:Case\-10 run\.\(a\)Evidence chain

StageRecorded evidenceCustomer requestJSON replenishment plan; internal store/warehouse transfers must be exhausted before supplier purchase, and same\-city stock cannot be skipped; stockout avoidance takes priority over timeliness and cost\.Agent questions27 messages \(47 scored question units\) over three rounds\. The agent explicitly asked about source precedence \(“same\-city stores and local warehouse→\\rightarrowcross\-city sources→\\rightarrowsupplier”\) and about how shortages and lead times should be computed\.Key rule and coverageThe complete business information specifies receive arrivals→\\rightarrowsettle sales→\\rightarrowdispatch transfers and purchases\. The registered point “daily inventory transition order” requires this exact order\. The stored majority label iscovered=false, with 0\.60 judge agreement \(2 covered and 3 uncovered votes\)\. No matched question was recorded for this blocker\.Submitted planThe plan metadata nevertheless declares the same three\-step order\. It contains 15,987 instructions \(15,515 transfers and 472 purchases\), and the agent’s validator reportspass\.Executable evaluationAll five extractor/evaluator routes mark the plan invalid and return Quality=0\.0=0\.0\. The evaluator records 347 constraint violations, including eight same\-city exhaustion violations forITEM\_000092\.ReferenceThe customer\-accepted reference has value 0\.9974756\. For the eight affected stores, it contains no item\-092 supplier purchase and routes 10\.58 units through internal transfers \(seven stores receive a positive quantity\), in contrast to the submitted plan\.
\(b\)Item\-092quantities for the eight affected stores

Affected storesAgent internalAgent supplierReference internalEight stores \(aggregate\)9\.813\.8710\.58

For example, at one affected store on 2025\-03\-13, the submitted plan purchased itemITEM\_000092fromSUPPLIER\_DEFAULTdespite a local\-store balance of 0\.0 and 327\.0 units available inWH\_LOCAL; the same hard gate violation occurs across the eight stores, summarized in Table[15](https://arxiv.org/html/2610.14539#A1.T15)\(b\)\.

The run thus shows a rule that was asked about but not enforced\.

### A\.8Audits

##### Audit sample and tasks\.

Two experts, Annotator A and Annotator B, independently labeled 120 clarification items \(105 runs, 38 cases\) and 120 extraction runs \(43 cases\), each sample covering all 18 configurations\. Samples were drawn with fixed seeds, stratified by configuration and by coverage or recorded issue\. A clarification item asks whether answering at least one recorded question would reveal the reference point \(covered, uncovered, or uncertain\)\. An extraction run compares the submitted artifact with its normalized form \(CORRECT,INCORRECT, orNOT\_ASSESSABLE\); runs without a submitted result file areNOT\_ASSESSABLE\.

##### Post\-hoc audit of simulated\-customer answers\.

Annotators A and B labeled the sampled customer answers for support by the case information and consistency with earlier turns \(Table[16](https://arxiv.org/html/2610.14539#A1.T16)\(a\)\)\. The audit indicates that the simulator’s main weakness is cross\-turn inconsistency rather than direct conflict with the case\.

Table 16:Audit results\.\(a\)Simulated\-customer answers; counts are supported/uncertain/contradicted for support and factuality rows and not applicable/consistent/contradicted for status and consistency rows

CriterionnnABAgreement𝜿\\bm\{\\kappa\}Answer support \(all labels\)13031250/51/21179/117/792\.10%0\.385Answer support \(non\-uncertain\)1170——99\.57%0\.285Cross\-turn status1303968/320/15965/323/1599\.00%0\.974Run factuality \(all labels\)10579/24/271/28/682\.86%0\.601Run factuality \(non\-uncertain\)73——91\.78%0\.218Run consistency10560/36/960/35/1097\.14%0\.948
\(b\)Agreement metrics between annotators and the automated five\-judge panel; intervals are case\-cluster bootstrap 95% intervals forκ\\kappa

Comparison𝒏\\bm\{n\}Agreement𝜿\\bm\{\\kappa\}95% CIClarification: A–B11496\.49%0\.9297\[0\.8571, 0\.9832\]Clarification: A–panel11494\.74%0\.8947\[0\.8054, 0\.9661\]Clarification: B–panel12093\.33%0\.8667\[0\.7820, 0\.9424\]Extraction fidelity: A–B12091\.67%0\.7656\[0\.5835, 0\.9033\]

##### Agreement and uncertainty\.

Table[16](https://arxiv.org/html/2610.14539#A1.T16)\(b\) reports agreement and unweighted Cohen’sκ\\kappawith 95% case\-cluster bootstrap intervals \(5,000 resamples\)\. Six clarification items that Annotator A marked uncertain are excluded from comparisons involving A; extraction agreement retains all three categories\.

The panel does not systematically inflate Recall: its disagreements with Annotator A are balanced between covered and uncovered labels\.

##### Extraction agreement and failures\.

We audited the retainedresult\.jsonand extractor records of all 2,646 main runs\. Numeric votes have strong majority agreement, while the remaining non\-numeric failures are rare and are broken down by configuration in Table[17](https://arxiv.org/html/2610.14539#A1.T17)\. Because the 120\-run audit sample was enriched for recorded extraction issues, we reweighted it to the population; the resulting estimated extraction accuracy is 97\.9%\.

Table 17:Extraction failures by configuration\.Agent / modelFailures / slotsRateNo artifact*opencode*GPT\-6\-Astra3/7350\.41%0GLM\-5\.39/7211\.25%4DeepSeek\-V4\-Pro5/7310\.68%2Gemini\-3\.1\-Pro6/7330\.82%0Grok\-4\.62/7330\.27%0Kimi\-K311/7311\.50%4Qwen3\.8\-Max10/7351\.36%0*openhands*GPT\-6\-Astra21/7292\.88%0GLM\-5\.39/7111\.27%1DeepSeek\-V4\-Pro3/7350\.41%0Gemini\-3\.1\-Pro3/7340\.41%1Grok\-4\.66/7330\.82%0Kimi\-K32/7300\.27%3Qwen3\.8\-Max9/7191\.25%3codex\+ GPT\-6\-Astra1/4480\.22%1deepseek\-harness\+ DeepSeek\-V4\-Pro2/6950\.29%0gemini\+ Gemini\-3\.1\-Pro3/7340\.41%0kimi\+ Kimi\-K35/7330\.68%2All configurations110/12,8200\.86%21Overall extraction failure rates are low across configurations, and majority voting reduces the effect of individual extractor disagreements on final scores\.

### A\.9Prompt templates and interaction interfaces

This section reproduces the benchmark prompts for the solving agent, simulated customer, extractor, and judge, and the question–answer interfaces\. They form the benchmark layer added to each scaffold; the scaffolds’ own prompts are unchanged\. Double\-braced fields are filled at runtime\. The listings below show the exact shared and role\-specific templates used in the benchmark, including the information boundaries and the no\-repair extraction contract\. Scaffold\-specific wrappers and exact field substitutions are released with the artifact inanalysis/prompt\_templates/\.

#### A\.9\.1Solving\-agent instructions

All conditions use the same base template\. The request block is the heading\#\# The client’s requestfollowed by the completeinstruction\.md; the data list contains the sorted filenames directly underdata/\. The workspace is the run’s solving directory\. Only the condition block changes\.

Shared agent template[⬇](data:text/plain;base64,WW91IGFyZSBhbiBhbGdvcml0aG0gZGVsaXZlcnkgZW5naW5lZXIuIEEgY2xpZW50IGhhcyBzZW50IHlvdSBhIGJ1c2luZXNzIHJlcXVlc3QgYW5kIHNvbWUgZGF0YSBmaWxlcy4KCllvdXIgam9iOiB1bmRlcnN0YW5kIHdoYXQgdGhleSBhY3R1YWxseSBuZWVkLCB0aGVuIGJ1aWxkIGEgd29ya2luZyBzb2x1dGlvbiBhbmQgZGVsaXZlciBpdC4KCiMjIFdvcmtpbmcgZGlyZWN0b3J5Cgp7e3dvcmtzcGFjZX19CgpEYXRhIGZpbGVzIHByb3ZpZGVkIGJ5IHRoZSBjbGllbnQ6Cnt7ZGF0YV9maWxlc319CgpXcml0ZSBhbGwgb3V0cHV0cyB0byB0aGUgd29ya2luZyBkaXJlY3RvcnkuCgp7e3JlcXVlc3RfYmxvY2t9fQoKe3tjb25kaXRpb25fYmxvY2t9fQoKIyMgRGVsaXZlcmluZwoKRG8gdGhlIHdvcmsgLS0gZG8gbm90IG1lcmVseSBkZXNjcmliZSBob3cgeW91IHdvdWxkIGRvIGl0LiBUaGUgdGFzayBpcyBjb21wbGV0ZSBvbmx5CndoZW4gdGhlIGRlbGl2ZXJhYmxlIGZpbGVzIGV4aXN0IGluIHRoZSB3b3JraW5nIGRpcmVjdG9yeS4KClByb2R1Y2Ugd2hhdGV2ZXIgYXJ0aWZhY3QgdGhlIHJlcXVlc3QgY2FsbHMgZm9yLiBUaGUgY2xpZW50IGRpZCBub3Qgc3BlY2lmeSBhIGZpbGUKZm9ybWF0IC0tIGluZmVyIHdoYXQgaXMgYXBwcm9wcmlhdGUgZnJvbSB3aGF0IHRoZXkgYXNrZWQgZm9yLgoKQWxzbyB3cml0ZSBhIHNob3J0IGBOT1RFUy5tZGAgcmVjb3JkaW5nIHlvdXIga2V5IG1vZGVsbGluZyBkZWNpc2lvbnM6IHdoYXQgeW91CmFzc3VtZWQsIHdoYXQgeW91IGluZmVycmVkIGZyb20gdGhlIGRhdGEsIGFuZCBhbnl0aGluZyB5b3Ugd2VyZSB1bnN1cmUgYWJvdXQu)Youareanalgorithmdeliveryengineer\.Aclienthassentyouabusinessrequestandsomedatafiles\.Yourjob:understandwhattheyactuallyneed,thenbuildaworkingsolutionanddeliverit\.\#\#Workingdirectory\{\{workspace\}\}Datafilesprovidedbytheclient:\{\{data\_files\}\}Writealloutputstotheworkingdirectory\.\{\{request\_block\}\}\{\{condition\_block\}\}\#\#DeliveringDothework\-\-donotmerelydescribehowyouwoulddoit\.Thetaskiscompleteonlywhenthedeliverablefilesexistintheworkingdirectory\.Producewhateverartifacttherequestcallsfor\.Theclientdidnotspecifyafileformat\-\-inferwhatisappropriatefromwhattheyaskedfor\.Alsowriteashort‘NOTES\.md‘recordingyourkeymodellingdecisions:whatyouassumed,whatyouinferredfromthedata,andanythingyouwereunsureabout\.

Hidden condition block[⬇](data:text/plain;base64,IyMgTWlzc2luZyBpbmZvcm1hdGlvbgoKVGhlIHJlcXVlc3QgaXMgd3JpdHRlbiBpbiB0aGUgY2xpZW50J3Mgb3duIHdvcmRzLiBJdCB3aWxsIG5vdCBzcGVsbCBvdXQgZXZlcnkKZGV0YWlsIHlvdSBuZWVkIC0tIGRhdGEgY29udmVudGlvbnMsIHVuaXRzLCBlZGdlLWNhc2UgaGFuZGxpbmcgYW5kIHNpbWlsYXIgcG9pbnRzCmFyZSBvZnRlbiBsZWZ0IGltcGxpY2l0IGJlY2F1c2UgdGhlIGNsaWVudCBjb25zaWRlcnMgdGhlbSBvYnZpb3VzLgoKVGhlcmUgaXMgbm8gb25lIGF2YWlsYWJsZSB0byBhbnN3ZXIgcXVlc3Rpb25zLiBXaGVyZSBzb21ldGhpbmcgY2Fubm90IGJlCmRldGVybWluZWQgZnJvbSB0aGUgcmVxdWVzdCBvciB0aGUgZGF0YSwgbWFrZSBhIHJlYXNvbmFibGUgYXNzdW1wdGlvbiwgcHJvY2VlZCwKYW5kIHJlY29yZCB0aGUgYXNzdW1wdGlvbiBpbiBgTk9URVMubWRgLg==)\#\#MissinginformationTherequestiswrittenintheclient’sownwords\.Itwillnotspellouteverydetailyouneed\-\-dataconventions,units,edge\-casehandlingandsimilarpointsareoftenleftimplicitbecausetheclientconsidersthemobvious\.Thereisnooneavailabletoanswerquestions\.Wheresomethingcannotbedeterminedfromtherequestorthedata,makeareasonableassumption,proceed,andrecordtheassumptionin‘NOTES\.md‘\.

Interact condition block[⬇](data:text/plain;base64,IyMgTWlzc2luZyBpbmZvcm1hdGlvbgoKVGhlIHJlcXVlc3QgaXMgd3JpdHRlbiBpbiB0aGUgY2xpZW50J3Mgb3duIHdvcmRzLiBJdCB3aWxsIG5vdCBzcGVsbCBvdXQgZXZlcnkKZGV0YWlsIHlvdSBuZWVkIC0tIGRhdGEgY29udmVudGlvbnMsIHVuaXRzLCBlZGdlLWNhc2UgaGFuZGxpbmcgYW5kIHNpbWlsYXIgcG9pbnRzCmFyZSBvZnRlbiBsZWZ0IGltcGxpY2l0IGJlY2F1c2UgdGhlIGNsaWVudCBjb25zaWRlcnMgdGhlbSBvYnZpb3VzLgoKVGhlIGNsaWVudCBpcyByZWFjaGFibGUgYW5kIGV4cGVjdHMgdG8gYmUgY29uc3VsdGVkIG9uIHJlcXVpcmVtZW50cy4gSWYgc29tZXRoaW5nCmNhbm5vdCBiZSBkZXRlcm1pbmVkIGZyb20gdGhlIHJlcXVlc3Qgb3IgdGhlIGRhdGEsIGFzayB0aGVtLiBZb3UgbWF5IGFzayBtb3JlCnRoYW4gb25jZSwgYW5kIGF0IGFueSBwb2ludCBpbiB0aGUgd29yay4KCldoZW4geW91IGFzaywgcHV0IHRoZSBxdWVzdGlvbnMgYXQgdGhlIGVuZCBvZiB5b3VyIHJlcGx5IHVuZGVyIGEgaGVhZGluZwpgIyMgUXVlc3Rpb25zIGZvciB0aGUgY2xpZW50YCwgb25lIHF1ZXN0aW9uIHBlciBsaW5lLCBhbmQgc3RvcCB0aGVyZSAtLSB0aGUgY2xpZW50CndpbGwgYW5zd2VyIGluIHRoZSBuZXh0IG1lc3NhZ2UuIChBc2tpbmcgaW4gcGxhaW4gcHJvc2UgYWxzbyB3b3JrczsgdGhlIGhlYWRpbmcKanVzdCBtYWtlcyB0aGUgZXhjaGFuZ2UgY2xlYW5lci4pCgpGb3IgYW55dGhpbmcgeW91IGNob29zZSBub3QgdG8gYXNrIGFib3V0LCBtYWtlIGEgcmVhc29uYWJsZSBhc3N1bXB0aW9uLCBwcm9jZWVkLAphbmQgcmVjb3JkIHRoZSBhc3N1bXB0aW9uIGluIGBOT1RFUy5tZGAu)\#\#MissinginformationTherequestiswrittenintheclient’sownwords\.Itwillnotspellouteverydetailyouneed\-\-dataconventions,units,edge\-casehandlingandsimilarpointsareoftenleftimplicitbecausetheclientconsidersthemobvious\.Theclientisreachableandexpectstobeconsultedonrequirements\.Ifsomethingcannotbedeterminedfromtherequestorthedata,askthem\.Youmayaskmorethanonce,andatanypointinthework\.Whenyouask,putthequestionsattheendofyourreplyunderaheading‘\#\#Questionsfortheclient‘,onequestionperline,andstopthere\-\-theclientwillanswerinthenextmessage\.\(Askinginplainprosealsoworks;theheadingjustmakestheexchangecleaner\.\)Foranythingyouchoosenottoaskabout,makeareasonableassumption,proceed,andrecordtheassumptionin‘NOTES\.md‘\.

Interact\-Req condition block[⬇](data:text/plain;base64,IyMgTWlzc2luZyBpbmZvcm1hdGlvbgoKVGhlIHJlcXVlc3QgaXMgd3JpdHRlbiBpbiB0aGUgY2xpZW50J3Mgb3duIHdvcmRzLiBJdCB3aWxsIG5vdCBzcGVsbCBvdXQgZXZlcnkKZGV0YWlsIHlvdSBuZWVkIC0tIGRhdGEgY29udmVudGlvbnMsIHVuaXRzLCBlZGdlLWNhc2UgaGFuZGxpbmcgYW5kIHNpbWlsYXIgcG9pbnRzCmFyZSBvZnRlbiBsZWZ0IGltcGxpY2l0IGJlY2F1c2UgdGhlIGNsaWVudCBjb25zaWRlcnMgdGhlbSBvYnZpb3VzLgoKVGhlIGNsaWVudCBpcyByZWFjaGFibGUuIEJlZm9yZSB3cml0aW5nIGFueSBjb2RlLCBhc2sgdGhlIGNsaWVudCBhYm91dCBhbnl0aGluZwp5b3UgbmVlZCBzZXR0bGVkLCBhbmQga2VlcCBhc2tpbmcgdW50aWwgeW91IGhhdmUgd2hhdCB5b3UgbmVlZC4gRG8gbm90IHN0YXJ0CnRoZSB3b3JrIHVudGlsIHRoZW4uCgpBc2sgYWJvdXQgYW55dGhpbmcgdGhlIHJlcXVlc3QgYW5kIHRoZSBkYXRhIGRvIG5vdCBzZXR0bGUuIFlvdSBtYXkgYXNrIGFjcm9zcwpzZXZlcmFsIHJvdW5kcy4KCldoZW4geW91IGFzaywgcHV0IHRoZSBxdWVzdGlvbnMgYXQgdGhlIGVuZCBvZiB5b3VyIHJlcGx5IHVuZGVyIGEgaGVhZGluZwpgIyMgUXVlc3Rpb25zIGZvciB0aGUgY2xpZW50YCwgb25lIHF1ZXN0aW9uIHBlciBsaW5lLCBhbmQgc3RvcCB0aGVyZSAtLSB0aGUgY2xpZW50CndpbGwgYW5zd2VyIGluIHRoZSBuZXh0IG1lc3NhZ2UuIChBc2tpbmcgaW4gcGxhaW4gcHJvc2UgYWxzbyB3b3JrczsgdGhlIGhlYWRpbmcKanVzdCBtYWtlcyB0aGUgZXhjaGFuZ2UgY2xlYW5lci4pCgpQcm9jZWVkaW5nIG9uIGFuIHVuY29uZmlybWVkIHJlYWRpbmcgY291bnRzIGFzIGEgZmFpbHVyZSBldmVuIGlmIHRoZSBndWVzcwp0dXJucyBvdXQgcmlnaHQuIFJlY29yZCBpbiBgTk9URVMubWRgIGFueSBhc3N1bXB0aW9uIHRoZSBjbGllbnQgZGlkIG5vdCBzZXR0bGUu)\#\#MissinginformationTherequestiswrittenintheclient’sownwords\.Itwillnotspellouteverydetailyouneed\-\-dataconventions,units,edge\-casehandlingandsimilarpointsareoftenleftimplicitbecausetheclientconsidersthemobvious\.Theclientisreachable\.Beforewritinganycode,asktheclientaboutanythingyouneedsettled,andkeepaskinguntilyouhavewhatyouneed\.Donotstarttheworkuntilthen\.Askaboutanythingtherequestandthedatadonotsettle\.Youmayaskacrossseveralrounds\.Whenyouask,putthequestionsattheendofyourreplyunderaheading‘\#\#Questionsfortheclient‘,onequestionperline,andstopthere\-\-theclientwillanswerinthenextmessage\.\(Askinginplainprosealsoworks;theheadingjustmakestheexchangecleaner\.\)Proceedingonanunconfirmedreadingcountsasafailureeveniftheguessturnsoutright\.Recordin‘NOTES\.md‘anyassumptiontheclientdidnotsettle\.

Full condition block[⬇](data:text/plain;base64,IyMgQWRkaXRpb25hbCBjb250ZXh0CgpUaGUgY2xpZW50IGhhcyBhbHJlYWR5IHN1cHBsaWVkIHRoZSBmb2xsb3dpbmcgZGV0YWlscywgY292ZXJpbmcgdGhlIGRhdGEKY29udmVudGlvbnMsIHVuaXRzIGFuZCBlZGdlLWNhc2UgaGFuZGxpbmcgdGhhdCB0aGUgcmVxdWVzdCBpdHNlbGYgbGVhdmVzIGltcGxpY2l0OgoKe3tpbmZvcm1hdGlvbn19CgpUaGVyZSBpcyBubyBvbmUgYXZhaWxhYmxlIHRvIGFuc3dlciBmdXJ0aGVyIHF1ZXN0aW9ucy4gV2hlcmUgc29tZXRoaW5nIGlzIHN0aWxsCnVuZGV0ZXJtaW5lZCwgbWFrZSBhIHJlYXNvbmFibGUgYXNzdW1wdGlvbiwgcHJvY2VlZCwgYW5kIHJlY29yZCBpdCBpbiBgTk9URVMubWRgLg==)\#\#AdditionalcontextTheclienthasalreadysuppliedthefollowingdetails,coveringthedataconventions,unitsandedge\-casehandlingthattherequestitselfleavesimplicit:\{\{information\}\}Thereisnooneavailabletoanswerfurtherquestions\.Wheresomethingisstillundetermined,makeareasonableassumption,proceed,andrecorditin‘NOTES\.md‘\.

Oracle answer\-coverage condition block[⬇](data:text/plain;base64,IyMgTWlzc2luZyBpbmZvcm1hdGlvbgoKVGhlIHJlcXVlc3QgaXMgd3JpdHRlbiBpbiB0aGUgY2xpZW50J3Mgb3duIHdvcmRzLiBJdCB3aWxsIG5vdCBzcGVsbCBvdXQgZXZlcnkKZGV0YWlsIHlvdSBuZWVkIC0tIGRhdGEgY29udmVudGlvbnMsIHVuaXRzLCBlZGdlLWNhc2UgaGFuZGxpbmcgYW5kIHNpbWlsYXIgcG9pbnRzCmFyZSBvZnRlbiBsZWZ0IGltcGxpY2l0IGJlY2F1c2UgdGhlIGNsaWVudCBjb25zaWRlcnMgdGhlbSBvYnZpb3VzLgoKVGhlIGNsaWVudCBoYXMgYWxyZWFkeSBzZXR0bGVkIHRoZSBmb2xsb3dpbmcgcG9pbnRzIGZvciB5b3U6Cgp7e2Fuc3dlcnN9fQoKVGhlcmUgaXMgbm8gb25lIGF2YWlsYWJsZSB0byBhbnN3ZXIgZnVydGhlciBxdWVzdGlvbnMuIFdoZXJlIHNvbWV0aGluZyBzdGlsbApjYW5ub3QgYmUgZGV0ZXJtaW5lZCBmcm9tIHRoZSByZXF1ZXN0LCB0aGUgZGF0YSwgb3IgdGhlIHBvaW50cyBhYm92ZSwgbWFrZSBhCnJlYXNvbmFibGUgYXNzdW1wdGlvbiwgcHJvY2VlZCwgYW5kIHJlY29yZCB0aGUgYXNzdW1wdGlvbiBpbiBgTk9URVMubWRgLg==)\#\#MissinginformationTherequestiswrittenintheclient’sownwords\.Itwillnotspellouteverydetailyouneed\-\-dataconventions,units,edge\-casehandlingandsimilarpointsareoftenleftimplicitbecausetheclientconsidersthemobvious\.Theclienthasalreadysettledthefollowingpointsforyou:\{\{answers\}\}Thereisnooneavailabletoanswerfurtherquestions\.Wheresomethingstillcannotbedeterminedfromtherequest,thedata,orthepointsabove,makeareasonableassumption,proceed,andrecordtheassumptionin‘NOTES\.md‘\.

ForFull,informationis the complete business\-information file\. For the oracle experiment,answersis the selected answer text, rendered as bullets in registry order, or\(none\)for zero coverage\. The nested subset selection is specified in Appendix[A\.6](https://arxiv.org/html/2610.14539#A1.SS6.SSS0.Px4); requirement identifiers and type labels are not inserted into the oracle prompt\.

#### A\.9\.2Simulated\-customer messages

The customer uses GLM\-5\.3 and the default answering style below\. Each call has this system message, the accumulated user\-question/assistant\-answer history, and the current question as a new user message\. The returned answer is appended to that history\. The two knowledge fields are the initial request and complete business information; clarification annotations and evaluator code are not included\. The retained implementation requests at most 10,000 output tokens per call and leaves temperature and seed unset\.

Customer system prompt \(English translation\)[⬇](data:text/plain;base64,WW91IGFyZSB0aGUgcGVyc29uIHJlc3BvbnNpYmxlIGZvciB0aGlzIGJ1c2luZXNzIHNjZW5hcmlvLCBhbnN3ZXJpbmcgdGhlIHRlY2huaWNhbCB0ZWFtJ3MgY2xhcmlmaWNhdGlvbiBxdWVzdGlvbnMgYWJvdXQgdGhlIHJlcXVpcmVtZW50cy4KVGhlIGZvbGxvd2luZyBpcyB5b3VyIGJ1c2luZXNzIGtub3dsZWRnZS4gQmVmb3JlIGFuc3dlcmluZywgcmVhZCBlYWNoIGJsb2NrIGluIGZ1bGwgYW5kIGNyb3NzLWNoZWNrIHRoZW0uIEluIHBhcnRpY3VsYXIsIDxpbmZvcm1hdGlvbj4gb2Z0ZW4gY29udGFpbnMgc3BlY2lmaWMgdmFsdWVzLCBmaWVsZCBuYW1lcywgY29vcmRpbmF0ZXMsIHRocmVzaG9sZHMsIGFuZCBjb25zdHJhaW50IGRldGFpbHMuIFdoZW4gYXNrZWQgYWJvdXQgYSBzY2hlbWEsIHBhcmFtZXRlciwgdGhyZXNob2xkLCBjb29yZGluYXRlLCBmb3JtdWxhLCBvciBzaW1pbGFyIGRldGFpbCwgZmlyc3QgbG9vayBmb3IgdGhlIGV4aXN0aW5nIGFuc3dlciBpbiA8aW5mb3JtYXRpb24+OyBkbyBub3Qgc2F5ICJJIHdpbGwgcHJvdmlkZSBpdCIgb3IgZGVjaWRlIGl0IG9uIHRoZSBzcG90OgoKPGluc3RydWN0aW9uPgp7e2luc3RydWN0aW9ufX0KPC9pbnN0cnVjdGlvbj4KCjxpbmZvcm1hdGlvbj4Ke3tpbmZvcm1hdGlvbn19CjwvaW5mb3JtYXRpb24+CgpOb3RlOiBBbnkgb2YgdGhlIGFib3ZlIG1heSBjb250YWluIGludGVybmFsIHRlY2huaWNhbCBkZXRhaWxzIHN1Y2ggYXMgZXZhbHVhdGlvbiBmb3JtdWxhcywgc2NvcmluZyB3ZWlnaHRzLCBvciBwZW5hbHR5IGNvZWZmaWNpZW50cy4gRG8gbm90IHZvbHVudGVlciB0aGVzZSB0byB0aGUgdGVjaG5pY2FsIHRlYW0uIElmIGFza2VkLCB5b3UgbWF5IHByb3ZpZGUgYSBxdWFsaXRhdGl2ZSBidXNpbmVzcy1sZXZlbCBleHBsYW5hdGlvbiwgYnV0IGRvIG5vdCBnaXZlIHNwZWNpZmljIG1hdGhlbWF0aWNhbCBmb3JtdWxhcyBvciBudW1lcmljYWwgd2VpZ2h0cy4KCk1hbmRhdG9yeSBhbnN3ZXJpbmcgcHJvY2VkdXJlIChmb2xsb3cgdGhlc2UgdGhyZWUgc3RlcHMgZm9yIGV2ZXJ5IHF1ZXN0aW9uKToKClN0ZXAgMTogQ2hlY2sgaW5mb3JtYXRpb24gZmlyc3QgKGludGVybmFsbHk7IGRvIG5vdCBpbmNsdWRlIHRoaXMgc3RlcCBpbiB0aGUgcmVwbHkpLgpCZWZvcmUgc2VsZWN0aW5nIG9yIHdyaXRpbmcgYW4gYW5zd2VyLCBmaW5kIHRoZSBjb3JyZXNwb25kaW5nIHRleHQgaW4gPGluZm9ybWF0aW9uPiBhbmQgYW5zd2VyIG9uIHRoYXQgYmFzaXMuCllvdSBhcmUgcGxheWluZyB0aGUgYnVzaW5lc3Mgb3duZXI6IHRoaXMgaXMgYnVzaW5lc3Mga25vd2xlZGdlIGluIHlvdXIgaGVhZCwgbm90IGEgZG9jdW1lbnQgeW91IGNhbiBjb25zdWx0LgpEbyBub3Qgc2F5ICJjaGVja2luZyBpbmZvcm1hdGlvbiIsICJhY2NvcmRpbmcgdG8gdGhlIGRvY3VtZW50L21hdGVyaWFscy9pbnN0cnVjdGlvbnMiLCAib3JpZ2luYWwgZXhjZXJwdCIsIG9yICJub3QgZm91bmQiLCBvciBpbXBseSB0aGF0IHlvdSBhcmUgY29uc3VsdGluZyBtYXRlcmlhbHMuIEFuc3dlciBkaXJlY3RseSBpbiB5b3VyIG93biB2b2ljZS4KClN0ZXAgMjogQ29tcGFyZSB0aGUgb3B0aW9ucyB3aXRoIHRoZSBmYWN0cyBpbiBpbmZvcm1hdGlvbi4KLSBJZiBhbiBvcHRpb24gZXhhY3RseSBtYXRjaGVzIHRoZSBmYWN0cyBpbiBpbmZvcm1hdGlvbiwgc2VsZWN0IGl0LgotIElmIG5vbmUgb2YgdGhlIG9wdGlvbnMgZXhhY3RseSBtYXRjaGVzLCBzZWxlY3QgIk90aGVyIChwbGVhc2UgZGVzY3JpYmUpIiBhbmQgY29weSB0aGUgcmFuZ2VzLCB2YWx1ZXMsIG9yIGZpZWxkIG5hbWVzIGZyb20gaW5mb3JtYXRpb24uCi0gT25seSBpZiBpbmZvcm1hdGlvbiBkb2VzIG5vdCBhZGRyZXNzIHRoZSBxdWVzdGlvbiBtYXkgeW91IGRlY2lkZSBiYXNlZCBvbiBpbnN0cnVjdGlvbi4KClN0ZXAgMzogRG8gbm90IHJlcGxhY2Ugc3BlY2lmaWMgdmFsdWVzIHdpdGggdmFndWUgcXVhbGlmaWNhdGlvbnMuCkRvIG5vdCBhbnN3ZXIgInVucmVzdHJpY3RlZCAvIG5vdCBjb3VudGVkIC8gdW5uZWNlc3NhcnkgLyBmbGV4aWJsZSAvIGZvciB0aGUgbW9kZWwgdG8gZGVjaWRlIiB1bmxlc3MgaW5mb3JtYXRpb24gZXhwbGljaXRseSBzYXlzIHNvLiBDb3B5IHNwZWNpZmljIHZhbHVlcywgcmFuZ2VzLCBhbmQgZmllbGQgbmFtZXMgZnJvbSBpbmZvcm1hdGlvbi4KCklmIHNlbGVjdGluZyBhbiBvcHRpb24gdGhhdCBzYXlzICJwbGVhc2UgZGVzY3JpYmUiLCBwcm92aWRlIHRoZSBzcGVjaWZpYyBjb250ZW50IGFmdGVyIHRoZSBvcHRpb24gbmFtZS4gSWYgdGhlcmUgaXMgbm90aGluZyBuZXcgdG8gYWRkLCBzZWxlY3QgYSBuZWdhdGl2ZSBvcHRpb24gcmF0aGVyIHRoYW4gbGVhdmluZyB0aGUgZGVzY3JpcHRpb24gZW1wdHkuCgpBbnN3ZXIgcmVxdWlyZW1lbnRzOgotIFNwZWFrIGFzIHRoZSBidXNpbmVzcyBvd25lci4gR2l2ZSBvbmx5IHRoZSBhbnN3ZXIgaXRzZWxmLCB3aXRob3V0IGV4cGxhaW5pbmcgdGhlIHJlYXNvbmluZyBvciByZXBlYXRpbmcgdGhlIHF1ZXN0aW9uLgotIElmIHRoZXJlIGFyZSBtdWx0aXBsZSBxdWVzdGlvbnMsIGFuc3dlciBlYWNoIGRpcmVjdGx5IG9uIGEgc2VwYXJhdGUgbGluZS4KLSBEbyBub3Qgdm9sdW50ZWVyIGltcGxlbWVudGF0aW9uIGRldGFpbHMgc3VjaCBhcyBldmFsdWF0aW9uIG1ldGhvZHMsIHNjb3JpbmcgZm9ybXVsYXMsIG9yIGV4YWN0IG9iamVjdGl2ZSB3ZWlnaHRzLiBJZiBhc2tlZCwgeW91IG1heSBnaXZlIGEgcXVhbGl0YXRpdmUgYnVzaW5lc3MtbGV2ZWwgZXhwbGFuYXRpb24sIGJ1dCBub3Qgc3BlY2lmaWMgbWF0aGVtYXRpY2FsIGZvcm11bGFzIG9yIG51bWVyaWNhbCB3ZWlnaHRzLgotIElmIG5vbmUgb2YgdGhlIGluZm9ybWF0aW9uIHNvdXJjZXMgY292ZXJzIHRoZSBxdWVzdGlvbiwgcHJvdmlkZSBhIHJlYXNvbmFibGUgYnVzaW5lc3MgZGVmYXVsdC4=)Youarethepersonresponsibleforthisbusinessscenario,answeringthetechnicalteam’sclarificationquestionsabouttherequirements\.Thefollowingisyourbusinessknowledge\.Beforeanswering,readeachblockinfullandcross\-checkthem\.Inparticular,<information\>oftencontainsspecificvalues,fieldnames,coordinates,thresholds,andconstraintdetails\.Whenaskedaboutaschema,parameter,threshold,coordinate,formula,orsimilardetail,firstlookfortheexistinganswerin<information\>;donotsay"Iwillprovideit"ordecideitonthespot:<instruction\>\{\{instruction\}\}</instruction\><information\>\{\{information\}\}</information\>Note:Anyoftheabovemaycontaininternaltechnicaldetailssuchasevaluationformulas,scoringweights,orpenaltycoefficients\.Donotvolunteerthesetothetechnicalteam\.Ifasked,youmayprovideaqualitativebusiness\-levelexplanation,butdonotgivespecificmathematicalformulasornumericalweights\.Mandatoryansweringprocedure\(followthesethreestepsforeveryquestion\):Step1:Checkinformationfirst\(internally;donotincludethisstepinthereply\)\.Beforeselectingorwritingananswer,findthecorrespondingtextin<information\>andansweronthatbasis\.Youareplayingthebusinessowner:thisisbusinessknowledgeinyourhead,notadocumentyoucanconsult\.Donotsay"checkinginformation","accordingtothedocument/materials/instructions","originalexcerpt",or"notfound",orimplythatyouareconsultingmaterials\.Answerdirectlyinyourownvoice\.Step2:Comparetheoptionswiththefactsininformation\.\-Ifanoptionexactlymatchesthefactsininformation,selectit\.\-Ifnoneoftheoptionsexactlymatches,select"Other\(pleasedescribe\)"andcopytheranges,values,orfieldnamesfrominformation\.\-Onlyifinformationdoesnotaddressthequestionmayyoudecidebasedoninstruction\.Step3:Donotreplacespecificvalueswithvaguequalifications\.Donotanswer"unrestricted/notcounted/unnecessary/flexible/forthemodeltodecide"unlessinformationexplicitlysaysso\.Copyspecificvalues,ranges,andfieldnamesfrominformation\.Ifselectinganoptionthatsays"pleasedescribe",providethespecificcontentaftertheoptionname\.Ifthereisnothingnewtoadd,selectanegativeoptionratherthanleavingthedescriptionempty\.Answerrequirements:\-Speakasthebusinessowner\.Giveonlytheansweritself,withoutexplainingthereasoningorrepeatingthequestion\.\-Iftherearemultiplequestions,answereachdirectlyonaseparateline\.\-Donotvolunteerimplementationdetailssuchasevaluationmethods,scoringformulas,orexactobjectiveweights\.Ifasked,youmaygiveaqualitativebusiness\-levelexplanation,butnotspecificmathematicalformulasornumericalweights\.\-Ifnoneoftheinformationsourcescoversthequestion,provideareasonablebusinessdefault\.

This prompt allows business defaults for unsupported questions, so answers are not guaranteed to follow from the case information \(Appendix[A\.8](https://arxiv.org/html/2610.14539#A1.SS8)\)\.

#### A\.9\.3Question and answer interfaces

The harness supports native question requests, a scaffold callback, and text exchanges\. These routes share the customer above but preserve each scaffold’s interaction mechanism\. The following examples illustrate the message shapes; angle\-bracket values are placeholders, not recorded dialogue\.

##### OpenCode native question tool\.

The harness receives a request containing aquestionsarray and replies throughPOST /question/<request\-id\>/reply\(or the corresponding session\-scoped route in the older API\)\. Each answer is a one\-element string array, preserving free\-text answers rather than forcing them into an agent\-proposed option\. The customer receives eachquestionstring, falling back toheaderif it is empty; option metadata is recorded but is not separately appended to that string\.

Native question request and reply[⬇](data:text/plain;base64,UmVxdWVzdDoKeyJpZCI6ICI8cmVxdWVzdC1pZD4iLCAicXVlc3Rpb25zIjogWwogIHsicXVlc3Rpb24iOiAiPHF1ZXN0aW9uIHRleHQ+IiwgImhlYWRlciI6ICI8c2hvcnQgaGVhZGVyPiIsCiAgICJvcHRpb25zIjogW3sibGFiZWwiOiAiPG9wdGlvbj4iLCAiZGVzY3JpcHRpb24iOiAiPG1lYW5pbmc+In1dfQpdfQoKUmVwbHk6CnsiYW5zd2VycyI6IFtbIjxjdXN0b21lciBhbnN3ZXI+Il1dfQ==)Request:\{"id":"<request\-id\>","questions":\[\{"question":"<questiontext\>","header":"<shortheader\>","options":\[\{"label":"<option\>","description":"<meaning\>"\}\]\}\]\}Reply:\{"answers":\[\["<customeranswer\>"\]\]\}

##### Text exchange and OpenHands callback\.

The CLI configurations use text exchanges\. OpenHands obtains a reply via its user\-response callback, connected to the customer by the harness\. Questions under the suggested heading are parsed directly when possible; otherwise a detector determines whether the message requests information\. Each detected question is answered separately, and the answers return as one user message in the form below\. Recorded question units used for scoring may be split further as described in Appendix[A\.3](https://arxiv.org/html/2610.14539#A1.SS3.SSS0.Px3)\.

Text\-channel request and response[⬇](data:text/plain;base64,QWdlbnQgbWVzc2FnZToKIyMgUXVlc3Rpb25zIGZvciB0aGUgY2xpZW50CjxxdWVzdGlvbiAxPgo8cXVlc3Rpb24gMj4KCkN1c3RvbWVyIHJlcGx5IGRlbGl2ZXJlZCBhcyBhIHVzZXIgbWVzc2FnZToKLSA8cXVlc3Rpb24gMT4KICA8YW5zd2VyIDE+Ci0gPHF1ZXN0aW9uIDI+CiAgPGFuc3dlciAyPg==)Agentmessage:\#\#Questionsfortheclient<question1\><question2\>Customerreplydeliveredasausermessage:\-<question1\><answer1\>\-<question2\><answer2\>

The detector sees only agent text, not hidden requirements, and is skipped when the question heading is present\. Replies without a question mark are treated as non\-questions; otherwise five models vote by strict majority of returned ballots on whether the agent is asking\. Overlong replies keep the first 2,000 and last 8,000 characters\.

Question detector system message[⬇](data:text/plain;base64,WW91IGp1ZGdlIHdoZXRoZXIgYSBjb2RpbmcgYWdlbnQncyByZXBseSBpcyBBU0tJTkcgdGhlIGNsaWVudCBmb3IgaW5mb3JtYXRpb24sIG9yIFJFUE9SVElORy9ERUxJVkVSSU5HIHdvcmsuCgpSZXR1cm4gSlNPTiBvbmx5Ogp7ImFza2luZyI6IHRydWV8ZmFsc2UsICJxdWVzdGlvbnMiOiBbIi4uLiIsICIuLi4iXX0KClJ1bGVzOgotICJhc2tpbmciIGlzIHRydWUgb25seSBpZiB0aGUgYWdlbnQgaXMgd2FpdGluZyBmb3IgdGhlIGNsaWVudCB0byBzdXBwbHkKICBpbmZvcm1hdGlvbiBpdCBjYW5ub3QgZGV0ZXJtaW5lIGl0c2VsZi4KLSBSaGV0b3JpY2FsIHF1ZXN0aW9ucyBpbnNpZGUgYW4gZXhwbGFuYXRpb24gb3IgcmVwb3J0IGFyZSBOT1QgYXNraW5nLgogICgiSG93IGRvIHdlIGJhbGFuY2UgdGltZSBhZ2FpbnN0IGZhaXJuZXNzPyBXZSB3ZWlnaHQgdGhlbSAxOjkwLiIgLT4gZmFsc2UpCi0gT2ZmZXJzIHRvIGNvbnRpbnVlIGFyZSBOT1QgYXNraW5nLiAoIlNoYWxsIEkgYWxzbyBwbG90IHRoZSByb3V0ZXM/IiAtPiBmYWxzZSkKLSBTcGxpdCBjb21wb3VuZCBxdWVzdGlvbnMgaW50byBhdG9taWMgb25lcywgb25lIGZhY3QgcGVyIGVudHJ5LgogICgiV2hhdCBzcGVlZCwgYW5kIGlzIGxvYWRpbmcgdGltZSBpbmNsdWRlZD8iIC0+IHR3byBlbnRyaWVzKQotIEtlZXAgZWFjaCBxdWVzdGlvbiBpbiBpdHMgb3JpZ2luYWwgbGFuZ3VhZ2UsIGNsb3NlIHRvIHRoZSBhZ2VudCdzIHdvcmRpbmcuCi0gSWYgImFza2luZyIgaXMgZmFsc2UsICJxdWVzdGlvbnMiIG11c3QgYmUgW10u)Youjudgewhetheracodingagent’sreplyisASKINGtheclientforinformation,orREPORTING/DELIVERINGwork\.ReturnJSONonly:\{"asking":true\|false,"questions":\["\.\.\.","\.\.\."\]\}Rules:\-"asking"istrueonlyiftheagentiswaitingfortheclienttosupplyinformationitcannotdetermineitself\.\-RhetoricalquestionsinsideanexplanationorreportareNOTasking\.\("Howdowebalancetimeagainstfairness?Weweightthem1:90\."\-\>false\)\-OfferstocontinueareNOTasking\.\("ShallIalsoplottheroutes?"\-\>false\)\-Splitcompoundquestionsintoatomicones,onefactperentry\.\("Whatspeed,andisloadingtimeincluded?"\-\>twoentries\)\-Keepeachquestioninitsoriginallanguage,closetotheagent’swording\.\-If"asking"isfalse,"questions"mustbe\[\]\.

Question detector user message[⬇](data:text/plain;base64,QWdlbnQgcmVwbHk6Cgp7e2NsaXBwZWRfYWdlbnRfcmVwbHl9fXt7aGludH19)Agentreply:\{\{clipped\_agent\_reply\}\}\{\{hint\}\}

The detector requests at most 2,000 output tokens\. On malformed JSON it retries once, prepending the following text to the original user message:

Question detector JSON\-repair retry prefix[⬇](data:text/plain;base64,WW91ciBwcmV2aW91cyByZXNwb25zZSB2aW9sYXRlZCB0aGUgSlNPTiBwcm90b2NvbCAoVmFsdWVFcnJvcikuIFJldHVybiBleGFjdGx5IG9uZSBKU09OIG9iamVjdCB3aXRoIGtleXMgYXNraW5nIGFuZCBxdWVzdGlvbnMsIGFuZCBubyBvdGhlciB0ZXh0Lg==)YourpreviousresponseviolatedtheJSONprotocol\(ValueError\)\.ReturnexactlyoneJSONobjectwithkeysaskingandquestions,andnoothertext\.

The retained text loop sends the following continuation when an agent has stopped without a deliverable, subject to its run budget, and allows up to three such pushes in that branch\. Repeated identical question batches or exhausted clarification limits also trigger a transition to solving\.

Continuation and budget messages[⬇](data:text/plain;base64,Q29udGludWF0aW9uOgpVbmRlcnN0b29kLiBQbGVhc2Ugbm93IGNhcnJ5IG91dCB0aGUgd29yayBhbmQgd3JpdGUgdGhlIGRlbGl2ZXJhYmxlIGZpbGVzIGludG8gdGhlIHdvcmtpbmcgZGlyZWN0b3J5LgoKVGV4dC1jaGFubmVsIGNsYXJpZmljYXRpb24gYnVkZ2V0IGV4aGF1c3RlZDoKKE5vIGZ1cnRoZXIgcXVlc3Rpb25zIGNhbiBiZSBhbnN3ZXJlZC4gUHJvY2VlZCB3aXRoIHlvdXIgYmVzdCBqdWRnZW1lbnQuKQoKTmF0aXZlLXRvb2wgLyBjYWxsYmFjayBjbGFyaWZpY2F0aW9uIGJ1ZGdldCBleGhhdXN0ZWQ6Ck5vIGZ1cnRoZXIgaW5mb3JtYXRpb24gaXMgYXZhaWxhYmxlLiBQcm9jZWVkIHdpdGggeW91ciBiZXN0IGp1ZGdlbWVudC4=)Continuation:Understood\.Pleasenowcarryouttheworkandwritethedeliverablefilesintotheworkingdirectory\.Text\-channelclarificationbudgetexhausted:\(Nofurtherquestionscanbeanswered\.Proceedwithyourbestjudgement\.\)Native\-tool/callbackclarificationbudgetexhausted:Nofurtherinformationisavailable\.Proceedwithyourbestjudgement\.

If a customer call fails, the OpenCode handler returns a placeholder telling the agent to proceed on its own judgment\.

#### A\.9\.4Artifact\-extraction instructions

The extractor runs after the solving agent has finished\. The shared prompt below inserts the case’s completesubmission\_schema\.jsonunderTarget schema \(authoritative\)\. If that file is absent, it uses the evaluator’sEXTRACTOR\_SPECdocumentation, or directs the extractor to infer the schema from evaluator source\. Paths identify the retained agent workspace, case data, evaluator, and a separate normalized\-output directory\.

Shared extractor template[⬇](data:text/plain;base64,WW91IGFyZSBhIGRhdGEgZXh0cmFjdG9yIGZvciBhIGJlbmNobWFyayBldmFsdWF0aW9uIHBpcGVsaW5lLgoKIyMgWW91ciB0YXNrCgoxLiAqKlJlYWQgYF9hZ2VudF9zdW1tYXJ5Lm1kYCBmaXJzdCoqIChpZiBwcmVzZW50IGluIHRoZSB3b3Jrc3BhY2UpLiBJdCB0ZWxscyB5b3Ugd2hpY2ggZmlsZQogICB0aGUgYWdlbnQgY29uc2lkZXJzIGl0cyBmaW5hbCBhbnN3ZXIuIFRoZSBhZ2VudCdzIG93biBkZXNpZ25hdGlvbiBkZWNpZGVzIHdoaWNoIGFydGlmYWN0CiAgIGdldHMgZ3JhZGVkIC0tIHlvdSBkbyBub3QgZ2V0IHRvIHBpY2sgYSBkaWZmZXJlbnQgKGJldHRlci1zY29yaW5nKSBvbmUuCjIuIFJlYWQgdGhlIHRhcmdldCBzY2hlbWEgYmVsb3cgdG8gbGVhcm4gd2hhdCBmaWxlcyBhbmQgZmllbGRzIHRoZSBldmFsdWF0b3IgYWNjZXB0cy4KMy4gRmluZCB0aGUgYWdlbnQncyBvdXRwdXQgaW4gdGhlIHdvcmtzcGFjZS4gRmlsZW5hbWVzIGFuZCBjb2x1bW4gbmFtZXMgd2lsbCBkaWZmZXIuCjQuIE5vcm1hbGl6ZSBpdCB0byB0aGUgc2NoZW1hIGFuZCB3cml0ZSB0aGUgcmVzdWx0IHRvIHRoZSBPVVRQVVQgZGlyZWN0b3J5Lgo1LiAqKklmIHRoZSBkZWxpdmVyYWJsZSBpcyBleGVjdXRhYmxlIGNvZGUqKiAodGhlIHNjaGVtYSBhc2tzIGZvciBhIGAucHlgIGRlZmluaW5nIGEgc3BlY2lmaWMKICAgZnVuY3Rpb24gb3IgY2xhc3MpOiB0aGUgZXZhbHVhdG9yIGltcG9ydHMgaXQgZnJvbSB0aGUgT1VUUFVUIGRpcmVjdG9yeSwgc28gY29weSB0aGUgZW50cnkKICAgZmlsZSB1bmRlciB0aGUgcmVxdWlyZWQgbmFtZSBBTkQgZXZlcnkgd29ya3NwYWNlLWxvY2FsIG1vZHVsZSBpdCBpbXBvcnRzLCBkaXJlY3RseSBvcgogICB0cmFuc2l0aXZlbHkuIENvcHlpbmcgb25seSB0aGUgZW50cnkgZmlsZSBsZWF2ZXMgaXQgb3JwaGFuZWQgLS0gTW9kdWxlTm90Rm91bmRFcnJvciwgYW5kIGEKICAgd29ya2luZyBzb2x1dGlvbiBzY29yZXMgMC4KNi4gV3JpdGUgYF9yZXN1bHQuanNvbmAgdG8gdGhlIE9VVFBVVCBkaXJlY3RvcnkgYXMgeW91ciBmaW5hbCBhY3Rpb24uCgp7e3NjaGVtYV9ibG9ja319Cgp7e3ZlcmlmaWNhdGlvbl9ibG9ja319CgoqKlN0b3AgYXMgc29vbiBhcyB0aGUgZXZhbHVhdG9yIHJldHVybnMgYSBzY29yZS4qKiBBIDAgaXMgYSBsZWdpdGltYXRlIGZpbmFsIGFuc3dlcjogdGhlCmFnZW50J3Mgc29sdXRpb24gcmFuIGFuZCBkaWQgbm90IG1lZXQgdGhlIGNvbnN0cmFpbnRzLiBOZXZlciBlZGl0IHZhbHVlcywgYWRkIHJvd3MsIGRyb3AKcm93cywgb3Igc3dhcCBpbiBhIGRpZmZlcmVudCBzb2x1dGlvbiBmaWxlIHRvIHJhaXNlIHRoZSBudW1iZXIuIFlvdSBub3JtYWxpemUKKnJlcHJlc2VudGF0aW9uKiAtLSBmaWxlbmFtZXMsIGNvbHVtbiBuYW1lcywgbGF5b3V0LCBlbmNvZGluZy4gVGhlIGRlY2lzaW9ucyBhcmUgdGhlIGFnZW50J3MKYW5kIG11c3Qgc3Vydml2ZSB1bmNoYW5nZWQuIElmIG5vIHNjb3JlIGNhbiBiZSBvYnRhaW5lZCwgdGhlIGFydGlmYWN0IGlzIHdyb25nOyBzYXkgc28gYW5kCnN0b3AgcmF0aGVyIHRoYW4gbWFraW5nIHNvbWV0aGluZyB1cC4KCioqRm9yYmlkZGVuIHRyYW5zZm9ybWF0aW9ucy4qKiBUaGUgZm9sbG93aW5nIHRyYW5zZm9ybWF0aW9ucyB3b3VsZCBjaGFuZ2UgdGhlCmFnZW50J3Mgc3Vic3RhbnRpdmUgYW5zd2VyIGFuZCBhcmUgdGhlcmVmb3JlIGZvcmJpZGRlbi4KCjEuICoqTnVkZ2luZyB2YWx1ZXMgaW5zaWRlIGEgc3RhdGVkIHRvbGVyYW5jZS4qKiBEbyBub3QgYWRqdXN0IGEgbnVtYmVyIHdpdGhpbiBhbiBldmFsdWF0b3IKICAgdG9sZXJhbmNlIChvciBhbnkgb3RoZXIgc2xhY2spIHNvIHRoYXQgYSBmYWlsaW5nIGNvbnN0cmFpbnQgcGFzc2VzLiBUb2xlcmFuY2UgZXhpc3RzIHRvCiAgIGFic29yYiAqdGhlIGFnZW50J3MqIHJvdW5kaW5nLCBub3QgdG8gZ2l2ZSB0aGUgZXh0cmFjdG9yIGEgc2VhcmNoIHNwYWNlLiBJZiB0aGUgYWdlbnQncwogICBudW1iZXJzIGxhbmQgb3V0c2lkZSBhIGxpbWl0LCB0aGF0IGlzIHRoZSBhbnN3ZXI6IHJlcG9ydCB0aGUgMC4KMi4gKipXcml0aW5nIGFuIGFkYXB0ZXIgd2hlbiB0aGUgYWdlbnQncyBpbnRlcmZhY2UgZG9lcyBub3QgbWF0Y2ggdGhlIGV2YWx1YXRvcidzLioqIFdoZW4gYQogICBkZWxpdmVyYWJsZSBpcyBjb2RlIGFuZCBpdHMgc2lnbmF0dXJlIG9yIGRhdGEgbW9kZWwgZGlzYWdyZWVzIHdpdGggd2hhdCB0aGUgZXZhbHVhdG9yCiAgIHBhc3NlcyBpbiAoZS5nLiB0aGUgcG9saWN5IGV4cGVjdHMgYSBjdXN0b20gb2JqZWN0LCB0aGUgZXZhbHVhdG9yIGhhbmRzIGl0IGEgZGljdCksIGRvCiAgICoqbm90KiogYXV0aG9yIGEgc2hpbSwgYSB3cmFwcGVyIGNsYXNzLCBvciBhIHJlLWltcGxlbWVudGF0aW9uLiBDb3B5IHRoZSBhZ2VudCdzIGZpbGUKICAgYXMtaXMsIGxldCBpdCBmYWlsLCBhbmQgcmVjb3JkIHRoZSBtaXNtYXRjaCBpbiBgbm90ZXNgLgozLiAqKkRyb3BwaW5nIHJvd3MgdGhhdCAibG9vayB3cm9uZyIuKiogUmVtb3Zpbmcgcm93cyB0aGUgYWdlbnQgZW1pdHRlZCAtLSBldmVuIG9uZXMgdGhhdAogICBzZWVtIGxpa2UgYW5ub3RhdGlvbnMsIHBsYWNlaG9sZGVycywgb3IgaW5mZWFzaWJsZSBsZWZ0b3ZlcnMgLS0gY2hhbmdlcyB0aGUgYW5zd2VyLiBJZiB5b3UKICAgZHJvcCBvciBhZGQgKiphbnkqKiByb3csIHN0YXRlIHRoZSBiZWZvcmUvYWZ0ZXIgY291bnRzIGFuZCB0aGUgcmVhc29uIGluIGBub3Rlc2AuCgo0LiAqKiJGaXhpbmciIGFuIGlkZW50aWZpZXIgdGhlIGFnZW50IHNwZWxsZWQgd3JvbmcuKiogV2hldGhlciB5b3UgbWF5IHJlcGFpciBhIGJhZCBuYW1lCiAgIGRlcGVuZHMgb24gb25lIHF1ZXN0aW9uOiAqKmRvZXMgdGhlIHRhc2sgc3BlY2lmeSB0aGUgbmFtaW5nPyoqCiAgIC0gKipUYXNrIGlzIHNpbGVudCBvbiBuYW1pbmcqKiAoZS5nLiB0aGUgYWdlbnQgbGFiZWxzIGEgcGxhdGUgYHBsYXRlMWAgd2hlcmUgdGhlIGV2YWx1YXRvcgogICAgIGtleXMgb24gYFAwMDFgKTogbWFwIGl0LCBwcm92aWRlZCB0aGUgdGFyZ2V0IGlzICoqdW5pcXVlKiogaW4gdGhlIHJlZmVyZW5jZSBkYXRhLiBUaGlzCiAgICAgaXMgb3JkaW5hcnkgaWRlbnRpZmllciBub3JtYWxpemF0aW9uLiBJZiB0aGUgbWFwcGluZyBpcyBhbWJpZ3VvdXMgb3IgaGFzIG5vIG1hdGNoLAogICAgIGRvIG5vdCBndWVzcyAtLSBsZWF2ZSBpdCBhbmQgc2F5IHNvLgogICAtICoqVGFzayBzdGF0ZXMgdGhlIG5hbWluZyBydWxlKiogKGUuZy4gYGluZm9ybWF0aW9uLm1kYCBzYXlzIGZlYXR1cmUgbmFtZXMgbXVzdCBiZQogICAgIHJlYWwgY29sdW1ucyBvZiB0aGUgdHJhaW5pbmcgZGF0YSwgaW4gYHt0cmFpdH1fRHtkYXl9YCBmb3JtKTogYSBuYW1lIHRoYXQgYnJlYWtzIHRoZQogICAgIHJ1bGUgaXMgYSAqKnN1YnN0YW50aXZlIGVycm9yIGJ5IHRoZSBhZ2VudCoqLCBub3QgYSBmb3JtYXR0aW5nIHNsaXAuIENvcHkgaXQgYXMtaXMgYW5kCiAgIGxldCBpdCBzY29yZSAwLgogICBDaGVjayBgaW5mb3JtYXRpb24ubWRgIC8gYGluc3RydWN0aW9uLm1kYCBmb3IgYSBuYW1pbmcgcnVsZSAqKmJlZm9yZSoqIHlvdSByZXBhaXIgYW55CiAgIG5hbWUuIEFsc28gY2hlY2sgdGhlIHNjaGVtYSdzIGByb2xlYDogcmVwYWlyaW5nIGlzIG9ubHkgZXZlciBpbiBzY29wZSBmb3IgYGlkZW50aWZpZXJgCiAgIGZpZWxkcywgbmV2ZXIgZm9yIGBkZWNpc2lvbmAgZmllbGRzIC0tIHRob3NlIGFyZSB0aGUgYW5zd2VyIGl0c2VsZi4KCjUuICoqVXNpbmcgdGhlIGV2YWx1YXRvcidzIG93biBmb3JtdWxhIHRvIGdlbmVyYXRlIGEgdmFsdWUgdGhlIGV2YWx1YXRvciBjaGVja3MuKiogV2hlbiB0aGUKICAgc2NoZW1hIGRvY3VtZW50cyBob3cgYSBmaWVsZCByZWxhdGVzIHRvIG90aGVycyAoYHN1Yl9pX2xvbmcgbXVzdCBlcXVhbAogICBtb3RoZXJfY3V0X3dlaWdodCAvIChtb3RoZXJfd2lkdGggeCA3LjY1KSB4IDEwMDAwYCksIHRoYXQgZm9ybXVsYSBpcyB0aGVyZSBzbyB5b3UgY2FuCiAgIHVuZGVyc3RhbmQgdGhlIGZpZWxkIC0tICoqbm90IHNvIHlvdSBjYW4gY29tcHV0ZSBpdCoqLiBGaWxsaW5nIHRoZSBmaWVsZCBmcm9tIHRoZSBmb3JtdWxhCiAgIG1ha2VzIHRoZSBjb25zaXN0ZW5jeSBjaGVjayBwYXNzIGJ5IGNvbnN0cnVjdGlvbjogdGhlIGV2YWx1YXRvciBlbmRzIHVwIHZlcmlmeWluZyBpdHMgb3duCiAgIGFyaXRobWV0aWMgaW5zdGVhZCBvZiB0aGUgYWdlbnQncy4gQSBkZXJpdmVkIHZhbHVlIHRoYXQgdGhlIGV2YWx1YXRvciB2YWxpZGF0ZXMgaXMKICAgbG9hZC1iZWFyaW5nIGV2ZW4gd2hlbiBpdCBpcyBub3QgaXRzZWxmIGEgZGVjaXNpb24gdmFyaWFibGUuCiAgIFRoZSB0ZXN0IGlzIG5vdCAiaXMgdGhpcyBmaWVsZCBhIGRlY2lzaW9uPyIgLS0gaXQgaXMgKioiZG9lcyB0aGUgZXZhbHVhdG9yIGxvb2sgYXQgaXQ/IioqCiAgIElmIHllcywgdGhlIHZhbHVlIG11c3QgY29tZSBmcm9tIHRoZSBhZ2VudCwgZXZlbiB3aGVuIHlvdSBjYW4gY29tcHV0ZSBhIGJldHRlciBvbmUsIGFuZAogICBldmVuIHdoZW4gdGhlIGFnZW50J3MgaXMgcHJvdmFibHkgaW5jb25zaXN0ZW50LiBBbiBpbmNvbnNpc3RlbmN5IHlvdSBmb3VuZCBpcyBhIGZpbmRpbmcgdG8KICAgcmVwb3J0IGluIGBub3Rlc2AsIG5vdCBhIGRlZmVjdCB0byByZXBhaXIuCgojIyBXaGVuIHRoZSBhZ2VudCdzIGRlc2lnbmF0ZWQgZmlsZSBpcyBpbmNvbXBsZXRlCgpSdWxlIDEgc2F5cyB0aGUgYWdlbnQncyBvd24gZGVzaWduYXRpb24gZGVjaWRlcyB3aGF0IGdldHMgZ3JhZGVkLiBUaGF0IGFzc3VtZXMgdGhlCmRlc2lnbmF0ZWQgYXJ0aWZhY3QgZXhpc3RzIGFuZCBpcyBjb2hlcmVudC4gV2hlbiBpdCBpcyAqKm5vdCoqOgoKLSAqKlRoZSBkZXNpZ25hdGlvbiBuYW1lcyBhIHN0YXRlIHRoZSBhZ2VudCBuZXZlciBmaW5pc2hlZC4qKiBJZiB0aGUgZGVzaWduYXRlZCBhcnRpZmFjdCBpcwogIGluY29tcGxldGUgb3IgaW50ZXJuYWxseSBjb250cmFkaWN0b3J5LCBmYWxsIGJhY2sgdG8gdGhlIG1vc3QgcmVjZW50IGNvbXBsZXRlIHN0YXRlIHRoYXQKICBwYXNzZXMgdGhlIGFnZW50J3Mgb3duIHZhbGlkYXRpb24sIHNldCBgc3RhdHVzYCB0byBgcGFydGlhbGAsIGFuZCBzYXkgaW4gYG5vdGVzYCB3aGljaCBzdGF0ZQogIHlvdSB1c2VkIGFuZCB3aHkgdGhlIGRlc2lnbmF0ZWQgb25lIHdhcyBsZWZ0IG91dC4gVGhpcyBpcyBub3QgInBpY2tpbmcgdGhlIGJldHRlci1zY29yaW5nCiAgZmlsZSIgLS0gaXQgaXMgc2VsZWN0aW5nIHRoZSBvbmx5IGZpbGUgdGhhdCBpcyBhIGNvaGVyZW50IGFuc3dlci4KLSAqKkV4dHJhIHJvd3MgYmV5b25kIHRoZSBncmFkZWQgc2V0LioqIFJvd3MgdGhlIGFnZW50IHByb2R1Y2VkIGZvciBrZXlzIHRoYXQgYXJlIG5vdCBpbiB0aGUKICBldmFsdWF0aW9uIHNldCBzaG91bGQgYmUgZHJvcHBlZCwgd2l0aCB0aGUgYmVmb3JlL2FmdGVyIGNvdW50cyBkaXNjbG9zZWQuIFRoYXQgaXMgZGlmZmVyZW50CiAgZnJvbSBkcm9wcGluZyByb3dzIHRoZSBhZ2VudCAqZGlkKiBwcm9kdWNlIGZvciBncmFkZWQga2V5cywgd2hpY2ggaXMgZm9yYmlkZGVuLgoKIyMgVmVyaWZ5IG9yZGVyLWJlYXJpbmcgZmllbGRzIGJlZm9yZSB5b3Ugd3JpdGUKCldoZW4gYSBmaWVsZCBlbmNvZGVzIGEgc2VxdWVuY2UgKGEgdmlzaXQgb3JkZXIsIGEgcGVybXV0YXRpb24sIG9yIGEgY29uY2F0ZW5hdGVkIHJvdXRlKSwgaXRzCm9yZGVyICoqaXMqKiB0aGUgZGVjaXNpb24uIFJlY29uc3RydWN0aW5nIGl0IGZyb20gYSBncm91cGVkIHNvdXJjZSBjYW4gc2lsZW50bHkgc2NyYW1ibGUgdGhlCmdyb3VwIG9yZGVyIGV2ZW4gd2hlbiBhbGwgbWVtYmVycyBhbmQgY291bnRzIG1hdGNoLgoKQmVmb3JlIHdyaXRpbmcgYW4gb3JkZXItYmVhcmluZyBmaWVsZCwgcmUtZGVyaXZlIGl0IGZyb20gdGhlIHNvdXJjZSBhIHNlY29uZCB3YXkgYW5kIGNvbXBhcmUKZWxlbWVudCBieSBlbGVtZW50LiBJZiB0aGUgc2NoZW1hIGdpdmVzIHlvdSBhIGNvbXBhbmlvbiBzdHJ1Y3R1cmUgKGEgYHRyaXBzYCBhcnJheSB3aXRoIHBlci0KdHJpcCBzdG9wIGNvdW50cywgc2F5KSwgc2xpY2UgeW91ciBzZXF1ZW5jZSBieSB0aG9zZSBjb3VudHMgYW5kIGNoZWNrIGVhY2ggc2xpY2UgYWdhaW5zdCB0aGUKc291cmNlIGdyb3VwIGl0IHNob3VsZCBtYXRjaC4KClJ1bGUgb2YgdGh1bWI6IGlmIGFuIGVkaXQgd291bGQgY2hhbmdlIHRoZSBzY29yZSwgaXQgaXMgbm90IG5vcm1hbGl6YXRpb24uIFJlZm9ybWF0dGluZywKcmVuYW1pbmcsIHJlc2hhcGluZywgcmUtZW5jb2RpbmcgbmV2ZXIgY2hhbmdlIGEgc2NvcmUgLS0gdGhvc2UgYXJlIHlvdXJzIHRvIG1ha2UgZnJlZWx5LgoKIyMgTm9ybWFsaXphdGlvbiBydWxlcwoKLSAqKlNlbWFudGljLWZpcnN0IG1hcHBpbmcqKjogdGhlIHNhbWUgY29sdW1uIG5hbWUgZG9lcyBub3QgZ3VhcmFudGVlIHRoZSBzYW1lIG1lYW5pbmcuCiAgQ2hlY2sgdGhlIHNjaGVtYSdzIGZpZWxkIGRlZmluaXRpb24gKHVuaXRzLCBzY29wZSwgdG90YWwgdnMuIG5ldCkgYmVmb3JlIHJlbmFtaW5nLgotICoqQ29sdW1uIHJlbmFtaW5nIC8gZm9ybWF0IGNvbnZlcnNpb24qKjogZnV6enktbWF0Y2ggYWdlbnQgY29sdW1ucyB0byBzY2hlbWEgZmllbGRzOwogIGNvbnZlcnQgd2lkZS9sb25nIGxheW91dHMgYXMgbmVlZGVkLgotICoqTWlzc2luZyByb3dzKio6IG9ubHkgZmlsbCBhIG1pc3NpbmcgY29tYmluYXRpb24gd2l0aCB6ZXJvIHdoZW4gdGhlIHNjaGVtYSBkZWZpbmVzIHRoYXQKICBkaW1lbnNpb24ncyBhYnNlbmNlIEFTIHplcm8uIERvIE5PVCBpbnZlbnQgcm93cyBmb3IgdGFza3MgdGhlIGFnZW50IGRpZCBub3QgcHJvZHVjZSAtLQogIGxlYXZlIHRoZW0gb3V0IGFuZCBzZXQgc3RhdHVzIHRvICJwYXJ0aWFsIi4KLSAqKk5ldmVyKiogcmUtcnVuIHRoZSBhZ2VudCdzIHNvbHZlciwgcmUtb3B0aW1pemUsIG9yIHJlLWNvbXB1dGUgaXRzIGRlY2lzaW9ucy4KLSAqKlByZWZlciB0aGUgaGlnaGVzdC1wcmVjaXNpb24gYXJ0aWZhY3QuKiogV2hlbiB0aGUgYWdlbnQgZXhwcmVzc2VzIHRoZSBzYW1lIGRlY2lzaW9ucyBpbgogIHNldmVyYWwgZmlsZXMgLS0gc2F5IGEgbWludXRlLXRydW5jYXRlZCBDU1YgYW5kIGEgZnVsbC1wcmVjaXNpb24gSlNPTiAtLSB0YWtlIHRoZSBsb3NzbGVzcwogIG9uZSwgZXZlbiBpZiBgX2FnZW50X3N1bW1hcnkubWRgIG5hbWVzIHRoZSBvdGhlci4gUHJlY2lzaW9uIGxvc3MgaXMgbm90IGEgZm9ybWF0dGluZwogIGNob2ljZTsgaXQgc2lsZW50bHkgY2hhbmdlcyB0aGUgYW5zd2VyIGJlaW5nIGdyYWRlZC4KLSAqKk5ldmVyIHRha2UgYW55dGhpbmcgZnJvbSB0aGUgRGF0YSBkaXJlY3RvcnkgYXMgdGhlIGFnZW50J3Mgb3V0cHV0LioqIGBkYXRhL2AgaG9sZHMgdGhlCiAgcHJvYmxlbSBpbnB1dCwgaW5jbHVkaW5nIGFueSBiYXNlbGluZSBvciBzZWVkIHBvbGljeSBzaGlwcGVkIHdpdGggdGhlIHRhc2suIFN1Ym1pdHRpbmcgb25lCiAgb2YgdGhvc2UgZ3JhZGVzIHRoZSBiZW5jaG1hcmsncyBvd24gcmVmZXJlbmNlIGluc3RlYWQgb2YgdGhlIGFnZW50IC0tIGlmIHRoZSB3b3Jrc3BhY2UgaGFzCiAgbm8gcmVhbCBkZWxpdmVyYWJsZSwgc2F5IHNvIGFuZCBzZXQgc3RhdHVzIHRvICJmYWlsZWQiLgoKIyMgUGF0aHMgJiBwZXJtaXNzaW9ucwoKfCBEaXJlY3RvcnkgfCBQYXRoIHwgUmVhZCB8IFdyaXRlIHwKfC0tLS0tLS0tLS0tfC0tLS0tLXwtLS0tLS18LS0tLS0tLXwKfCBXb3Jrc3BhY2UgfCBge3t3b3Jrc3BhY2V9fWAgfCBZRVMgfCAqKk5PKiogfAp8IERhdGEgICAgICB8IGB7e2RhdGFfZGlyfX1gIHwgWUVTIHwgKipOTyoqIHwKfCBFdmFsdWF0b3IgfCBge3tldmFsdWF0b3J9fWAgfCBZRVMgfCAqKk5PKiogfAp8IE91dHB1dCAgICB8IGB7e291dHB1dF9kaXJ9fWAgfCBZRVMgfCAqKllFUyoqIHwKCkFsbCB3cml0ZXMgZ28gdG8gdGhlIE91dHB1dCBkaXJlY3Rvcnkgb25seS4KCiMjIF9yZXN1bHQuanNvbiBmb3JtYXQKCnsKICAic3RhdHVzIjogInN1Y2Nlc3MiLAogICJub3RlcyI6ICJ3aGF0IHdhcyBmb3VuZCBhbmQgd2hhdCB0cmFuc2Zvcm1hdGlvbnMgd2VyZSBhcHBsaWVkIiwKICAiZmlsZXNfZm91bmQiOiBbInJlbGF0aXZlIHBhdGhzIG9mIGFnZW50IG91dHB1dCBmaWxlcyBkaXNjb3ZlcmVkIl0sCiAgImZpbGVzX25vcm1hbGl6ZWQiOiBbImZpbGVuYW1lcyB3cml0dGVuIHRvIHRoZSBvdXRwdXQgZGlyZWN0b3J5IChleGNsdWRpbmcgX3Jlc3VsdC5qc29uKSJdCn0KClN0YXR1czogInN1Y2Nlc3MiID0gYWxsIHJlcXVpcmVkIGZpbGVzIHdyaXR0ZW47ICJwYXJ0aWFsIiA9IGZvdW5kIG91dHB1dCBidXQgaXQgaXMKaW5jb21wbGV0ZSBvciBzZWxmLWNvbnRyYWRpY3RvcnkgKGRlc2NyaWJlIHdoYXQgaXMgbWlzc2luZyk7ICJmYWlsZWQiID0gbm8gcmVsZXZhbnQgYWdlbnQKb3V0cHV0IGZvdW5kLiBXaGVuIHRoZSBhZ2VudCdzIG91dHB1dCBpcyBnZW51aW5lbHkgaW5jb21wbGV0ZSwgInBhcnRpYWwiIGlzIHRoZSBjb3JyZWN0Cm91dGNvbWUgLS0gZG8gTk9UIHJlY29uc3RydWN0IG9yIHJlLXNvbHZlIHRvIHJlYWNoICJzdWNjZXNzIi4=)Youareadataextractorforabenchmarkevaluationpipeline\.\#\#Yourtask1\.\*\*Read‘\_agent\_summary\.md‘first\*\*\(ifpresentintheworkspace\)\.Ittellsyouwhichfiletheagentconsidersitsfinalanswer\.Theagent’sowndesignationdecideswhichartifactgetsgraded\-\-youdonotgettopickadifferent\(better\-scoring\)one\.2\.Readthetargetschemabelowtolearnwhatfilesandfieldstheevaluatoraccepts\.3\.Findtheagent’soutputintheworkspace\.Filenamesandcolumnnameswilldiffer\.4\.NormalizeittotheschemaandwritetheresulttotheOUTPUTdirectory\.5\.\*\*Ifthedeliverableisexecutablecode\*\*\(theschemaasksfora‘\.py‘definingaspecificfunctionorclass\):theevaluatorimportsitfromtheOUTPUTdirectory,socopytheentryfileundertherequirednameANDeveryworkspace\-localmoduleitimports,directlyortransitively\.Copyingonlytheentryfileleavesitorphaned\-\-ModuleNotFoundError,andaworkingsolutionscores0\.6\.Write‘\_result\.json‘totheOUTPUTdirectoryasyourfinalaction\.\{\{schema\_block\}\}\{\{verification\_block\}\}\*\*Stopassoonastheevaluatorreturnsascore\.\*\*A0isalegitimatefinalanswer:theagent’ssolutionrananddidnotmeettheconstraints\.Nevereditvalues,addrows,droprows,orswapinadifferentsolutionfiletoraisethenumber\.Younormalize\*representation\*\-\-filenames,columnnames,layout,encoding\.Thedecisionsaretheagent’sandmustsurviveunchanged\.Ifnoscorecanbeobtained,theartifactiswrong;saysoandstopratherthanmakingsomethingup\.\*\*Forbiddentransformations\.\*\*Thefollowingtransformationswouldchangetheagent’ssubstantiveanswerandarethereforeforbidden\.1\.\*\*Nudgingvaluesinsideastatedtolerance\.\*\*Donotadjustanumberwithinanevaluatortolerance\(oranyotherslack\)sothatafailingconstraintpasses\.Toleranceexiststoabsorb\*theagent’s\*rounding,nottogivetheextractorasearchspace\.Iftheagent’snumberslandoutsidealimit,thatistheanswer:reportthe0\.2\.\*\*Writinganadapterwhentheagent’sinterfacedoesnotmatchtheevaluator’s\.\*\*Whenadeliverableiscodeanditssignatureordatamodeldisagreeswithwhattheevaluatorpassesin\(e\.g\.thepolicyexpectsacustomobject,theevaluatorhandsitadict\),do\*\*not\*\*authorashim,awrapperclass,orare\-implementation\.Copytheagent’sfileas\-is,letitfail,andrecordthemismatchin‘notes‘\.3\.\*\*Droppingrowsthat"lookwrong"\.\*\*Removingrowstheagentemitted\-\-evenonesthatseemlikeannotations,placeholders,orinfeasibleleftovers\-\-changestheanswer\.Ifyoudroporadd\*\*any\*\*row,statethebefore/aftercountsandthereasonin‘notes‘\.4\.\*\*"Fixing"anidentifiertheagentspelledwrong\.\*\*Whetheryoumayrepairabadnamedependsononequestion:\*\*doesthetaskspecifythenaming?\*\*\-\*\*Taskissilentonnaming\*\*\(e\.g\.theagentlabelsaplate‘plate1‘wheretheevaluatorkeyson‘P001‘\):mapit,providedthetargetis\*\*unique\*\*inthereferencedata\.Thisisordinaryidentifiernormalization\.Ifthemappingisambiguousorhasnomatch,donotguess\-\-leaveitandsayso\.\-\*\*Taskstatesthenamingrule\*\*\(e\.g\.‘information\.md‘saysfeaturenamesmustberealcolumnsofthetrainingdata,in‘\{trait\}\_D\{day\}‘form\):anamethatbreakstheruleisa\*\*substantiveerrorbytheagent\*\*,notaformattingslip\.Copyitas\-isandletitscore0\.Check‘information\.md‘/‘instruction\.md‘foranamingrule\*\*before\*\*yourepairanyname\.Alsochecktheschema’s‘role‘:repairingisonlyeverinscopefor‘identifier‘fields,neverfor‘decision‘fields\-\-thosearetheansweritself\.5\.\*\*Usingtheevaluator’sownformulatogenerateavaluetheevaluatorchecks\.\*\*Whentheschemadocumentshowafieldrelatestoothers\(‘sub\_i\_longmustequalmother\_cut\_weight/\(mother\_widthx7\.65\)x10000‘\),thatformulaistheresoyoucanunderstandthefield\-\-\*\*notsoyoucancomputeit\*\*\.Fillingthefieldfromtheformulamakestheconsistencycheckpassbyconstruction:theevaluatorendsupverifyingitsownarithmeticinsteadoftheagent’s\.Aderivedvaluethattheevaluatorvalidatesisload\-bearingevenwhenitisnotitselfadecisionvariable\.Thetestisnot"isthisfieldadecision?"\-\-itis\*\*"doestheevaluatorlookatit?"\*\*Ifyes,thevaluemustcomefromtheagent,evenwhenyoucancomputeabetterone,andevenwhentheagent’sisprovablyinconsistent\.Aninconsistencyyoufoundisafindingtoreportin‘notes‘,notadefecttorepair\.\#\#Whentheagent’sdesignatedfileisincompleteRule1saystheagent’sowndesignationdecideswhatgetsgraded\.Thatassumesthedesignatedartifactexistsandiscoherent\.Whenitis\*\*not\*\*:\-\*\*Thedesignationnamesastatetheagentneverfinished\.\*\*Ifthedesignatedartifactisincompleteorinternallycontradictory,fallbacktothemostrecentcompletestatethatpassestheagent’sownvalidation,set‘status‘to‘partial‘,andsayin‘notes‘whichstateyouusedandwhythedesignatedonewasleftout\.Thisisnot"pickingthebetter\-scoringfile"\-\-itisselectingtheonlyfilethatisacoherentanswer\.\-\*\*Extrarowsbeyondthegradedset\.\*\*Rowstheagentproducedforkeysthatarenotintheevaluationsetshouldbedropped,withthebefore/aftercountsdisclosed\.Thatisdifferentfromdroppingrowstheagent\*did\*produceforgradedkeys,whichisforbidden\.\#\#Verifyorder\-bearingfieldsbeforeyouwriteWhenafieldencodesasequence\(avisitorder,apermutation,oraconcatenatedroute\),itsorder\*\*is\*\*thedecision\.Reconstructingitfromagroupedsourcecansilentlyscramblethegrouporderevenwhenallmembersandcountsmatch\.Beforewritinganorder\-bearingfield,re\-deriveitfromthesourceasecondwayandcompareelementbyelement\.Iftheschemagivesyouacompanionstructure\(a‘trips‘arraywithper\-tripstopcounts,say\),sliceyoursequencebythosecountsandcheckeachsliceagainstthesourcegroupitshouldmatch\.Ruleofthumb:ifaneditwouldchangethescore,itisnotnormalization\.Reformatting,renaming,reshaping,re\-encodingneverchangeascore\-\-thoseareyourstomakefreely\.\#\#Normalizationrules\-\*\*Semantic\-firstmapping\*\*:thesamecolumnnamedoesnotguaranteethesamemeaning\.Checktheschema’sfielddefinition\(units,scope,totalvs\.net\)beforerenaming\.\-\*\*Columnrenaming/formatconversion\*\*:fuzzy\-matchagentcolumnstoschemafields;convertwide/longlayoutsasneeded\.\-\*\*Missingrows\*\*:onlyfillamissingcombinationwithzerowhentheschemadefinesthatdimension’sabsenceASzero\.DoNOTinventrowsfortaskstheagentdidnotproduce\-\-leavethemoutandsetstatusto"partial"\.\-\*\*Never\*\*re\-runtheagent’ssolver,re\-optimize,orre\-computeitsdecisions\.\-\*\*Preferthehighest\-precisionartifact\.\*\*Whentheagentexpressesthesamedecisionsinseveralfiles\-\-sayaminute\-truncatedCSVandafull\-precisionJSON\-\-takethelosslessone,evenif‘\_agent\_summary\.md‘namestheother\.Precisionlossisnotaformattingchoice;itsilentlychangestheanswerbeinggraded\.\-\*\*NevertakeanythingfromtheDatadirectoryastheagent’soutput\.\*\*‘data/‘holdstheprobleminput,includinganybaselineorseedpolicyshippedwiththetask\.Submittingoneofthosegradesthebenchmark’sownreferenceinsteadoftheagent\-\-iftheworkspacehasnorealdeliverable,saysoandsetstatusto"failed"\.\#\#Paths&permissions\|Directory\|Path\|Read\|Write\|\|\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\|\-\-\-\-\-\-\|\-\-\-\-\-\-\-\|\|Workspace\|‘\{\{workspace\}\}‘\|YES\|\*\*NO\*\*\|\|Data\|‘\{\{data\_dir\}\}‘\|YES\|\*\*NO\*\*\|\|Evaluator\|‘\{\{evaluator\}\}‘\|YES\|\*\*NO\*\*\|\|Output\|‘\{\{output\_dir\}\}‘\|YES\|\*\*YES\*\*\|AllwritesgototheOutputdirectoryonly\.\#\#\_result\.jsonformat\{"status":"success","notes":"whatwasfoundandwhattransformationswereapplied","files\_found":\["relativepathsofagentoutputfilesdiscovered"\],"files\_normalized":\["filenameswrittentotheoutputdirectory\(excluding\_result\.json\)"\]\}Status:"success"=allrequiredfileswritten;"partial"=foundoutputbutitisincompleteorself\-contradictory\(describewhatismissing\);"failed"=norelevantagentoutputfound\.Whentheagent’soutputisgenuinelyincomplete,"partial"isthecorrectoutcome\-\-doNOTreconstructorre\-solvetoreach"success"\.

Theverification\_blockuses the first block below for ordinary cases\. Case 13, whose evaluator launches expensive solver subprocesses, uses the second block and is scored after the extractor exits\.

Standard extractor verification block[⬇](data:text/plain;base64,IyMgVmVyaWZ5IGJ5IHJ1bm5pbmcgdGhlIGV2YWx1YXRvciAocmVxdWlyZWQpCgpBIHNjaGVtYSBjYW5ub3QgZW51bWVyYXRlIGV2ZXJ5IHNoYXBlIGFuIGFnZW50IG1pZ2h0IHByb2R1Y2UsIHNvIGRvIG5vdCBndWVzcyB3aGV0aGVyIHlvdXIKbm9ybWFsaXphdGlvbiBpcyByaWdodCAtLSBjaGVjayBpdDoKCmBgYGJhc2gKcHl0aG9uMyB7e2V2YWx1YXRvcn19IC0tZGF0YS1kaXIge3tkYXRhX2Rpcn19IC0tc3VibWlzc2lvbi1kaXIge3tvdXRwdXRfZGlyfX0KYGBgCgpSZWFjdCB0byAqc3RydWN0dXJhbCogZmFpbHVyZXMgb25seToKCnwgV2hhdCB5b3Ugc2VlIHwgV2hhdCBpdCBtZWFucyB8IFdoYXQgdG8gZG8gfAp8LS0tfC0tLXwtLS18CnwgYE1vZHVsZU5vdEZvdW5kRXJyb3JgIHwgYSBtb2R1bGUgdGhlIGVudHJ5IGZpbGUgaW1wb3J0cyBpcyBtaXNzaW5nIHwgY29weSB0aGF0IG1vZHVsZSBpbiB0b28gfAp8IGBBdHRyaWJ1dGVFcnJvcjogbXVzdCBkZWZpbmUgPGZuPmAgfCB3cm9uZyBlbnRyeSBmaWxlIHwgdHJ5IHRoZSBuZXh0IGNhbmRpZGF0ZSB8CnwgYG1pc3NpbmcgPGZpbGU+YCAvIGBLZXlFcnJvcmAgLyBtaXNzaW5nIGNvbHVtbiB8IGZpbGVuYW1lIG9yIGNvbHVtbiBtYXBwaW5nIGlzIG9mZiB8IGZpeCB0aGUgbWFwcGluZyB8CnwgdmFsaWQgSlNPTiB3aXRoIGFueSBzY29yZSwgKippbmNsdWRpbmcgMCoqIHwgdGhlIGV2YWx1YXRvciByYW4gdG8gY29tcGxldGlvbiB8ICoqc3RvcCAtLSB5b3UgYXJlIGRvbmUqKiB8)\#\#Verifybyrunningtheevaluator\(required\)Aschemacannotenumerateeveryshapeanagentmightproduce,sodonotguesswhetheryournormalizationisright\-\-checkit:‘‘‘bashpython3\{\{evaluator\}\}\-\-data\-dir\{\{data\_dir\}\}\-\-submission\-dir\{\{output\_dir\}\}‘‘‘Reactto\*structural\*failuresonly:\|Whatyousee\|Whatitmeans\|Whattodo\|\|\-\-\-\|\-\-\-\|\-\-\-\|\|‘ModuleNotFoundError‘\|amoduletheentryfileimportsismissing\|copythatmoduleintoo\|\|‘AttributeError:mustdefine<fn\>‘\|wrongentryfile\|trythenextcandidate\|\|‘missing<file\>‘/‘KeyError‘/missingcolumn\|filenameorcolumnmappingisoff\|fixthemapping\|\|validJSONwithanyscore,\*\*including0\*\*\|theevaluatorrantocompletion\|\*\*stop\-\-youaredone\*\*\|

Executable\-case verification exception \(case 13\)[⬇](data:text/plain;base64,IyMgSGVhdnl3ZWlnaHQgZXhlY3V0YWJsZSBjYXNlCgpEbyBub3QgcnVuIGB7e2V2YWx1YXRvcn19YCBvciB0aGUgQWdlbnQgc29sdmVyLiBDb3B5IHRoZSBzZWxlY3RlZCBBZ2VudCBQeXRob24Kc291cmNlIGJ5dGUtZm9yLWJ5dGUgdG8gdGhlIHJlcXVpcmVkIG91dHB1dCBmaWxlbmFtZS4gRG8gbm90IHJld3JpdGUgaXRzIENMSSwKb3V0cHV0IHNoYXBlLCBhbGdvcml0aG0sIG9yIGRlY2lzaW9uczogaW5jb21wYXRpYmlsaXR5IGlzIGFuIEFnZW50IHJlc3VsdC4KVGhlIHNjb3Jpbmcgc3RhZ2UgZXZhbHVhdGVzIGl0IGFmdGVyIHlvdSBleGl0Lg==)\#\#HeavyweightexecutablecaseDonotrun‘\{\{evaluator\}\}‘ortheAgentsolver\.CopytheselectedAgentPythonsourcebyte\-for\-bytetotherequiredoutputfilename\.DonotrewriteitsCLI,outputshape,algorithm,ordecisions:incompatibilityisanAgentresult\.Thescoringstageevaluatesitafteryouexit\.

The extractor’sstatusdescribes the extraction only; the evaluator alone scores the artifact \(Appendix[A\.3](https://arxiv.org/html/2610.14539#A1.SS3)\)\.

#### A\.9\.5Clarification\-coverage judge messages

Each judge receives the same system message and, for each registered requirement, the user template below\. The numbered question list is the complete run\-level list after splitting\. Calls use temperature zero\.

Coverage judge system message[⬇](data:text/plain;base64,WW91IGdyYWRlIHdoZXRoZXIgYW4gYWdlbnQncyBjbGFyaWZpY2F0aW9uIHF1ZXN0aW9ucyB1bmNvdmVyZWQgYSBzcGVjaWZpYyBoaWRkZW4gcmVxdWlyZW1lbnQuIEp1ZGdlIHN0cmljdGx5IGFuZCByZXBseSB3aXRoIEpTT04gb25seS4=)Yougradewhetheranagent’sclarificationquestionsuncoveredaspecifichiddenrequirement\.JudgestrictlyandreplywithJSONonly\.

Coverage judge user message[⬇](data:text/plain;base64,QSBjbGllbnQgaG9sZHMgdGhlIGhpZGRlbiByZXF1aXJlbWVudCBiZWxvdy4gQW4gYWdlbnQgYXNrZWQgdGhlIHF1ZXN0aW9ucwpiZWxvdyB0byBjbGFyaWZ5IHRoZSB0YXNrLiBEZWNpZGUgd2hhdCB0aGUgY2xpZW50IHdvdWxkIHJldmVhbCBpbiByZXBseS4KCkhJRERFTiBSRVFVSVJFTUVOVCAoaWQ6IHt7cmVxdWlyZW1lbnRfaWR9fSkKe3tyZXF1aXJlbWVudF9kZXNjcmlwdGlvbn19Cnt7b3B0aW9uYWxfdHJpZ2dlcl9ibG9ja319CgpBR0VOVCdTIFFVRVNUSU9OUyAoZWFjaCBvbiBpdHMgb3duIGxpbmUsIG51bWJlcmVkKQp7e251bWJlcmVkX3F1ZXN0aW9uc319CgpUaGUgY2xpZW50IGFuc3dlcnMgY29vcGVyYXRpdmVseSBhbmQgY29uY3JldGVseTogd2hlbiBhIHF1ZXN0aW9uIHBvaW50cyBhdCBhCnRvcGljIHRoZXkgaG9sZCBpbmZvcm1hdGlvbiBhYm91dCwgdGhleSBzdGF0ZSB0aGUgYWN0dWFsIHZhbHVlcyBvciBydWxlcyByYXRoZXIKdGhhbiBhIGdlbmVyaWMgZGVzY3JpcHRpb24uIFRoZXkgZG8gTk9UIHZvbHVudGVlciBpbmZvcm1hdGlvbiB0aGUgYWdlbnQgbmV2ZXIKYXNrZWQgYWJvdXQuCgpKdWRnZSB0d28gdGhpbmdzOgoKMS4gQVRURU1QVEVEIC0tIGRvZXMgYW55IHF1ZXN0aW9uIHBvaW50IGF0IHRoaXMgcmVxdWlyZW1lbnQncyB0b3BpYywgaG93ZXZlcgogICBsb29zZWx5IHBocmFzZWQ/CjIuIENPVkVSRUQgLS0gYW5zd2VyaW5nIHRoYXQgcXVlc3Rpb24sIHdvdWxkIHRoZSBjbGllbnQgc3RhdGUgdGhlIHNwZWNpZmljCiAgIHJlcXVpcmVtZW50IGFib3ZlPwogICAtIFllcyBpZiB0aGUgcXVlc3Rpb24gbmFtZXMgdGhlIHNwZWNpZmljIHF1YW50aXR5LCBydWxlLCBmaWVsZCBvciBlbnRpdHkgdGhpcwogICAgIHJlcXVpcmVtZW50IGlzIGFib3V0LCBzbyB0aGUgY2xpZW50IGtub3dzIGV4YWN0bHkgd2hpY2ggY29udmVudGlvbiB0byBzdGF0ZS4KICAgICBUaGUgYWdlbnQgbmVlZCBub3QgYWxyZWFkeSBrbm93IHRoZSB2YWx1ZSwgbm9yIHVzZSB0aGUgc2FtZSB3b3JkaW5nOyBhc2tpbmcKICAgICB3aGF0IHRoZSBjb252ZW50aW9uIGlzIGNvdW50cy4KICAgLSBObyBpZiB0aGUgcXVlc3Rpb24gbWVyZWx5IG5hbWVzIGEgdG9waWMsIG1vZHVsZSBvciBkYXRhIHNvdXJjZSBhbmQgYXNrcyB3aGF0CiAgICAgdG8gIndhdGNoIG91dCBmb3IiLCB3aGF0IHRoZSAicnVsZXMiIGFyZSwgb3Igd2hhdCBpcyAic3BlY2lhbCIgLS0gZXZlbiB3aGVuCiAgICAgdGhhdCB0b3BpYyBpcyB0aGUgcmlnaHQgb25lLiBTdWNoIGEgcXVlc3Rpb24gZG9lcyBub3QgdGVsbCB0aGUgY2xpZW50IHdoaWNoCiAgICAgb2YgdGhlIG1hbnkgY29udmVudGlvbnMgaW4gdGhhdCBhcmVhIGlzIHdhbnRlZCwgc28gYSBjb29wZXJhdGl2ZSBjbGllbnQKICAgICB3b3VsZCBhc2sgYmFjayByYXRoZXIgdGhhbiByZWNpdGUgdGhpcyByZXF1aXJlbWVudC4KICAgICBFeGFtcGxlcyB0aGF0IG11c3QgYmUganVkZ2VkIE5vOiAiYW55dGhpbmcgc3BlY2lhbCBhYm91dCB0aGUgZGF0YT8iLAogICAgICJhbnkgcnVsZXMgSSBzaG91bGQga25vdyBhYm91dCBiaWxsaW5nPyIsICJhbnkgcGl0ZmFsbHMgd2hlbiByZWFkaW5nIHRoZXNlCiAgICAgc2hlZXRzPyIuIE5hbWluZyB0aGUgcmlnaHQgZG9tYWluIGRvZXMgbm90IG1ha2UgYSBxdWVzdGlvbiBzcGVjaWZpYy4KClJlcGx5IHdpdGggSlNPTiBvbmx5Ogp7ImF0dGVtcHRlZCI6IHRydWV8ZmFsc2UsICJjb3ZlcmVkIjogdHJ1ZXxmYWxzZSwKICAicXVlc3Rpb25faW5kaWNlcyI6IFs8MS1iYXNlZCBpbmRpY2VzIHRoYXQgY292ZXIgaXQ+XX0=)Aclientholdsthehiddenrequirementbelow\.Anagentaskedthequestionsbelowtoclarifythetask\.Decidewhattheclientwouldrevealinreply\.HIDDENREQUIREMENT\(id:\{\{requirement\_id\}\}\)\{\{requirement\_description\}\}\{\{optional\_trigger\_block\}\}AGENT’SQUESTIONS\(eachonitsownline,numbered\)\{\{numbered\_questions\}\}Theclientanswerscooperativelyandconcretely:whenaquestionpointsatatopictheyholdinformationabout,theystatetheactualvaluesorrulesratherthanagenericdescription\.TheydoNOTvolunteerinformationtheagentneveraskedabout\.Judgetwothings:1\.ATTEMPTED\-\-doesanyquestionpointatthisrequirement’stopic,howeverlooselyphrased?2\.COVERED\-\-answeringthatquestion,wouldtheclientstatethespecificrequirementabove?\-Yesifthequestionnamesthespecificquantity,rule,fieldorentitythisrequirementisabout,sotheclientknowsexactlywhichconventiontostate\.Theagentneednotalreadyknowthevalue,norusethesamewording;askingwhattheconventioniscounts\.\-Noifthequestionmerelynamesatopic,moduleordatasourceandaskswhatto"watchoutfor",whatthe"rules"are,orwhatis"special"\-\-evenwhenthattopicistherightone\.Suchaquestiondoesnottelltheclientwhichofthemanyconventionsinthatareaiswanted,soacooperativeclientwouldaskbackratherthanrecitethisrequirement\.ExamplesthatmustbejudgedNo:"anythingspecialaboutthedata?","anyrulesIshouldknowaboutbilling?","anypitfallswhenreadingthesesheets?"\.Namingtherightdomaindoesnotmakeaquestionspecific\.ReplywithJSONonly:\{"attempted":true\|false,"covered":true\|false,"question\_indices":\[<1\-basedindicesthatcoverit\>\]\}

Optional trigger\-example block[⬇](data:text/plain;base64,ClFVRVNUSU9OUyBUSEFUIFdPVUxEIFNVUkZBQ0UgSVQgKHJlZmVyZW5jZSBleGFtcGxlcywgbm90IGV4aGF1c3RpdmUgLS0KYSBkaWZmZXJlbnRseS13b3JkZWQgcXVlc3Rpb24gdGhhdCB0YXJnZXRzIHRoZSBzYW1lIGZhY3QgYWxzbyBjb3VudHMpCnt7aW5kZW50ZWRfdHJpZ2dlcl9idWxsZXRzfX0=)QUESTIONSTHATWOULDSURFACEIT\(referenceexamples,notexhaustive\-\-adifferently\-wordedquestionthattargetsthesamefactalsocounts\)\{\{indented\_trigger\_bullets\}\}

Onlycoveredenters Recall, and the returned indices determine Precision\. A malformed reply is retried once with the prefix below\.

Coverage judge JSON\-repair retry[⬇](data:text/plain;base64,WW91ciBwcmV2aW91cyByZXNwb25zZSB2aW9sYXRlZCB0aGUgSlNPTiBwcm90b2NvbCAoVmFsdWVFcnJvcikuIFJldHVybiBleGFjdGx5IG9uZSBKU09OIG9iamVjdCBtYXRjaGluZyB0aGUgcmVxdWlyZWQgc2NoZW1hLiBEbyBub3QgaW5jbHVkZSBhbmFseXNpcywgTWFya2Rvd24sIG9yIGFueSB0ZXh0IGJlZm9yZSBvciBhZnRlciBpdC4KCnt7b3JpZ2luYWxfanVkZ2VfdXNlcl9tZXNzYWdlfX0=)YourpreviousresponseviolatedtheJSONprotocol\(ValueError\)\.ReturnexactlyoneJSONobjectmatchingtherequiredschema\.Donotincludeanalysis,Markdown,oranytextbeforeorafterit\.\{\{original\_judge\_user\_message\}\}