LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation

# LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation
Source: [https://arxiv.org/html/2607.27353](https://arxiv.org/html/2607.27353)
###### Abstract

Agentic retrieval\-augmented generation systems can produce answers that appear grounded while failing at the evidence, tool\-contract, authorization, or session\-state layer\. We introduce LayerRAG\-Bench, a controlled cross\-layer reliability benchmark with 8 enterprise domains, 240 tasks, 9 fault scenarios, 2 contract modes, and 38,880 live task\-level records across nine models from OpenAI, Anthropic, and Gemini\. Schema normalization raises schema\-drift success from 0\.000 to 0\.913, but stale evidence, missing tool output, denied permissions, and wrong\-session context are not recovered by schema normalization\. Groundedness\-only evaluation also produces substantial false positives under stale and wrong\-session evidence\. These results support a layer\-specific evaluation principle: a reliability intervention should be credited for repairing its target layer without being mistaken for a universal fix\.

††footnotetext:Large language model tools were used to assist with editing, code review, and manuscript preparation\. The author is responsible for all ideas, claims, experiments, analyses, and final text\.## 1Introduction

Retrieval\-augmented generation conditions model outputs on external evidence, while agentic RAG adds tool calls, structured interfaces, permissions, and persistent state\(Lewiset al\.,[2020](https://arxiv.org/html/2607.27353#bib.bib1); Gaoet al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib4); Yaoet al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib6)\)\. Reliability therefore depends on more than whether an answer is textually supported\. A response can faithfully summarize a superseded policy, cite evidence from the wrong session, or proceed from incomplete tool output and still appear grounded\.

Existing retrieval and faithfulness metrics are necessary but can conflate failures from different layers\. This matters operationally because schema normalization may repair a changed tool payload, whereas it cannot restore missing evidence, override authorization, refresh a stale index, or correct tenant context\. Evaluation should distinguish those cases rather than collapse them into one answer\-quality score\.

LayerRAG\-Bench introduces controlled faults across retrieval, freshness, metadata, tool contracts, permissions, completeness, and session state\. It combines deterministic regression tests with a live matrix of nine models, nine scenarios, two contract modes, and 240 task rows per cell\. The study makes three contributions:

1. 1\.a reproducible enterprise\-style benchmark that maps nine scenarios to distinct operational layers;
2. 2\.matched strict and repair conditions that test whether schema repair improves its intended layer without masking negative controls; and
3. 3\.privacy\-safe archives and hierarchical uncertainty analysis for recomputing all reported aggregate results\.

The main finding is structural rather than leaderboard\-oriented\. Contract repair consistently recovers schema drift, while stale, unavailable, unauthorized, incomplete, and wrong\-session evidence requires separate controls\. LayerRAG\-Bench is therefore complementary to retrieval, groundedness, and general factuality benchmarks: it evaluates whether a mitigation fixes the layer it claims to address\.

## 2Benchmark and Methods

### 2\.1Benchmark design and threat model

Theenterprise\_v2corpus contains 8 enterprise domains, 80 policy specifications, 160 documents, and 240 tasks\. Each policy unit has a current and superseded document plus three templated question variants\. We treat these variants as correlated coverage cases rather than 240 independent content units\. Documents expose policy prose and operational metadata; task\-level expected answers and required document identifiers remain evaluator\-only\.

LayerRAG\-Bench targets non\-adversarial operational faults inside a RAG stack: stale indexing, schema drift, unavailable or partial tool output, conflicting evidence, permission filtering, corrupted metadata, and wrong session state\. It does not claim to evaluate prompt injection, jailbreaks, data exfiltration, or production access\-control enforcement\. A failure should not always be “repaired” by the model: denied evidence should trigger a safe non\-answer, and stale or wrong\-session evidence should not receive credit merely because it supports the generated text\.

Table 1:Fault scenarios, targeted reliability layers, and expected mitigation behavior\.Table[1](https://arxiv.org/html/2607.27353#S2.T1)defines the intended failure layer and mitigation boundary for each scenario\. This mapping is the basis for the paper’s layer\-specific interpretation\.

### 2\.2Evaluation protocol

The deterministic harness evaluates all 240 tasks without model calls using the reportedtfidf\_domain\_validityretriever\. It combines TF–IDF matching with query\-derived domain cues and status/year validity reranking; it does not receive the task’s labeled domain\.

The live harness applies the same task definitions, retriever, scenarios, contract modes, and scorer to nine models spanning OpenAI, Anthropic, and Gemini\. Crossing 9 models, 9 scenarios, 2 contract modes, and 240 task rows produces 162 cells and 38,880 records\. Strict mode rejects tool outputs that do not match the expected schema\. Repair mode applies bounded schema normalization; it is not designed to reconstruct missing evidence, bypass permissions, refresh an index, or repair session context\.

A separate focused LangChain study compares uncontrolled retrieval and generation with evidence\-aware allow, abstain, reject, and bounded\-retry controls\. It uses one model, eight tasks, nine evidence conditions, and two arms, yielding 144 evaluated cells\. Strict success treats abstention as failure; safe resolution credits either a correct answer on answerable conditions or a structured non\-answer on unsafe conditions\.

### 2\.3Metrics

For tasktt, strict success is

St=Et∧Rt∧Gt∧Vt,S\_\{t\}=E\_\{t\}\\land R\_\{t\}\\land G\_\{t\}\\land V\_\{t\},whereEtE\_\{t\}is normalized exact match,RtR\_\{t\}requires all task\-specific evidence identifiers to be retrieved,GtG\_\{t\}requires at least one citation and restricts citations to retrieved evidence, andVtV\_\{t\}requires every tool call to satisfy its schema\. This conjunction is intentionally stricter than answer correctness or groundedness alone\. Abstentions remain strict failures but are retained in diagnostic labels and, in the focused integration study, the separate safe\-resolution metric\.

The historical live runner serialized a version\-local answer field from document metadata alongside answer\-bearing prose\. It did not expose task\-level expected answers or required document identifiers\. Current code removes that field\. We therefore interpret historical live results as end\-to\-end pipeline measurements, not as an isolated test of metadata\-free reading comprehension\.

### 2\.4Artifacts

Privacy\-safe scored\-record archives contain all 38,880 cross\-provider records and 144 focused\-integration cells needed to recompute scenario aggregates, repair gains, groundedness false positives, and bootstrap intervals without provider calls\. They omit raw model prose, free\-form rationale, answer text, document identifiers, latency, and cost\. Full protocol settings and additional audits appear in Appendix[B](https://arxiv.org/html/2607.27353#A2)\.

## 3Related Work

Retrieval benchmarks such as DPR and BEIR evaluate relevance and ranking quality\(Karpukhinet al\.,[2020](https://arxiv.org/html/2607.27353#bib.bib2); Thakuret al\.,[2021](https://arxiv.org/html/2607.27353#bib.bib3)\)\. RAG evaluation extends this toward answer correctness, citation quality, and faithfulness; RAGAS, ARES, CRAG, and Self\-RAG automate or adapt several of these dimensions\(Lewiset al\.,[2020](https://arxiv.org/html/2607.27353#bib.bib1); Gaoet al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib4); Eset al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib9); Saad\-Falconet al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib10); Yanget al\.,[2024](https://arxiv.org/html/2607.27353#bib.bib11); Asaiet al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib12)\)\. These properties remain necessary, but evidence can be faithful and still be superseded, unauthorized, incomplete, or associated with the wrong session\.

Tool\-use benchmarks study action selection, schema compliance, and structured outputs\(Yaoet al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib6); Schicket al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib7); Qinet al\.,[2023](https://arxiv.org/html/2607.27353#bib.bib8)\)\. Recent diagnostic work localizes intermediate RAG and agent failures, while reliability and enterprise benchmarks introduce production\-like faults or multidimensional evaluation\(Linet al\.,[2025](https://arxiv.org/html/2607.27353#bib.bib14); Youet al\.,[2026](https://arxiv.org/html/2607.27353#bib.bib15); Jiaoet al\.,[2026](https://arxiv.org/html/2607.27353#bib.bib16); Gupta,[2026](https://arxiv.org/html/2607.27353#bib.bib13); Sunet al\.,[2026](https://arxiv.org/html/2607.27353#bib.bib18); Naritaet al\.,[2026](https://arxiv.org/html/2607.27353#bib.bib17)\)\.

LayerRAG\-Bench is complementary to these lines of work\. Its distinct object is a matched cross\-layer fault matrix with negative controls: a mitigation is credited when it repairs its intended operational layer without appearing to solve unrelated freshness, authorization, completeness, or state failures\. This is consistent with broader systematizations of risks across retrieval, memory, and tool\-execution loops\(Mishraet al\.,[2026](https://arxiv.org/html/2607.27353#bib.bib19)\)\.

## 4Results

### 4\.1Deterministic benchmark behavior

Table[2](https://arxiv.org/html/2607.27353#S4.T2)reports the full deterministic evaluation over 240 tasks and nine scenarios\. It verifies that the benchmark separates answerable conditions, contract failures, and evidence or state failures before introducing model variability\.

Table 2:Deterministic enterprise v2 scenario results\.Clean and conflicting\-evidence conditions are largely answerable, whereas stale index, schema drift, missing output, permission denial, and wrong session state fail for distinct reasons\. In particular, stale and wrong\-session answers can remain grounded in retrieved documents while being operationally incorrect\.

### 4\.2Retriever and context sensitivity

A deterministic post\-hoc audit evaluates the exact retriever interfaces on all 240 tasks without model calls\. Table[3](https://arxiv.org/html/2607.27353#S4.T3)reports required\-document top\-1 recall under clean, conflicting\-evidence, and corrupted\-metadata conditions\.

Table 3:Required\-document top\-1 recall under deterministic retriever ablations\. Each cell evaluates all 240 task rows\. The explicit task\-domain filter is a sensitivity condition, not the reported retriever configuration\.Validity awareness is the main source of current\-version preference\. Removing validity reranking lowers clean top\-1 recall from 0\.896 to 0\.483\. Under corrupted metadata, the reported retriever’s required\-document top\-1 recall is 0\.554 and its current\-version preference rate is 0\.562\. The query\-derived domain heuristic is correct for 88 tasks, incorrect for 14, and unresolved for 138\.

One exact standalone query appears in both returns and legal tasks with different answers\. The reported retriever ranks the required document first for only one of the two; explicit task\-domain filtering resolves both\. However, explicit filtering is itself brittle to corrupted domain metadata, where top\-1 and top\-3 recall both fall to zero\. This establishes a central boundary of the benchmark: query text alone may not identify the correct domain or session, and contextual metadata is useful only when its integrity is trustworthy\.

### 4\.3Live cross\-provider matrix

The primary live result is the scenario–contract pattern in Table[4](https://arxiv.org/html/2607.27353#S4.T4)\. Each row aggregates 2,160 records: nine models times 240 task rows\.

Table 4:Cross\-provider success rates by scenario and contract mode\.All 38,880 records have resolved binary success labels\. Schema drift is the dominant repairable fault: strict success is 0\.000 and repair success is 0\.913\. Stale index, missing tool output, permission denial, and wrong session state remain at 0\.000 under both modes\. Partial timeout remains degraded under both modes \(0\.314 strict; 0\.308 repair\)\.

The paired schema\-drift improvement is 91\.3 percentage points with a hierarchical 95% interval of 85\.4–95\.6 points\. Its largest absolute off\-target contract change is 0\.7 points, giving a specificity margin of 90\.6 points\.

Table 5:Groundedness false positives and paired strict\-to\-repair effects\. GF denotes records grounded in retrieved documents that nevertheless fail strict benchmark success\. Differences and intervals are percentage points\.Table[5](https://arxiv.org/html/2607.27353#S4.T5)shows why groundedness alone is insufficient\. Under wrong session state, 77\.6% of strict records and 78\.8% of repair records are grounded yet unsuccessful; every grounded stale\-index record also fails strict success\. Schema repair, by contrast, improves paired task success by 91\.3 points because the required evidence remains present and only the contract shape is broken\.

### 4\.4Representative failure mechanisms

Table[6](https://arxiv.org/html/2607.27353#S4.T6)contrasts one repairable contract fault with evidence, authorization, and context failures\.

Table 6:Representative task\-level outcomes\. “Unknown” denotes no usable answer under the benchmark contract\.The paired schema\-drift rows isolate the intended mechanism: repair restores a changed representation of available evidence\. The remaining cases require freshness checks, access\-aware abstention, timeout recovery, or session isolation rather than schema normalization\.

### 4\.5Focused LangChain integration

Table 7:Focused LangChain integration\. Safe resolution credits correct answers on answerable conditions and structured non\-answers on unsafe ones\.Within the LayerRAG arm, strict success is 24/24 on answerable conditions and safe resolution is 48/48 on unsafe conditions\. Clean\-task strict success remains 100% in both arms, with no incorrect LayerRAG control interventions\. The study is intentionally narrow and supports safe handling, not a general strict\-accuracy claim\.

### 4\.6Interpretation

The evidence supports three conclusions\. First, contract repair should be evaluated against contract faults: it recovers schema drift but does not restore missing evidence or state\. Second, groundedness alone is insufficient when freshness, authorization, completeness, and context matter\. Third, similar failure shapes across providers indicate that stronger base models do not eliminate the need for layer\-specific system controls\. Provider\-level and repair\-gain breakdowns appear in Appendix[A](https://arxiv.org/html/2607.27353#A1)\.

## 5Interpreting Fixed\-Budget Results

The live matrix is a fixed\-budget systems evaluation, not a powered commercial model ranking\. Each model contributes 4,320 evaluations, each provider 12,960, and each scenario–contract row 2,160\. The strongest claims concern repeated failure structure rather than small provider differences\.

Strict and repair outcomes are paired by provider, model, scenario, and task\. The repair\-minus\-strict interval uses a hierarchical bootstrap that resamples provider–model units and then task pairs within each selected unit, preserving the nested design\. Schema drift improves by 0\.913 with a 95% interval of \[0\.854, 0\.956\]\. Clean, conflicting\-evidence, corrupted\-metadata, and partial\-timeout intervals include zero; missing output, permission denial, stale index, and wrong session effects are exactly zero under both modes\.

These results support a mechanism\-level interpretation\. Schema drift is repairable because the needed evidence remains present and the failure is localized to the tool contract\. The negative controls remain unresolved because their missing conditions are not schema\-format problems\.

## 6Reproducibility

The public repository includes benchmark construction, deterministic evaluation, table export, tests, and aggregate\-recomputation code\. Fixed specifications generate the corpus, tasks, and scenarios\. Live execution uses explicit call budgets, while deterministic workflows and public\-result recomputation require no provider access\.

The committed privacy\-safe archives reproduce scenario and provider aggregates, repair gains, groundedness false\-positive counts, hierarchical bootstrap intervals, and focused LangChain results\. Prompt hashes are present for 28,080 of 38,880 records; the remaining records terminate before prompt construction because retrieval is unavailable or invalid\. The released archives exclude credentials, raw model prose, unrestricted traces, and non\-allowlisted identifying metadata\.

Appendix[B](https://arxiv.org/html/2607.27353#A2)reports provider settings, archive boundaries, and historical\-run caveats\. The manuscript source and public artifacts are versioned together in the repository linked on the first page\.

## 7Limitations and Future Work

The corpus is synthetic and policy\-like\. This enables controlled faults but does not reproduce the ambiguity, document diversity, user behavior, or organizational complexity of production deployments\. The 240 task rows are three variants of 80 policy units, so linguistic variants are correlated rather than independent samples\.

The reported retriever infers domain from query wording; it was unresolved on 138 tasks and incorrect on 14 in the deterministic audit\. Historical live prompts also included a version\-local answer field in document metadata, although answer\-bearing prose contained the same value and task\-level labels were not exposed\. Current code removes the field\. The historical cross\-provider and focused\-integration results should therefore be interpreted as measurements of the pre\-hardening end\-to\-end pipeline, not metadata\-free reading comprehension\.

The cross\-provider matrix is broad in task count but limited to nine evaluated models and a fixed budget\. The focused LangChain study is narrower still: one model, eight selected tasks, and one completion per cell\. Neither study establishes provider rankings, production latency advantages, or general security guarantees\.

Future extensions should evaluate freshness\-aware retrieval, session isolation, timeout recovery, policy\-based abstention, human escalation, multi\-turn and multi\-hop tasks, hierarchical permissions, adversarial document conflicts, open\-source models, and repeated temporal replications\. The current contribution is the layer\-specific evaluation framework, not exhaustive coverage of every RAG reliability or security failure\.

## 8Conclusion

LayerRAG\-Bench treats agentic RAG reliability as a layered systems problem\. In 38,880 cross\-provider records, schema repair raises schema\-drift success from 0\.000 to 0\.913 but does not repair stale evidence, missing output, denied permissions, partial timeouts, or wrong\-session context\. Groundedness can therefore be necessary while remaining operationally insufficient\.

The broader principle is that mitigations should be evaluated against the layer they are designed to repair\. Retrieval improvements, schema normalization, freshness validation, access\-aware abstention, timeout recovery, and session isolation address different mechanisms\. LayerRAG\-Bench makes those boundaries measurable and provides reproducible negative controls against overstating a narrow intervention as a universal reliability solution\.

## References

- Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.arXiv preprint arXiv:2310\.11511\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- S\. Es, J\. James, L\. Espinosa\-Anke, and S\. Schockaert \(2023\)RAGAS: automated evaluation of retrieval augmented generation\.arXiv preprint arXiv:2309\.15217\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, and H\. Wang \(2023\)Retrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.10997\.Cited by:[§1](https://arxiv.org/html/2607.27353#S1.p1.1),[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- A\. Gupta \(2026\)ReliabilityBench: evaluating LLM agent reliability under production\-like stress conditions\.arXiv preprint arXiv:2601\.06112\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- S\. Jiao, C\. Huang, S\. Qi, X\. Wang, Y\. Li, Q\. Weng, L\. Liu, X\. Cai, and L\. Yao \(2026\)Doctor\-RAG: a failure\-aware repair framework for agentic retrieval\-augmented generation\.arXiv preprint arXiv:2604\.00865\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.27353#S1.p1.1),[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- J\. Lin, C\. Zhang, S\. Y\. Liu, and H\. Li \(2025\)RAGCap\-Bench: benchmarking capabilities of LLMs in agentic retrieval augmented generation systems\.arXiv preprint arXiv:2510\.13910\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- S\. Mishra, S\. Niroula, U\. Yadav, D\. Thakur, S\. Gyawali, and S\. Gaire \(2026\)SoK: agentic retrieval\-augmented generation \(RAG\): taxonomy, architectures, evaluation, and research directions\.arXiv preprint arXiv:2603\.07379\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p3.1)\.
- K\. Narita, S\. Peng, T\. Fukui, M\. Yamada, S\. Munakata, and S\. Takahashi \(2026\)Overcoming the “impracticality” of RAG: proposing a real\-world benchmark and multi\-dimensional diagnostic framework\.arXiv preprint arXiv:2604\.02640\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- Y\. Qin, S\. Cai, X\. Wang, Y\. Chen, Z\. Liu, Y\. Liang, Z\. Liu, X\. Han, X\. Han, Z\. Liu, M\. Sun, and J\. Zhou \(2023\)ToolBench: towards tool\-augmented large language models\.arXiv preprint arXiv:2307\.16789\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. Zaharia \(2023\)ARES: an automated evaluation framework for retrieval\-augmented generation systems\.arXiv preprint arXiv:2311\.09476\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- Y\. Sun, J\. Rahmfeld, C\. Weaver, W\. Chen, R\. Desai, W\. Huang, and M\. H\. Butler \(2026\)EnterpriseRAG\-Bench: a RAG benchmark for company internal knowledge\.arXiv preprint arXiv:2605\.05253\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. Gurevych \(2021\)BEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- X\. Yang, K\. Sun, H\. Xin, Y\. Sun, N\. Bhalla, X\. Chen, S\. Choudhary, R\. D\. Gui, Z\. W\. Jiang, Z\. Jiang, L\. Kong, B\. Moran, J\. Wang, Y\. E\. Xu, A\. Yan, C\. Yang, E\. Yuan, H\. Zha, N\. Tang, L\. Chen, N\. Scheffer, Y\. Liu, N\. Shah, R\. Wanga, A\. Kumar, W\. Yih, and X\. L\. Dong \(2024\)CRAG – comprehensive RAG benchmark\.arXiv preprint arXiv:2406\.04744\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.27353#S1.p1.1),[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.
- Q\. You, W\. Yu, and W\. Zhang \(2026\)AgenticRAGTracer: a hop\-aware benchmark for diagnosing multi\-step retrieval reasoning in agentic RAG\.arXiv preprint arXiv:2602\.19127\.Cited by:[§3](https://arxiv.org/html/2607.27353#S3.p2.1)\.

## Appendix AAdditional Live\-Matrix Results

Table 8:Largest strict\-to\-repair gains\.Table 9:Provider\-level averages, reported as supporting context rather than commercial\-model rankings\.Provider averages combine clean tasks, repairable contract failures, and negative controls that should remain unresolved\. They are therefore not ordinary capability scores\. The central repeated pattern is the scenario\-level specificity of schema repair\.

## Appendix BProtocol and Artifact Details

The eight benchmark domains are returns, paid time off, support, security, finance, human resources, legal, and IT operations\. The reported retriever usestop\_k=3\. Provider clients use a 180\-second request timeout, a 600\-token output cap, three total attempts, and a 2\-second linear retry backoff\. Request temperature was not explicitly supplied, so endpoints used their API defaults; Anthropic requests use API version2023\-06\-01\.

The reported matrix contains 54 model–scenario–contract cells for each of OpenAI, Anthropic, and Gemini \(162 total\)\. Run manifests prevent double counting of mirrored outputs\. The privacy\-safe archives contain fields needed for aggregate recomputation but omit raw model output, answer text, free\-form rationale, document identifiers, latency, cost, and non\-allowlisted metadata\.

The benchmark does not treat permission denial, stale evidence, or wrong session state as faults that a model should answer through\. Those scenarios test whether system boundaries remain visible\. Prompt injection, malicious documents, exfiltration, and production authorization certification are outside the present threat model\.