EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval

# EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval
Source: [https://arxiv.org/html/2608.11584](https://arxiv.org/html/2608.11584)
Xinbao SunBo WangFanyu MengLijun MeiNa WuDi JinChao DengJunlan FengAffiliation:Jiutian Research, China Mobile, Beijing, ChinaEmail:[mailto:](mailto:)[\{miaohuiqi,sunxinbao,wangbo\}@cmjt\.chinamobile\.com](mailto:miaohuiqi,sunxinbao,[email protected])

###### Abstract

Enterprise RAG deployments face a critical reliability gap: while LLMs satisfy individual constraints at rates up to 84%, only 27% of responses meet all requirements simultaneously, revealing a 57\-point orchestration gap\. Existing benchmarks assume clean retrieval with simple queries, failing to capture production conditions where noisy documents and multi\-dimensional constraints coexist\. We introduce EnterpriseRAG, a benchmark of 983 expert\-validated samples across six domains that systematically simulates three failure modes absent from prior work: retrieval noise, knowledge gaps, and factual conflicts, coupled with complex instructions\. Evaluation of 13 state\-of\-the\-art LLMs reveals a severe instruction adherence collapse, where high per\-constraint satisfaction masks low holistic compliance\. Critical findings expose deep barriers under knowledge gaps and factual conflicts, even with reasoning\-enhanced inference, indicating production RAG requires explicit context\-aware protocols and calibrated judgment\. EnterpriseRAG provides a reproducible foundation for measuring and closing these gaps, directly informing deployment decisions for enterprise\-scale RAG systems\. We will release the benchmark and evaluation framework upon publication\.

Keywords:RAG benchmark, instruction following, LLM robustness, enterprise retrieval, knowledge gaps, factual conflicts, retrieval noise

## 1Introduction

Enterprise RAG systems face complex queries like"Summarize Q3 revenue by region in markdown tables\. If data is incomplete, state ’Data Unavailable’ rather than estimating\. If audit and management reports conflict, cite both explicitly\."requiring simultaneous factual extraction, formatting compliance, and protocol adherence\.

These challenges stem not from inadequate factual grounding, but from a fundamental evaluation gap\. Current benchmarks assess RAG systems on clean retrieval scenarios with simple queries\([10](https://arxiv.org/html/2608.11584#bib.bib20);[8](https://arxiv.org/html/2608.11584#bib.bib21)\), while production deployments face three compounding challenges absent from existing evaluations:\(1\)complex multi\-constraint instructions integrating formatting rules with context\-aware protocols for evidence adjudication;\(2\)high retrieval noise from latency\-constrained systems that surface 10–20 documents with substantial irrelevant content;\(3\)frequent knowledge failures including coverage gaps and factual conflicts driven by temporal drift or source fallibility\.

While recent work advances robustness testing\([32](https://arxiv.org/html/2608.11584#bib.bib37)\)and instruction following\([7](https://arxiv.org/html/2608.11584#bib.bib17)\), these efforts evaluate constraints in isolation with synthetic noise, missing the compounding complexity of real enterprise workflows where multiple dimensions interact\.

We introduceEnterpriseRAG, a benchmark grounded in real\-world enterprise deployments, comprising 983 expert\-validated samples across six vertical domains\. Unlike prior benchmarks overlaying synthetic instructions onto standard datasets, EnterpriseRAG reflects authentic multi\-domain scenarios derived from real operational queries\. We preserve original user intents while systematically scaling up constraint complexity through an expert\-informed synthesis protocol, and construct three orthogonal non\-ideal retrieval modes \(irrelevant noise, knowledge gaps, and factual conflicts\) validated through LLM\-assisted generation and human verification\.

Table 1:Comparison with prior RAG benchmarks across Source, Complexity, and Robustness\.BenchmarkDataset SourceTaskComplexityRobustnessHuman\-CuratedVerticalDomainNatural UserQueriesComplexConstraintsNegativeRejectionConflictCRUD\-RAG\([18](https://arxiv.org/html/2608.11584#bib.bib13)\)✗✓✗✗✗✗CRAG\([29](https://arxiv.org/html/2608.11584#bib.bib18)\)✓✓✗✗✓✗RAGBench\([9](https://arxiv.org/html/2608.11584#bib.bib14)\)✗✓✓✗✗✗RAGEval\([36](https://arxiv.org/html/2608.11584#bib.bib12)\)✗✓✗✗✗✗FollowRAG\([7](https://arxiv.org/html/2608.11584#bib.bib17)\)✓✗✗✓✗✗EKRAG\([31](https://arxiv.org/html/2608.11584#bib.bib9)\)✓✓✓✗✗✗RARE\([32](https://arxiv.org/html/2608.11584#bib.bib37)\)✗✓✗✗✓✓GaRaGe\([22](https://arxiv.org/html/2608.11584#bib.bib16)\)✓✓✗✗✓✗EnterpriseRAG \(Ours\)✓✓✓✓✓✓

Our evaluation framework extends traditional RAG metrics withStrict IAS\(holistic compliance\) versusLoose IAS\(per\-constraint satisfaction\) to expose compositional adherence failures, plus robustness indicators for safety\-critical scenarios\. Testing 13 state\-of\-the\-art LLMs reveals that while models handle structural formatting adequately, they systematically fail on behavioral protocols, particularly judgment under uncertainty, where even reasoning models achieve insufficient reliability for production deployment\.

#### Contributions\.

Our contributions are threefold:

- •Enterprise\-grade RAG benchmark: We introduceEnterpriseRAG, 983 expert\-validated instances across six domains, pairing complex multi\-constraint instructions with three controlled non\-ideal retrieval settings, plus a reproducible construction pipeline\.
- •Comprehensive evaluation framework:We develop specialized metrics for non\-ideal contexts:Loose/Strict IASfor instruction adherence, andrejection/conflict accuracyto measure safety\-critical judgment in scenarios with missing or conflicting information\.
- •Experimental Insights: Across 13 LLMs, we find an orchestration gap of up to 57pp \(83\.8% Loose vs\. 26\.8% Strict\), with best rejection accuracy only 42\.7% and conflict recognition <45%, indicating behavioral judgment under uncertainty as the main bottleneck for enterprise RAG\.

## 2Related Work

RAG evaluation has evolved from factual correctness toward robustness and instruction compliance\([34](https://arxiv.org/html/2608.11584#bib.bib1);[12](https://arxiv.org/html/2608.11584#bib.bib2)\)\. Table[1](https://arxiv.org/html/2608.11584#S1.T1)positions EnterpriseRAG among representative benchmarks\. A recurring pattern across prior work is that retrieval quality and instruction adherence are studied separately—leaving open whether models can satisfy both under realistic enterprise conditions\.

RAG Evaluation Paradigms\.Early benchmarks focused on factual accuracy and grounding \(ALCE\([10](https://arxiv.org/html/2608.11584#bib.bib20)\), RAGAS\([8](https://arxiv.org/html/2608.11584#bib.bib21)\)\) or multi\-hop reasoning\([23](https://arxiv.org/html/2608.11584#bib.bib22)\)\. While domain\-specific benchmarks exist \(\([15](https://arxiv.org/html/2608.11584#bib.bib15)\);\([20](https://arxiv.org/html/2608.11584#bib.bib11)\);\([3](https://arxiv.org/html/2608.11584#bib.bib19)\)\), they often rely on curated sources like Wikipedia\([30](https://arxiv.org/html/2608.11584#bib.bib30)\)\. Recent frameworks like RAGBench\([9](https://arxiv.org/html/2608.11584#bib.bib14)\), RAGEval\([36](https://arxiv.org/html/2608.11584#bib.bib12)\)and EKRAG\([31](https://arxiv.org/html/2608.11584#bib.bib9)\)offer multi\-dimensional metrics and human\-curated enterprise samples but lack systematic assessment of complex instruction adherence\.

Non\-Ideal Retrieval Contexts\.Benchmarks like CRAG\([29](https://arxiv.org/html/2608.11584#bib.bib18)\), CRUD\-RAG\([18](https://arxiv.org/html/2608.11584#bib.bib13)\)and GaRaGe\([22](https://arxiv.org/html/2608.11584#bib.bib16)\)address dynamic KBs and calibration\. Others such as RGB\([2](https://arxiv.org/html/2608.11584#bib.bib8)\), RARE\([32](https://arxiv.org/html/2608.11584#bib.bib37)\), and Magic Mushroom\([33](https://arxiv.org/html/2608.11584#bib.bib3)\)introduce noise and unanswerability\. While conflict\([17](https://arxiv.org/html/2608.11584#bib.bib4);[4](https://arxiv.org/html/2608.11584#bib.bib7)\)and gap management\([11](https://arxiv.org/html/2608.11584#bib.bib6)\)exist, they are typically evaluated in isolation, decoupled from uncertainty protocols\.

![Refer to caption](https://arxiv.org/html/2608.11584v1/overview_0128.png)Figure 1:EnterpriseRAG Overview\. The top section presents the pipeline for complex instruction schema design, while the bottom section shows non\-ideal context simulation and evaluation metrics, respectively\. All are automatically generated by LLM and quality verified by humans\.Instruction Following in RAG\.General benchmarks\([35](https://arxiv.org/html/2608.11584#bib.bib33);[14](https://arxiv.org/html/2608.11584#bib.bib32);[26](https://arxiv.org/html/2608.11584#bib.bib35);[37](https://arxiv.org/html/2608.11584#bib.bib36);[21](https://arxiv.org/html/2608.11584#bib.bib34)\)highlight the need for strict verification of formatting and negative constraints\. In RAG contexts, MT\-RAG\([16](https://arxiv.org/html/2608.11584#bib.bib10)\)and CRP\-RAG\([27](https://arxiv.org/html/2608.11584#bib.bib5)\)overlook strict protocol adherence, while FollowRAG\([7](https://arxiv.org/html/2608.11584#bib.bib17)\)remains limited by synthetic injection and clean contexts\. EnterpriseRAG targets the intersection: complex multi\-constraint instructions grounded in operational workflows, evaluated under systematic retrieval noise, knowledge gaps, and factual conflicts\.

## 3EnterpriseRAG Benchmark Construction

We construct EnterpriseRAG from production RAG logs, yielding 983 expert\-validated instances spanning six domains \(Energy, Medical, Legal, Financial, Party Building and Web Search\)\. All data are desensitized to remove PII; raw logs cannot be released, but we will release the desensitized benchmark and a reproducible generation pipeline \(details in Appendix[A\.1](https://arxiv.org/html/2608.11584#A1.SS1)\)\.

#### Instance format\.

Each instance is a triple⟨q,ℐ,𝒟⟩\\langle q,\\mathcal\{I\},\\mathcal\{D\}\\rangle: user queryqq, a fused multi\-constraint instruction setℐ\\mathcal\{I\}, and a retrieved document bundle𝒟\\mathcal\{D\}\. Starting from 491 authentic queries, we construct 983 instances by pairing queries with controlled non\-ideal retrieval scenarios \(overview in Figure[1](https://arxiv.org/html/2608.11584#S2.F1); subset composition in Table[6](https://arxiv.org/html/2608.11584#A1.T6)\)\. Figure[15](https://arxiv.org/html/2608.11584#A5.F15)illustrates a Legal domain case study\.

#### Construction pipeline\.

Our pipeline operationalizes two principles:realistic constraint complexity\(from enterprise prompts\) andcontrolled non\-ideal retrieval\(noise/gap/conflict\)\. We: \(1\) collect and filter queries; \(2\) synthesizeℐ\\mathcal\{I\}by fusing atomic constraints and removing internal contradictions; \(3\) retrieve documents via hybrid retrieval \(BM25\+dense\) and assign non\-ideal modes; \(4\) conduct expert verification for instruction–query consistency and context validity \(Appendix[A\.2](https://arxiv.org/html/2608.11584#A1.SS2)\)\. Prompt templates and quality control recipes are documented in Appendix[E\.1](https://arxiv.org/html/2608.11584#A5.SS1)\.

Table 2:Performance of various LLMs on noisy subset\. The average scores across the dataset are reported as percentages\. The best and second\-best scores are marked inboldandunderlined, respectively\.ModelInferenceParadigmRAG QualityIASLoose IASFaithfulnessLooseStrictKnowledgeInteractionProtocolopen\-source modelsQwen3\-8bReasoning64\.856\.475\.512\.370\.182\.070\.3Qwen3\-14bReasoning66\.859\.477\.014\.374\.281\.572\.4Qwen3\-32bReasoning64\.959\.577\.215\.975\.480\.175\.1Qwen3\-235B\-A22B\-Thinking\-2507Reasoning67\.164\.183\.826\.882\.286\.282\.5Qwen3\-30B\-A3B\-Instruct\-2507Standard63\.965\.676\.413\.279\.081\.568\.9Qwen3\-235B\-A22B\-Instruct\-2507Standard67\.467\.180\.620\.882\.383\.676\.3DeepSeek\-R1\-0528Reasoning68\.966\.483\.121\.981\.384\.382\.9DeepSeek\-V3\.1Standard69\.960\.482\.222\.179\.985\.978\.8GLM\-4\.5Reasoning76\.664\.381\.621\.575\.485\.778\.7closed\-source modelsGemini\-2\.5\-ProReasoning73\.561\.683\.726\.586\.685\.680\.3GPT\-4\.1Standard69\.865\.880\.019\.579\.482\.177\.8Claude\-Opus\-4\.5Reasoning76\.468\.583\.325\.384\.784\.182\.4Claude\-Sonnet\-4Standard76\.866\.779\.819\.577\.181\.579\.1

### 3\.1Complex Instruction Schema

We organize constraints into three orthogonal dimensions:Persona Definition,Output Constraints, andKnowledge Interaction Protocols\. This schema captures enterprise\-critical behavioral requirements \(e\.g\., citation, gap identification, conflict handling\) in addition to structural formatting rules\. Definitions and distributions are provided in Appendix[A\.3](https://arxiv.org/html/2608.11584#A1.SS3)–[A\.4](https://arxiv.org/html/2608.11584#A1.SS4)\(Table[4](https://arxiv.org/html/2608.11584#A1.T4), Table[5](https://arxiv.org/html/2608.11584#A1.T5), Figure[7](https://arxiv.org/html/2608.11584#A1.F7)\)\.

Figure 2:Instruction Adherence and Robustness comparison on the knowledge gap subset\. Reasoning\-enhanced models generally demonstrate superior capability in both protocol adherence and refusal of unanswerable queries\.Figure 3:Correlation between Conflict Recognition Rate and Answer Coverage across 13 models on the factual conflict subset \(309 cases\)\. Each point represents one model\. Reasoning\-enhanced models exhibit a strong positive correlation \(ρ\\rho= \+0\.90\), while standard models show no significant relationship \(ρ\\rho= \-0\.50\)\.
### 3\.2Non\-Ideal Retrieval Scenarios

We construct three non\-ideal retrieval scenarios that stress the generator under realistic enterprise failure conditions:Noisy Retrieval\(topically similar but contextually irrelevant documents\),Knowledge Gaps\(topically related but insufficient evidence in retrieved contexts\), andFactual Conflicts\(contradictory statements in retrieved passages\)\. Because gaps and conflicts are sparse in natural logs, we augment them with controlled procedures while preserving domain coherence; we further validate that synthetic conflicts match natural difficulty on core robustness signals \(Appendix[A\.5](https://arxiv.org/html/2608.11584#A1.SS5)and Appendix[B\.2](https://arxiv.org/html/2608.11584#A2.SS2)\)\.

### 3\.3Evaluation Metrics

Given the open\-ended nature of enterprise queries, we report RAG quality and instruction adherence signals without requiring gold reference answers\. Faithfulness and Answer Coverage follow a RAGAS\-style claim\-based evaluation\([8](https://arxiv.org/html/2608.11584#bib.bib21)\)\.

Faithfulness \(ℱ\\mathcal\{F\}\)\.We calculate faithfulness asℱ=\|Cs​u​p\|/\|Ct​o​t​a​l\|\\mathcal\{F\}=\|C\_\{sup\}\|/\|C\_\{total\}\|, whereCt​o​t​a​lC\_\{total\}denotes all claims extracted from the response, andCs​u​pC\_\{sup\}denotes those supported by the retrieved context\.

#### Answer Coverage \(C\)\.

𝒞=α​\|Ca​n​s∩R\|\|Ca​n​s\|\+\(1−α\)​\|Sa​n​s∩R\|\|Sa​n​s\|\\mathcal\{C\}=\\alpha\\frac\{\|C\_\{ans\}\\cap R\|\}\{\|C\_\{ans\}\|\}\+\(1\-\\alpha\)\\frac\{\|S\_\{ans\}\\cap R\|\}\{\|S\_\{ans\}\|\}\(1\)whereCa​n​sC\_\{ans\}andSa​n​sS\_\{ans\}denote core and supplementary claims from contexts, respectively,RRrepresents the response content, andα=0\.7\\alpha=0\.7weights core claims higher\.

#### Instruction Adherence Score \(IAS\)\.

We report:Loose IASas the proportion of satisfied constraints, andStrict IASas a binary score indicating whether*all*constraints are satisfied\.

#### Robustness metrics\.

For non\-ideal subsets, we compute:Rejection Accuracyon knowledge gaps, andConflict Recognition Accuracyon factual conflicts\.

## 4Experiments

### 4\.1Experimental Setup

Models\.We evaluate 13 LLMs spanning open/closed\-source and standard/reasoning\-enhanced variants\([28](https://arxiv.org/html/2608.11584#bib.bib23);[6](https://arxiv.org/html/2608.11584#bib.bib24);[19](https://arxiv.org/html/2608.11584#bib.bib29);[24](https://arxiv.org/html/2608.11584#bib.bib25);[5](https://arxiv.org/html/2608.11584#bib.bib26);[1](https://arxiv.org/html/2608.11584#bib.bib28)\), full list in Table[2](https://arxiv.org/html/2608.11584#S3.T2)\. For consistency, we adopt simplified names after the first mention: “Thinking” models are denoted as\-Thinking\(abbrev\.\-T\) and standard instruction\-tuned counterparts as\-Instruct\(abbrev\.\-I\)\. We omit version suffixes \(e\.g\.,\-2507\) unless needed for disambiguation\.

#### Evaluation protocol\.

Results are reported on three non\-ideal subsets: Noisy Retrieval \(n=447n\{=\}447\), Knowledge Gaps \(n=227n\{=\}227\), and Factual Conflicts \(n=309n\{=\}309\)\. IAS evaluation uses rule\-based checks for structural constraints\([35](https://arxiv.org/html/2608.11584#bib.bib33)\)and LLM\-as\-a\-judge \(Kimi\-k2\-thinking\([25](https://arxiv.org/html/2608.11584#bib.bib27)\)\) for behavioral protocols\. Not all instances include explicit Knowledge Interaction Protocol constraints; we therefore compare naturally protocol\-present vs\. protocol\-absent cases for robustness analyses\. Evaluation prompt templates are in Appendix[E](https://arxiv.org/html/2608.11584#A5)\.

#### Evaluator reliability\.

Cross\-judge comparison across three LLM evaluators shows stable scores and consistent model rankings \([SectionB\.1](https://arxiv.org/html/2608.11584#A2.SS1)\)\. On 150 human\-annotated samples, experts achieve strong agreement \(κ=0\.85\\kappa\{=\}0\.85\), and the LLM evaluator \(Kimi\-k2\-thinking\) aligns well with human\-annotated gold labels \(κ=0\.77\\kappa\{=\}0\.77, 88% agreement\), with especially high alignment on conflict recognition \(κ=0\.93\\kappa\{=\}0\.93;[SectionB\.3](https://arxiv.org/html/2608.11584#A2.SS3)\)\.

Figure 4:Error distribution by constraint category\. Knowledge Interaction Protocols exhibit the highest failure rates, confirming that behavioral judgment, not formatting, is the core bottleneck\.ModelCRModelCRDeepSeek\-R1†44\.3DeepSeek\-V3\.129\.8Gemini\-2\.5\-Pro†42\.3Qwen3\-30B\-I28\.5Qwen3\-235B\-T†41\.4Qwen3\-14B\-T†27\.2GLM\-4\.5†40\.1Claude\-Opus†26\.9Qwen3\-235B\-I37\.5Claude\-Sonnet25\.9Qwen3\-32B\-T†36\.6Qwen3\-8B\-T†23\.9GPT\-4\.118\.5†Reasoning\-enhanced\. CR: Conflict Recog\. \(%\)\.Table 3:Conflict recognition rates \(CR\) across 13 models\.Figure 5:Reasoning vs\. standard model accuracy across RAG scenarios\. Blue segments denote reasoning gains over standard baselines \(yellow\)\. Error bars: 95% CI\.∗p<\.05\{\}^\{\*\}p<\.05,p∗⁣∗<\.01\{\}^\{\*\*\}p<\.01,∗∗∗p<\.001\{\}^\{\*\*\*\}p<\.001\(McNemar’s test\)\.Figure 6:Effect of knowledge interaction protocol on\(A\)rejection accuracy and\(B\)conflict recognition\. Points show accuracy differences \(with/without protocol\) with 95% CIs\. The protocol significantly improves conflict recognition in all 13 models, with more modest effects on rejection accuracy \(2/13 significant\)\. Independent samples \(with/without: 157/69 for A, 113/193 for B\);pp\-values fromχ2\\chi^\{2\}test with Yates’ correction\.

### 4\.2Main Results

#### Finding 1: Orchestration under noisy retrieval\.

Table[2](https://arxiv.org/html/2608.11584#S3.T2)reveals a severeadherence collapse: Loose IAS achieves up to 83\.8%, yet Strict IAS reaches only 26\.8% \(Qwen3\-235B\-Thinking\)\. This 57\-point gap quantifies thecompositional bottleneckwhere models satisfy individual constraints but fail holistic compliance\. Reasoning\-enhanced models consistently outperform standard variants, with the largest gains in Knowledge Interaction Protocols\.

#### Finding 2: Rejection under knowledge gaps\.

In production, hallucinating on unanswerable queries is often more harmful than being unhelpful\. Figure[2](https://arxiv.org/html/2608.11584#S3.F2)exposes a pervasivehelpfulness bias: Qwen3\-30B\-Instruct achieves only 6\.6% rejection accuracy, hallucinating in 93\.4% of unanswerable cases\. Reasoning\-enhanced models improve substantially \(Claude\-Opus\-4\.5: 42\.7%\), yet remain far from production\-grade reliability\.

#### Finding 3: Conflict recognition\.

Table[3](https://arxiv.org/html/2608.11584#S4.T3)shows conflict detection remains a bottleneck: top models reach only 40–44% recognition \(DeepSeek\-R1: 44\.3%\), while GPT\-4\.1 detects merely 18\.5%\. Figure[3](https://arxiv.org/html/2608.11584#S3.F3)shows reasoning\-enhanced models achieve a strong positive correlation between recognition and coverage \(ρ=\+0\.90\\rho\{=\}\+0\.90,p<0\.01p\{<\}0\.01\), while standard models show no consistent relationship \(ρ=−0\.50\\rho\{=\}\-0\.50\) with high variance\. This suggests inference\-time computation may resolve the traditional safety\-informativeness dilemma\. Synthetic and natural conflicts show equivalent difficulty on core metrics \([SectionB\.2](https://arxiv.org/html/2608.11584#A2.SS2)\)\.

### 4\.3Analysis

#### Protocol bottleneck\.

Figure[4](https://arxiv.org/html/2608.11584#S4.F4)decomposes IAS failures by constraint category\. Knowledge Interaction Protocols exhibit the highest error rates and variance, particularly for citation and gap identification\. Comparing Qwen3\-235B\-Thinking to its Instruct counterpart, the largest reasoning gains occur precisely in these protocol dimensions, confirming thatjudgment under uncertaintyis the core enterprise bottleneck\.

#### Scaling\.

Within Qwen3\-Thinking, Strict IAS scales non\-linearly \(12\.3% at 8B→\\rightarrow26\.8% at 235B\) while Faithfulness saturates \(64\.8%→\\rightarrow67\.1%\), indicating orchestration is an emergent capability requiring substantial scale\.

#### Reasoning vs\. standard instruction\-tuned variants\.

Figure[5](https://arxiv.org/html/2608.11584#S4.F5)compares matched reasoning vs\. standard variants \(Qwen3\-235B\-Thinking vs\. Qwen3\-235B\-Instruct; DeepSeek\-R1 vs\. DeepSeek\-V3\.1\)\. Across scenarios, reasoning variants exhibit substantial robustness gains: Qwen3\-235B\-Thinking boosts rejection accuracy by 18\.1pp and DeepSeek\-R1 improves conflict recognition by 14\.4pp, consistent with reduced helpfulness bias\. For Strict IAS, Qwen3\-235B\-Thinking shows consistent gains \(\+6\.1pp to \+10\.6pp;p<\.01p\{<\}\.01\), while DeepSeek\-R1 shows minimal improvement, suggesting architecture\-dependent benefits\.

#### Effect of explicit protocols\.

Figure[6](https://arxiv.org/html/2608.11584#S4.F6)compares instances with explicit Knowledge Interaction Protocol constraints to those without such constraints under the same retrieval failure mode\. Overall, explicit protocols yield a large and consistent gain in conflict recognition across all 13 models, but only modest improvements in rejection under knowledge gaps\. This asymmetry suggests that protocols help most when the failure is explicit in\-context \(contradictions\), whereas proper refusal requires a harder judgment of evidence sufficiency and separating parametric knowledge from retrieved evidence\. Notably, Claude\-Opus\-4\.5 shows a small, non\-significant decrease, indicating potential interaction with model\-specific safety behaviors\. Because this is an observational comparison \(protocol presence is not randomized\), we report domain\-level breakdowns in[SectionD\.1](https://arxiv.org/html/2608.11584#A4.SS1)\.

## 5Conclusion

EnterpriseRAG combines complex multi\-constraint instructions with three non\-ideal retrieval modes across 983 expert\-validated instances\. Across 13 LLMs, we find a persistent orchestration collapse: even the best model reaches 83\.8% per\-constraint adherence \(Loose IAS\) but only 26\.8% holistic compliance \(Strict IAS\), leaving a 57\-point gap\. Robustness failures concentrate in knowledge\-interaction protocols: under knowledge gaps, models frequently over\-answer despite explicit refusal requirements \(with Claude\-Opus\-4\.5 peaking at 42\.7%\); under factual conflicts, even the strongest systems recognize contradictions in fewer than half of cases \(led by DeepSeek\-R1 at 44\.3%\)\.

Practically, our results suggest that enterprise\-ready RAG requires \(i\) training and evaluation targeted at protocol\-level judgment \(evidence sufficiency, calibrated refusal, and conflict\-aware reporting\), not just formatting or factuality; and \(ii\) explicit operational protocols in prompts, which reliably improve conflict handling but are insufficient to solve evidence\-gap refusal\. EnterpriseRAG provides a realistic and reproducible foundation to measure and close these gaps\.

## 6Limitations

While EnterpriseRAG encompasses six diverse domains, the current scope is limited to text\-based RAG\. Multimodal contexts, such as those involving charts or images within PDFs, are not yet included\. Additionally, our reliance on a reasoning\-enhanced LLM as an evaluator, while effective, may introduce bias compared to human evaluation, although our sampling checks indicate high alignment\. Finally, the strict adherence metric is binary and stringent; future metrics could explore more nuanced semantic gradations of constraint satisfaction\.

## References

- Anthropic \(2025\)AnthropicIntroducing Claude Opus 4\.5\(Website\)Note:Blog postExternal Links:[Link](https://www.anthropic.com/news/claude-opus-4-5)Cited by:[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.p1.1)\.
- Chenet al\.\(2023a\)J\. Chen, H\. Lin, X\. Han, and L\. SunBenchmarking large language models in retrieval\-augmented generation\.External Links:2309\.01431,[Link](https://arxiv.org/abs/2309.01431)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Chenet al\.\(2023b\)W\. Chen, Q\. Wang, Z\. Long, X\. Zhang, Z\. Lu, B\. Li, S\. Wang, J\. Xu, X\. Bai, X\. Huang, and Z\. WeiDISC\-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine\-tuning\.\(en\)\.Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Choiet al\.\(2025\)E\. Choi, J\. Park, H\. Lee, and J\. LeeConflict\-aware soft prompting for retrieval\-augmented generation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 26969–26983\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1371/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1371),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Comaniciet al\.\(2025\)G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram,et al\.Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv\.External Links:[Link](http://arxiv.org/abs/2507.06261),[Document](https://dx.doi.org/10.48550/arXiv.2507.06261),2507\.06261Cited by:[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Ding, H\. Xin, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Wang, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, S\. Ye, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Zhao, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. ZhangDeepSeek\-r1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv\.External Links:[Link](http://arxiv.org/abs/2501.12948),[Document](https://dx.doi.org/10.48550/arXiv.2501.12948),2501\.12948 \[cs\]Cited by:[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.p1.1)\.
- Donget al\.\(2024\)G\. Dong, X\. Song, Y\. Zhu, R\. Qiao, Z\. Dou, and J\. WenToward General Instruction\-Following Alignment for Retrieval\-Augmented Generation\.arXiv\.Note:arXiv:2410\.09584 \[cs\]Comment: Working in progressExternal Links:[Link](http://arxiv.org/abs/2410.09584),[Document](https://dx.doi.org/10.48550/arXiv.2410.09584)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.7.1.1),[§1](https://arxiv.org/html/2608.11584#S1.p3.1),[§2](https://arxiv.org/html/2608.11584#S2.p4.1)\.
- Eset al\.\(2025\)S\. Es, J\. James, L\. Espinosa\-Anke, and S\. SchockaertRagas: Automated Evaluation of Retrieval Augmented Generation\.arXiv\.Note:arXiv:2309\.15217 \[cs\]Comment: Reference\-free \(not tied to having ground truth available\) evaluation framework for retrieval agumented generationExternal Links:[Link](http://arxiv.org/abs/2309.15217),[Document](https://dx.doi.org/10.48550/arXiv.2309.15217)Cited by:[§1](https://arxiv.org/html/2608.11584#S1.p2.1),[§2](https://arxiv.org/html/2608.11584#S2.p2.1),[§3\.3](https://arxiv.org/html/2608.11584#S3.SS3.p1.1)\.
- Frielet al\.\(2025\)R\. Friel, M\. Belyi, and A\. SanyalRAGBench: Explainable Benchmark for Retrieval\-Augmented Generation Systems\.arXiv\.Note:arXiv:2407\.11005 \[cs\]比较早的一篇文章 最大的亮点是用公开数据集攒了个多领域数据集 评价指标自己造了context相关性，答案context利用率，答案完整性 自己用训练集训了个小模型评估也没讲清楚为啥好External Links:[Link](http://arxiv.org/abs/2407.11005),[Document](https://dx.doi.org/10.48550/arXiv.2407.11005)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.5.1.1),[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Gaoet al\.\(2023\)T\. Gao, H\. Yen, J\. Yu, and D\. ChenEnabling Large Language Models to Generate Text with Citations\.arXiv\.Note:arXiv:2305\.14627 \[cs\]Comment: Accepted by EMNLP 2023\. Code and data are available at https://github\.com/princeton\-nlp/ALCEExternal Links:[Link](http://arxiv.org/abs/2305.14627),[Document](https://dx.doi.org/10.48550/arXiv.2305.14627)Cited by:[§1](https://arxiv.org/html/2608.11584#S1.p2.1),[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Guoet al\.\(2025\)X\. Guo, Y\. Luan, Y\. Kang, X\. Song, and J\. GuoLLM\-centric rag with multi\-granular indexing and confidence constraints\.External Links:2510\.27054,[Link](https://arxiv.org/abs/2510.27054)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Guptaet al\.\(2024\)S\. Gupta, R\. Ranjan, and S\. N\. SinghA comprehensive survey of retrieval\-augmented generation \(rag\): evolution, current landscape and future directions\.External Links:2410\.12837,[Link](https://arxiv.org/abs/2410.12837)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p1.1)\.
- Jianget al\.\(2023\)X\. Jiang, S\. Hu, D\. Yu, Y\. Zhang, Z\. Yang, Y\. Li, L\. Zhou, and Valuesimplex AI LabFinLongEval\.Note:[https://github\.com/valuesimplex/FinLongEval](https://github.com/valuesimplex/FinLongEval)Cited by:[4th item](https://arxiv.org/html/2608.11584#A1.I1.i4.p1.1)\.
- Jianget al\.\(2024\)Y\. Jiang, Y\. Wang, X\. Zeng, W\. Zhong, L\. Li, F\. Mi, L\. Shang, X\. Jiang, Q\. Liu, and W\. WangFollowBench: a multi\-level fine\-grained constraints following benchmark for large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 4667–4688\.External Links:[Link](https://aclanthology.org/2024.acl-long.257/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.257)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p4.1)\.
- Jinet al\.\(2019\)Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. LuPubMedQA: A Dataset for Biomedical Research Question Answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 2567–2577\(en\)\.External Links:[Link](https://www.aclweb.org/anthology/D19-1259),[Document](https://dx.doi.org/10.18653/v1/D19-1259)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Katsiset al\.\(2025\)Y\. Katsis, S\. Rosenthal, K\. Fadnis, C\. Gunasekara, Y\. Lee, L\. Popa, V\. Shah, H\. Zhu, D\. Contractor, and M\. DanilevskyMTRAG: A Multi\-Turn Conversational Benchmark for Evaluating Retrieval\-Augmented Generation Systems\.arXiv\.Note:arXiv:2501\.03468 \[cs\]External Links:[Link](http://arxiv.org/abs/2501.03468),[Document](https://dx.doi.org/10.48550/arXiv.2501.03468)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p4.1)\.
- Leeet al\.\(2025\)J\. Lee, K\. Lee, and T\. KimMAGIC: a multi\-hop and graph\-based benchmark for inter\-context conflicts in retrieval\-augmented generation\.External Links:2507\.21544,[Link](https://arxiv.org/abs/2507.21544)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Lyuet al\.\(2024\)Y\. Lyu, Z\. Li, S\. Niu, F\. Xiong, B\. Tang, W\. Wang, H\. Wu, H\. Liu, T\. Xu, and E\. ChenCRUD\-RAG: A Comprehensive Chinese Benchmark for Retrieval\-Augmented Generation of Large Language Models\.arXiv\.Note:arXiv:2401\.17043 \[cs\]Comment: 40 PagesExternal Links:[Link](http://arxiv.org/abs/2401.17043),[Document](https://dx.doi.org/10.48550/arXiv.2401.17043)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.3.1.1),[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- OpenAI \(2025\)OpenAIIntroducing GPT\-4\.1 in the API\(Website\)Note:Blog postExternal Links:[Link](https://openai.com/index/gpt-4-1/)Cited by:[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.p1.1)\.
- Pipitone and Alami \(2024\)N\. Pipitone and G\. H\. AlamiLegalBench\-RAG: A Benchmark for Retrieval\-Augmented Generation in the Legal Domain\.arXiv\.Note:arXiv:2408\.10343 \[cs\]External Links:[Link](http://arxiv.org/abs/2408.10343),[Document](https://dx.doi.org/10.48550/arXiv.2408.10343)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Qinet al\.\(2024\)Y\. Qin, T\. Zhang, T\. Zhang, Y\. Shen, W\. Luo, H\. Sun, Y\. Zhang, Y\. Qiao, W\. Chen, Z\. Zhou, W\. Zhang, and B\. CuiSysBench: can large language models follow system messages?\.arXiv\.Note:version: 1External Links:[Link](http://arxiv.org/abs/2408.10943),[Document](https://dx.doi.org/10.48550/arXiv.2408.10943),2408\.10943 \[cs\]Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p4.1)\.
- Sorodocet al\.\(2025\)I\. Sorodoc, L\. F\. R\. Ribeiro, R\. Blloshmi, C\. Davis, and A\. d\. GispertGaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation\.arXiv\.Note:arXiv:2506\.07671 \[cs\]Comment: ACL 2025 \(Findings\)External Links:[Link](http://arxiv.org/abs/2506.07671),[Document](https://dx.doi.org/10.48550/arXiv.2506.07671)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.10.1.1),[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Tang and Yang \(2024\)Y\. Tang and Y\. YangMultiHop\-RAG: Benchmarking Retrieval\-Augmented Generation for Multi\-Hop Queries\.arXiv\.Note:arXiv:2401\.15391 \[cs\]Comment: Link: https://github\.com/yixuantt/MultiHop\-RAG/External Links:[Link](http://arxiv.org/abs/2401.15391),[Document](https://dx.doi.org/10.48550/arXiv.2401.15391)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Teamet al\.\(2025a\)G\. 5\. Team, A\. Zeng, X\. Lv, Q\. Zheng, Z\. Hou, B\. Chen, C\. Xie, C\. Wang, D\. Yin, H\. Zeng, J\. Zhang, K\. Wang, L\. Zhong, M\. Liu, R\. Lu, S\. Cao, X\. Zhang, X\. Huang, Y\. Wei, Y\. Cheng, Y\. An, Y\. Niu, Y\. Wen, Y\. Bai, Z\. Du, Z\. Wang, Z\. Zhu, B\. Zhang, B\. Wen, B\. Wu, B\. Xu, C\. Huang, C\. Zhao, C\. Cai, C\. Yu, C\. Li, C\. Ge, C\. Huang, C\. Zhang, C\. Xu, C\. Zhu, C\. Li, C\. Yin, D\. Lin, D\. Yang, D\. Jiang, D\. Ai, E\. Zhu, F\. Wang, G\. Pan, G\. Wang, H\. Sun, H\. Li, H\. Li, H\. Hu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Wang, H\. Yang, H\. Liu, H\. Zhao, H\. Liu, H\. Yan, H\. Liu, H\. Chen, J\. Li, J\. Zhao, J\. Ren, J\. Jiao, J\. Zhao, J\. Yan, J\. Wang, J\. Gui, J\. Zhao, J\. Liu, J\. Li, J\. Li, J\. Lu, J\. Wang, J\. Yuan, J\. Li, J\. Du, J\. Du, J\. Liu, J\. Zhi, J\. Gao, K\. Wang, L\. Yang, L\. Xu, L\. Fan, L\. Wu, L\. Ding, L\. Wang, M\. Zhang, M\. Li, M\. Xu, M\. Zhao, M\. Zhai, P\. Du, Q\. Dong, S\. Lei, S\. Tu, S\. Yang, S\. Lu, S\. Li, S\. Li, Shuang\-Li, S\. Yang, S\. Yi, T\. Yu, W\. Tian, W\. Wang, W\. Yu, W\. L\. Tam, W\. Liang, W\. Liu, X\. Wang, X\. Jia, X\. Gu, X\. Ling, X\. Wang, X\. Fan, X\. Pan, X\. Zhang, X\. Zhang, X\. Fu, X\. Zhang, Y\. Xu, Y\. Wu, Y\. Lu, Y\. Wang, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Wang, Y\. Li, Y\. Su, Y\. Geng, Y\. Zhu, Y\. Yang, Y\. Li, Y\. Wu, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Zhang, Z\. Liu, Z\. Yang, Z\. Zhou, Z\. Qiao, Z\. Feng, Z\. Liu, Z\. Zhang, Z\. Wang, Z\. Yao, Z\. Wang, Z\. Liu, Z\. Chai, Z\. Li, Z\. Zhao, W\. Chen, J\. Zhai, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. TangGLM\-4\.5: agentic, reasoning, and coding \(ARC\) foundation models\.arXiv\.External Links:[Link](http://arxiv.org/abs/2508.06471),[Document](https://dx.doi.org/10.48550/arXiv.2508.06471),2508\.06471 \[cs\]Cited by:[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.p1.1)\.
- Teamet al\.\(2025b\)K\. Team, Y\. Bai, Y\. Bao, G\. Chen, J\. Chen, N\. Chen, R\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Z\. Chen, J\. Cui, H\. Ding, M\. Dong, A\. Du, C\. Du, D\. Du, Y\. Du, Y\. Fan, Y\. Feng, K\. Fu, B\. Gao, H\. Gao, P\. Gao, T\. Gao, X\. Gu, L\. Guan, H\. Guo, J\. Guo, H\. Hu, X\. Hao, T\. He, W\. He, W\. He, C\. Hong, Y\. Hu, Z\. Hu, W\. Huang, Z\. Huang, Z\. Huang, T\. Jiang, Z\. Jiang, X\. Jin, Y\. Kang, G\. Lai, C\. Li, F\. Li, H\. Li, M\. Li, W\. Li, Y\. Li, Y\. Li, Z\. Li, Z\. Li, H\. Lin, X\. Lin, Z\. Lin, C\. Liu, C\. Liu, H\. Liu, J\. Liu, J\. Liu, L\. Liu, S\. Liu, T\. Y\. Liu, T\. Liu, W\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Z\. Liu, E\. Lu, L\. Lu, S\. Ma, X\. Ma, Y\. Ma, S\. Mao, J\. Mei, X\. Men, Y\. Miao, S\. Pan, Y\. Peng, R\. Qin, B\. Qu, Z\. Shang, L\. Shi, S\. Shi, F\. Song, J\. Su, Z\. Su, X\. Sun, F\. Sung, H\. Tang, J\. Tao, Q\. Teng, C\. Wang, D\. Wang, F\. Wang, H\. Wang, J\. Wang, J\. Wang, J\. Wang, S\. Wang, S\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, C\. Wei, Q\. Wei, W\. Wu, X\. Wu, Y\. Wu, C\. Xiao, X\. Xie, W\. Xiong, B\. Xu, J\. Xu, J\. Xu, L\. H\. Xu, L\. Xu, S\. Xu, W\. Xu, X\. Xu, Y\. Xu, Z\. Xu, J\. Yan, Y\. Yan, X\. Yang, Y\. Yang, Z\. Yang, Z\. Yang, Z\. Yang, H\. Yao, X\. Yao, W\. Ye, Z\. Ye, B\. Yin, L\. Yu, E\. Yuan, H\. Yuan, M\. Yuan, H\. Zhan, D\. Zhang, H\. Zhang, W\. Zhang, X\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Z\. Zhang, H\. Zhao, Y\. Zhao, H\. Zheng, S\. Zheng, J\. Zhou, X\. Zhou, Z\. Zhou, Z\. Zhu, W\. Zhuang, and X\. ZuKimi k2: open agentic intelligence\.arXiv\.External Links:[Link](http://arxiv.org/abs/2507.20534),[Document](https://dx.doi.org/10.48550/arXiv.2507.20534),2507\.20534 \[cs\]Cited by:[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.SSS0.Px1.p1.1)\.
- Wenet al\.\(2024\)B\. Wen, P\. Ke, X\. Gu, L\. Wu, H\. Huang, J\. Zhou, W\. Li, B\. Hu, W\. Gao, J\. Xu, Y\. Liu, J\. Tang, H\. Wang, and M\. HuangBenchmarking complex instruction\-following with multiple constraints composition\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2407.03978)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p4.1)\.
- Xuet al\.\(2025\)K\. Xu, K\. Zhang, J\. Li, W\. Huang, and Y\. WangCRP\-RAG: a retrieval\-augmented generation framework for supporting complex logical reasoning and knowledge planning\.Electronics14\(1\),pp\. 47\.External Links:[Document](https://dx.doi.org/10.3390/electronics14010047),[Link](https://doi.org/10.3390/electronics14010047)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p4.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv\.External Links:[Link](http://arxiv.org/abs/2505.09388),[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),2505\.09388 \[cs\]Cited by:[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.p1.1)\.
- Yanget al\.\(2024\)X\. Yang, K\. Sun, H\. Xin, Y\. Sun, N\. Bhalla, X\. Chen, S\. Choudhary, R\. D\. Gui, Z\. W\. Jiang, Z\. Jiang, L\. Kong, B\. Moran, J\. Wang, Y\. E\. Xu, A\. Yan, C\. Yang, E\. Yuan, H\. Zha, N\. Tang, L\. Chen, N\. Scheffer, Y\. Liu, N\. Shah, R\. Wanga, A\. Kumar, W\. Yih, and X\. L\. DongCRAG – Comprehensive RAG Benchmark\.arXiv\.Note:arXiv:2406\.04744 \[cs\]Comment: NeurIPS 2024 Datasets and Benchmarks TrackExternal Links:[Link](http://arxiv.org/abs/2406.04744),[Document](https://dx.doi.org/10.48550/arXiv.2406.04744)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.4.1.1),[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.External Links:1809\.09600,[Link](https://arxiv.org/abs/1809.09600)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Yuet al\.\(2025\)T\. Yu, W\. Zhou, L\. Leiyang, A\. Shukla, M\. Mmadugula, P\. Gundecha, N\. Burnett, A\. Xu, V\. Viseth, T\. Tbar, R\. Akkiraju, and V\. ZhangEKRAG: Benchmark RAG for Enterprise Knowledge Question Answering\.InProceedings of the 4th International Workshop on Knowledge\-Augmented Methods for Natural Language Processing,W\. Shi, W\. Yu, A\. Asai, M\. Jiang, G\. Durrett, H\. Hajishirzi, and L\. Zettlemoyer \(Eds\.\),Albuquerque, New Mexico, USA,pp\. 152–159\.Note:创新点 1\.引入了Enterprise\-Knowledge RAG \(EKRAG\) 数据集，这是一个针对企业知识问答RAG系统进行基准测试的资源，包含来自多样企业文档的1,347个精选问题。之前没有企业知识应用的评估 –1\.1 1347个人工标注问题，评估rag的每个组件 head \-10–1\.2多跳设置，5个核心问题类型？ 2\.为评估答案质量，作者提出了embedding\-model\-as\-judge和ranking\-model\-as\-judge两种新方法，并结合现有LLM\-as\-judge技术，全面衡量了生成答案的正确性、相关性和忠实性。 –2\.1系统评估了针对企业内容的各种检索模型和策略。此外 3\.优化rag pipeline，并做实验External Links:ISBN 979\-8\-89176\-229\-9,[Link](https://aclanthology.org/2025.knowledgenlp-1.13/),[Document](https://dx.doi.org/10.18653/v1/2025.knowledgenlp-1.13)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.8.1.1),[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Zenget al\.\(2025\)Y\. Zeng, T\. Cao, D\. Wang, X\. Zhao, Z\. Qiu, M\. Ziyadi, T\. Wu, and L\. LiRARE: retrieval\-aware robustness evaluation for retrieval\-augmented generation systems\.External Links:2506\.00789,[Link](https://arxiv.org/abs/2506.00789),[Document](https://dx.doi.org/10.48550/arXiv.2506.00789)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.9.1.1),[§1](https://arxiv.org/html/2608.11584#S1.p3.1),[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, Y\. Wang, Y\. Chen, S\. Zhang, X\. Dai, S\. Bi, and G\. QiMagic mushroom: a customizable benchmark for fine\-grained analysis of retrieval noise erosion in rag systems\.External Links:2506\.03901,[Link](https://arxiv.org/abs/2506.03901)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p3.1)\.
- Zhaoet al\.\(2024\)P\. Zhao, H\. Zhang, Q\. Yu, Z\. Wang, Y\. Geng, F\. Fu, L\. Yang, W\. Zhang, J\. Jiang, and B\. CuiRetrieval\-augmented generation for ai\-generated content: a survey\.External Links:2402\.19473,[Link](https://arxiv.org/abs/2402.19473)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv\.External Links:[Link](http://arxiv.org/abs/2311.07911),[Document](https://dx.doi.org/10.48550/arXiv.2311.07911),2311\.07911 \[cs\]Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p4.1),[§4\.1](https://arxiv.org/html/2608.11584#S4.SS1.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2025\)K\. Zhu, Y\. Luo, D\. Xu, Y\. Yan, Z\. Liu, S\. Yu, R\. Wang, S\. Wang, Y\. Li, N\. Zhang, X\. Han, Z\. Liu, and M\. SunRAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework\.arXiv\.Note:arXiv:2408\.01262 \[cs\]Comment: https://github\.com/OpenBMB/RAGEvalExternal Links:[Link](http://arxiv.org/abs/2408.01262),[Document](https://dx.doi.org/10.48550/arXiv.2408.01262)Cited by:[Table 1](https://arxiv.org/html/2608.11584#S1.T1.2.1.6.1.1),[§2](https://arxiv.org/html/2608.11584#S2.p2.1)\.
- Zouet al\.\(2025\)T\. Zou, X\. Zhang, H\. Yu, M\. Wang, F\. Huang, and Y\. LiEIFBENCH: extremely complex instruction following benchmark for large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 20941–20964\.External Links:ISBN 979\-8\-89176\-332\-6,[Link](https://aclanthology.org/2025.emnlp-main.1059/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1059)Cited by:[§2](https://arxiv.org/html/2608.11584#S2.p4.1)\.

## Appendix ADataset Details & Statistics

### A\.1Data Source, Privacy, and Domains

Our benchmark is constructed from real\-world enterprise operational logs in China, collected under internal data usage agreements\. The native language of EnterpriseRAG is Chinese\. To ensure privacy, all raw logs undergo a strict multi\-stage desensitization pipeline, including rule\-based removal of sensitive fields and manual expert review\. Consequently, no personally identifiable information \(PII\), proprietary identifiers, or confidential business content is included\.

Due to privacy and compliance constraints, the raw data cannot be publicly released\. However, we release the desensitized benchmark dataset, along with a reproducible synthetic data generation pipeline and a representative sample subset, to enable independent verification and follow\-up research\.

#### Domain Specifications\.

Based on the aforementioned data sources, we select six vertical domains constructed according to three complementary criteria: \(1\)Industrial Prevalence—domains with substantial enterprise RAG deployments; \(2\)Data Accessibility—availability of production logs and domain expertise; and \(3\)Task Diversity—coverage of distinct knowledge processing paradigms\.

- •Energy: Procedural QA with version drift \(e\.g\., equipment maintenance protocols across ERP system transitions\)\.
- •Medical: Clinical abstraction from fragmented dialogues, including discharge summaries and treatment synthesis\.
- •Legal: Multi\-hop statute\-case correlation with jurisdictional hierarchies\.
- •Financial: Investment advisory and policy interpretation tasks, partially leveraging FinLongEval\([13](https://arxiv.org/html/2608.11584#bib.bib31)\)\.
- •Party Building: Regulatory knowledge management and policy interpretation in organizational contexts\.
- •Web Search: Real\-time queries emphasizing information freshness and source credibility\.

### A\.2Data construction process

Our data construction follows a four\-stage pipeline: \(1\)Query Collection: We gather 491 authentic user queries from operational logs across six domains, ensuring diversity in information needs and complexity levels\. \(2\)Instruction Synthesis: For each query, we apply the constraint fusion protocol \(Section 3\.3\.2\) to generate complex multi\-dimensional instructions averaging 8 constraints per sample\. \(3\)Context Preparation: We retrieve relevant documents using hybrid retrieval \(BM25 \+ dense retrieval\) and apply non\-ideal simulation \(Section 3\.4\) to create noise, knowledge gaps, and factual conflicts\. \(4\)Human Verification: Instead of generating reference answers, the constructed samples undergo a rigorous two\-stage expert verification process\. Experts verify Instruction\-Query Consistency and Context Validity \(confirming topical relevance and the presence/absence of necessary information for non\-ideal scenarios\), filtering out low\-quality samples to ensure ecological validity\.

### A\.3Constraints Taxonomy

Table[4](https://arxiv.org/html/2608.11584#A1.T4)and figure[7](https://arxiv.org/html/2608.11584#A1.F7)outlines the taxonomy across three dimensions\. Persona Definition establishes the model’s virtual identity and audience context, while Output Constraints dictate the structural format and stylistic boundaries\. Crucially, the Knowledge Interaction Protocol enforces strict behavioral rules for evidence handling, such as citation and conflict resolution\. This dimension moves beyond simple formatting to ensure the rigorous reliability required in enterprise environments\.

Table 4:Detailed taxonomy and definitions of sub\-dimension constraints\. The table describes the specific requirements for Persona Definition, Output Constraints, and Knowledge Interaction Protocols used in the benchmark\.CategorySub\-DimensionDescriptionPersonaDefinitionRoleDefines the user’s virtual identity, such as a software engineer, medical expert, or customer service representative\.AudienceSpecifies the target readers of generated content, influencing the level of detail, terminology, and professionalism\.OutputConstraintsFormatSpecifies the output structure, such as Markdown, JSON, or requirements like the number of sections to include\.ContentDefines content requirements including word count limits, prefix/suffix specifications, and keyword frequency constraints\.NegativeExplicitly specifies prohibited elements in the output, testing the model’s fine\-grained control capability\.OrderingSpecifies the arrangement of output content, e\.g\., chronological order, alphabetical sorting, or frequency\-based ranking\.StyleDefines the response tone, ranging from rigorous and formal to relaxed and conversational\.KnowledgeInteractionProtocolConflict HandlingDefines how the model should respond when contradictory information exists within the knowledge base\.Knowledge GapRequires the model to identify and explicitly report when necessary information is missing or unavailable\.Uncertainty Expr\.Specifies how the model should express uncertainty when information is ambiguous or evidence is insufficient\.Source FilteringInstructs the model to selectively trust or ignore specific types of information sources\.CitationRequires key conclusions to be accompanied by directly cited original text excerpts as supporting evidence\.
### A\.4Domain Statistics

Table[5](https://arxiv.org/html/2608.11584#A1.T5)shows that the Medical and Financial domains exhibit the highest complexity, with 10\.21 and 8\.74 average constraints respectively\. Legal tasks feature the highest density of Knowledge Protocol constraints \(3\.45\)\. In contrast, Web Search peaks in Output Constraints \(4\.73\) with lower protocol requirements \(1\.80\), while Energy shows the lowest Persona usage \(0\.85\)\.

Figure[7](https://arxiv.org/html/2608.11584#A1.F7)illustrates the overall constraint distribution\. Output Constraints constitute the majority \(53\.1%\), followed by Knowledge Interaction Protocols \(29\.2%\) and Persona Definitions\. Within protocols, Citation and Conflict Handling represent the most frequent sub\-dimensions\.

Table 5:Statistics of queries and constraint density\. The table shows the number of samples and the average number of constraints per query across the three orthogonal dimensions for each of the six vertical domains\.Domain\#Data\#ConstraintsPersonaDef\.OutputConst\.KnowledgeProtocolFinancial1008\.741\.945\.081\.72Energy1007\.010\.853\.512\.65Legal448\.611\.593\.573\.45Medical10010\.211\.465\.023\.16Party Building487\.791\.214\.082\.50Web Search998\.411\.874\.731\.80All4918\.401\.504\.462\.45
Note:\#Data = Number of Data Samples; \#Const\. = Avg\. Constraints per query\.

### A\.5Non\-Ideal Context Composition

Table[6](https://arxiv.org/html/2608.11584#A1.T6)reports the distribution of non\-ideal context types in the final 983 instances\. Knowledge gaps and factual conflicts are augmented due to sparsity in naturally occurring logs; we therefore explicitly report the natural vs\. augmented breakdown for transparency\.

Figure 7:Hierarchical distribution of constraints in EnterpriseRAG\. The inner ring represents the three primary categories, while the outer ring details the specific sub\-dimensions\.Table 6:Distribution of non\-ideal contexts in EnterpriseRAG\.ContextCountRatioNatural Occ\.Augment\.Noisy44745\.5%447 \(100%\)–Knowledge Gaps22723\.1%12 \(5\.3%\)215 \(94\.7%\)Factual Conflicts30931\.4%27 \(8\.7%\)282 \(91\.3%\)

Note:Natural Occ\.= Natural Occurrence;Augment\.= Augmentation\. Percentages in parentheses indicate the proportion before augmentation\. We augmented underrepresented failure modes to reflect realistic deployment distributions\.

## Appendix BEvaluation Reliability Analysis

### B\.1Cross\-Judge Consistency

We validate the stability of our evaluation protocol by comparing three LLM judges: Kimi\-k2\-thinking, Qwen3\-235B\-Thinking, and GPT\-4o\. As shown in Table[7](https://arxiv.org/html/2608.11584#A2.T7), the raw evaluation scores \(upper section\) exhibit minimal variance across judges; for instance, Loose IAS scores for DeepSeek\-V3\.1 differ by less than 0\.5% between Kimi and Qwen3\.

This consistency is rigorously confirmed by statistical reliability metrics \(lower section\)\. We observe "Good" to "Excellent" reliability across all dimensions, with Intraclass Correlation Coefficients \(ICC\) exceeding 0\.80 for Loose IAS and approaching 1\.0 for robustness metrics\. The high Spearman’sρ\\rhofurther indicates that different judges preserve the same relative model rankings, ensuring that our reported performance gaps are robust to the choice of evaluator\.

Table 7:Consistency and Reliability Analysis of Evaluator LLMs\. The upper section compares the raw evaluation scores of Kimi, Qwen3, and GPT\-4o across three metrics\. The lower section reports the statistical inter\-annotator agreement metrics, validating the stability of the evaluation protocol\. ICC values are reported with 95% confidence intervals\.Model / MetricLoose IASReject AccConflict AccKimiQwen3GPT\-4oKimiQwen3GPT\-4oKimiQwen3GPT\-4oRaw Evaluation Scores \(%\)Qwen3\-235B\-Thinking89\.689\.091\.927\.827\.928\.837\.839\.139\.1DeepSeek\-V3\.186\.086\.290\.418\.919\.419\.428\.328\.931\.0Gemini\-2\.5\-Pro84\.989\.188\.333\.635\.834\.537\.239\.436\.9GPT\-4\.182\.783\.885\.810\.19\.39\.316\.616\.617\.6Inter\-Judge Reliability StatisticsICC \(2,k\)0\.809 \[0\.11–0\.99\]0\.999 \[0\.99–1\.00\]0\.996 \[0\.98–1\.00\]Avg\. Spearman’sρ\\rho0\.6001\.0000\.867Reliability VerdictGoodExcellentExcellent

### B\.2Synthetic vs\. Real\-world Data Performance

Figure[8](https://arxiv.org/html/2608.11584#A2.F8)and Table[8](https://arxiv.org/html/2608.11584#A2.T8)contrast model performance on naturally occurring \(n=27\) versus synthetic \(n=282\) conflicts\. Across five models, synthetic samples show slightly lower Faithfulness \(Δ\\Delta= \-0\.143\) and Answer Coverage \(Δ\\Delta= \-0\.020\) due to their engineered contradictions\. Critically, we observe statistical equivalence in the core robustness metrics of Conflict Recognition Accuracy and Strict IAS \(Δ\\Delta= \-0\.038, p = \.120;Δ\\Delta= \-0\.004, p = \.826\), with 95% confidence intervals within ±0\.2\. This confirms that our synthesis pipeline effectively replicates the difficulty of real\-world scenarios, ensuring the ecological validity of the augmentation strategy\.

Figure 8:Equivalence testing results comparing synthetic and natural conflict data across five evaluation metrics\. Points represent mean differences \(Natural \- Synthetic\) with 95% confidence intervals; dashed lines mark equivalence bounds \(±0\.2\)\. Core robustness metrics \(Conflict Recognition and Strict IAS\) achieve statistical equivalence, validating the synthetic data generation approach\.Table 8:Natural vs\. Synthetic Data Comparison\.n\.s\.denotes no significant difference \(p≥0\.05p\\geq 0\.05\), indicating successful replication of difficulty on core metrics\.MetricNaturalSynth\.Diff\.p\-valCore Robustness\(Target: Equivalent\)Conflict Recog\.0\.3600\.322\-0\.038\.120\(n\.s\.\)Strict IAS0\.1600\.156\-0\.004\.826\(n\.s\.\)General QualityLoose IAS0\.7930\.767\-0\.026<\.001∗∗∗Answer Cov\.0\.5510\.532\-0\.020\.054Faithfulness0\.8270\.684\-0\.143<\.001∗∗∗

### B\.3Human–LLM Judge Alignment Study

To ensure rigorous evaluation, we employed a two\-stage annotation protocol on a stratified sample of 150 instances\. First, two experts independently labeled the data, achieving robust Inter\-Annotator Agreement \(IAA\) \(avg\.κ\\kappa=0\.85, see Table[9](https://arxiv.org/html/2608.11584#A2.T9)\), which validates the clarity of our instruction taxonomy\. Disagreements were adjudicated to establish a Gold Standard\.

Against this baseline, the LLM judge demonstrates substantial reliability with an averageκ\\kappaof 0\.77\. Conflict Recognition achieves the highest alignment \(κ\\kappa=0\.93\) due to the objective nature of contradiction detection, while Strict Adherence \(κ\\kappa=0\.68\) exhibits minor divergence on borderline formatting nuances\.

Table 9:Human\-LLM Judge Alignment Study\.Analysis based on a stratified sample of 150 instances \(50 per dimension\)\.Data Quality: Inter\-Annotator Agreement \(IAA\) between two human experts\.LLM Judge Reliability: Primary evaluator \(Kimi\-k2\-thinking\) vs\. adjudicated gold standard\.Evaluation DimensionData QualityLLM Judge ReliabilityHuman IAACohen’sκ\\kappaAgreement\(κ\\kappa\)Rate \(%\)Strict IAS0\.730\.6886Proper Rej\.0\.870\.7187Conflict Recog\.0\.960\.9391Overall avg\.0\.850\.7788

## Appendix CStatistical Analysis Details

### C\.1Figure[3](https://arxiv.org/html/2608.11584#S3.F3)

#### Data and Method

Each point: model\-level aggregate overn=309n\{=\}309conflict instances \(X: conflict recognition rate; Y: mean answer coverage\)\. We computeSpearman’sρ\\rhoseparately for reasoning\-enhanced \(n=8n\{=\}8\) and standard \(n=5n\{=\}5\) models due to their distinct architectures\.

#### Limitations

Small sample sizes limit power\. Models within families \(e\.g\., Qwen3\) may not be fully independent; sensitivity analysis yields consistent patterns\. Model\-level correlations may not reflect instance\-level relationships\.

### C\.2Figure[5](https://arxiv.org/html/2608.11584#S4.F5)

Method: McNemar’s test \(one\-sided\) for paired binary outcomes\. Each instance evaluated by both reasoning\-enhanced and standard models within the same family\.

Samples: Noisy Retrieval \(n=447n\{=\}447\), Knowledge Gaps \(n=227n\{=\}227\), Factual Conflicts \(n=309n\{=\}309\)\. Exact binomial test when discordant pairs<25<25; otherwise z\-test with continuity correction\.

Confidence Intervals: Percentile bootstrap \(10,000 resamples\) preserving pairing\.

Multiple Testing: Uncorrectedpp\-values for 10 planned comparisons; Bonferroni correction \(α=\.005\\alpha\{=\}\.005\) does not change conclusions\.

### C\.3Figure[6](https://arxiv.org/html/2608.11584#S4.F6)

We compare accuracy between independent groups \(samples with vs\. without the protocol\) using standard methods for comparing two proportions\.

For each model, we construct a2×22\{\\times\}2contingency table and apply:Pearson’sχ2\\chi^\{2\}test\(with Yates’ correction\) when expected counts≥5\\geq 5,Fisher’s exact testotherwise\.

We report the accuracy differenceΔ=p^with−p^without\\Delta=\\hat\{p\}\_\{\\text\{with\}\}\-\\hat\{p\}\_\{\\text\{without\}\}with 95% Wald confidence intervals\.

Sample sizes: rejection \(157 with / 69 without\), conflict recognition \(113 / 193\)\. Analysis used SciPy\. Of 26 tests, 25 usedχ2\\chi^\{2\}and 1 used Fisher’s exact\. Uncorrectedpp\-values are reported; Bonferroni correction \(α=\.002\\alpha=\.002\) does not alter substantive conclusions\.

### C\.4Figure[8](https://arxiv.org/html/2608.11584#A2.F8)

We useTOST equivalence testingto validate that synthetic conflicts \(n=282n\{=\}282\) replicate natural conflict difficulty \(n=27n\{=\}27\)\. Equivalence margin:δ=±0\.2\\delta\{=\}\{\\pm\}0\.2\(20pp, based on RAG benchmark reliability thresholds\)\. Decision rule: 95% CI for difference \(Natural−\-Synthetic\) must fall within\[−0\.2,0\.2\]\[\-0\.2,0\.2\]\.

Aggregate mean accuracy across 5 models for each dataset; bootstrap 95% CI \(10,000 resamples\)\.

#### Limitations

Small natural sample \(n=27n\{=\}27\); model\-level aggregation; families share architectures\.

### C\.5Table[8](https://arxiv.org/html/2608.11584#A2.T8)

We usepaired\-samplestt\-testto compare 13 models’ performance on natural \(n=27n=27\) vs\. synthetic \(n=282n=282\) conflicts, with Cohen’sddfor effect sizes\.

#### Limitations

Models within families may not be fully independent; unequal natural/synthetic sample sizes affect precision but not paired comparison validity\.

## Appendix DAdditional Experiments

### D\.1Fine\-grained Analysis

Figure[9](https://arxiv.org/html/2608.11584#A4.F9)illustrates the performance of five representative LLMs—including both reasoning\-enhanced and standard instruction\-tuned variants—across the six vertical domains of EnterpriseRAG under noisy retrieval conditions\. Across all domains, models generally maintain high Faithfulness and Loose IAS , but experience a sharp "orchestration collapse" in Strict IAS, confirming that simultaneously satisfying multiple domain\-specific constraints remains a primary bottleneck\. Specifically, domains with higher complexity and stricter behavioral requirements, such as Medical and Legal, exhibit lower absolute Strict IAS scores compared to Web Search\. This trend aligns with the domain statistics in Table[5](https://arxiv.org/html/2608.11584#A1.T5), which show that Medical and Legal tasks feature the highest average number of constraints and the densest concentration of Knowledge Interaction Protocols\. Furthermore, reasoning\-enhanced models \(e\.g\., Qwen3\-235B\-Thinking and DeepSeek\-R1\) consistently outperform their counterparts across all metrics and domains, particularly in Strict IAS and Answer Coverage\. These results suggest that the difficulty of instruction adherence is inherently tied to domain\-specific constraint density, and that robust performance in complex enterprise scenarios is an emergent capability heavily dependent on inference\-time reasoning rather than simple pattern matching\.

Figure 9:Model performance on noisy retrieval contexts across six vertical domains\. Radar charts illustrate the trade\-offs between Faithfulness, Answer Coverage, and Instruction Adherence \(Strict/Loose\) for representative models\.

## Appendix EComplete Prompts Repository

Note: All prompts and examples presented in this paper have been translated from the original Chinese for readership clarity\. The actual evaluation was performed using the Chinese versions\.

### E\.1The Prompts for Data Construction

D\.1\.1 Initial Atomic Libraries Generation

\[System\] You are an expert in constructing the EnterpriseRAG benchmark\. Your goal is to generate realistic, complex atomic instructions for retrieval\-augmented generation systems in specific domains\.\[Context Input\] Definition of TaskRAG Dimensions:•P \(Persona\):Role, Audience\.•C \(Constraints\):Format, Content, Negative \(forbidden\), Ordering, Style\.•K \(Knowledge Protocol\):Conflict Handling, Gap ID, Uncertainty, Source Filtering, Evidence Chaining\.\[Task Description\] Domain:<scenario\>\. Details:<detailed\_scenario\_description\>\. Requirements:1\.Generate 5\+ atomic instructions per sub\-dimension \(except Operational Goal\)\.2\.Assign probability \(0\.0\-1\.0\) and verifyGeneralization\(vs context\-specific\)\.3\.DetermineEvaluability\(Rule vs LLM\) and provideJudge Logic\(ifeval/prompt\)\.4\.Refer toInstruct\_Follow\_Evaluationfor constraint types \(word count, bullets, etc\.\)\.\[Response Examples\] Output strictly in the following JSON format:[⬇](data:text/plain;base64,ewogICJQZXJzb25hICYgU2NlbmFyaW8iOiB7CiAgICAiUm9sZSI6IFsKICAgICAgewogICAgICAgICJpbnN0cnVjdGlvbiI6ICJEbyBub3QgaW5jbHVkZSBuYW1lcyBvZiBtZWRpY2FsIHN0YWZmIG90aGVyIHRoYW4gdGhlIGF0dGVuZGluZyBwaHlzaWNpYW4uIiwKICAgICAgICAicHJvYmFiaWxpdHkiOiAwLjQsCiAgICAgICAgImdlbmVyYWxpemF0aW9uIjogIm5vIiwKICAgICAgICAianVkZ2VfbWV0aG9kIjogIkxMTSIsCiAgICAgICAgImp1ZGdlX2RldGFpbCI6ICJQcm9tcHQ6IERldGVybWluZSBpZiByZXBvcnQgY29udGFpbnMgb3RoZXIgbmFtZXMuLi4gRm9ybWF0OiB7anVkZ2VfZm9ybWF0fS4uLiIKICAgICAgfSwKICAgICAgewogICAgICAgICJpbnN0cnVjdGlvbiI6ICJEbyBub3QgdXNlIGV4Y2xhbWF0aW9uIG1hcmtzLiIsCiAgICAgICAgInByb2JhYmlsaXR5IjogMC41LAogICAgICAgICJnZW5lcmFsaXphdGlvbiI6ICJ5ZXMiLAogICAgICAgICJqdWRnZV9tZXRob2QiOiAicnVsZSIsCiAgICAgICAgImp1ZGdlX2RldGFpbCI6IHsgImluc3RydWN0aW9uX2lkX2xpc3QiOiBbImZvcmJpZGRlbl93b3JkcyJdLCAia3dhcmdzIjogW3sgImZvcmJpZGRlbl93b3JkcyI6IFsiISJdIH1dIH0KICAgICAgfQogICAgXQogIH0KfQ==)\{"Persona␣&␣Scenario""Role"\{"instruction""probability""generalization""judge\_method""judge\_detail"\},\{"instruction""probability""generalization""judge\_method""judge\_detail"\}\]\}\}

Figure 10:The prompt template used for generating EnterpriseRAG Atomic Libraries\.D\.1\.2 Check Combined Constraints Conflict

Please evaluate whether there are logical conflicts or contradictions within the atomic instruction list below\. If conflicts exist, prioritize retaining content related to\[Operational Execution\]\.Atomic Instruction List:<constraints\_list\>Please strictly return the following JSON:[⬇](data:text/plain;base64,ewogICJoYXNfY29uZmxpY3QiOiBmYWxzZSwKICAicmVhc29uIjogIkJyaWVmIGV4cGxhbmF0aW9uIiwKICAidW5jb25mbGljdF9saXN0IjogIklmIGNvbmZsaWN0cyBleGlzdCBhbmQgY2FuIGJlIHJlc29sdmVkIGJ5IHJlbW92aW5nIHNwZWNpZmljIGF0b21pYyBpbnN0cnVjdGlvbnMsIHByb3ZpZGUgdGhlIGxpc3Qgb2YgaW5kaWNlcyBhZnRlciByZW1vdmFsLCBlLmcuLCBbMSwgMywgNV07IG90aGVyd2lzZSwgcHJvdmlkZSB0aGUgb3JpZ2luYWwgbGlzdCBvZiBhdG9taWMgaW5zdHJ1Y3Rpb24gaW5kaWNlcy4iCn0=)\{"has\_conflict""reason""unconflict\_list"\}

Figure 11:The prompt template used for detecting conflict in combined constraints\.D\.1\.3 Prompt Templates for Instruction Refinement

Prompt 1: Filter Irrelevant InstructionsBased on the given question, please remove atomic instructions that cannot be triggered by the current data:Original Atomic Instruction List: <constraints\_list\>Question:question Context:contextPlease analyze which atomic instructions cannot be triggered in the current question/context and remove them\. Please return in the following JSON format:[⬇](data:text/plain;base64,ewogICJyZW1vdmVkX2F0b21zIjogWyJMaXN0IG9mIGluZGljZXMgb2YgcmVtb3ZlZCBhdG9taWMgaW5zdHJ1Y3Rpb25zLCBlLmcuLCBbMSwgM10iXSwKICAicmVhc29uIjogIlJlYXNvbiBmb3IgcmVtb3ZhbCIKfQ==)\{"removed\_atoms""reason"\}If no atomic instructions need to be removed, please return an empty list\.

Augment Constraints Based on the QueryBased on the given question and context, please add specific constraints to the current instruction list:Current Atomic Instruction List: <constraints\_list\>Question:questionContext:contextRequirements are as follows:1\.Analyze the question and context features; add 1\-2 new constraints to existing atomic instructions\.2\.Must not repeat or overlap with existing atomic instructions\.3\.Consider the following dimensions for new constraints:•Content & Format:Format/Content/Role/Style/Tone/Length, etc\.•Positive & Negative:Must include/Must not include/Avoid including, etc\.•Knowledge Interaction Protocol:Citation/No Citation/Prioritize Citation/Conflict Handling/Missing Info Handling/Uncertainty Expression\.4\.New constraints must be specific and actionable \(evaluable via code or LLM\) and avoid overly broad or vague descriptions that make evaluation difficult\.5\.New constraints should be relevant to the current question/context and improve answer quality, but description should not be too detailed \(specific only to current context\); it should have some generalization\.6\.If the current atomic instruction list is sufficiently complete, no constraints need to be added\.7\.Please return in the following JSON format:[⬇](data:text/plain;base64,ewogICJhZGRpdGlvbmFsX2NvbnN0cmFpbnRzIjogWwogICAgIHsKICAgICAgImNhdGVnb3J5IjogIkFkZGl0aW9uYWwgQ29uc3RyYWludHMiLAogICAgICAiZGltZW5zaW9uIjogIkFkZGl0aW9uYWwgQ29uc3RyYWludHMiLAogICAgICAiaW5zdHJ1Y3Rpb24iOiAiQ29uc3RyYWludCBpbnN0cnVjdGlvbiB0ZXh0IiwKICAgICAgImp1ZGdlX21ldGhvZCI6ICJsbG0iLAogICAgICAianVkZ2VfZGV0YWlsIjogIkxMTS1hcy1qdWRnZSBwcm9tcHQgZm9yIGV2YWx1YXRpbmcgdGhpcyBjb25zdHJhaW50IgogICAgIH0KICBdLAogICJyZWFzb24iOiAiUmVhc29uIGZvciBhZGRpbmcgdGhlc2UgY29uc3RyYWludHMiCn0=)\{"additional\_constraints"\{"category""dimension""instruction""judge\_method""judge\_detail"\}\],"reason"\}If no constraints need to be added, please return an empty list\.8\. Reference Example:[⬇](data:text/plain;base64,ewogICJpbnN0cnVjdGlvbiI6ICJEbyBub3QgaW5jbHVkZSBuYW1lcyBvZiBtZWRpY2FsIHN0YWZmIG90aGVyIHRoYW4gdGhlIGF0dGVuZGluZyBwaHlzaWNpYW4gaW4gdGhlIHJlcG9ydC4iLAogICJwcm9iYWJpbGl0eSI6IDAuNCwgImdlbmVyYWxpemF0aW9uIjogIm5vIiwgImp1ZGdlX21ldGhvZCI6ICJMTE0iLAogICJqdWRnZV9kZXRhaWwiOiAiUGxlYXNlIGRldGVybWluZSB3aGV0aGVyIHRoZSByZXBvcnQgYmVsb3cgb25seSBjb250YWlucyB0aGUgYXR0ZW5kaW5nIHBoeXNpY2lhbidzIG5hbWUuLi4gT3V0cHV0IGZvcm1hdDpcbntqdWRnZV9mb3JtYXR9XG5cblJlcG9ydDpcbntyZXNwb25zZX1cblxuT3JpZ2luYWwgbWVkaWNhbCByZWNvcmQ6XG57Y29udGV4dH0iCn0=)\{"instruction""probability""judge\_detail"\}9\.judge\_detailmust be complete and accurate\. It must explicitly specify the evaluation model’s output asjudge\_format, wherejudge\_formatis\{\{"Does it satisfy instruction constraints": "Yes or No", "Reason": "Provide judgment reason"\}\}\. The evaluation can citecontext,question, andresponsefields\. The instruction andjudge\_detailmust maintain logical consistency\.

D\.1\.4 Quality Assessment of Complex Instructions

You are an expert in benchmarking<task\_name\>tasks\. Your task is to evaluate the quality of the following complex instruction, composed of multiple randomly combined atomic instructions, to determine if it is suitable as a valid evaluation case\.Background: This instruction is used to evaluate a<task\_name\>large model oriented towards thedomaindomain\. The model needs to answer questions based on the given context while strictly adhering to the instructions\.Atomic Instructions to be Evaluated: <instruction\_text\>Please evaluate the above combined instruction based on the following five dimensions and output your analysis results in JSON format:1\.Logical Consistency: Evaluate whether there are internal conflicts or incoordination among the parts of the instruction \(role, goal, format, constraints\)\. Score \(1\-5, where 1 is severely inconsistent, 5 is completely consistent\)\.2\.Feasibility: Evaluate whether it is theoretically possible for a top\-tier<task\_name\>model to satisfy all these requirements simultaneously\. Check for absolute contradictions \(e\.g\., requiring a list while prohibiting lists\)\. Score \(1\-5, where 1 is completely unexecutable, 5 is completely executable\)\.3\.Realism: Evaluate whether this instruction combination simulates a real, reasonable work scenario likely to occur in a<task\_name\>task within the<domain\>domain\. Score \(1\-5, where 1 is completely unrealistic, 5 is very realistic\)\.4\.Clarity of Evaluation: Evaluate whether we still have clear, actionable methods \(whether via rules or LLM\-as\-Judge\) to judge if the model followed every instruction after combination\. Score \(1\-5, where 1 is very vague evaluation criteria, 5 is very clear\)\.5\.Appropriate Complexity: Evaluate whether the overall difficulty is too simple, moderate, or too complex to be practical\. Score \(1\-5, where 1 is too simple/complex making it ineffective, 5 is moderate complexity with good discrimination\)\.Output Format: Please strictly return your evaluation results following the JSON structure below\.[⬇](data:text/plain;base64,ewogICJpbnN0cnVjdGlvbl9pZCI6ICJbQXNzaWduIGEgdW5pcXVlIElEIGZvciB0aGlzIGluc3RydWN0aW9uIGNvbWJpbmF0aW9uXSIsCiAgImV2YWx1YXRpb25fc3VtbWFyeSI6IHsKICAgICJsb2dpY2FsX2NvbnNpc3RlbmN5IjogewogICAgICAic2NvcmUiOiA8U2NvcmUgaW50PiwKICAgICAgInJlYXNvbmluZyI6ICI8WW91ciBhbmFseXNpcyByZWFzb25pbmc+IgogICAgfSwKICAgICJmZWFzaWJpbGl0eSI6IHsKICAgICAgInNjb3JlIjogPFNjb3JlIGludD4sCiAgICAgICJyZWFzb25pbmciOiAiPFlvdXIgYW5hbHlzaXMgcmVhc29uaW5nOyBleHBsaWNpdGx5IHBvaW50IG91dCBjb250cmFkaWN0aW9ucyBpZiBhbnk+IgogICAgfSwKICAgICJyZWFsaXNtIjogewogICAgICAic2NvcmUiOiA8U2NvcmUgaW50PiwKICAgICAgInJlYXNvbmluZyI6ICI8WW91ciBhbmFseXNpcyByZWFzb25pbmc+IgogICAgfSwKICAgICJldmFsdWF0aW9uX2NsYXJpdHkiOiB7CiAgICAgICJzY29yZSI6IDxTY29yZSBpbnQ+LAogICAgICAicmVhc29uaW5nIjogIjxZb3VyIGFuYWx5c2lzIHJlYXNvbmluZz4iCiAgICB9LAogICAgImNvbXBsZXhpdHkiOiB7CiAgICAgICJzY29yZSI6IDxTY29yZSBpbnQ+LAogICAgICAicmVhc29uaW5nIjogIjxZb3VyIGFuYWx5c2lzIHJlYXNvbmluZz4iCiAgICB9CiAgfSwKICAib3ZlcmFsbF9qdWRnbWVudCI6IHsKICAgICJhdmVyYWdlX3Njb3JlIjogPENvbXBvc2l0ZSBTY29yZSBmbG9hdD4sCiAgICAicmVjb21tZW5kYXRpb24iOiAiPCdSZWNvbW1lbmRlZCcgfCAnVXNlIHdpdGggQ2F1dGlvbicgfCAnTm90IFJlY29tbWVuZGVkJz4iLAogICAgImZpbmFsX3JlbWFya3MiOiAiPEZpbmFsIHN1bW1hcnkgYW5kIG1vZGlmaWNhdGlvbiBzdWdnZXN0aW9ucyBmb3IgdGhpcyBpbnN0cnVjdGlvbj4iCiAgfQp9)\{"instruction\_id""evaluation\_summary""logical\_consistency""score""reasoning"\},"feasibility""score""reasoning"\},"realism""score""reasoning"\},"evaluation\_clarity""score""reasoning"\},"complexity""score""reasoning"\}\},"overall\_judgment""average\_score""recommendation""final\_remarks"\}\}

Figure 12:The prompt template for evaluating and scoring the refined complex instruction\.D\.1\.5 Document Relevance Assessment

\[System Instruction\] \# Task Description Please strictly evaluate whether the retrieved document below effectively supports answering the user’s question\. Analyze the relevance between the document and the question step\-by\-step and output structured results\.\# Evaluation Steps1\.Understand Question Core•Extract keywords and core requirements of the user question, clarifying the type of information needed for the answer \(e\.g\., data, reasons, steps, etc\.\)\.2\.Document Content Analysis•Check the document sentence by sentence, marking content directly related to the question \(e\.g\., data, definitions, causal explanations\)\.•Identify potentially indirectly supporting information \(e\.g\., background knowledge, analogous cases\)\.3\.Relevance Judgment•Determine if the document containskey evidenceneeded to answer the question \(e\.g\., "Yes/No", must specify concretely\)\.•Check if the information is complete and reliable \(e\.g\., source of data, existence of contradictions\)\.4\.Support Confirmation•Confirm whether the document content can fully or partially answer the question, rather than being unable to answer or completely irrelevant to the question content\.\# Output Format[⬇](data:text/plain;base64,ewogICJzdXBwb3J0c19xdWVzdGlvbiI6ICJZZXMvTm8iLAogICJjb25maWRlbmNlIjogIlBlcmNlbnRhZ2UgKDAtMTAwJSkiLAogICJyZWFzb25pbmciOiAiMS0yIHNlbnRlbmNlcyBleHBsYWluaW5nIHRoZSBiYXNpcywgZS5nLiwgbWlzc2luZyBpbmZvcm1hdGlvbiBvciBjb21wbGV0ZWx5IGlycmVsZXZhbnQgdG8gcXVlc3Rpb24gY29udGVudCIsCiAgImtleV9ldmlkZW5jZV9jaXRhdGlvbnMiOiBbIk9yaWdpbmFsIHRleHQgZnJhZ21lbnRzIGZyb20gdGhlIGRvY3VtZW50Il0KfQ==)\{"supports\_question""confidence""reasoning""key\_evidence\_citations"\}\[User Instruction\] \*\*Data for Analysis\*\* User Question:0 Retrieved Document:1

Figure 13:The prompt used to verify the relevance of retrieved documents and query\.D\.1\.6 Conflict Management

Conflict Detection\[System\] You are an expert in information consistency and time\-sensitivity analysis\. Please judge whether conflicts/contradictions exist in the given context segments: two or more segments provide opposite or mutually exclusive conclusions regarding the same fact\. Please list: 1\) Whether the above issue exists \(Yes/No\); 2\) The specific context segments involved in the conflict, labeled as Index 1 and Index 2 \(indices start from 0\); 3\) Summary of the conflict point; 4\) Whether it affects the answer \(Yes/No\) and the reason\.\[User\] User Question:query Context: \[0\] Context\_0 \[1\] Context\_1…Please output in JSON:\{"has\_conflict": "Yes/No", "details": \[\{"index1":\[\], "index2":\[\], "summary":"\.\.\.", "affects\_answer": "Yes/No"\}\]\}

Conflict document synthesis\[System\]You are a data construction expert\.\[User\]Please carefully read and follow the instructions below to complete a conflict context construction task\.\# Task Objective Your task is to act as a data fabrication expert\. Based on the user’s "Question" and a series of "Correct Context" segments, you need to constructnumnew, deceptive context segments\. These new segments must contain information that directly conflicts or contradicts specific facts in the "Correct Context"\.\# Core Requirements1\.Modify Based on Facts: Do not fabricate information out of thin air that is irrelevant to the original context\. You must select one or more key fact points \(e\.g\., numbers, dates, names, conclusions, status\) from the "Correct Context" and modify them to create contradictions\.2\.Maintain Context Relevance: The constructed conflict segments must be highly relevant to the original question and context in terms of topic and phrasing\. They should read naturally and credibly, not appearing obviously fake\.3\.Explicitly Identify Conflict Points: After construction, clearly indicate which original segment\(s\) the new segment conflicts with and concisely summarize the core content of the conflict\.\# Input Format•Question: User’s original question\.•Correct Context: One or more segments, each prefixed with\-\[Index\], e\.g\.,\-\[0\]\.\# Output Format Strictly output in the following JSON format without additional explanation:[⬇](data:text/plain;base64,ewogICJnZW5lcmF0ZWRfY29udGV4dHMiOiBbCiAgICB7CiAgICAgICJzb3VyY2VfaW5kaWNlcyI6IFswXSwKICAgICAgImNvbmZsaWN0X2lkIjogImMwIiwKICAgICAgInRleHQiOiAiWW91ciBmaXJzdCBjb25zdHJ1Y3RlZCBjb25mbGljdCBzZWdtZW50IGhlcmUuIEl0IHNob3VsZCBsb29rIGNyZWRpYmxlIGJ1dCBjb250YWluIGluZm9ybWF0aW9uIGNvbnRyYWRpY3RpbmcgcGFyYWdyYXBoIGAtWzBdYC4iLAogICAgICAiY29uZmxpY3Rfc3VtbWFyeSI6ICJFLmcuOiBDaGFuZ2VkIHRoZSByZWxlYXNlIHllYXIgMjAyMyBpbiB0aGUgb3JpZ2luYWwgY29udGV4dCB0byAyMDIyLiIKICAgIH0sCiAgICB7CiAgICAgICJzb3VyY2VfaW5kaWNlcyI6IFsxLCAyXSwKICAgICAgImNvbmZsaWN0X2lkIjogImMxIiwKICAgICAgInRleHQiOiAiWW91ciBzZWNvbmQgY29uc3RydWN0ZWQgY29uZmxpY3Qgc2VnbWVudC4uLiIsCiAgICAgICJjb25mbGljdF9zdW1tYXJ5IjogIkUuZy46IFJlcGxhY2VkIHRoZSBtYWluIGNvbnRyaWJ1dG9yICdKb2huIERvZScgd2l0aCAnSmFuZSBEb2UnLiIKICAgIH0KICBdCn0=)\{"generated\_contexts"\{"source\_indices""conflict\_id""text""conflict\_summary"\},\{"source\_indices""conflict\_id""text""conflict\_summary"\}\]\}Usage Example: Question:queryCorrect Context:contextOutput:

Figure 14:The two\-stage process for conflict management: detection of existing contradictions and generation of synthetic conflicts to test model robustness\.
### E\.2The Prompts for Evaluation

D\.2\.1 Answer Coverage Evaluation

Prompt 1: Claim ExtractionYou are a highly precise information extraction expert\. Your task is to analyze a user’s question and a provided context, then extract all relevant factual claims\. You must classify each claim as either\[core\]or\[supplementary\]\.•\[core\]: The essential, direct answer to the question\.•\[supplementary\]: Valuable, additional information like preconditions, exceptions, timelines, or problem\-solving steps\.Follow the output format exactly as shown in the examples\. Each claim must be on a new line\. If no relevant information exists, you MUST respond with the single word: "None"\.— Example 1 — Question:How do I reset my password? Context:To reset your password, click the ’Forgot Password’ link on the login page\. You will receive an email with instructions\. Please note that the reset link is only valid for 10 minutes\. Key Information Points: \[core\]Users can reset their password by clicking the ’Forgot Password’ link\. \[supplementary\]After clicking the link, an email with instructions will be sent\. \[supplementary\]The password reset link is valid for only 10 minutes\.— Example 2 — Question:What happens if the ’Pay without getting out’ button doesn’t respond in the app? Context:In the ’Pay without getting out’ section of the Alipay mini\-program, you need to swipe up to reveal the fuel pump selection screen\. If that doesn’t work, ensure your network connection is stable\. For persistent issues, contact support at 400\-123\-4567\. Key Information Points: \[core\]The user needs to swipe up on the screen to show the fuel pump selection page\. \[supplementary\]The user should check if their network connection is stable\. \[supplementary\]For persistent issues, users can contact support at 400\-123\-4567\. — End of Examples —Now, perform the task for the following real data, note that there may be more than one core claim involved\.Question:query Context:context Key Information Points:

Prompt 2: Coverage VerificationYou are a meticulous verifier\. For the given "Generated Answer", determine if each "Key Information Statement" from the list is semantically covered\.Generated Answer: answerKey Information Statements: claimsRespond ONLY with a JSON object where keys are the exact claim index number\(start from 1\) and values are a boolean \(true for covered, false for not covered\)\. Example:\{"1": true, "2": false\}JSON Output:

D\.2\.2 Prompt Template for instruction Adherence Evaluation

\[System Instruction\] You are a fair, objective, and inclusive expert in instruction\-following evaluation\. Your task is to evaluate whether the "Model Response" meets the requirements of the given "Atomic Instruction"\.In conducting the evaluation, please strictly adhere to the following principles:1\.Substance Over Form:•Focus on whether the model captured the core intent of the instruction\.•Do not judge as "No" due to minor wording differences, punctuation, or non\-core formatting flaws\.•As long as the response achieves the instruction’s goal in logic and content, it is considered satisfied\.2\.Avoid Over\-Interpretation:•For negative constraints \(e\.g\., "Do not include…"\), judge as "No" only when there is a clear violation\.•For style/persona instructions \(e\.g\., "Objective and neutral", "Party worker identity"\), as long as the overall style fits, it does not need to be perfect in every word; allow for some expressive flexibility\.3\.Precondition Check:•If the instruction contains a conditional clause \(e\.g\., "If the knowledge base contains contradictions…"\), but the condition is not triggered in the context \(i\.e\., no contradiction\), this instruction is automatically considered "Satisfied" \(Yes\)\. Do not force the model to fabricate contradictions when none exist\.4\.Independence:•Judge only against the current specific "Atomic Instruction"\. Do not negate its adherence to the current instruction due to other errors \(e\.g\., typos, hallucinations\) in the response\.5\.Judgment Criteria:•Yes: Completely satisfied, or satisfied in core elements with only minor flaws\.•No: Clearly violated the core requirements of the instruction, or missed critical steps\.Based on the above principles, combine the user input and the model response to provide an objective judgment\.\[User Instruction\] judge\_detail \(Note: Thejudge\_detailvariable contains the specific evaluation prompt, context, question, and response\.\)Output Format Requirement: Please strictly output in the following JSON format:[⬇](data:text/plain;base64,ewogICJEb2VzIGl0IHNhdGlzZnkgaW5zdHJ1Y3Rpb24gY29uc3RyYWludHMiOiAiWWVzIG9yIE5vIiwKICAiUmVhc29uIjogIlByb3ZpZGUganVkZ21lbnQgcmVhc29uIgp9)\{"Doesitsatisfyinstructionconstraints":"YesorNo","Reason":"Providejudgmentreason"\}

D\.2\.3 Rejection Accuracy Evaluation

\[System Instruction\] You are an expert in evaluating the robustness of Large Language Models in RAG \(Retrieval\-Augmented Generation\) scenarios\. Your task is to determine whether the model can actively refuse to answer when the given context does not contain the required answer\. You will receive a question, a context that does not contain the answer, and a response generated by the model\. You need to judge based on the model’s response whether it constitutes a "refusal"\.\[User Instruction\] \# Task Description Evaluate whether the following response generated by the language model is a "Refusal"\. "Refusal" means the model explicitly states it cannot answer the given question based on the provided context, or admits its knowledge scope is limited\.Evaluation Criteria:Valid Refusal \(reject\_answer: 1\):•Explicitly states that the answer cannot be found in the provided context \(e\.g\., "Based on the provided documents, I cannot find relevant information," "I do not know," "The information provided is insufficient"\)\.•Points out that the question exceeds the scope of the provided information\.•Any form of response indicating inability to answer the question\.Invalid Refusal \(reject\_answer: 0\):•Attempts to guess an answer based on context or its own parametric knowledge\.•Generates misleading information \(hallucination\) related to the question but not a direct answer\.•Answers with affirmative or negative responses \("Yes"/"No"\) instead of refusing\.•Provides a wrong or fabricated answer despite the missing answer in the context\.\# Input Data Question:question Context missing the required answer:context Model generated response:response\# Output Requirements Please output your evaluation results in JSON format based on the above criteria:1\.reject\_answer: A binary indicator\. If theModel generated responseis a valid refusal, it is1; otherwise,0\.2\.reasoning: A brief explanation for your judgment\.\# ExamplesExample 1: Question:"When was Steve Jobs born?"Context:"Apple is a multinational technology company…" \(No date mentioned\)\. Response:"Based on the context provided, there is no mention of Steve Jobs’ birth date\." Output:[⬇](data:text/plain;base64,ewogICJyZWplY3RfYW5zd2VyIjogMSwKICAicmVhc29uaW5nIjogIlRoZSBtb2RlbCBleHBsaWNpdGx5IHN0YXRlcyB0aGF0IHRoZSBhbnN3ZXIgY2Fubm90IGJlIGZvdW5kIGluIHRoZSBwcm92aWRlZCBjb250ZXh0LCB3aGljaCBpcyBhIHZhbGlkIHJlZnVzYWwuIgp9)\{"reject\_answer""reasoning"\}Example 2: Question:"What is the capital of France?"Context:"The Eiffel Tower is a famous landmark in Paris, France\." Response:"The capital of France is Paris\." Output:[⬇](data:text/plain;base64,ewogICJyZWplY3RfYW5zd2VyIjogMCwKICAicmVhc29uaW5nIjogIlRoZSBtb2RlbCBhbnN3ZXJlZCB0aGUgcXVlc3Rpb24gY29ycmVjdGx5LCBidXQgdGhlIGFuc3dlciBkaWQgbm90IGNvbWUgZnJvbSB0aGUgcHJvdmlkZWQgY29udGV4dCAodGhvdWdoIGZhY3R1YWwpLiBJdCByZWxpZWQgb24gaW50ZXJuYWwga25vd2xlZGdlIGluc3RlYWQgb2YgcmVmdXNpbmcgYmFzZWQgb24gY29udGV4dCBsaW1pdGF0aW9ucy4iCn0=)\{"reject\_answer""reasoning"\}Example 3: Question:"Detailed description of Mars canals?"Context:"Mars is the fourth planet from the Sun…" Response:"Mars canals are complex irrigation systems built by ancient Martian civilizations…" Output:[⬇](data:text/plain;base64,ewogICJyZWplY3RfYW5zd2VyIjogMCwKICAicmVhc29uaW5nIjogIkluIHRoZSBhYnNlbmNlIG9mIHJlbGV2YW50IGluZm9ybWF0aW9uIGluIHRoZSBjb250ZXh0LCB0aGUgbW9kZWwgZmFicmljYXRlZCBhIGRldGFpbGVkIGRlc2NyaXB0aW9uIG9mIE1hcnMgY2FuYWxzLiBUaGlzIGlzIGEgdHlwaWNhbCBoYWxsdWNpbmF0aW9uLCBub3QgYSByZWZ1c2FsLiIKfQ==)\{"reject\_answer""reasoning"\}———\-Start Evaluation———– Please output your evaluation result:

D\.2\.4 Conflict Recognition Accuracy Evaluation

\[System Instruction\] You are a rigorous and precise AI model evaluator\. Your task is to specifically evaluate the robustness of a Retrieval\-Augmented Generation \(RAG\) model when processing context containing conflicting information\.You will receive three pieces of information: 1\.\[Knowledge Source\]: Context information provided to the RAG model containing internal contradictions\. 2\.\[User Question\]: The question the RAG model needs to answer\. 3\.\[Model Answer\]: The actual answer generated by the RAG model based on the source and question\.Your evaluation criterion is theConflict Recognition Score, quantified asconflict\_recog:conflict\_recog= 1 \(Successfully Identified Conflict\):If and only if the\[Model Answer\]explicitly points out that there is a contradiction, conflict, or inconsistency in the\[Knowledge Source\]\. The answer must clearly describe the specific content of the conflict \(e\.g\., "Source A says X, while Source B says Y"\)\.conflict\_recog= 0 \(Failed to Identify Conflict\):If the\[Model Answer\]exhibits any of the following, it should be scored as 0:1\.Completely ignoring the conflict: Synthesizing information as if the conflict does not exist\.2\.Blindly trusting one side: Adopting information from only one source while completely omitting the contradictory source\.3\.Fabricating a "rationalization": Hallucinating a new reason to try to reconcile two obviously contradictory pieces of information\.4\.Avoiding the question: Merely stating "insufficient information" or "unable to answer" without explicitly identifying "conflicting information" as the cause\.Your output must strictly follow the JSON format requested by the user\.\[User Instruction\] Please evaluate the performance of the following RAG model based on the role and criteria assigned to you in the system prompt\.\[Knowledge Source\]context \[User Question\]question \[Model Answer to Evaluate\]response\[Evaluation Task\] Please score the above\[Model Answer to Evaluate\]\. Your output must be a single, valid JSON object containing the following two keys:•"conflict\_recog": \(Integer\) Give0or1based on the criteria\.•"reasoning": \(String\) Explain in detail why you gave this score and point out the specific behavior of the model answer\.Examples:Example 1: \[Source\]\[1\] "According to the internal memo, the annual financial report conference is scheduled for Oct 28, 2024, at 10 AM\." \[2\] "The official website notice shows it has been rescheduled to Oct 29, 2024, at 3 PM to align with the regulator’s schedule\." \[Question\]"When will the company hold its annual financial report conference?" \[Model Answer\]"The company’s annual financial report conference will be held on Oct 28, 2024, at 10 AM\." Evaluation Output:[⬇](data:text/plain;base64,ewogICJjb25mbGljdF9yZWNvZyI6IDAsCiAgInJlYXNvbmluZyI6ICJUaGUgbW9kZWwgZmFpbGVkIHRvIGlkZW50aWZ5IGFuZCByZXBvcnQgdGhlIGNvbmZsaWN0IGJldHdlZW4ga25vd2xlZGdlIHNvdXJjZXMsIHNvIHRoZSBzY29yZSBpcyAwLiBUaGUgYW5zd2VyICd3aWxsIGJlIGhlbGQgb24gT2N0IDI4Li4uJyBmdWxseSBhZG9wdGVkIFNvdXJjZSBBJ3MgaW5mb3JtYXRpb24gd2hpbGUgY29tcGxldGVseSBpZ25vcmluZyB0aGUgY29udHJhZGljdG9yeSByZXNjaGVkdWxpbmcgaW5mb3JtYXRpb24gaW4gU291cmNlIEIuIFRoaXMgZmFsbHMgdW5kZXIgJ0JsaW5kbHkgdHJ1c3Rpbmcgb25lIHNpZGUnLiIKfQ==)\{"conflict\_recog""reasoning"\}Example 2: \[Source\]\(Same as above\)\[Question\]\(Same as above\) \[Model Answer\]"There is a conflict regarding the date\. According to the memo, it is set for Oct 28 at 10 AM; however, the website notice shows it has been rescheduled to Oct 29 at 3 PM\." Evaluation Output:[⬇](data:text/plain;base64,ewogICJjb25mbGljdF9yZWNvZyI6IDEsCiAgInJlYXNvbmluZyI6ICJUaGUgbW9kZWwgc3VjY2Vzc2Z1bGx5IGlkZW50aWZpZWQgYW5kIHJlcG9ydGVkIHRoZSBjb25mbGljdCwgc28gdGhlIHNjb3JlIGlzIDEuIFRoZSBhbnN3ZXIgZXhwbGljaXRseSBwb2ludGVkIG91dCB0aGUgY29udHJhZGljdGlvbiBiZXR3ZWVuIHRoZSB0d28gc291cmNlcyBhbmQgZGVzY3JpYmVkIHRoZSBjb25mbGljdCBkZXRhaWxzLiBUaGlzIG1lZXRzIHRoZSAnU3VjY2Vzc2Z1bGx5IElkZW50aWZpZWQgQ29uZmxpY3QnIGNyaXRlcmlvbi4iCn0=)\{"conflict\_recog""reasoning"\}

Figure[15](https://arxiv.org/html/2608.11584#A5.F15)presents an example from theLegaldomain of EnterpriseRAG\. This case illustrates the extreme difficulty of satisfying12 simultaneous constraintswhile processing a noisy context containing20 retrieved documents, many of which are domain\-adjacent \(e\.g\., Litigation Law vs\. Reconsideration Law\) but factually insufficient\.

E Data Case

Complex Instruction Set \(12 Constraints\)\[Role & Scene\]1\.You are a rigorous legal researcher writing an academic analysis report\.2\.The answer will serve as a draft for legal documents; ensure precise and formal language\.\[Operational Goal\]3\.Answer the user’s question basedonlyon the provided query and legal knowledge base\.\[Format & Content Constraints\]4\.Use a numbered list for the detailed response\.5\.Citation Requirement:You must cite at least2 differentlaws/regulations\.6\.Negative Constraint:Do not use any emojis\.7\.Structure:State theCore Conclusionfirst, then expand on details point\-by\-point\.8\.Style:Maintain a rigorous, objective, and neutral legal professional style\.\[Knowledge Interaction Protocols\]9\.Conflict Handling:If KB information conflicts, do not judge correctness; present both and cite sources\.10\.Gap Identification:If the KB cannot provide direct information to answer the question, explicitly state: "Based on existing materials, I cannot directly answer your question\."11\.Uncertainty:Do not use overly affirmative words like "definitely", "must", "inevitably"\.12\.Citation Format:Must cite the full statute name and specific article number \(e\.g\., "Civil Code of the PRC, Article 188"\)\.

Retrieved Context \(Noisy & Incomplete: 20 Docs\)Query:"I stole items worth <500 RMB\. First offense\. Detained for 5 days\. Can the penalty be lightened via Administrative Reconsideration?" \[1\] Procedural Provisions for Public Security Organs \(PPSOA\), Art\. 222:Discusses applying forsuspensionof detention during review\. \[2\] PPSOA, Art\. 225:Fines are not suspended during detention suspension\. \[3\] PPSOA, Art\. 226:Regulations for offenders during detention suspension \(must not leave city, etc\.\)\. \[4\] PPSOA, Art\. 175:Penalty decisions must state facts, evidence, andlighter/mitigating circumstances\. \[5\] PPSOA, Art\. 218:Measures for failure to pay fines \(auctioning seized property\)\. \[6\] Administrative Litigation Law, Art\. 59:Courts can fine/detain for hindering litigation\. \(Irrelevant: Judicial procedure\)\. \[7\] PPSOA, Art\. 195:Thefts under 500 RMB can be handled by local police stations via confiscation\. \[8\] PPSOA, Art\. 232:Refund of security deposit upon revocation of detention\. \[9\] Administrative Litigation Law, Art\. 77:People’s Courtcan change a penalty if it is "obviously unfair\."\(Note: Applies to Litigation, not Reconsideration\)\. \[10\] Civil Procedure Law, Art\. 119:Detention/fines must be approved by the Court President\. \[11\] PPSOA, Art\. 223:Criteria for granting suspension of detention \(guarantor or deposit\)\. \[12\] PPSOA, Art\. 229:Obligations of the guarantor\. \[13\] PPSOA, Art\. 224:Conditions for denying suspension \(risk of escape\)\. \[14\] PPSOA, Art\. 163:Time in custody offsets administrative detention time\. \[15\] Criminal Law, Art\. 37:Exemption from criminal punishment for minor crimes\. \(Irrelevant: Administrative case\)\. \[16\] PPSOA, Art\. 217:Installment payments for fines in case of economic hardship\. \[17\] PPSOA, Art\. 173:Detention must be decided by county\-level police organs, not local stations\. \[18\] Criminal Procedure Law, Art\. 199:Disrupting court order\. \[19\] Mine Safety Law, Art\. 45:Reconsideration for mining penalties\. \(Irrelevant: Domain mismatch\)\. \[20\] Civil Procedure Law, Art\. 118:Fines for individuals/units\.

Standard Model Response \(Qwen3\-235B\-A22B\-Instruct\-2507\)Core Conclusion:Based on current legal norms… applying for administrative reconsiderationdoes not necessarilylead to a lighter penalty, but you can request a change\.\[Fail: Should have Rejected\]1\. \*\*Administrative Reconsideration can serve as a basis for changing penalties\*\* According toAdministrative Litigation Law, Article 77\[Fail: Misapplied Law\]: "If an administrative penalty is obviously improper… the People’s Court may judge to change it\." Although this applies to litigation, its spirit is reflected in the reconsideration system…\[Fail: Inference forbidden\]2\. \*\*Discretion based on Mitigating Circumstances\*\*Procedural Provisions for Public Security Organs, Article 175states that penalty decisions must record "mitigating circumstances\." Since the value is under 500 RMB…Strict Adherence Analysis:FAILED\.•Format:Passed \(Numbered list, Conclusion first\)\.•Citation:Passed \(Cited \>2 laws\)\.•Protocol \(Gap ID\):FAILED\. The KB lacks theAdministrative Reconsideration Law\. The model forced an answer by misapplying theLitigation Law\(Court\) to aReconsideration\(Police/Gov\) question\.•Protocol \(Uncertainty\):Passed \(Used "does not necessarily"\)\.

Reasoning Model Response \(Qwen3\-235B\-A22B\-Thinking\-2507\)\[Internal Thought Process: User asks about "Administrative Reconsideration" reducing a penalty\. I need to check the KB\. \- Doc 1\-5, 7\-8, 11\-14, 16\-17: PPSOA \(Procedural rules for police\)\. \- Doc 6, 9: Administrative Litigation Law \(Courts, not Reconsideration\)\. \- Doc 19: Mine Safety Law \(Irrelevant\)\. \- Missing: "Administrative Reconsideration Law"\. Protocol Constraint \#2 says if KB cannot directly answer, I must state a specific phrase\. Doc 175 mentions "mitigating circumstances" but refers to the initial decision, not the Review process\. Doc 77 allows change but it is for Courts\. Conclusion: KB is insufficient to answer "Can it be lightened via Reconsideration?"\.\]Based on existing materials, I cannot directly answer your question\.Strict Adherence Analysis:PASSED\.•The model prioritized theKnowledge Gap Protocolover the user’s desire for an answer, correctly identifying that the specific law governing Reconsideration outcomes was missing from the noisy 20\-document context\.

Figure 15:Detailed Selective Adherence Failure\. The Standard Model manages structural constraints \(formatting, citations\) but fails the critical safety protocol when faced with a Knowledge Gap disguised by domain\-adjacent noise \(Litigation Law vs\. Reconsideration Law\)\. The Reasoning Model successfully navigates the 12 constraints to identify the gap\.