Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy

# Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy
Source: [https://arxiv.org/html/2610.29338](https://arxiv.org/html/2610.29338)
Michael KuoKyana PollasAffiliation:PolicingLab, Miami, Florida, United StatesEmail:[\{baker,kuo,pollas\}@policinglab\.org](mailto:)

###### Abstract

Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections*conflict*, are*redundant*, are merely*similar*, or are*different*\. We describe the system and compare two architectures\.Retrieve\-Judge, a retrieve\-then\-classify pipeline, runs as a cheap library\-wide health signal;Screen\-Confirm, a screen\-then\-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another\. On a 302\-pair benchmark drawn from one department’s ~84 real policy documents, given each pair Screen\-Confirm reaches 0\.795 aggregate recall and 0\.885 on Conflict, so current models judge well\. Finding the pair is the hard part: Retrieve\-Judge’s composed recall falls to 0\.457, and Screen\-Confirm recovers 0\.472 \(0\.615 on Conflict\)\. Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re\-checked after every revision\. A scan that returns half of them in minutes, at low cost, and can be re\-run on demand changes the economics of policy review even before recall improves\.

## 1Introduction

Police departments run on written policy: standard operating procedures, general orders, and directives, written by each agency for itself\([Sutton and Dahir, 2025](https://arxiv.org/html/2610.29338#bib.bib7)\)and edited over many years\. The result routinely contains internal tension: one order requires what another forbids, or two procedures cover the same ground in different words\. Finding them by hand is slow, and getting it wrong has real consequences in the field\.

We introduce Policy Central, a system in pilot at one U\.S\. police department with broader rollout underway, which surfaces these relationships automatically\. It classifies pairs of policy sections into one of four relationships:Conflict\(the sections cannot both be followed\),Redundancy\(they say the same thing and one is unnecessary\),Similarity\(they are related but not redundant\), orDifferent\(unrelated\)\. Conflict is the high\-value, low\-frequency case; most pairs in a real library are Different\.

Policy Central serves two operational needs with two architectures\. Given the pair, current models judge well, Conflict included; when they must find the pair themselves, both architectures lose much of that recall, in different places\. This paper reports where, and what it took to recover some of it\. Our contributions:

1. 1\.A deployed system with two architectures,Retrieve\-JudgeandScreen\-Confirm: the first a retrieve\-then\-classify pipeline run as a library\-wide health signal, the second a screen\-then\-confirm cascade run as a selective audit of the documents a reviewer chooses\.
2. 2\.A 302\-pair benchmark of real policy pairs, on which we compare both architectures and two baselines \(Llama zero\-shot, DeBERTa NLI\), given the pair and having to find it, and report the production cost and time of both modes\.

## 2Related Work

Figure 1:Policy Central, traced on one real benchmark pair \(hp\-0026, Appendix[A](https://arxiv.org/html/2610.29338#A1)\)\. One ingestion pipeline \(A\) writes three artifacts\.Retrieve\-Judge\(B\) reads only the vector index and judges each section’s top\-kkneighbours; for this pair, the partner section is the nearest neighbour\.Screen\-Confirm\(C\) reads the structural index and provisions store, never the vector index, and sweeps every section through a cheap\-screen / expensive\-confirm cascade\. Every model output shown is the scored pipeline’s actual output, abbreviated with “…”\.The problem\.Legal and regulatory text grows steadily\([Coupette et al\., 2021a](https://arxiv.org/html/2610.29338#bib.bib2)\), and with it the room for contradiction\. Policing shows it at scale: a review of over 11,000 pages of use\-of\-force policy from the 100 largest U\.S\. cities found policies varying widely in structure and length, some departments spreading their force rules across multiple sections of their general orders\([Sutton and Dahir, 2025](https://arxiv.org/html/2610.29338#bib.bib7)\), and a systematic review of NLP in policing covers no work on policy documents themselves\([Sarzaeim et al\., 2023](https://arxiv.org/html/2610.29338#bib.bib6)\)\.

Prior systems\.Approaches range from formal machinery to text\-level methods\. Knowledge\-graph systems detect conflict over a structured representation, extracted automatically from text in PoliGraph\([Cui et al\., 2023](https://arxiv.org/html/2610.29338#bib.bib3)\)or hand\-encoded as deontic axioms\([Robaldo and Pozzato, 2025](https://arxiv.org/html/2610.29338#bib.bib4)\)\. Benchmark work targets conflict directly: LegalWiz generates legal\-style documents with injected contradictions\([Mantravadi et al\., 2025](https://arxiv.org/html/2610.29338#bib.bib1)\), and Clash\-of\-Leges and a Chinese legal\-conflict dataset benchmark statutory conflict detection\([Italiani et al\., 2026](https://arxiv.org/html/2610.29338#bib.bib11);[Zhao et al\., 2026](https://arxiv.org/html/2610.29338#bib.bib12)\)\. Applied compliance systems work against an external standard rather than within one library: a procurement\-audit framework checks purchase records against compliance policy on real procurement data\([Wang et al\., 2020](https://arxiv.org/html/2610.29338#bib.bib30)\), and a maintenance\-manual retrieval system keeps every answer traceable to certified sources\([Jo, 2026](https://arxiv.org/html/2610.29338#bib.bib33)\)\. Neither judges whether two sections of one library conflict\. Duplication in the U\.S\. Code\([Coupette et al\., 2021b](https://arxiv.org/html/2610.29338#bib.bib31)\)and an open\-source NLI pipeline over U\.S\. defence policy documents\([Addison, 2021](https://arxiv.org/html/2610.29338#bib.bib32)\)measure relations directly\.

Framing\.Natural language inference is the natural framing\([Koreeda and Manning, 2021](https://arxiv.org/html/2610.29338#bib.bib9);[Yin et al\., 2021](https://arxiv.org/html/2610.29338#bib.bib10)\), adapted in Section[3\.1](https://arxiv.org/html/2610.29338#S3.SS1)\. Conflict is hard for models: even GPT\-4 is unreliable on self\-contradictions that need context to see\([Li et al\., 2024](https://arxiv.org/html/2610.29338#bib.bib13)\), and current LLMs fail to detect conflicts across clinical guidelines\([Xie and Du, 2026](https://arxiv.org/html/2610.29338#bib.bib8)\)\. Our two architectures rest on existing patterns: Screen\-Confirm is a cost cascade\([Chen et al\., 2024a](https://arxiv.org/html/2610.29338#bib.bib14);[Rabanser et al\., 2025](https://arxiv.org/html/2610.29338#bib.bib15);[Khalili et al\., 2022](https://arxiv.org/html/2610.29338#bib.bib16);[Viola and Jones, 2001](https://arxiv.org/html/2610.29338#bib.bib17)\); Retrieve\-Judge combines similarity retrieval\([Robertson and Zaragoza, 2009](https://arxiv.org/html/2610.29338#bib.bib18);[Muennighoff et al\., 2023](https://arxiv.org/html/2610.29338#bib.bib19)\)with a prompt\-based LLM judge, as in[Schumann et al\. \(2026\)](https://arxiv.org/html/2610.29338#bib.bib26)\. Retrieving by similarity alone can miss what a task needs\([Gan et al\., 2024](https://arxiv.org/html/2610.29338#bib.bib20)\), and a rule that conflicts with another need not read like it\.

## 3System Design

A reviewer picks a policy document and the system reports which sections elsewhere in the library relate to it, each pair labelled with a one\-line reason\. Two architectures produce that report\.*Retrieve\-Judge*embeds every section, retrieves its nearest neighbours, and judges each pair \(panel B of Figure[1](https://arxiv.org/html/2610.29338#S2.F1), ending in the library\-health report\)\.*Screen\-Confirm*checks every section of another document \(the*compared*document\) against every requirement of the reviewer’s \(the*anchor*\), cheaply first and carefully on what survives \(panel C, ending in the audit report\)\. Both read the artifacts that one ingestion pipeline writes \(panel A\)\.

### 3\.1Relation Taxonomy

Table[1](https://arxiv.org/html/2610.29338#S3.T1)defines the four labels with real excerpt pairs; each maps to a different reviewer action\.

Table 1:The four relation labels, each with a verbatim excerpt pair from the pilot department’s corpus that the deployed system labelled correctly \(provenance in Appendix[A](https://arxiv.org/html/2610.29338#A1)\)\.The product exists to surfaceConflict, because two sections that cannot both be followed mean the library is wrong somewhere\.Redundancymatters for upkeep, since a requirement stated twice can be cut to one copy so that future edits happen once\.Similarityexists so that a reviewer drafting or revising one policy sees the related sections elsewhere in the library, to reference or reconcile rather than delete\. Redundancy and Similarity are separated by a deletion test: if one section can go because the other carries its content, the pair is Redundant; if both must stay, it is Similar\. Similarity versus Different is likewise a question about the reviewer, not the text: two sections are Similar if a reviewer revising one would need to read the other, and Different if not, however much wording they share\.

The four labels are natural language inference \(NLI\) labels adapted to what a reviewer does next\. Building on NLI gives “contradicts” a precise meaning, connects to prior work on legal documents\([Koreeda and Manning, 2021](https://arxiv.org/html/2610.29338#bib.bib9);[Yin et al\., 2021](https://arxiv.org/html/2610.29338#bib.bib10)\), and supplies trained models as a free baseline\. The mapping is: contradiction→\\rightarrowConflict; entailment in both directions→\\rightarrowRedundancy; neutral→\\rightarrowSimilarity*or*Different, split by whether a reviewer must act\. Conflict drops NLI’s direction: NLI asks whether a premise contradicts a hypothesis, whereas two rules that cannot both be followed conflict whichever is read first\. The last line is where policy review departs from NLI, since two of our labels share one NLI label\. An off\-the\-shelf NLI model can therefore find Conflict at best and collapses the rest, which is what DeBERTa NLI shows \(Section[4\.3](https://arxiv.org/html/2610.29338#S4.SS3)\): 0\.654 Conflict recall and near zero on Redundancy and Similarity\.

### 3\.2Data Ingestion

A single pipeline \(panel A\) serves both architectures but writes*three*artifacts, each read by a different consumer\. The parse \(page\-by\-page extraction, Unicode normalisation, and outline\-aware segmentation that records each chunk’shierarchy\_path\) is shown in the figure\. Chunking is a deterministic length\-based split \(∼\{\\sim\}600 tokens\) that keeps the source text verbatim\.

The artifacts are: \(i\) astructural index: raw chunk text andhierarchy\_path, no embeddings, read by Screen\-Confirm for its*compared*document; \(ii\) adistilled provisions store, read by Screen\-Confirm for its*anchor*document; and \(iii\) avector index\(Qdrant\): each leaf chunk embedded once \(Section[3\.3](https://arxiv.org/html/2610.29338#S3.SS3)\) for similarity search, read*only*by Retrieve\-Judge\. The provisions store is built by a separate ingest\-time LLM pass \(Claude Haiku, once per document\) that rewrites a document’s chunks into one standalone sentence per requirement \(prompt in Appendix[G\.1](https://arxiv.org/html/2610.29338#A7.SS1)\), each grounded back to its source chunk, an obligation\-level analogue of proposition retrieval units\([Chen et al\., 2024b](https://arxiv.org/html/2610.29338#bib.bib35)\)and atomic facts\([Min et al\., 2023](https://arxiv.org/html/2610.29338#bib.bib36)\)\. It is cached and reused by every later scan\. A requirement it fails to extract is invisible on the anchor side; Section[4\.3](https://arxiv.org/html/2610.29338#S4.SS3)measures that loss\.

Retrieve\-Judge’s retrieval unit is the leaf chunk \(19,737 across the department’s∼84\{\\sim\}84documents, Table[2](https://arxiv.org/html/2610.29338#S4.T2)\); Screen\-Confirm judges whole sections, so the two compare different\-sized units \(Limitations\)\.

### 3\.3Retrieve\-Judge

Retrieve\-Judge \(panel B\) takes each leaf chunk of the reviewer’s document in turn: each is embedded, its top\-kknearest neighbours by embedding similarity, drawn from every document in the library, are retrieved, and each \(chunk, neighbour\) pair is judged\. Chunks are embedded with Cohere Embed v4 via Bedrock \(1024 dimensions\)\. The deployed audit retrievesk=20k\{=\}20under cosine distance and keeps neighbours above a 0\.75 similarity threshold\. The judge returns one JSON object per pair,\{summary, findings\[\{type, description\}\]\}: a label and a one\-sentence description that must name the relation \(“deadline is 4 hours in A versus 24 hours in B”\), with no reasoning trace and no quotations \(Appendix[G](https://arxiv.org/html/2610.29338#A7)\)\. Retrieval is dense\-only; a reranker, hypothetical\-document queries, BM25, embedding\-side changes, and a largerkkwere all tried and none moved recall \(Appendix[C](https://arxiv.org/html/2610.29338#A3)\), and Section[4\.3](https://arxiv.org/html/2610.29338#S4.SS3)shows that judge swaps do not either\. The constraint is structural: Conflict is a*logical*relation, not a similarity one\. Similarity surfaces contradictions that share wording: the 20\-hour and 16\-hour minimums in Figure[1](https://arxiv.org/html/2610.29338#S2.F1)are each other’s top\-1 neighbour\. It has no signal for conflicts that share no wording: a rule that narcotics officers never wear uniform off duty and one that uniform must be worn on extra\-duty jobs conflict, yet neither is in the other’s top\-20 \(a benchmark pair\)\. Retrieve\-Judge therefore samples the library rather than covering it, and we use it as a library\-health scan rather than an audit \(Limitations\)\.

### 3\.4Screen\-Confirm

Screen\-Confirm \(panel C\) replaces similarity sampling with an exhaustive two\-stage cascade: discard the Different majority cheaply, reason carefully about the survivors, the Viola–Jones intuition\([Viola and Jones, 2001](https://arxiv.org/html/2610.29338#bib.bib17)\)as a cost cascade\([Wang et al\., 2011](https://arxiv.org/html/2610.29338#bib.bib27)\)\. No model ever sees a whole PDF\. The compared document is walked one section at a time, and each section is matched against the anchor document’s*provision list*, its distilled one\-line\-per\-requirement form, rather than its raw text\. Each screen call is therefore short, and the provisions block is the same on every call, so it can be prompt\-cached\. Headings, definitions, and repeated administrative language that distillation left out are never sent to the model\.

Screen\.One Claude Haiku call per compared\-document section receives the numbered provisions list plus that section and returns a JSON array of candidate hits \(as in the figure\)\. There is no score or threshold: any hit labelled Conflict, Redundancy, or Similarity escalates that provision–section pair, and only Different does not\.

Confirm\.Each escalated pair goes to one Claude Sonnet call that sees three things: the flagged provision as its distilled sentence rather than its source chunk, the compared document’s raw section, and the screen’s label, which it may use only as a tie\-breaker\. It returns a confirmed label withquote\_aandquote\_b; a downgrade to Different drops the hit\.quote\_ais therefore drawn from the distilled sentence andquote\_bfrom the section, and neither is code\-validated against the source, so we call them model\-attributed\. Confirm does not cache \(each pair is unique\); findings render as an audit report \(panel C\), typically tens of rows for a document pair\.

A scan costsNNscreen calls plusMMconfirm calls \(NNsections,MMescalations\); distillation is a once\-per\-document ingest cost\.

## 4Evaluation

### 4\.1Benchmark and Its Construction

Our benchmark is 302 pairs \(26 Conflict, 57 Redundancy, 44 Similarity, 175 Different\), a distribution that reflects real policy libraries, where Different is the large majority\. Every pair is two real sections from the pilot department’s policies\. Labels are*semi\-automatic*rather than subject\-matter expert: a Haiku pass extracts checkable claims per document and a pure\-code stage \(no LLM\) pairs them, a*non\-Claude*judge \(GPT\-5\.6 Sol\) labels each pair, an Opus model attacks each label, and people check the disputed rows, following the extract\-then\-verify pattern of[Bahaj and Ghogho \(2026\)](https://arxiv.org/html/2610.29338#bib.bib34)\(Figure[2](https://arxiv.org/html/2610.29338#A1.F2), Appendix[A](https://arxiv.org/html/2610.29338#A1)\)\.

We do not field the mining pipeline as a detector, since scoring it on a benchmark it produced would be circular; one dependency remains, in that its claim extraction and production distillation are both Haiku passes \(Limitations\)\.

### 4\.2Systems and Methodology

We compare four systems on the 302 pairs\.Screen\-ConfirmandRetrieve\-Judgeare as described in Section[3](https://arxiv.org/html/2610.29338#S3)\.Llama zero\-shotis a single call with a four\-line prompt, no examples and no retrieval, a floor for judgment\.DeBERTa NLIis an off\-the\-shelf zero\-shot NLI checkpoint, not fine\-tuned on our data \(Appendix[G\.6](https://arxiv.org/html/2610.29338#A7.SS6)\)\.

Table[3](https://arxiv.org/html/2610.29338#S4.T3)asks two questions\. The lower block asks how well each system judges a pair it is handed; the upper block asks how much of that survives when the system must find the pair itself\. Each architecture appears in both, so the gap between its two rows is what finding costs it\. Table[2](https://arxiv.org/html/2610.29338#S4.T2)lists what each system sees in each setting\. Handed the pair, the cascade’s screen still has to pick the right provision, but out of 16 rather than out of the whole anchor, and Retrieve\-Judge’s judge simply reads the two sections\. To find the pair, Retrieve\-Judge must first retrieve it, so we estimate its recall \(RR, the fraction of gold pairs caught\) asRfind=Rretrieve×RjudgeR\_\{\\text\{find\}\}=R\_\{\\text\{retrieve\}\}\\times R\_\{\\text\{judge\}\}, whereRretrieveR\_\{\\text\{retrieve\}\}is the fraction of gold pairs whose partner is in the top 10 andRjudgeR\_\{\\text\{judge\}\}the fraction the judge labels correctly when handed the pair; for Conflict,0\.571×0\.731=0\.4180\.571\\times 0\.731=0\.418\. Multiplying assumes the two are independent \(Limitations\)\. Retrieval recall is measured on the 137 pairs whose two quotes both map to an indexed leaf chunk \(14 Conflict, 13 Redundancy, 25 Similarity\), each queried in both directions; judge recall is over every pair of the class\. Screen\-Confirm’s end\-to\-end recall is measured directly, one pass per pair, but reads the same way:Re2e=Rreach×Rscreen×RconfirmR\_\{\\text\{e2e\}\}=R\_\{\\text\{reach\}\}\\times R\_\{\\text\{screen\}\}\\times R\_\{\\text\{confirm\}\}, the fractions of gold positives that reach the screen, that the screen then escalates, and that confirm then labels correctly\. Over all classes that is0\.945×0\.742×0\.674=0\.4720\.945\\times 0\.742\\times 0\.674=0\.472\(Appendix[D](https://arxiv.org/html/2610.29338#A4)gives the counts\)\.

Table 2:What each system is shown in the two settings of Table[3](https://arxiv.org/html/2610.29338#S4.T3)\.All model calls use temperature 0 and pinned model IDs; prompts are in Appendix[G](https://arxiv.org/html/2610.29338#A7)\.

Table 3:Recall by relation class\.†Support\-weighted recall over Conflict, Redundancy, Similarity \(n=127n=127\); Different is excluded, since predicting Different for everything would score well on it while catching nothing\. Upper block: the system must find the partner section\. Retrieve\-Judge rows are retrieval recall \(R@10, over the 137 pairs whose quotes map to indexed leaf chunks, Section[4\.2](https://arxiv.org/html/2610.29338#S4.SS2)\) alone, then retrieval times judge recall for two of eight judges \(Appendix[B](https://arxiv.org/html/2610.29338#A2)\);∗measured on the production data path\. Lower block: the system is handed the pair\. Confl\., Redun\., Simil\.: Conflict, Redundancy, Similarity\.
### 4\.3Results by Relation Class

Given the pair, judgment is largely solved: a plain zero\-shot Llama reaches 0\.764 and Screen\-Confirm 0\.795, with the cascade’s gain concentrated on Similarity \(0\.682 vs\. 0\.455\), and Retrieve\-Judge’s own judge 0\.654\. Having to find the pair is what costs, and the end\-to\-end figures should be read against the alternative, a reviewer reading the library\. We did not choose this recall\. The cascade is tuned to over\-flag rather than miss, so its low precision is by design, but every missed pair is a loss, and Appendix[D](https://arxiv.org/html/2610.29338#A4)shows at which stage each one was lost\. The rest of this section reports what we tried to raise it and what did not move\. Retrieve\-Judge falls from 0\.654 to 0\.457, and the loss is retrieval: its retrieval alone recovers 0\.663, swapping its judge across eight models moves the composed figure by only about 0\.12, and replacing the dense retriever with BM25 barely moves it \(0\.455 against 0\.468 on the same pairs\)\. Screen\-Confirm falls from 0\.795 to 0\.472 and pays at its screen \(Appendix[D](https://arxiv.org/html/2610.29338#A4)\); on Conflict it keeps 0\.615, above Retrieve\-Judge’s 0\.418–0\.462\.

Read by class, the pattern is consistent\. Redundancy is the easy class everywhere: duplicated text shares wording, so retrieval reaches 0\.846 and most judges score it highest; when the cascade loses a Redundancy pair end to end, the largest single cause is confirm calling it Similarity \(8 of 25 misses\)\. Conflict is harder to find than to judge: embeddings rank by how alike two chunks are overall, and two rules that contradict each other differ at one detail while the chunk holding the partner rule often sits among unrelated requirements, so a cosine neighbourhood ranks it behind chunks that merely discuss the same topic\. Retrieval reaches only 0\.571, and end\-to\-end the cascade keeps 0\.615, with the screen the larger loss; handed the pair, the capable judges reach 0\.73 to 0\.89\. Similarity is the hard class in every setting \(retrieval 0\.480, judges 0\.43 to 0\.68, end\-to\-end 0\.273\), because it is the residual class, bounded on one side by Redundancy and on the other by Different, so every call on it is a call on two judgment boundaries at once, and confirm’s errors go both ways \(5 Similarity pairs read as Redundancy, 8 as Conflict\)\.

Swapping the confirm model did not help: every alternative lowered Conflict recall or lost the ability to walk a false escalation back to Different \(Table[6](https://arxiv.org/html/2610.29338#A2.T6), Appendix[B](https://arxiv.org/html/2610.29338#A2)\)\.

### 4\.4Time and Cost on the Department’s Library

Table[4](https://arxiv.org/html/2610.29338#S4.T4)gives what the two modes cost on the 78 of the department’s 84 documents that were indexed when these runs were made \(19,646 sections, Screen\-Confirm’s unit\), from production run records and Bedrock’s token metrics\. Retrieve\-Judge has audited the whole library: one pass judges the roughly 4,800 neighbour pairs above the 0\.75 threshold for $13 in five minutes, with the judge’s static prompt cached \(8\.9M of 10\.4M input tokens were cache reads\)\. Screen\-Confirm has run only as targeted scans of one document against one or two others, at list price because a provision store that small is below the screen model’s cacheable minimum; scaled by sections, one SOP against the whole library would take about an hour at ten workers and cost $40 to $80\. Full\-library scans are not the intended use: in the pilot, reviewers told us their main use is comparing a document against the few documents they are revising or need to align, which is the targeted\-scan case in Table[4](https://arxiv.org/html/2610.29338#S4.T4)\. Cheaper models would cut cost further: zero\-shot Llama 3\.3 70B scores 0\.764 given the pair against Screen\-Confirm’s 0\.795 at about one two\-hundredth of the cost \(Appendix[E](https://arxiv.org/html/2610.29338#A5)\)\. The two modes therefore serve different cases: the thresholded audit is the cheap library\-wide signal, the exhaustive scan the per\-pair check, and at today’s recall either gives a reviewer a starting point that reading the library does not\.

Table 4:Cost and wall\-clock of production scans on the deployment \(ten scan workers\)\. Calls are screen \+ confirm for Screen\-Confirm and judge calls for Retrieve\-Judge; times are from the run records; costs are Bedrock’s logged token counts for each run at list rate, cache reads at the cache rate\. The projected row scales the measured per\-section cost and time; it is an upper bound, not the intended use\.

## 5Conclusion and Future Work

Finding rules that contradict or repeat one another is currently a manual review task\. Policy Central does not replace that review, but it gives the reviewer a report of candidate pairs to start from\. Conflict is a logical relation, not a similarity, so a retrieve\-then\-judge pipeline has a ceiling that none of the judge or retriever swaps we tried lifts\. Handed a pair, current models label it well, and the losses come from reaching the pair\. An exhaustive cascade over distilled requirements fits the operational need: it trades precision for recall, since a reviewer dismisses a false flag quickly but never sees a missed one; its cost is linear in the number of sections, low enough to re\-run after each edit; and it reports on a document pair in minutes, so a reviewer can fix and re\-check in one sitting\.

In deployment, the next steps are improving finding quality against reviewer feedback from the pilot, extending the rollout under way to further departments, and reporting usage data\. Ingestion is the main barrier to other corpora: the parser is pattern\-based and tuned to one department’s directive formats, and layout\-aware document models\([Xu et al\., 2020](https://arxiv.org/html/2610.29338#bib.bib22);[Kim et al\., 2022](https://arxiv.org/html/2610.29338#bib.bib23)\)are one possible route to ingesting any agency’s PDFs unchanged\. Beyond policing, other rule\-bound domains report the same inconsistency problem: clinical guidelines\([Bowles et al\., 2019](https://arxiv.org/html/2610.29338#bib.bib28)\), contracts\([Koc and Gurgun, 2022](https://arxiv.org/html/2610.29338#bib.bib29)\), and immigration rules\([Vigneswaran and de León, 2023](https://arxiv.org/html/2610.29338#bib.bib5)\)\. Recall matters most for a possible step after detection: an agent that runs the scan, resolves the clear cases, and brings the reviewer the edge cases, an extension that Ethical Considerations flags for separate review\. Rewrites reopen evaluation, since a proposed text must be graded for grammaticality, naturalness, and completeness as in the QA2D evaluation of[Demszky et al\. \(2018\)](https://arxiv.org/html/2610.29338#bib.bib37)\.

## Limitations

Benchmark\.The most important limitation is label provenance: the 302 pairs are real, but their labels are semi\-automatic, not subject\-matter expert\. Our SME annotators did label an earlier pool, whose underlying pairs turned out too narrowly distributed to report; the annotators and protocol are in place and labelling this benchmark is the next step\. We have no inter\-annotator agreement figure for this benchmark\. The label set has no “partly redundant” class, so pairs that share only part of their content are forced to Redundancy or Similarity; a fifth label would only move that boundary\. The benchmark is also one department’s library: 302 pairs from a single corpus of about 84 documents, with 26 Conflicts, so the Conflict figures carry wide intervals \(Appendix[F](https://arxiv.org/html/2610.29338#A6)\) and transfer to other departments’ formats and drafting conventions is untested\. The end\-to\-end Conflict recall of 0\.615 is a reported finding about the production path, not a property of the models; Appendix[D](https://arxiv.org/html/2610.29338#A4)attributes the shortfall stage by stage\.

Development and evaluation share one benchmark\.The definition fix, the prompt iterations and two design fixes of Appendix[D](https://arxiv.org/html/2610.29338#A4)were found and measured on the same 302 pairs, with no held\-out split\. Absolute recall is therefore optimistic, and the number to trust is the direction of each change rather than its size\. The model\-swap and retriever ablations vary one component with everything else fixed, so they are relative comparisons and less exposed\. Every model\-swap result was obtained under prompts tuned against Sonnet 4\.5, so swaps measure prompt transfer as much as model capability\. The SME labelling above will provide the held\-out set\.

Measurement\.Retrieve\-Judge judges leaf chunks and Screen\-Confirm whole sections, so unit size is an uncontrolled variable in the comparison\. Parsing errors cascade\([Zhang et al\., 2024](https://arxiv.org/html/2610.29338#bib.bib21)\), and here they are silent: a wrong section boundary changes what confirm reads and can turn a correct judgment into a wrong one with no trace in the output\. Our stage\-by\-stage attribution of misses \(Appendix[D](https://arxiv.org/html/2610.29338#A4)\) checks that gold text is present in the index and that a section can be located, not that boundaries are right, and we report no boundary\-level accuracy for the parser; the three ingest faults were found only because that attribution placed misses before any model call\. The cascade itself is a design choice with a counter\-argument: escalation pays only when the screen’s decision to escalate carries information about whether confirm will be right\. Our screen escalates 36 to 71% of sections in the end\-to\-end run depending on anchor size, and the gain we measure over a single pass is in recall, not in cost per correct finding, which we have not measured\. Distillation recall is a lower bound: we ask whether each gold quote is contained in some provision, a token\-overlap test, so a requirement the distiller restated in its own words counts as missed\. The misses we did see are mostly descriptive statements that a requirement\-extraction prompt does not treat as rules, such as*“Sworn personnel work ten hours a day, four days a week”*, a Redundancy target in the benchmark that never became a provision\. Confirm also judges the distilled sentence rather than the source chunk it was distilled from, whereas the given\-pair test handed it source text; each provision retains its chunk id, so passing the chunk to confirm is a one\-line change that would also makequote\_averbatim from the document, and we have not yet measured it\.

What the benchmark cannot see\.Benchmark candidates were mined by a Claude Haiku pass, and production distillation is also a Haiku pass, with separate code and prompts\. A kind of requirement that Haiku reliably fails to extract, for instance one stated as a schedule or in a table rather than as a “shall”, would be missing from the benchmark and from the provisions store alike, so the benchmark would score the system as complete where it is not\. Candidates were also paired by shared terms, so the benchmark contains no pair with zero lexical overlap, and Retrieve\-Judge’s retrieval gap is measured on pairs that do share wording\. And because the benchmark is enriched for the rare classes, precision will be lower in deployment, while recall, the figure we report, does not depend on how rare a class is \(Appendix[F](https://arxiv.org/html/2610.29338#A6)works the numbers\)\.

Retrieval churn\.A distinct limitation concerns using Retrieve\-Judge*repeatedly*as a library\-health check\. Because its top\-kkretrieval is corpus\-state\-dependent, re\-running an audit after document edits changes the embeddings and therefore the neighbor sets: fixed pairs can drop out before their resolution is confirmed, genuinely new pairs surface, and the reported issue count can stay flat or even rise despite real improvement, purely from retrieval churn rather than any statement about the underlying corpus\. This is a known property to design around, and it is the main reason we route iterative, check\-your\-work usage to Screen\-Confirm, whose exhaustive per\-section screen re\-checks the full space on every run and so drops reliably as issues are fixed\.

Composed estimates\.Retrieve\-Judge’s find\-the\-pair figures are a product of two separately measured quantities, retrieval recall and judge recall given retrieval, which assumes the two difficulties are independent for a given pair, a simplification prior RAG work has named\([Zamani and Bendersky, 2024](https://arxiv.org/html/2610.29338#bib.bib25)\)\. If they are correlated, for instance if ambiguous pairs are both harder to retrieve and harder to judge, the true figure could sit on either side of the estimate, and retrieval metrics are known to track downstream performance only weakly\([Salemi and Zamani, 2024](https://arxiv.org/html/2610.29338#bib.bib24)\)\. The retrieval factor is measured only on the 137 pairs whose quotes could be mapped to indexed leaf chunks, so the composition applies a subset’s retrieval rate to the whole class’s judge rate\. The estimate also uses recall at 10 with no score threshold, whereas the production audit keeps only neighbours above 0\.75 cosine, so it is an upper bound on what the deployed audit retrieves\. Measuring Retrieve\-Judge end\-to\-end in one pass, as we do for Screen\-Confirm, is the next measurement step\.

## Ethical Considerations

Policy Central operates on police policy, a sensitive domain\. The system is advisory: it surfaces candidate relationships for human policy reviewers and never edits, approves, or enforces policy\. Whether two policies in fact conflict, and what to do about it, remains a determination for the agency’s policy and legal personnel; the system only proposes\. Every finding carries model\-attributed quotes so a reviewer can check it against the source text, and reviewers are told that Conflict findings are low\-precision \(0\.404\) by design\. The failure mode that matters more is the false negative: on the production path the system recovers 0\.615 of benchmark Conflicts, so a clean report is not evidence that a library is clean, and the tool is presented to departments as a complement to review, not a certification of it\. The policies are public records of the department, and the department reviewed Policy Central’s output and gave feedback throughout the pilot\. Policy text is processed within the operating organisation’s own AWS environment\. The only policy text reproduced here is the short excerpts in Table[1](https://arxiv.org/html/2610.29338#S3.T1), Figure[1](https://arxiv.org/html/2610.29338#S2.F1), and Appendix[A](https://arxiv.org/html/2610.29338#A1), with document identifiers removed\. The benchmark was labeled semi\-automatically rather than by subject\-matter experts, so its labels are not an authoritative reading of any policy\. Finally, a tool that flags policy conflicts could be misread as evidence about an individual officer’s compliance; the system is scoped to document review and is neither designed nor validated for adjudicating individual conduct\. Systems built for a narrow advisory purpose tend to expand into adjacent decision areas without a corresponding reassessment, so any extension of Policy Central toward proposing or applying fixes, or toward enforcement\-relevant determinations, should trigger a fresh risk assessment rather than inherit this one\.

## References

- Addison \(2021\)P\. AddisonPolicy\-contradictions\-nlp: pipeline to detect contradictions in policy documents using NLP and transformers\.Note:GitHub repositoryExternal Links:[Link](https://github.com/parkeraddison/policy-contradictions-nlp)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Bahaj and Ghogho \(2026\)A\. Bahaj and M\. GhoghoMizanQA: a benchmark for multi\-answer Moroccan legal QA\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 5: Industry Track\),Rabat, Morocco,pp\. 132–144\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-industry.10),[Link](https://aclanthology.org/2026.eacl-industry.10/)Cited by:[§4\.1](https://arxiv.org/html/2610.29338#S4.SS1.p1.1)\.
- Bowleset al\.\(2019\)J\. Bowles, M\. B\. Caminati, S\. Cha, and J\. MendozaA framework for automated conflict detection and resolution in medical guidelines\.Science of Computer Programming182,pp\. 42–63\.External Links:[Document](https://dx.doi.org/10.1016/j.scico.2019.07.002),[Link](https://doi.org/10.1016/j.scico.2019.07.002)Cited by:[§5](https://arxiv.org/html/2610.29338#S5.p2.1)\.
- Chenet al\.\(2024a\)L\. Chen, M\. Zaharia, and J\. ZouFrugalGPT: how to use large language models while reducing cost and improving performance\.Transactions on Machine Learning Research\.Note:Preprint: arXiv:2305\.05176External Links:2305\.05176,[Link](https://openreview.net/forum?id=cSimKw5p6R)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Chenet al\.\(2024b\)T\. Chen, H\. Wang, S\. Chen, W\. Yu, K\. Ma, X\. Zhao, H\. Zhang, and D\. YuDense X retrieval: what retrieval granularity should we use?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.845)Cited by:[§3\.2](https://arxiv.org/html/2610.29338#S3.SS2.p2.1)\.
- Coupetteet al\.\(2021a\)C\. Coupette, J\. Beckedorf, D\. Hartung, M\. Bommarito, and D\. M\. KatzMeasuring law over time: a network analytical framework with an application to statutes and regulations in the united states and germany\.Frontiers in Physics9\.External Links:[Document](https://dx.doi.org/10.3389/fphy.2021.658463),[Link](https://arxiv.org/abs/2101.11284)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p1.1)\.
- Coupetteet al\.\(2021b\)C\. Coupette, J\. Singh, and H\. SpamannSimplify your law: using information theory to deduplicate legal documents\.In2021 International Conference on Data Mining Workshops \(ICDMW\),pp\. 631–638\.External Links:[Document](https://dx.doi.org/10.1109/ICDMW53433.2021.00083),[Link](https://arxiv.org/abs/2110.00735)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Cuiet al\.\(2023\)H\. Cui, R\. Trimananda, A\. Markopoulou, and S\. JordanPoliGraph: automated privacy policy analysis using knowledge graphs\.InProceedings of the 32nd USENIX Security Symposium \(USENIX Security 23\),External Links:[Link](https://www.usenix.org/conference/usenixsecurity23/presentation/cui)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Demszkyet al\.\(2018\)D\. Demszky, K\. Guu, and P\. LiangTransforming question answering datasets into natural language inference datasets\.Note:arXiv preprint arXiv:1809\.02922External Links:[Link](https://arxiv.org/abs/1809.02922)Cited by:[§5](https://arxiv.org/html/2610.29338#S5.p2.1)\.
- Ganet al\.\(2024\)C\. Gan, D\. Yang, B\. Hu, H\. Zhang, S\. Li, Z\. Liu, Y\. Shen, L\. Ju, Z\. Zhang, J\. Gu, L\. Liang, and J\. ZhouSimilarity is not all you need: endowing retrieval augmented generation with multi layered thoughts\.External Links:2405\.19893,[Link](https://arxiv.org/abs/2405.19893)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Italianiet al\.\(2026\)P\. Italiani, G\. Moro, and L\. RagazziClash\-of\-leges: a bilingual dataset for conflict detection and explanation in statutory law\.Expert Systems with Applications300,pp\. 130182\.External Links:[Document](https://dx.doi.org/10.1016/j.eswa.2025.130182)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Jo \(2026\)B\. JoA compliance\-preserving retrieval system for aircraft MRO task search\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 5: Industry Track\),Rabat, Morocco,pp\. 317–329\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-industry.23),[Link](https://aclanthology.org/2026.eacl-industry.23/)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Khaliliet al\.\(2022\)L\. Khalili, Y\. You, and J\. BohannonBabyBear: cheap inference triage for expensive language models\.arXiv preprint arXiv:2205\.11747\.External Links:[Link](https://arxiv.org/abs/2205.11747)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Kimet al\.\(2022\)G\. Kim, T\. Hong, M\. Yim, J\. Nam, J\. Park, J\. Yim, W\. Hwang, S\. Yun, D\. Han, and S\. ParkOCR\-free document understanding transformer\.InComputer Vision – ECCV 2022,External Links:[Document](https://dx.doi.org/10.1007/978-3-031-19815-1%5F29),[Link](https://arxiv.org/abs/2111.15664)Cited by:[§5](https://arxiv.org/html/2610.29338#S5.p2.1)\.
- Koc and Gurgun \(2022\)K\. Koc and A\. P\. GurgunAmbiguity factors in construction contracts entailing conflicts\.Engineering, Construction and Architectural Management29\(5\),pp\. 1946–1964\.External Links:[Document](https://dx.doi.org/10.1108/ECAM-04-2020-0254),[Link](https://doi.org/10.1108/ECAM-04-2020-0254)Cited by:[§5](https://arxiv.org/html/2610.29338#S5.p2.1)\.
- Koreeda and Manning \(2021\)Y\. Koreeda and C\. D\. ManningContractNLI: a dataset for document\-level natural language inference for contracts\.InFindings of the Association for Computational Linguistics: EMNLP 2021,pp\. 1907–1919\.External Links:[Link](https://aclanthology.org/2021.findings-emnlp.164/)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1),[§3\.1](https://arxiv.org/html/2610.29338#S3.SS1.p3.1)\.
- Liet al\.\(2024\)J\. Li, V\. Raheja, and D\. KumarContraDoc: understanding self\-contradictions in documents with large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6509–6523\.External Links:[Link](https://aclanthology.org/2024.naacl-long.362)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Mantravadiet al\.\(2025\)A\. Mantravadi, S\. Dalmia, O\. Pospelova, A\. Mukherji, N\. Dave, and A\. MittalLegalWiz: a multi\-agent generation framework for contradiction detection in legal documents\.External Links:2510\.03418,[Link](https://arxiv.org/abs/2510.03418)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.741)Cited by:[§3\.2](https://arxiv.org/html/2610.29338#S3.SS2.p2.1)\.
- Muennighoffet al\.\(2023\)N\. Muennighoff, N\. Tazi, L\. Magne, and N\. ReimersMTEB: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics \(EACL 2023\),Dubrovnik, Croatia,pp\. 2014–2037\.External Links:[Link](https://aclanthology.org/2023.eacl-main.148/)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Rabanseret al\.\(2025\)S\. Rabanser, N\. Rauschmayr, A\. Kulshrestha, P\. Poklukar, W\. Jitkrittum, S\. Augenstein, C\. Wang, and F\. TombariGatekeeper: improving model cascades through confidence tuning\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS 2025\),External Links:2502\.19335,[Link](https://arxiv.org/abs/2502.19335)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Robaldo and Pozzato \(2025\)L\. Robaldo and G\. L\. PozzatoHandling irresolvable conflicts in the semantic web: an RDF\-based conflict\-tolerant version of the deontic traditional scheme\.Journal of Logic and Computation35\(8\),pp\. exaf054\.External Links:[Document](https://dx.doi.org/10.1093/logcom/exaf054),[Link](https://doi.org/10.1093/logcom/exaf054)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Robertson and Zaragoza \(2009\)S\. E\. Robertson and H\. ZaragozaThe probabilistic relevance framework: bm25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.External Links:[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Salemi and Zamani \(2024\)A\. Salemi and H\. ZamaniEvaluating retrieval quality in retrieval\-augmented generation\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR ’24\),pp\. 2395–2400\.External Links:[Document](https://dx.doi.org/10.1145/3626772.3657957),[Link](https://arxiv.org/abs/2404.13781)Cited by:[Limitations](https://arxiv.org/html/2610.29338#Sx1.p6.1)\.
- Sarzaeimet al\.\(2023\)P\. Sarzaeim, Q\. H\. Mahmoud, A\. Azim, G\. Bauer, and I\. BowlesA systematic review of using machine learning and natural language processing in smart policing\.Computers12\(12\),pp\. 255\.External Links:[Document](https://dx.doi.org/10.3390/computers12120255),[Link](https://www.mdpi.com/2073-431X/12/12/255)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p1.1)\.
- Schumannet al\.\(2026\)G\. Schumann, J\. Marx Gómez, and H\. PargmannDetection of contradictions and inconsistencies in regulatory documents using prompt\-engineering\.InProceedings of the 59th Hawaii International Conference on System Sciences \(HICSS\-59\),pp\. 1766–1775\.External Links:[Link](https://hdl.handle.net/10125/111606)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Sutton and Dahir \(2025\)D\. Sutton and F\. DahirPolice use of force policies across america\.Technical reportStanford Center for Racial Justice, Stanford Law School\.Note:Analysis of 11,000 pages of policy documents and 2,200 policy provisions across the 100 largest U\.S\. citiesExternal Links:[Link](https://law.stanford.edu/wp-content/uploads/2025/06/Police-Use-of-Force-Policies-Across-America-1.pdf)Cited by:[§1](https://arxiv.org/html/2610.29338#S1.p1.1),[§2](https://arxiv.org/html/2610.29338#S2.p1.1)\.
- Vigneswaran and de León \(2023\)D\. Vigneswaran and E\. de LeónThe inconsistency of immigration policy: the limits of “top\-down” approaches\.Ethnic and Racial Studies47\(10\),pp\. 2060–2084\.External Links:[Document](https://dx.doi.org/10.1080/01419870.2023.2263076),[Link](https://doi.org/10.1080/01419870.2023.2263076)Cited by:[§5](https://arxiv.org/html/2610.29338#S5.p2.1)\.
- Viola and Jones \(2001\)P\. Viola and M\. J\. JonesRapid object detection using a boosted cascade of simple features\.InProceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.1,pp\. I–511–I–518\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2001.990517)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1),[§3\.4](https://arxiv.org/html/2610.29338#S3.SS4.p1.1)\.
- Wanget al\.\(2020\)K\. Wang, M\. Zipperle, M\. Becherer, F\. Gottwalt, and Y\. ZhangAn AI\-based automated continuous compliance awareness framework \(CoCAF\) for procurement auditing\.Big Data and Cognitive Computing4\(3\),pp\. 23\.External Links:[Document](https://dx.doi.org/10.3390/bdcc4030023),[Link](https://www.mdpi.com/2504-2289/4/3/23)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.
- Wanget al\.\(2011\)L\. Wang, J\. Lin, and D\. MetzlerA cascade ranking model for efficient ranked retrieval\.InProceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR\),pp\. 105–114\.External Links:[Document](https://dx.doi.org/10.1145/2009916.2009934)Cited by:[§3\.4](https://arxiv.org/html/2610.29338#S3.SS4.p1.1)\.
- Xie and Du \(2026\)S\. Xie and J\. DuNeuro\-symbolic resolution of recommendation conflicts in multimorbidity clinical guidelines\.InProceedings of the 40th AAAI Conference on Artificial Intelligence \(Bridge Program on Logic and AI\),External Links:2604\.17340,[Link](https://arxiv.org/abs/2604.17340)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1)\.
- Xuet al\.\(2020\)Y\. Xu, M\. Li, L\. Cui, S\. Huang, F\. Wei, and M\. ZhouLayoutLM: pre\-training of text and layout for document image understanding\.InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining \(KDD ’20\),pp\. 1192–1200\.External Links:[Document](https://dx.doi.org/10.1145/3394486.3403172),[Link](https://arxiv.org/abs/1912.13318)Cited by:[§5](https://arxiv.org/html/2610.29338#S5.p2.1)\.
- Yinet al\.\(2021\)W\. Yin, D\. Radev, and C\. XiongDocNLI: a large\-scale dataset for document\-level natural language inference\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 4913–4922\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.435),[Link](https://aclanthology.org/2021.findings-acl.435)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p3.1),[§3\.1](https://arxiv.org/html/2610.29338#S3.SS1.p3.1)\.
- Zamani and Bendersky \(2024\)H\. Zamani and M\. BenderskyStochastic rag: end\-to\-end retrieval\-augmented generation through expected utility maximization\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR ’24\),pp\. 2641–2646\.External Links:[Document](https://dx.doi.org/10.1145/3626772.3657923),[Link](https://arxiv.org/abs/2405.02816)Cited by:[Limitations](https://arxiv.org/html/2610.29338#Sx1.p6.1)\.
- Zhanget al\.\(2024\)Q\. Zhang, V\. S\. Huang, B\. Wang, J\. Zhang, Z\. Wang, H\. Liang, S\. Wang, M\. Lin, C\. He, and W\. ZhangDocument parsing unveiled: techniques, challenges, and prospects for structured information extraction\.External Links:2410\.21169,[Link](https://arxiv.org/abs/2410.21169)Cited by:[Limitations](https://arxiv.org/html/2610.29338#Sx1.p3.1)\.
- Zhaoet al\.\(2026\)S\. Zhao, Y\. Xu, Z\. Chen, F\. Qiao, H\. Chen, X\. Li, S\. Lin, Z\. Ji, Y\. Li, and W\. WangBridging the gap in Chinese legal conflict review: a dataset, benchmark tasks, and framework\.Scientific Data13,pp\. 835\.External Links:[Document](https://dx.doi.org/10.1038/s41597-026-07195-2)Cited by:[§2](https://arxiv.org/html/2610.29338#S2.p2.1)\.

## Appendix ABenchmark Construction and Example Provenance

Figure[2](https://arxiv.org/html/2610.29338#A1.F2)shows the four\-stage construction of the 302\-pair benchmark described in Section[4\.1](https://arxiv.org/html/2610.29338#S4.SS1): mechanical candidate mining with no model judgment at the pairing step, labelling by a non\-Claude judge, an adversarial review pass by a separate model, and human verification of the disputed subset only\. Candidates entered the pool three ways: a stratified random sample of extracted claim pairs \(174\), a value\-mismatch gate that pairs claims on one topic carrying different numbers or thresholds \(75\), and a same\-topic gate that pairs claims restating one value \(53\), which supplies most Redundancy candidates\. Every passage is verbatim document text; only the pairing step differs\. 33 of the 57 Redundancy pairs are character\-identical text appearing in two places\.

Figure 2:Benchmark construction\. Claims are extracted per document and paired by shared terms with no model judgment; every pair is labelled by a non\-Claude judge, each label is attacked by a separate model, and only disputed rows are hand\-checked; the hand\-checked label is final and replaces the judge’s\.
## Appendix BJudge\-Model Leaderboard

Table[5](https://arxiv.org/html/2610.29338#A2.T5)gives the full Retrieve\-Judge leaderboard across the eight judge models we evaluated\. The headline point is the tight clustering: every model lands between 0\.46 and 0\.57 aggregate recall, a spread of about 0\.12, so the choice of judge model moves Retrieve\-Judge far less than the architecture does\. This is the evidence behind the claim in Section[4](https://arxiv.org/html/2610.29338#S4)that the ceiling is intrinsic to the retrieve\-then\-judge architecture rather than to any one model\.

Table 5:Full eight\-model Retrieve\-Judge leaderboard \(composed retrieval\+judge aggregate recall, 302\-pair benchmark\)\. Sonnet 4\.5, Sonnet 5, and Llama 3\.3 70B are measured directly; cells marked∗are composed from each model’s measured per\-class judge recall and the shared leaf\-corrected per\-class retrieval recall \(Conflict 0\.571, Redundancy 0\.846, Similarity 0\.480\), support\-weighted over Conflict/Redundancy/Similarity\. The retrieval factor is common to every row, so it sets the band’s absolute level but not the model ordering\.The Llama 3\.3 70B row also measures prompt transfer: under the Sonnet\-tuned judge prompt its judge\-only Conflict recall was 0\.654, and a from\-scratch Llama\-specific prompt raised it to 0\.808 while Conflict precision held near 0\.49, a label\-boundary limit rather than a wording one\.

The confirm\-model swap of Section[4\.3](https://arxiv.org/html/2610.29338#S4.SS3)is in Table[6](https://arxiv.org/html/2610.29338#A2.T6)\.

Table 6:Screen\-Confirm with its*confirm*model swapped \(Haiku screen fixed\): per\-class recall \(including Different\) and four\-class accuracy on the 302 pairs\. Conflict recall is not monotonic in general model strength; the Different column tracks each model’s ability to walk a false escalation back to Different, where Llama is weakest \(0\.474\)\.
## Appendix CNegative Ablations

For completeness we list experiments that produced no usable improvement and are not part of the deployed system; each was run in full before being set aside\.

- •Embedding\-side retrieval fixes \(five, all negative\)\.Parent\-section index alongside leaves, LLM\-generated context concatenated into the embedding, section/document title prepended to embedded text, 1024→\\rightarrow1536 output dimension, and float32 versus int8 Cohere\-side quantization\. None improved Conflict or Similarity R@10; two regressed it\. The retrieval gap is structural \(chunk boundaries and the leaf filter\), not an embedding parameter\.
- •Reranking and hypothetical\-document queries\.Cohere Rerank over the dense top\-20 moved per\-class R@10 by at most±\\pm0\.05 and changed no composed cell, so the scored configuration is dense\-only\. Hypothetical partner passages generated by Haiku and embedded as the query \(HyDE\) landed on the right topic but not the annotated partner among many same\-topic chunks, with no net R@10 gain\.
- •Chain\-of\-thought judge prompt\.A forced reasoning protocol before the label lowered Retrieve\-Judge Conflict precision on the benchmark \(0\.444→\\rightarrow0\.304\): it applied the new rule inconsistently across near\-duplicate phrasings\.
- •Long\-context, no\-retrieval baseline\.Feeding whole documents \(up to 533K tokens\) to the judge scored within noise of quote\-only judging on an 18\-pair pilot at roughly 39×\\timesthe input tokens per pair\.
- •Agentic tool\-use judge\.Letting the judge fetch surrounding context at its own discretion was a wash on 302 pairs \(24 rows fixed, 24 broken\) at about 3\.7×\\timescost, and introduced a new failure: irrelevant adjacent context turning correct Different calls into false Conflicts\.

## Appendix DWhich Fixes Moved Recall

Screen\-Confirm’s end\-to\-end row is scored on the production index: for each benchmark pair the screen reads the anchor document’s entire provisions store \(614 provisions on average over the 78 indexed documents; 13,352 for the 957\-page Departmental Orders\) against the section of the compared document that holds the partner text, and confirm follows\. It scores 0\.472 aggregate and 0\.615 on Conflict\. Table[7](https://arxiv.org/html/2610.29338#A4.T7)shows where each gold positive was lost: 7 of 127 positives never reach a model, the screen drops 31, confirm relabels 29, and 20 of 175 Different pairs survive as false positives\. Cross\-document pairs recover 0\.541; within\-document pairs 0\.377\. Precision with confidence intervals is in Appendix[F](https://arxiv.org/html/2610.29338#A6)\.

Table 7:Where each of the 127 gold positives ended on the production data path, final run\. Absent: its text is not in the index\. No prov\. \(no provision\): indexed, but distillation produced no provision for it\. Screen, Confirm: lost at that stage\. Found: labelled correctly\. Recall: over pairs that reached the screen\.This stage\-by\-stage attribution was re\-run after each fix, and each run said what to fix next\. The first run lost 67 of 127 positives before any model call, to three ingest faults: split\-PDF parts overwrote each other’s chunks, short chunks were dropped, and distillation batches were truncated\. Fixing them took recall from 0\.260 to 0\.354 and left 4 positives absent\. Two design faults followed\. The screen read about 6,000 provisions per call; windows of about 1,500 took recall to 0\.449 \(Conflict 12→\\rightarrow17 of 26\) at about 2\.3×\\timesthe run’s cost\. Confirm judged a provision against a whole section, so a sentence duplicated inside a longer section read as Similarity; asking it to label against the matching passage took recall to 0\.472 \(Redundancy 27→\\rightarrow32 of 57\)\. Both cost precision, with Different pairs surviving confirm rising from 12 to 20 of 175\.

## Appendix EBenchmark\-Scale Cost and Latency

Table[8](https://arxiv.org/html/2610.29338#A5.T8)gives cost and latency for the 302\-pair benchmark run, with the pair given; deployment\-scale costs are in Section[4\.4](https://arxiv.org/html/2610.29338#S4.SS4)\.

Table 8:Benchmark\-scalecost and latency, sequential single worker, not production cost\. Costs are summed from logged token counts at list rate\.‡Retrieve\-Judge output tokens estimated from a 40\-pair sample; with prompt caching its cost falls to∼\{\\sim\}$0\.91\.§Wall\-clock from an earlier 376\-pair run of the same pipelines; the 302\-pair run did not log it\.
## Appendix FPrecision and Confidence Intervals

Screen\-Confirm’s per\-class precision and recall with the pair given \(the setting of Table[3](https://arxiv.org/html/2610.29338#S4.T3)\), with 95% Wilson intervals on recall\. Redundancy precision is 1\.000 and Similarity’s 0\.612: every pair the cascade called Redundancy was Redundancy, and its mistakes were Redundancy pairs called Similarity\. It did not mislabel any pair as Conflict or Different\.

Table 9:Screen\-Confirm precision/recall per class, with 95% Wilson intervals on recall\.Because the benchmark is enriched for the rare classes, precision will be lower in deployment\. Conflict is 26 of the 302 pairs, about 9%\. Given the pair, the cascade flags 57 Conflicts, 23 right and 34 wrong \(precision 0\.404\)\. If true Conflicts are ten times rarer in a real corpus, the 23 become about 2 while the 34 false flags remain, and precision falls to about 0\.06\. Recall does not move: 0\.885 of true Conflicts are caught whatever their number\.

## Appendix GPrompts

The five prompts below are reproduced verbatim\. Variable slots filled at call time are shown in\{braces\}; section headers within a prompt are set in bold for readability and em dashes in the originals are shown as hyphens, with no other change to the text\. The Retrieve\-Judge judge prompt is the longest because it carries the label definitions, decision order, and five worked examples; the distillation and screen prompts are short and rely on the provisions list for context\.

### G\.1Provision Distillation Prompt \(Ingest, Fast Model\)

System prompt; each call receives a batch of up to 100 chunks, each prefixed with its header\.

Provision distillation \(system prompt, fast model\)You are a policy analyst\. Extract every distinct requirement, rule, or obligation from the policy text\. Each chunk of input is prefixed with a header like \[chunk:<id\> \| <section\> \| p\.<page\>\]\. Return one bullet per requirement\. Each bullet must begin with the exact chunk header from its source chunk, then the requirement as a complete, standalone sentence\. Example: \[chunk:abc \| I \> I\.A \| p\.3\] Officers must maintain their service weapon\. No headings, no grouping, no commentary\. Return only prefixed bullets, one per line\.

### G\.2Retrieve\-Judge Judge Prompt

Retrieve\-Judge judge \(system prompt\)You are a senior police policy analyst verifying whether two policy chunks have a genuine relationship that officers need to know about\.These chunks were pre\-screened as potentially related\. Your job is to CONFIRM the relationship type or REJECT as Different\. Decide EVERY boundary by normative CONTENT, not by wording overlap\. Apply the DECISION ORDER below in exact order; Similarity is NOT a leftover bucket \- confirm its three positive conditions\.DEFINITIONS\(decide by normative CONTENT, never by shared word count\): \- Conflict: THE SAME OFFICER in THE SAME SITUATION cannot comply with both \- obeying one puts them in breach of the other\. Time frames, thresholds, or procedures directly contradict\. \- Redundancy: the two impose materially the SAME obligation \- same actor, same scope, same threshold \- so deleting one loses NO requirement\. Strip formatting, casing, punctuation, and filler words first; if the only remaining differences are non\-normative \(reworded, an inserted "the", a unicode variant, reordering\), it is Redundancy even when the wording is not word\-for\-word\. \- Similarity: ALL THREE hold \- \(1\) SAME SUBJECT \(same act, object, event, or decision; an officer handling one would foreseeably meet the other\), \(2\) COMPATIBLE \(both can be obeyed together; neither breaches the other\), and \(3\) MATERIAL NORMATIVE DELTA \(each imposes normative content the other lacks: a different duty, actor, recipient, threshold, phase, or added condition\)\. Related rules that neither restate nor contradict each other\. \- Different: no shared act, object, event, or decision \- one carries nothing that bears on the other\.DECISION ORDER\(apply in this exact order\): 1\. Redundancy check FIRST, by content: same obligation on the same actor/scope/threshold, differing only in non\-normative ways? \-\> Redundancy\. 2\. Conflict check: could obeying one ever breach the other for the same subject in one situation? \-\> Conflict\. A difference in wording, or different\-but\-compatible details, is NOT a Conflict\. 3\. Different check: does one carry nothing that bears on the other? \-\> Different\. 4\. Otherwise \-\> Similarity \(same subject \+ compatible \+ material normative delta\)\.EXCEPTION HANDLING\(a general rule paired with a narrower passage\): \- The narrower passage EXPLICITLY subordinates itself and fully dissolves the tension \("unless", "except", or it names the sub\-population it exempts\) \- the tension is resolved \-\> Different\. \- It is a conditional override the reader must CROSS\-REFERENCE to apply correctly \(e\.g\. a default threshold plus a different threshold when a condition holds\) \-\> Similarity\.NOTIFICATION ROUTING\- apply this before classifying any communication/reporting pair as Conflict: Requiring notification to Person A \(one policy\) and Person B \(another policy\) is NOT a Conflict \- the officer can notify both\. It is Similarity\. A Conflict exists only when both policies specify the SAME notification slot but assign it to mutually exclusive recipients\.REDUNDANCY vs SIMILARITY\- the content test \(never decide this by shared word count\): Does either passage impose normative content the other lacks \- a different duty, actor, recipient, threshold, phase, or added condition? \- No material delta \(only wording/formatting differs\) \-\> Redundancy, even if the two are not word\-for\-word identical\. \- At least one material delta \-\> Similarity, even when long spans of wording are shared\.HARD REJECTION CRITERIAfor Conflict \(output Similarity or Different if ANY apply\): \- Different actors \(officer vs supervisor vs dispatcher vs records staff\) are a MATERIAL DELTA: an otherwise\-duplicate rule that assigns the duty to a different actor is Similarity, not Redundancy\. CARVE\-OUT \- Conflict: same procedure, contradictory parameters for situations involving the same officer \-\> Conflict regardless of unit labels\. \- Different phases \(initial response vs follow\-up report vs administrative review\) \- Same broad topic but different specific situations \- General guidance vs specific procedure \- One policy sets a standard, the other implements it in a different context \- SAME\-DOCUMENT PAIRS \- apply extra caution: apparent contradictions are often complementary steps or sequential procedures\. Default to Similarity over Conflict unless a specific numeric value, deadline, or threshold directly contradicts with no possible sequential or role\-based reading\.AUDIENCE AND UNIT CHECK\- before any Conflict: Confirm both chunks apply to the same unit, role, and operational context\. If different audiences, units, or operational phases are plausible, classify as Different or Similarity rather than Conflict\.FINAL CHECK: \- Conflict: Name the specific scenario where the same officer cannot comply with both simultaneously\. If you cannot \-\> Similarity or Different\. \- Redundancy: Confirm NO material normative delta remains after stripping formatting and wording\. If a delta remains \(different actor, threshold, phase, added condition\) \-\> Similarity\.EXAMPLES:Example 1\- Conflict \(time limit\): Chunk A: "Officers involved in a use\-of\-force incident shall submit a written report to their immediate supervisor within four \(4\) hours of the incident\." Chunk B: "A Use of Force Report shall be completed and submitted by the involved officer no later than twenty\-four \(24\) hours following any use of force occurrence\." Output: \{"summary": "Conflict", "findings": \[\{"type": "Conflict", "description": "Use of force report deadline is 4 hours \(Chunk A\) versus 24 hours \(Chunk B\) \- the same officer filing the same report faces contradictory deadlines\."\}\]\}Example 2\- Conflict \(authorisation threshold\): Chunk A: "Purchases of evidence\-related supplies exceeding $500\.00 require prior written approval from the unit commander\." Chunk B: "Unit expenditures on evidence supplies up to $1,000\.00 may be authorised by the division supervisor without additional approval\." Output: \{"summary": "Conflict", "findings": \[\{"type": "Conflict", "description": "A $750 evidence supply purchase requires prior commander approval \(Chunk A\) but falls within the supervisor’s self\-authorisation limit \(Chunk B\) \- the same officer cannot determine which approval pathway applies\."\}\]\}Example 3\- Redundancy \(near\-verbatim\): Chunk A: "Officers must complete a Use of Force Report \(Form UF\-1\) within twenty\-four \(24\) hours of any use of force incident\." Chunk B: "A Use of Force Report \(Form UF\-1\) shall be completed by the involved officer within twenty\-four \(24\) hours following any use of force occurrence\." Output: \{"summary": "Redundancy", "findings": \[\{"type": "Redundancy", "description": "Both sections require Form UF\-1 within 24 hours of a use of force incident using near\-identical wording \- this requirement is duplicated across two documents\."\}\]\}Example 4\- Similarity \(notification to different people \- NOT a Conflict\): Chunk A: "Officers shall notify their immediate supervisor within one \(1\) hour of any use of force incident\." Chunk B: "Following any use of force, the officer shall notify the on\-duty watch commander as soon as practicable\." Output: \{"summary": "Similarity", "findings": \[\{"type": "Similarity", "description": "Both policies require post\-use\-of\-force notifications but to different recipients \(supervisor vs watch commander\) \- an officer must satisfy both, so awareness of each is operationally relevant\."\}\]\}Example 5\- Different: Chunk A: "The Records Unit shall maintain all arrest reports for a minimum of seven \(7\) years\." Chunk B: "Officers responding to domestic disturbance calls shall request a supervisor to respond to the scene\." Output: \{"summary": "Different", "findings": \[\]\}OUTPUT ONLY valid JSON\- no other text: \{"summary": "Conflict\|Redundancy\|Similarity\|Different", "findings": \[\{"type": "Conflict\|Redundancy\|Similarity", "description": "concise description of the issue"\}\]\}For Different: \{"summary": "Different", "findings": \[\]\}WRITING GUIDANCE\- officers review thousands of these: \- State the specific contradiction, duplication, or connection in one sentence \- Embed section references in parentheses \- do not start with "Section X\.X requires\.\.\." \- Do not use: "appears in both", "creating divergence risk", "mutually exclusive", "cannot simultaneously comply" \- Do not add a second sentence unless critical information will not fit in one

### G\.3Screen\-Confirm Screen Prompt \(Fast Model\)

Sent after the cached, numbered provisions block and the Policy B section \(“Policy B section \(<key\>\):” followed by the section text\)\. This is the version scored in Section[4](https://arxiv.org/html/2610.29338#S4); the deployed version at the time of writing is a shorter earlier one with the same output contract\.

Screen\-Confirm screen \(fast model\)Below is a list of numbered provisions from Policy A, then a section from Policy B\. This is a fast first\-pass screen\. Flag every provision that plausibly relates to the Policy B section so it can be re\-checked later; when in doubt, INCLUDE it\. Use these labels: \- Conflict: the provision and the section give directives that cannot both be followed for the same subject \- one requires what the other forbids, or they set different thresholds/deadlines/numbers for the same thing\. \- Redundancy: the provision and the section impose the same obligation on the same actor and scope \- the same requirement restated, differing only in wording or formatting \(decide by content, not by shared word count\)\. \- Similarity: same subject and compatible, but each carries some normative content the other lacks \- a different duty, actor, recipient, threshold, phase, or added condition \- so neither restates nor contradicts the other\. Examples of what to flag: \- Provision ’Reports are due by the 5th of the month’ vs section ’Reports shall be submitted by the 5th of each month’ \-\> Redundancy \(same requirement, near\-identical text\)\. \- Provision ’Officers must always wear a uniform’ vs section ’Plain\-clothes officers may work in civilian dress’ \-\> Conflict \(cannot both hold for the same officers\)\. Skip only provisions that are completely unrelated \- a different subject with no overlap \- and output nothing for them\. Return a JSON array\. Each item: \{"provision\_index": int, "label": "Conflict\|Redundancy\|Similarity", "why": str\}\. If nothing matches, return \[\]\. Return JSON array only, no other text\.

### G\.4Screen\-Confirm Confirm Prompt \(Strong Model\)

Cross\-document version, scored in Section[4](https://arxiv.org/html/2610.29338#S4); as above, the deployed version is a shorter earlier one\. The screen’s label enters only at step 3 as a tie\-breaker\.

Screen\-Confirm confirm \(strong model\), cross\-documentPolicy A text: \{policy\_a\_text\}Policy B section text: \{policy\_b\_text\}Classify the relationship between these two passages\. Work in order; do NOT consider any prior system’s guess until step 3\. Step 1\- In one clause each, state the specific requirement each passage imposes \(who must do what, when, to what threshold\)\. If a passage imposes no requirement, say so\. Step 2\- Using ONLY your step\-1 reading, choose the label by CONTENT \(never by shared word count\), checking in this order: \- Redundancy \(check FIRST\): the two impose materially the SAME obligation \- same actor, scope, and threshold \- so deleting one loses no requirement\. Strip formatting and wording; if the only differences are non\-normative \(rewording, casing, an inserted ’the’, a unicode variant\), it is Redundancy even if not word\-for\-word\. \- Conflict: the two requirements CANNOT both be satisfied for the same subject in the same situation\. Apply this concrete test \- would obeying Policy A put someone in breach of Policy B in that same situation? Only ’yes’ is a Conflict\. Shared topic, overlapping wording, or different\-but\-compatible details are NOT a Conflict\. \- Similarity: ALL THREE hold \- same subject, compatible \(both can be obeyed together\), and a MATERIAL NORMATIVE DELTA \(each imposes content the other lacks: a different duty, actor, recipient, threshold, phase, or added condition\)\. A different actor or threshold is a material delta \- Similarity, not Redundancy, even when wording is largely shared\. \- Different: no shared act, object, event, or decision\. Finding ONE distinguishing detail does not by itself make a pair Different when both passages still address the same underlying subject \- that is Similarity, not Different\. Contrastive examples\(same format, opposite outcomes\): \- A ’Officers shall document any use of force within 24 hours’ vs B ’Supervisors shall review all use\-of\-force reports’ \-\> Similarity: same subject \(use of force\), each imposes a different but compatible duty; neither contradicts or restates the other\. \- A ’Officers shall document any use of force within 24 hours’ vs B ’Fleet vehicles shall be refueled at the end of each shift’ \-\> Different: unrelated subjects, no shared requirement\. Step 3\- A first\-pass screen guessed:\{label\}\(reason:\{why\}\)\. Use this only as a tie\-breaker: keep your step\-2 label as\-is; adopt the guess ONLY if you were genuinely undecided between two labels and it breaks the tie\. Do not switch to it merely because it was provided\. Reply with JSON only: \{"confirmed": true/false, "label": "Conflict\|Redundancy\|Similarity\|Different", "why": str, "quote\_a": str, "quote\_b": str\} confirmed: true if your final label equals the first\-pass guess, false otherwise\. quote\_a: short verbatim phrase from Policy A most relevant to the finding \(1\-2 sentences, empty string if Different\)\. quote\_b: short verbatim phrase from Policy B most relevant to the finding \(1\-2 sentences, empty string if Different\)\.

End\-to\-end runs \(Section[4\.3](https://arxiv.org/html/2610.29338#S4.SS3)\) replace the opening instruction with:*The Policy B section may contain several requirements\. First find the single passage \(1–3 sentences\) in it that addresses the same subject as the Policy A text\. Classify the relationship between the Policy A text and THAT passage only; ignore requirements elsewhere in the section\.*

### G\.5Screen\-Confirm Confirm Prompt, Same\-Document Variant

Used when both passages come from one document \(the exception\-handling rules differ\)\.

Screen\-Confirm confirm \(strong model\), same\-documentBoth passages below are drawn from the SAME policy document\. Treat them as parts of one governing authority that is meant to be internally consistent\.Passage A: \{policy\_a\_text\}Passage B: \{policy\_b\_text\}Classify the relationship\. Work in order; do NOT consider any prior system’s guess until step 3\. Step 1\- In one clause each, state the specific requirement each passage imposes \(who must do what, when, under what circumstances\)\. Step 2\- Using ONLY your step\-1 reading, choose the label: \- Conflict: the two passages issue directives that CANNOT both be complied with for the same subject under the same circumstances\. Apply this concrete test \- would obeying Passage A put someone in breach of Passage B in that same situation? Only ’yes’ is a Conflict\. \- NOT a conflict \(label Different\): a general rule paired with a narrower exception that is EXPLICITLY subordinated to it \- the narrower passage signals it is a carve\-out \(e\.g\. ’unless’, ’except’, ’shall not \.\.\. unless’, or it names the specific sub\-population it exempts from the general rule\)\. The exception refines the rule and both can be followed together\. But if the two passages are each free\-standing rules that independently reach OPPOSITE determinations about the same subject \(neither is written as an exception to the other\), that remains a Conflict\. \- NOT a conflict \(label Different\): passages that govern different subjects, roles, units, materials, or circumstances, even when they share wording or cite different numbers/thresholds\. \- Redundancy: the two impose materially the SAME obligation \- same actor, scope, and threshold \- differing only in non\-normative ways \(wording, casing, formatting\)\. Decide by content, not by shared word count\. \- Similarity: ALL THREE hold \- same subject, compatible, and a MATERIAL NORMATIVE DELTA \(each imposes content the other lacks: a different duty, actor, threshold, phase, or added condition\)\. This INCLUDES a general default plus a conditional override the reader must CROSS\-REFERENCE to apply correctly \(e\.g\. a different threshold when some condition holds, neither written as an explicit exception to the other\)\. A different actor or threshold is a material delta \- Similarity, not Redundancy\. Finding ONE distinguishing detail does not by itself make a pair Different when both still address the same subject\. \- Different: no shared act, object, event, or decision, or an apparent contradiction resolved by an explicit exception or by differing scope as above\. Contrastive examples\(same format, opposite outcomes\): \- A ’All prisoners will be handcuffed during transport’ vs B ’Juveniles 12 and under shall not be handcuffed unless they pose a threat’ \-\> Different: B is an explicit exception \(’unless’\) carved out of A; both can be followed together\. \- A ’Use\-of\-force reports are due within 24 hours’ vs B ’Use\-of\-force reports are due within 72 hours’ \-\> Conflict: two free\-standing rules set incompatible deadlines for the same report, neither written as an exception to the other\. Step 3\- A first\-pass screen guessed:\{label\}\(reason:\{why\}\)\. Use this only as a tie\-breaker: keep your step\-2 label as\-is; adopt the guess ONLY if you were genuinely undecided between two labels and it breaks the tie\. Do not switch to it merely because it was provided\. Reply with JSON only: \{"confirmed": true/false, "label": "Conflict\|Redundancy\|Similarity\|Different", "why": str, "quote\_a": str, "quote\_b": str\} confirmed: true if your final label equals the first\-pass guess, false otherwise\. quote\_a: short verbatim phrase from Passage A most relevant to the finding \(1\-2 sentences, empty string if Different\)\. quote\_b: short verbatim phrase from Passage B most relevant to the finding \(1\-2 sentences, empty string if Different\)\.

### G\.6Baseline Setups

Llama zero\-shot\.us\.meta\.llama3\-3\-70b\-instruct\-v1:0via Bedrock, one call per pair, quote\-only, temperature 0, no examples\. System instruction:

Llama zero\-shot baseline \(system instruction\)You classify the relationship between two police policy statements into exactly one of four categories: \- Conflict: an officer cannot comply with both at once; they contradict\. \- Redundancy: they impose the same requirement; one duplicates the other\. \- Similarity: related and compatible, but each adds something the other lacks\. \- Different: unrelated; neither bears on the other\. Answer with only one word: Conflict, Redundancy, Similarity, or Different\.

DeBERTa NLI\.MoritzLaurer/deberta\-v3\-large\-zeroshot\-v2\.0, not fine\-tuned, run as zero\-shot classification over the concatenated pair with the hypothesis template “These two policy statements are \{\}\.” and one candidate per label:*in conflict*,*redundant*,*similar*,*unrelated*\. The highest entailment score wins; ties break in that order\.