ContractScrub: A benchmark for final review of legal contracts

# ContractScrub: A benchmark for final review of legal contracts
Source: [https://arxiv.org/html/2608.20204](https://arxiv.org/html/2608.20204)
1\]Thomson Reuters Foundational Research, London, UK 2\]Imperial College London, UK\\correspondence\{first\.last\}@thomsonreuters\.com\\contribution\[\*\]Equal contribution

Kirsty Fielding∗Brandan Oliver∗Brian Birke∗Nabeel Seedat∗Andrew M\. Bean∗Affiliation:\[Affiliation:\[

###### Abstract

Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs\. Contract “scrubbing,” the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents\. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long\-context reasoning, consistency checking, and named entity recognition \(NER\)\. Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted\. We introduceContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand\-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language\. Frontier models perform surprisingly poorly with only one model reaching 0\.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain\-specific benchmarks for measuring real\-world impact\.

## 1Introduction

SERVICE AGREEMENT January 1, 20241\. DEFINITIONS “Agreement” means this current agreement, inclusive of Schedules A\-C\.“Services” means consulting services as listed inSchedule D\.2\. OBLIGATIONS 2\.1 The Supplier shall provide Services to the Client at the prices specified in the Schedules contained in theagreement\.3\. TERMINATION Either party may terminate the Agreement if the other commits aMaterial Breach\.— 1 —Identified issuesNot using defined termsUsing “agreement” in §2\.1 introduces questions about what agreement is at issueIncorrect reference“Schedule D” in §1 is not part of the AgreementMissing definition“Material Breach” referenced in §3 is capitalized, implying it is a defined term, but it is not defined

Figure 1:An example of errors that should be found in the scrubbing process\.The practice of law ranks highly among the professions where AI is expected to have the most potential for economic impact\[[14](https://arxiv.org/html/2608.20204#bib.bib16),[7](https://arxiv.org/html/2608.20204#bib.bib15)\]\. Benchmarks such as LEXam\[[5](https://arxiv.org/html/2608.20204#bib.bib9)\]and Stanford Legal Bench\[[9](https://arxiv.org/html/2608.20204#bib.bib10)\]are typically used to assess progress in legal capabilities like reasoning\. However, translation between capability evaluations and real\-world impact can often be limited\[[21](https://arxiv.org/html/2608.20204#bib.bib13),[1](https://arxiv.org/html/2608.20204#bib.bib14)\]\. By directly testing models on economically valuable tasks, narrowly targeted benchmarks with high ecological validity offer a more reliable measure of the potential of current legal AI systems\.

Transactional law practices involve the negotiation, drafting, and execution of legally binding agreements and navigation of the complex relationships of the parties\. Contracts, as a subset of transactional law, place heavy demands on precision and nuance\. Small details can affect legal interpretation, a famous example being the lack of an Oxford comma in the statutes governing Maine employment contracts, which led to a multi\-million dollar settlement111See O’Connor v\. Oakhurst Dairy, 851 F\.3d 69 \(1st Cir\. 2017\); see also https://www\.bbc\.co\.uk/worklife/article/20180723\-the\-commas\-that\-cost\-companies\-millions\. Contract scrubbing occupies a peculiar position in transactional law practices, widely recognized as essential but also generally tedious to carry out\. At the very end of \(and often throughout\) a deal, when negotiations are complete but before signatures are exchanged, lawyers and paralegals will “scrub” a contract; making a final pass to remove any outstanding errors or inconsistencies\. This exercise offers high ecological validity as it is a discrete, well\-scoped task with a clear ground truth directly replicating work performed by professionals under realistic conditions\.

Though critically necessary, scrubbing is meticulous, repetitive work often performed under intense pressure from the demands of closing a deal, making this routine task error\-prone in practice\. At the same time, many elements of contract scrubbing \(e\.g\. identifying defined terms, checking consistency of usage, and verifying section references\) seem to fit naturally within the capabilities of LLMs around long\-context reasoning, consistency checking, and named entity recognition \(NER\)\. Legal practitioners frequently approach this risk\-control function as essential but tedious work\. If the scrub could be automated reliably, practitioners could redirect their attention to other matters: identifying issues that demand legal judgment, advising clients on risk, and advancing negotiations\. Attorneys could also perform automated scrubbing more frequently throughout the negotiation of a contract, reducing the risk of last\-minute changes disrupting an otherwise settled deal\.

The clear potential for AI\-assisted contract scrubbing makes it a natural domain for a targeted, task\-specific benchmark, but, to the best of our knowledge, no such benchmark exists\. Current contract\-related benchmarks focus largely on reasoning about contracts rather than reviewing them for precision and consistency\. CUAD\[[10](https://arxiv.org/html/2608.20204#bib.bib1)\]tests whether models can identify legally significant provisions such as governing law, exclusivity, and non\-compete clauses\.\[[17](https://arxiv.org/html/2608.20204#bib.bib3)\]introduced ContractEval, a benchmark focused on evaluating LLMs’ ability to identify clause\-level legal risks in commercial contracts\. And, general\-purpose tasks in NLP like "needle\-in\-a\-haystack"\[[12](https://arxiv.org/html/2608.20204#bib.bib8)\], are more similar to the actual work required for scrubbing, but do not share the idiosyncrasies of the legal domain\.

Table 1:Scrub Review Categories\. Each tested contract contains different categories of errors\.CategoryExplanationDefined TermsWords or phrases formally defined in the agreement with a designated meaning, often capitalized, underlined, bolded, italicized, or in quotations\.Undefined Capitalized TermsTerms that are capitalized or otherwise treated as a defined term in an agreement but not formally defined\.Uncapitalized Defined TermsFormally defined terms appearing in lowercase when they should be capitalized\.Incorrectly Capitalized Terms In ContextTerms that have a definition but are capitalized in a context where they are not being used as a defined term\.Unused Defined TermsDefined terms that are not used outside of the specific instance where they are defined\.Terms Defined Multiple TimesTerms that are defined more than one time in an agreement\.Incorrect Section, Article, or Paragraph ReferencesInternal cross\-references that assign an incorrect section, article, or paragraph of the agreement\.Incorrect Party ReferencesInstances where a party is referred to by the wrong party name or role\.Inconsistent LanguageLanguage in the agreement that directly contradicts itself or other language elsewhere in the agreement\.To address this gap, we introduceContractScrub, the first benchmark designed to evaluate LLM performance on the various \(including final\) review stages of the contract or deal lifecycle\. ContractScrub comprises 3,014 annotated tasks across 44 contracts drawn from CUAD\[[10](https://arxiv.org/html/2608.20204#bib.bib1)\], including scrubbing elements and errors, e\.g\., defined term inconsistencies, capitalization errors, and cross\-reference failures\. The contracts are hand annotated by experienced lawyers to include representative test issues\.

We evaluate 9 frontier and open\-weight models of varying families and sizes onContractScruband find that contract scrubbing remains a substantially harder task for LLMs than its individual components might suggest\. The best\-performing model, GPT\-5\.5, reaches a macro\-average recall score of only 0\.750, and all F1 scores are below 0\.650\. Performance is uneven across issue types: models handle categories with explicit lexical signals \(e\.g\., defined term;μ=\\mu=\.835\) much better than those requiring inference of intent within context \(e\.g\., Incorrect Capitalization in Context;μ=\\mu=\.427\)\. Beyond the practical value of filling the measurement gap, our benchmark offers a valuable theoretical insight\. Despite many of the tasks required for scrubbing sharing a strong similarity with conventional NLP tasks, most frontier models fall short of the performance that would be expected based on their general capabilities, highlighting the value of domain\-specific benchmarking as a practice\. Furthermore, enabling reasoning yields only moderate gains, concentrated in categories that require full\-document term consistency rather than deeper legal interpretation\. These findings underscore the need for targeted benchmarks in professional domains, and we releaseContractScrubto contribute to that effort\.

## 2Related Work

#### Legal Benchmarks for LLMs

Evaluating the legal capabilities of LLMs is an active area of research, with recent benchmarks aimed at measuring broad legal competence across diverse tasks\. LEXam\[[5](https://arxiv.org/html/2608.20204#bib.bib9)\], Stanford LegalBench\[[9](https://arxiv.org/html/2608.20204#bib.bib10)\], LawBench\[[6](https://arxiv.org/html/2608.20204#bib.bib12)\], and LexGLUE\[[2](https://arxiv.org/html/2608.20204#bib.bib18)\]exemplify this general\-purpose approach, evaluating LLMs in collections of legal problems that test skills such as legal knowledge, reasoning, and interpretation\. These suites are valuable for measuring overall progress in legal AI, but their tasks typically probe narrow skills in isolation rather than end\-to\-end review of a single long document\. Contract scrubbing is one such workflow: it requires sustained attention to long, highly structured documents, where small textual inconsistencies can have legal or commercial consequences, or both\.

A closer line of work does evaluate LLMs in contract\-specific tasks\. CUAD\[[10](https://arxiv.org/html/2608.20204#bib.bib1)\]tests whether models can identify important clauses and legal attributes in thousands of annotated examples of commercial contracts, while MAUD\[[22](https://arxiv.org/html/2608.20204#bib.bib2)\]provides a similarly large\-scale benchmark for locating deal\-points in merger agreements\. Both cast contract review largely as reading comprehension over predefined legal categories, requiring models to locate relevant provisions or determine whether particular deal points are present — e\.g “is there a non\-compete clause?” ContractNLI\[[13](https://arxiv.org/html/2608.20204#bib.bib17)\]evaluates contractual reasoning asking whether contract provisions entail, contradict, or leave undetermined a set of hypotheses\. The Lease benchmark of\[[15](https://arxiv.org/html/2608.20204#bib.bib4)\]evaluates NER and red\-flag detection on residential lease agreements, and ContractEval\[[17](https://arxiv.org/html/2608.20204#bib.bib3)\]extends this to legal\-risk identification across a wider set of commercial agreements\. Both move closer to practical review, but still assess classification of provisions against a fixed taxonomy of risk types\. In contrast,ContractScrubevaluates defect identification as a recall\-sensitive, full\-document task\. Rather than answering a supplied question, classifying a provision against a known taxonomy, or evaluating a predefined hypothesis, the model is expected to surface drafting, consistency, and document\-hygiene defects in a contract, which, if missed, can have material legal and practical consequences\. In short, prior contract benchmarks largely evaluate “is X here?” under a known schema\.ContractScrubevaluates over the whole document “what is wrong with this relative to the contract’s own internal conventions?”

#### Related Non\-Legal Tasks

Contract scrubbing requires the joint application of several general\-purpose LLM capabilities, including long\-context reasoning, precise localization, consistency checking, and referential understanding, in a structured legal setting\. Existing non\-legal benchmarks probe related capabilities, but generally in more isolated settings\. Long\-context benchmarks\[[11](https://arxiv.org/html/2608.20204#bib.bib6),[16](https://arxiv.org/html/2608.20204#bib.bib7)\]evaluate whether models can reason over extended inputs, a prerequisite for contract scrubbing because relevant errors may be sparsely distributed across a lengthy document\. Needle\-in\-a\-Haystack\-style evaluations\[[12](https://arxiv.org/html/2608.20204#bib.bib8),[3](https://arxiv.org/html/2608.20204#bib.bib21)\]require models to find a single embedded fact within a long context, which parallels the localization demands of scrubbing but not the requirement to surface*all*relevant defects based on the broader context of the document\. FaithEval\[[19](https://arxiv.org/html/2608.20204#bib.bib5)\]includes inconsistency detection over short passages, resembling conflicting\-definition or inconsistent\-term errors\. However, it treats consistency primarily as a binary classification problem, rather than requiring models to find, locate, and contextualize inconsistencies across a long document\. IdentifyMe\[[18](https://arxiv.org/html/2608.20204#bib.bib11)\]benchmarks entity reference resolution, a capability relevant to detecting incorrect party names and section references, but operates on general\-domain text\. Each of these benchmarks isolates an individual component capability\.ContractScrubis designed to evaluate their collective application to comprehensive defect identification and localization in full\-length contracts\.

## 3ContractScrubBenchmark

Contract review involves reading and understanding a contract thoroughly to identify errors, analyse risks, and ensure consistency\. Throughout the negotiation process and certainly at the final stage of review, attorneys conduct a dedicated pass to eliminate residual errors and inconsistencies – a process commonly known as “scrubbing\.” Seemingly small errors in contracts are time\-intensive to identify and review and can be highly consequential, requiring legal practitioners to catch them in the contract\-drafting lifecycle\. However, they remain difficult to identify under the time pressures typically accompanying such a routine task \(that is, nearing deal closing or contract execution\) and because manual review is repetitive, subject to fatigue, and complicated by multiple negotiated drafts, schedules, and amendments\. Automating or augmenting legal practitioners at this stage with LLMs offers a practical path to freeing practitioners for higher\-value work by reducing avoidable drafting mistakes, improving first\-pass review, supporting junior attorneys, accelerating quality control, and allowing senior lawyers to spend more time on issues requiring their judgment and experience, all the while upholding the high standards commensurate with the legal profession\.

### 3\.1Task Overview

Figure 2:ContractScrub construction pipeline\. Contracts are sourced and screened for existing structural issues \(Stage 1\), annotated for existing drafting errors \(Stage 2\), and augmented with targeted additional errors \(Stage 3\) to produce the gold answer\. All steps are carried out by experienced lawyers\.![Refer to caption](https://arxiv.org/html/2608.20204v1/figures/contractscrub_pipeline.png)

ContractScrubis the first benchmark evaluating LLM performance on the scrubbing pass\. The task is defined as follows\. Let𝒞=\{c1,…,cn\}\\mathcal\{C\}=\\\{c\_\{1\},\\dots,c\_\{n\}\\\}be a corpus ofnncontracts\. For each contractcic\_\{i\}, a gold annotationRiR\_\{i\}is a multiset of tuples, where each tuple𝐫=\(κ,𝐟\)\\mathbf\{r\}=\(\\kappa,\\mathbf\{f\}\)consists of a category labelκ∈𝒦\\kappa\\in\\mathcal\{K\}and a category\-specific field vector𝐟\\mathbf\{f\}\. The category set𝒦\\mathcal\{K\}comprises nine elements covering one defined\-term extraction category and eight drafting\-error categories \(Table[1](https://arxiv.org/html/2608.20204#S1.T1)\)\. The field vector𝐟\\mathbf\{f\}encodes the category\-specific fields for each instance — for example,𝐟=\(term,location\)\\mathbf\{f\}=\(\\textit\{term\},\\textit\{location\}\)for most term\-level categories, and𝐟=\(location1,location2\)\\mathbf\{f\}=\(\\textit\{location\}\_\{1\},\\textit\{location\}\_\{2\}\)for relational categories such as inconsistent terms\. The task is designed to simulate the scrubbing pass as performed by actual attorneys in practice: given contractcic\_\{i\}and an instruction promptIIas input, a model must produce a predicted multisetR^i\\hat\{R\}\_\{i\}of tuples drawn from the same schema\. Formally, each modelℳ\\mathcal\{M\}produces a predicted annotation:

R^i=ℳ⁡\(I,ci\),\\hat\{R\}\_\{i\}=\\mathcal\{M\}\(I,c\_\{i\}\),Performance is measured by comparingR^i\\hat\{R\}\_\{i\}against the goldRiR\_\{i\}across all categories and contracts\.

### 3\.2Dataset

To systematically evaluate scrubbing capability of LLMs, we construct dedicated annotated data \(Figure[2](https://arxiv.org/html/2608.20204#S3.F2)\)\. The schema of categories is designed by licensed attorneys with professional experience practicing and litigating contract law\. Then, the further annotation and creation of the dataset is conducted by 9 different lawyers with experience in various forms of commercial, corporate, and contract law\. All the lawyers have been practicing law for at least 8 years, with 8 having more than 10 years experience and 6 having more than 15 years\.

ContractScrubconsists of 3,014 annotated tasks across 9 categories drawn from 44 contracts\. Each contractcic\_\{i\}has corresponding gold annotationRiR\_\{i\}, which consists of different categories explained in Table[1](https://arxiv.org/html/2608.20204#S1.T1)\. The guiding principles in creating this benchmark are improving contract hygiene \(precision and consistency\), reducing the risk of misinterpretation and ambiguity, and preserving client confidence by uncovering drafting defects that can have outsized impacts on contractual meaning, clarity, and outcomes\.

#### Scrub Categories

Each gold answer covers nine annotation categories listed in Table[1](https://arxiv.org/html/2608.20204#S1.T1): one defined\-term extraction category and eight drafting\-error categories\. The categories were selected because they are common, concrete, and potentially impactful on a contract with these types of issues\. These errors are also often overlooked because they are embedded in otherwise innocuous contract language and surfaced only with both deep contextual awareness and exacting attention to detail\.

Each category carries distinct legal implications\. To illustrate, consider the ‘Uncapitalized Defined Terms’\. Where a contracting party defines "Representative" narrowly, encompassing only company officers and legal counsel, but subsequently uses "representative" in lowercase within a confidentiality provision, a counterparty may reasonably interpret the inconsistency as intentional and apply the broader, ordinary meaning of the term\. The result is that the countererparty may share sensitive information with a substantially wider group than the drafting party intended\. By contrast, some errors may be characterized as more mundane in nature, with comparatively limited legal consequence in isolation\. Nonetheless, even errors that appear minor can introduce ambiguity about agreed terms, generate friction between contracting parties, delay deal execution, and erode client confidence – outcomes that carry real commercial cost regardless of their legal characterization\. See Appendix[4](https://arxiv.org/html/2608.20204#A1.T4)for implication of each category\.

#### Construction Pipeline

The process consists of three stages as shown in Figure[2](https://arxiv.org/html/2608.20204#S3.F2): \(i\) selecting and reviewing source contracts for structural flaws; \(ii\) annotating existing issues; and \(iii\) inserting additional issues to create the final gold answers\. Source contracts are drawn from the open\-source CUAD dataset\[[10](https://arxiv.org/html/2608.20204#bib.bib1)\], which is in turn drawn from EDGAR222https://www\.sec\.gov/search\-filings, an open repository of documents from publicly\-owned US companies\.ContractScrubdoesnotuse CUAD’s labels: each contract is independently reviewed and annotated by legal subject matter experts \(SMEs\) for categories relevant to scrubbing contracts\. We describe the pipeline in greater detail:

1. 1\.Contract sourcing and review\.Contracts are selected from the source pool to span a range of subject matter and drafting styles\. Each candidate is reviewed by an SME for fundamental drafting flaws that would render it unusable, and for suitable length \- short enough to validate within reasonable effort, but long enough to host the full set of targeted issues without becoming structurally invalid, which in practice means approximately 10\-15 pages\. Where necessary, contracts were also revised to improve overall coherence by, for example, removing empty exhibits or improving consistency\. This process yields an initial corpus𝒞^=\{c^1,…,c^n\}\\mathcal\{\\hat\{C\}\}=\\\{\\hat\{c\}\_\{1\},\\dots,\\hat\{c\}\_\{n\}\\\}\.
2. 2\.Annotation of existing issues\.For eachc^i\\hat\{c\}\_\{i\}, the SME records every defined term and any pre\-existing instances of error categoriesk∈𝒦k\\in\\mathcal\{K\}already present in the contract as tuples\(κ,𝐟\)\(\\kappa,\\mathbf\{f\}\)\. The quality of contracts in this dataset varies widely, and some contracts will have several existing errors to be identified\.
3. 3\.Insertion of new issues\.The SME then purposefully introduces additional drafting errors across the error categories \(Table[1](https://arxiv.org/html/2608.20204#S1.T1)\), reflecting realistic mistakes seen in transactional practices\. These annotations aim for broad and approximately balanced representation across error types𝒦\\mathcal\{K\}, while preserving the coherence and legal plausibility of the edited contract\. Each inserted issue is recorded as a tuple\(κ,𝐟\)\(\\kappa,\\mathbf\{f\}\)under the same schema as pre\-existing issues\. Every occurrence is logged separately: repeated terms or errors in the same section are not de\-duplicated\. The final result is a new contract,ci∈𝒞c\_\{i\}\\in\\mathcal\{C\}for each original contractc^i\\hat\{c\}\_\{i\}, and gold answers,RiR\_\{i\}, including all defined terms, any identified pre\-existing issues and all SME\-inserted issues\.

After the data creation pipeline, two of the lawyers performed targeted reviews of the contracts and gold answers for quality, and suggested changes to the contracts, gold answers, and task prompts as necessary to increase alignment\.

### 3\.3Metrics

We evaluate performance via multiset comparison between gold and predicted tuples\. For each categoryk∈𝒦k\\in\\mathcal\{K\}, tuples fromRiR\_\{i\}andR^i\\hat\{R\}\_\{i\}are normalized and matched across all contracts; true positives \(T​PkTP\_\{k\}\) are matched pairs, false positives \(F​PkFP\_\{k\}\) are unmatched tuples inR^i\\hat\{R\}\_\{i\}, and false negatives \(F​NkFN\_\{k\}\) are unmatched tuples inRiR\_\{i\}, with counts pooled across allci∈𝒞c\_\{i\}\\in\\mathcal\{C\}\.

We focus on recall,Rk=T​PkT​Pk\+F​NkR\_\{k\}=\\frac\{TP\_\{k\}\}\{TP\_\{k\}\+FN\_\{k\}\}, as the primary metric, though we also report precision and F1 scores\. For contract scrubbing, the potential costs of false negatives are significantly higher than the cost of false positives, since it is much easier to check whether a flagged issue is real than to identify issues that were not previously known\. Practically, since the CUAD contracts can also contain pre\-existing errors, using recall allows us to focus primarily on the known issues that experts have identified and inserted without being impacted by any potential remaining unknown issues\.

Overall performance is reported as the macro\-average across all\|𝒦\|=9\|\\mathcal\{K\}\|=9categories:

Macro​\-​R=1\|𝒦\|​∑k∈𝒦Rk\.\\small\\mathrm\{Macro\\text\{\-\}R\}=\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\sum\_\{k\\in\\mathcal\{K\}\}R\_\{k\}\.We additionally report aword\-onlyvariant in which location fields are removed from each tuple prior to comparison\. This isolates errors of*identification*– whether the model found the correct term, reference, or party pair – from errors of*localization*, i\.e\., whether it cited the correct section\.

### 3\.4Models

We evaluate a range of proprietary and open\-source models: GPT\-5\.5, GPT\-5\.2, o4\-mini; Claude Opus 4\.7, Claude Sonnet 4\.6, Claude Haiku 4\.5; Qwen 3\.5\-397B; and Gemini 3\.1 pro, Gemini 2\.5 pro, Gemma\-4\-26B\. This selection spans multiple model families and capability tiers – from frontier flagship models to smaller, efficient variants – enabling us to assess how model scale and family affect performance on the legal document scrubbing task\. We provide a full list of the models tested and their hyperparameters in Appendix[F](https://arxiv.org/html/2608.20204#A6)\.

### 3\.5Implementation Details

Each model is prompted to scrub a given contract, with instructions describing each of the nine categories and the output and reference formats prompt \(See the full prompts in Appendix[G](https://arxiv.org/html/2608.20204#A7)\)\. We prompt the models to identify issues from each category in separate instances to help reduce the competing task demands\. Each response is expected to return a single JSON object with one key per categoryk∈𝒦k\\in\\mathcal\{K\}\. Model outputs are parsed into category\-specific tuples – for example,\(term,location\)\(\\textit\{term\},\\textit\{location\}\)for defined terms and\(wrong reference,correct reference,location\)\(\\textit\{wrong reference\},\\textit\{correct reference\},\\textit\{location\}\)for incorrect references\. These tuples are compared deterministically against gold annotationRiR\_\{i\}as multisets, so repeated occurrences are scored independently as the same term or error type may occur multiple times in different locations, and each occurrence imposes a separate burden on a reviewer\.

#### Normalisation before scoring

To avoid penalizing superficial formatting differences, all tuple fields are normalized on both the gold and prediction sides before comparison\. We apply normalization as follow: \(1\)Lower\-casing of Terms:Term and word fields are lowercased and stripped of whitespace \(‘Licensor’ and ‘licensor’ match\)\. \(2\)Location field Canonicalization:Location fields are canonicalized by collapsing parenthetical section levels while preserving major/minor dots: e\.g\.,1\(a\)\(i\)→\\rightarrow1ai,1\.1\(h\)\(vii\)→\\rightarrow1\.1hvii, so1\.1\(d\)≠\\neq11\(d\)\. Special labels such asP/Preamble,R/Recitals,Exhibit X,Schedule X, andSignature Blockare also canonicalized\. \(3\)Symmetric Tuple Matching:Paired\-location categories, such as Conflicting Definitions and Inconsistent Terms, are compared order\-independently: a gold tuple containing locations\(1e, 7a\)is treated as equivalent to a predicted tuple containing\(7a, 1e\)\.

## 4Results

Table 2:Main Results\. Precision \(P\), Recall \(R\), F1 scores, and cost for all evaluated models\. The best score in each column isin bold; the second best isunderlined\. Cost column includes the mean price \($\) and time \(sec\.\) per contract\.ModelRPF1Cost \($ \| s\)GPT\-5\.50\.7500\.5800\.6321\.38 \| 533Gemini 3\.1 Pro0\.7440\.6160\.6550\.19 \| 76Claude Sonnet 4\.60\.6860\.6200\.6371\.52 \| 258Gemini 2\.5 Pro0\.6320\.5270\.5570\.13 \| 48Claude Opus 4\.70\.6160\.6440\.6210\.66 \| 23GPT\-5\.20\.5890\.5400\.5530\.32 \| 192Qwen 3\.5 \(397B\)0\.4380\.2680\.3160\.04 \| 194Claude Haiku 4\.50\.4450\.5920\.4920\.68 \| 108o4\-mini0\.4090\.5480\.4530\.22 \| 301WithoutReasoningGPT\-5\.50\.6430\.4630\.5250\.42 \| 22Claude Opus 4\.70\.5260\.4780\.4900\.42 \| 5Gemma 4 \(26B\)0\.3650\.3150\.3260\.01 \| 26#### Overall

The benchmark is reasonably challenging, with GPT\-5\.5 having a recall score of 0\.750, and the precision and F1 scores across models nearly all less than 0\.650 as shown in Table[2](https://arxiv.org/html/2608.20204#S4.T2)\. Most models have higher recall than precision, but there are tradeoffs between the two scores, with the model having the highest recall, GPT\-5\.5, actually having being fourth in precision\. Focusing on recall, performance drops off quickly from the top few models to the weaker ones\. The most expensive model, GPT\-5\.5, costs only $1\.38 per contract, significantly less than a lawyer or paralegal, though at scale the cost differences to other models may be more relevant\. Runtime is also a consideration, with the top performance of GPT\-5\.5 also requiring the most latency, taking nearly 9 minutes to finish while Gemini 3\.1 Pro returns within 90 seconds\.

As expected, smaller and older models generally lag behind their larger counterparts\. However, Qwen3\.5\-397b \(\.438 recall\) performs poorly despite the large parameter count, comparable to the much smaller o4\-mini \(\.409\) and Haiku 4\.5 \(\.445\)\. This suggests that raw model scale does not straightforwardly translate to legal document analysis ability\.

Table 3:Recall score comparison across models and scrub categories\.Color heatmap applied to all metric rows\. Recall Overall shown without heatmap for reference\. All models use reasoning variants\.MetricModelGPT\-5\.5GeminiClaudeGeminiClaudeGPT\-5\.2ClaudeQwen3\.5o4\-mini3\.1 ProSonnet 4\.62\.5 ProOpus 4\.7Haiku 4\.5\(397B\)Overall Recall0\.7500\.7440\.6860\.6320\.6160\.5890\.4450\.4380\.409Defined Terms0\.9050\.9010\.8620\.8620\.8350\.8760\.7530\.7930\.730Undef\. Capitalized Terms0\.5140\.4400\.3670\.3190\.4470\.5430\.1410\.2160\.171Uncapitalized Defined Terms0\.8680\.8680\.7030\.6620\.5960\.6280\.3060\.3310\.227Incorr\. Capitalized in Context0\.7440\.6900\.5270\.3570\.4810\.4190\.1710\.3490\.101Unused Defined Terms0\.9360\.9310\.8960\.7820\.8710\.8220\.5990\.4850\.703Terms Defined Multiple Times0\.7530\.8350\.8250\.7220\.7630\.6700\.5260\.5770\.526Incorr\. Sec\./Art\./Para\. Refs0\.7600\.8000\.7470\.7330\.7730\.6670\.6870\.5330\.533Incorrect Party References0\.5620\.5690\.5310\.5690\.2000\.0080\.3150\.2000\.308Inconsistent Language0\.7130\.6670\.7130\.6780\.5750\.6670\.5060\.4600\.379

#### Performance by Category

As shown in Table[3](https://arxiv.org/html/2608.20204#S4.T3), model performance varies widely across the different scrub categories\. The gap in recall scores between the easiest \(Defined Terms\) and hardest category \(Undefined Capitalized Terms\) is 0\.484\. Tasks which involve understanding legal relationships between different parts of the contracts tend to be more difficult than those which only require spotting repetitions or omissions\. Easier categories – Defined Terms \(mean recall \.835\), Terms Defined Multiple Times \(\.689\), Unused Defined Terms \(\.781\) – have explicit lexical signals and require less legal reasoning\. In contrast, Incorrect Party References \(\.362\), Incorrect Capitalization in Context \(\.427\), and Undefined Capitalized Terms \(\.351\) require inferring intent from context rather than matching surface form\. Catching capitalized undefined terms requires the ability to flag inconsistencies against an internally maintained registry of definitions in combination with understanding which capitalizations are legally significant\. Even the strongest performing model, GPT\-5\.5, achieves only 0\.514 on Undefined Capitalized Terms and 0\.562 on Party References\.

Correlations \(Pearson’s R\) between the category scores, shown in Figure[3a](https://arxiv.org/html/2608.20204#S4.F3.sf1), are also relatively low, but positive\. There is a weak cluster of categories around incorrect usage of definitions, with uncapitalized defined terms, unused defined terms, incorrectly capitalized terms, and incorrect references forming a group, but overall the different subtasks appear to measure mostly independent capabilities\.

![Refer to caption](https://arxiv.org/html/2608.20204v1/final_sections/figures/clustermap.png)\(a\)Correlation of Subtask Performance\. Task categories show clusters, between those related to definitions, with less correlation between the other categories \(Pearson R\)\.
![Refer to caption](https://arxiv.org/html/2608.20204v1/final_sections/figures/recall_vs_distance_by_model.png)\(b\)Long Distance References\. Recall vs\. reference distance, by model\.

#### Reasoning

As some of the tasks involved in contract scrubbing may appear not to require heavy reasoning, we tested whether turning off reasoning impacts model performance for the top models\. The scores without reasoning are shown at the end of Table[2](https://arxiv.org/html/2608.20204#S4.T2)\. For the two models we tested in both modes, including reasoning increased performance moderately at the cost of slower and more expensive inference\. We include an analysis by issue category in Appendix[B](https://arxiv.org/html/2608.20204#A2)\.

#### Long\-Distance Referencing

In reviewing the errors made by the models, we observed a qualitative tendency to struggle more with identifying incorrect section references that were further away \(in terms of characters\) from the section that they were meant to reference\. As a targeted quantitative assessment, we manually annotated the distance in characters between the references and the referred sections for half of the “Incorrect Section References” gold standard items\. As shown in Figure[3b](https://arxiv.org/html/2608.20204#S4.F3.sf2), the general tendency is that predicted recall decreases as the distance between two target entities grows, with the effect becoming most pronounced past ten thousand characters \(roughly 5\-6 pages\)\.

## 5Discussion

#### Contract scrubbing in deployment

Contract scrubbing is a practical task requiring reasoning over long documents with close attention to detail\. Although the mechanics of the task \(e\.g\. spotting capitalization errors and checking for duplicative definitions\) seem simple, legal knowledge and judgement are still an important aspect\. Such foundational tasks, though routine, are also load\-bearing: errors at this level propagate upward, undermining careful negotiations, expected outcomes, and client relationships\. Full automation of contract scrubbing requires a high bar for model performance, likely to be higher than the 75% recall currently attained\. However, the speed and low cost of all of the evaluated models points to potential for integrating LLMs into existing workflows alongside experts, as well as adding more frequent scrubbing into contract lifecycles to reduce the issues that remain to be found at the end\.

Sources of task difficulty for LLMsThe difficulty of contract scrubbing stems from the simultaneous demands of several core tasks: \(1\)long\-context reasoning, as errors must be detected across lengthy documents; \(2\)structured consistency checking, which requires the model to maintain and query an implicit index of defined terms throughout the document; \(3\)sparse error detection, where errors are rare, demanding high precision to avoid hallucination while retaining sufficient sensitivity to catch true positives; \(4\)heterogeneous error types, meaning a single context includes various types of errors; and \(5\)document\-internal context dependence, where correctness is determined not by external legal standards but by conventions established within the document itself\.

A number of existing benchmarks focus on measuring one or more of these abilities in more generic and isolated contexts, and no benchmark combines all of them, and the scores on our benchmark are lower than might be expected based on other existing benchmarks\. For example, our inconsistent term detection category resembles the inconsistency detection evaluated in FaithEval\[[19](https://arxiv.org/html/2608.20204#bib.bib5)\], where models determine whether a passage contains internal contradictions\. On FaithEval, GPT\-4 and Claude Sonnet 3\.5 achieved 89\.4% and 92\.2%, respectively, while more recent models from the same family achieve only 75\.0% and 68\.6% recall on inconsistency detection inContractScrub\. Similarly, our term\-finding tasks are similar in concept to named entity recognition or needle in a haystack tasks, where models search for specific target words or phrases with long documents\.\[[3](https://arxiv.org/html/2608.20204#bib.bib21)\]evaluate retrieval from long contexts, where Claude 3 achieves 98\.28% at 128K tokens; Gemini 1\.5 likewise reports near\-perfect recall on needle\-in\-a\-haystack probes\[[8](https://arxiv.org/html/2608.20204#bib.bib19)\]\. We do find in our results that performance across the tasks related to defined terms is correlated, but all of the tasks prove to be substantially harder in our setting than standard needle in a haystack\. In addition to requiring many capabilities at once, we expect that the domain specific terminology and structure of legal contracts adds in a layer of difficulty further lowering scores\.

Broader implications

Our findings inContractScrubhighlight two important themes\. First, they demonstrate a disconnect between perceived difficulty of a task within a professional domain and its difficulty for LLMs\. This echoes observations that models can struggle on tasks that appear relatively simple\[[20](https://arxiv.org/html/2608.20204#bib.bib20),[4](https://arxiv.org/html/2608.20204#bib.bib22)\], even when they are capable of passing the SAT or the bar exam\. Building on this, the gap between model performance onContractScruband on existing benchmarks targeting individual capabilities suggests that evaluations of isolated abilities may not fully capture how models perform when those abilities are exercised jointly\. Targeted benchmarks may focus on skills that seem cognitively significant to humans, while potentially overlooking factors that humans find trivial but which actually pose challenges for an LLM\. Our analysis of the relationship between the distance between inconsistencies and model performance highlights one such example, where many models struggled with checking section references across longer spans\. Contract scrubbing also requires following conventions internal to the context of a document rather than external world or domain\-specific knowledge, which we believe is relatively under\-represented in current benchmark efforts\.

#### Limitations

ContractScrubhas a few limitations worth considering\. First, while the benchmark has a large number of issues for the models to identify, they are drawn from only 44 contracts, a more modest scale\. This size is sufficient to surface meaningful performance differences across models, but a larger corpus would reduce the effects of idiosyncrasies in any particular contract and better represent the universe of corporate contracts\. Second, the benchmark’s dataset contains only English\-language contracts with an approximate length of 10–15 pages\. This design choice reflects the practical setting of a closing\-stage scrub in many real use cases, but limits generalisability to other legal traditions, languages, and deal types involving different document lengths such as shorter term sheets or multi\-hundred\-page complex financings\. Third, our evaluation protocol requires models to produce structured JSON output, which may depress raw performance scores relative to a free\-form setting and introduces sensitivity to instruction\-following ability as a confounder\. That said, structured output is arguably a realistic requirement for any production scrubbing tool, where downstream parsing and issue tracking depend on machine\-readable responses; the restriction, therefore, reflects a genuine constraint of the deployment context rather than an arbitrary evaluation choice\.

## 6Conclusion

We introduceContractScrub, a novel evaluation for assessing the ability of LLMs to scrub legal contracts for errors\. We found that performance varied widely among models, with many scoring well on tasks that are primarily lexical in nature and scoring poorly when the tasks require more complex legal reasoning on top of contextual document analysis skills\. These findings contrast with results on related general\-domain benchmarks, where frontier models are generally very capable\. Our results reinforce the case for narrowly targeted, ecologically valid, benchmarks in professional domains\. As AI systems are increasingly deployed in professional settings, benchmarks that directly measure economically relevant, task\-specific performance are important for both scientific progress and responsible deployment\. We offerContractScrubas one step in that direction\.

## References

- \[1\]A\. M\. Bean, R\. O\. Kearns, A\. Romanou, F\. S\. Hafner, H\. Mayne, J\. Batzner, N\. Foroutan, C\. Schmitz, K\. Korgul, H\. Batra, O\. Deb, E\. Beharry, C\. Emde, T\. Foster, A\. Gausen, M\. Grandury, S\. Han, V\. Hofmann, L\. Ibrahim, H\. Kim, H\. R\. Kirk, F\. Lin, G\. K\. Liu, L\. Luettgau, J\. Magomere, J\. Rystrøm, A\. Sotnikova, Y\. Yang, Y\. Zhao, A\. Bibi, A\. Bosselut, R\. Clark, A\. Cohan, J\. Foerster, Y\. Gal, S\. A\. Hale, I\. D\. Raji, C\. Summerfield, P\. H\. S\. Torr, C\. Ududec, L\. Rocher, and A\. Mahdi\(2025\)Measuring what matters: construct validity in large language model benchmarks\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2511.04703),2511\.04703,ISSN 2331\-8422,[Link](https://arxiv.org/abs/2511.04703)Cited by:[§1](https://arxiv.org/html/2608.20204#S1.p1.1)\.
- \[2\]I\. Chalkidis, A\. Jana, D\. Hartung, M\. Bommarito, I\. Androutsopoulos, D\. Katz, and N\. Aletras\(2022\)LexGLUE: a benchmark dataset for legal language understanding in English\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 4310–4330\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.297),[Link](https://aclanthology.org/2022.acl-long.297/)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]P\. Chen, H\. Jin, C\. Lee, R\. Shao, J\. Yang, M\. Zhao, Z\. Zhang, Q\. Lu, K\. Men, N\. Xie, H\. Li, B\. Yin, H\. Li, and L\. Wang\(2025\)LongLeader: a comprehensive leaderboard for large language models in long\-context scenarios\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 8734–8750\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.439),ISBN 979\-8\-89176\-189\-6,[Link](https://aclanthology.org/2025.naacl-long.439/)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.20204#S5.SS0.SSS0.Px1.p3.1)\.
- \[4\]F\. Dell’Acqua, E\. McFowland III, E\. R\. Mollick, H\. Lifshitz\-Assaf, K\. Kellogg, S\. Rajendran, L\. Krayer, F\. Candelon, and K\. R\. Lakhani\(2023\)Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality\.SSRN Scholarly Paper,Social Science Research Network,Rochester, NY\.External Links:[Document](https://dx.doi.org/10.2139/ssrn.4573321),4573321Cited by:[§5](https://arxiv.org/html/2608.20204#S5.SS0.SSS0.Px1.p5.1)\.
- \[5\]Y\. Fan, J\. Ni, J\. Merane, Y\. Tian, Y\. Hermstrüwer, Y\. Huang, M\. Akhtar, E\. Salimbeni, F\. Geering, O\. Dreyer,et al\.\(2025\)Lexam: benchmarking legal reasoning on 340 law exams\.Cited by:[§1](https://arxiv.org/html/2608.20204#S1.p1.1),[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]Z\. Fei, X\. Shen, D\. Zhu, F\. Zhou, Z\. Han, A\. Huang, S\. Zhang, K\. Chen, Z\. Yin, Z\. Shen,et al\.\(2024\)Lawbench: benchmarking legal knowledge of large language models\.InProceedings of the 2024 conference on empirical methods in natural language processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),pp\. 7933–7962\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.452.pdf)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]E\. Felten, M\. Raj, and R\. Seamans\(2023\)How will language modelers like chatgpt affect occupations and industries?\.External Links:2303\.01157,[Link](https://arxiv.org/abs/2303.01157)Cited by:[§1](https://arxiv.org/html/2608.20204#S1.p1.1)\.
- \[8\]Google Cloud\(2024\)The needle in the haystack test and how Gemini 1\.5 Pro solves it\.Note:Google Cloud BlogCited by:[§5](https://arxiv.org/html/2608.20204#S5.SS0.SSS0.Px1.p3.1)\.
- \[9\]N\. Guha, J\. Nyarko, D\. Ho, C\. Ré, A\. Chilton, A\. Chohlas\-Wood, A\. Peters, B\. Waldon, D\. Rockmore, D\. Zambrano,et al\.\(2023\)Legalbench: a collaboratively built benchmark for measuring legal reasoning in large language models\.Advances in neural information processing systems36,pp\. 44123–44279\.External Links:[Document](https://dx.doi.org/10.52202/075280-1915)Cited by:[§1](https://arxiv.org/html/2608.20204#S1.p1.1),[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]D\. Hendrycks, C\. Burns, A\. Chen, and S\. Ball\(2021\)CUAD: an expert\-annotated nlp dataset for legal contract review\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,J\. Vanschoren and S\. K\. Yeung \(Eds\.\),External Links:[Document](https://dx.doi.org/10.48550/arxiv.2103.06268),[Link](https://arxiv.org/pdf/2103.06268)Cited by:[§1](https://arxiv.org/html/2608.20204#S1.p4.1),[§1](https://arxiv.org/html/2608.20204#S1.p5.1),[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p2.1),[§3\.2](https://arxiv.org/html/2608.20204#S3.SS2.SSS0.Px2.p1.1)\.
- \[11\]C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. Ginsburg\(2024\)RULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px2.p1.1)\.
- \[12\]G\. Kamradt\(2023\)LLMTest\_NeedleInAHaystack: pressure testing LLMs\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)GitHub repositoryCited by:[§1](https://arxiv.org/html/2608.20204#S1.p4.1),[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]Y\. Koreeda and C\. Manning\(2021\)ContractNLI: a dataset for document\-level natural language inference for contracts\.InFindings of the Association for Computational Linguistics: EMNLP 2021,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Punta Cana, Dominican Republic,pp\. 1907–1919\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.164),[Link](https://aclanthology.org/2021.findings-emnlp.164/)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p2.1)\.
- \[14\]M\. Lane and A\. Saint\-Martin\(2021\)The impact of artificial intelligence on the labour market: what do we know so far?\.Organisation for Economic Co\-Operation and Development \(OECD\)\.External Links:[Document](https://dx.doi.org/10.1787/7c895724-en),ISSN 1815\-199XCited by:[§1](https://arxiv.org/html/2608.20204#S1.p1.1)\.
- \[15\]S\. Leivaditi, J\. Rossi, and E\. Kanoulas\(2020\)A benchmark for lease contract review\.arXiv preprint arXiv:2010\.10386abs/2010\.10386\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2010.10386),ISSN 2331\-8422,[Link](https://arxiv.org/pdf/2010.10386)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p2.1)\.
- \[16\]Z\. Ling, K\. Liu, K\. Yan, Y\. Yang, W\. Lin, T\. Fan, L\. Shen, Z\. Du, and J\. Chen\(2025\)Longreason: a synthetic long\-context reasoning benchmark via context expansion\.arXiv preprint arXiv:2501\.15089\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2501.15089),ISSN 2331\-8422,[Link](https://arxiv.org/pdf/2501.15089)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px2.p1.1)\.
- \[17\]S\. Liu, Z\. Li, R\. Ma, H\. Zhao, and M\. Du\(2025\)ContractEval: benchmarking LLMs for clause\-level legal risk identification in commercial contracts\.InProceedings of the Natural Legal Language Processing Workshop 2025,N\. Aletras, I\. Chalkidis, L\. Barrett, C\. Goanță, D\. Preoțiuc\-Pietro, and G\. Spanakis \(Eds\.\),Suzhou, China,pp\. 291–291\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.nllp-1.19),ISBN 979\-8\-89176\-338\-8,[Link](https://aclanthology.org/2025.nllp-1.19/)Cited by:[§1](https://arxiv.org/html/2608.20204#S1.p4.1),[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p2.1)\.
- \[18\]K\. Manikantan, M\. Tapaswi, V\. Gandhi, and S\. Toshniwal\(2025\)IdentifyMe: a challenging long\-context mention resolution benchmark for LLMs\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 768–777\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-short.64),ISBN 979\-8\-89176\-190\-2,[Link](https://aclanthology.org/2025.naacl-short.64/)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px2.p1.1)\.
- \[19\]Y\. Ming, S\. Purushwalkam, S\. Pandit, Z\. Ke, X\. Nguyen, C\. Xiong, and S\. Joty\(2025\)Faitheval: can your language model stay faithful to context, even if" the moon is made of marshmallows"\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 29430–29456\.Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.20204#S5.SS0.SSS0.Px1.p3.1)\.
- \[20\]M\. Nezhurina, L\. Cipolina\-Kun, M\. Cherti, and J\. Jitsev\(2024\)Alice in wonderland: simple tasks showing complete reasoning breakdown in state\-of\-the\-art large language models\.arXiv preprint arXiv:2406\.02061\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2406.02061),[Link](https://arxiv.org/pdf/2406.02061)Cited by:[§5](https://arxiv.org/html/2608.20204#S5.SS0.SSS0.Px1.p5.1)\.
- \[21\]R\. Schwartz, R\. Chowdhury, A\. Kundu, H\. Frase, M\. Fadaee, T\. David, G\. Waters, A\. Taik, M\. Briggs, P\. Hall,et al\.\(2025\)Reality check: a new evaluation ecosystem is necessary to understand ai’s real world effects\.arXiv preprint arXiv:2505\.18893\.Cited by:[§1](https://arxiv.org/html/2608.20204#S1.p1.1)\.
- \[22\]S\. Wang, A\. Scardigli, L\. Tang, W\. Chen, D\. Levkin, A\. Chen, S\. Ball, T\. Woodside, O\. Zhang, and D\. Hendrycks\(2023\)MAUD: an expert\-annotated legal NLP dataset for merger agreement understanding\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 16369–16382\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.1019),[Link](https://aclanthology.org/2023.emnlp-main.1019/)Cited by:[§2](https://arxiv.org/html/2608.20204#S2.SS0.SSS0.Px1.p2.1)\.

Appendix

PART 1Additional Context and ExperimentsAPotential Implications of Each Error Category[4](https://arxiv.org/html/2608.20204#A1.T4)BReasoning Ablation[B](https://arxiv.org/html/2608.20204#A2)CTerm\-only Scoring Ablation[C](https://arxiv.org/html/2608.20204#A3)DPer\-category Precision & Recall[D](https://arxiv.org/html/2608.20204#A4)PART 2Reproducibility DetailsEData Access and Usage Notes[E](https://arxiv.org/html/2608.20204#A5)FInference Details[F](https://arxiv.org/html/2608.20204#A6)GPrompt Templates[G](https://arxiv.org/html/2608.20204#A7)
PART 1: Additional Context and Experiments

## Appendix APotential implications of each error category

Table 4:Scrub error category counts inContractScruband per\-contract averagesCategoryTotalAvg/ContractDefined Terms1,50534\.2Undefined Capitalized Terms68915\.7Uncapitalized Defined Terms3177\.2Unused Defined Terms2024\.6Incorrect Section, Article, or Paragraph References1503\.4Incorrect Party References1303\.0Incorrectly Capitalized Terms In Context1292\.9Terms Defined Multiple Times972\.2Inconsistent Language872\.0Each category has different potential implications for the parties agreeing to the contract\. Not all consequences are strictly "legal" in nature such that a law is broken or a contract is breached\. Courts may, in certain circumstances, correct obvious clerical or scrivener’s errors where the parties’ mutual intent is evident from the surrounding context\. Nevertheless, such errors remain undesirable, and those that introduce substantive ambiguity are of greater legal concern\. From a practical standpoint, however, the threshold for harm is lower\. Any error that delays deal execution or introduces friction between parties, including ostensibly minor clerical mistakes that create confusion about the operative terms, represents a failure with real professional consequences, up to and including reputational damage and client attrition\.

Below, we describe what each category represents and provide examples of possible impacts that could arise from missing this type of error:

Undefined Capitalized Terms: Terms that are capitalized or otherwise treated as a defined term in an agreement but not formally defined\. For example, an agreement provides: “Party A shall comply with all applicable requirements in the Approved Specification,” but Approved Specification is not defined\. This matters because an undefined capitalized term indicates that a contracting party specifically intended a special contractual meaning\. As a result, an ambiguity is created and a party may later dispute what specification was approved and whether it was applicable, whether a breach occurred, and whether outside evidence is admissible to contradict, vary, or add to the terms of the agreement\.

Uncapitalized Defined TermsFormally defined terms appearing in lowercase when they should be capitalized\. For example, defining “Representative” to mean only company officers and legal counsel, rather than the everyday meaning of anyone acting on another’s behalf\. If the confidentiality section of an agreement then says that a party can disclose confidential information to “representatives” in lowercase, a counterparty may treat the inconsistency as intentional and apply the broader, everyday definition\. The result is that the other party could be permitted to share sensitive information with a much wider group of people than was intended\.Incorrectly Capitalized Terms in ContextTerms that have a definition but are capitalized in a context where they are not being used as a defined term\. For example, an agreement provides: “‘Services’ shall mean the software implementation services provided by Vendor to Customer as described in Exhibit A\.” The agreement later provides that: “Vendor shall not provide similar Services to any competitor of Customer\.” This matters because the later capitalized term can change the scope of a covenant, restriction, exclusion, or permission\. Here, the later erroneous capitalization improperly restricts what should be “services” to the implementation services in Exhibit A whereas a drafter could have intended a broader scope of similar services generally provided by the Vendor\.Unused Defined TermsDefined terms that appear only once in the context where they are defined and are not used elsewhere in the agreement\. For example, an agreement provides: “‘Change Order’ means additional or different specifications from the project terms set out in Scope of Construction\.” However, the term “Change Order” never appears again in the agreement\. Because courts aim to avoid any interpretation of contractual language that renders it surplusage, disputes over intent and scope of additional or different specifications can arise if a party attributes meaning to the stranded definition and can invite an opportunity to seek the admission of extrinsic evidence to provide competing interpretations of the agreement\.Terms Defined Multiple TimesTerms that are defined more than one time in an agreement with conflicting or inconsistent definitions\. A term that is merely repeated with the same meaning is NOT an error — only flag a term when its multiple definitions genuinely conflict\. For example, if the Effective Date is defined in two places with conflicting dates in an agreement, the parties may disagree about which one controls\. Consider a company hired to manage a property that is liable for any accidents occurring while the agreement is in effect\. If the agreement defines the Effective Date as both March 1 and March 30, and an accident occurs on March 15, it becomes genuinely unclear who is responsible for that incident, the property owner or the management company\. What should be a straightforward question can turn into a costly dispute\.Incorrect Section, Article, or Paragraph ReferencesInternal cross\-references that assign an incorrect section, article, or paragraph of the agreement\. Agreements are often reorganized during negotiation, with sections added, removed, or reordered\. If an internal reference isn’t updated to reflect those changes, it may end up pointing to the wrong section entirely\. For example, suppose an agreement states that a party will face enhanced damages for breaches of “Section 3,” which covers confidentiality\. During negotiation, the sections are reshuffled and Section 3 now covers product liability instead, but the reference is never updated\. That party is now potentially exposed to enhanced damages for product liability incidents, a potentially broader and more expensive risk\.Incorrect Party ReferencesInstances where a party is referred to by the wrong party name or role, such as “Licensor” instead of “Licensee” or “Receiving Party” instead of “Disclosing Party”\. Referencing the wrong party name in a contract can shift responsibilities in ways that a party may not want\. If a clause requires a specific action, such as paying for shipping, but names the wrong party, the obligation could fall on that party regardless of what was originally discussed during contract negotiations\. This type of error can end up costing a party time and money and potentially lead to a dispute over who is actually responsible\.Inconsistent LanguageLanguage in the agreement that directly contradicts itself or other language elsewhere in the agreement\. When two provisions in a contract directly contradict each other, it creates uncertainty about which one actually applies\. For example, if one section states that payment is due 30 days after receiving an invoice, but another states 45 days, neither party can be fully confident about when payment is expected\. For the seller, this kind of ambiguity can complicate cash flow planning; they may be counting on payment at 30 days, while the buyer believes they have until 45\. What starts as a drafting oversight can quickly become a source of friction or dispute\.
![Refer to caption](https://arxiv.org/html/2608.20204v1/final_sections/figures/recall_by_category.png)Figure 4:Recall by categories for two models with and without reasoning\. Solid lines show per\-section recall across nine defined\-term and reference\-checking tasks; dashed horizontal lines indicate each condition’s mean\.
## Appendix BReasoning Ablation

As some of the tasks involved in contract scrubbing may appear not to require heavy thinking, we tested whether turning off reasoning impacts model performance for the top models\. The scores with and without reasoning are shown in Figure[4](https://arxiv.org/html/2608.20204#A1.F4)\.

Across the two models tested, enabling reasoning markedly improved performance for Claude Opus 4\.7 \(\+0\.090 avg\. recall\) and GPT\-5\.5 \(\+0\.107\)\. The effect was particularly pronounced for Unused Defined Term category and Uncapitlized Defined Term categories\. Although these categories rely on lexical signals rather than deep legal interpretation, verifying them requires holistic reasoning over the full document rather than local pattern\-matching, which likely explains why reasoning\-enabled variants benefit more\. By contrast, categories that depend on inferring intent, for example, Incorrect Party References, saw little to no improvement from reasoning, suggesting that current reasoning traces help with structured cross\-referencing more than with the deeper semantic judgments these categories demand in this task\. For this reason, we report the main results with reasoning enabled\.

## Appendix CTerm\-only Scoring Ablation

To distinguish between failure to identify issues in each contract and failure to provide their locations, we conducted a scoring ablation\. Rather than multiset comparison across tuples fromRiR\_\{i\}andR^i\\hat\{R\}\_\{i\}, we limit the comparison to the categories,κ\\kappa, and the elements of𝐟\\mathbf\{f\}which are terms, allowing mismatches in the locations\.

By construction, scores matching only on the terms are higher than the scores requiring both term and location matches \(Table[5](https://arxiv.org/html/2608.20204#A3.T5)\)\. The differences are mostly on the order of five percentage points, though Qwen 3\.5 sees much larger improvements and actually passes Claude 4\.5 Haiku, indicating that it struggles more with location references within the document than the other models\.

Table 5:Recall score comparison across models and scrub categories\.Color heatmap applied to all metric rows\. Recall Overall shown without heatmap for reference\. All models use reasoning variants\.MetricModelGPT\-5\.5GeminiClaudeGeminiClaudeGPT\-5\.2ClaudeQwen3\.5o4\-mini3\.1 ProSonnet 4\.6 ‡2\.5 ProOpus 4\.7Haiku 4\.5\(397B\)Overall Recall0\.7990\.7930\.7460\.6980\.6610\.6310\.4780\.5290\.466Defined Terms0\.9400\.9410\.9190\.9440\.8900\.9330\.8090\.8770\.791Undef\. Capitalized Terms0\.5520\.4600\.3920\.3470\.5050\.5820\.1650\.2530\.205Uncapitalized Defined Terms0\.9240\.9150\.7760\.7760\.6560\.7070\.3690\.5330\.297Incorr\. Capitalized in Context0\.7980\.7360\.6200\.3950\.5190\.4810\.2170\.5190\.140Unused Defined Terms0\.9600\.9550\.9410\.8370\.9060\.8760\.6290\.5400\.772Terms Defined Multiple Times0\.7940\.8450\.8660\.7940\.7940\.7530\.5880\.6800\.608Incorr\. Sec\./Art\./Para\. Refs0\.7930\.8400\.8200\.8130\.8070\.7130\.7270\.5730\.580Incorrect Party References0\.6310\.6540\.6310\.6770\.2080\.0080\.3230\.2540\.338

## Appendix DPer\-category Precision & F1

We provide per\-categoy precision and F1 performance of all models we tested for completeness\.

### D\.1Per\-category Precision

Table 6:Precision score comparison across models and scrub categories\.Color heatmap applied to all metric rows\. Precision Overall shown without heatmap for reference\.MetricModelGPT\-5\.5GeminiClaudeGeminiClaudeGPT\-5\.2ClaudeQwen3\.5o4\-mini3\.1 ProSonnet 4\.6 ‡2\.5 ProOpus 4\.7Haiku 4\.5\(397B\)Overall Precision0\.5800\.6160\.6200\.5270\.6440\.5400\.5920\.2680\.548Defined Terms0\.8780\.9090\.9010\.8600\.9160\.8990\.8900\.5750\.906Undef\. Capitalized Terms0\.6320\.5960\.6340\.5640\.6150\.5700\.6180\.1850\.608Uncapitalized Defined Terms0\.3270\.2990\.4080\.3250\.5430\.3750\.4060\.0730\.369Incorr\. Capitalized in Context0\.4900\.5560\.5230\.3540\.7050\.4700\.3550\.0190\.232Unused Defined Terms0\.7470\.7520\.6700\.6400\.8040\.7310\.6650\.3590\.582Terms Defined Multiple Times0\.7770\.8710\.8160\.6730\.8410\.7650\.7970\.4630\.797Incorr\. Sec\./Art\./Para\. Refs0\.7450\.6820\.7570\.6110\.8290\.7250\.7860\.3620\.734Incorrect Party References0\.3760\.5480\.4960\.4430\.2280\.0070\.4230\.1700\.377Inconsistent Language0\.2460\.3310\.3780\.2690\.3120\.3140\.3930\.2070\.327

### D\.2Per\-category F1

Table 7:F1 score comparison across models and scrub categories\.Color heatmap applied to all metric rows\. F1 Overall shown without heatmap for reference\.MetricModelGPT\-5\.5GeminiClaudeGeminiClaudeGPT\-5\.2ClaudeQwen3\.5o4\-mini3\.1 ProSonnet 4\.6 ‡2\.5 ProOpus 4\.7Haiku 4\.5\(397B\)Overall F10\.6320\.6550\.6370\.5570\.6210\.5530\.4920\.3160\.453Defined Terms0\.8910\.9050\.8810\.8610\.8730\.8880\.8160\.6670\.809Undef\. Capitalized Terms0\.5670\.5060\.4650\.4080\.5180\.5560\.2290\.1990\.267Uncapitalized Defined Terms0\.4750\.4450\.5170\.4360\.5680\.4700\.3490\.1190\.281Incorr\. Capitalized in Context0\.5910\.6160\.5250\.3550\.5710\.4430\.2300\.0360\.141Unused Defined Terms0\.8310\.8320\.7670\.7040\.8360\.7740\.6300\.4130\.637Terms Defined Multiple Times0\.7640\.8530\.8210\.6970\.8000\.7140\.6340\.5140\.634Incorr\. Sec\./Art\./Para\. Refs0\.7520\.7360\.7520\.6670\.8000\.6940\.7330\.4310\.618Incorrect Party References0\.4510\.5580\.5130\.4980\.2130\.0080\.3610\.1840\.339Inconsistent Language0\.3660\.4430\.4940\.3860\.4050\.4260\.4420\.2860\.351

PART 2: Reproducibility Details

## Appendix EData Access and Usage Notes

The dataset will be made publicly available upon acceptance\. The CUAD dataset is used under a CC\-BY\-4\.0 license\.

## Appendix FInference Details

We evaluated 10 models on the contract scrub benchmark\. Details regarding hyperparameters and compute resources are listed in Table[8](https://arxiv.org/html/2608.20204#A6.T8)\. Empty cells indicate that an option is not available for a particular model\.

Table 8:Model Inference DetailsNameProviderTemperatureReasoning EffortClaude Opus 4\.7Bedrock–HighClaude Sonnet 4\.6Bedrock0\.6HighClaude Haiku 4\.5Bedrock0\.6HighGPT 5\.5OpenAI–MediumGPT 5\.2OpenAI–Mediumo4 miniAzure0\.6–Qwen 3\.5 397B A17BSelf\-hosted0\.6–Gemini 3\.1 ProVertex AI0\.6–Gemini 2\.5 ProVertex AI0\.6–Gemma 4 26B A4BVertex AI0\.6–
## Appendix GPrompt Templates

We provide the full prompt text used to test each model in the main results\. The prompts for each of the nine categories were built from an overall prompt template, with a targeted section inserted for each category\. Each category has four items, including ‘title’, ‘definition’, ‘instruction’ and ‘schema’\.

Overall Scoring Prompt TemplateYou are a legal editor reviewing an agreement for drafting errors and issues\. Review the agreement below and identify ONLY the following category: \{title\} \- \{definition\}\{instructions\}If you find no items in this category, respond with an empty list\. Follow these rules for the location format\. This is critical: \- Always give the most specific sub\-section reference, NEVER just the parent section number\.\- Concatenate sub\-section letters/numerals together; preserve dots between major\.minor section numbers\. \- If a defined term appears in section 1\(a\), the location is "1a", NOT "1"\. \- If something is in section 11\(b\)\(i\), the location is "11bi", NOT "11" or "11b"\. \- If something is in section 7\(a\)\(i\), the location is "7ai", NOT "7" or "7a"\. \- If something is in Attachment X Section Y, the location is "attachment xy", NOT "attachment x\.y"\. \- For nested numbering: section 1\.1\(d\) is "1\.1d", section 3\.7\(a\) is "3\.7a", section 1\.1\(h\)\(vii\) is "1\.1hvii"\. \- Use "P" for the Preamble \(the introductory section that identifies the parties, date and general background\)\. \- Use "Recitals" for the WHEREAS clauses; if the item is in a specific recital clause, append its letter/number, e\.g\. "Recitals c" for recital \(c\)\. \- Use "Exhibit A", "Exhibit B" and "Schedule A", "Schedule B" etc\. for exhibits and schedules\. \- Use "Signature Block" for the section where parties execute the agreement\. Respond with JSON only \- no explanation, no markdown fences\. Use this exact schema:\{schema\}

Defined Termstitle:Defined Terms definition:Words or phrases that are formally defined in the agreement with a designated meaning within the context of the agreement, often set out as a word or words that are capitalized, underlined, bolded, italicized, or in quotations\. instruction:For each term, provide the section where it is defined\. schema:\{"defined\_terms": \[\{"term": "…", "location": "…"\}\]\}

Undefined Capitalized Termstitle:Undefined Capitalized Terms definition:Terms that are capitalized or otherwise treated as a defined term in an agreement but not formally defined\. For example, an agreement provides: “Party A shall comply with all applicable requirements in the Approved Specification,” but Approved Specification is not defined\. This matters because an undefined capitalized term indicates that a contracting party specifically intended a special contractual meaning\. As a result, an ambiguity is created and a party may later dispute what specification was approved and whether it was applicable, whether a breach occurred, and whether outside evidence is admissible to contradict, vary, or add to the terms of the agreement\.However, the following shall NOT be flagged as undefined capitalized terms:1\.Terms of art — words or terms with a specific, precise, technical, and specialized meaning commonly understood and used within a particular field, profession, or discipline that is germane or applicable to the contract\.2\.Proper nouns used in common parlance — names of specific people, places, brands, or things that have been adopted into everyday, informal speech and are widely understood by the general population, even outside of their original specialized or formal context\.3\.Morphological variants of defined terms — instances where a defined term appears in a grammatically inflected form that is not itself defined\.4\.Scientific and technical nomenclature — standardized words or terms with a specific, precise, and specialized meaning commonly understood and used in scientific and technical fields\.5\.References to agreement sections, exhibits, and schedules\.6\.The title of the agreement and the names of any associated documents as they appear in the agreement’s header or title section\.7\.Titles, roles, and positions\.instruction:List each occurrence separately with the section where each term is located\.schema:\{"undefined\_capitalized\_terms": \[\{"term": "…", "location": "…"\}\]\}

Uncapitalized Defined Termstitle:Uncapitalized Defined Termsdefinition:Formally defined terms appearing in lowercase when they should be capitalized\. For example, defining “Representative” to mean only company officers and legal counsel, rather than the everyday meaning of anyone acting on another’s behalf\. If the confidentiality section of an agreement then says that a party can disclose confidential information to “representatives” in lowercase, a counterparty may treat the inconsistency as intentional and apply the broader, everyday definition\. The result is that the other party could be permitted to share sensitive information with a much wider group of people than was intended\.instruction:List each occurrence separately with the section where each term is located\.schema:\{"uncapitalized\_defined\_terms": \[\{"term": "…", "location": "…"\}\]\}

Incorrectly Capitalized Terms In Contexttitle:Incorrectly Capitalized Terms In Contextdefinition:Terms that have a definition but are capitalized in a context where they are not being used as a defined term\. For example, an agreement provides: “‘Services’ shall mean the software implementation services provided by Vendor to Customer as described in Exhibit A\.” The agreement later provides that: “Vendor shall not provide similar Services to any competitor of Customer\.” This matters because the later capitalized term can change the scope of a covenant, restriction, exclusion, or permission\. Here, the later erroneous capitalization improperly restricts what should be “services” to the implementation services in Exhibit A whereas a drafter could have intended a broader scope of similar services generally provided by the Vendor\.instruction:List each occurrence separately with the section where each term is located\.schema:\{"incorrectly\_capitalized\_terms\_in\_context": \[\{"term": "…", "location": "…"\}\]\}

Unused Defined Termstitle:Unused Defined Termsdefinition:Defined terms that appear only once in the context where they are defined and are not used elsewhere in the agreement\. For example, an agreement provides: “‘Change Order’ means additional or different specifications from the project terms set out in Scope of Construction\.” However, the term “Change Order” never appears again in the agreement\. Because courts aim to avoid any interpretation of contractual language that renders it surplusage, disputes over intent and scope of additional or different specifications can arise if a party attributes meaning to the stranded definition and can invite an opportunity to seek the admission of extrinsic evidence to provide competing interpretations of the agreement\.instruction:List each occurrence separately with the section where each term is located\.schema:\{"unused\_defined\_terms": \[\{"term": "\.\.\.", "location": "\.\.\."\}\]\}

Terms Defined Multiple Timestitle:Terms Defined Multiple Timesdefinition:Terms that are defined more than one time in an agreement with conflicting or inconsistent definitions\. A term that is merely repeated with the same meaning is NOT an error — only flag a term when its multiple definitions genuinely conflict\. For example, if the Effective Date is defined in two places with conflicting dates in an agreement, the parties may disagree about which one controls\. Consider a company hired to manage a property that is liable for any accidents occurring while the agreement is in effect\. If the agreement defines the Effective Date as both March 1 and March 30, and an accident occurs on March 15, it becomes genuinely unclear who is responsible for that incident, the property owner or the management company\. What should be a straightforward question can turn into a costly dispute\.instruction:List each occurrence separately along with each of the sections where the multiple definitions are located\.schema:\{"terms\_defined\_multiple\_times": \[\{"term": "…", "location1": "…", "location2": "…"\}\]\}

Incorrect Section, Article, or Paragraph Referencestitle:Incorrect Section, Article, or Paragraph Referencesdefinition:Internal cross\-references that assign an incorrect section, article, or paragraph of the agreement\. Agreements are often reorganized during negotiation, with sections added, removed, or reordered\. If an internal reference isn’t updated to reflect those changes, it may end up pointing to the wrong section entirely\. For example, suppose an agreement states that a party will face enhanced damages for breaches of “Section 3,” which covers confidentiality\. During negotiation, the sections are reshuffled and Section 3 now covers product liability instead, but the reference is never updated\. That party is now potentially exposed to enhanced damages for product liability incidents, a potentially broader and more expensive risk\.instruction:For each, list the incorrect reference used, the correct reference, and the section where the error is located\. Use only the reference identifier \(e\.g\. “9\.1”\), without descriptive section names\. List each reference as a SEPARATE item: if one cross\-reference covers several sections, report each pair on its own \(e\.g\. “9\.1 or 9\.2→\\rightarrow10\.1 or 10\.2” becomes two items, 9\.1→\\rightarrow10\.1 and 9\.2→\\rightarrow10\.2\), and if the same error recurs in multiple places, report each location separately\.schema:\{"incorrect\_section\_article\_paragraph\_references": \[\{"wrong": "…", "correct": "…", "location": "…"\}\]\}

Incorrect Party Referencestitle:Incorrect Party Referencesdefinition:Instances where a party is referred to by the wrong party name or role, such as “Licensor” instead of “Licensee” or “Receiving Party” instead of “Disclosing Party”\. Referencing the wrong party name in a contract can shift responsibilities in ways that a party may not want\. If a clause requires a specific action, such as paying for shipping, but names the wrong party, the obligation could fall on that party regardless of what was originally discussed during contract negotiations\. This type of error can end up costing a party time and money and potentially lead to a dispute over who is actually responsible\.instruction:List each item identifying the incorrect party reference, the correct party reference, and the section where the error is located\.schema:\{"incorrect\_party\_references": \[\{"wrong": "…", "correct": "…", "location": "…"\}\]\}

Inconsistent Languagetitle:Inconsistent Languagedefinition:Language in the agreement that directly contradicts itself or other language elsewhere in the agreement\.When two provisions in a contract directly contradict each other, it creates uncertainty about which one actually applies\. For example, if one section states that payment is due 30 days after receiving an invoice, but another states 45 days, neither party can be fully confident about when payment is expected\. For the seller, this kind of ambiguity can complicate cash flow planning; they may be counting on payment at 30 days, while the buyer believes they have until 45\. What starts as a drafting oversight can quickly become a source of friction or dispute\.instruction:For each occurrence, identify both sections where the conflicting language appears\. schema:\{"inconsistent\_terms": \[\{"section1": "…", "section2": "…"\}\]\}