Redesigning and Auditing Deep Research Writing for Faithful Reports

# Redesigning and Auditing Deep Research Writing for Faithful Reports
Source: [https://arxiv.org/html/2608.28643](https://arxiv.org/html/2608.28643)
###### Abstract

Rubric\-based evaluations of deep\-research \(DR\) systems often obscure fine\-grained factual failures in generated reports\. We introduceClaimProbe, a claim\-level audit that decomposes DR reports into claims and measures hallucination, misattribution, citation hygiene, and necessary\-fact recall against retrieved evidence\. UsingClaimProbe, we find that strong DR pipelines can omit key evidence and misattribute claims even when their rubric scores remain stable\. We then proposeClaimWriter, a hierarchical claim\-based writer that extracts source facts, maps them to a query\-derived outline, and drafts each section from a source\-linked claim representation\. Across three prior DR frameworks, replacing only the report writer withClaimWriterreduces hallucination by2\.6×2\.6\\times–4\.5×4\.5\\timesand improves necessary\-fact recall by1\.2×1\.2\\times–1\.7×1\.7\\times, while largely preserving overall report quality\.ClaimWriteralso enables localized revision: when sources change, it propagates changed source facts into revised reports at the highest rate among update methods, while also being more cost\-effective\.

## 1Introduction

Deep\-research \(DR\) systems are agentic pipelines that search across heterogeneous sources, extract evidence, reason over that evidence, and synthesize long\-form reports with citations\([15](https://arxiv.org/html/2608.28643#bib.bib12)\)\. These systems are increasingly used in enterprise workflows such as sales pitch generation, market and competitor analysis, and customer research\([3](https://arxiv.org/html/2608.28643#bib.bib1)\)\. In these settings, the final report is the primary artifact used by downstream decision makers\. A useful report must therefore do three things simultaneously: include the decision\-relevant facts available in the collected evidence, attribute claims to the right sources, and remain maintainable when only a small part of the underlying evidence changes\.

Existing DR benchmarks primarily score final reports with holistic rubrics\([4](https://arxiv.org/html/2608.28643#bib.bib2);[20](https://arxiv.org/html/2608.28643#bib.bib6);[18](https://arxiv.org/html/2608.28643#bib.bib18)\)\. While useful for overall quality, these scores often hide claim\-level failures: reports may omit key facts, cite sources that do not support their claims, or introduce unsupported inferences\. In enterprise DR, such errors are consequential, yet rubric scores provide little visibility into whether claims are supported, correctly attributed, and complete with respect to the available evidence\([8](https://arxiv.org/html/2608.28643#bib.bib13);[11](https://arxiv.org/html/2608.28643#bib.bib10);[19](https://arxiv.org/html/2608.28643#bib.bib4)\)\.

We address this gap withClaimProbe, a claim\-level auditing framework for DR reports\.ClaimProbedecomposes each report into claims, checks them against the collected source evidence, and measures four dimensions:hallucination,misattribution,citation hygiene, andnecessary\-fact recall\. It then aggregates these judgments into report\-level metrics while preserving claim\-level diagnostics, making it possible to identify which claims failed and why\. The evaluator is an LLM judge calibrated through a three\-track human study and grounded in explicit evidence comparisons\([22](https://arxiv.org/html/2608.28643#bib.bib11)\)\. UsingClaimProbe, we find that strong DR pipelines still miss key facts, misattribute evidence, and introduce unsupported claims despite producing fluent reports\. These failures point to a limitation of direct report generation from retrieved evidence: facts can lose structure and provenance during writing\.

Motivated by this finding, we introduceClaimWriter, a hierarchical claim\-based writer that replaces only the report\-generation module of existing DR systems\.ClaimWriterfirst organizes source\-grounded facts into an explicit claim hierarchy and then drafts the report from this structure\. By separating evidence organization from report generation,ClaimWriterimproves recall of important facts while preserving links between claims and their supporting sources\.

We evaluateClaimWriterby replacing only the original writer in three DR hosts: Enterprise Deep Research\([3](https://arxiv.org/html/2608.28643#bib.bib1)\), NVIDIA AI\-Q\([16](https://arxiv.org/html/2608.28643#bib.bib5)\), and OpenDeepResearch\([9](https://arxiv.org/html/2608.28643#bib.bib3)\)\. Upstream search and evidence collection are kept fixed\. Across both evaluation settings,ClaimWriterconsistently improves claim\-level quality over the original writers, reducing hallucination and misattribution while increasing necessary\-fact recall\. These gains are largely hidden by rubrics\-based scores, which move only slightly\.

Beyond one\-shot generation,ClaimWriteralso supports low\-cost report maintenance when source evidence changes\. It uses the claim hierarchy to map changed source facts to affected claims, revise only the corresponding sections, and preserve the rest of the report\. This matches enterprise settings, where the task and report structure often remain stable while individual facts evolve\([14](https://arxiv.org/html/2608.28643#bib.bib17)\)\. Across different levels of source churn, our update system propagates changed facts into the report at the highest rate among update methods, while keeping output\-token cost comparable to a single diff\-based rewrite\.

In summary, we make three main contributions\.\(1\)We introduceClaimProbe, a calibrated claim\-level auditing framework for measuring faithfulness, attribution, citation quality, and coverage in DR reports\.\(2\)We introduceClaimWriter, a hierarchical claim\-based writer that delivers superior claim\-level report quality across DR hosts and writer models\.\(3\)We develop an incremental revision method that uses claim structure to update reports efficiently and preserve unaffected content as source changes\.

## 2ClaimProbe: A Claim\-Level Report Auditing Framework

ClaimProbeis afour\-phase pipelinethat takes the outputs collected through tool calls, such as web search, together with a report, evaluates each report claim against the collected evidence, and computes four report\-level measures designed to capture failure modes identified in commercial answer engines\([19](https://arxiv.org/html/2608.28643#bib.bib4);[11](https://arxiv.org/html/2608.28643#bib.bib10)\)and prior work on long\-form factuality evaluation\([13](https://arxiv.org/html/2608.28643#bib.bib7);[21](https://arxiv.org/html/2608.28643#bib.bib8)\)\.

Hallucination \(Hall↓\\downarrow\)\.Does the report make claims that no retrieved source supports?

Misattribution \(Mis↓\\downarrow\)\.When the report cites sourcekk, does sourcekkactually support the claim, or is the claim only supported by some other retrieved source?

Uncited\-but\-supported \(Unc\)\.Does the report omit citations for claims that retrieved sources do support? A highUnctogether with lowHallandMisindicates that the writer is grounded but does not always cite\.

Necessary fact recall \(Rec↑\\uparrow\)\.Of the source facts that are highly relevant to the task, what fraction appear in the report?

Enterprise Deep ResearchNVIDIA AI\-QOpen Deep ResearchHost deep\-research pipelines \(interchangeable\)Source factsself\-contained, scored, deduplicatedReport fact graphtopics, sections,\{\{CITE:fN\}\}markersAssembled reportreport\.mdwith numbered citations\[N\]\[N\]ClaimProbeauditHallMisUncRec\(\+\+strict,\+\+direct\)Fact\-level updatesfact\-patch, fact\-diff\(touch only changed facts\)Updatedreportreport\.mdwith numbered citations\[N\]\[N\]ClaimWritersourcesassembleevaluatesource delta

Figure 1:System overview\.A selected host deep\-research pipeline supplies sources toClaimWriter, which extracts source facts, materializes a report fact graph with\{\{CITE:fN\}\}markers, and assembles the user\-facingreport\.md\.ClaimProbeaudits this fact\-linked representation \(Section[2](https://arxiv.org/html/2608.28643#S2)\); when sources change,ClaimWriterapplies fact\-level updates and emits an updated report \(Section[6\.2](https://arxiv.org/html/2608.28643#S6.SS2)\)\.### 2\.1Report Audit Procedure

ClaimProbefirst processes the collected source outputs and the report separately, then aligns report claims with source facts, and finally computes report\-level measures\.

Source processing:converts collected source outputs into a standard format and extracts short, self\-contained factual statements\. It then labels each fact asHigh,Related, orUnrelatedwith respect to the task\. We computeRecusing onlyHigh\-relevance facts\.

Report processing:parses the report’s reference block to recover cited references\. It then processes the report section by section, excluding the references section, and uses an LLM to extract report claim statements from each section together with their cited references\.

Alignment:For each report claim,ClaimProbeembeds the claim and all source facts usingtext\-embedding\-3\-small, retrieves the top\-K=20K\{=\}20source facts by cosine similarity, and uses an LLM judge to label the claim asSupported,Partially Supported, orNot Supported\. For recall, it applies the same retrieval\-and\-judging procedure in the reverse direction, starting from each source fact and matching it to report claims\. Claim\-to\-source matching is strict and requires precise factual support, including for numeric values\. Source\-to\-report matching is more lenient and treats paraphrase, summarization, and coverage as sufficient when the core information is preserved\.

Metrics Computations:Using the cached alignment judgments,ClaimProbecomputes the four report\-level measures\. For report\-to\-source metrics,Halltreats a report claim as hallucinated only when no retrieved source fact supports it, i\.e\., when the claim is labeledNot Supported;Hallstris stricter and treats bothPartially SupportedandNot Supportedreport claims as not fully grounded\. For source\-to\-report metrics,Reccredits aHigh\-relevance source fact when the report fully or partially covers it, whileRecstrcredits only fully covered source facts\. Thus, the default metrics count bothSupportedandPartially Supportedas support, but in opposite alignment directions: report claims forHalland source facts forRec\. Appendix[A](https://arxiv.org/html/2608.28643#A1)provides further details on the metric definitions and human calibration\.

Judge calibration\.We calibrate GPT\-5\.4\-mini against human annotations\. Three annotators labeled disjoint pools drawn from a stratified task sample \(n=100\), on hallucination, misattribution, and relevance, each in a blind setting\. The AI judge agrees with the unanimous\-human label \(Cohen’sκ=0\.484,0\.797,0\.743\\kappa=0\.484,0\.797,0\.743; details in Appendix[B](https://arxiv.org/html/2608.28643#A2)\)\. The judge is asked to score one claim against a short shortlist at a time, the regime under which LLM\-as\-judge calibration is most stable\([22](https://arxiv.org/html/2608.28643#bib.bib11)\)\.

## 3ClaimWriter: Hierarchical Claim\-Based Writer Module

A claim\-centric audit naturally motivates a claim\-centric writer\. Instead of concatenating retrieved content and using a long\-context LLM to synthesize a report,ClaimWritermaintains*source facts*as first\-class objects\. It organizes these facts into a structured topic hierarchy before drafting and generates report text with explicit*claim\-to\-source linkage*, which it resolves only in the final report construction stage\([6](https://arxiv.org/html/2608.28643#bib.bib9)\)\.

Specifically,ClaimWriterfirstdeduplicatesthe retrieved source content and generates an initial topic outline directly from the user query, without conditioning on extracted evidence\. This choice prevents partial retrieval coverage from determining the report structure and keeps the outline aligned with the information requirements of the query\. It thenextracts self\-contained factsfrom each source page in parallel, scores them for relevance and specificity, deduplicates them, and assigns each fact to a single outline topic\. Given the resulting organization of topics and facts,ClaimWriterdrafts each section independentlyusing fact\-level citation markers \(e\.g\.,CITE:fN\) that are converted into numeric references only when the report is constructed\. The system then constructs thefinal reportby concatenating sections and resolving these into a global citation list\.

Efficient Handling of Source Changes

ClaimWriterproduces a fact\-linked report representation in which each extracted fact is tied to a topic assignment, a citation marker, and the section it supports\. This representation allows the system to localize updates to only those sections affected by changes in source content, unlike current deep research pipelines that require a complete rerun of the process\.

For each changed source,ClaimWritercompares the previous and updated source content together with the facts extracted from the previous version of that source\. It then determines which existing fact ids should be updated or removed and which new facts should be added\. For updated or removed facts,ClaimWriteruses the existing fact\-to\-topic assignments to identify the affected outline topics and revises the corresponding report sections\. For newly added facts,ClaimWriterassigns each fact to the appropriate topic in the existing outline and updates the corresponding section to incorporate the new information\.

This update procedure preserves the report structure while restricting rewriting to sections whose assigned facts changed, reducing token cost and avoiding unnecessary edits to unaffected text\.

Table 1:RACE scores \(n=100n\{=\}100, judge GPT\-5\.5\)\. Columns are comprehensiveness, insight, instruction\-following, readability, and overall \(higher is better\)\.
## 4Experiments

We report three main experiments that address the key questions an industry adopter would ask before deploying a deep\-research stack: \(i\) Can the writer preserve holistic report quality under an established rubric \(Section[5\.1](https://arxiv.org/html/2608.28643#S5.SS1)\)? \(ii\) Can the writer retain necessary facts without fabricating new ones, across host pipelines and writer models \(Section[5\.2](https://arxiv.org/html/2608.28643#S5.SS2)\)? and \(iii\) Can the system update reports quickly, consistently, and accurately when sources change \(Section[6\.2](https://arxiv.org/html/2608.28643#S6.SS2)\)?

Setup\.We conduct our experiments on DeepResearch Bench\([4](https://arxiv.org/html/2608.28643#bib.bib2)\), which contains 50 English and 50 Chinese deep research tasks; the task scope for each evaluation is specified below\. We use three open\-source deep research pipelines as host systems for source consolidation: Enterprise Deep Research \(EDR\)\([3](https://arxiv.org/html/2608.28643#bib.bib1)\), NVIDIA AI\-Q\([16](https://arxiv.org/html/2608.28643#bib.bib5)\), and OpenDeepResearch \(ODR\)\([9](https://arxiv.org/html/2608.28643#bib.bib3)\)\. For each pipeline, we replace its original report\-writing step with the proposedClaimWriterto generate the final report\. We use GPT\-5\.5 as the underlying model for all pipeline components, including report generation\.

Evaluation\.ForClaimProbe\-based evaluation, we usetext\-embedding\-3\-smallas the embedding model, set retrieval toK=20K\{=\}20in both directions, and use a single LLM judge\. We evaluateClaimProbewith two backbone LLMs: GPT\-5\.4\-mini on all 100 tasks \(ids 1–100\) and GPT\-5\.5 on a 10\-task subset \(ids 1–5, 51–55\)\. All results in Table[2](https://arxiv.org/html/2608.28643#S5.T2)use microaveraging, computed by pooling numerators and denominators across a run’s tasks\. We also report RACE scores\([4](https://arxiv.org/html/2608.28643#bib.bib2)\)on all 50 English tasks using the original GPT\-5\.5 judge configuration\.

Baseline systems\.We compareClaimWriteragainst the system\-default report writer step \(Baseline\)\. For each DR pipeline, we run the full pipeline to obtain the final report\. We also instantiate a naive variant ofClaimWriter, namelyClaimWriter\-BU, that follows a bottom\-up writing strategy: it performs agglomerative clustering on the source facts to form topic groups, prunes query\-irrelevant groups, then uses the remaining groups and the query to derive the report outline\. Finally, relevant facts to the outline are assigned and the report is generated for each section\.

## 5Results

### 5\.1RACE scores stay close underClaimWriter

We show the RACE scores for the baseline andClaimWriterin Table[1](https://arxiv.org/html/2608.28643#S3.T1)\. On this report\-level rubric, swapping in the claim\-based writer leaves comprehensiveness and insight essentially unchanged but lowers readability \(by3\.83\.8–5\.95\.9points\), yielding a small net drop in Overall \(≤1\.8\\leq 1\.8points\) in every host\. This is a positive result: the claim\-based writer largely preserves the holistic qualities that RACE measures while delivering the claim\-level gains in Table[2](https://arxiv.org/html/2608.28643#S5.T2)\.

### 5\.2ClaimProbereveals claim\-level report quality differences

We report theClaimProberesults for the3×3×23\{\\times\}3\{\\times\}2\(Host, Writer, Judge model\) combinations in Table[2](https://arxiv.org/html/2608.28643#S5.T2)\. Unlike RACE,ClaimProbereveals large discrepancy between writers: the same writer swap that moves holistic scores only slightly can sharply reduce hallucination and misattribution while increasing necessary\-fact recall\. For example,ClaimWriterreducesHallfrom15\.8915\.89to5\.025\.02andMisfrom18\.9418\.94to5\.435\.43, while improvingRecfrom36\.8336\.83to45\.8545\.85on EDR with GPT\-5\.4\-mini\. Across hosts and judge models,ClaimWriterandClaimWriter\-BUare consistently the strongest writers, trading the lead across cells, while the baseline is weakest\. We observe that GPT\-5\.4\-mini is sufficient for ourClaimProbesetup: it gives nearly the same relative ranking of report writers as GPT\-5\.5\. However, absolute scores requires cautious interpretation since they differ moderately between the judge models\. Since all writers share a single backbone model \(GPT\-5\.5\), any judge self\-preference due to matching generator & evaluator pairs applies to all cases equally\. We verify this by also runningClaimProbewith Gemini\-3\.5\-flash as the backbone and demonstrate the same relative system rankings in Appendix[C](https://arxiv.org/html/2608.28643#A3)\.

Table 2:ClaimProbeaudit on the host×\\timeswriter×\\timesjudge\-model factorial\.All values are percentages, micro\-averaged over pooled \(numerator, denominator\) pairs\. Green cells are the best writer in each \(host, judge\-model\) block;Uncis contextual and left uncolored\.

## 6Discussion

### 6\.1Holistic quality and source faithfulness can diverge

The RACE result in Table[1](https://arxiv.org/html/2608.28643#S3.T1)and theClaimProberesult in Table[2](https://arxiv.org/html/2608.28643#S5.T2)show that surface\-level report quality and source faithfulness are partly separable\. A writer can keep the holistic evaluation score close to the baseline while changing the claim\-to\-source relationship substantially\. In other words, the broad report\-quality measures are useful but insufficient in practice: they should be paired with a claim\-level audit that checks whether the report cites the right sources and retains the necessary facts\.

Why claim\-level judging is better scoped\.ClaimProbealso uses an LLM judge, but each decision is local\. AHallorMisverdict compares one report claim against a top\-K=20K\{=\}20evidence shortlist; aRecverdict asks whether oneHigh\-relevance source fact is paraphrased or subsumed in the report\. Since the judge never evaluates either artifact end to end, page\-level confounds such as long\-prose anchoring, length bias, and presentation effects are reduced\. This matches the observation from[22](https://arxiv.org/html/2608.28643#bib.bib11)that LLM\-as\-a\-judge is more reliable under tightly scoped prompts, and our human study \(Section[2](https://arxiv.org/html/2608.28643#S2)\) confirms higher agreement in this restricted setting\.

Table 3:Dynamic\-update cost and update yield\.Output tokens are per\-report averages over the tasks\.Update Yieldis the*clean*rate: the percentage of relevant perturbed facts whose new value reaches the regenerated report while the old value is removed \(higher is better\)\.\+\\mathbf\{\+\}ClaimWriteris the fact\-diff \(top\-down\) updater and\+\\mathbf\{\+\}ClaimWriter\-BUthe bottom\-up fact\-diff updater\. Cell colour: green favourable, yellow mid, red unfavourable, dark\-red catastrophic \(the cost ceiling\)\.
### 6\.2Dynamic Report Update: Yield and Cost

Enterprise reports often keep the task and report structure fixed while the underlying sources drift\. To model this setting, we use five DeepResearch Bench tasks \(ids 51–55\) and evaluate two source\-change budgets: 5 changed sources and 25 changed sources\. In each setting, we update one specific numerical fact per changed source document with GPT\-5\.5, covering years, quantities, trends, or combinations thereof, and validate each edit for consistency with the surrounding source context\. Each perturbation defines a new \(perturbed\) and an old \(original\) target value; over the query\-relevant facts, an LLM judge labels the regenerated report as containing theNewvalue, theOldvalue,Both, orAbsent\.

We compareClaimWriteragainst two update baselines:AI\-Q native \(replay\), which stores the full DR trace and replays the pipeline after replacing only the changed source contents, andBase \(diff\), which compares each old source document with its updated version, extracts the changed facts, aggregates them, and rewrites the existing report in a single diff\-based update\.

As shown in Table[3](https://arxiv.org/html/2608.28643#S6.T3), we report output\-token cost and update yield\. We define yield as the*clean*rate: the percentage of relevant perturbed facts for which the regenerated report states the new value*and*no longer states the old one\. Since deep\-research report updates are typically scheduled jobs rather than latency\-critical interactive calls, we optimize primarily for output\-token cost\.

We find that update quality depends less on rewriting strength than on whether the system exposes a fine\-grained edit handle\. Full replay reruns the pipeline and sets the cost ceiling, but it still misses many small source changes and produces very low yield\. A generic diff over a monolithic report reduces cost, but it lacks a stable object corresponding to the changed fact and often edits around the change rather than propagating it\. In contrast,ClaimWritercasts source changes into fact changes and uses the constructed fact graph to localize revisions to affected sections\. As a result,ClaimWriterachieves substantially higher update yield while staying close to diff\-style output\-token costs and preserving report faithfulness\.111Refer to Appendix[D](https://arxiv.org/html/2608.28643#A4)forClaimProbeevaluation on reports from perturbed source\.

Steady\-state cost note\.ClaimWriteruses more first\-run input tokens than the host’s native writer because it re\-extracts facts from the retrieved sources \(Table[7](https://arxiv.org/html/2608.28643#A5.T7)in Appendix[E](https://arxiv.org/html/2608.28643#A5)\)\. This one\-time cost materializes the fact graph; subsequent updates amortize it through the lower incremental\-update budget in Table[3](https://arxiv.org/html/2608.28643#S6.T3)\. For workflows that update the same report multiple times, this amortization dominates\. For one\-shot reports, users should weigh the higher first\-run cost against the benefits of claim\-level structure and maintainability\.

### 6\.3Interpreting secondary metrics

Why highUncis a benign signal\.The uncited\-but\-supported rate \(Unc\) sits at6464–91%91\\%everywhere and tends to*rise*withClaimWriter\. Read together with the simultaneous drops inHallandMisand the lifts in strict\-recall \(Recstr\{\}\_\{\\text\{str\}\}\), this means the uncited residue is increasingly composed of grounded background and connective tissue \(general knowledge, framing sentences\), not fabrications\.Uncalone should therefore be read as a citation\-hygiene gap, not as a fabrication signal\.

## 7Related Work

Production deep\-research systems include Google Gemini Deep Research\([7](https://arxiv.org/html/2608.28643#bib.bib14)\), OpenAI Deep Research\([17](https://arxiv.org/html/2608.28643#bib.bib15)\), and open\-source efforts such as Enterprise Deep Research\([3](https://arxiv.org/html/2608.28643#bib.bib1)\), NVIDIA AI\-Q\([16](https://arxiv.org/html/2608.28643#bib.bib5)\), and Open Deep Research\([9](https://arxiv.org/html/2608.28643#bib.bib3)\)\. Existing systems emphasize planning, search, and stopping criteria, while treating report generation largely as a final long\-context synthesis step\. Recent benchmarks, including DeepResearch Bench\([4](https://arxiv.org/html/2608.28643#bib.bib2)\), LiveResearch Bench\([20](https://arxiv.org/html/2608.28643#bib.bib6)\), ResearchRubrics\([18](https://arxiv.org/html/2608.28643#bib.bib18)\), DeepResearch Bench II\([10](https://arxiv.org/html/2608.28643#bib.bib16)\), DRBench\([1](https://arxiv.org/html/2608.28643#bib.bib19)\), and GAIA\([12](https://arxiv.org/html/2608.28643#bib.bib22)\), evaluate final reports, full agents, or trajectories using rubrics, human\-authored criteria, or ground\-truth insight recall\. These benchmarks improve coverage, realism, and enterprise grounding, but do not isolate the report writer under fixed retrieved evidence; as Section 5\.1 shows, aggregate scores can mask large claim\-level faithfulness differences\.

A parallel line of work evaluates factuality and attribution below the document level\. FActScore\([13](https://arxiv.org/html/2608.28643#bib.bib7)\)and SAFE\([21](https://arxiv.org/html/2608.28643#bib.bib8)\)decompose long\-form generations into atomic facts or claims, while ALCE\([6](https://arxiv.org/html/2608.28643#bib.bib9)\), attributed QA\([2](https://arxiv.org/html/2608.28643#bib.bib20)\), and RAGAS\([5](https://arxiv.org/html/2608.28643#bib.bib21)\)evaluate citation quality or retrieval\-grounded faithfulness\. Prior studies also document hallucinations and citation errors in deployed search and QA systems\([11](https://arxiv.org/html/2608.28643#bib.bib10);[19](https://arxiv.org/html/2608.28643#bib.bib4);[15](https://arxiv.org/html/2608.28643#bib.bib12)\), and surveys summarize hallucination phenomena more broadly\([8](https://arxiv.org/html/2608.28643#bib.bib13)\)\.ClaimProbeextends this line to multi\-source, citation\-heavy deep\-research reports, isolating writer\-attributable failures through claim\-level verdicts over fixed retrieved evidence\.

## 8Conclusion

We introduceClaimProbe, a claim\-level audit for DR reports, andClaimWriter, a hierarchical claim\-based DR report writer\. Across three DR hosts, replacing only their writer withClaimWriterpreserves rubrics\-based performance while improving claim\-level faithfulness and recall\. The same fact\-linked representation supports localized report revision under source drift, producing more accurate updates at lower output\-token cost\. These results motivate focused work on DR report writing and update systems that preserve evidence structure, attribution, and maintainability\. We have made public our pipeline and benchmark222https://github\.com/SalesforceAIResearch/claimwriter\-deep\-research\.

## Limitations

Our evaluation has two main limitations\. First,ClaimProbeuses an LLM judge\. We mitigate this by scoring local decisions: one claim against a top\-K=20K\{=\}20evidence shortlist, and by validating the judge against human annotations\. Still, users who deployClaimProbein new domains should recalibrate the judge on their own task distributions\.

Second, our dynamic\-update study uses a controlled source\-drift setting: five tasks, two source\-change budgets, and one numerical fact update per changed source\. This design lets us isolate whether a system propagates changed facts into the report, but it is only a first step toward studying report maintenance under evolving evidence\. We hope it motivates further research on broader update types, including new sources, removed evidence, qualitative changes, and updates that require restructuring the report\. Finally,ClaimWriterpays a higher first\-run cost to materialize the fact graph, so its cost advantage is strongest in workflows where reports are updated over time\.

## References

- Abaskohiet al\.\(2026\)A\. Abaskohi, T\. Chen, M\. Muñoz\-Mármol, C\. Fox, A\. V\. Ramesh, É\. Marcotte, X\. H\. Lù, N\. Chapados, S\. Gella, P\. West, G\. Carenini, C\. Pal, A\. Drouin, and I\. H\. LaradjiDRBench: a realistic benchmark for enterprise deep research\.External Links:2510\.00172,[Link](https://arxiv.org/abs/2510.00172)Cited by:[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Bohnetet al\.\(2022\)B\. Bohnet, V\. Q\. Tran, P\. Verga, R\. Aharoni, D\. Andor, L\. B\. Soares, M\. Ciaramita, J\. Eisenstein, K\. Ganchev, J\. Herzig,et al\.Attributed question answering: evaluation and modeling for attributed large language models\.arXiv preprint arXiv:2212\.08037\.Cited by:[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- Choubeyet al\.\(2026\)P\. K\. Choubey, K\. Huang, P\. N\. Venkit, J\. Zhang, V\. Vats, Y\. Li, X\. Peng, and C\. WuDon’t stop early: scalable enterprise deep research with controlled information flow and evidence\-aware termination\.arXiv preprint arXiv:2604\.24978\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p1.1),[§1](https://arxiv.org/html/2608.28643#S1.p5.1),[§4](https://arxiv.org/html/2608.28643#S4.p2.1),[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Duet al\.\(2025\)M\. Du, B\. Xu, C\. Zhu, X\. Wang, and Z\. MaoDeepresearch bench: a comprehensive benchmark for deep research agents\.arXiv preprint arXiv:2506\.11763\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p2.1),[§4](https://arxiv.org/html/2608.28643#S4.p2.1),[§4](https://arxiv.org/html/2608.28643#S4.p3.1),[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Eset al\.\(2024\)S\. Es, J\. James, L\. E\. Anke, and S\. SchockaertRagas: automated evaluation of retrieval augmented generation\.InProceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations,pp\. 150–158\.Cited by:[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- Gaoet al\.\(2023\)T\. Gao, H\. Yen, J\. Yu, and D\. ChenEnabling large language models to generate text with citations\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 6465–6488\.Cited by:[§3](https://arxiv.org/html/2608.28643#S3.p1.1),[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- Google \(2025\)GoogleGemini Deep Research\.Note:Google product pageExternal Links:[Link](https://gemini.google/overview/deep-research/)Cited by:[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Jiet al\.\(2023\)Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. FungSurvey of hallucination in natural language generation\.ACM computing surveys55\(12\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p2.1),[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- LangChain \(2025\)LangChainOpen deep research\.Note:GitHub repositoryExternal Links:[Link](https://github.com/langchain-ai/open_deep_research)Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p5.1),[§4](https://arxiv.org/html/2608.28643#S4.p2.1),[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Liet al\.\(2026\)R\. Li, M\. Du, B\. Xu, C\. Zhu, X\. Wang, and Z\. MaoDeepResearch bench ii: diagnosing deep research agents via rubrics from expert report\.External Links:2601\.08536,[Link](https://arxiv.org/abs/2601.08536)Cited by:[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Liuet al\.\(2023\)N\. F\. Liu, T\. Zhang, and P\. LiangEvaluating verifiability in generative search engines\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 7001–7025\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p2.1),[§2](https://arxiv.org/html/2608.28643#S2.p1.1),[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. ScialomGaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFactscore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12076–12100\.Cited by:[§2](https://arxiv.org/html/2608.28643#S2.p1.1),[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- Mukherjee and Mao \(2004\)R\. Mukherjee and J\. MaoEnterprise search: tough stuff: why is it that searching an intranet is so much harder than searching the web?\.Queue2\(2\),pp\. 36–46\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p6.1)\.
- Narayanan Venkitet al\.\(2025\)P\. Narayanan Venkit, P\. Laban, Y\. Zhou, K\. Huang, Y\. Mao, and C\. WuDeepTRACE: auditing deep research AI systems for tracking reliability across citations and evidence\.arXiv preprint arXiv:2509\.04499\.External Links:[Link](https://arxiv.org/abs/2509.04499)Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p1.1),[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- NVIDIA \(2025\)NVIDIAAI\-q blueprint: enterprise deep research agent\.Note:GitHub repositoryExternal Links:[Link](https://github.com/NVIDIA-AI-Blueprints/aiq)Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p5.1),[§4](https://arxiv.org/html/2608.28643#S4.p2.1),[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- OpenAI \(2025\)OpenAIIntroducing deep research\.Note:OpenAI blog postExternal Links:[Link](https://openai.com/index/introducing-deep-research/)Cited by:[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Sharmaet al\.\(2025\)M\. Sharma, C\. B\. C\. Zhang, C\. Bandi, C\. Wang, A\. Aich, H\. Nghiem, T\. Rabbani, Y\. Htet, B\. Jang, S\. Basu, A\. Balwani, D\. Peskoff, M\. Ayestaran, S\. M\. Hendryx, B\. Kenstler, and B\. LiuResearchRubrics: a benchmark of prompts and rubrics for evaluating deep research agents\.External Links:2511\.07685,[Link](https://arxiv.org/abs/2511.07685)Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p2.1),[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Venkitet al\.\(2024\)P\. N\. Venkit, P\. Laban, Y\. Zhou, Y\. Mao, and C\. WuSearch engines in an ai era: the false promise of factual and verifiable source\-cited responses\.arXiv preprint arXiv:2410\.22349\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p2.1),[§2](https://arxiv.org/html/2608.28643#S2.p1.1),[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- Wanget al\.\(2025\)J\. Wang, Y\. Ming, R\. Dulepet, Q\. Chen, A\. Xu, Z\. Ke, F\. Sala, A\. Albarghouthi, C\. Xiong, and S\. JotyLiveresearchbench: a live benchmark for user\-centric deep research in the wild\.arXiv preprint arXiv:2510\.14240\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p2.1),[§7](https://arxiv.org/html/2608.28643#S7.p1.1)\.
- Weiet al\.\(2024\)J\. Wei, C\. Yang, X\. Song, Y\. Lu, N\. Hu, J\. Huang, D\. Tran, D\. Peng, R\. Liu, D\. Huang,et al\.Long\-form factuality in large language models\.Advances in Neural Information Processing Systems37,pp\. 80756–80827\.Cited by:[§2](https://arxiv.org/html/2608.28643#S2.p1.1),[§7](https://arxiv.org/html/2608.28643#S7.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2608.28643#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.28643#S2.SS1.p6.1),[§6\.1](https://arxiv.org/html/2608.28643#S6.SS1.p2.1)\.

## Appendix AClaimProbepipeline details

#### Extract\.

Tool logs become a normalizedsourcestable with URL canonicalization \(scheme,wwwprefix, trailing slash, and fragments removed\) and a boilerplate stripper\. The report’s reference list is parsed by three regular expressions \(canonical\[N\] url, numbered MarkdownN\. \[Title\]\(url\), and descriptive\[N\] description: url\) to tolerate real\-world citation conventions\. The report is segmented twice:report\_units\.json\(sentence units, table summaries, contextualized table\-cell atoms\) forRec, andreport\_claims\.json\(claims\) forHall,Mis,Unc\. Source contents are segmented into source facts by an LLM extractor instructed to make facts self\-sufficient, replace pronouns with entity names, and drop boilerplate\.

#### Relevance\.

Each source fact receives a label in \{High,Related,Unrelated\} conditioned on the task statement\.Highfacts directly answer the task or provide a central statistic the report is expected to use;Relatedfacts provide background;Unrelatedfacts are off\-topic\.Unrelatedfacts are excluded from the embedding index\.

#### Alignment\.

For each query item we retrieve the top\-K=20K\{=\}20candidates and ask a holistic judge for a strict JSON verdict in \{S,PS,NS\} with a 1\-to\-5 confidence score and a 20\-word justification\. The judge is direction\-aware: the report\-to\-source prompt is strict \(numeric precision required\); the source\-to\-report prompt is lenient \(paraphrase and subsumption count as coverage\)\.

#### Metrics\.

With𝒞\\mathcal\{C\}the set of report claims andℱhigh\\mathcal\{F\}\_\{\\text\{high\}\}theHighsource\-fact set,

Hall=\|\{c:r2s​\(c\)=NS\}\|/\|𝒞\|\\displaystyle=\|\\\{c:\\texttt\{r2s\}\(c\)\{=\}\\textsc\{NS\}\\\}\|\\,/\\,\|\\mathcal\{C\}\|Hallstr\\displaystyle\\textsc\{Hall\}\_\{\\text\{str\}\}=\|\{c:r2s​\(c\)≠S\}\|/\|𝒞\|\\displaystyle=\|\\\{c:\\texttt\{r2s\}\(c\)\{\\neq\}\\textsc\{S\}\\\}\|\\,/\\,\|\\mathcal\{C\}\|Mis=\|\{c∈𝒞cited:¬by\_cited​\(c\)∧by\_other​\(c\)\}\|/\|𝒞cited\|\\displaystyle=\|\\\{c\{\\in\}\\mathcal\{C\}\_\{\\text\{cited\}\}:\\neg\\text\{by\\\_cited\}\(c\)\\wedge\\text\{by\\\_other\}\(c\)\\\}\|\\,/\\,\|\\mathcal\{C\}\_\{\\text\{cited\}\}\|Unc=\|\{c∈𝒞uncited:supported​\(c\)\}\|/\|𝒞uncited\|\\displaystyle=\|\\\{c\{\\in\}\\mathcal\{C\}\_\{\\text\{uncited\}\}:\\text\{supported\}\(c\)\\\}\|\\,/\\,\|\\mathcal\{C\}\_\{\\text\{uncited\}\}\|Rec=\|\{f∈ℱhigh:s2r​\(f\)∈\{S,PS\}\}\|/\|ℱhigh\|\.\\displaystyle=\|\\\{f\{\\in\}\\mathcal\{F\}\_\{\\text\{high\}\}:\\texttt\{s2r\}\(f\)\{\\in\}\\\{\\textsc\{S\},\\textsc\{PS\}\\\}\\\}\|\\,/\\,\|\\mathcal\{F\}\_\{\\text\{high\}\}\|\.A site\-prefix fallback \(netloc \+ first two path segments\) avoids inflatingMison government and statistical mega\-domains\.

## Appendix BHuman validation of the LLM judge

Three annotators labeled disjoint pools drawn from a stratified task sample\. We report Cohen’sκ\\kappafor human\-human agreement and for the AI judge against the human majority and the unanimous\-human labels\.

Table 4:Human and LLM judge agreement on the three judging tasks \(judge GPT\-5\.4\-mini\)\.
## Appendix CRobustness to the judge model

TheClaimProbejudge we use was GPT\-5\.5, drawn from the same family as the backbone that writes the reports\. While this may raise a concern regarding self\-preference in LLM judges, we note that such effect is applied toallof them equally and leaves their relative ranking unchanged\. To show this, we also ranClaimProbewith Gemini\-3\.5\-flash as an independent judge on the 10\-task subset \(ids 1–5, 51–55\), holding the writers, sources, and extraction steps fixed\. Table[5](https://arxiv.org/html/2608.28643#A3.T5)demonstrates a similar observation: our proposed report writing systems significantly improve over the baselines in every metric\. On AI\-Q, for instance, replacing the baseline withClaimWriter\-BUdropsHallfrom57\.1457\.14to19\.8119\.81andMisfrom15\.8915\.89to0\.860\.86, and raisesRecfrom15\.9715\.97to41\.2241\.22\.

Table 5:ClaimProbeaudit with the Gemini\-3\.5\-flash judge\(n=10n\{=\}10; ids 1–5, 51–55\)\. All values are percentages, micro\-averaged over pooled \(numerator, denominator\) pairs\. Green cells are the best writer in each host block;Uncis contextual and left uncolored\.
## Appendix DProbe under perturbation

We re\-runClaimProbeon every perturbed regeneration of Section[6\.2](https://arxiv.org/html/2608.28643#S6.SS2)\. Table[6](https://arxiv.org/html/2608.28643#A4.T6)shows that all EDR update methods \(Base, FP, FD\) cluster within±1\\pm 1percentage point of the unperturbed EDR baseline on everyClaimProbemetric; the surgical updates do not introduce drift\. The AI\-Q systems hallucinate and misattribute markedly more in every configuration \(Hallstr≈66−68\\textsc\{Hall\}\_\{\\text\{str\}\}\\approx 66\{\-\}68,Mis≈44−45\\textsc\{Mis\}\\approx 44\{\-\}45\) and recall fewerHighfacts \(Rec≈31−38\\textsc\{Rec\}\\approx 31\{\-\}38\), independently of the update strategy\.

Table 6:ClaimProbeon perturbed regenerations \(%\\%, judge GPT\-5\.5\)\. EDR update strategies cluster within±1\\pm 1of the unperturbed EDR baseline; updates do not drift\.
## Appendix ESteady\-state writing cost

Per\-report writing\-stage tokens \(model GPT\-5\.5\), averaged over deepresearchbench tasks\. Counts the LLM calls from processing retrieved tool output to the final report, including intermediate artifacts; excludes search and tool calls and query and plan generation\.

Table 7:Steady\-state writing\-stage token cost \(averages per report\)\.\+\+ClaimWriterand\+\+ClaimWriter\-BUre\-extract facts from the entire retrieved corpus, so their input cost is higher than the host baselines \(which only process what they synthesize\)\. The fact\-graph cost is paid once per first\-run and amortizes against the incremental\-update budget in Table[3](https://arxiv.org/html/2608.28643#S6.T3)\.
## Appendix FJudge and extractor prompts

We reproduce the system prompts used by theClaimProbejudge and the extractors, taken verbatim from the released code \(pipeline/current/eval\_pilot/\)\. Long extractor prompts are shown as their core skeleton plus the hard\-rule list, with the full text released with the code\.

### Alignment judge: shared base

The same base prompt powers bothHall/Mis/Unc\(report\-to\-source\) andRec\(source\-to\-report\); only the direction\-specific context and rules differ\.

Alignment judge \(shared base\)You are given aClaimand a numbered list ofEvidencestatements \(1 to N\)\. The evidence was retrieved using semantic similarity and may include irrelevant, partially relevant, or redundant items\. Each evidence line is prefixed with its cosine similarity score to the claim, higher scores indicate closer semantic match but do not guarantee factual support\.\{direction\_context\}Verdict definitions\.•Supported: One or more evidence statements clearly back the claim\. The core factual content of the claim is present in the evidence, even if wording, structure, or level of detail differs\. Minor differences in framing are acceptable as long as the central meaning is preserved\.•Partially Supported: The evidence covers some but not all of the claim, or the match is incomplete or ambiguous \(for example, confirms part of a compound statement but is silent on the rest, or provides weaker or less specific data than the claim asserts\)\.•Not Supported: The evidence does not meaningfully back the claim\. Use this when the claim’s core assertion is absent, materially changed, or contradicted\.Rules \(shared\)\.\(1\) Use only the provided evidence text\. Do not rely on outside knowledge\. \(2\) Treat paraphrasing, summarisation, and synthesis as valid support when the factual meaning matches\. \(3\) Ignore duplicate or redundant evidence lines, they do not strengthen support\. \{direction\_rules\}Output \(strict JSON, no markdown fences\):\{"verdict": "SUPPORTED \| PARTIALLY SUPPORTED \| NOT SUPPORTED", "confidence": <1\-5\>, "justification": "<5\-20 word explanation\>"\}

### Direction: report\-to\-source \(Hall,Mis,Unc\)

Report\-to\-source directionDirection context\.TheClaimis extracted from a generated report\. TheEvidencecomes from original source documents\. Your task: determine whether the source evidence factually supports the report’s claim\. Be strict, the report must be grounded in source data\.Additional rules\.\(4\) Numeric precision matters: if the claim states a specific number and the evidence gives a materially different number, that part is not supported\. \(5\) The claim may synthesise across sources, require that each key assertion has at least one evidence match\.

### Direction: source\-to\-report \(Rec\)

Source\-to\-report directionDirection context\.TheClaimis a fact extracted from an original source document\. TheEvidenceconsists of passages from a generated report\. Your task: determine whether the report covers this source fact\. The report may paraphrase, summarise, or incorporate the fact into broader analysis, this counts as coverage\.Additional rules\.\(4\) The report may express the fact at a different granularity \(e\.g\. rounded numbers, broader category\)\. Accept this as coverage if the core information is preserved\. \(5\) If the report subsumes the fact into a higher\-level summary that clearly encompasses it, treat it asSupported\.

### Relevance labeler \(High/Related/Unrelated\)

The relevance step has two implementations\. The default \(ClaimWriterexperiments\) is a deterministic mapping from the extraction\-time relevance field \(directbecomesHigh,off\_topicbecomesUnrelated, otherwiseRelated\)\. The optional*refine*path uses the following batched LLM relevance prompt\.

Batched source\-fact relevance labelerYou are labeling the relevance of factual sentences to a research task\. For each fact, choose exactly one label:High,Related, orUnrelated\.Task:\{task\}Definitions and examples\.High: directly answers the task, provides key evidence for the main question, or states a central statistic the report would need\.*Example*\(task: “What is Japan’s elderly population trend?”\): “The proportion of people aged 65 and over in Japan reached 28\.6% in 2020 and is projected to peak around 2040\.” \(labelHigh\)\.Related: same topic or useful background but does not directly answer or strongly support the task\.*Example*\(same task\): “The National Institute of Population and Social Security Research publishes official projections every five years\.” \(labelRelated\)\.Unrelated: does not help answer the task; off\-topic or different subject\.*Example*\(same task\): “The company reported strong earnings in the retail segment last quarter\.” \(labelUnrelated\)\.Facts to label\(each line is one fact; the leading number is the fact index, 0\-based\): \{numbered\_facts\}Output\.Exactly one line per fact index in the format “index: LABEL”\. Use onlyHigh,Related, orUnrelated\. Example:0: HIGH \\n 1: RELATED \\n 2: UNRELATED\. Output nothing else\.

### Inconsistency judge \(intra\-report contradictions\)

Pairwise inconsistency judgeYou are givenClaim\_AandClaim\_B, both claims extracted from the same generated report\. Decide whether they contradict each other\.Verdict definitions\.•Contradicts: the two claims assert at least one specific fact about the same entity, quantity, time, and scope that cannot both be true \(different numbers for the same measurement, opposite polarity, mutually exclusive categories, incompatible peak/trough years\)\. A single concrete factual conflict is sufficient\.•Partially Contradicts: the claims address overlapping subject matter and one tensions with the other without strictly conflicting \(different granularity that could be reconciled, ambiguous scope, borderline rounding\)\.•Consistent: both claims can be true together\. They may be unrelated, orthogonal, complementary, or actually agreeing\.Rules\.\(1\) Different time references about the same quantity areConsistent\. \(2\) Different scopes \(Japan vs Tokyo, total vs working\-age\) areConsistent\. \(3\) Rounded vs precise numbers areConsistent\. \(4\) Paraphrases of the same fact areConsistent\. \(5\) Numeric disagreement on the same quantity, entity, time, and scope isContradicts, even when the claims share other consistent content\. \(6\) Use only the text given\.Output\.Strict JSON:\{"verdict": "CONTRADICTS \| PARTIALLY\_CONTRADICTS \| CONSISTENT", "confidence": <1\-5\>, "justification": "<5\-20 word explanation\>"\}\.

### Claim and fact extractor \(shared core\)

Source\-side fact extraction and report\-side claim extraction share a common system prompt \(\_CLAIM\_PROMPT\_CORE\) of roughly four hundred lines; the report\-side variant additionally requests inline citation reference IDs \(cited\_refs\)\. We reproduce the core’s skeleton; the full text is released with the code\.

Claim and fact extractor \(skeleton\)A claim is a self\-contained, specific, information\-dense statement that can stand alone in a report without any surrounding context\. Include all materially useful claims; if nothing is useful for the task, return an empty list\.Language\.Output in English only, regardless of the language of the task description, report, or source content\.Task\-driven relevance\.Label each claim’s relevance as one ofdirect\(answers, proves, or quantifies a task question\),supporting\(evidence or proof point for a direct answer\),contextual\(useful background\), oroff\_topic\(no meaningful connection\)\. Score salience 0\-10 \(10 = answers a primary task question with hard data; 0 paired with off\_topic\)\.Self\-sufficiency\.Each claim must includetext\(fully standalone, no pronouns or generic placeholders, named subject inline\),time\_reference\(concrete calendar anchor only, never placeholder prose\),entities\(specific proper\-noun named entities, no generic categories\),relevance, andsalience\_score\.Hard rules\.No vague references \(“it”, “the company”, “the study”\); no relative time unless converted to a concrete year/date; no meta placeholders intime\_reference\(e\.g\. “report publication year” without digits\); no bare metrics without a named subject; no claim that depends on its heading or neighbours for meaning; preserve numeric precision; exclude boilerplate, navigation, and legal/privacy text\.Quality gate\.Apply five checks to every claim before output: subject identifiable, event identifiable, time anchored \(or correctly empty\), scope present when available, claim serves a task section\. If any fails, rewrite or drop\.Output \(JSON\)\.\{"claims": \[\{"text", "time\_reference", "entities", "relevance", "salience\_score"\}\]\}\. Report\-side addscited\_refs: list of integer reference IDs found in or near the claim text\.

### Fact\-diff \(Stage 1\) source\-change classifier

This prompt is part of the writer’s incremental\-update path \(Section[3](https://arxiv.org/html/2608.28643#S3)\); the model receives the updated source page text and the prior facts extracted from that page restricted to the report\-used candidates, and returns only facts whose value has changed as\(i​d,c​h​a​n​g​e\)\(id,change\)pairs wherechangeis a concise natural\-language instruction\. The exact text lives with the writer release; the schema we evaluate against is reproduced below\.

Source\-change classifier \(schema\)Input\.Updated source page text \(capped at the 200K\-character extraction budget\) and a listing of prior facts from that page, restricted to the candidates the report actually cites:\[\{id: fN, text: \.\.\.\}, \.\.\.\]\.Task\.Return only the facts whose value the edit changed, each as an\(id, change\)pair, wherechangeis a concise natural\-language instruction describing the modification \(for example, “speech year changed from 1995 to 1998”\) rather than a full rewritten fact\.Output \(JSON\)\.\{"changes": \[\{"id": "fN", "change": "\.\.\."\}\]\}\. Emitting an instruction rather than a rewritten fact keeps Stage 1’s completion\-token cost near the floor\.