Can AI Agents Synthesize Scientific Conclusions?

# Can AI Agents Synthesize Scientific Conclusions?
Source: [https://arxiv.org/html/2606.11337](https://arxiv.org/html/2606.11337)
Hayoung Jung♠Pedro Viana Diniz♣José Reinaldo Corrêa Roveda♣ Abner Fernandes da Silva♣Haeun Jung♡Enoch Tsai♢ Aleksandra Korolova♠Manoel Horta Ribeiro♠11footnotemark:1 ♠Princeton University♣Universidade Federal de Minas Gerais ♡Stony Brook University♢Hackensack Meridian School of Medicine \{hayoung, korolova, manoel\}@cs\.princeton\.edu ![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/icons/github.png)Code:[https://github\.com/hayoungjungg/SciConBench](https://github.com/hayoungjungg/SciConBench) ![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/x1.png)SciConBenchDataset:[hayoungjung/SciConBench](https://huggingface.co/datasets/hayoungjung/SciConBench)

###### Abstract

Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions\. Yet, their ability to do so in high\-stakes domains such as health remains unclear\. We introduceSciConBench, a large\-scalelivebenchmark of 9\.11K questions and expert\-written conclusions from systematic reviews to evaluate open\-domain scientific conclusion synthesis\. The benchmark draws on an expert\-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall\. To mitigate data leakage, we further introduceSciConHarness, aclean\-roomevaluation harness that equips agents with controlled web interaction to ensure valid measurement\. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean\-room settings, the best agent achieves only a factual F1 of 0\.337\. Our clean\-room settingconsistentlyreduces performance relative to unconstrained evaluation, suggesting that leakageinflatesestimates of models’ true synthesis capabilities\. Finally, we audit consumer\-facing agents \(e\.g\., Google AI Overview, OpenEvidence\) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground\-truth answer is available\. Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and thatclean\-roomevaluation is essential for assessing open\-domain AI agents\.

## 1Introduction

AI agents are transforming how individuals and institutions access and act on scientific knowledge\[[5](https://arxiv.org/html/2606.11337#bib.bib134),[80](https://arxiv.org/html/2606.11337#bib.bib129),[128](https://arxiv.org/html/2606.11337#bib.bib125)\]\. Unlike traditional search engines \(e\.g\., Google Search\), which retrieve relevant documents and leave the synthesis to users, agentic systems from Anthropic\[[6](https://arxiv.org/html/2606.11337#bib.bib131)\], Google DeepMind\[[44](https://arxiv.org/html/2606.11337#bib.bib133)\], OpenAI\[[81](https://arxiv.org/html/2606.11337#bib.bib44)\], and Perplexity\[[93](https://arxiv.org/html/2606.11337#bib.bib132)\]increasingly synthesize conclusions from scientific evidence\. They retrieve relevant evidence from the open web, filter irrelevant sources, reconcile conflicting findings, assess the evidence quality, and produce a long\-form expert\-level conclusion\. This long\-horizon task ofscientific conclusion synthesisis increasingly delegated to such systems, accelerating decision\-making and shaping decisions in health, science, and policy\[[80](https://arxiv.org/html/2606.11337#bib.bib129),[128](https://arxiv.org/html/2606.11337#bib.bib125)\]\.

One of the most consequential areas for scientific synthesis is health, and its impact is already evident in practice\. OpenAI reportsbillionsof weekly ChatGPT messages concern healthcare, with 40 million daily users, including the general public, many of whom trust AI\-generated health information, as well as physicians, who rely on AI for symptom and treatment exploration\[[80](https://arxiv.org/html/2606.11337#bib.bib129)\]\. More recently, specialized platforms like OpenEvidence serve as clinical AI copilots for high\-stakes decision\-making, reporting over 200 million AI\-powered health consultations and widespread use among U\.S\. clinicians\[[87](https://arxiv.org/html/2606.11337#bib.bib3)\]\.

![Refer to caption](https://arxiv.org/html/2606.11337v1/x2.png)Figure 1:Overview\.\(1\) We constructSciConBench, a live benchmark of 9\.11K questions and expert\-written conclusions\. \(2\) The benchmark evaluates AI agents’ capability for scientific synthesis by using web tools\. \(3\)SciConHarnessenforces clean\-room evaluation by blocking ground\-truth artifacts\. \(4\) Generated conclusions are evaluated against ground\-truth references using an expert\-validated pipeline that decomposes both into atomic facts and computes factual precision, recall, and F1\. \(5\) Results suggest that frontier systems achieve low factual F1 under clean\-room evaluation, highlighting the difficulty of reliable scientific conclusion synthesis\.However, prior work falls short in evaluating AI agents on the fulllong\-horizontask of synthesizinglong\-formscientific conclusions from theopen web\. Existing works focus on intermediate artifacts, such as retrieval and citation grounding\[[2](https://arxiv.org/html/2606.11337#bib.bib145),[39](https://arxiv.org/html/2606.11337#bib.bib146),[68](https://arxiv.org/html/2606.11337#bib.bib147)\], summarization\[[54](https://arxiv.org/html/2606.11337#bib.bib92),[107](https://arxiv.org/html/2606.11337#bib.bib70),[131](https://arxiv.org/html/2606.11337#bib.bib143)\], short\-form factuality\[[119](https://arxiv.org/html/2606.11337#bib.bib126),[120](https://arxiv.org/html/2606.11337#bib.bib127)\], or multiple\-choice QA\[[53](https://arxiv.org/html/2606.11337#bib.bib64),[90](https://arxiv.org/html/2606.11337#bib.bib144),[115](https://arxiv.org/html/2606.11337#bib.bib142),[116](https://arxiv.org/html/2606.11337#bib.bib71)\]—rather thanscientific conclusions\. As such, they fail to capture the core challenges and real\-world complexity of scientific conclusion synthesis\. Recent work moves closer by using expert\-curated datasets to evaluate open\-web synthesis\[[14](https://arxiv.org/html/2606.11337#bib.bib24),[34](https://arxiv.org/html/2606.11337#bib.bib23),[62](https://arxiv.org/html/2606.11337#bib.bib18),[71](https://arxiv.org/html/2606.11337#bib.bib16),[92](https://arxiv.org/html/2606.11337#bib.bib130),[99](https://arxiv.org/html/2606.11337#bib.bib22)\]\. However, these benchmarks remain limited: they are often small due to the high cost of expert curation \(N≤100N\\leq 100\), become outdated as new information emerges, and fail to address benchmark leakage, where models may be pre\-trained on or retrieve ground\-truth artifacts\.

In this work, we introduceSciConBench, alivebenchmark of 9\.11K questions and expert\-written conclusions, derived from theCochrane Database of Systematic Reviews\(CDSR\)\.SciConBenchevaluates whether agents can synthesize scientific conclusions from open\-web evidence, and is updated monthly with newCDSRreviews to reduce benchmark leakage\. To further mitigate leakage, we introduceSciConHarness, aclean\-roomevaluation harness with controlled web search and browsing tools\. Finally, we develop a factual evaluation pipeline that decomposes generated conclusions into atomic facts111Defined as “statements containing single pieces of information”\[[73](https://arxiv.org/html/2606.11337#bib.bib38),[78](https://arxiv.org/html/2606.11337#bib.bib141)\]\.and uses LLM\-based judges to measurefactual precision\(correctness\),factual recall\(coverage\), and F1 \(overall quality\), showing strong agreement with expert judgments\.

Evaluating 8 frontier models and deep research agents onSciConBench, we find scientific conclusion synthesis remains an open challenge underclean\-roomevaluation: the best system,o3\-deep\-research, achieves only F1=0\.337=0\.337\. Across systems,clean\-roomevaluation reduces factual F1 by0\.020\.02–0\.1720\.172relative to unconstrained settings \(agents can access ground\-truth artifacts\), indicating that much of the apparent performance arises from retrieving ground\-truth artifacts rather than from genuine synthesis\. This highlights the importance ofclean\-roomevaluation for valid measurement of open\-domain AI agent capabilities\. Finally, we audit consumer\-facing agents \(e\.g\., Google AI Overview, OpenEvidence\) increasingly used in health contexts\[[4](https://arxiv.org/html/2606.11337#bib.bib113),[88](https://arxiv.org/html/2606.11337#bib.bib111),[113](https://arxiv.org/html/2606.11337#bib.bib112)\]\. Despite access to ground\-truth artifacts, these systems remain unreliable \(F1=0\.361=0\.361–0\.5220\.522\), often generating incomplete and sometimes contradictory conclusions\. Our main contributions are:

1. 1\.We introduceSciConBench, a large\-scale,livebenchmark of 9\.11K questions paired with expert\-written scientific conclusions, capturing real\-world, open\-domain scientific synthesis tasks\.
2. 2\.We developSciConHarness, aclean\-roomevaluation harness that provides controlled web tools for AI agents, mitigating leakage and enabling valid measurement of synthesis capabilities\.
3. 3\.Using ourexpert\-validatedfactual evaluation pipelinethat decomposes conclusions into facts and measuresfactual precisionandrecall, our benchmark evaluation of frontier models and deep research agents reveals thatreliable scientific conclusion synthesis remains unresolved\.
4. 4\.Weauditwidely\-deployed consumer\-facing agents, including Google AI Overview and OpenEvidence, and findthey synthesize incomplete and sometimes contradictory scientific conclusions, raising concerns for high\-stakes decision\-making relying on them in real\-world health contexts\.

## 2TheSciConBenchDataset

SciConBenchleverages theCochrane Database of Systematic Reviewsto evaluate models on the long\-horizon task of scientific conclusion synthesis: retrieving relevant sources, appraising evidence quality, and integrating heterogeneous evidence to construct a long\-form, expert\-level conclusion\.

Background\.TheCochrane Database of Systematic Reviews \(CDSR\)is a peer\-reviewed collection of systematic reviews that synthesize evidence to answer well\-defined clinical and public health questions\[[110](https://arxiv.org/html/2606.11337#bib.bib1)\]\. Each review identifies and evaluates a body of relevant studies—from a few to hundreds of publications—to answer a well\-defined clinical or public health question\. The review appraises the quality of the evidence, reconciles conflicting findings, and synthesizes the overall evidence into a concise, paragraph\-long conclusion\[[46](https://arxiv.org/html/2606.11337#bib.bib103),[72](https://arxiv.org/html/2606.11337#bib.bib101),[111](https://arxiv.org/html/2606.11337#bib.bib4)\]\. To ensure conclusions remain current as new scientific evidence emerges,CDSRre\-evaluates the literature every two years and updates the conclusions\[[26](https://arxiv.org/html/2606.11337#bib.bib7)\], thoughFrenchet al\.\[[36](https://arxiv.org/html/2606.11337#bib.bib6)\]finds that most conclusions remain stable over time\. As the “gold\-standard” for evidence\-based synthesis\[[104](https://arxiv.org/html/2606.11337#bib.bib100)\],CDSR’s expert\-written conclusions inform real\-world clinical decisions and health policy, making it a valuable data source for evaluating AI agents’ ability to synthesize conclusions from the scientific literature\.

Automated Data Collection: ALiveBenchmark\.We construct alivebenchmark by drawing from systematic reviews from the regularly updatedCDSR\. In total, as of January 1st, 2026, we collected 9,531 systematic reviews, of which 424 were withdrawn, yielding 9,107 valid reviews\. Given growing concerns around benchmark leakage during the pre\-training of frontier models\[[126](https://arxiv.org/html/2606.11337#bib.bib99)\], we design our benchmark to be continuously updated as newCDSRreviews are released\. In contrast to static benchmarks\[[13](https://arxiv.org/html/2606.11337#bib.bib8),[63](https://arxiv.org/html/2606.11337#bib.bib98),[71](https://arxiv.org/html/2606.11337#bib.bib16)\], this ensures timely evaluations of latest agents, while mitigating leakage\.

Data Preprocessing\.We convert the expert\-authored systematic reviews into structured question–answer \(QA\) evaluation units\. For each review, we use theObjectivesas the basis for the question and employ theAuthors’ Conclusionsas the answer\. See Figure[S7](https://arxiv.org/html/2606.11337#A2.F7)for an example\. TheObjectivesdefine the research question and scope the review aims to address—typically structured around the population, intervention, comparator, and outcomes \(PICO\) framework—while theConclusionsprovide the corresponding evidence\-based synthesis, including key findings and their certainty\.

Question Generation\.SinceObjectivesare typically written as declarative statements rather than questions, we transform them into clinically grounded questions with the PICO framework \(Participants, Interventions, Comparisons, Outcomes\), which is widely used to formulate clinical research questions and guide evidence retrieval\[[112](https://arxiv.org/html/2606.11337#bib.bib48)\]\. Usinggpt\-5\-chat, we convert eachObjectiveinto a sentence\-style question, consistent with prior work showing that users prefer sentence\-style queries over keyword\-based inputs when using LLMs\[[23](https://arxiv.org/html/2606.11337#bib.bib51),[130](https://arxiv.org/html/2606.11337#bib.bib55)\]\. This formulation aligns with real\-world usage, where clinicians and scientists increasingly ask scientific and medical questions to AI systems for decision\-making\[[12](https://arxiv.org/html/2606.11337#bib.bib36),[40](https://arxiv.org/html/2606.11337#bib.bib45)\]\. Applying this pipeline, we construct a QA\-style benchmark comprising 9,107 samples spanning nearly 30 years of systematic reviews across diverse scientific and clinical domains, from neonatal care to kidney disease\. We provide additional details in §[B\.1](https://arxiv.org/html/2606.11337#A2.SS1), including prompts and an example question \(Figure[S3](https://arxiv.org/html/2606.11337#A2.F3)\)\.

Validation\.We validate whether the generated questions faithfully reflect the intent and scope of eachCDSRreview with annotations by two medical students with extensive clinical research experience\. Given the generated question,Objectives, andBackgroundof theCDSRreview, annotators evaluate question quality along three dimensions grounded in prior works\[[57](https://arxiv.org/html/2606.11337#bib.bib31),[74](https://arxiv.org/html/2606.11337#bib.bib15),[91](https://arxiv.org/html/2606.11337#bib.bib83)\]: Faithfulness, PICO Completeness, and Clarity & Answerability\. In the calibration phase, the annotators label 10 questions to validate the task guidelines and resolve disagreements\. They then independently label an additional 10 questions to assess reliability, measured using Gwet’s AC1, which is robust to skewed label distributions\[[79](https://arxiv.org/html/2606.11337#bib.bib13),[124](https://arxiv.org/html/2606.11337#bib.bib14)\]\. Agreement is high across dimensions \(AC1: 0\.756–1\.00; see Table[S4](https://arxiv.org/html/2606.11337#A2.T4)\), comparable to or exceeding prior work\[[57](https://arxiv.org/html/2606.11337#bib.bib31),[91](https://arxiv.org/html/2606.11337#bib.bib83)\]\. Given high agreement, each annotator independently labeled 40 questions \(N=100N=100total\)\. We find generated questions to be faithful \(92%\), PICO\-complete \(92%\), and clear & answerable \(96%\)\. Appendix §[B\.2](https://arxiv.org/html/2606.11337#A2.SS2)provides details, including annotation guidelines \(Figure[S4](https://arxiv.org/html/2606.11337#A2.F4)\) and interface \(Figures[S5](https://arxiv.org/html/2606.11337#A2.F5)–[S6](https://arxiv.org/html/2606.11337#A2.F6)\)\.

## 3SciConHarness: Controlled Evaluation in the Clean\-Room

Synthesizingscientific conclusionswith open\-domain information access is challenging, as models may find sources that contain the already synthesized information wholesale, thus turning the harder synthesis task into a simpler retrieval task\. We address this challenge withSciConHarness, an MCP\-based harness that provides controlled access to web search and browsing tools\.SciConHarnessenforces a “clean\-room setting” preventing models’ access to ground\-truth systematic reviews, while retrieving, integrating, and reasoning over scientific evidence from the open web\[[48](https://arxiv.org/html/2606.11337#bib.bib12)\]\.

Overview\.SciConHarnessorchestrates the full model\-tool interactions, executing iterative model\-driven tool calls with web access, and appending tool outputs to the model context\. The harness was built using Ai2’s open\-sourcedr\-agent\-lib\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\], and supports iterative model\-tool interaction to discover and obtain scientific evidence from the open web, following prior work\[[38](https://arxiv.org/html/2606.11337#bib.bib27),[66](https://arxiv.org/html/2606.11337#bib.bib25),[67](https://arxiv.org/html/2606.11337#bib.bib26),[100](https://arxiv.org/html/2606.11337#bib.bib30)\]: \(1\)google\_search\(query→\\rightarrowtop search results\) via Serper API, \(2\)web\_browse\(URL→\\rightarrowpage text\) via Jina Browsing API, and \(3\)paper\_search\(query→\\rightarrowrelevant paragraphs from open\-access papers\) via Semantic Scholar Full\-Text Search API\.222Links:[https://serper\.dev/](https://serper.dev/),[https://jia\.ai/reader/](https://jia.ai/reader/),[https://api\.semanticscholar\.org/api\-docs](https://api.semanticscholar.org/api-docs)Like prior works\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\],google\_searchandpaper\_searchretrieve the top 10 results by default, though models may request more\. Forweb\_browse, whose outputs can be long, we usegpt\-5\-minito summarize webpage text, reducing context usage and cost\[[66](https://arxiv.org/html/2606.11337#bib.bib25),[100](https://arxiv.org/html/2606.11337#bib.bib30)\]\. For brevity, we leave additional implementation details in §[C](https://arxiv.org/html/2606.11337#A3)\.

The Clean\-Room\.Analogous to controlled environments in the life sciences to prevent contamination\[[1](https://arxiv.org/html/2606.11337#bib.bib104)\], we designSciConHarnessto support aclean\-roomevaluation protocol that mitigates benchmark leakage\. To prevent access to the ground\-truth artifacts \(e\.g\.,CDSRarticles\), we implement a filtering middleware inSciConHarnessthat inspects all tool outputs and enforces explicitclean\-roomprotocols before returning them to the model\. Concretely, we filter outputs fromgoogle\_searchandpaper\_searchusing: \(1\) URLs fromCDSRdomains, \(2\) result titles containing “cochrane” or matching aCDSRreview title, or \(3\) results published after the ground\-truth review’s publication date \(to prevent indirect leakage from derivative content\)\. Forweb\_browse, we filter content containing both “cochrane” or the ground\-truthCDSRreview title\. Ourclean\-roomprotocol is conservative: it may occasionally filter legitimate content, but it ensures that the benchmark measures synthesis rather than memorization or shortcut retrieval\.

Validation\.We validateSciConHarness’sclean\-roomfiltering with a random sample ofN=150N\{=\}150tool outputs from logged tool calls from the full benchmark evaluations \(§[6](https://arxiv.org/html/2606.11337#S6)\) \(5050per tool; e\.g\., search result forgoogle\_search, page text forweb\_browse\)\. These are stratified evenly between filtered and unfiltered cases\. We manually annotated each output, measuring false positives \(over\-filtering benign content\) and false negatives \(missed leakage\), with the latter being more critical as they allow access to the reference conclusions\. As shown in Table[S6](https://arxiv.org/html/2606.11337#A3.T6), the filtering achieves high precision \(0\.88–0\.92; avg\. 0\.933\) and recall \(0\.957–1\.00; avg\. 0\.972\) across tools, indicating effective leakage mitigation with minimal over\-filtering\. Notably,allground\-truthCDSRarticles were successfully removed across all tools, demonstrating robust prevention of direct leakage\. The remaining false negatives arise fromindirectleakage \(e\.g\., news coverage regarding the ground\-truthCDSRarticle\)\.

## 4Measuring Factual Quality

How to evaluate thefactual qualityof scientific conclusions? We decompose both generated and reference conclusions intoatomic facts\(§[4\.1](https://arxiv.org/html/2606.11337#S4.SS1)\)\. Then, we compare the extent to which the generated claims: \(1\) are supported and non\-contradictory to the reference \(factual precision\) and \(2\) support facts necessary to answer the question \(factual recall; see §[4\.2](https://arxiv.org/html/2606.11337#S4.SS2)\)\. Finally, we scale our evaluation procedure using LLM\-based judges \(§[4\.3](https://arxiv.org/html/2606.11337#S4.SS3)\), finding strong agreement with domain experts\.

### 4\.1Decomposing Conclusions into Atomic Facts\.

Pipeline Overview\.We design a modular pipeline that transforms long\-form scientific conclusions into high\-quality, self\-contained atomic facts\. Building on prior works on long\-form factuality evaluations\[[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38),[123](https://arxiv.org/html/2606.11337#bib.bib140)\], our pipeline comprises six steps\. We tokenize paragraph\-length conclusions into sentences \(step 0: preprocessing\) and decompose each into atomic facts usinggpt\-5\.1\(step 1: decomposition\)\. Then, we decontextualize each fact to resolve implicit references \(step 2: decontextualization\) and ensure self\-containment by rewriting incomplete facts to fill in missing references, comparisons, and conditions \(step 3: incomplete fact rewriting\)\. Finally, we filter facts for relevance to the original question, retaining only those that directly contribute to answering it or provide necessary context \(step 4: relevance filtering\) and filter out redundant facts within each sentence \(step 5: redundancy filtering\)\. For details, see §[D\.1](https://arxiv.org/html/2606.11337#A4.SS1)and Figures[S9](https://arxiv.org/html/2606.11337#A4.F9)\-[S13](https://arxiv.org/html/2606.11337#A4.F13)\.

Cost ConsiderationSince generating atomic facts from paragraph\-length conclusions is financially and computationally expensive at scale, the pipeline is designed to bemodular, enabling per\-step model selection and hyperparameter selection to balance quality and cost\. We allocate higher\-capability models \(e\.g\.,gpt\-5\.1\) to early, formative stages \(steps 1\-2\) to ensure high\-quality initial facts, and smaller models \(e\.g\.,gpt\-5\-mini\) to later classification\-style steps \(steps 3\-5\)\. OnCDSRconclusions, this reduces cost from $0\.35 to $0\.13 per instance\. Full design decisions are in §[D\.2](https://arxiv.org/html/2606.11337#A4.SS2)\.

Validation\.We evaluate the quality of generated facts via expert annotations from two medical doctors with substantial clinical practice and research experience\. Given a source sentence, its paragraph context, and its extracted atomic facts, annotators assess fact quality along dimensions grounded in prior work\[[57](https://arxiv.org/html/2606.11337#bib.bib31),[69](https://arxiv.org/html/2606.11337#bib.bib37),[74](https://arxiv.org/html/2606.11337#bib.bib15),[91](https://arxiv.org/html/2606.11337#bib.bib83)\]:FaithfulnessandCompletenessat the fact\-level \(e\.g\., per fact\), andComprehensivenessandRedundancyat the sentence\-level \(e\.g\., per sentence and its set of atomic facts\)\. See §[D\.3](https://arxiv.org/html/2606.11337#A4.SS3)for details, including annotation guidelines \(Figure[S8](https://arxiv.org/html/2606.11337#A4.F8)\) and interface \(Figures[S19](https://arxiv.org/html/2606.11337#A5.F19)\-[S21](https://arxiv.org/html/2606.11337#A5.F21)\)\. We find agreement to be high across all dimensions \(AC1: 0\.597–0\.955\), comparable to prior work \(Table[S7](https://arxiv.org/html/2606.11337#A4.T7)\)\. Given high agreement, annotators independently label 90 sentences each, stratified sampled from generated and reference conclusions, yielding a total ofNsent=200N\_\{\\text\{sent\}\}\{=\}200sentences withNfacts=469N\_\{\\text\{facts\}\}\{=\}469\. We find that generated facts are largely faithful \(96\.4%\), complete \(96\.0%\), comprehensive \(98\.0%\), and non\-redundant \(90\.5%\); see Table[S8](https://arxiv.org/html/2606.11337#A4.T8)\.

### 4\.2Factual Precision and Recall Metrics

Scientific conclusions should be accurate and comprehensive\[[109](https://arxiv.org/html/2606.11337#bib.bib40)\]\. Thus we define two metrics to measure factual quality, capturing correctness and coverage\[[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38)\]\. Following prior work\[[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38),[116](https://arxiv.org/html/2606.11337#bib.bib71)\], we adopt a source\-grounded view of factuality, defining correctness of statements with respect to a trusted reference—here,CDSRreviews, a “gold standard” in evidence\-based science\[[104](https://arxiv.org/html/2606.11337#bib.bib100)\]\.

Factual precisionmeasures the extent to which facts from generated conclusions are supported and non\-contradictory with respect to theCDSRreview,RR\. Letεx\\varepsilon\_\{x\}represent the extracted facts from a generated conclusionxx\. Each facte∈εxe\\in\\varepsilon\_\{x\}is labeled asContradicted,Supported, orNot Supported\. We compute the factual precision of generated conclusionxxas:\(1\|εx\|​∑e∈εx𝟏​\[e​isSupported by​R\]\)⋅\(1−1\|εx\|​∑e∈εx𝟏​\[e​isContradicted by​R\]\)\.\(\\frac\{1\}\{\|\\varepsilon\_\{x\}\|\}\\sum\_\{e\\in\\varepsilon\_\{x\}\}\\mathbf\{1\}\[e\\text\{ is \}\\textsc\{Supported by \}R\]\)\\cdot\(1\-\\frac\{1\}\{\|\\varepsilon\_\{x\}\|\}\\sum\_\{e\\in\\varepsilon\_\{x\}\}\\mathbf\{1\}\[e\\text\{ is \}\\textsc\{Contradicted by \}R\]\)\.This formulation rewards conclusions whose facts are supported, while explicitly penalizing contradictions, consistent with prior work\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38)\]\.

Factual recallmeasures the extent to which generated conclusions cover facts from theAuthors’ ConclusionsofCDSRreviews, treated as the authoritative set required to answer the question\. Letxxrepresent the generated conclusion andεA′\\varepsilon\_\{A\}^\{\\prime\}be the corresponding reference facts from theAuthors’ Conclusions, A\. Each facte′∈εA′e^\{\\prime\}\\in\\varepsilon\_\{A\}^\{\\prime\}is labeled asSupportedorNot Supported\. We compute the factual recall of the generated conclusionxxas:\(1\|εA′\|​∑e′∈εA′𝟏​\[e′​isSupported by​x\]\)\(\\frac\{1\}\{\|\\varepsilon\_\{A\}^\{\\prime\}\|\}\\sum\_\{e^\{\\prime\}\\in\\varepsilon\_\{A\}^\{\\prime\}\}\\mathbf\{1\}\[e^\{\\prime\}\\text\{ is \}\\textsc\{Supported by \}x\]\)\. This formulation rewards conclusions that cover the reference facts necessary to answer the question\.

Factual F1measures the overall factual quality of the generated conclusions as the harmonic mean of factual precision and recall; thus, a high factual F1\-score requires both high correctness \(precision\) and strong coverage \(recall\) of the generated conclusion\.

### 4\.3Measuring Factual Precision and Recall at Scale\.

Annotating for factual precision and recall is challenging, requiring annotators to understandCDSRsystematic reviews and reason over complex clinical evidence with substantial domain expertise\. However, with dozens of atomic facts per conclusion, manual evaluation is expensive and infeasible\. We therefore carefully construct annotation guidelines, develop an expert\-annotated gold\-standard dataset, and validate LLM\-based judges against it\. Full details are provided in §[E](https://arxiv.org/html/2606.11337#A5)\.

Creating the Gold\-Standard Dataset\.For both factual precision and recall tasks, we develop annotation guidelines \(§[E\.1\.1](https://arxiv.org/html/2606.11337#A5.SS1.SSS1); Figures[S16](https://arxiv.org/html/2606.11337#A5.F16)–[S17](https://arxiv.org/html/2606.11337#A5.F17)\) and conduct multiple rounds of annotation with two medical doctors, with a third independently adjudicating disagreements to produce consensus labels \(§[E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2)\)\. This process resulted in a gold\-standard dataset ofN=129N=129facts for precision andN=119N=119for recall, representing a substantial annotation effort, exceeding or matching the scale of prior expert\-annotated evaluations of LLM judges\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[27](https://arxiv.org/html/2606.11337#bib.bib9),[28](https://arxiv.org/html/2606.11337#bib.bib35),[56](https://arxiv.org/html/2606.11337#bib.bib11),[97](https://arxiv.org/html/2606.11337#bib.bib10)\]\. Experts reported an average annotation time of 6 minutes per fact\. See annotation interfaces in Figures[S19](https://arxiv.org/html/2606.11337#A5.F19)–[S21](https://arxiv.org/html/2606.11337#A5.F21)\.

Table 1:Percentage agreement \(%\), Cohen’sκ\\kappa, and Gwet’s AC1 between experts and the LLM judge for factual precision and recall usinggpt\-5\.4\-mini, the best\-performing judge\.\)\.Agreement and resulting dataset\.Between the two experts, we observe moderate agreement for factual precision \(Cohen’sκ=0\.517\\kappa=0\.517, Gwet’s AC1=0\.544=0\.544\) and substantial agreement for factual recall \(Cohen’sκ=0\.658\\kappa=0\.658, Gwet’s AC1=0\.671=0\.671\)\[[60](https://arxiv.org/html/2606.11337#bib.bib108)\]\. Agreement is comparable to or exceeds prior work\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38)\], indicating that our annotation task is well\-defined and yields reliable labels\. See Table[S9](https://arxiv.org/html/2606.11337#A5.T9)for the full list of agreement scores across annotation rounds\.333Expert disagreement can reflectlegitimate differencesin evaluations that reflect background and expertise\[[50](https://arxiv.org/html/2606.11337#bib.bib97)\]\.The resulting gold\-standard dataset contains 19Contradicted, 54Supported, and 56Not Supportedlabels for factual precision, and 48Supportedand 71Not Supportedlabels for factual recall\.

Validating LLM Judge\.Using the gold\-standard dataset, we validate LLM\-based judges through extensive prompt design and systematic evaluation across three models \(gpt\-5\.4\-mini,claude\-haiku\-4\.5, andgemini\-3\-flash\), varying reasoning levels, temperature settings, and prompts \(zero\-shot, few\-shot\)\. In the few\-shot setting, we include six annotated examples from our gold\-standard dataset,444For each task, we exclude few\-shot examples from evaluation to avoid data leakage\.following prior work\[[76](https://arxiv.org/html/2606.11337#bib.bib123)\]\. We detail the input features and prompt design in §[E\.2\.1](https://arxiv.org/html/2606.11337#A5.SS2.SSS1)\-[E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2)and show prompts in Figures[S22](https://arxiv.org/html/2606.11337#A5.F22)\-[S23](https://arxiv.org/html/2606.11337#A5.F23)\.

Validation Results\.For both tasks,gpt\-5\.4\-miniachieves the strongest performance in its best configurations \(macro F1 of 0\.837 for precision and 0\.868 for recall\), which we use in our downstream evaluation\.555Factual precision: few\-shot prompt, no reasoning, temp 0\.2; factual recall: zero\-shot prompt, no reasoning, temp 1\.0\.In addition,gpt\-5\.4\-minidemonstrates strong alignment with the expert annotators\. As shown in Table[1](https://arxiv.org/html/2606.11337#S4.T1), agreement betweengpt\-5\.4\-minias a judge and individual experts is comparable to—and in some cases exceeds—the agreement between the experts themselves\.gpt\-5\.4\-minialso passes theAlternative Annotator Test\[[21](https://arxiv.org/html/2606.11337#bib.bib34)\], a leave\-one\-out statistical test for evaluating substitute annotators, with a winning rate of 1\.0 on both tasks, statistically supporting its use as a reliable substitute annotator\. Together, high task performance and strong agreement with experts validate both the quality of our prompts and the use of frontier LLMs as reliable evaluators for factual precision and recall\. To identify common failure modes and areas to improve LLM judge accuracy and reliability, we conduct an error analysis of the LLM judge in §[E\.2\.4](https://arxiv.org/html/2606.11337#A5.SS2.SSS4)\. Tables[S10](https://arxiv.org/html/2606.11337#A5.T10)\-[S11](https://arxiv.org/html/2606.11337#A5.T11)present the full evaluation results for both factual precision and recall, with details in §[E\.2\.3](https://arxiv.org/html/2606.11337#A5.SS2.SSS3)\.

## 5Evaluation Details

Overview\.We benchmark eight state\-of\-the\-art models and deep research agents equipped withSciConHarnessonSciConBench\. We evaluate three frontier LLMs at the time of our experiment:gpt\-5\.1\[[82](https://arxiv.org/html/2606.11337#bib.bib119)\],claude\-sonnet\-4\.5\[[10](https://arxiv.org/html/2606.11337#bib.bib121)\], andgemini\-3\-pro\[[42](https://arxiv.org/html/2606.11337#bib.bib120)\]—under threeSciConHarnesssettings: \(1\)base\(parametric\-only, no tools\), \(2\)SciConHarnesstools\(without the clean\-room protocol, allowing retrieval access to ground\-truth artifacts\), and \(3\)SciConHarnesstools \+ clean\-room\(filters ground\-truth leakage\)\. These settings enable comparisons among parametric models, nonparametric models with unrestricted retrieval, and nonparametric models under our controlledclean\-roomretrieval\. We also evaluate other models and deep research agents, including Ai2’s open\-sourceDR Tulu\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\], OpenAI’so3\-deep\-research\[[84](https://arxiv.org/html/2606.11337#bib.bib116)\]ando4\-mini\-deep\-research\[[85](https://arxiv.org/html/2606.11337#bib.bib115)\], and Perplexity’ssonar\-deep\-research\[[94](https://arxiv.org/html/2606.11337#bib.bib118)\]andsonar\-reasoning\-pro\[[95](https://arxiv.org/html/2606.11337#bib.bib117)\]\.666We only consider closed\-source deep research agents that provide an API and support integration withSciConHarnessat the time of writing, excluding Gemini Deep Research and Grok Deep Research\.As these agents natively use tools, we evaluate them undertoolsandtools \+ clean\-roomsettings\.

Setup\.Across all evaluations, we use the same system prompt \(Figure[S25](https://arxiv.org/html/2606.11337#A6.F25)\); except forDR Tulu, for which we adapt their default system prompt \(see Figure[S26](https://arxiv.org/html/2606.11337#A6.F26)\)\. All systems must produce a paragraph\-length synthesized conclusion\. We use default, recommended hyperparameters \(e\.g\., temperature\) and the highest available reasoning level\. To support long\-form synthesis across potentially hundreds of web sources, we do not limit the number of tool calls\.777Except foro3\-deep\-researchando4\-mini\-deep\-research, which were capped atN=30N\{=\}30tool calls per query due to cost constraints; only 2% of their queries reached this limit during evaluation\.Some provider\-hosted agents do not support direct integration withSciConHarness, requiring alternative strategies to enforce ourclean\-roomprotocol, e\.g\., remote MCP endpoints\. See §[F](https://arxiv.org/html/2606.11337#A6)for details\. To mitigate data contamination and benchmark leakage from pretraining, we obtainN=268N=268samples fromSciConBench, restricting toCDSRreviews published after the latest model knowledge cutoff \(e\.g\., the end of January 2025 forgemini\-3\-pro\)\. For benchmark performance comparison, this sample size provides sufficient power to detect a statistically significant difference between two model performances in factual F1\-scores of at leastΔ≈0\.037\\Delta\\approx 0\.037atα=0\.05\\alpha=0\.05with power0\.80\.8\(see §[G](https://arxiv.org/html/2606.11337#A7)for the power analysis\)\. After generating conclusions, we decompose both reference and generated conclusions into atomic facts \(§[4\.1](https://arxiv.org/html/2606.11337#S4.SS1)\) and evaluate factual precision, recall, and F1 using our expert\-validated LLM judge\. See §[H](https://arxiv.org/html/2606.11337#A8)for cost analysis and Tables[S12](https://arxiv.org/html/2606.11337#A8.T12)–[S14](https://arxiv.org/html/2606.11337#A8.T14)for the cost breakdowns of the end\-to\-end benchmark evaluation\.

Table 2:Benchmark performance of models and deep research \(DR\) acrossSciConHarnesssettings\. We report the macro factual precision, recall, and F1 \(variance in parentheses\)\.†\\daggerindicates evaluation on a subset \(N=100N=100of 268\) due to high cost \(\>$2/query\)\.Bolddenotes the best performance without clean\-room, andunderlinewith clean\-room\.ΔTools\\Delta\_\{\\text\{Tools\}\}F1is the F1 change from Base toSciConHarness, andΔClean\\Delta\_\{\\text\{Clean\}\}F1denotes the change when adding clean\-room toSciConHarness\.
## 6Results

Models and deep research agents have substantial room for improvement\.Table[2](https://arxiv.org/html/2606.11337#S5.T2)shows the benchmark performance of models and deep research agents acrossSciConHarnesssettings\. Across all systems, factual F1\-score remains far from reliable for scientific conclusion synthesis\. Even in the favorable setting without the clean\-room, no system exceeds 0\.63 on any metric\. The best\-performingo3\-deep\-researchachieves the highest precision \(0\.628\), recall \(0\.483\), and F1 \(0\.508\), significantly outperforming the second\-besto4\-mini\-deep\-research\(F1 0\.427; paired t\-test:t​\(267\)=6\.1t\(267\)=6\.1,p<0\.001p<0\.001, Cohen’sd=0\.37d=0\.37\)\. Under clean\-room evaluation, which better isolates true synthesis capability,o4\-mini\-deep\-researchachieves the highest precision \(0\.467\),gpt\-5\.1the highest recall \(0\.408\), ando3\-deep\-researchthe highest F1 \(0\.337\), again significantly outperformingo4\-mini\-deep\-research\(F1 0\.315;t​\(267\)=2\.36t\(267\)=2\.36,p<0\.05p<0\.05,d=0\.14d=0\.14\)\.DR Tulu, despite being the most cost\-efficient fully open agent, shows the weakest performance\.

At the conclusion level, factual quality issues were pervasive across models and deep research agents: 44\.8\-84\.0% of generated conclusions contained at least one fact contradicting the referenceCDSRreview, and nearly all contained at least one fact not supported by the reference review \(Table[S16](https://arxiv.org/html/2606.11337#A9.T16)\)\. The generated conclusions were also incomplete, failing to support and cover 55\.4–84\.7% of reference facts fromCDSRreviews \(Table[S15](https://arxiv.org/html/2606.11337#A9.T15)\)\. These findings indicate that current models and agents often produce scientific conclusions that are incomplete, contradictory, or not supported by the reference review\. As AI agents are increasingly used to synthesize evidence for clinical and scientific decision\-making, such errors may distort high\-stakes judgments\.

Tool augmentations require precise use and effective evidence integration to improve synthesis\.Among base models,gpt\-5\.1achieves the highest F1 score \(0\.332\) compared toclaude\-sonnet\-4\.5\(0\.291\) andgemini\-3\-pro\(0\.239\), highlighting limited synthesis capability from parametric knowledge alone\. AddingSciConHarnesstools \(no clean\-room\) improves performance in most cases, but unevenly:claude\-sonnet\-4\.5achieves large gains in recall \(\+0\.139\) and F1 \(\+0\.091\) with relatively efficient tool use \(8\.97 calls/query\), despite a drop in precision \(\-0\.029\), whilegpt\-5\.1shows only a marginal gain \(\+0\.012 F1\) despite heavier tool usage \(14\.05 calls/query; see Table[S17](https://arxiv.org/html/2606.11337#A9.T17)\)\.gemini\-3\-prodegrades with tools and uses them least \(5\.59 calls/query\), indicating poor integration\. This demonstrates the challenges ofscientific conclusion synthesis: even without clean\-room constraints—where agents can access ground\-truth artifacts—strong performance requires not just access to tools, but disciplined, intentional tool use and effective evidence integration\.

Our clean\-room evaluationconsistentlyattenuates performance\.Applying our clean\-room protocol consistently reduces F1 by0\.020\.02–0\.1720\.172across all systems, even eliminating gains from unconstrained tool use\. For example,claude\-sonnet\-4\.5drops by−0\.085\-0\.085, nearly offsetting its\+0\.091\+0\.091gain with tools, whilesonar\-reasoning\-proshows the largest decline \(−0\.172\-0\.172\)\. Even top\-performingo3\-deep\-research\(F1 0\.508 without clean\-room\) degrades sharply by−0\.17\-0\.17\. These results indicate that much of the observed performance without clean\-room constraints is driven byretrievalof ground\-truth artifacts rather than genuinesynthesis\. In our experiment, we observe that agents actively exploit open\-web access toretrievethe ground\-truthCDSRconclusion—even when instructed not to—shortcutting synthesis and creating leakage\[[126](https://arxiv.org/html/2606.11337#bib.bib99)\]\. By mitigating leakage with our clean\-room evaluation, we enforce controlled evaluation that prevents conflating retrieval with synthesis, avoiding overestimation of true capability and maintaining construct validity\[[133](https://arxiv.org/html/2606.11337#bib.bib47)\]\.

For brevity, we summarize additional analyses from Appendix §[I](https://arxiv.org/html/2606.11337#A9), which includes label distributions \(§[I\.1](https://arxiv.org/html/2606.11337#A9.SS1)\), tool usage \(§[I\.2](https://arxiv.org/html/2606.11337#A9.SS2)\), failure mode analysis \(§[I\.3](https://arxiv.org/html/2606.11337#A9.SS3)\), robustness to conclusion length \(§[I\.4](https://arxiv.org/html/2606.11337#A9.SS4)\), and Pareto frontiers of performance vs\. cost and time \(§[I\.5](https://arxiv.org/html/2606.11337#A9.SS5)\)\. Tool usage varies substantially across systems, with OpenAI agents relying heavily ongoogle\_searchandweb\_browse, while Claude and Gemini usepaper\_searchmore\. Across all systems,google\_searchexhibits high clean\-room filtering rates \(49\.6%–81\.8%\), highlighting the importance of clean\-room evaluation to mitigate benchmark leakage\. Failure mode analysis further reveals that models often invert treatment effects, mischaracterize evidence quality, and generate overly broad conclusions lacking outcome\-level specificity, which may mislead scientific interpretations and downstream clinical decisions\. Finally, longer conclusions generally trade higher recall for lower precision, indicating that simply generating longer outputs does not improve factual F1 and that agent quality, rather than verbosity, drives performance\.

### 6\.1Auditing Consumer\-Facing Agents

Figure 2:Performance of consumer\-facing AI agents \(precision, recall, F1; variance in parentheses\)\.UsingSciConBench, we audit proprietary, consumer\-facing agents increasingly used by laypeople and clinicians to synthesize scientific conclusions in high\-stakes health contexts\[[4](https://arxiv.org/html/2606.11337#bib.bib113),[88](https://arxiv.org/html/2606.11337#bib.bib111),[113](https://arxiv.org/html/2606.11337#bib.bib112)\]\. We evaluate Google AI Overview, Google AI Mode, and OpenEvidence on the sameN=268N=268benchmark samples without the clean\-room protocol, decompose the conclusions into facts, and evaluate them \(§[5](https://arxiv.org/html/2606.11337#S5)\)\. See §[J](https://arxiv.org/html/2606.11337#A10)for details on data collection\.

Consumer\-facing agents generate unreliable scientific conclusions despite access to ground\-truth artifacts\.As shown in Table[2](https://arxiv.org/html/2606.11337#S6.F2), Google AI Mode \(F1: 0\.361\) and Google AI Overview \(F1: 0\.384\) perform poorly, with weak recall \(0\.380 and 0\.367\) indicating limited coverage of key reference facts\. More concerningly, a substantial share of generated conclusions containing at least one fact contradicting theCDSRreview: 56\.3% for Google AI Overview and 59% for Google AI Mode \(Table[S16](https://arxiv.org/html/2606.11337#A9.T16)\)—despite access to the ground\-truthCDSRreview\. This suggests failures from these consumer\-facing agents come from unreliable synthesis, not missing information\. Given these agents’ widespread use in health contexts\[[113](https://arxiv.org/html/2606.11337#bib.bib112)\], these contradiction rates are particularly concerning\. OpenEvidence performs better \(F1: 0\.522\), achieving the highest precision \(0\.580\) and recall \(0\.541\)\. However, it remains far from reliable: 50\.8% of generated conclusions from OpenEvidence contain at least one fact contradicting theCDSRreview \(Table[S16](https://arxiv.org/html/2606.11337#A9.T16)\) and cover only half \(51\.7%\) of the reference facts \(Table[S15](https://arxiv.org/html/2606.11337#A9.T15)\)\. Thus, even the strongest agent focused on health context frequently omits critical information and occasionally contradicts established conclusions, which may pose a risk for high\-stakes decision\-making in real\-world health contexts\.

## 7Related Work

Table 3:Comparison ofSciConBenchwith prior benchmarks\.Evaluations for Science, Health, and Factuality\.LLMs are increasingly deployed in science and health contexts\[[25](https://arxiv.org/html/2606.11337#bib.bib41),[109](https://arxiv.org/html/2606.11337#bib.bib40)\]\. To evaluate their capabilities, prior work has developed benchmarks for biomedical and scientific question answering\[[30](https://arxiv.org/html/2606.11337#bib.bib39),[53](https://arxiv.org/html/2606.11337#bib.bib64),[114](https://arxiv.org/html/2606.11337#bib.bib63)\], clinical and scientific reasoning\[[61](https://arxiv.org/html/2606.11337#bib.bib79),[64](https://arxiv.org/html/2606.11337#bib.bib80),[106](https://arxiv.org/html/2606.11337#bib.bib81)\], scientific text summarization and simplification\[[16](https://arxiv.org/html/2606.11337#bib.bib52),[33](https://arxiv.org/html/2606.11337#bib.bib75),[107](https://arxiv.org/html/2606.11337#bib.bib70)\], risk of bias assessment\[[49](https://arxiv.org/html/2606.11337#bib.bib93),[70](https://arxiv.org/html/2606.11337#bib.bib95),[108](https://arxiv.org/html/2606.11337#bib.bib94),[118](https://arxiv.org/html/2606.11337#bib.bib96)\], factuality evaluation\[[54](https://arxiv.org/html/2606.11337#bib.bib92),[58](https://arxiv.org/html/2606.11337#bib.bib65),[59](https://arxiv.org/html/2606.11337#bib.bib61),[115](https://arxiv.org/html/2606.11337#bib.bib142),[116](https://arxiv.org/html/2606.11337#bib.bib71)\], citation grounding and verifiability\[[37](https://arxiv.org/html/2606.11337#bib.bib69),[39](https://arxiv.org/html/2606.11337#bib.bib146),[68](https://arxiv.org/html/2606.11337#bib.bib147)\], and literature review generation\[[11](https://arxiv.org/html/2606.11337#bib.bib88),[22](https://arxiv.org/html/2606.11337#bib.bib90),[31](https://arxiv.org/html/2606.11337#bib.bib91),[77](https://arxiv.org/html/2606.11337#bib.bib89)\]\. Concurrently, small\-scale studies comparing AI\-assisted and expert\-written scientific summaries suggest that LLMs may help scale evidence communication and systematic review workflows\[[20](https://arxiv.org/html/2606.11337#bib.bib74),[32](https://arxiv.org/html/2606.11337#bib.bib73)\]\. Other works examine whether LLMs can support scientific discovery and research ideation\[[45](https://arxiv.org/html/2606.11337#bib.bib67),[102](https://arxiv.org/html/2606.11337#bib.bib68),[103](https://arxiv.org/html/2606.11337#bib.bib66)\]\. However, existing benchmarks largely evaluate intermediate artifacts \(e\.g\., risk of bias, citation quality\) rather than the final scientific conclusion\. They also fail to capture the realistic long\-horizon task of scientific synthesis with current AI agents, which iteratively retrieve sources from the open web, reason across them, and integrate heterogeneous evidence to produce long\-form, multi\-source scientific conclusions\.

Long\-Horizon Synthesis Benchmarks for AI Agents\.Recent work has introduced increasingly capable deep research agents, includingOpenScholar\[[14](https://arxiv.org/html/2606.11337#bib.bib24)\],DR Tulu\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\],OpenResearcher\[[132](https://arxiv.org/html/2606.11337#bib.bib107)\], andWebThinker\[[65](https://arxiv.org/html/2606.11337#bib.bib59)\]\. In parallel, prior benchmarks have studied long\-form QA and factuality\[[35](https://arxiv.org/html/2606.11337#bib.bib78),[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38),[105](https://arxiv.org/html/2606.11337#bib.bib76)\]and open\-domain QA\[[3](https://arxiv.org/html/2606.11337#bib.bib77),[123](https://arxiv.org/html/2606.11337#bib.bib140)\], but are largely non\-agentic or fail to reflect realistic long\-horizon settings where complex open\-ended questions require open\-web tool use and synthesis across multiple sources\. More recent agentic benchmarks\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[29](https://arxiv.org/html/2606.11337#bib.bib19),[98](https://arxiv.org/html/2606.11337#bib.bib56),[117](https://arxiv.org/html/2606.11337#bib.bib57),[119](https://arxiv.org/html/2606.11337#bib.bib126),[120](https://arxiv.org/html/2606.11337#bib.bib127),[125](https://arxiv.org/html/2606.11337#bib.bib58)\]begin to address this gap\. However, as shown in Table[3](https://arxiv.org/html/2606.11337#S7.T3), many benchmarks remain small\-scale, static, and poorly suited to evaluating robust synthesis from noisy open\-web evidence\. They often do not test whether agents can filter irrelevant or unreliable sources and produce high\-quality long\-form conclusions from heterogeneous evidence\. Existing benchmarks also rarely control for benchmark leakage, where web\-enabled agents retrieve ground\-truth artifacts directly from the open web\[[8](https://arxiv.org/html/2606.11337#bib.bib28)\]\.

## 8Conclusion

We introduceSciConBench, a large\-scalelivebenchmark for the long\-horizon task of open\-domain scientific conclusion synthesis;SciConHarness, aclean\-roomevaluation harness with controlled web tools to mitigate benchmark leakage; and an expert\-validated factual evaluation pipeline based on atomic facts, factual precision, and recall\. Through our benchmark evaluation and audits of deployed consumer\-facing systems, we show that current frontier models and AI agents still struggle to synthesize accurate and comprehensive scientific conclusions\. We hope this work guides the development of AI agents that can more reliably synthesize scientific evidence from the open web and support trustworthy scientific and medical decision\-making in high\-stakes contexts\.

## Acknowledgments and Disclosure of Funding

This work was supported in part by the National Science Foundation grants CNS\-1956435 and CNS\-2344925, and by the Alfred P\. Sloan Research Fellowship for A\. Korolova\. We thank Francesco Salvi, Alejandro Cuevas, Max Springer, Chung Peng Lee, Elijah Fullerton, Jane Castleman, Blossom Metevier, Jason Greenfield, and members of the Center for Information Technology Policy at Princeton University for their insightful feedback and discussions\.

## References

- \[1\]ACH Engineering\(n\.d\.\)What is a biotech cleanroom?\.Note:[https://www\.achengineering\.com/what\-is\-a\-biotech\-cleanroom/](https://www.achengineering.com/what-is-a-biotech-cleanroom/)Accessed: 2026\-04\-01Cited by:[§3](https://arxiv.org/html/2606.11337#S3.p3.1)\.
- \[2\]A\. Ajith, M\. Xia, A\. Chevalier, T\. Goyal, D\. Chen, and T\. Gao\(2024\-11\)LitSearch: a retrieval benchmark for scientific literature search\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 15068–15083\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.840/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.840)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1)\.
- \[3\]S\. Amouyal, T\. Wolfson, O\. Rubin, O\. Yoran, J\. Herzig, and J\. Berant\(2023\-12\)QAMPARI: a benchmark for open\-domain questions with many answers\.InProceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics \(GEM\),S\. Gehrmann, A\. Wang, J\. Sedoc, E\. Clark, K\. Dhole, K\. R\. Chandu, E\. Santus, and H\. Sedghamiz \(Eds\.\),Singapore,pp\. 97–110\.External Links:[Link](https://aclanthology.org/2023.gem-1.9/)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[4\]Annenberg Public Policy Center\(2024\)Annenberg science and public health knowledge survey \(asaph\): results\.Note:[https://www\.annenbergpublicpolicycenter\.org/](https://www.annenbergpublicpolicycenter.org/)Survey results and reports on public health attitudes and knowledgeCited by:[§1](https://arxiv.org/html/2606.11337#S1.p5.5),[§6\.1](https://arxiv.org/html/2606.11337#S6.SS1.p1.1)\.
- \[5\]Anthropic\(2026\-January 11\)Advancing claude in healthcare and the life sciences\.Note:[https://www\.anthropic\.com/news/healthcare\-life\-sciences](https://www.anthropic.com/news/healthcare-life-sciences)Accessed March 3, 2026Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p1.1)\.
- \[6\]Anthropic\(2026\)Claude research\.Note:[https://claude\.com/blog/research](https://claude.com/blog/research)Accessed: 2026\-05\-05Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p1.1)\.
- \[7\]Anthropic\(2026\)Create a message — claude api reference\.Note:[https://platform\.claude\.com/docs/en/api/messages/create](https://platform.claude.com/docs/en/api/messages/create)Accessed: 2026\-04\-14Cited by:[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1)\.
- \[8\]Anthropic\(2026\-03\)Eval awareness in claude opus 4\.6’s browsecomp performance\.Note:[https://www\.anthropic\.com/engineering/eval\-awareness\-browsecomp](https://www.anthropic.com/engineering/eval-awareness-browsecomp)Accessed: 2026\-04\-01Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[9\]Anthropic\(2026\)Prompt engineering overview\.Note:[https://platform\.claude\.com/docs/en/build\-with\-claude/prompt\-engineering/overview](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview)Accessed: 2026\-04\-14Cited by:[2nd item](https://arxiv.org/html/2606.11337#A5.I3.i2.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1)\.
- \[10\]Anthropic\(2026\)System prompts — claude api docs \(release notes\)\.Note:[https://platform\.claude\.com/docs/en/release\-notes/system\-prompts](https://platform.claude.com/docs/en/release-notes/system-prompts)Accessed: 2026\-04\-23Cited by:[§5](https://arxiv.org/html/2606.11337#S5.p1.1)\.
- \[11\]S\. A\. Antu, H\. Chen, and C\. K\. Richards\(2023\)Using llm \(large language model\) to improve efficiency in literature review for undergraduate research\.\.Llm@ Aied,pp\. 8–16\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[12\]H\. Armitage\(2025\-02\)Study suggests physician’s medical decisions benefit from chatbot\.Note:[https://med\.stanford\.edu/news/all\-news/2025/02/physician\-decision\-chatbot\.html](https://med.stanford.edu/news/all-news/2025/02/physician-decision-chatbot.html)Stanford Medicine NewsCited by:[§2](https://arxiv.org/html/2606.11337#S2.p5.1)\.
- \[13\]R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel,et al\.\(2025\)Healthbench: evaluating large language models towards improved human health\.arXiv preprint arXiv:2505\.08775\.Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.7.6.1)\.
- \[14\]A\. Asai, J\. He, R\. Shao, W\. Shi, A\. Singh, J\. C\. Chang, K\. Lo, L\. Soldaini, S\. Feldman, M\. D’Arcy, D\. Wadden, M\. Latzke, M\. Tian, P\. Ji, S\. Liu, H\. Tong, B\. Wu, Y\. Xiong, L\. S\. Zettlemoyer, G\. Neubig, D\. Weld, D\. Downey, W\. Yih, P\. W\. Koh, and H\. Hajishirzi\(2024\)OpenScholar: synthesizing scientific literature with retrieval\-augmented lms\.ArXivabs/2411\.14199\.External Links:[Link](https://api.semanticscholar.org/CorpusID:274166189)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.9.8.1),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[15\]A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi\(2023\)Self\-rag: learning to retrieve, generate, and critique through self\-reflection\.InThe Twelfth International Conference on Learning Representations,Cited by:[§I\.3](https://arxiv.org/html/2606.11337#A9.SS3.p1.1),[§I\.4](https://arxiv.org/html/2606.11337#A9.SS4.p1.1)\.
- \[16\]J\. Bakker and J\. Kamps\(2024\-11\)Cochrane\-auto: an aligned dataset for the simplification of biomedical abstracts\.InProceedings of the Third Workshop on Text Simplification, Accessibility and Readability \(TSAR 2024\),M\. Shardlow, H\. Saggion, F\. Alva\-Manchego, M\. Zampieri, K\. North, S\. Štajner, and R\. Stodden \(Eds\.\),Miami, Florida, USA,pp\. 41–51\.External Links:[Link](https://aclanthology.org/2024.tsar-1.5/),[Document](https://dx.doi.org/10.18653/v1/2024.tsar-1.5)Cited by:[§A\.2](https://arxiv.org/html/2606.11337#A1.SS2.p3.1),[2nd item](https://arxiv.org/html/2606.11337#A5.I1.i2.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[17\]M\. Ballon, A\. Algaba, and V\. Ginis\(2025\)The relationship between reasoning and performance in large language models – o3 \(mini\) thinks harder, not longer\.External Links:2502\.15631,[Link](https://arxiv.org/abs/2502.15631)Cited by:[§E\.2\.3](https://arxiv.org/html/2606.11337#A5.SS2.SSS3.p4.1)\.
- \[18\]S\. Bird and E\. Loper\(2004\-07\)NLTK: the natural language toolkit\.InProceedings of the ACL Interactive Poster and Demonstration Sessions,Barcelona, Spain,pp\. 214–217\.External Links:[Link](https://aclanthology.org/P04-3031/)Cited by:[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p2.2)\.
- \[19\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[4th item](https://arxiv.org/html/2606.11337#A5.I3.i4.p1.1)\.
- \[20\]A\. Cadiente, C\. Implicito, A\. Udaiyar, A\. Ho, C\. Wan, J\. Chen, C\. Palmer, Q\. Cao, M\. Raver, K\. Lembrikova,et al\.\(2024\)Evaluating incontinence abstracts: artificial intelligence\-generated versus cochrane review\.Urogynecology,pp\. 10–1097\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[21\]N\. Calderon, R\. Reichart, and R\. Dror\(2025\-07\)The alternative annotator test for LLM\-as\-a\-judge: how to statistically justify replacing human annotators with LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 16051–16081\.External Links:[Link](https://aclanthology.org/2025.acl-long.782/),ISBN 979\-8\-89176\-251\-0Cited by:[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p5.1)\.
- \[22\]C\. Cao, R\. Arora, P\. Cento, A\. Budak, K\. Manta, E\. Farahani, M\. Cecere, A\. Selemon, J\. Sang, L\. X\. Gong,et al\.\(2025\)Automation of systematic reviews with large language models\.medRxiv,pp\. 2025–06\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[23\]K\. M\. Caramancion\(2024\)Large language models vs\. search engines: evaluating user preferences across varied information retrieval scenarios\.arXiv preprint arXiv:2401\.05761\.Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p5.1)\.
- \[24\]A\. Cheng, A\. Jacovi, A\. Globerson, B\. Golan, C\. Kwong, C\. Alberti, C\. Tao, E\. Ben\-David, G\. S\. Tomar, L\. Haas,et al\.\(2025\)The facts leaderboard: a comprehensive benchmark for large language model factuality\.arXiv preprint arXiv:2512\.10791\.Cited by:[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p2.2),[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p3.2),[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p4.1),[Table S9](https://arxiv.org/html/2606.11337#A5.T9),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p2.6),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p2.2),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p3.4),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[25\]B\. Costa\-Gomes, P\. Tolmachev, E\. Taysom, V\. Sounderajah, H\. Richardson, P\. Schoenegger, X\. Liu, M\. M\. Nour, S\. Spielman, S\. F\. Way,et al\.\(2026\)Public use of a generalist llm chatbot for health queries\.Nature Health,pp\. 1–8\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[26\]M\. Cumpston and E\. Flemyng\(2024\)Chapter iv: updating a review\.InCochrane Handbook for Systematic Reviews of Interventions version 6\.5,J\. P\. T\. Higgins, J\. Thomas, J\. Chandler, M\. Cumpston, T\. Li, M\. J\. Page, and et al\. \(Eds\.\),Note:Last updated August 2023\. Available from[https://www\.cochrane\.org/authors/handbooks\-and\-manuals/handbook/current/chapter\-iv](https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-iv)External Links:[Link](https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-iv)Cited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p3.1),[§2](https://arxiv.org/html/2606.11337#S2.p2.1)\.
- \[27\]M\. Dahl, V\. Magesh, M\. Suzgun, and D\. E\. Ho\(2024\-06\)Large legal fictions: profiling legal hallucinations in large language models\.Journal of Legal Analysis16\(1\),pp\. 64–93\.External Links:ISSN 2161\-7201,[Document](https://dx.doi.org/10.1093/jla/laae003),[Link](https://doi.org/10.1093/jla/laae003),https://academic\.oup\.com/jla/article\-pdf/16/1/64/58336922/laae003\.pdfCited by:[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p2.2),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p2.2)\.
- \[28\]P\. P\. S\. Dammu, H\. Jung, A\. Singh, M\. Choudhury, and T\. Mitra\(2024\-11\)“They are uncultured”: unveiling covert harms and social threats in LLM generated conversations\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 20339–20369\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1134/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1134)Cited by:[§B\.1](https://arxiv.org/html/2606.11337#A2.SS1.p1.1),[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1),[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p2.2),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p2.2)\.
- \[29\]P\. P\. S\. Dammu, A\. Palkhiwala, T\. Roosta, and C\. Shah\(2026\)IAgentBench: benchmarking sensemaking capabilities of information\-seeking agents on high\-traffic topics\.arXiv preprint arXiv:2603\.04656\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[30\]P\. Dasigi, K\. Lo, I\. Beltagy, A\. Cohan, N\. A\. Smith, and M\. Gardner\(2021\-06\)A dataset of information\-seeking questions and answers anchored in research papers\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 4599–4610\.External Links:[Link](https://aclanthology.org/2021.naacl-main.365/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.365)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[31\]F\. M\. Delgado\-Chaves, M\. J\. Jennings, A\. Atalaia, J\. Wolff, R\. Horvath, Z\. M\. Mamdouh, J\. Baumbach, and L\. Baumbach\(2025\)Transforming literature screening: the emerging role of large language models in systematic reviews\.Proceedings of the National Academy of Sciences122\(2\),pp\. e2411962122\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2411962122),[Link](https://www.pnas.org/doi/abs/10.1073/pnas.2411962122),https://www\.pnas\.org/doi/pdf/10\.1073/pnas\.2411962122Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[32\]D\. Devane, J\. Pope, P\. Byrne, E\. Forde, S\. Woloshin, E\. Culloty, D\. Dahly, I\. H\. Elgersma, H\. Munthe\-Kaas, C\. Judge, M\. O’Donnell, F\. Krewer, S\. Galvin, N\. Burke, T\. Tierney, K\. Saif\-Ur\-Rahman, T\. Conway, and J\. Thomas\(2025\)Comparison of ai\-assisted and human\-generated plain language summaries for cochrane reviews: protocol for a randomised trial \(hiet\-1\) \[registered report \- stage i\]\.Journal of Clinical Epidemiology185,pp\. 111894\.External Links:ISSN 0895\-4356,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.jclinepi.2025.111894),[Link](https://www.sciencedirect.com/science/article/pii/S0895435625002276)Cited by:[§A\.2](https://arxiv.org/html/2606.11337#A1.SS2.p3.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[33\]A\. Devaraj, I\. Marshall, B\. Wallace, and J\. J\. Li\(2021\-06\)Paragraph\-level simplification of medical texts\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 4972–4984\.External Links:[Link](https://aclanthology.org/2021.naacl-main.395/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.395)Cited by:[2nd item](https://arxiv.org/html/2606.11337#A5.I1.i2.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[34\]M\. Du, B\. Xu, C\. Zhu, X\. Wang, and Z\. Mao\(2025\)Deepresearch bench: a comprehensive benchmark for deep research agents\.arXiv preprint arXiv:2506\.11763\.Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.12.11.1)\.
- \[35\]A\. Fan, Y\. Jernite, E\. Perez, D\. Grangier, J\. Weston, and M\. Auli\(2019\-07\)ELI5: long form question answering\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 3558–3567\.External Links:[Link](https://aclanthology.org/P19-1346/),[Document](https://dx.doi.org/10.18653/v1/P19-1346)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[36\]S\. D\. French, S\. McDonald, J\. E\. McKenzie, and S\. E\. Green\(2005\)Investing in updating: how do conclusions change when cochrane systematic reviews are updated?\.BMC Medical Research Methodology5\(1\),pp\. 33\.Cited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p3.1),[§2](https://arxiv.org/html/2606.11337#S2.p2.1)\.
- \[37\]M\. Funkquist, I\. Kuznetsov, Y\. Hou, and I\. Gurevych\(2023\-12\)CiteBench: a benchmark for scientific citation text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 7337–7353\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.455/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.455)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[38\]J\. Gao, W\. Fu, M\. Xie, S\. Xu, C\. He, Z\. Mei, B\. Zhu, and Y\. Wu\(2025\)Beyond ten turns: unlocking long\-horizon agentic search with large\-scale asynchronous rl\.arXiv preprint arXiv:2508\.07976\.Cited by:[§3](https://arxiv.org/html/2606.11337#S3.p2.3)\.
- \[39\]T\. Gao, H\. Yen, J\. Yu, and D\. Chen\(2023\-12\)Enabling large language models to generate text with citations\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 6465–6488\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.398/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.398)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[40\]E\. Goh, R\. J\. Gallo, E\. Strong, Y\. Weng, H\. Kerman, J\. A\. Freed, J\. A\. Cool, Z\. Kanjee, K\. P\. Lane, A\. S\. Parsons,et al\.\(2025\)GPT\-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial\.Nature Medicine31\(4\),pp\. 1233–1238\.Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p5.1)\.
- \[41\]Google Cloud\(2025\)Gemini 3 flash\.Note:[https://docs\.cloud\.google\.com/vertex\-ai/generative\-ai/docs/models/gemini/3\-flash](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-flash)Accessed: 2026\-04\-14Cited by:[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1)\.
- \[42\]Google Cloud\(2026\)Gemini 3 pro — generative ai on vertex ai\.Note:[https://docs\.cloud\.google\.com/vertex\-ai/generative\-ai/docs/models/gemini/3\-pro](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-pro)Accessed: 2026\-04\-23Cited by:[§5](https://arxiv.org/html/2606.11337#S5.p1.1)\.
- \[43\]Google Cloud\(2026\)What is prompt engineering?\.Note:[https://cloud\.google\.com/discover/what\-is\-prompt\-engineering](https://cloud.google.com/discover/what-is-prompt-engineering)Accessed: 2026\-04\-14Cited by:[2nd item](https://arxiv.org/html/2606.11337#A5.I3.i2.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1)\.
- \[44\]Google\(2026\)Gemini deep research\.Note:[https://gemini\.google/overview/deep\-research/](https://gemini.google/overview/deep-research/)Accessed: 2026\-05\-05Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p1.1)\.
- \[45\]T\. Gupta and D\. Pruthi\(2025\)All that glitters is not novel: plagiarism in ai generated research\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 25721–25738\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[46\]A\. Hevia, S\. Chintalapati, V\. K\. W\. Lai, N\. T\. Tam, W\. Wong, T\. P\. Klassen, and L\. L\. Wang\(2025\-11\)ROBOTO2: an interactive system and dataset for LLM\-assisted clinical trial risk of bias assessment\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,I\. Habernal, P\. Schulam, and J\. Tiedemann \(Eds\.\),Suzhou, China,pp\. 12–25\.External Links:[Link](https://aclanthology.org/2025.emnlp-demos.2/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-demos.2),ISBN 979\-8\-89176\-334\-0Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p2.1)\.
- \[47\]A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi\(2019\)The curious case of neural text degeneration\.arXiv preprint arXiv:1904\.09751\.Cited by:[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1),[§E\.2\.3](https://arxiv.org/html/2606.11337#A5.SS2.SSS3.p5.1)\.
- \[48\]X\. Hou, Y\. Zhao, S\. Wang, and H\. Wang\(2025\)Model context protocol \(mcp\): landscape, security threats, and future research directions\.ACM Transactions on Software Engineering and Methodology\.Cited by:[§3](https://arxiv.org/html/2606.11337#S3.p1.1)\.
- \[49\]J\. Huang, H\. Lai, W\. Zhao, D\. Xia, C\. Bai, M\. Sun, J\. Liu, J\. Liu, B\. Pan, J\. Tian,et al\.\(2025\)Large language model–assisted risk\-of\-bias assessment in randomized controlled trials using the revised risk\-of\-bias tool: evaluation study\.Journal of Medical Internet Research27,pp\. e70450\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[50\]J\. D\. Hwang, V\. Kishore, A\. Singh, D\. Haddad, A\. Naik, M\. Hamada, J\. Bragg, M\. D’Arcy, D\. S\. Weld, L\. L\. Wang, D\. Downey, and S\. Feldman\(2026\)Deep research, shallow evaluation: a case study in meta\-evaluation for long\-form qa benchmarks\.External Links:2603\.06942,[Link](https://arxiv.org/abs/2603.06942)Cited by:[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p3.2),[§E\.2\.4](https://arxiv.org/html/2606.11337#A5.SS2.SSS4.p5.1),[footnote 3](https://arxiv.org/html/2606.11337#footnote3)\.
- \[51\]D\. Jiang, Y\. Lu, Z\. Li, Z\. Lyu, P\. Nie, H\. Wang, A\. Su, H\. Chen, K\. Zou, C\. Du,et al\.\(2025\)Verltool: towards holistic agentic reinforcement learning with tool use\.arXiv preprint arXiv:2509\.01055\.Cited by:[Appendix C](https://arxiv.org/html/2606.11337#A3.p3.1)\.
- \[52\]D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. Szolovits\(2021\)What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\),pp\. 6421\.Cited by:[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.4.3.1)\.
- \[53\]Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. Lu\(2019\-11\)PubMedQA: a dataset for biomedical research question answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2567–2577\.External Links:[Link](https://aclanthology.org/D19-1259/),[Document](https://dx.doi.org/10.18653/v1/D19-1259)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.2.1.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[54\]S\. Joseph, L\. Chen, J\. Trienes, H\. Göke, M\. Coers, W\. Xu, B\. Wallace, and J\. J\. Li\(2024\-08\)FactPICO: factuality evaluation for plain language summarization of medical evidence\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 8437–8464\.External Links:[Link](https://aclanthology.org/2024.acl-long.459/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.459)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[55\]H\. Jung, P\. Juneja, and T\. Mitra\(2025\-Jun\.\)Algorithmic behaviors across regions: a geolocation audit of youtube search for covid\-19 misinformation between the united states and south africa\.Proceedings of the International AAAI Conference on Web and Social Media19\(1\),pp\. 935–964\.External Links:[Link](https://ojs.aaai.org/index.php/ICWSM/article/view/35854),[Document](https://dx.doi.org/10.1609/icwsm.v19i1.35854)Cited by:[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1),[5th item](https://arxiv.org/html/2606.11337#A5.I3.i5.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1)\.
- \[56\]H\. Jung, S\. Mittal, A\. Aatreya, N\. Kaur, M\. De Choudhury, and T\. Mitra\(2025\-11\)MythTriage: scalable detection of opioid use disorder myths on a video\-sharing platform\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 2948–2982\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.146/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.146),ISBN 979\-8\-89176\-332\-6Cited by:[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p2.2),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p2.2)\.
- \[57\]N\. Kaur, H\. Ayad, H\. Jung, S\. Mittal, M\. D\. Choudhury, and T\. Mitra\(2025\)Who’s asking? simulating role\-based questions for conversational ai evaluation\.External Links:2510\.16829,[Link](https://arxiv.org/abs/2510.16829)Cited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p10.1),[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p2.1),[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p5.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p2.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p7.2),[§2](https://arxiv.org/html/2606.11337#S2.p6.1),[§4\.1](https://arxiv.org/html/2606.11337#S4.SS1.p3.2)\.
- \[58\]N\. Kaur, M\. Choudhury, and D\. Pruthi\(2024\-08\)Evaluating large language models for health\-related queries with presuppositions\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14308–14331\.External Links:[Link](https://aclanthology.org/2024.findings-acl.850/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.850)Cited by:[§I\.3](https://arxiv.org/html/2606.11337#A9.SS3.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[59\]T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov\(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Link](https://aclanthology.org/Q19-1026/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[60\]J\. R\. Landis and G\. G\. Koch\(1977\)The measurement of observer agreement for categorical data\.Biometrics33\(1\),pp\. 159–174\.External Links:ISSN 0006341X, 15410420,[Link](http://www.jstor.org/stable/2529310)Cited by:[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p3.4)\.
- \[61\]Y\. Lee, K\. Lee, S\. Park, D\. Hwang, J\. Kim, H\. Lee, and M\. Lee\(2023\)QASA: advanced question answering on scientific articles\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[62\]M\. Li, Y\. Zeng, Z\. Cheng, C\. Ma, and K\. Jia\(2025\)Reportbench: evaluating deep research agents via academic survey tasks\.arXiv preprint arXiv:2508\.15804\.Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.11.10.1)\.
- \[63\]M\. Li, Y\. Zeng, Z\. Cheng, C\. Ma, and K\. Jia\(2025\)ReportBench: evaluating deep research agents via academic survey tasks\.External Links:2508\.15804,[Link](https://arxiv.org/abs/2508.15804)Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p3.1)\.
- \[64\]S\. S\. Li, V\. Balachandran, S\. Feng, J\. S\. Ilgen, E\. Pierson, P\. W\. Koh, and Y\. Tsvetkov\(2024\)Mediq: question\-asking llms and a benchmark for reliable interactive clinical reasoning\.Advances in Neural Information Processing Systems37,pp\. 28858–28888\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[65\]X\. Li, J\. Jin, G\. Dong, H\. Qian, Y\. Zhu, Y\. Wu, J\. Wen, and Z\. Dou\(2025\)WebThinker: empowering large reasoning models with deep research capability\.CoRRabs/2504\.21776\.External Links:[Link](https://doi.org/10.48550/arXiv.2504.21776),[Document](https://dx.doi.org/10.48550/ARXIV.2504.21776),2504\.21776Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[66\]Z\. Li, X\. Guan, B\. Zhang, S\. Huang, H\. Zhou, S\. Lai, M\. Yan, Y\. Jiang, P\. Xie, F\. Huang,et al\.\(2025\)Webweaver: structuring web\-scale evidence with dynamic outlines for open\-ended deep research\.arXiv preprint arXiv:2509\.13312\.Cited by:[Appendix C](https://arxiv.org/html/2606.11337#A3.p4.1),[§3](https://arxiv.org/html/2606.11337#S3.p2.3)\.
- \[67\]J\. Liu, Y\. Li, C\. Zhang, J\. Li, A\. Chen, K\. Ji, W\. Cheng, Z\. Wu, C\. Du, Q\. Xu,et al\.\(2025\)Webexplorer: explore and evolve for training long\-horizon web agents\.arXiv preprint arXiv:2509\.06501\.Cited by:[§3](https://arxiv.org/html/2606.11337#S3.p2.3)\.
- \[68\]N\. Liu, T\. Zhang, and P\. Liang\(2023\-12\)Evaluating verifiability in generative search engines\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 7001–7025\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.467/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.467)Cited by:[§I\.4](https://arxiv.org/html/2606.11337#A9.SS4.p1.1),[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[69\]X\. Liu, L\. Zhang, S\. Munir, Y\. Gu, and L\. Wang\(2025\-11\)VeriFact: enhancing long\-form factuality evaluation with refined fact extraction and reference facts\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 17908–17925\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.905/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.905),ISBN 979\-8\-89176\-332\-6Cited by:[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p1.1),[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p4.1),[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p5.1),[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p6.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p2.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p7.2),[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p3.2),[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p4.1),[Table S9](https://arxiv.org/html/2606.11337#A5.T9),[§4\.1](https://arxiv.org/html/2606.11337#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.11337#S4.SS1.p3.2),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p2.6),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p3.4),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[70\]I\. Marshall, J\. Kuiper, E\. Banner, and B\. C\. Wallace\(2017\-07\)Automating biomedical evidence synthesis: RobotReviewer\.InProceedings of ACL 2017, System Demonstrations,M\. Bansal and H\. Ji \(Eds\.\),Vancouver, Canada,pp\. 7–12\.External Links:[Link](https://aclanthology.org/P17-4002/)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[71\]G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom\(2023\)Gaia: a benchmark for general ai assistants\.arXiv preprint arXiv:2311\.12983\.Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§2](https://arxiv.org/html/2606.11337#S2.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.5.4.1)\.
- \[72\]Mike Clarke\(2025\)Guide to the contents of a cochrane methodology protocol and review\.The Cochrane Collaboration\.Note:Accessed: 2026\-02\-19External Links:[Link](https://www.cochrane.org/sites/default/files/uploads/PDFs/guide_to_the_contents_of_a_cochrane_methodology_protocol_and_review.pdf)Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p2.1)\.
- \[73\]S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi\(2023\-12\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12076–12100\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.741/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.741)Cited by:[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p1.1),[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p2.2),[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p3.2),[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p4.1),[§E\.2\.4](https://arxiv.org/html/2606.11337#A5.SS2.SSS4.p4.1),[Table S9](https://arxiv.org/html/2606.11337#A5.T9),[§I\.3](https://arxiv.org/html/2606.11337#A9.SS3.p1.1),[§4\.1](https://arxiv.org/html/2606.11337#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p2.6),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p3.4),[§7](https://arxiv.org/html/2606.11337#S7.p2.1),[footnote 1](https://arxiv.org/html/2606.11337#footnote1)\.
- \[74\]R\. Mir, B\. Felbo, N\. Obradovich, and I\. Rahwan\(2019\-06\)Evaluating style transfer for text\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 495–504\.External Links:[Link](https://aclanthology.org/N19-1049/),[Document](https://dx.doi.org/10.18653/v1/N19-1049)Cited by:[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p2.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p2.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p7.2),[§2](https://arxiv.org/html/2606.11337#S2.p6.1),[§4\.1](https://arxiv.org/html/2606.11337#S4.SS1.p3.2)\.
- \[75\]S\. Mishra and P\. Chatterjee\(2023\)Exploring chatgpt for toxicity detection in github\.arXiv preprint arXiv:2312\.13105\.Cited by:[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1)\.
- \[76\]S\. Mittal, H\. Jung, M\. ElSherief, T\. Mitra, and M\. De Choudhury\(2025\-Jun\.\)Online myths on opioid use disorder: a comparison of reddit and large language model\.Proceedings of the International AAAI Conference on Web and Social Media19\(1\),pp\. 1224–1245\.External Links:[Link](https://ojs.aaai.org/index.php/ICWSM/article/view/35870),[Document](https://dx.doi.org/10.1609/icwsm.v19i1.35870)Cited by:[5th item](https://arxiv.org/html/2606.11337#A5.I3.i5.p1.1),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p4.1)\.
- \[77\]M\. Mostafapour, J\. H\. Fortier, K\. Pacheco, H\. Murray, and G\. Garber\(2024\)Evaluating literature reviews conducted by humans versus chatgpt: comparative study\.Jmir ai3,pp\. e56537\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[78\]A\. Nenkova and R\. Passonneau\(2004\-May2 \-May7\)Evaluating content selection in summarization: the pyramid method\.InProceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT\-NAACL 2004,Boston, Massachusetts, USA,pp\. 145–152\.External Links:[Link](https://aclanthology.org/N04-1019/)Cited by:[footnote 1](https://arxiv.org/html/2606.11337#footnote1)\.
- \[79\]T\. Ohyama\(2021\)Statistical inference of gwet’s ac1 coefficient for multiple raters and binary outcomes\.Communications in Statistics\-Theory and Methods50\(15\),pp\. 3564–3572\.Cited by:[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p4.1),[§2](https://arxiv.org/html/2606.11337#S2.p6.1)\.
- \[80\]OpenAI\(2026\-01\)AI as a Healthcare Ally: How Americans Are Navigating the System with ChatGPT\.Technical ReportOpenAI\.Note:Accessed 2026External Links:[Link](https://cdn.openai.com/pdf/2cb29276-68cd-4ec6-a5f4-c01c5e7a36e9/OpenAI-AI-as-a-Healthcare-Ally-Jan-2026.pdf)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p1.1),[§1](https://arxiv.org/html/2606.11337#S1.p2.1)\.
- \[81\]OpenAI\(2026\)Deep research\.Note:[https://developers\.openai\.com/api/docs/guides/deep\-research](https://developers.openai.com/api/docs/guides/deep-research)OpenAI API documentation\. Accessed: 2026\-04\-03Cited by:[§F\.3\.2](https://arxiv.org/html/2606.11337#A6.SS3.SSS2.p1.1),[§1](https://arxiv.org/html/2606.11337#S1.p1.1)\.
- \[82\]OpenAI\(2026\)GPT\-5\.1 model — openai api documentation\.Note:[https://developers\.openai\.com/api/docs/models/gpt\-5\.1](https://developers.openai.com/api/docs/models/gpt-5.1)Accessed: 2026\-04\-23Cited by:[§5](https://arxiv.org/html/2606.11337#S5.p1.1)\.
- \[83\]OpenAI\(2026\)Latest: gpt\-5\.4\.Note:[https://developers\.openai\.com/api/docs/guides/latest\-model](https://developers.openai.com/api/docs/guides/latest-model)Accessed: 2026\-04\-14Cited by:[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1)\.
- \[84\]OpenAI\(2026\)O3\-deep\-research model — openai api documentation\.Note:[https://platform\.openai\.com/docs/models/o3\-deep\-research](https://platform.openai.com/docs/models/o3-deep-research)Accessed: 2026\-04\-23Cited by:[§5](https://arxiv.org/html/2606.11337#S5.p1.1)\.
- \[85\]OpenAI\(2026\)O4\-mini\-deep\-research model — openai api documentation\(Website\)External Links:[Link](https://developers.openai.com/api/docs/models/o4-mini-deep-research)Cited by:[§5](https://arxiv.org/html/2606.11337#S5.p1.1)\.
- \[86\]OpenAI\(2026\)Prompt engineering\.Note:[https://platform\.openai\.com/docs/guides/prompt\-engineering](https://platform.openai.com/docs/guides/prompt-engineering)Accessed: 2026\-04\-14Cited by:[1st item](https://arxiv.org/html/2606.11337#A5.I3.i1.p1.1),[2nd item](https://arxiv.org/html/2606.11337#A5.I3.i2.p1.1),[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1)\.
- \[87\]OpenEvidence\(2025\)About OpenEvidence\.Note:[https://www\.openevidence\.com/about](https://www.openevidence.com/about)Accessed: 2026\-03\-03\. OpenEvidence is an AI\-powered medical information platform that aggregates and synthesizes peer\-reviewed clinical evidence to support clinician decision\-making and evidence accessCited by:[§1](https://arxiv.org/html/2606.11337#S1.p2.1)\.
- \[88\]OpenEvidence\(2026\-03\-12\)OpenEvidence achieves historic milestone: 1 million clinical consultations between verified doctors and an artificial intelligence system in a single day\.Note:https://www\.prnewswire\.com/news\-releases/openevidence\-achieves\-historic\-milestone\-1\-million\-clinical\-consultations\-between\-verified\-doctors\-and\-an\-artificial\-intelligence\-system\-in\-a\-single\-day\-302712459\.htmlPress releaseCited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p2.1),[§1](https://arxiv.org/html/2606.11337#S1.p5.5),[§6\.1](https://arxiv.org/html/2606.11337#S6.SS1.p1.1)\.
- \[89\]A\. Oxman, I\. Chalmers, and A\. Dahlgren\(2022\)Key concepts for informed health choices\. 1\.1: assumptions that treatments are safe or effective can be misleading\.Journal of the Royal Society of Medicine115\(9\),pp\. 354–359\.Cited by:[§I\.3](https://arxiv.org/html/2606.11337#A9.SS3.p2.1)\.
- \[90\]A\. Pampari, P\. Raghavan, J\. Liang, and J\. Peng\(2018\-October\-November\)EmrQA: a large corpus for question answering on electronic medical records\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2357–2368\.External Links:[Link](https://aclanthology.org/D18-1258/),[Document](https://dx.doi.org/10.18653/v1/D18-1258)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1)\.
- \[91\]C\. Y\. Park, S\. S\. Li, H\. Jung, S\. Volkova, T\. Mitra, D\. Jurgens, and Y\. Tsvetkov\(2024\-11\)ValueScope: unveiling implicit norms and values via return potential model of social interactions\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 16659–16695\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.972/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.972)Cited by:[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p2.1),[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p5.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p2.1),[§D\.3](https://arxiv.org/html/2606.11337#A4.SS3.p7.2),[3rd item](https://arxiv.org/html/2606.11337#A5.I3.i3.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1),[§2](https://arxiv.org/html/2606.11337#S2.p6.1),[§4\.1](https://arxiv.org/html/2606.11337#S4.SS1.p3.2)\.
- \[92\]L\. Patel, N\. Arabzadeh, H\. Gupta, A\. Sundar, I\. Stoica, M\. Zaharia, and C\. Guestrin\(2025\)Deepscholar\-bench: a live benchmark and automated evaluation for generative research synthesis\.arXiv preprint arXiv:2508\.20033\.Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.13.12.1)\.
- \[93\]Perplexity AI\(2025\)Introducing perplexity deep research\.Note:[https://www\.perplexity\.ai/hub/blog/introducing\-perplexity\-deep\-research](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research)Accessed: 2026\-05\-05Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p1.1)\.
- \[94\]Perplexity AI\(2025\)Sonar deep research — perplexity api documentation\.Note:[https://docs\.perplexity\.ai/docs/sonar/models/sonar\-deep\-research](https://docs.perplexity.ai/docs/sonar/models/sonar-deep-research)Accessed: 2026\-04\-23Cited by:[§5](https://arxiv.org/html/2606.11337#S5.p1.1)\.
- \[95\]Perplexity AI\(2025\)Sonar reasoning pro — perplexity api documentation\.Note:[https://docs\.perplexity\.ai/docs/sonar/models/sonar\-reasoning\-pro](https://docs.perplexity.ai/docs/sonar/models/sonar-reasoning-pro)Accessed: 2026\-04\-23Cited by:[§5](https://arxiv.org/html/2606.11337#S5.p1.1)\.
- \[96\]L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.6.5.1)\.
- \[97\]M\. Phutane, H\. Jung, M\. Kim, T\. Mitra, and A\. Vashistha\(2025\)ABLEIST: intersectional disability bias in llm\-generated hiring scenarios\.External Links:2510\.10998,[Link](https://arxiv.org/abs/2510.10998)Cited by:[§E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2.p2.2),[§E\.2\.3](https://arxiv.org/html/2606.11337#A5.SS2.SSS3.p4.1),[§4\.3](https://arxiv.org/html/2606.11337#S4.SS3.p2.2)\.
- \[98\]C\. Polzak, A\. Lozano, M\. W\. Sun, J\. Burgess, Y\. Zhang, K\. Wu, and S\. Yeung\-Levy\(2025\)Can large language models match the conclusions of systematic reviews?\.arXiv preprint arXiv:2505\.22787\.Cited by:[§A\.2](https://arxiv.org/html/2606.11337#A1.SS2.p3.1),[2nd item](https://arxiv.org/html/2606.11337#A5.I1.i2.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[99\]J\. Ruan, I\. Nair, S\. Cao, A\. Liu, S\. Munir, M\. Pollens\-Dempsey, T\. Chiang, L\. Kates, N\. David, S\. Chen,et al\.\(2025\)Expertlongbench: benchmarking language models on expert\-level long\-form generation tasks with structured checklists\.arXiv preprint arXiv:2506\.01241\.Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.8.7.1)\.
- \[100\]R\. Shao, A\. Asai, S\. Z\. Shen, H\. Ivison, V\. Kishore, J\. Zhuo, X\. Zhao, M\. Park, S\. G\. Finlayson, D\. Sontag, T\. Murray, S\. Min, P\. Dasigi, L\. Soldaini, F\. Brahman, W\. Yih, T\. Wu, L\. Zettlemoyer, Y\. Kim, H\. Hajishirzi, and P\. W\. Koh\(2025\)DR tulu: reinforcement learning with evolving rubrics for deep research\.External Links:2511\.19399,[Link](https://arxiv.org/abs/2511.19399)Cited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p5.1),[Appendix C](https://arxiv.org/html/2606.11337#A3.p3.1),[Appendix C](https://arxiv.org/html/2606.11337#A3.p4.1),[Figure S26](https://arxiv.org/html/2606.11337#A6.F26),[§F\.2](https://arxiv.org/html/2606.11337#A6.SS2.p1.1),[Appendix H](https://arxiv.org/html/2606.11337#A8.p1.2),[§3](https://arxiv.org/html/2606.11337#S3.p2.3),[§5](https://arxiv.org/html/2606.11337#S5.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[101\]K\. G\. Shojania, M\. Sampson, M\. T\. Ansari, J\. Ji, S\. Doucette, and D\. Moher\(2007\)How quickly do systematic reviews go out of date? a survival analysis\.Annals of Internal Medicine147\(4\),pp\. 224–233\.Note:PMID: 17638714External Links:[Document](https://dx.doi.org/10.7326/0003-4819-147-4-200708210-00179),[Link](https://doi.org/10.7326/0003-4819-147-4-200708210-00179),https://doi\.org/10\.7326/0003\-4819\-147\-4\-200708210\-00179Cited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p3.1)\.
- \[102\]C\. Si, T\. Hashimoto, and D\. Yang\(2025\)The ideation\-execution gap: execution outcomes of llm\-generated versus human research ideas\.arXiv preprint arXiv:2506\.20803\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[103\]C\. Si, D\. Yang, and T\. Hashimoto\(2024\)Can llms generate novel research ideas? a large\-scale human study with 100\+ nlp researchers\.arXiv preprint arXiv:2409\.04109\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[104\]R\. Smith\(2013\)The cochrane collaboration at 20\.Vol\.347,British Medical Journal Publishing Group\.Cited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p3.1),[§2](https://arxiv.org/html/2606.11337#S2.p2.1),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p1.1)\.
- \[105\]I\. Stelmakh, Y\. Luan, B\. Dhingra, and M\. Chang\(2022\-12\)ASQA: factoid questions meet long\-form answers\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 8273–8288\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.566/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.566)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[106\]Y\. Sun, X\. Qian, W\. Xu, H\. Zhang, C\. Xiao, L\. Li, D\. Zhao, W\. Huang, T\. Xu, Q\. Bai, and Y\. Rong\(2025\-11\)ReasonMed: a 370K multi\-agent generated dataset for advancing medical reasoning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 26446–26467\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1344/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1344),ISBN 979\-8\-89176\-332\-6Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[107\]S\. Takeshita, T\. Green, I\. Reinig, K\. Eckert, and S\. Ponzetto\(2024\-06\)ACLSum: a new dataset for aspect\-based summarization of scientific publications\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 6660–6675\.External Links:[Link](https://aclanthology.org/2024.naacl-long.371/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.371)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[108\]P\. E\. Taneri\(2025\)Human versus artificial intelligence: comparing cochrane authors’ and chatgpt’s risk of bias assessments\.Cochrane Evidence Synthesis and Methods3\(5\),pp\. e70044\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/cesm.70044),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1002/cesm.70044),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1002/cesm\.70044Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[109\]L\. Tang, Z\. Sun, B\. Idnay, J\. G\. Nestor, A\. Soroush, P\. A\. Elias, Z\. Xu, Y\. Ding, G\. Durrett, J\. F\. Rousseau,et al\.\(2023\)Evaluating large language models on medical evidence summarization\.NPJ digital medicine6\(1\),pp\. 158\.Cited by:[2nd item](https://arxiv.org/html/2606.11337#A5.I1.i2.p1.1),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[110\]The Cochrane Collaboration\(2026\)About the cochrane database of systematic reviews\(Website\)Cochrane\.Note:Accessed: 2026\-02\-16External Links:[Link](https://www.cochranelibrary.com/cdsr/about-cdsr)Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p2.1)\.
- \[111\]The Cochrane Collaboration\(2026\)Cochrane methods gradeing\(Website\)Cochrane\.Note:Accessed: 2026\-02\-16External Links:[Link](https://methods.cochrane.org/gradeing/)Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p2.1)\.
- \[112\]The Hong Kong Polytechnic University Library\(2025\)Formulate research question using pico\.Note:[https://libguides\.lb\.polyu\.edu\.hk/syst\_review/PICO](https://libguides.lb.polyu.edu.hk/syst_review/PICO)Accessed: 2026\-03\-31Cited by:[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p2.1),[§2](https://arxiv.org/html/2606.11337#S2.p5.1)\.
- \[113\]The New York Times\(2024\-05\-31\)Google’s a\.i\. answers about health can be inaccurate or misleading\.The New York Times\.Note:Accessed: 2026\-05\-04External Links:[Link](https://www.nytimes.com/2024/05/31/well/live/google-ai-health-information.html)Cited by:[§A\.1](https://arxiv.org/html/2606.11337#A1.SS1.p2.1),[§1](https://arxiv.org/html/2606.11337#S1.p5.5),[§6\.1](https://arxiv.org/html/2606.11337#S6.SS1.p1.1),[§6\.1](https://arxiv.org/html/2606.11337#S6.SS1.p2.1)\.
- \[114\]J\. Vladika, P\. Schneider, and F\. Matthes\(2024\-08\)MedREQAL: examining medical knowledge recall of large language models via question answering\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14459–14469\.External Links:[Link](https://aclanthology.org/2024.findings-acl.860/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.860)Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[115\]D\. Wadden, S\. Lin, K\. Lo, L\. L\. Wang, M\. van Zuylen, A\. Cohan, and H\. Hajishirzi\(2020\-11\)Fact or fiction: verifying scientific claims\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 7534–7550\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.609/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.609)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.3.2.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[116\]D\. Wadden, K\. Lo, B\. Kuehl, A\. Cohan, I\. Beltagy, L\. L\. Wang, and H\. Hajishirzi\(2022\-12\)SciFact\-open: towards open\-domain scientific claim verification\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 4719–4734\.External Links:[Link](https://aclanthology.org/2022.findings-emnlp.347/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.347)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§4\.2](https://arxiv.org/html/2606.11337#S4.SS2.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[117\]H\. Wan, C\. Yang, J\. Yu, M\. Tu, J\. Lu, D\. Yu, J\. Cao, B\. Gao, J\. Xie, A\. Wang,et al\.\(2026\)Deep research arena: the first exam of llms’ research abilities via seminar\-grounded tasks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33341–33349\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[118\]J\. Wang, W\. Cao, L\. Bao, Y\. Zheng, G\. Pasternak, K\. Wang, X\. Wang, R\. Paturi, and L\. Bergen\(2025\-11\)Measuring risk of bias in biomedical reports: the RoBBR benchmark\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 3220–3248\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.160/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.160),ISBN 979\-8\-89176\-332\-6Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p1.1)\.
- \[119\]J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus\(2024\)Measuring short\-form factuality in large language models\.arXiv preprint arXiv:2411\.04368\.Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[120\]J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. Glaese\(2025\)Browsecomp: a simple yet challenging benchmark for browsing agents\.arXiv preprint arXiv:2504\.12516\.Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[121\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[5th item](https://arxiv.org/html/2606.11337#A5.I3.i5.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1)\.
- \[122\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§B\.1](https://arxiv.org/html/2606.11337#A2.SS1.p1.1)\.
- \[123\]J\. Wei, C\. Yang, X\. Song, Y\. Lu, N\. Hu, J\. Huang, D\. Tran, D\. Peng, R\. Liu, D\. Huang, C\. Du, and Q\. V\. Le\(2024\)Long\-form factuality in large language models\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p1.1),[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p4.1),[§D\.1](https://arxiv.org/html/2606.11337#A4.SS1.p6.1),[§4\.1](https://arxiv.org/html/2606.11337#S4.SS1.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[124\]N\. Wongpakaran, T\. Wongpakaran, D\. Wedding, and K\. L\. Gwet\(2013\)A comparison of cohen’s kappa and gwet’s ac1 when calculating inter\-rater reliability coefficients: a study conducted with personality disorder samples\.BMC medical research methodology13\(1\),pp\. 61\.Cited by:[§B\.2](https://arxiv.org/html/2606.11337#A2.SS2.p4.1),[§2](https://arxiv.org/html/2606.11337#S2.p6.1)\.
- \[125\]J\. Wu, W\. Yin, Y\. Jiang, Z\. Wang, Z\. Xi, R\. Fang, L\. Zhang, Y\. He, D\. Zhou, P\. Xie,et al\.\(2025\)Webwalker: benchmarking llms in web traversal\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10290–10305\.Cited by:[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[126\]R\. Xu, Z\. Wang, R\. Fan, and P\. Liu\(2024\)Benchmarking benchmark leakage in large language models\.arXiv preprint arXiv:2404\.18824\.Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p3.1),[§6](https://arxiv.org/html/2606.11337#S6.p4.6)\.
- \[127\]T\. Xu, P\. Lu, L\. Ye, X\. Hu, and P\. Liu\(2025\)Researcherbench: evaluating deep ai research systems on the frontiers of scientific inquiry\.arXiv preprint arXiv:2507\.16280\.Cited by:[Table 3](https://arxiv.org/html/2606.11337#S7.T3.7.1.10.9.1)\.
- \[128\]J\. Yang, N\. Yonack, K\. Zyskowski, D\. Yarats, J\. Ho, and J\. Ma\(2025\)The Adoption and Usage of AI Agents: Early Evidence from Perplexity\.External Links:2512\.07828,[Link](https://arxiv.org/abs/2512.07828)Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p1.1)\.
- \[129\]Y\. Yang, C\. P\. Lee, S\. Feng, D\. Zhao, B\. Wen, A\. Z\. Liu, Y\. Tsvetkov, and B\. Howe\(2025\)Escaping the spuriverse: can large vision\-language models generalize beyond seen spurious correlations?\.arXiv preprint arXiv:2506\.18322\.Cited by:[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1)\.
- \[130\]P\. Zhang, Q\. Ye, Z\. Peng, K\. Garimella, and G\. Tyson\(2025\)Source coverage and citation bias in llm\-based vs\. traditional search engines\.arXiv preprint arXiv:2512\.09483\.Cited by:[§2](https://arxiv.org/html/2606.11337#S2.p5.1)\.
- \[131\]X\. Zhang, Y\. Xie, J\. Huang, J\. Ma, Z\. Pan, Q\. Liu, Z\. Xiong, T\. Ergen, D\. Shim, H\. Lee, and Q\. Mei\(2025\-04\)MASSW: a new dataset and benchmark tasks for AI\-assisted scientific workflows\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 2373–2394\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.127/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.127),ISBN 979\-8\-89176\-195\-7Cited by:[§1](https://arxiv.org/html/2606.11337#S1.p3.1)\.
- \[132\]M\. Zheng, J\. Pei, L\. Logeswaran, M\. Lee, and D\. Jurgens\(2024\-11\)When “a helpful assistant” is not really helpful: personas in system prompts do not improve performances of large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 15126–15154\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.888/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.888)Cited by:[1st item](https://arxiv.org/html/2606.11337#A5.I3.i1.p1.1),[§E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2.p1.1),[§7](https://arxiv.org/html/2606.11337#S7.p2.1)\.
- \[133\]L\. Zhou, L\. Pacchiardi, F\. Martínez\-Plumed, K\. M\. Collins, Y\. Moros\-Daval, S\. Zhang, Q\. Zhao, Y\. Huang, L\. Sun, J\. E\. Prunty, Z\. Li, P\. Sánchez\-García, K\. J\. Chen, P\. A\. M\. Casares, J\. Zu, J\. Burden, B\. Mehrbakhsh, D\. Stillwell, M\. Cebrian, J\. Wang, P\. Henderson, S\. T\. Wu, P\. C\. Kyllonen, L\. Cheke, X\. Xie, and J\. Hernández\-Orallo\(2026\)General scales unlock ai evaluation with explanatory and predictive power\.Nature\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10303-2),[Link](https://www.nature.com/articles/s41586-026-10303-2)Cited by:[§6](https://arxiv.org/html/2606.11337#S6.p4.6)\.

Appendices

## Appendix ADiscussions

### A\.1Limitations

WhileSciConBenchaddresses current pitfalls of agentic benchmarks, some limitations remain:

Focus on Science and Health\.BecauseSciConBenchis derived from theCDSR, our benchmark primarily reflects scientific and health\-related conclusion synthesis tasks grounded in evidence\-based medicine\. While these clinical and health domains are consequential and increasingly targeted by AI agents\[[113](https://arxiv.org/html/2606.11337#bib.bib112),[88](https://arxiv.org/html/2606.11337#bib.bib111)\], they do not fully capture the diversity of synthesis tasks across other disciplines \(e\.g\., social science, legal\)\. As such, performance onSciConBenchmay not generalize to other long\-form synthesis in different domains\. Nevertheless,CDSRprovides a large\-scale corpus of expert\-written scientific conclusions spanning nearly 30 years and over 9\.11K benchmark samples, enabling substantially broader and more realistic evaluations than many existing synthesis benchmarks\.

Dependence on Cochrane Reviews\.Our benchmark treatsCDSRreviews as the reference standard for scientific conclusions\. While Cochrane reviews are widely regarded as a gold standard for evidence synthesis\[[104](https://arxiv.org/html/2606.11337#bib.bib100)\], they are still products of human judgment and may inherit biases, including emphasis on Western clinical research practices\. While systematic reviews themselves can become outdated as new evidence emerges\[[101](https://arxiv.org/html/2606.11337#bib.bib110)\], French et al\.\[[36](https://arxiv.org/html/2606.11337#bib.bib6)\]found that only a small portion of updated reviews change conclusions, suggesting that stale conclusions are a small fraction ofSciConBenchdataset\. To mitigate this issue,CDSRrecommends periodic literature updates approximately every two years and withdraws reviews deemed excessively outdated\[[26](https://arxiv.org/html/2606.11337#bib.bib7)\]\. Our benchmark continuously updates with new reviews, and our evaluations focus on recent reviews published after model knowledge cutoffs to reduce data contamination from pre\-training and conclusion staleness\.

Residual benchmark leakage may still exist\.AlthoughSciConHarnessintroduces a clean\-room protocol to mitigate retrieval of ground\-truth artifacts \(with 100% filtering rate as in Table[S6](https://arxiv.org/html/2606.11337#A3.T6)\), leakage may still occur indirectly in the noisy open\-web environment\. For example, derivative webpages \(e\.g\., news coverage\) or summaries may paraphrase Cochrane conclusions without explicitly mentioning the original review\. As such, occasional residual leakage may still exist\.

Evolving Agentic Frameworks and Tooling\.SciConHarnessuses the same set of agentic tools as prior works\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\]\. While this enables controlled and reproducible evaluation, AI agents and tool ecosystems are evolving rapidly\. Future systems may employ substantially different retrieval infrastructures, reasoning paradigms, memory mechanisms, or tool orchestration strategies beyond our current setup\. At the same time, our benchmark evaluation results \(e\.g\.,o3\-deep\-researchscoring 0\.508 F1\) are broadly consistent with our audit of consumer\-facing agents using proprietary harnesses \(e\.g\.,OpenEvidencescoring 0\.522 F1\), suggesting that the relatively low performance observed inSciConBenchis unlikely to be solely an artifact of our harness design\. We makeSciConHarnessfully open\-source to support extensibility and enable future practitioners to improve upon the framework with new tools, reasoning processes, and orchestration protocols\.

LLM Error Cascades\.We use LLMs to generateSciConBenchquestions, decompose conclusions into atomic facts, and evaluate factual quality\. While domain experts largely validate the quality of the generated questions \(§[B\.2](https://arxiv.org/html/2606.11337#A2.SS2)\) and atomic facts \(§[D\.3](https://arxiv.org/html/2606.11337#A4.SS3)\), the small error rates may potentially influence our downstream evaluations\. In addition, our factual evaluation pipeline relies on expert\-validated LLM judges \(§[E\.2\.3](https://arxiv.org/html/2606.11337#A5.SS2.SSS3)\), but may still introduce errors that may propagate into downstream evaluations\. To better understand this error, we conduct an error analysis of the LLM judge in §[E\.2\.4](https://arxiv.org/html/2606.11337#A5.SS2.SSS4)\.

Decision Loss\.WhileSciConBenchmeasures factual precision and recall, these metrics do not directly capture downstream decision impact\. In practice, missing, incomplete, or contradictory information becomes most consequential when itchangesscientific, clinical, or policy decisions\. For example, omissions in treatment effectiveness, uncertainty, or harms may lead users toward different conclusions or actions despite only modest changes in factual F1\. The impact of synthesis errors may differ across populations or user contexts\. Future work should investigate how factual errors translate into downstreamdecision loss, particularly in personalized or high\-stakes health contexts\.

Focus on Scientific Conclusion Synthesis\.We focus on scientific conclusion synthesis, a consequential output that shapes decisions in health, science, and policy\. However, our benchmark does not directly evaluate intermediate reasoning processes, such as multi\-hop reasoning, evidence selection, or assessment of evidence quality\. Although many proprietary, frontier models do not expose detailed reasoning traces,SciConHarnesslogs reasoning summaries and tool calls\. We leave the evaluation of intermediate reasoning artifacts to future work\.

Alternative Clean\-Room Evaluations\.Some provider\-hosted deep research agents do not support direct integration withSciConHarness, requiring alternative clean\-room protocols for comparability with other evaluated systems \(§[F\.3](https://arxiv.org/html/2606.11337#A6.SS3)\)\. In particular, Perplexity agents rely on provider\-native search APIs without custom MCP support, for which we use provider\-side search filters as a “best\-effort” clean\-room protocol\. As a result, these evaluations may not be perfectly comparable to directSciConHarnessintegration\. Nevertheless, both Perplexity and OpenAI deep research agents consistently exhibit substantial performance decreases under clean\-room settings \(§[6](https://arxiv.org/html/2606.11337#S6)\), suggesting that our central finding on leakage\-driven performance inflation is robust across implementations\.

Real\-World Query Formulation\.SciConBenchquestions are derived from the Objectives sections of publishedCDSRreviews, reflecting real\-world scientific synthesis tasks\. However, these questions may not fully capture how diverse users—such as patients, caregivers, and the general public\[[57](https://arxiv.org/html/2606.11337#bib.bib31)\]—naturally formulate queries in health contexts\. Real\-world information seeking is often underspecified, conversational, personalized, or shaped by incomplete domain knowledge, whereas our benchmark questions are grounded in well\-defined review objectives and PICO\-style framing\. While the current study focuses on the task of scientific conclusion synthesis rather than noisy or ambiguous real\-world queries, evaluating agent behavior under such settings represents an important direction for future work, as AI agents are increasingly deployed in real\-world health and scientific contexts\.

### A\.2Ethical Considerations & Broader Impact

As AI agents are increasingly used to access and synthesize scientific conclusions,SciConBenchcan guide the development of more reliable, transparent, and evidence\-grounded AI agents for health and science in open\-domain settings\. By evaluating and identifying weaknesses in current frontier agents, our work can help researchers and practitioners better understand the limitations of existing AI agents before deployment in high\-stakes health settings\. Broadly, our benchmark may support the development of safer benchmark evaluation practices for AI agents on the open web and encourage future work on trustworthy scientific synthesis, evidence integration, and uncertainty communication\.

At the same time, our work carries ethical considerations\. As a publicly available and open\-source benchmark,SciConBenchmay itself become a target for benchmark contamination or overfitting, potentially inflating future agent performance through memorization or retrieval of benchmark artifacts rather than genuine synthesis capability\. To mitigate this risk, we designSciConBenchas a continuously updated live benchmark and introduceSciConHarness, an open\-source clean\-room evaluation harness that restricts access to ground\-truth artifacts\. In addition, improvements onSciConBenchmay be misinterpreted as evidence that AI\-generated scientific conclusions are sufficiently reliable for real\-world deployment\. However, our results show that current systems frequently omit necessary facts and sometimes generate contradictory conclusions, raising concerns about overtrust and automation bias in high\-stakes health contexts\. We therefore emphasize thatSciConBenchis intended as an evaluation benchmark rather than a deployment endorsement\.

Our benchmark does not assign new licenses toCDSR\-authored content, which remains copyrighted by the original authors and/or Cochrane and governed by Cochrane/Wiley terms and applicable Creative Commons licensing terms\. We claim no ownership over Cochrane\-authored content\. The benchmark was developed through non\-commercial academic research and constructed from publicly accessibleCDSRcomponents, including Abstract sections \(e\.g\.,Objectives,Authors’ Conclusions\) and Plain Language Summaries\. These components are available without paywall access and have been used in prior research\[[16](https://arxiv.org/html/2606.11337#bib.bib52),[32](https://arxiv.org/html/2606.11337#bib.bib73),[98](https://arxiv.org/html/2606.11337#bib.bib56)\]\.

## Appendix BDetails on Question Generation

Here, we discuss the technical details of the question generation pipeline \(§[B\.1](https://arxiv.org/html/2606.11337#A2.SS1)\) and the validation results of the generated question \(§[B\.2](https://arxiv.org/html/2606.11337#A2.SS2)\)\.

### B\.1Technical Details and Methods

For each systematic review, we generate a single question from theObjectivessection, using theBackgroundsection as additional context\. We employgpt\-5\-chatvia Microsoft Azure AI Foundry888[https://ai\.azure\.com/](https://ai.azure.com/)with a few\-shot prompt \(Figure[S3](https://arxiv.org/html/2606.11337#A2.F3)\)\. The prompt includes three examples of objective\-to\-question conversions and instructs the model to output a JSON object containing the generated question and a brief step\-by\-step justification, which have been to improve performance\[[28](https://arxiv.org/html/2606.11337#bib.bib35),[122](https://arxiv.org/html/2606.11337#bib.bib33)\]\. We set temperature to 0 and a maximum of 1,024 tokens\.

To ensure well\-formed questions, we apply a rules\-based check: 1\) strip whitespace, 2\) enforce a trailing question mark, 3\) verify the presence of interrogative structure \(e\.g\.,what,how,which,is, etc\), and 4\) discard outputs that are empty or shorter than 10 characters\. Invalid outputs are regenerated until they pass validation\. See Figure[S3](https://arxiv.org/html/2606.11337#A2.F3)for an example of a generated question\.

### B\.2Validation of Generated Questions

Overview\.We validate the quality of generated questions through domain\-expert annotations conducted by two medical student annotators\. The task is to assess whether each question faithfully reflects the intent and scope of the underlying systematic review\. These medical students are from established medical schools in the U\.S\. with extensive clinical and scientific research experiences\. Annotators are provided with theObjectives, generated question, and supporting context from theBackgroundandAuthors’ Conclusions, with optional access to the full article\.

Evaluation Dimensions\.We evaluate question quality along three dimensions grounded in prior works\[[57](https://arxiv.org/html/2606.11337#bib.bib31),[74](https://arxiv.org/html/2606.11337#bib.bib15),[91](https://arxiv.org/html/2606.11337#bib.bib83),[112](https://arxiv.org/html/2606.11337#bib.bib48)\]:

- •Faithfulness:Whether the generated question preserve the meaning of theObjectiveswithout distortions\.Answer Choices: Faithful, Unfaithful
- •PICO Completeness:Whether the generated question captures all key PICO components present in theObjectives\.Answer Choices: Complete, Partially Complete, Incomplete
- •Clarity & Answerability:Whether the generated question is clearly phrased and answerable by a systematic review\.Answer Choices: Clear and Answerable, Unclear / Unanswerable

The full annotation guidelines are provided in Figure[S4](https://arxiv.org/html/2606.11337#A2.F4), and the annotation interface used by annotators is shown in Figures[S5](https://arxiv.org/html/2606.11337#A2.F5)and[S6](https://arxiv.org/html/2606.11337#A2.F6)\.

Table S4:Agreement between two expert annotators on evaluating generated questions \(N=10\) across three dimensions: Faithfulness \(Faith\.\), PICO Completeness \(PICO\), and Clarity & Answerability \(Clar\.\)\.Annotation Procedure\.Based on the annotation guidelines, we first conduct a calibration phase in which annotators jointly label 10 generated questions to refine the task, validate the evaluation dimensions, and resolve disagreements\. During this process, we clarify edge cases \(e\.g\., allowing additional background context in the generated questions when it does not alter meaning for Faithfulness\) and update the codebook accordingly\. After refining and validating the annotation task, annotators then independently label an additional 10 generated questions to assess reliability\. To assess their reliability, we compute the inter\-annotator agreement using Gwet’s AC1, which is more robust than Cohen’sκ\\kappaunder a skewed label distribution\[[124](https://arxiv.org/html/2606.11337#bib.bib14),[79](https://arxiv.org/html/2606.11337#bib.bib13)\]\.

Results\.As shown in Table[S4](https://arxiv.org/html/2606.11337#A2.T4), the inter\-annotator agreement is high across all three evaluation dimensions \(Gwet’s AC1: 0\.756 – 1\.00; Percentage Agreement: 0\.80 – 1\.00\), comparable to or exceeding prior work\[[57](https://arxiv.org/html/2606.11337#bib.bib31),[91](https://arxiv.org/html/2606.11337#bib.bib83)\]\. After measuring agreement, the expert annotators resolved disagreements through discussion\. Following high agreement, the two annotators independently labeled 40 generated questions each, yieldingN=100N=100annotated questions total\.

Table S5:Label distribution across evaluation dimensions for the generated questions \(N=100N=100\)\. Abbreviations: Partial = Partially Complete; Clarity = Clarity & Answerability; Clear = Clear and Answerable; Unclear = Unclear / Unanswerable\.Table[S5](https://arxiv.org/html/2606.11337#A2.T5)summarizes the validation results, in which the annotators evaluated the generated questions to be largely faithful \(92%\), complete with key PICO elements \(92%\), and clear and answerable \(96%\) These results indicate that the question generation pipeline reliably produces faithful, complete, and answerable research questions that accurately reflect the intent and scope of the underlying systematic reviews, validating their use in downstream evaluation\.

Question Generation PromptSystem Prompt:You are an expert specialized in converting research objectives into their corresponding research question format\. Instruction:Given the following research objective, please convert it into a single question, using the provided study background as context to inform your answer\.\*\*\*RESEARCH OBJECTIVE STARTS HERE\*\*\* To assess the benefits and risks of medical treatments prior to surgery for uterine fibroids\. \*\*\*RESEARCH OBJECTIVE ENDS HERE\*\*\* \*\*\*STUDY BACKGROUND CONTEXT STARTS HERE\*\*\* Uterine fibroids occur in up to 40% of women over 35 years of age\. Up to 50% of uterine fibroids cause symptoms that warrant treatment: anaemia caused by heavy menstrual bleeding, pelvic pain, dysmenorrhoea, infertility and poor quality of life\. Surgery is the first choice of treatment… \*\*\*STUDY BACKGROUND CONTEXTS ENDS HERE\*\*\* \*\*\*EXAMPLE 1 STARTS HERE\*\*\* Research Objective: …Generated Question: …\*\*\*EXAMPLE 1 ENDS HERE\*\*\* …\*\*\*EXAMPLE 3 STARTS HERE\*\*\* Research Objective: …Generated Question: …\*\*\*EXAMPLE 3 ENDS HERE\*\*\* Now, given what you learned from the examples and using the study background as context, please convert the provided objective into a single research question, justifying your answer and thinking step\-by\-step about your answer\. Use the guidelines below to inform your output question and justification\.\*\*\*GUIDELINES FOR GENERATING RESEARCH QUESTIONS STARTS HERE:\*\*\* \- The generated question should be specific, answerable, and capture the main research focus in the objective\.\- Do not include any extraneous information or context in your answer that were not provided in the study background\- Do not add additional information beyond what is stated in the objective\. Only use the background as context to inform your answer\.\- Avoid copying the objective verbatim and aim to generate a question that is Google\-Proof\. Preserve the objective’s semantics and the objective to maintain the full population, intervention, comparison, and outcome \(PICO\) of the study and objective in the generated question\.\*\*\*GUIDELINES FOR GENERATING RESEARCH QUESTIONS ENDS HERE\.\*\*\* Output should be in JSON format with the following structure:\- “question”: string research question\- “justification”: string justification for your answer regarding the converted question\.

Example Generated QuestionFor women with uterine fibroids undergoing surgery, what are the benefits and risks of using medical treatments before surgery?

Figure S3:Few\-shot prompt used to convertCDSRObjectivesinto equivalent research questions\. TheBackgroundsection is included to provide additional context during generation\. Text inblueshows an exampleObjectiveandBackgroundfrom aCDSRsystematic review, along with the corresponding generated question below\.Annotation Guideline for Evaluating Generated QuestionsTask Overview\.PICOstands forPopulation,Intervention,Comparator, andOutcome—a standard framework for structuring research questions in health evidence\. Population is the patient group or condition; Intervention is the treatment or exposure; Comparator is the alternative \(e\.g\., placebo or another treatment\); Outcome is what is measured or of interest\.Your task is to evaluate the quality of questions generated from theObjectivessection in Cochrane review articles\. According to prior work, both research questions and theObjectivessection in systematic review articles are closely related and should be rooted in the PICO framework\. Since research questions are often framed within PICO, we convertObjectivesinto aQuestionformat\.Inputs\.You will be provided with the following:•Question: Unit of evaluation\. An overarching research question of the review\.•Objectives: Direct source of theQuestion\. Outlines the overarching goal of the systematic review\.•Background and Conclusion: Paragraph\(s\) from the Background section and the Conclusion paragraph to provide context and motivation\.Use these as context when evaluating the Question\.•Cochrane Article Link: Provided for additional context if needed\.Evaluation Criteria\.When evaluating eachQuestion, use the Background and Conclusion as context\.1\.Faithfulness\.Does the generatedQuestionaccurately reflect the meaning of theObjectives?•Faithful: Preserves the meaning of the Objectives\. Minor rephrasing or inclusion of Background context is acceptable as long as it does not deviate from theObjective\.•Unfaithful: Misrepresents the Objectives by introducing new or altered elements or changing the meaning\.Note: Additional background context in the generated question is fine as long as they do not change the meaning of the originalObjective\.2\.PICO Completeness\.Does the generatedQuestioncapture the key Population, Intervention, Comparator, and Outcome \(PICO\) elements present in theObjectives?•Complete: All relevant PICO elements are correctly represented without distortion or hallucination\.•Partially Complete: Some PICO elements are missing or underspecified/distorted\.•Incomplete: PICO elements are missing, distorted, or extraneous elements are introduced that alter meaning\.Note: Evaluate only PICO present in the Objectives; do not penalize PICO elements that are also missing from the Objectives\.3\.Clarity and Answerability\.Is theQuestionclearly phrased and answerable by a systematic review?•Clear and Answerable: Clearly phrased, unambiguous, and can be answered through a systematic review\.•Unclear / Unanswerable: Vague, ambiguous, overly broad, or not suitable for systematic review \(e\.g\., requires primary data, is normative, or lacks operational clarity\)\.Figure S4:Annotation Guideline for evaluating generated questions derived fromObjectivesinCDSRsystematic reviews\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/question-generation-interface/annotator_guidelines_overview.png)\(a\)Overview Page
![Refer to caption](https://arxiv.org/html/2606.11337v1/x3.png)\(b\)Introduction to Evaluation Dimensions
![Refer to caption](https://arxiv.org/html/2606.11337v1/images/question-generation-interface/annotation_guidelines_faithfulness.png)\(c\)Faithfulness Definition
![Refer to caption](https://arxiv.org/html/2606.11337v1/x4.png)\(d\)Labeled Examples of Faithfulness

Figure S5:Overview of the annotation interface for evaluating generated questions\. Panels show the task introduction and guidelines: \(a\) overview page, \(b\) evaluation dimensions, \(c\) definition of faithfulness, and \(d\) labeled examples of faithfulness\. Panels are cropped or resized for space\. Additional panels of the annotation interface are in Figure[S6](https://arxiv.org/html/2606.11337#A2.F6)\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/question-generation-interface/annotation_guidelines_pico_completeness.png)\(a\)PICO Completeness Definition
![Refer to caption](https://arxiv.org/html/2606.11337v1/x5.png)\(b\)Labeled Examples of PICO Completeness
![Refer to caption](https://arxiv.org/html/2606.11337v1/images/question-generation-interface/annotation_guidelines_clarity.png)\(c\)Clarity & Answerability Definition
![Refer to caption](https://arxiv.org/html/2606.11337v1/images/question-generation-interface/annotation_interface.png)\(d\)Annotation Interface

Figure S6:Additional panels of the annotation interface for evaluating generated questions\. Panels present the interface defining the evaluation dimensions and labeling annotations: \(a\) definition of PICO completeness, \(b\) labeled examples of PICO completeness, \(c\) definition of clarity & answerability, and \(d\) annotation interface\. Panels are cropped or resized for space\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/cochrane-example.png)Figure S7:Example of aCDSRsystematic review article used inSciConBench\. We derive benchmark questions from the publicly availableObjectivessection and reference scientific conclusions from theAuthors’ Conclusionsin theCDSRreview\. Note that theAuthors’ Conclusionssection is partially cut off in the screenshot due to the figure crop, but is fully available in the original review\.

## Appendix CSciConHarnessImplementation Details

SciConHarnessis implemented as a unified MCP\-based control layer that separates*orchestration*from*tool execution*\. The MCP client side handles any LLM providers, managing the model context and tool\-calling state\. The MCP server side exposes API\-based web search and browsing tools over MCP, and, if specified, enforcesclean\-roomfiltering protocols before evidence is returned to the model\. This separation enables consistent evaluation across heterogeneous model APIs while preserving a common tool interface\. Below, we describe the coreSciConHarnessdesign: a unified client–server framework for orchestrating tool use, executing retrieval, and enforcingclean\-roomfiltering during evaluation\.

MCP Client: The Orchestration Engine\.The MCP client is the orchestration engine ofSciConHarness\. The client abstracts LLM provider\-specific semantics \(e\.g\., OpenAI, Anthropic, etc\) into a unified execution loop, formats tool schemas, and maintains intermediate reasoning and structured tool outputs in the context across iterations\. The client also handles retries, error handling, and logs detailed metadata \(e\.g\., iterations, tool invocations, token usage\)\. This design provides a consistent interface for evaluating tool\-using models\.

MCP Server: The Data Plane for Tool Executions\.The MCP server acts as the data plane for API\-based web search and browsing tools\. The server is built onFastMCP999[https://github\.com/prefecthq/fastmcp](https://github.com/prefecthq/fastmcp)and Ai2’sdr\-agent\-lib\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\], which features global caching and asynchronous request handling\[[51](https://arxiv.org/html/2606.11337#bib.bib29)\]\. As described in §[3](https://arxiv.org/html/2606.11337#S3), the server exposesgoogle\_search,paper\_search, andweb\_browse, returning standardized outputs consumable by any client\-supported model\.

To enforce the clean\-room protocol,paper\_searchis temporally constrained to return only papers published before the ground\-truth review, preventing access to post\-publication evidence\. Forweb\_browse, the page text output can be very long—sometimes exceeding the context limits of models such asGemini\-3\-pro\. Following prior work\[[66](https://arxiv.org/html/2606.11337#bib.bib25),[100](https://arxiv.org/html/2606.11337#bib.bib30)\], we summarize retrieved content usinggpt\-5\-mini101010Configured with medium verbosity and reasoning effortto reduce context usage and cost while preserving relevant information\. See Figure[S24](https://arxiv.org/html/2606.11337#A6.F24)for the summarization prompt\. Under theclean\-roomprotocol, filtering is applied server\-side before results are returned to the MCP client\. Centralizing filtering within the MCP server ensures that models cannot access the reference conclusions via retrieval leakage\.

MCP Client–Server Interaction\.At runtime, the client initializes a session with the server and provides the schemas of available tools to the model\. The client then facilitates an iterative loop: \(1\) model inference \(including tool calls\), \(2\) tool\-call parsing, \(3\) tool execution, and \(4\) appending results back into context\. This continues until the model produces a final conclusion wrapped in triple brackets\. All interactions are logged \(e\.g\., tool calls, outputs, reasoning traces\)\. Figure[S25](https://arxiv.org/html/2606.11337#A6.F25)shows the system prompt used to evaluate all models onSciConBench\.

Table S6:Validation results of theSciConHarness’sclean\-roomsetup\. All ground\-truthCDSRarticles \(GT\) were successfully filtered across all tools\.
## Appendix DDetails on Atomic Fact Generation

### D\.1Pipeline Details\.

Our atomic fact generation pipeline decomposes long\-form scientific conclusions \(e\.g\., model\-generated conclusions,Authors’ ConclusioninCDSRreviews\) into a set of self\-contained, complete atomic facts\. Our design draws from prior works on long\-form factuality evaluations\[[73](https://arxiv.org/html/2606.11337#bib.bib38),[123](https://arxiv.org/html/2606.11337#bib.bib140),[69](https://arxiv.org/html/2606.11337#bib.bib37)\], comprising six sequential steps: \(0\) preprocessing, \(1\) decomposition, \(2\) decontextualization, \(3\) incomplete fact rewriting, \(4\) Relevance Filtering, and \(5\) Redundancy filtering\.

Step 0: Preprocessing Module\.Each paragraph\-length conclusion is first tokenized into sentences using NLTK’ssent\_tokenize\[[18](https://arxiv.org/html/2606.11337#bib.bib106)\]\. Following Min et al\.\[[73](https://arxiv.org/html/2606.11337#bib.bib38)\], we apply multi\-pass corrections to fix common tokenization errors: 1\) merge spurious splits from initials \(e\.g\., “J\.K\. Rowling”\), and \(2\) combine bullet lists into a single sentence to preserve context\. We then filter out non\-informative sentences using simple heuristics, removing those that are too short \(<<10 characters or<2<2words\), contain only punctuation or formatting characters, or consist of conversational or meta\-commentary \(e\.g\., “Happy to help,” “Here is a breakdown”\)\.

Step 1: Decomposition Module\.After preprocessing, we decompose each sentence intoatomic factsusinggpt\-5\.1\. For each sentence, the model is given the sentence, its parent paragraph, and the generated question, and outputs a JSON object containing a set of atomic facts with brief justifications for its outputs\. The few\-shot prompt contained detailed guidelines instructinggpt\-5\.1to \(i\) produce independent, self\-contained facts; \(ii\) avoid inferred or indirect facts; \(iii\) avoid writing review\-specific meta\-statements \(e\.g\., “The review found…”\); \(iv\) reframe authorial judgments into direct claims \(e\.g\., “The certainty of the evidence was downgraded due to imprecision” becomes “Imprecision reduced the certainty of the evidence on \[TOPIC\]”\); and \(v\) skip non\-content elements \(conversational markers, formatting\)\. The prompt includes five worked examples fromCDSRAuthors’ Conclusions, illustrating correct decomposition across simple and multi\-clause sentence examples\. See Figure[S9](https://arxiv.org/html/2606.11337#A4.F9)for the prompt\.

Step 2: Decontextualization Module\.We thendecontextualizeeach atomic fact usinggpt\-5\.1so it can be self\-contained without additional context\. Given an atomic fact, its parent paragraph, and the generated question, the model outputs a JSON object containing the decontextualized fact and a brief justification\. Following prior work\[[69](https://arxiv.org/html/2606.11337#bib.bib37),[123](https://arxiv.org/html/2606.11337#bib.bib140)\], this step replaces vague references \(e\.g\., pronouns, shortened names, review\-specific mentions\) with the specific entities they denote, using the parent paragraph and generated question as context while preserving the original claim\. We adapt the prompt from Wei et al\.\[[123](https://arxiv.org/html/2606.11337#bib.bib140)\]and use a few\-shot setup with three worked examples\. See Figure[S10](https://arxiv.org/html/2606.11337#A4.F10)for the prompt\.

Step 3: Incomplete Fact Rewriting Module\.Following Liu et al\.\[[69](https://arxiv.org/html/2606.11337#bib.bib37)\], we identifyincomplete facts—those that require additional context or other facts to be correctly interpreted\. Given the original fact and its parent paragraph,gpt\-5\-miniclassifies each fact asIndependent\(self\-contained\) orDependent\(context\-dependent\)\. Dependent facts are further categorized into three subtypes: \(1\)Ambiguous Concepts/Pronounse\.g\., “this method,” “they” without clear referents, \(2\)Missing Comparisonimplies comparison \(e\.g\., “better”\) without explicit comparison target, and \(3\)Lack of Conditionindicates a missing temporal, hypothetical, or qualifying context \(e\.g\., missing “if” conditions\)\. For dependent facts, the model rewrites the fact by incorporating the missing context from the source paragraph\. See Figure[S11](https://arxiv.org/html/2606.11337#A4.F11)for the prompt\.

Step 4: Relevance Filtering Module\.Usinggpt\-5\-mini, we filter atomic facts for relevance with respect to the generated question, following\[[69](https://arxiv.org/html/2606.11337#bib.bib37),[123](https://arxiv.org/html/2606.11337#bib.bib140)\]\. Following the SAFE approach\[[123](https://arxiv.org/html/2606.11337#bib.bib140)\], we use substitute labels “Foo” and “Not Foo” instead of “Relevant” and “Irrelevant” to force the model to follow our definition of relevance instead of the model relying on its own prior notion of relevance\. A fact is labeled “Foo” \(relevant\) if it directly contributes to answering the question or provides useful background context; otherwise \(e\.g\., generic responses, indirect inferences, or meta\-commentary\), it is labeled “Not Foo\.” The prompt includes three worked examples fromCDSRreviews\. Given a fact, the question, and its parent paragraph as context,gpt\-5\-minioutputs a classification with justification; only “Foo” facts are retained\. This module is disabled forAuthors’ ConclusionsfromCDSRreviews, as they are expert\-written to directly address the question\. See Figure[S12](https://arxiv.org/html/2606.11337#A4.F12)for the prompt\.

Step 5: Redundancy Filtering Module\.After preprocessing and context restoration, some atomic facts became highly redundant\. This module removes redundancy among the set of facts extracted from a single sentence\. For sentences with more than one atomic fact,gpt\-5\-minitakes the sentence and its full fact set as input and returns a maximally non\-redundant subset, preserving the most atomic and specific facts\. See Figure[S13](https://arxiv.org/html/2606.11337#A4.F13)for the prompt\.

### D\.2Cost & Quality Considerations

We design the pipeline to bemodular, enabling per\-component model assignment to balance quality and cost\. Atomic fact generation is financially and computationally expensive at ourscale—processing paragraph\-length conclusions with several sentences—so careful allocation is critical\. We route early, high\-impact stages to more capable models \(e\.g\.,gpt\-5\.1\) to ensure high\-quality facts, and later defined, classification\-style stages to smaller models \(e\.g\.,gpt\-5\-mini\) to reduce cost\.

Specifically, the decomposition \(Step 1\) and decontextualization \(Step 2\) modules usegpt\-5\.1\(without reasoning\), as we observed higher fact quality without any reasoning\. The remaining steps \(Steps 3–5\) usegpt\-5\-miniwith minimal reasoning effort to efficiently handle classification and filtering while maintaining quality\. All components use low verbosity with automatic reasoning summaries\. The pipeline is implemented via the Azure OpenAI Responses API\.

Across the entire five\-step pipeline, for a conclusion withSScontent sentences that decompose into a total ofFFatomic facts, the total number of LLM calls isS\+3​F\+S′S\+3F\+S^\{\\prime\}, whereS′S^\{\\prime\}is the number of sentences with more than one fact after prior filtering stages\. When applying the atomic fact generation pipeline toAuthors’ ConclusionsfromCDSRreviews, agpt\-5\.1\-only pipeline with medium reasoning effort costs $0\.35 per conclusion, while our cost\-optimized pipeline reduces this to $0\.13 per conclusion\.

### D\.3Validating the Pipeline

Overview\.We validate the quality of the atomic fact generations through domain\-expert annotations by two medical doctors\. The task assesses whether each atomic fact represents a single, independent piece of information from its source sentence\. Annotators are based at a leading federal university medical school in Brazil with extensive research experience\. As inputs, they are given the source sentence, its set of atomic facts, and the full paragraph containing the sentence for context\.

Evaluation Dimensions\.We evaluate the atomic fact quality using four dimensions across two granularities \(e\.g\., fact\-level, sentence\-level\), grounded in prior works\[[57](https://arxiv.org/html/2606.11337#bib.bib31),[69](https://arxiv.org/html/2606.11337#bib.bib37),[74](https://arxiv.org/html/2606.11337#bib.bib15),[91](https://arxiv.org/html/2606.11337#bib.bib83)\]\.

At the fact\-level \(per atomic fact\), we evaluate:

- •Faithfulness:Whether the fact accurately reflects the meaning of the source sentence\.Answer Choices: Faithful, Unfaithful
- •Completeness:Whether the fact represents a single complete piece of information\.Answer Choices: Complete Fact, Incomplete Fact, Compound Fact

Table S7:Agreement between two expert annotators on evaluating generated atomic facts \(Ns​e​n​t=20N\_\{sent\}=20,Nf​a​c​t​s=46N\_\{facts\}=46\) across four dimensions: Faithfulness \(Faith\.\), Completeness \(Compl\.\), Comprehensiveness \(Compr\.\), and Redundancy \(Redun\.\)\.At the sentence\-level \(per sentence and its set of atomic facts\), we evaluate:

- •Comprehensiveness:Whether the set of facts completely and accurately captures the meaning of the source sentence\.Answer Choices: Comprehensive, Partially Comprehensive, Not Comprehensive
- •Redundancy:Whether any facts are duplicated or substantially overlap in content\.Answer Choices: No Redundancy, Redundancy

The annotation guidelines were iteratively refined with feedback from graduate students in computer science, clinicians, and medical students, who reviewed the annotation guidelines and worked through sample annotations\. The full annotation guidelines are provided in Figure[S8](https://arxiv.org/html/2606.11337#A4.F8), and the annotation interface used by annotators is shown in Figures[S19](https://arxiv.org/html/2606.11337#A5.F19)\-[S21](https://arxiv.org/html/2606.11337#A5.F21)\.

Annotation Procedure\.To validate our annotation task, two expert annotators independently follow the annotation guidelines to label 20 sentences and their corresponding atomic facts across all evaluation dimensions\. The sentences and atomic facts are stratified and sampled from both generated and reference conclusions fromCDSRreviews\. In total, the 20 sentences contain 46 atomic facts\. We assess annotator reliability using Gwet’s AC1, as done in §[B\.2](https://arxiv.org/html/2606.11337#A2.SS2)\. As mentioned above, fact\-level dimensions \(e\.g\., faithfulness, completeness\) are evaluated per atomic fact \(Nf​a​c​t​s=46N\_\{facts\}\{=\}46\), while sentence\-level dimensions \(e\.g\., comprehensiveness, redundancy\) are evaluated per sentence and its set of atomic facts \(Ns​e​n​t=20N\_\{sent\}\{=\}20\)\.

Table S8:Label distribution across evaluation dimensions for atomic facts\. Overall, the results are reported overNsent=200N\_\{\\text\{sent\}\}\{=\}200sentences \(evenly stratified between generated and reference conclusions fromCDSR\), corresponding toNfacts=469N\_\{\\text\{facts\}\}\{=\}469atomic facts\. Abbreviations: Compre\. = Comprehensive, Partial = Partially Comprehensive, Not Compre\. = Not Comprehensive\.Results\.As shown in Table[S7](https://arxiv.org/html/2606.11337#A4.T7), the inter\-annotator agreement is high across the evaluation dimensions \(Gwet’s AC1: 0\.597 – 0\.955; Percentage Agreement: 0\.7 – 0\.957\), comparable or even exceeding prior works\[[57](https://arxiv.org/html/2606.11337#bib.bib31),[69](https://arxiv.org/html/2606.11337#bib.bib37),[74](https://arxiv.org/html/2606.11337#bib.bib15),[91](https://arxiv.org/html/2606.11337#bib.bib83)\]\. These results validate both annotator reliability and the overall annotation setup\. Following this, the two expert annotators independently label 90 sentences each, stratified evenly between generated and reference conclusions fromCDSRreviews, yieldingNsent=200N\_\{\\text\{sent\}\}\{=\}200sentences andNfacts=469N\_\{\\text\{facts\}\}\{=\}469atomic facts\.

Table[S8](https://arxiv.org/html/2606.11337#A4.T8)summarizes the validation results\. Overall, annotators find the generated atomic facts to be largely faithful \(96\.4%\), complete \(96\.0%\), comprehensive with respect to the source sentence \(98\.0%\), and non\-redundant \(90\.5%\)\. These findings indicate that the atomic fact generation pipeline reliably produces high\-quality atomic facts, supporting their use in downstream decomposition for measuring factual quality in long\-form scientific conclusions\.

Annotation Guideline for Evaluating Generated Atomic FactsTask Overview\.Your task is to evaluate the quality of each atomic fact extracted from a given scientific conclusion or generated LLM output\. Each atomic fact should represent a single independent piece of information from an underlying sentence and should not require additional context\.Inputs\.You will be provided with the following:•Underlying Sentence: Direct source for atomic facts\.•Extracted Atomic Facts: Unit of evaluation, extracted from the sentence•Full Paragraph: Paragraph containing the underlying sentence for additional context\.Evaluation Criteria\.Below, we list two sets of evaluation criteria across different granularities \(e\.g\., fact\-level vs\. sentence\-level\)\. You can optionally leave a note for each annotations\.Atomic Fact\-Level Criteria:For each atomic fact, please evaluate:•Faithfulness\.Does the atomic fact accurately reflect the meaning of the underlying sentence?–Faithful: Correctly represents the meaning of the underlying sentence\. Note that adding additional context is fine\.–Unfaithful: Incorrectly represents or alters the meaning of the underlying sentence\. Examples include changing details \(e\.g\., PICO\) and exaggerating certainty\.•Completeness\.Does the atomic fact represent a single complete piece of information?–Complete Fact: A fact that expresses one fully formed claim with all essential contexts\.–Incomplete Fact: Missing essential information required to interpret the claim on its own, making the fact unclear without additional context\.–Compound Fact: Contains multiple distinct claims or outcomes within a single statement, even after accounting for necessary contextual details\.Note: Some facts may appear lengthy\. If they reflect a single claim overall, they should be marked complete \(e\.g\., “If X, Y” may seem like two pieces of information on X and Y, but it is just one\)\.Sentence\-Level Criteria:For each sentence & its set of atomic fact\(s\), please evaluate:•Comprehensiveness\.Does the full set of atomic facts completely and accurately capture the meaning of the underlying sentence?–Comprehensive: All essential elements in the underlying sentence are accurately represented by the set of atomic facts\.–Partially Comprehensive: The atomic facts capture the core meaning of the sentence but omit, weaken, or inaccurately reflect minor elements that do not fundamentally change the meaning of the sentence\.–Not Comprehensive: The atomic facts omit or misrepresent essential meaning to the extent that the overall meaning of the sentence is distorted\.Note: Please focus on whether essential details are reflected rather than contextual or minor details\.•Redundancy\.Are any atomic facts substantially duplicated or redundant with another fact in information content?–No Redundancy:All facts are not redundant or contribute distinct information\. A set of a single atomic fact should be marked with this\.–Redundancy:One or more atomic facts repeat information already expressed in another fact \(e\.g\., duplicates or rephrasing\) and do not contribute any additional meaning, even minor distinctions\.Note: Some atomic facts may share partial information\. If each fact contributes distinct information, even if minor, and is not a rephrasing, it should be “No Redundancy\.”Figure S8:Annotation guidelines for evaluating atomic facts extracted from scientific conclusions\. The instruction defines fact\-level criteria \(faithfulness, completeness\) and sentence\-level criteria \(comprehensiveness, redundancy\) to assess the quality of the decomposed facts\.Atomic Fact Generation \(Step 1\): Decomposition PromptSystem Prompt:You are an expert in breaking down complex scientific sentences into atomic facts–short statements that each contain one piece of information\. Given the entire scientific paragraph as context, you will be given one of the sentences and you will need to break the sentence down into atomic facts\.
Instruction:Using the full scientific paragraph as context, please breakdown one of the sentences from the paragraph into independent atomic facts and justify your answer\. The atomic facts should be short statements that each contain one piece of information\.\*\*\*GUIDELINES FOR GENERATING ATOMIC FACTS STARTS HERE:\*\*\*
\- Make sure that each atomic fact is independent\. Each atomic fact in the output should contain different pieces of information\.\- Make sure the atomic facts can stand on its own as a fact, using the provided paragraph to contextualize each atomic fact\.\- Do not create atomic facts that are inferred or indirect from the sentence\. Focus directly on the sentence and the information it contains\.\- Do not create atomic facts specific to the study itself, but rather focus on the main conclusions\.\- Do not assume that atomic facts inform one another in terms of context\. Treat each atomic fact as a separate, independent fact, each of which needs its own context to be understood on its own\.\- Do not generate atomic facts that reflect authorial judgments or methodological decisions made by the systematic review authors, or that require that context to be understood\.\- DO NOT generate atomic facts for non\-content elements…\- If the sentence contains only non\-content elements, return an empty list of atomic facts\.\*\*\*GUIDELINES FOR GENERATING ATOMIC FACTS ENDS HERE\.\*\*\*
Here are five examples of how to breakdown a sentence into independent atomic facts:\*\*\*EXAMPLE 1 STARTS HERE\*\*\*
\*Full Paragraph:\* …\*Sentence:\* …\*Atomic facts:\* …\*Justification\*: …\*\*\*EXAMPLE 1 ENDS HERE\*\*\*
……Now, given what you learned from the guidelines and examples, please breakdown the sentence into independent atomic facts, using the provided paragraph and question as context and providing justification and thinking step\-by\-step about your answer\. Avoid creating atomic facts specific to the study itself, but rather focus on the main conclusions\.\*\*\*QUESTION STARTS HERE\*\*\* Use this question as context for your task\.
\{question\}\*\*\*QUESTION ENDS HERE\*\*\*
\*\*\*FULL PARAGRAPH STARTS HERE\*\*\* Use this paragraph as context for your task\.
\{paragraph\}\*\*\*FULL PARAGRAPH ENDS HERE\*\*\*
\*\*\*SENTENCE STARTS HERE\*\*\*\{sentence\}\*\*\*SENTENCE ENDS HERE\*\*\*
The atomic facts should be short statements that each contain one piece of information\. Output should be in JSON format with the following keys:\- “atomic\_facts”: list of atomic facts as strings\- “justification”: string justification for your atomic fact breakdown\.Figure S9:Decomposition prompt used to decompose sentences intoatomic factsusinggpt\-5\.1\. The sentence, its parent paragraph, and the generated question were provided as input, filling in their bracketed components in the prompt\. The prompt was shortened to fit within the page\. Full prompt available in supplementary material / code\.Atomic Fact Generation \(Step 2\): Decontextualization PromptSystem Prompt:You are an expert in decontextualizing vague references in scientific sentences into more specific entities, ensuring that any statement relations are not out of context and are self\-contained\. Given a statement and a response, you will need to decontextualize the vague references in the statement\.
Instruction:Evaluate the provided statement in context of the response\. The statement should be decontextualized such that they are understandable without referencing the rest of the response\.Instructions:
1\. The following STATEMENT has been extracted from the broader context of the given RESPONSE\.2\. Modify the STATEMENT by replacing vague references with the proper entities from the RESPONSE that they are referring to\.3\. You MUST NOT change any of the factual claims made by the original STATEMENT\.4\. You MUST NOT add any additional factual claims to the original STATEMENT\.5\. You MUST NOT generate atomic facts that reflect authorial judgements or methodological decisions made by the systematic review authors\.6\. Before giving your revised statement, think step\-by\-step and show your reasoning\. As part of your reasoning, be sure to identify the subjects in the STATEMENT and determine whether they are vague references\. If they are vague references, identify the proper entity that they are referring to and be sure to revise this subject in the revised statement\.7\. Your task is to do this for the STATEMENT and RESPONSE under “Your Task”\. Some examples have been provided for you to learn how to do this task\.Vague references include but are not limited to:\- Pronouns \(e\.g\., “his”, “they”, “her”\)\- Unknown entities \(e\.g\., “this event”, “the research”, “the invention”, “this review”, “the study”, “many studies”\)\- Non\-full names \(e\.g\., “Jeff…” or “Bezos…” when referring to Jeff Bezos\)\- Systematic review, specific studies, or specific evidence \(e\.g\., “the review”, “the study”, “many studies”, “A review”, “Our review”, “this evidence”, “in the review”, “in this evidence”\)You SHOULD NOT generate atomic facts that reference the systematic review or specific studies in any shape or form e\.g\., “The certainy of evidence in this systematic review…”Example 1:
STATEMENT: …RESPONSE: …REVISED STATEMENT: ………Your Task:
QUESTION:
\{question\}STATEMENT:
\{individual\_fact\}RESPONSE:
\{response\}The atomic fact should be short statements that each contain one piece of information\. Use the QUESTION as additional context to help understand what the STATEMENT and RESPONSE are referring to\. Output should be in JSON format with the following keys:\- “decontextualized\_fact”: a string of the atomic fact decontextualized to be self\-contained\.\- “justification”: string justification for your decontextualization process of the atomic fact\.Figure S10:Decontextualization prompt for makingatomic factsself\-contained usinggpt\-5\.1\. Inputs include the fact, its parent paragraph, and the generated question, which replaced their bracketed component in the prompt\. The prompt was shortened to fit within the page\. Full prompt available in supplementary material / code\.Atomic Fact Generation \(Step 3\): Incomplete Fact Rewriting PromptInstruction:Given a context and a claim extracted from the context, determine whether the claim is Dependent or Independent of the context\.\*Independent: If the claim itself precisely reflects the original meaning of the context without further explanation\.\*Dependent: If the claim requires additional context or detail to precisely reflect its original meaning\.Categorize independent claims into one of three types:\*Ambiguous Concepts/Pronouns
The claim contains vague terms \(e\.g\., “this method,” “they”\) or pronouns lacking clear referents from the context\.Example:Context: ‘Decarbonizing aviation requires SAFs\.”Claim: “They reduce emissions\.” → Dependent \(Ambiguous pronoun “they”\)\.\*Missing Comparison
The claim implies a comparison \(e\.g\., “more,” “better”\) but omits the explicit comparison target stated in the context\.Example:Context: “SAFs reduce emissions by 80% compared to jet fuels\.”Claim: “SAFs reduce emissions by 80%\.” → Dependent \(Missing “compared to jet fuels”\)\.\*Lack of Condition
The claim omits critical contextual details, such as:\- Temporal conditions \(e\.g\., “as of 2023”\)\.\- Hypothetical scenarios \(e\.g\., “if regulations are adopted”\)\.Example:Context: “As of 2023, the U\.S\. top 1% net worth is $10M \(Smith et al\., 2023\)\.”Claim: “The U\.S\. top 1% net worth is $10M\.” → Dependent \(Missing time\)\.ONE CAVEAT: Do not mark claim as Dependent based on not having enough context regarding the scope of claim\. The atomic fact is intentionally designed to be self\-contained and avoid having any specific details regarding the review or specific studies\.If the claim is Dependent, you must also provide a rewritten version that makes it Independent by incorporating the missing context\. Some guidelines for rewriting dependent claims:\- You MUST NOT not rewrite atomic facts that reflect authorial judgements or methodological decisions made by the systematic review authors\.Output should be in JSON format with the following keys:\- “classification”: string, either “Independent” or “Dependent”\- “dependent\_type”: string, one of “Ambiguous Concepts/Pronouns”, “Missing Comparison”, “Lack of Condition”, or “None” \(if Independent\)\- “explanation”: string, explanation for your classification\- “rewritten\_claim”: string, if classification is “Dependent”, provide a rewritten version that makes the claim Independent by incorporating missing context\. If “Independent”, this should be empty\.\# ExampleContext:…Claim:…Your Response:……\# Your Task Context:\{context\}Claim:\{claim\}Your Response:Figure S11:Prompt for incomplete fact rewriting\. Given an atomic fact as the “claim” and its parent paragraph as the “context,”gpt\-5\-miniclassifies the fact asIndependentorDependent; dependent facts are rewritten by integrating the missing context\. The prompt was shortened to fit within the page\. Full prompt available in supplementary material / code\.Atomic Fact Generation \(Step 4\): Relevance Filtering PromptInstruction:A STATEMENT is considered “Foo” if the STATEMENT is directly relevant or provides beneficial background context to addressing the QUESTION in context of the RESPONSE\.Instructions:
1\. The following STATEMENT has been extracted from the broader context of the given RESPONSE to the given QUESTION\.2\. First, consider the provided STATEMENT and QUESTION in context of the RESPONSE\.3\. Next, determine whether the STATEMENT is considered “Foo” e\.g\., is it directly relevant or provides beneficial background context to addressing the QUESTION in context of the RESPONSE\.4\. Before showing your answer, think step\-by\-step and show your specific reasoning\.5\. If the STATEMENT is considered “Foo”, say “\[Foo\]” after showing your reasoning\. Otherwise show “\[Not Foo\]” after showing your reasoning\.6\. Your task is to do this for the STATEMENT and RESPONSE under “Your Task”\. Some examples have been provided for you to learn how to do this task\.\*\*General Rule of Thumb\*\*
\- If a statement is generic response \(e\.g\., “I cannot help with that,” “I am not familiar with that,” etc\.\) or does not provide any information that is directly relevant to addressing the QUESTION in context of the RESPONSE, then the STATEMENT is “\[Not Foo\]”\.\- If a statement is more indirect or inferred from the RESPONSE, then the STATEMENT is considered “\[Not Foo\]”\. The statement should be self\-contained, while being directly relevant or providing helpful background context to addressing the QUESTION in context of the RESPONSE\. Err to the side of\[Foo\]if the statement provides background context to the RESPONSE\. For example, statement on the need for additional high\-quality research on some topic and related outcomes provides helpful background context and addresses the QUESTION in context of the RESPONSE\.Example 1:
QUESTION: …RESPONSE: …STATEMENT: …SOLUTION: ……Your Task:
QUESTION:\{question\}RESPONSE:\{response\}STATEMENT:\{fact\}Output should be in JSON format with the following keys:\- “reasoning”: string, step\-by\-step reasoning for your determination\- “classification”: string, either “\[Foo\]” or “\[Not Foo\]”Figure S12:Prompt for Relevance Filtering\. Given an atomic fact, its parent paragraph as the response, and the question,gpt\-5\-miniclassifies the fact asrelevant\(e\.g\.,Foo\) orirrelevant\(e\.g\.,Not Foo\); only relevant facts are retained\. The prompt was shortened to fit within the page\. Full prompt available in supplementary material / code\.Atomic Fact Generation \(Step 5\): Redundancy Filtering PromptInstruction:You are given a RESPONSE and a list of STATEMENTS extracted from that RESPONSE\. Your task is to select the most atomic set of facts by removing redundant statements, while keeping as many statements that provide as much information about the RESPONSE as possible\.Instructions:
1\. You are given a RESPONSE and a numbered list of STATEMENTS extracted from that RESPONSE\.2\. Your goal is to select the most atomic set of facts by:\- Identifying groups of redundant statements \(statements that convey exactly the same meaning\)\- For each redundant group, keeping ONLY the best statement\(s\)\. Keep the set that retains as many atomic facts as possible while accurately representing the original RESPONSE\.3\. A statement is redundant with another if:\- It expresses the EXACT SAME meaning as another statement \(even if worded differently\)\- It does not provide any further information beyond what is already covered by other statements\.4\. When choosing which statement to keep from a redundant group:\- STRONGLY prefer more atomic facts: If a statement combines multiple pieces of information \(e\.g\., “diarrhea, nausea, and vomiting”\), prefer keeping individual atomic statements \(e\.g\., separate statements for diarrhea, nausea, vomiting\) over the combined statement\- Prefer more specific statements over general ones: If a statement is too broad and does not accurately represent the original RESPONSE, prefer more specific statements that provide comprehensive information5\. A statement should be kept if:\- It provides unique information not covered by any other statement\- It is the best version in its redundant group \(most atomic, most specific, most accurate to RESPONSE\)\- It accurately represents the original RESPONSE with specific details\- When in doubt, err on the side of keeping the fact\.6\. A statement should be removed if:\- It is redundant with another statement and does not provide any further information beyond what is already covered\- It is overly general and does not accurately represent the original RESPONSE7\. Before showing your answer, think step\-by\-step about:\- Which statements are redundant with each other\- Which statements are more atomic \(individual facts vs combined facts\)\- Which statements are more specific and accurately represent the RESPONSE\- Which statements are overly general and should be removedTHINK CAREFULLY STEP\-BY\-STEP BEFORE EXCLUDING REDUNDANT STATEMENTS\.…Example 1:
RESPONSE: …STATEMENTS: …SOLUTION: …Selected statements: ……Your Task:
RESPONSE:\{response\}STATEMENTS:\{all\_facts\}Output should be in JSON format with the following keys:\- “reasoning”: string, step\-by\-step reasoning explaining which statements are redundant, which ones you’re keeping, and why\- “selected\_statements”: array of integers, the statement numbers \(1\-based\) from the STATEMENTS list that should be kept in the final atomic set\. This should contain no redundant statements and preserve all unique information\.Figure S13:Prompt for redundancy filtering\. Given a sentence as the response and its full set of atomic facts,gpt\-5\-minioutputs the maximally non\-redundant subset\. The prompt was shortened to fit within the page\. Full prompt available in supplementary material / code\.

## Appendix EDetails on Measuring Factual Precision and Recall

We employ LLM\-based judges to measure factual precision and recall atscaleby assigning labels to individual facts\. To validate these judges, we construct an expert\-annotated gold\-standard dataset with medical doctors with extensive clinical practice and research experience \(§[E\.1](https://arxiv.org/html/2606.11337#A5.SS1)\) and evaluate judge performance against these annotations \(§[E\.2](https://arxiv.org/html/2606.11337#A5.SS2)\)\.

### E\.1Creating the Expert\-Annotated Gold\-Standard Dataset\.

We describe the annotation guidelines \(§[E\.1\.1](https://arxiv.org/html/2606.11337#A5.SS1.SSS1)\) and the annotation procedure with three medical doctors, including inter\-annotator agreement \(§[E\.1\.2](https://arxiv.org/html/2606.11337#A5.SS1.SSS2)\), used to construct the gold\-standard dataset\.

#### E\.1\.1Annotation Guidelines

To construct our gold\-standard dataset, annotators perform two tasks: factual precision and factual recall\. For both tasks, annotators are not informed whether facts originate from model\-generated orCDSRAuthors’ Conclusionsto mitigate potential priming effects that may lead annotators to be overly cautious or systematically biased against model\-generated conclusions\.

Factual Precision Task\.As described in §[4\.2](https://arxiv.org/html/2606.11337#S4.SS2), factual precision measures thecorrectnessof the generated conclusion\. In this task, annotators assess whether each fact is factually supported and non\-contradictory with respect to a trusted source text—theCDSRreview\. For each fact, annotators carefully examine the fullCDSRreview article on the web to assign one of the following labels, along with supporting excerpts and a brief justification:Supported,Contradicted, andNot Supported\. All facts evaluated in this task are extracted from generated conclusions, though annotators are not informed of their origin\. See Figure[S16](https://arxiv.org/html/2606.11337#A5.F16)for the full annotation guidelines and label definitions\.

Factual Recall Task\.Factual recall measures thecoverageof the generated conclusion, evaluating the extent to which generated conclusionscoverfacts from theAuthors’ ConclusionofCDSRreviews, which are treated as the authoritative set of facts required to answer the question\. In this task, annotators assess whether each fact is supported by a conclusion\. For each fact, annotators examine the conclusion to assign one of two labels, along with supporting excerpts and a brief justification:SupportedandNot Supported\. All facts evaluated in this task are derived from theAuthors’ ConclusionofCDSRreviews, while all evaluated conclusions are model\-generated; their origins are not disclosed to annotators\. See Figure[S17](https://arxiv.org/html/2606.11337#A5.F17)for the full annotation guidelines and label definitions\.

See Figures[S19](https://arxiv.org/html/2606.11337#A5.F19)–[S21](https://arxiv.org/html/2606.11337#A5.F21)for the annotation interface used in both tasks\.

#### E\.1\.2Annotation Procedure\.

Verifying each atomic fact requires reading a corresponding long\-form text \(e\.g\.,CDSRreview for factual precision, generated conclusion for factual recall\)\. Naively sampling facts across conclusions would require annotators to repeatedly switch between different long\-form texts, making the process inefficient\. To improve tractability while preserving coverage, we adopt a conclusion\-level subsampling strategy: for our initial annotation batch, we sample 5 generated conclusions, select up to 10 facts per conclusion for factual precision \(all verified against the sameCDSRarticle\), and use all facts from theAuthors’ Conclusionsfor factual recall on the same generated conclusion\.

Two medical doctors annotated the initial batch of 5 generated conclusions, corresponding to 50 facts for factual precision and 46 facts for factual recall\. Following this round, annotators engaged in discussions to resolve disagreements, improve their agreement, and iteratively refine the annotation guidelines\. Afterward, the two expert annotators conducted a second round of annotations on 8 additional generated conclusions and their corresponding facts for both tasks\. This resulted in a total ofN=129N=129facts for precision andN=119N=119for recall, comparable to or larger than sample sizes in prior work relying on expert annotations to evaluate LLM judge performance\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[27](https://arxiv.org/html/2606.11337#bib.bib9),[28](https://arxiv.org/html/2606.11337#bib.bib35),[56](https://arxiv.org/html/2606.11337#bib.bib11),[97](https://arxiv.org/html/2606.11337#bib.bib10)\]\. For both rounds, a third medical doctor independently reviewed and resolved any remaining disagreements to produce final labels\. Overall, the annotation process was time\-intensive, requiring approximately 6 minutes per fact for the expert annotators\.

Table S9:Inter\-annotator agreement between two expert annotators for factual precision \(N=129N=129\) and recall \(N=119N=119\) across two annotation rounds and overall\. Agreement is reported using Cohen’sκ\\kappa, Gwet’s AC1, and percentage agreement, all of which improve from Round 1 to Round 2 following adjudication and discussion of disagreement cases\. These agreement rates are comparable to, or exceed, those reported in prior work\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38)\]\.Agreement Results\.As shown in Table[S9](https://arxiv.org/html/2606.11337#A5.T9), we observe high agreement across all metrics for both factual precision \(Cohen’sκ\\kappa: 0\.517; AC1: 0\.544; Percentage Agreement: 69%\) and factual recall \(Cohen’sκ\\kappa: 0\.658; AC1: 0\.671; Percentage Agreement: 83\.2%\), comparable to or exceeding prior work\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38)\]\. Importantly, expert disagreement does not necessarily indicate noise; it can reflectlegitimate differencesin how experts prioritize aspects in scientific tasks based on their expertise, background, and inferred goals\[[50](https://arxiv.org/html/2606.11337#bib.bib97)\]\. Overall, despite the challenging and expertise\-intensive nature of the tasks, these results indicate that our annotation task is well\-defined and yields reliable labels\.

Following discussion of disagreement cases, the agreement consistently improves from Round 1 to Round 2 across all metrics \(e\.g\., by approximately 0\.12–0\.17 in Cohen’sκ\\kappaand Gwet’s AC1\), indicating improved annotator calibrations and annotation consistency\. In addition, we observe lower agreement for factual precision compared to factual recall\. Factual precision is inherently more challenging: annotators must verify each fact against the fullCDSRarticle and assign finer\-grained labels \(e\.g\., supported, contradicted, not supported\), whereas factual recall involves checking coverage against a single conclusion paragraph with a binary decision\. Despite this increased complexity, agreement for factual precision remains strong and comparable to prior work\[[24](https://arxiv.org/html/2606.11337#bib.bib72),[69](https://arxiv.org/html/2606.11337#bib.bib37),[73](https://arxiv.org/html/2606.11337#bib.bib38)\]\.

Most importantly, we employ a third medical doctor to independently adjudicate disagreements between the two expert annotators, producing consensus labels\. This consensus\-based process is important for this expertise\-intensive task and further improves the quality of the final labels, particularly in cases of disagreement\. The resulting gold\-standard dataset contains 19Contradicted, 54Supported, and 56Not Supportedlabels for factual precision, and 48Supportedand 71Not Supportedlabels for factual recall\.

### E\.2Validation of LLM Judges\.

We describe the features \(§[E\.2\.1](https://arxiv.org/html/2606.11337#A5.SS2.SSS1)\), the prompt engineering process \(§[E\.2\.2](https://arxiv.org/html/2606.11337#A5.SS2.SSS2)\), report performance against our expert\-annotated gold\-standard dataset across LLMs \(§[E\.2\.3](https://arxiv.org/html/2606.11337#A5.SS2.SSS3)\), and conduct error analysis on the LLM judge \(§[E\.2\.4](https://arxiv.org/html/2606.11337#A5.SS2.SSS4)\)\.

#### E\.2\.1Feature Descriptions\.

Our prompts provide the same input features used in the expert annotation tasks \(§[E\.1\.1](https://arxiv.org/html/2606.11337#A5.SS1.SSS1)\)\.

Factual Precision Task:The model assesses whether each atomic fact is factually supported and non\-contradictory with respect to a trusted source text\.

- •Atomic Fact: An atomic fact extracted from a model\-generated conclusion, evaluated against a trusted source text\.
- •Source Text: The full abstract and plain\-language summary sections of theCDSRsystematic review article\. These sections summarize the key methods, results, and conclusions of the review, includingObjectives,Main Results, andAuthors’ Conclusions, without introducing new information beyond the main body\. As full reviews may be paywalled with copyright restrictions, we rely on these publicly available, open\-access sections, such as the Abstracts and Plain\-Language Summaries\. Prior work has similarly adopted this approach\[[16](https://arxiv.org/html/2606.11337#bib.bib52),[33](https://arxiv.org/html/2606.11337#bib.bib75),[98](https://arxiv.org/html/2606.11337#bib.bib56),[109](https://arxiv.org/html/2606.11337#bib.bib40)\]\.

Factual Recall Task:The model assesses whether each atomic fact is supported by the conclusion \(e\.g\., source text\)\.

- •Atomic Fact: An atomic fact extracted from the referenceAuthors’ Conclusionsof aCDSRreview\.
- •Source Text: The model\-generated conclusion, used to assess whether it contains or directly supports the atomic fact\.

#### E\.2\.2Prompt Design Considerations\.

Our prompt design is guided by prompt\-engineering recommendations from OpenAI\[[86](https://arxiv.org/html/2606.11337#bib.bib86)\], Anthropic\[[9](https://arxiv.org/html/2606.11337#bib.bib84)\], Google Gemini\[[43](https://arxiv.org/html/2606.11337#bib.bib85)\], as well as prior works\[[28](https://arxiv.org/html/2606.11337#bib.bib35),[55](https://arxiv.org/html/2606.11337#bib.bib87),[75](https://arxiv.org/html/2606.11337#bib.bib82),[91](https://arxiv.org/html/2606.11337#bib.bib83),[121](https://arxiv.org/html/2606.11337#bib.bib139),[129](https://arxiv.org/html/2606.11337#bib.bib53),[132](https://arxiv.org/html/2606.11337#bib.bib107)\]\. For each task, we design both zero\-shot and few\-shot prompts following these considerations\. See Figure[S22](https://arxiv.org/html/2606.11337#A5.F22)for the factual precision prompt and Figure[S23](https://arxiv.org/html/2606.11337#A5.F23)for the factual recall prompt\. Below, we outline the key prompt design features considered:

- •System Roles:While personas can improve model performance\[[86](https://arxiv.org/html/2606.11337#bib.bib86)\], their effects are often inconsistent\[[132](https://arxiv.org/html/2606.11337#bib.bib107)\]\. However,Zhenget al\.\[[132](https://arxiv.org/html/2606.11337#bib.bib107)\]suggests that “gender\-neutral, in\-domain, and work\-related roles” yield more reliable improvements than other persona types\[[132](https://arxiv.org/html/2606.11337#bib.bib107)\]\. Given the evaluation\-oriented and evidence\-based nature of our tasks, we adopt an expert evaluator persona: “You are an expert evaluator with deep expertise in evidence\-based medicine and clinical research\.”
- •Contextual Details:Providing sufficient context improves LLM reasoning and justification\[[9](https://arxiv.org/html/2606.11337#bib.bib84),[43](https://arxiv.org/html/2606.11337#bib.bib85),[86](https://arxiv.org/html/2606.11337#bib.bib86)\]\. Accordingly, we include detailed input descriptions, explicit output specifications, and comprehensive guidelines and decision criteria for each output label\.
- •Temperature:Temperature influences how models generate text\[[86](https://arxiv.org/html/2606.11337#bib.bib86)\]\. Lower values \(e\.g\., 0\) produce more deterministic and consistent responses, while higher values \(e\.g\., 1\) yield more diverse outputs\. Forgpt\-5\.4\-mini,claude\-haiku\-4\.5, andgemini\-3\-flash, the default temperature is 1\[[7](https://arxiv.org/html/2606.11337#bib.bib135),[41](https://arxiv.org/html/2606.11337#bib.bib137),[83](https://arxiv.org/html/2606.11337#bib.bib136)\]\. Prior work\[[75](https://arxiv.org/html/2606.11337#bib.bib82),[28](https://arxiv.org/html/2606.11337#bib.bib35),[91](https://arxiv.org/html/2606.11337#bib.bib83),[55](https://arxiv.org/html/2606.11337#bib.bib87)\]suggests that moderate temperatures \(e\.g\., 0\.2\) perform best for structured and defined evaluation tasks such as misinformation detection and factuality assessment\. Although a temperature of 0 may lead to text degeneration\[[47](https://arxiv.org/html/2606.11337#bib.bib138)\], we include it to assess performance under fully deterministic settings\. We evaluate performance across temperatures\{0,0\.2,1\}\\\{0,0\.2,1\\\}\.
- •Zero\-Shot vs\. Few\-Shot:For each task, we evaluate both zero\-shot and few\-shot prompting\. Zero\-shot prompts present the task without examples, while few\-shot prompts provide labeled examples to enable in\-context learning without updating model weights\[[19](https://arxiv.org/html/2606.11337#bib.bib124)\]\. For few\-shot prompting, we manually construct six examples\. Each example includes the input features \(e\.g\., source text and atomic fact; see §[E\.2\.1](https://arxiv.org/html/2606.11337#A5.SS2.SSS1)\), along with the output label, a supporting excerpt from the source text, and a detailed justification\.
- •Reasoning:Prompting LLMs to reason step by step and justify their decisions has been shown to improve performance across a range of tasks\[[121](https://arxiv.org/html/2606.11337#bib.bib139)\], including factuality assessment and misinformation detection\[[55](https://arxiv.org/html/2606.11337#bib.bib87),[76](https://arxiv.org/html/2606.11337#bib.bib123)\]\. Following this approach, we instruct models to reason step by step before producing an output label, a brief supporting excerpt, and a justification\. Beyond standard step\-by\-step prompting, we also vary the reasoning level across models to assess whether additional reasoning improves performance on factual precision and recall\. This is especially relevant because these labeling tasks are challenging, reasoning\-intensive, and require substantial domain expertise, even for medical doctor annotators\. We therefore evaluate all available reasoning settings for each model\. For most models, reasoning is not constrained by a separate token budget; forclaude\-haiku\-4\.5, extended thinking uses a budget of 1,024 thinking tokens to stay within reasonable costs\.

#### E\.2\.3Evaluation Results

Using both zero\-shot and few\-shot prompts across varying reasoning and temperature settings, we evaluate three LLMs—gpt\-5\.4\-mini,claude\-haiku\-4\.5, andgemini\-3\-flash—on the gold\-standard dataset for both factual precision and recall tasks\. Full results are reported in Table[S10](https://arxiv.org/html/2606.11337#A5.T10)for factual precision and Table[S11](https://arxiv.org/html/2606.11337#A5.T11)for factual recall\. To mitigate evaluation leakage, we exclude the six few\-shot examples used in the prompts from the evaluation set for both tasks\.

gpt\-5\.4\-miniachieved the strongest overall performance\.Across both tasks,gpt\-5\.4\-minioutperformed the other models in the best\-performing configuration \(e\.g\., prompt, reasoning, temperature\)\. For factual precision, its best setting achieved a macro F1 of 0\.837 with accuracy 0\.830; for factual recall, it achieved a macro F1 of 0\.868 with accuracy 0\.903\.gemini\-3\-flashwas the next strongest model, reaching a best macro F1 of 0\.777 with accuracy 0\.797 for factual precision, and 0\.844 with accuracy 0\.876 for factual recall\.

For both tasks, expert annotators agree withgpt\-5\.4\-minimore than with each other\.Beyond absolute performance, these results are supported by strong agreement with our expert annotators\. As shown in Table[1](https://arxiv.org/html/2606.11337#S4.T1), the LLM judge \(gpt\-5\.4\-mini, in its best configuration\) agrees with each expert at rates comparable to—and in some cases higher than—the agreement between the experts themselves\. Notably, for factual recall, the average Expert–LLM agreement \(Cohen’sκ=0\.695\\kappa=0\.695, Gwet’s AC1=0\.723=0\.723\) exceeds Expert–Expert agreement \(Cohen’sκ=0\.658\\kappa=0\.658, Gwet’s AC1=0\.67=0\.67\), and the same pattern holds for factual precision\. These findings suggest that the LLM judge operates at a level comparable to expert annotators on our gold\-standard dataset\. Overall, the strong agreement with multiple experts—alongside high task performance—validates the quality of our prompts and supports the use of frontier LLMs as effective and reliable evaluators for our tasks\.

Higher reasoning levels did not consistently improve performance\.Increasing reasoning effort yielded limited benefit and often reduced performance\. Forgpt\-5\.4\-mini, the best results on both tasks were obtained without reasoning: factual precision peaked at reasoningNonewith temperature 0\.2, and factual recall peaked at reasoningNonewith temperature 1\. Under matched model, prompt, and temperature settings, increasing reasoning can substantially reduce performance\. For example, forgpt\-5\.4\-mini, moving from no reasoning to higher reasoning under the same configuration reduces factual precision from 0\.837 to 0\.706 and factual recall from 0\.868 to 0\.704\. This pattern is consistent with prior work showing that increasing reasoning levels do not always improve performance and can even degrade it in tasks such as mathematical reasoning and text classification\[[17](https://arxiv.org/html/2606.11337#bib.bib109),[97](https://arxiv.org/html/2606.11337#bib.bib10)\]\.

Non\-zero temperatures often performed better than temperature 0\.The best\-performing configurations forgpt\-5\.4\-minioccurred at non\-zero temperatures \(0\.2 for factual precision and 1 for factual recall\)\. A similar trend holds forgemini\-3\-flashandclaude\-haiku\-4\.5, whose strongest performance also appears at temperatures 0\.2 or 1 rather than 0\. This suggests that a small degree of sampling variability can be beneficial, even for evaluator\-style prompting\. Our result is consistent with prior work showing that deterministic decoding \(e\.g\., temperature 0\) can lead to text degeneration\[[47](https://arxiv.org/html/2606.11337#bib.bib138)\]\.

Few\-shot prompting had mixed effects, and did not consistently improve performance\.Few\-shot prompting did not uniformly outperform zero\-shot prompting under matched model, reasoning, and temperature settings\. For factual precision, the change in macro F1 from zero\-shot to few\-shot ranged from−0\.034\-0\.034to\+0\.395\+0\.395, indicating that few\-shot examples could either slightly hurt performance or substantially improve it depending on the configuration\. In contrast, factual recall showed even more mixed results: the corresponding change ranged from−0\.182\-0\.182to\+0\.075\+0\.075, with smaller gains and larger degradations overall\. This instability was especially consistent forclaude\-haiku\-4\.5, which consistently degraded under few\-shot prompting on the factual recall task \(range:−0\.097\-0\.097to−0\.019\-0\.019\)\. Taken together, these results suggest that few\-shot prompting is more sensitive to task, model choice, and hyperparameter selection \(e\.g\., temperature, reasoning level\) for LLM\-based evaluation than is often assumed\.

#### E\.2\.4Error Analysis

Whilegpt\-5\.4\-minias an LLM judge offers strong performance and expert\-level agreement with medical doctors, we conduct detailed error analysis on both factual precision and recall tasks to identify common failure modes and highlight areas for improving LLM judge accuracy and reliability\. In both tasks, the two common errors are:

Error \#1: Missing contextual equivalence and clinical synonymy\.In both tasks, the LLM judge occasionally fails to recognize clinically equivalent contexts and terminology\. As shown in Figure[S14](https://arxiv.org/html/2606.11337#A5.F14), while the source text provides strong evidence that antibiotics do not improve symptom persistence in closely related conditions \(e\.g\., common cold, acute purulent rhinitis\), the model treats differences in wording—such as “acute rhinosinusitis” and “longer\-term symptom duration”—as substantive mismatches and thus incorrectly labels the fact asNot Supported\. In contrast, expert annotators correctly interpret these as medically aligned concepts, grounding their judgment in a broader clinical context rather than strict lexical overlap, and label the atomic fact asSupported\. This suggests that the LLM judge can be sensitive to surface\-level discrepancies \(e\.g\., specific clinical jargon\) and lacks robust handling of medical synonymy and outcome equivalence, leading to false negatives in factual precision evaluation\.

Atomic Fact:For children with acute upper respiratory tract infections or acute rhinosinusitis diagnosed within the first 10 days, antibiotics do not meaningfully reduce longer\-term symptom duration compared with placebo\.
Source Text:[https://www\.cochranelibrary\.com/cdsr/doi/10\.1002/14651858\.CD000247\.pub4](https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD000247.pub4)Both Expert Annotators:1\.Label:Supported2\.Excerpt from Source Text: “Participants receiving antibiotics for the common cold did no better in terms of lack of cure or persistence of symptoms than those on placebo \(risk ratio \(RR\) 0\.83, 95% confidence interval \(CI\) 0\.60 to 1\.14, \(random\-e@ects\)”3\.Justification: “Results applicable to children and adults\.”LLM Judge\(gpt\-5\.4\-mini\):1\.Label:Not Supported2\.Excerpt from Source Text: “Participants receiving antibiotics for the common cold did no better in terms of lack of cure or persistence of symptoms than those on placebo \(risk ratio \(RR\) 0\.83, 95% confidence interval \(CI\) 0\.60 to 1\.14, \(random\-e@ects\)”, “There is no evidence of benefit from antibiotics for the common cold or for persisting acute purulent rhinitis in children or adults\.”3\.Justification: “The source text does support a null effect for antibiotics in common cold and acute purulent rhinitis/acute upper respiratory infection generally, including pediatric participants… However, the atomic fact is more specific than the source in two important ways: it narrows the condition to “acute rhinosinusitis” and claims effects on “longer\-term symptom duration,” whereas the source discusses common cold and acute purulent rhinitis with outcomes of lack of cure/persistence of symptoms, not longer\-term symptom duration, and it does not clearly identify rhinosinusitis as the studied condition\. Because the comparator, population, and especially the outcome wording do not directly match, the claim cannot be directly verified from the source text…”Figure S14:Error example from a factual precision task, where the LLM judge fails to recognize contextual equivalence\. Although the source text supports a lack of antibiotic benefit for closely related conditions \(e\.g\., common cold, acute purulent rhinitis\) and symptom persistence outcomes, the LLM judge incorrectly rejects the fact due to mismatches in terminology \(“acute rhinosinusitis”\) and outcome phrasing \(“longer\-term symptom duration”\), overlooking clinically equivalent concepts\.Error \#2: Different interpretation of the fact and the source text\.

Another common source of error arises from differing interpretations of the atomic fact and the source text\. Because both the fact and the source text can be interpreted in multiple ways, it can be debatable whether the source text sufficiently supports the fact, making such judgments subjective and challenging\[[73](https://arxiv.org/html/2606.11337#bib.bib38)\]\. As shown in Figure[S15](https://arxiv.org/html/2606.11337#A5.F15), the source text indicates that higher intensity or doses of rehabilitation improve outcomes in specific phases \(e\.g\., subacute, chronic\), but does not explicitly frame this as an “adjunct” intervention or generalize to all people after stroke\.

Expert annotator A interprets the fact in light of its hedged phrasing \(“may provide”\) and labels it asSupported, justifying that the source text sufficiently supports this non\-definitive fact\. In contrast, the other expert and the LLM judge adopt a stricter interpretation, requiring an explicit statement of an “adjunct” intervention in the source text, and therefore label it asNot Supported\. Thus, this highlights how differing interpretations of the atomic fact and source text make it debatable whether the source text sufficiently supports the fact\. In such cases, the LLM judge seems to default to stricter, literal criteria forSupported, leading to errors when the evidence in the source text is implicit or when the atomic fact is hedged\. Consistent with prior work\[[50](https://arxiv.org/html/2606.11337#bib.bib97)\], such variation in interpretation can lead to disagreement—even among experts—but also systematically contributes to failure modes in LLM judge decisions\.

Atomic Fact:Additional physical rehabilitation delivered as an adjunct to usual physical rehabilitation after stroke may provide added benefits for people after stroke\.
Source Text:Physical rehabilitation effectively improves motor function, balance, and activities of daily living \(ADL\) across all stages of recovery, but the timing and intensity of interventions are critical modulators of success\. Evidence indicates that while "early" rehabilitation \(commencing \>24 hours post\-stroke\) harnesses neuroplasticity for optimal recovery, "very early" high\-intensity mobilization \(within 24 hours\) can be detrimental and reduce favorable functional outcomes\. In the subacute and chronic phases, higher doses and intensities of therapy generally yield superior motor outcomes, with some guidelines suggesting 30–60 minutes daily\.\.\.
Expert Annotator A:1\.Label:Supported2\.Excerpt from Source Text: “In the subacute and chronic phases, higher doses and intensities of therapy generally yield superior motor outcome”3\.Justification: “Source states that there is benefit for "subacute and chronic phases", not for all "people after stroke\." But since the atomic fact says "may provide" \(not exaggerates certainty\) the source text does indeed support the atomic fact\.”Expert Annotator B:1\.Label:Not Supported2\.Excerpt from Source Text: N/A3\.Justification: “No description in the source text\.”LLM Judge\(gpt\-5\.4\-mini\):1\.Label:Not Supported2\.Excerpt from Source Text: “Physical rehabilitation effectively improves motor function, balance, and activities of daily living \(ADL\) across all stages of recovery, but the timing and intensity of interventions are critical modulators of success”3\.Justification: “The source text supports that physical rehabilitation is beneficial and that some delivery models are non\-inferior to conventional care, but it does not state that \*additional\* rehabilitation given as an adjunct to usual rehabilitation after stroke provides added benefits\. The required adjunct/comparator context is missing, so the atomic fact is not directly entailed\.”Figure S15:Error example a from factual recall task, illustrating differing interpretations of the fact and source text\. The atomic fact claims that additional rehabilitation may provide added benefits when used alongside usual physical rehabilitation, while the source text discusses the general benefits of rehabilitation intensity and delivery without explicitly framing an adjunct intervention\. One expert annotator interprets the source text more broadly and labels the fact asSupported, whereas the other expert and the LLM judge do not identify a sufficient description of the atomic fact in the source text and label it asNot Supported\.Annotation Guideline for Evaluating Factual PrecisionTask Overview\.Is the inference text factually supported by the source text \(e\.g\., Cochrane article\)?You evaluate whether eachinference textis factually supported by thesource text\. The inference text contains claims that you will evaluate, and the source text \(e\.g\., Cochrane article\) serves as the authoritative source against which you compare these claims\.Inputs\.You will receive:•Inference text: Pieces of information that you need to evaluate\. These appear in the “Inference Text to Evaluate” section\.•Source text: The authoritative source document \(e\.g\., Cochrane article; accessed via URL in the “Source Text” section\) that you will use to verify the inference text\.How they work together:You read each piece of inference text and carefully & rigorously search through the source text to determine whether the source text supports, contradicts, or does not address the inference text\.Outputs\.For each inference text, you will provide:•Label:Contradicted/Supported/Not Supported— Your judgment about whether the inference text is factually supported by the source text\.•Excerpts: Minimal sentence\(s\) from the source text that justify your label\.•Short Rationale\( 10 words\): A brief explanation of why you chose that label\.Label Definitions\.1\.Contradicted\.Choose ifany of the following is true:•The inference text explicitly states an opposite, conflicting/refuting, or contradictory claim made in the source text\.•The inference text states information that is inconsistent or incompatible with the source text\.•The inference text overstates or overgeneralizes beyond the source text\.2\.Supported\.Choose ifall of the following is true:•The inference text is explicitly stated or unambiguously entailed by the source text, with no additional inference or assumptions required\.•The inference text matches the scope, certainty in language, specificity, and comparison group/time\-frame as the source text3\.Not Supported\.Choose ifany of the following is true:•The inference text is not mentioned, clearly conveyed, and/or addressed in the source text•The inference text is neither supported nor contradicted by the source text\. This includes when the source text has ambiguity on whether it supports or contradicts the inference text•The inference text cannot be verified or refuted based solely on the source text, including when the inference text requires external knowledge or reasoning beyond the source text to be verified or refutedFigure S16:Annotation Guideline for evaluating factual precision in generated conclusions\.Annotation Guideline for Evaluating Factual RecallTask Overview\.Does the source text support the inference text?You evaluate whether thesource textsupports theinference text\. In other words, you will search through the source text to determine if it unambiguously contains or explicitly entails the same information as the inference text\.Inputs\.You will receive:•Inference text: Pieces of information that you need to evaluate\. These appear in the “Inference Text to Evaluate” section\.•Source text: The document \(shown in the “Source Text” section\) that you will search through to see if the source text supports the inference text\.How they work together:You read each inference text and search through the source text to determine whether the source text supports the inference text\.Outputs\.For each inference text, you will provide:•Label:Supported/Not Supported— Your judgment about whether the inference text is factually supported by the source text\.•Excerpts: Minimal sentence\(s\) from the source text that justify your label\.•Short Rationale\( 10 words\): A brief explanation of why you chose that label\.Label Definitions\.1\.Supported\.Choose ifall of the following is true:•The inference text is explicitly stated or unambiguously entailed by the source text, with no additional inference or assumptions required\.•The inference text matches the scope, certainty in language, specificity, and comparison group/time\-frame as the source text2\.Not Supported\.Choose ifany of the following is true:•Absent: The source text does not contain the fact stated in the inference text\.•Too vague / underspecified:The source text is too vague/underspecified to determine support against the fact stated in the inference text\.•Different claim:The source text discusses a related or different claim but does not state or convey the same fact as in the inference text\.•Overgeneralized:The source text overgeneralizes and makes a broader claim than the fact stated in the inference text \(e\.g\. broader PICO/effect/certainty than the fact\)\. For example, inference text indicates X affects Y in context Z, but the source text states X affects Y in context Z AND W, overgeneralizing how X effects•Contradicted or mixed:The source text contradicts or conflicts with the fact stated in the inference text \(including partial or full conflict\)\.Figure S17:Annotation Guideline for evaluating factual recall in generated conclusions\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/atomic-fact-generation-interface/afg_guidelines_instruction.png)\(a\)Task Overview Page
![Refer to caption](https://arxiv.org/html/2606.11337v1/)\(b\)Definition and Examples of Completeness
![Refer to caption](https://arxiv.org/html/2606.11337v1/x7.png)\(c\)Definition and Examples of Comprehensiveness
![Refer to caption](https://arxiv.org/html/2606.11337v1/x8.png)\(d\)Annotation Interface

Figure S18:Annotation interface for evaluating atomic facts\. Panels show the task introduction and guidelines: \(a\) task overview page, \(b\) definition and examples of completeness, \(c\) definition and examples of comprehensiveness, and \(d\) annotation interface\. Panels are cropped or resized for space\.![Refer to caption](https://arxiv.org/html/2606.11337v1/x9.png)\(a\)Annotation Interface for Factual Coverage
![Refer to caption](https://arxiv.org/html/2606.11337v1/x10.png)\(b\)Task Description for Factual Coverage
![Refer to caption](https://arxiv.org/html/2606.11337v1/x11.png)\(c\)Definition of “SUPPORTED” class for Factual Coverage
![Refer to caption](https://arxiv.org/html/2606.11337v1/x12.png)\(d\)Definition of “NOT SUPPORTED” class for Factual Coverage

Figure S19:Annotation interface for evaluating the factual coverage of scientific conclusions\. Panels show \(a\) annotation interface for factual coverage, \(b\) factual coverage description, \(c\) “Supported” class definition and examples, and \(d\) “Not Supported” class definition and examples\. Panels are cropped or resized for space\. Note that factual coverage corresponds to our factual recall task\.![Refer to caption](https://arxiv.org/html/2606.11337v1/x13.png)\(a\)Task Description for Factual Correctness
![Refer to caption](https://arxiv.org/html/2606.11337v1/x14.png)\(b\)Definition for “CONTRADICTED” category
![Refer to caption](https://arxiv.org/html/2606.11337v1/x15.png)\(c\)Definition for “SUPPORTED” category
![Refer to caption](https://arxiv.org/html/2606.11337v1/x16.png)\(d\)Definition for “NOT SUPPORTED” category

Figure S20:Annotation interface for evaluating the factual correctness of scientific conclusions\. Panels show \(a\) factual correctness description, \(b\) “CONTRADICTED” class definition and examples, \(c\) “Supported” class definition and examples, and \(d\) “Not Supported” class definition and examples\. Panels are cropped or resized for space\. Note that factual correctness corresponds to our factual precision task\. See Figure[S21](https://arxiv.org/html/2606.11337#A5.F21)for the annotation interface for the factual correctness task\.![Refer to caption](https://arxiv.org/html/2606.11337v1/x17.png)Figure S21:The annotation interface for evaluating factual correctness\.LLM Judge Prompt for Factual Precision EvaluationSystem Prompt:You are an expert evaluator with deep expertise in evidence\-based medicine and clinical research\.
Instruction:Please carefully evaluate whether the provided ATOMIC FACT is factually SUPPORTED, CONTRADICTED, or NOT SUPPORTED by the SOURCE TEXT\. Follow the evaluation criteria, tips, and examples below carefully and think step\-by\-step before answering\.\#\# Inputs
\- SOURCE TEXT: An authoritative source document \(texts from source, web\-retrieved document\) that you will search through to verify the ATOMIC FACT\.\- ATOMIC FACT: A piece of information that you need to evaluate against the SOURCE TEXT\#\# Outputs
Output should be in JSON format in the following fields:\- LABEL \(string\): CONTRADICTED / SUPPORTED / NOT SUPPORTED\- EXCERPTS \(list of strings\): Minimal sentence\(s\) from the SOURCE TEXT that justify your label\.\- JUSTIFICATION \(string\): A detailed justification regarding why you chose the labelBelow, we define the decision criteria for the labels…\#\#\# CONTRADICTEDThe ATOMIC FACT is CONTRADICTED by the SOURCE TEXT if \*\*ANY\*\* of the following is true:\- The ATOMIC FACT states opposite, conflicting, or incompatible info with the SOURCE TEXT\.\- The ATOMIC FACT makes a directional claim, but the SOURCE TEXT reports otherwise…\#\#\# SUPPORTEDThe ATOMIC FACT is SUPPORTED only if it is not CONTRADICTED & ALL hold:\- From the SOURCE TEXT, one can deduce the ATOMIC FACT with no additional inference or assumptions required OR the SOURCE TEXT explicitly contains or directly entails the ATOMIC FACT\.\- SOURCE TEXT matches or has a broader scope, PICO, or specificity than the ATOMIC FACT\.…\#\#\# NOT SUPPORTEDThe ATOMIC FACT is NOT SUPPORTED if ANY of the following is true:\- Absent: The SOURCE TEXT does not contain the ATOMIC FACT\- The SOURCE TEXT is too vague/underspecified…\- Neither SUPPORTED nor CONTRADICTED……Below, we present six examples of the evaluation tasks…\#\#\# EXAMPLE 1:
\- SOURCE TEXT: …\- ATOMIC FACT: …OUTPUT:\- LABEL: …\- EXCERPT: …\- JUSTIFICATION: … …Now, based on what you learned, evaluate whether the input ATOMIC FACT is factually SUPPORTED, CONTRADICTED, or NOT SUPPORTED… Carefully thinking step\-by\-step about your answer…\#\# Task
SOURCE TEXT:\{ground\_truth\_text\}ATOMIC FACT:\{llm\_fact\}Figure S22:Few\-shot prompt used by LLM judge for factual precision\. We usegpt\-5\.4\-mini\(no reasoning, temperature 0\.2\)\. The prompt was shortened to fit within the page\. Full prompt available in supplementary material / code\.LLM Judge Prompt for Factual Recall EvaluationSystem Prompt:You are an expert evaluator with deep expertise in evidence\-based medicine and clinical research\.
Instruction:Please carefully evaluate whether the provided ATOMIC FACT is present in, or directly supported by, the SOURCE TEXT\. Read the evaluation criteria, tips, and examples below carefully to think through your answer and make your judgement\.\#\# Inputs
\- SOURCE TEXT: The paragraph document of information that you will search through to see if the ATOMIC FACT is present in, or directly supported by this SOURCE TEXT\- ATOMIC FACT: A piece of information that you need to evaluate against the SOURCE TEXT\#\# Outputs
Output should be in JSON format in the following fields:\- LABEL \(string\):Supported/NOT SUPPORTED\- Your judgment about whether the ATOMIC FACT present in, or directly supported by, the SOURCE TEXT\.\- EXCERPTS \(list of strings\): Minimal sentence\(s\) from the SOURCE TEXT that justify your label\.\- JUSTIFICATION \(string\): A brief explanation of why you chose the label, based on the excerpt\.Below, we define the decision criteria for choosing the labels SUPPORTED or NOT SUPPORTED\.\#\#\# SUPPORTEDThe ATOMIC FACT is SUPPORTED by the SOURCE TEXT if ALL of the following is true:\- From the SOURCE TEXT, one can deduce the ATOMIC FACT with no additional inference or assumptions required OR the SOURCE TEXT explicitly contains or directly entails the ATOMIC FACT\.\- SOURCE TEXT matches or has a broader scope, PICO, or specificity than the ATOMIC FACT\.\- Both SOURCE TEXT and ATOMIC FACT match the certainty in language\.\- If stated / contained in the ATOMIC FACT: Match the comparison group \(e\.g\., improves outcome compared to placebo\), or time frame \(e\.g\., within 24 hours\) of an intervention\.\#\#\# NOT SUPPORTEDThe ATOMIC FACT is NOT SUPPORTED by the SOURCE TEXT if ANY of the following is true:\- Absent: The SOURCE TEXT does not contain the ATOMIC FACT\- Too vague or underspecified: The SOURCE TEXT is too vague/underspecified to determine support\.\- Different claim: The SOURCE TEXT discusses a related claim but does not convey the FACT\.\- Source text does not fully entail the ATOMIC FACT\.\- Contradicted or mixed: The SOURCE TEXT contradicts or conflicts with the ATOMIC FACT \(including partial or full conflict\)\.\- If stated / contained in the ATOMIC FACT: Mismatch or missing comparison group, treatment effect / outcome and their strengths \(e\.g\., significant effect\) or time frame in the SOURCE TEXT\.\#\# Important Evaluation Tips
\- Be rigorous and precise\.\- Focus on meaning and context, not exact wording\.\- Interpret scope and qualifiers carefully, especially in ATOMIC FACTS\.\- Consider hierarchical relationships between terms\.Now, based on what you learned from the evaluation criteria and tips, evaluate whether the input ATOMIC FACT is present in, or directly supported by, the SOURCE TEXT\. Justify and carefully thinking step\-by\-step about your answer\.\#\# Task
SOURCE TEXT:\{llm\_response\_text\}ATOMIC FACT:\{article\_facts\_text\}Figure S23:Zero\-shot prompt used by LLM judge for factual recall\. We usegpt\-5\.4\-mini\(no reasoning, temperature 1\)\. The prompt was shortened to fit within the page\. Full prompt available in supplementary material / code\.Table S10:Full evaluation performance ofgpt\-5\.4\-mini,claude\-haiku\-4\.5, andgemini\-3\-flashon the factual precision labeling task \(three classes:Supported,Contradicted, andNot Supported\) under zero\-shot and few\-shot prompting, across different reasoning settings and temperatures\. We evaluate performance onN=123N=123expert\-labeled examples, excluding six few\-shot examples used in the prompt to mitigate evaluation leakage\. The best performance is marked inbold\.ModelReasoningTemp\.PrecisionRecallF1Accuracygpt\-5\.4\-miniZero\-shot PromptNone00\.8010\.7160\.7320\.7560\.20\.8300\.7150\.7370\.75610\.8460\.7410\.7620\.772Low00\.7190\.6980\.6620\.6830\.20\.7460\.6790\.6460\.65910\.6940\.6290\.5730\.594Medium00\.7280\.6850\.6420\.6670\.20\.7290\.6800\.6310\.64210\.7060\.6610\.6190\.634High00\.7010\.6800\.6430\.6590\.20\.6900\.6540\.6180\.64210\.6540\.6410\.6040\.626Few\-shot PromptNone00\.8400\.7150\.7430\.7890\.20\.8820\.8130\.8370\.83010\.8610\.7000\.7280\.781Low00\.8080\.7500\.7430\.7480\.20\.7170\.6720\.6510\.66710\.7360\.6720\.6600\.667Medium00\.7560\.7050\.6800\.6910\.20\.7530\.6720\.6500\.66710\.7200\.6800\.6510\.659High00\.7100\.6660\.6350\.6590\.20\.7500\.7300\.7060\.72410\.7200\.6930\.6630\.675claude\-haiku\-4\.5Zero\-shot PromptNone00\.6880\.6740\.6120\.6340\.20\.7240\.7180\.6630\.69110\.7000\.6930\.6320\.659Ext\. Thinking00\.7060\.6860\.6270\.6500\.20\.6520\.6480\.5890\.61810\.7160\.6860\.6390\.667Few\-shot PromptNone00\.6860\.7130\.6610\.6830\.20\.6710\.7000\.6450\.66710\.7090\.7180\.6760\.691Ext\. Thinking00\.6830\.6870\.6320\.6500\.20\.7070\.6860\.6450\.66710\.7020\.6860\.6420\.667gemini\-3\-flashZero\-shot PromptMinimal00\.7340\.7510\.7140\.7320\.20\.7440\.7700\.7260\.74010\.7460\.7830\.7400\.756Low00\.7710\.7760\.7530\.7640\.20\.7710\.7760\.7530\.76410\.7670\.7700\.7460\.756Medium00\.6190\.3860\.3150\.4720\.20\.6940\.4370\.3950\.52010\.7210\.4700\.4430\.545High00\.4850\.3590\.2560\.4720\.20\.6520\.3790\.3010\.48010\.8220\.3910\.3180\.496Few\-shot PromptMinimal00\.7300\.7570\.7300\.7560\.20\.7140\.7450\.7130\.74010\.7220\.7570\.7220\.756Low00\.7520\.7830\.7600\.7890\.20\.7430\.7760\.7510\.78110\.7680\.8020\.7770\.797Medium00\.7150\.7460\.6860\.7070\.20\.6720\.7150\.6500\.66710\.6780\.7140\.6650\.683High00\.6680\.6680\.6280\.6420\.20\.7040\.7590\.6960\.70710\.6890\.7270\.6640\.683

Table S11:Full evaluation performance ofgpt\-5\.4\-mini,claude\-haiku\-4\.5, andgemini\-3\-flashon the factual recall labeling task \(two classes:Supported,Not Supported\) under zero\-shot and few\-shot prompting, across different reasoning settings and temperatures\. We evaluate performance onN=113N=113expert\-labeled examples, excluding six few\-shot examples used in the prompt to mitigate evaluation leakage\. The best performance is marked inbold\.ModelReasoningTemp\.PrecisionRecallF1Accuracygpt\-5\.4\-miniZero\-shot PromptNone00\.8950\.7560\.8190\.8670\.20\.8750\.7780\.8240\.86710\.9470\.8000\.8680\.903Low00\.9620\.5560\.7040\.8140\.20\.9260\.5560\.6940\.80510\.9290\.5780\.7120\.814Medium00\.9640\.6000\.7400\.8320\.20\.9630\.5780\.7220\.82310\.9570\.4890\.6470\.788High00\.9260\.5560\.6940\.8050\.20\.9260\.5560\.6940\.80510\.9620\.5560\.7040\.814Few\-shot PromptNone00\.7870\.8220\.8040\.8410\.20\.7920\.8440\.8170\.85010\.7920\.8440\.8170\.850Low00\.9000\.6000\.7200\.8140\.20\.9060\.6440\.7530\.83210\.9660\.6220\.7570\.841Medium00\.9000\.6000\.7200\.8140\.20\.9310\.6000\.7300\.82310\.9290\.5780\.7120\.814High00\.9090\.6670\.7690\.8410\.20\.9660\.6220\.7570\.84110\.9640\.6000\.7400\.832claude\-haiku\-4\.5Zero\-shot PromptNone00\.8180\.6000\.6920\.7880\.20\.8000\.6220\.7000\.78810\.7650\.5780\.6580\.761Ext\. Thinking00\.9600\.5330\.6860\.8050\.20\.9230\.5330\.6760\.79710\.92590\.5560\.6940\.805Few\-shot PromptNone00\.9200\.5110\.6570\.7880\.20\.8570\.5330\.6580\.77910\.9090\.4400\.5970\.761Ext\. Thinking00\.8890\.5330\.6670\.7880\.20\.9170\.4890\.6380\.77910\.9090\.4440\.5970\.761gemini\-3\-flashZero\-shot PromptMinimal00\.8180\.8000\.8090\.8500\.20\.8180\.8000\.8090\.85010\.8370\.8000\.8180\.858Low00\.8220\.8220\.8220\.8580\.20\.8410\.8220\.8320\.86710\.8040\.8220\.8130\.850Medium00\.9470\.4000\.5630\.7520\.21\.0000\.4890\.6570\.79711\.0000\.5110\.6770\.805High00\.9550\.4670\.6270\.7790\.20\.9550\.4440\.6060\.77011\.0000\.5780\.7320\.832Few\-shot PromptMinimal00\.8330\.7780\.8050\.8500\.20\.8330\.7780\.8050\.85010\.8330\.7780\.8050\.850Low00\.8440\.8440\.8440\.8760\.20\.8440\.8440\.8440\.87610\.8610\.8220\.8410\.876Medium01\.0000\.4000\.5710\.7610\.21\.0000\.3110\.4750\.72610\.9640\.6000\.7400\.832High01\.0000\.3560\.5250\.7430\.21\.0000\.4220\.5940\.77010\.9230\.5330\.6760\.797

## Appendix FFull Evaluation Details

In this section, we describe the selected hyperparameters \(§[F\.1](https://arxiv.org/html/2606.11337#A6.SS1)\), system prompts and output preprocessing \(§[F\.2](https://arxiv.org/html/2606.11337#A6.SS2)\), and theclean\-roomevaluation protocols for closed\-source agents \(§[F\.3](https://arxiv.org/html/2606.11337#A6.SS3)\)\.

### F\.1Selected Hyperparameters

In our evaluation, we configure each model and deep research agent to its default recommended hyperparameter settings and, when available, set the reasoning level to the highest setting, with a small number of targeted adjustments\. For Anthropic’sclaude\-sonnet\-4\.5, to stay within a reasonable cost, we enableextendedthinking with a budget of 4,096 tokens for reasoning and tool use, and allow up to 8,192 tokens per turn\. For Perplexity modelssonar\-deep\-researchandsonar\-reasoning\-pro, we configure retrieval to support controlled, high\-quality synthesis by enablingsearch\_mode=academic,web\_search\_context\_size=high,reasoning\_effort=high, andsearch\_type=auto\.

### F\.2System Prompts & Output Processing

ForDR Tulu, we adapt the original system prompt\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\]to align with our benchmark task\. However,DR Tuluoften struggle with instruction\-following, often failing to produce a paragraph\-long conclusion within the required \\boxed\{\} format\. To address this, we post\-process its structured outputs by extracting sections under relevant markdown headers \(e\.g\., “Conclusion,” “Summary,” “Bottom line”\) and use the corresponding text as the synthesized conclusion\.

For other models and agents, we use the same system prompt described in Figure[S25](https://arxiv.org/html/2606.11337#A6.F25)\. For base settings \(i\.e\., models without tool\-use integration inSciConHarness\), we append an additional instruction to the system prompt indicating that tool calls are unavailable, preventing unnecessary tool\-calling behavior, as some models otherwise attempt to invoke tools even when none are provided\.

### F\.3Clean\-room Evaluation Protocols for Closed\-Sourced Agents

Some provider\-hosted agents do not support direct integration withSciConHarness, requiring alternative strategies to enforce ourclean\-roomevaluation protocol\. In particular, \(1\) Perplexity’ssonar\-deep\-researchandsonar\-reasoning\-prorely on provider\-native search APIs and do not support custom MCP tool calling, preventing direct integration ofSciConHarness, and \(2\) OpenAI Deep Research agents operate within a provider\-controlled execution environment, where custom tools must be supplied via remote MCP server endpoints rather than direct integration withSciConHarness\. Below, we describe how we adapt clean\-room protocols under these constraints\.

#### F\.3\.1Clean\-room for Perplexity

Perplexity models such assonar\-deep\-researchandsonar\-reasoning\-prorely on native web search and do not support custom MCP tool calling, making direct integration withSciConHarnessinfeasible\. Instead, we enforce ourclean\-roomprotocol in best effort using provider\-side search filters\. Specifically, we applysearch\_before\_date\_filterto restrict all web search outcomes to those published before the ground\-truthCDSRreview publication date, andsearch\_domain\_filter\(a denylist of up to 20 URLs\) to exclude web search results that leak the ground\-truth artifacts \(e\.g\.,CDSRreview\)\.

Forsonar\-reasoning\-pro, we iteratively expand thesearch\_domain\_filterlist by repeating the same query until no new leakage from the search results is detected \(using the same filtering heuristic inSciConHarness’sclean\-roomevaluation protocol\) or the 20\-URL cap is reached\. Forsonar\-deep\-research, iterative expansion via repeated querying is cost\-prohibitive, so we instead create a fixedsearch\_domain\_filterlist per query\. Specifically, for each query, we identify the top 18 most frequently filtered URLs across prior evaluations ofclaude\-sonnet\-4\.5,gemini\-3\-pro, andgpt\-5\.1on that same query, clean and format URLs, deduplicate, and combine them with two defaultCDSRdomains \(cochrane\.org,cochranelibrary\.com\)\. To avoid over\-filtering, entries insearch\_domain\_filterare handled at the URL \(article\) level rather than blocking entire domains \(e\.g\., PubMed/PMC, a major repository of scientific papers\), preserving retrieval utility while preventing access to known leakage URLs\.

#### F\.3\.2Clean\-room for OpenAI Deep Research\.

OpenAI’s deep research agents, such aso3\-deep\-researchando4\-mini\-deep\-research, run autonomously in the background within a provider\-controlled execution environment, requiring any custom tools to be supplied via remote MCP server endpoints\[[81](https://arxiv.org/html/2606.11337#bib.bib44)\]\. To enableclean\-roomevaluation under these constraints, we implement remote MCP servers111111[https://developers\.openai\.com/api/docs/guides/tools\-connectors\-mcp](https://developers.openai.com/api/docs/guides/tools-connectors-mcp)with endpoints that exposeSciConHarness’s search and browsing tools with integratedclean\-roomfiltering\. This allows controlled open\-web access while preserving the agent’s native workflow\.

Under OpenAI’s Deep Research interface requirements for remote MCP servers, each remote MCP server must expose two primitives \(search,fetch\)\. To provide coverage over our full tool suite \(e\.g\.,google\_search,paper\_search,web\_browse\) while respecting thesearchandfetchprimitives, we implement two HTTP\-based remote MCP servers: \(1\)Serper\+Jinaforgoogle\_searchandweb\_browsetools and \(2\)SemanticScholar\+Jinaforpaper\_searchandweb\_browsetools\. Both remote MCP servers impose the sameclean\-roomfiltering protocol across the tools as done inSciConHarness\. These remote MCP servers are hosted behind an nginx gateway121212[https://nginx\.org/](https://nginx.org/)and exposed via ngrok131313[https://ngrok\.com/](https://ngrok.com/)for external API access, with bearer\-token authentication and standardized tool schemas for OpenAI remote MCP compatibility\.

Before eachSciConBenchquery, the client configuresclean\-roomfilters \(e\.g\., ground\-truthCDSRtitle and publication date\) on both remote MCP servers via POST requests and verifies the configuration via GET requests, retrying on mismatch\. During inference, Deep Research agents issue tool calls to the remote MCP endpoints, which enforceclean\-roomfiltering across the tools and return only compliant results\. Inference proceeds asynchronously with polling\. All runs produce structured logs capturing filter decisions, excluded URLs, and tool usage\. Filtering can also be disabled via the configuration endpoint, enabling evaluation of OpenAI Deep Research agents withoutclean\-roomconstraints\.

Summarization PromptSystem Prompt:You are a helpful assistant that creates summaries of web content focusing on main details\.
Instruction:Summarize the following web content, focusing only on the main details and key information\. Preserve important facts, numbers, dates, and conclusions\. Aim to filter out any noisy characters \(e\.g\., HTML tags, social media links, random strings, etc\.\) and outputting only important information\. Specific details, including but not limited to metrics, deltas, definitions, settings, limitations, and citations and references should be preserved\. Make sure not to lose any key information\.Content to summarize:\{content\}Figure S24:Prompt used to summarize long page text returned by theweb\_browsetool\. We usegpt\-5\-miniwith default settings \(e\.g\., medium verbosity and reasoning effort\)\.System Prompt Used for EvaluationYou are a research assistant who answers scientific questions by identifying relevant sources, assessing their evidence quality and certainty, and synthesizing the evidence into evidence\-backed conclusions\.Task Requirements:\- Synthesize a comprehensive, paragraph\-long conclusion that directly answers the question\. The conclusion must be clear, well\-supported, and WRAPPED with THREE SQUARE BRACKETS\. While you may generate additional content beyond the conclusion, the conclusion must be the main focus\.\- Focus on synthesizing the overall body of evidence \(e\.g\., highlighting relationships across sources, identifying contradictions, etc\) to form a coherent conclusion rather than just enumerating information\. Weigh the synthesis more heavily toward higher\-quality evidence when formulating the conclusion\.\- In your conclusion, explicitly describe both strengths and limitations of the evidence quality \(e\.g\., risk of bias, imprecision, inconsistency\), including uncertainty, gaps, or conflicts across sources\. Explicitly state when evidence is limited, low quality, or inconsistent and explain what additional research would help resolve these gaps\.\- Only provide the final answer when ready\. If available, tool calls are permitted without any hard limits, but should be used judiciously with a clear purpose to gather sufficient information to derive a conclusion to the question\. \- Please prefer high\-quality sources as evidence \(peer\-reviewed papers, journals, sources like PubMed, etc\) and prioritize recent work for fast\-moving areas\. Do not rely on or use Cochrane reviews for this task\.\- Cite all claims from search results\. You should ground every nontrivial claim in retrieved snippets and sources, if available\. Please include the sources cited in the form of references at the end of the answer\.\- Most importantly, DO NOT invent snippets or citations and never fabricate content\.Synthesize the conclusion, with the text being at most a paragraph\-long\. MAKE SURE to enclose the entire paragraph within exactly three square brackets on each side, like this: \[\[\[Enter your conclusion here\]\]\]\. Do not include any additional formatting outside the triple brackets\.IMPORTANT: Tool calls are NOT available\. Do NOT attempt to make any function calls or tool calls\. Answer the question directly using only your knowledge and reasoning\.Figure S25:System prompt for all models evaluated onSciConBench\. Theorange textis appended for base models without tool access\.System Prompt for DR Tulu EvaluationsYou are a research assistant who answers scientific questions through iterative reasoning and research\. You identify relevant sources, assess their evidence quality and certainty, and synthesize the evidence into evidence\-backed conclusions\.\#\# Process\- Use <think\></think\> tags to show your reasoning at any point\.\- Use <call\_tool name="…"\>query</call\_tool\> when you need information \(see tools below\)\.\- You can alternate between thinking and searching multiple times\.\- Only provide <answer\></answer\> tags when you have enough information for a complete response\. Within the <answer\> tags, you MUST synthesize a comprehensive, paragraph\-long conclusion that directly answers the question\. \*\*REQUIRED FORMAT\*\*: The synthesized conclusion \(at most a paragraph long\) MUST be placed in \\boxed\{\} format: \\boxed\{your synthesized conclusion here\}\. This is mandatory \- your answer must include \\\{\} with your conclusion\.…\#\# Calling Tools \(<call\_tool name="…"\>query</call\_tool\>\)\- You can use the following tools:1\. google\_search…2\. browse\_webpage…3\. snippet\_search…\#\# Tool Output\- After you issue a tool call, we will execute it and return results wrapped in <tool\_output\> tags\.…\#\# Requirements\- Focus on synthesizing the overall body of evidence \(e\.g\., highlighting relationships across sources, identifying contradictions, etc\) to form a coherent conclusion rather than just enumerating information\. Ideally, you should synthesize rather than enumerate content: it’s helpful to group findings across papers, explain relationships, and build a coherent narrative that answers the question, supported by citations\. Weigh the synthesis more heavily toward higher\-quality evidence when formulating the conclusion\.\- In your conclusion, explicitly describe both strengths and limitations of the evidence quality \(e\.g\., risk of bias, imprecision, inconsistency\), including uncertainty, gaps, or conflicts across sources\. Explicitly state when evidence is limited, low quality, or inconsistent and explain what additional research would help resolve these gaps\. You should acknowledge uncertainty and conflicts; if evidence is thin or sources disagree, state it and explain what additional evidence would resolve it\.\- Please prefer high\-quality sources as evidence \(peer\-reviewed papers, journals, sources like PubMed, etc\)\. Please prefer authoritative sources \(peer\-reviewed papers, reputable benchmarks/docs\) and prioritize recent work for fast\-moving areas\. Do not simply focus on Cochrane reviews for this task\.\- Most importantly, DO NOT invent snippets or citations and never fabricate content\.\- Tool calls are permitted without any hard limits, but should be used judiciously with a clear purpose to gather sufficient information to derive a conclusion to the question\.\#\# Answer and Citation Format\- Once you collect all of the necessary information, generate the final answer, and mark your answer with answer tags: <answer\></answer\>\.\- \*\*CRITICAL REQUIREMENT\*\*: Within the <answer\></answer\> tags, you MUST place your synthesized conclusion paragraph in \\boxed\{\} format……IMPORTANT:Before you return your final answer, you should ensure that your final answer contains a \\boxed\{\} tag, which contains a paragraph\-long synthesized conclusion\. Make sure it’s concise and to the point\.Figure S26:System prompt used forDR Tuluevaluations onSciConBench, adapted from the original system prompt in\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\]\. The system prompt shown is shortened to fit within a page and omits components already specified in the original system prompt\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\]; see the original for full details\.

## Appendix GPower Analysis

Here, we compute the minimum detectable effect \(MDE\) given our evaluation sample size ofN=268N=268\. In our case, the MDE is the smallest change in our metrics \(e\.g\., factual precision, recall\) that our benchmark would be able to detect with high probability \(80%\)\.

Preliminaries\.Let there bennpaired observations\(Xt,Yt\)\(X\_\{t\},Y\_\{t\}\)on the same items, e\.g\., two models evaluated on the samennitems\. For each pairt=1,…,nt=1,\.\.\.,n, let the difference between outcomes beDt=Xt−YtD\_\{t\}=X\_\{t\}\-Y\_\{t\}\.

Let the sample mean and SD of difference be:D¯=1n​∑t=1nDt\\bar\{D\}=\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}D\_\{t\}andsD=1n−1​∑t=1n\(Dt−D¯\)2\.s\_\{D\}=\\sqrt\{\\frac\{1\}\{n\-1\}\\sum\_\{t=1\}^\{n\}\(D\_\{t\}\-\\bar\{D\}\)^\{2\}\}\.We testH0:μD=0H\_\{0\}:\\mu\_\{D\}=0vs\.H1:μD≠0H\_\{1\}:\\mu\_\{D\}\\neq 0usingt=D¯sD/nt=\\frac\{\\bar\{D\}\}\{s\_\{D\}/\\sqrt\{n\}\}, which follows att\-distribution withn−1n\-1degrees of freedom underH0H\_\{0\}\. We rejectH0H\_\{0\}if\|t\|\>tn−1,1−α/2\|t\|\>t\_\{n\-1,1\-\\alpha/2\}for a two sided test\.

Power analysis\.First, we assume thattn−1,x≈zx,∀xt\_\{n\-1,x\}\\approx z\_\{x\},\\forall xfor large enoughnn\. Since our sample size isN=268N=268, we compute the MDE represented byΔ\\Deltawith significanceα\\alphaand power1−β1\-\\beta:

Δ=\(z1−α/2\+z1−β\)2​σD2N\.\\Delta=\\sqrt\{\\frac\{\(z\_\{1\-\\alpha/2\}\+z\_\{1\-\\beta\}\)^\{2\}\\sigma\_\{D\}^\{2\}\}\{N\}\}\.whereσD2=Var​\(Dt\)\\sigma\_\{D\}^\{2\}=\\mathrm\{Var\}\(D\_\{t\}\)\.

To estimate the population varianceσD2\\sigma\_\{D\}^\{2\}, we use the observed variance from a pilot study as a proxy\. Specifically, we evaluateclaude\-sonnet\-4\.5andgpt\-5\.1onN=100N=100SciConBenchsamples usingSciConHarnessunder theclean\-roomprotocol, compute factual F1\-scores via atomic fact decomposition, and estimate the variance \(σD,p​i​l​o​t2=0\.0457\\sigma\_\{D,pilot\}^\{2\}=0\.0457\) of their paired differences\.

Assuming power1−β=0\.81\-\\beta=0\.8and significance levelα=0\.05\\alpha=0\.05\(withz1−α/2=1\.96z\_\{1\-\\alpha/2\}=1\.96andz1−β=0\.8416z\_\{1\-\\beta\}=0\.8416\), and usingσD,pilot2\\sigma\_\{D,\\text\{pilot\}\}^\{2\}as an estimate ofσD2\\sigma\_\{D\}^\{2\}, the MDE for our benchmark evaluation sample \(N=268N=268\) is:

Δ≈\(1\.96\+0\.8416\)2⋅0\.0457268≈0\.037\.\\Delta\\approx\\sqrt\{\\frac\{\(1\.96\+0\.8416\)^\{2\}\\cdot 0\.0457\}\{268\}\}\\approx 0\.037\.
Thus, withN=268N=268, our evaluation is powered to detect differences in factual F1\-scores between models of at leastΔ=0\.037\\Delta=0\.037atα=0\.05\\alpha=0\.05and power0\.80\.8\.

## Appendix HCost Analysis

Model Querying Costs\.Table[S12](https://arxiv.org/html/2606.11337#A8.T12)reports the cost breakdown for querying models and deep research agents across differentSciConHarnesssettings\. To estimate inference costs for proprietary systems, we use the billed costs reported in their API consoles\. For open\-sourced systems \(e\.g\.,DR Tulu\), we follow Shao et al\.\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\]and estimate the inference cost using OpenRouter with Qwen3\-8B pricing \($0\.2 per 1M input/output tokens\)\.141414https://openrouter\.ai/qwen/qwen3\-8bUnderSciConHarnesstools\(no clean\-room\),DR Tuluuses an average of 9,940 input and 3,467 output tokens per query, corresponding to $0\.0027 per query; withSciConHarnesstools \+ clean\-room, it uses an average of 10,731 input and 3,362 output tokens per query, corresponding to $0\.0028 per query\. To estimate tool costs, we track the tool usage inSciConHarnessacross the evaluated systems and compute the tool cost, following Shao et al\.\[[100](https://arxiv.org/html/2606.11337#bib.bib30)\], in whichpaper\_searchis free to use due to the Semantic Scholar API,web\_browseis estimated to be $0\.00005 per use andgoogle\_searchis estimated to be USD $0\.00075 per use\. We sum per\-query inference and tool costs to obtain the total cost per query, and compute the overall cost by multiplying by the number of evaluated queries \(N=268N=268\)\. An exception issonar\-deep\-research, which is evaluated on a random subset \(N=100N=100\) due to its high per\-query cost \($2\.22–$2\.314\)\. The total querying cost across all systems is $1,569\.946\.

Table S12:Cost breakdown for querying models and deep research agents \(denoted asDR\) across differentSciConHarnesssettings:Base\(no tools\),SciConHarnesstools, andSciConHarnesstools \+ clean\-room\. For each system, we indicate inference cost \(e\.g\., token usage\), average number of tool calls per query \(e\.g\.,web\_browse,google\_search,paper\_search\), corresponding tool costs per query, and total cost per query \(inference cost \+ tool cost\)\. Total cost is aggregated across all evaluated queries \(N=268N=268\) for the system\. “\-” denotes this information was not applicable, while†\\daggerindicates thatsonar\-deep\-researchis evaluated on a random subset \(N=100N=100out of 268\) due to high inference cost per query \($2\.22\-$2\.314 per query\)\. All costs are in USD\. More details on the specific cost estimations are available in the §[H](https://arxiv.org/html/2606.11337#A8)\.Inference Costweb\_browsegoogle\_searchpaper\_searchTool CostTotal / QueryTotal\($ / Query\)\(\# / Query\)\(\# / Query\)\(\# / Query\)\($ / Query\)\($ / Query\)Cost \($\)Base Models \(No Tools\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.10\.046\-\-\-00\.04612\.328![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.50\.032\-\-\-00\.0328\.576![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro0\.030\-\-\-00\.0308\.040Subtotal28\.944Models \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.10\.13588\.385\.150\.520\.00430\.140137\.552![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.50\.82501\.470\.696\.810\.00060\.8256221\.261![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro0\.1040\.130\.824\.650\.00060\.104628\.033![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro0\.0143––––0\.01433\.832Subtotal290\.678Models \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.10\.1257\.914\.650\.550\.0040\.12934\.570![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.50\.9081\.551\.027\.230\.00080\.9088243\.558![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro0\.1180\.180\.904\.550\.00070\.118731\.812![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro0\.013––––0\.01253\.484Subtotal313\.424DR \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu0\.00280\.020\.215\.480\.00020\.0030\.804![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger2\.314––––2\.314231\.400![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research0\.12156\.787\.572\.690\.0060\.127534\.17![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research0\.79136\.543\.781\.100\.00320\.7945212\.926Subtotal479\.3DR \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu0\.00270\.020\.345\.090\.00030\.0030\.804![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger2\.220––––2\.220222\.000![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research0\.0926\.719\.082\.870\.00710\.099126\.559![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research0\.7737\.855\.231\.840\.0040\.777208\.236Subtotal457\.60Total \($\)1,569\.946

Atomic Fact Generation Costs\.Table[S13](https://arxiv.org/html/2606.11337#A8.T13)reports the cost breakdown for atomic fact generation across models and deep research agents under differentSciConHarnesssettings\. We calculate the costs using billed API usage from provider consoles\. Models such asgpt\-5\.1ando3\-deep\-researchgenerate longer conclusions, resulting in more atomic facts and higher token usage, which increases the cost per conclusion\. In total, atomic fact generation across all evaluated conclusions costs $1,239\.48\.

Table S13:Cost breakdown for atomic fact generation across models and deep research agents \(denoted asDR\) under differentSciConHarnesssettings\. After querying these systems and generating conclusions, we decompose the conclusions into atomic facts using our pipeline \(§[4\.1](https://arxiv.org/html/2606.11337#S4.SS1)\)\. For each system, we report the output characteristic of their generated conclusions: average number of tokens \(Avg Tokenscolumn\), sentences \(Avg Sent\.column\), and atomic facts \(Avg Factscolumn\)\. We also report the average pipeline cost per conclusion \(in API costs; seeCost / Conclusioncolumn\) and total cost aggregated across all processed conclusions \(N=268N=268\)\. As in Table[S12](https://arxiv.org/html/2606.11337#A8.T12),†\\daggernext tosonar\-deep\-researchdenotes evaluation on a subset \(N=100N=100\)\. All costs are in USD; see §[H](https://arxiv.org/html/2606.11337#A8)for details\.Avg TokensAvg Sent\.Avg FactsCost / ConclusionTotal Cost\(± STD\)\(± STD\)\(± STD\)\($ / Conclusion\)\($\)Base Models \(No Tools\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.1612\.6 ± 147\.58\.5 ± 5\.347\.4 ± 12\.70\.397728106\.5912![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.5298\.8 ± 47\.56\.8 ± 1\.327\.1 ± 5\.90\.21944458\.8111![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro278\.6 ± 49\.14\.8 ± 2\.318\.7 ± 4\.30\.15647241\.9346Subtotal207\.3369Models \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.1896\.7 ± 205\.211\.7 ± 6\.353\.6 ± 13\.40\.505919135\.5864![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.5455\.8 ± 82\.28\.5 ± 2\.134\.5 ± 8\.30\.29846079\.9873![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro278\.2 ± 615\.5 ± 1\.219\.4 ± 4\.80\.16527944\.2948![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro238\.8 ± 74\.55\.0 ± 1\.918\.2 ± 5\.50\.14806139\.6803Subtotal299\.5488Models \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.1892\.6 ± 19512\.4 ± 7\.353\.1 ± 14\.00\.510080136\.7014![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.5462\.7 ± 898\.8 ± 2\.235\.4 ± 8\.80\.30570481\.9287![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro282\.4 ± 58\.75\.6 ± 1\.319\.8 ± 5\.20\.16856245\.1746![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro230\.1 ± 79\.44\.7 ± 1\.817\.2 ± 5\.60\.13907837\.2728Subtotal301\.0775DR \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu368\.9 ± 163\.73\.9 ± 2\.521\.3 ± 10\.80\.16814245\.0621![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger370\.6 ± 90\.75\.8 ± 2\.130\.1 ± 8\.20\.23981123\.9811![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research541\.3 ± 168\.69\.5 ± 2\.825\.6 ± 8\.10\.24651366\.0654![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research810\.4 ± 33812\.0 ± 5\.534\.2 ± 16\.30\.36605898\.1035Subtotal233\.2121DR \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu369\.0 ± 151\.14\.3 ± 2\.615\.7 ± 8\.10\.11238430\.1188![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger349\.4 ± 106\.45\.7 ± 2\.128\.6 ± 8\.90\.23099523\.0995![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research499\.9 ± 146\.39\.4 ± 2\.525\.6 ± 7\.30\.24219464\.9079![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research789\.5 ± 315\.412\.2 ± 4\.629\.9 ± 11\.70\.29917480\.1787Subtotal198\.3049Total \($\)1,239\.4802

Table S14:Cost breakdown for measuring factual precision and recall of generated conclusions from models and deep research agents \(denoted asDR\) under differentSciConHarnesssettings\. We decompose conclusions into atomic facts, then assess precision and recall using our expert\-validatedgpt\-5\.4\-minijudge \(§[4\.3](https://arxiv.org/html/2606.11337#S4.SS3)\)\. We report the API billing costs of usinggpt\-5\.4\-mini\. For each system, we report the total number of facts evaluated \(\# Factscolumn\), cost to evaluate per fact \(Cost \($\) / Factscolumn\), and the total cost of evaluating all the facts \(Total Cost \($\)column\) for both factual precision and recall\. As in Table[S12](https://arxiv.org/html/2606.11337#A8.T12),†\\daggernext tosonar\-deep\-researchdenotes evaluation on a subset \(N=100N=100\)\. All costs are in USD; see §[H](https://arxiv.org/html/2606.11337#A8)for details\.PrecisionRecall\# FactsCost \($\) / FactTotal Cost \($\)\# FactsCost \($\) / FactTotal Cost \($\)Base Models \(No Tools\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.1127090\.00327641\.634728200\.0015484\.3654![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.572530\.00321523\.318428200\.0013443\.7901![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro50090\.00328116\.434528200\.0014514\.0918Subtotal81\.3876Subtotal12\.2473Models \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.1143780\.00330947\.576828200\.0016524\.6586![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.592570\.00319829\.603928200\.0013933\.9283![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro52000\.00327717\.040428200\.0014133\.9847![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro48700\.00324315\.793428200\.0013223\.7280Subtotal110\.0145Subtotal16\.2974Models \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.1142190\.00331247\.093328200\.0016704\.7094![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.594810\.00320630\.396128200\.0014194\.0016![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro52980\.00328117\.382728200\.0014123\.9818![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro46060\.00326215\.024828200\.0013653\.8493Subtotal109\.8969Subtotal16\.5421DR \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu57140\.00328618\.776228200\.0015194\.2836![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger30050\.0031789\.549910750\.0013111\.4093![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research68720\.00330922\.739428200\.0013533\.8155![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research91740\.00330930\.356828200\.0014194\.0002Subtotal81\.4223Subtotal13\.5086DR \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu42090\.00330913\.927628200\.0015144\.2695![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger28550\.0031819\.081810750\.0013601\.4620![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research68560\.00329622\.597428200\.0013703\.8634![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research80060\.00330926\.491928200\.0014534\.0975Subtotal72\.0987Subtotal13\.6924Total \(Precision\)454\.82Total \(Recall\)72\.2878Total \($\)527\.1078

Measuring Factual Precision and Recall Costs\.Table[S14](https://arxiv.org/html/2606.11337#A8.T14)reports the cost of evaluating factual precision and recall across models and deep research agents under differentSciConHarnesssettings\. Costs are computed from billed API usage\. The factual precision task involves more facts than the factual recall task, as generated conclusions are typically longer thanCDSRAuthors’ Conclusions\. Factual precision is also more expensive per fact \($0\.00318–$0\.00331 vs\. $0\.00131–$0\.00167 for recall\), since it requires longer inputs \(e\.g\., the full abstracts and plain\-language summary ofCDSRreview\)\.

In total, precision evaluation costs $454\.82 and recall costs $72\.29, for a combined $527\.11\. Usinggpt\-5\.4instead ofgpt\-5\.4\-miniwould exceed $1,500 as it is over 3×\\timesmore expensive in token usage costs\. The LLM judge processes each fact in∼\\sim2 seconds, compared to∼\\sim6 minutes for domain expert annotation \(§[4\.3](https://arxiv.org/html/2606.11337#S4.SS3)\), yielding over 180×\\timesspeedup\. In terms of cost, assuming U\.S\. federal minimum wage \($7\.25/hour\) as a conservative lower bound, annotating a fact manually takes 6 minutes \($0\.725 per fact\)\. In contrast, our LLM judge evaluation costs at most $0\.00331 \(precision\) and $0\.00167 \(recall\) per fact, corresponding to at least 219–434×\\timescost savings\.

Total Evaluation Cost\.In total, our entire end\-to\-end benchmark evaluation cost $1,569\.946 \+ $1,239\.4802 \+ $527\.1078 = $3,336\.534\.

## Appendix IAdditional Analysis

In this section, we provide additional analysis on the label distribution \(§[I\.1](https://arxiv.org/html/2606.11337#A9.SS1)\),SciConHarnesstool usage patterns \(§[I\.2](https://arxiv.org/html/2606.11337#A9.SS2)\), and Pareto frontier between performance vs\. time and cost \(§[I\.5](https://arxiv.org/html/2606.11337#A9.SS5)\)\.

### I\.1Label Distribution\.

Table[S15](https://arxiv.org/html/2606.11337#A9.T15)shows the full label distribution for factual precision and recall across models and deep research agents\. In terms of precision, base models generate relatively low proportions of facts supported byCDSRreviews \(e\.g\.,gpt\-5\.1: 36\.9%,gemini\-3\-pro: 35\.8%\) with non\-trivial contradiction rates \(3\.8%\-7\.7%\)\. With the exception ofgemini\-3\-pro, tool augmentation viaSciConHarnessgenerally improves recall \(e\.g\.,gpt\-5\.1: 35\.6%→\\rightarrow41\.8% reference facts supported by generated conclusions\) but decreases precision by increasing unsupported generations \(e\.g\.,gpt\-5\.1: 59\.1%→\\rightarrow62\.3% generated facts not supported by the referenceCDSRreview\)\. Under clean\-room evaluation constraints, supported rates consistently decline for both precision and recall, while contradiction rates increase \(except forgemini\-3\-pro\), revealing degraded performance when models must genuinely synthesize rather than retrieve ground\-truth reviews\.

Table S15:Label distribution for factual precision and recall across models and deep research agents \(denoted asDR\)\.†\\daggerdenotes evaluation on a subset \(N=100N=100\)\. Values are percentages\. “Supp\.” = Supported, “Not Supp\.” = Not Supported, and “Contr\.” = Contradicted \(precision only\)\. Among benchmarked models and deep research agents \(excluding consumer\-facing agents\),bluehighlights the highest supported proportion for precision and recall, whileredhighlights the highest contradicted proportion for precision and the highest not\-supported proportion for recall\.PrecisionRecall\# FactsSupp\.Not Supp\.Contr\.\# FactsSupp\.Not Supp\.Base Models \(No Tools\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.11270936\.959\.13\.9282035\.664\.4![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.5725347\.848\.43\.8282024\.375\.7![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro500935\.856\.57\.7282021\.778\.3Models \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.11437833\.462\.34\.3282041\.858\.2![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.5925743\.552\.93\.6282037\.762\.3![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro520032\.361\.06\.7282019\.880\.2![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro487057\.237\.05\.9282033\.366\.7Models \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.11421929\.765\.84\.5282038\.261\.8![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.5948136\.159\.54\.5282030\.769\.3![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro529830\.163\.66\.3282018\.681\.4![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro460640\.850\.88\.4282017\.982\.1DR \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu571429\.066\.74\.3282016\.483\.6![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger300538\.757\.63\.8107536\.064\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research687260\.435\.24\.4282035\.264\.8![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research917460\.935\.53\.6282044\.655\.4DR \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu420924\.969\.85\.3282015\.384\.7![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger285534\.062\.13\.9107523\.976\.1![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research685648\.744\.46\.9282027\.672\.4![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research800645\.048\.76\.3282031\.568\.5Consumer\-Facing AgentsGoogle AI Overview440452\.442\.15\.5274933\.666\.4Google AI Mode466146\.846\.96\.2282035\.264\.8OpenEvidence758459\.837\.23\.0268651\.748\.3

Compared to frontier models, deep research agents achieve higher supported rates \(e\.g\.,o3\-deep\-research: 60\.9%\) but remain sensitive to clean\-room evaluation\. Under the clean\-room,o3\-deep\-researchshows sharp drops in generated facts supported by referenceCDSRreviews \(60\.9%→\\rightarrow45%\), decrease in reference facts supported by the generated conclusions \(44\.6%→\\rightarrow31\.5%\), increases in contradiction rates \(3\.6%→\\rightarrow6\.3%\)\. Notably, consumer\-facing agents like OpenEvidence achieve the strongest balance in correctness and coverage in generated conclusions \(59\.8% precision\-supported; 51\.7 recall\-supported\), suggesting more effective evidence integration\. Overall, these trends reinforce that current systems—even without clean\-room constraints—have substantial room to improve factual precision and recall\.

Table[S16](https://arxiv.org/html/2606.11337#A9.T16)reports the percentage of generated conclusions containing at least one contradictory or unsupported fact with respect toCDSRreviews\. Across all evaluated agents, contradictory facts were common: even the lowest\-performing\-error setting,DR Tuluunder the clean\-room condition, produced at least one contradiction in 44\.8% of conclusions, while several systems exceeded 70%, includinggpt\-5\.1underSciConHarness\(84\.0%\) and under the clean\-room setting \(80\.6%\)\. Facts not supported byCDSRreviews were even more pervasive, appearing in 94\.0–100\.0% of generated conclusions across models, deep research agents, and consumer\-facing agents\. These findings suggest that current AI agents frequently synthesize conclusions that mix supported facts with unsupported or contradictory facts\. This has potential implications in clinical and scientific contexts where users may rely on such agents to interpret evidence and inform downstream decisions\.

Table S16:Percentage of generated conclusions containing at least one fact that contradicts \(≥1 Contr\.\\geq\\textbf\{1 Contr\.\}\) and is not supported \(≥1 Not Supp\.\\geq\\textbf\{1 Not Supp\.\}\) with respect toCDSRreviews across models and deep research agents \(denoted asDR\)\.†\\daggerdenotes evaluation on a subset \(N=100N=100\)\. Excluding consumer\-facing agents,bluehighlights the lowest percentage in each column, whileredhighlights the highest percentage\.Model≥𝟏\\mathbf\{\\geq 1\}Contr\.≥𝟏\\mathbf\{\\geq 1\}Not Supp\.Base Models \(No Tools\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.177\.2100\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.556\.7100\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro70\.1100\.0Models \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.184\.0100\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.560\.899\.3![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro62\.799\.3![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro56\.794\.0Models \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)gpt\-5\.180\.6100\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/claude_icon.jpeg)claude\-sonnet\-4\.573\.9100\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/gemini_icon.png)gemini\-3\-pro61\.299\.3![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-reasoning\-pro70\.997\.4DR \(SciConHarness\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu47\.898\.9![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger61\.0100\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research61\.997\.4![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research55\.295\.9DR \(SciConHarness\+ clean\-room\)![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/ai2_icon.jpeg)DR Tulu44\.899\.3![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/perplexity_icon.png)sonar\-deep\-research†\\dagger62\.0100\.0![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o4\-mini\-deep\-research75\.498\.5![[Uncaptioned image]](https://arxiv.org/html/2606.11337v1/images/company-logos/openai_icon.png)o3\-deep\-research73\.199\.6Consumer\-Facing AgentsGoogle AI Overview56\.398\.9Google AI Mode59\.099\.6OpenEvidence50\.8100\.0
### I\.2SciConHarnessTool Usage Patterns\.

Table[S17](https://arxiv.org/html/2606.11337#A9.T17)shows the average number of tool calls acrossSciConHarnesstools, along with the percentage filtered under clean\-room evaluation\. Tool selection and usage varies substantially across systems\. OpenAI models and agents \(e\.g\.,gpt\-5\.1,o3\-deep\-research,o4\-mini\-deep\-research\) make the heaviest use of tools overall, particularlyweb\_browseandgoogle\_search, often issuing the highest total number of calls per query\. In contrast,claude\-sonnet\-4\.5uses tools more moderately and relies more heavily onpaper\_searchcompared togoogle\_searchandweb\_browse—a pattern also observed forgemini\-3\-pro, which uses tools sparingly\.DR Tulucalls the fewest tools overall, though it is smaller scale \(baseQwen3\-8B\) and has limited context window \(32,768 tokens\), which may constrain extensive tool interaction\. Across systems, the highest rates of clean\-room filtering occur ingoogle\_search\(49\.6%–81\.8%\), indicating frequent retrieval of ground\-truth artifacts; for instance, up to 81\.8% ofgoogle\_searchcalls forclaude\-sonnet\-4\.5was filtered\. Meanwhile,web\_browseandpaper\_searchalso exhibit non\-trivial filtering rates, withweb\_browsereaching up to 7\.3% andpaper\_searchup to 11\.9% across systems\. This highlight the susceptibility of web search and browsing tools to benchmark leakage\.

Table S17:For each model and deep research agent, we show the average number of tool calls per query \(mean±\\pmstd\) acrossSciConHarnesstools:web\_browse,paper\_search,google\_search, and the total tool calls\. We also report the percentage of the tool calls filtered by the clean\-room evaluation protocol\.
### I\.3Failure Mode Analysis

To better understand failure modes in scientific conclusion synthesis, we conduct a small\-scale manual analysis of generated conclusions\. Following prior work\[[15](https://arxiv.org/html/2606.11337#bib.bib114),[73](https://arxiv.org/html/2606.11337#bib.bib38),[58](https://arxiv.org/html/2606.11337#bib.bib65)\], we sampleN=30N=30generated conclusions and analyze why agents produce contradictory or incomplete conclusions that contribute to lower factual precision and recall\. While not exhaustive, common failure modes are:

Failure Mode \#1: Incorrect Direction\-of\-Effect\.One particularly concerning failure mode is that agents misrepresent the direction of treatment effects relative to theCDSRreviews\. Agents sometimes invert the core conclusion of the evidence itself: presenting null or uncertain findings as clinically beneficial or harmful, or reversing positive and negative effects\. In evidence\-based medicine, the direction of effect is often the central conclusion used to guide downstream clinical interpretation and decision\-making\[[89](https://arxiv.org/html/2606.11337#bib.bib32)\]\. As a result, these errors are especially consequential because they can transform a cautious or beneficial finding into a harmful one \(or vice versa\) while still appearing fluent and evidence\-grounded\. Consider the following example:

Generated Conclusion\(o3\-deep\-research, tools\):Large trials \(e\.g\. STICH and STICH II\) and meta\-analyses foundno statistically significant improvement in survival or functional recoverywith early open surgery compared to medical management aloneCDSRReference Article \([https://www\.cochranelibrary\.com/cdsr/doi/10\.1002/14651858\.CD015387\.pub2](https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD015387.pub2)\):For people with spontaneous supratentorial ICH, surgery aimed at clot removal mayincrease the chance of achieving good functional outcome and may reduce all‐cause mortality and 30‐day case fatalitycompared to standard medical management\.Failure Mode \#2: Evidence Quality Mischaracterization\.Another common failure mode is the mischaracterization of evidence quality and certainty\. Agents frequently distort the confidence level expressed in theCDSRreviews, for example, describing findings supported by high\- or moderate\-certainty evidence as “low” or “very low” certainty, or overstating weak and uncertain evidence as highly certain\. Unlike factual omissions alone, these errors can mislead users on how to interpret and act upon the evidence itself\. In evidence\-based medicine, certainty assessments communicate how much confidence clinicians, researchers, and policymakers should place in a conclusion and whether additional evidence is likely to change the finding\. Consequently, misrepresenting evidence quality may lead users to either overtrust weak findings or dismiss well\-supported conclusions\. Example:

Generated Conclusion\(gemini\-3\-pro, tools\):The DASH diet is a powerful intervention for preventing the onset of cardiovascular disease, withhigh\-quality evidencedemonstrating it significantly outperforms standard or ’usual’ dietary practices in reducing cardiovascular risk factors and incidence\.CDSRReference Article \([https://www\.cochranelibrary\.com/cdsr/doi/10\.1002/14651858\.CD013729\.pub2](https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD013729.pub2):The effect of the DASH diet on major cardiovascular outcomes—including myocardial infarction, stroke, cardiovascular mortality, and all‐cause mortality—remains inconclusivedue to a lack of robust long‐term evidence…The certainty of evidence is low to very low, primarily due to design limitations such as high risk of bias, small sample sizes, and short follow‐up periods in the included trials\.Failure Mode \#3: Lack of Specificity\.Another common failure mode is overly general and non\-specific synthesis relative to theCDSRreviews\. Agents often collapse nuanced outcome\-level findings into overly broad review\-level summaries, failing to preserve important distinctions in effects and certainty estimates across different outcomes and population groups\. For example, AI agents often emphasize a few salient outcomes \(e\.g\., mortality, pain reduction, primary endpoints\) while omitting secondary but clinically important outcomes \(e\.g\., quality of life, functional outcomes\)\. As a result, generated conclusions may appear fluent and broadly correct while still failing to communicate these essential details for comprehensive scientific interpretation and real\-world clinical decision\-making, particularly when weighing trade\-offs, patient\-centered outcomes, and downstream risks\. Example:

Generated Conclusion\(sonar\-reasoning\-pro, tools \+ clean\-room\):Multimodal health behavior\-changing interventions targeting children under 10 years with obesity produce modest but clinically meaningful reductions in BMI and obesity prevalence… These interventions consistently improve secondary outcomes including physical activity levels, dietary habits, and obesity\-related knowledge…CDSRReference Article\([https://www\.cochranelibrary\.com/cdsr/doi/10\.1002/14651858\.CD016063](https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD016063):\)For children under 10 years living with obesity, multimodal health behaviour\-changing interventions may slightly improvehealth\-related quality of life…
### I\.4Impact of Conclusion Length on Factual Precision and Recall

Figure[S27](https://arxiv.org/html/2606.11337#A9.F27)shows the relationship between conclusion length and factual precision/recall\. Longer conclusions consistently exhibit lower precision, indicating that additional generated facts are less likely to be supported by the referenceCDSRreviews, aligning with prior works\[[15](https://arxiv.org/html/2606.11337#bib.bib114),[68](https://arxiv.org/html/2606.11337#bib.bib147)\]\. At the same time, recall generally increases with length, reflecting improved coverage of the reference conclusions\. This trade\-off highlights that longer outputs do not necessarily improve factual F1, the harmonic mean of both factual precision and recall\. Importantly, this suggests that the factual F1 is robust to conclusion length: increases in length are penalized through reduced precision, preventing models and agents from inflating performance by generating longer conclusions\. These suggest that model and agent quality matter more than the length of conclusions to obtain a high score onSciConBench\.

![Refer to caption](https://arxiv.org/html/2606.11337v1/x18.png)Figure S27:Relationship between conclusion length in words vs\. factual precision and recall across models and deep research agents\.
### I\.5Pareto Frontier Between Performance vs\. Time and Cost

Performance vs\. Cost\.Figures[S28](https://arxiv.org/html/2606.11337#A9.F28)\-[S30](https://arxiv.org/html/2606.11337#A9.F30)show the Pareto frontier of performance vs\. cost for models and deep research agents across factual F1, precision, and recall\. Clean\-room constraints consistently flatten the frontier, aligning with our earlier findings on performance attenuation\. For factual F1,DR Tulu,sonar\-reasoning\-pro,o4\-mini\-deep\-research, ando3\-deep\-researchlie on the Pareto frontier, representing the most efficient trade\-offs at different cost levels\.DR Tuluis the most cost\-efficient but lowest\-performing point, whileo3\-deep\-researchachieves the highest performance at the greatest cost\.

Performance vs\. Time\.Figures[S31](https://arxiv.org/html/2606.11337#A9.F31)\-[S33](https://arxiv.org/html/2606.11337#A9.F33)show the Pareto frontier of performance vs\. time for models and deep research agents across factual F1, precision, and recall\. Clean\-room constraints flatten and shift the frontier rightward, indicating both attenuated performance and increased time to synthesize conclusions\. This increase in latency suggests that agents are engaging in genuine synthesis rather than shortcut retrieval\. Under clean\-room evaluation,sonar\-reasoning\-pro,claude\-sonnet\-4\.5, ando3\-deep\-researchlie on the Pareto frontier, representing efficient trade\-offs at different latency levels\.sonar\-reasoning\-prois the most time\-efficient but lowest\-performing, whileo3\-deep\-researchachieves the highest performance at the greatest time cost\.

![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/cost_performance_tools_only/cost_vs_macro_f1.png)

![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/cost_performance_tools_filter/cost_vs_macro_f1.png)

Figure S28:Performance vs\. cost of frontier models and deep research agents\. We plot factual F1 against cost \(USD per query\)\. Left:SciConHarnesswithout clean\-room constraints\. Right:SciConHarnesswith clean\-room evaluation\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/cost_performance_tools_only/cost_vs_macro_precision.png)

![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/cost_performance_tools_filter/cost_vs_macro_precision.png)

Figure S29:Performance vs\. cost of frontier models and deep research agents\. We plot factual precision against cost \(USD per query\)\. Left:SciConHarnesswithout clean\-room constraints\. Right:SciConHarnesswith clean\-room evaluation\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/cost_performance_tools_only/cost_vs_macro_recall.png)

![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/cost_performance_tools_filter/cost_vs_macro_recall.png)

Figure S30:Performance vs\. cost of frontier models and deep research agents\. We plot factual recall against cost \(USD per query\)\. Left:SciConHarnesswithout clean\-room constraints\. Right:SciConHarnesswith clean\-room evaluation\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/time_performance_tools_only/time_vs_macro_f1.png)

![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/time_performance_tools_filter/time_vs_macro_f1.png)

Figure S31:Performance vs\. time of frontier models and deep research agents\. We plot factual F1 against time \(seconds per query\)\. Left:SciConHarnesswithout clean\-room constraints\. Right:SciConHarnesswith clean\-room evaluation\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/time_performance_tools_only/time_vs_macro_precision.png)

![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/time_performance_tools_filter/time_vs_macro_precision.png)

Figure S32:Performance vs\. time of frontier models and deep research agents\. We plot factual precision against time \(seconds per query\)\. Left:SciConHarnesswithout clean\-room constraints\. Right:SciConHarnesswith clean\-room evaluation\.![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/time_performance_tools_only/time_vs_macro_recall.png)

![Refer to caption](https://arxiv.org/html/2606.11337v1/images/analysis/time_performance_tools_filter/time_vs_macro_recall.png)

Figure S33:Performance vs\. time of frontier models and deep research agents\. We plot factual recall against time \(seconds per query\)\. Left:SciConHarnesswithout clean\-room constraints\. Right:SciConHarnesswith clean\-room evaluation\.

## Appendix JDetails on Auditing Consumer\-Facing Agents

We audit three consumer\-facing AI agents on commercial platforms: Google AI Overview, Google AI Mode, and OpenEvidence\. Each system was queried over our benchmark set ofN=268N=268samples\. Each question was augmented with a standardized benchmark suffix instructing the system to synthesize a paragraph\-length conclusion drawing on the highest\-quality and most up\-to\-date evidence, explicitly discussing strengths, limitations, uncertainty, and contradictions across the body of evidence, with the conclusion paragraph delimited by triple square brackets for downstream extraction\. See Figure[S34](https://arxiv.org/html/2606.11337#A10.F34)for the prompt used to standardize query formatting\. We detail the data collection below:

Google AI Overview & AI Mode\.For both Google AI Overview and AI Mode, we collect their synthesized conclusions using the SerpAPI library151515[https://serpapi\.com/](https://serpapi.com/)\. Since Google does not consistently generate an AI Overview for every query on the first attempt, the pipeline retried the same query up to three times, with a 15\-second wait in\-between before recording a null response\.

OpenEvidence\.As a proprietary platform for clinicians, OpenEvidence does not provide a public API\. We therefore collect responses via automated browsing with SeleniumBase, issuing queries and scraping the generated conclusions\. To respect rate limits, we introduce delays of up to 265 seconds between queries, allowing sufficient time for response generation\. Given automated browsing, OpenEvidence did not consistently return a response; the pipeline therefore retried each query before recording a null output\.

Audit Prompt\{question\}Synthesize a paragraph\-long conclusion using the highest\-quality and most up\-to\-date scientific evidence available, and explicitly discuss the strengths, limitations, uncertainty, and contradictions across the body of evidence\. Wrap the conclusion paragraph in three square brackets\.Figure S34:Prompt for auditing consumer\-facing agents: Google AI Overview, Google AI Mode, and OpenEvidence\. For eachN=268N=268query, we replace \{question\} placeholder with the query\.
## Appendix KNeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The main claims made in the abstract and introduction accurately reflect the main paper’s contributions and scope\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: We include a thorough limitation discussion in §[A](https://arxiv.org/html/2606.11337#A1)\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[N/A\]
14. Justification: There are no theoretical results in this paper\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper, and we will release the full benchmark dataset and code to assist the reproducibility of our experimental results\. The Appendix sections contain all the necessary details for reproducing our results\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
24. Justification: This paper provides open access to the data and code, including instructions and sample code to run various modules in the paper\. We will include links to the dataset collection and our code in the first page of the abstract\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: All appendix sections contain all the necessary experimental details\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification:We report statistical significance analyses of factual F1 differences between the best\- and second\-best\-performing agents in §[6](https://arxiv.org/html/2606.11337#S6), with power analyses in §[G](https://arxiv.org/html/2606.11337#A7)\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: We include all experiments, compute, API costs, and human annotation resources in the Appendix, covering all resources used for preprocessingSciConBench, generating model response, and measuring factual precision and recall\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: We confirm the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification: We discuss broader impacts in §[A](https://arxiv.org/html/2606.11337#A1)\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[Yes\]
54. Justification: We discuss safeguards that have been put in place for responsible release of data in §[A](https://arxiv.org/html/2606.11337#A1)\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: The creators or original owners of assets used in the paper are properly credited and are respected for the license \(e\.g\.,CDSRuses license CC\-BY\-NC 4\.0\) and terms of use explicitly mentioned\. See §[A\.2](https://arxiv.org/html/2606.11337#A1.SS2)\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://arxiv.org/html/2606.11337v1/paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[Yes\]
64. Justification: We document all assets\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[Yes\]
69. Justification: We include details for human annotations in §[B\.2](https://arxiv.org/html/2606.11337#A2.SS2), §[D\.3](https://arxiv.org/html/2606.11337#A4.SS3), and §[E\.1](https://arxiv.org/html/2606.11337#A5.SS1)\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: Our human annotation is innocuous and thus does not require IRB approval\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[Yes\]
79. Justification: We describe how LLMs are used to constructSciConBench\(§[2](https://arxiv.org/html/2606.11337#S2)\), generate conclusions withSciConHarness\(§[3](https://arxiv.org/html/2606.11337#S3)\), and evaluate the factual quality of generated conclusions \(§[4](https://arxiv.org/html/2606.11337#S4)\)\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.