科学检索智能体在何处失败：暴露与查阅尝试的决策检查点审计

# Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts
Source: [https://arxiv.org/html/2609.38670](https://arxiv.org/html/2609.38670)
Hongmin LiAffiliation:School of Life Science and Technology, Institute of Science TokyoTokyo, JapanAffiliation:Department of Computational Biology and Medical SciencesGraduate School of Frontier Sciences, The University of TokyoKashiwa, JapanEmail:[lihongmin@edu\.k\.u\-tokyo\.ac\.jp](mailto:[email protected])Wanli ZhaoAffiliation:Department of Information ScienceGraduate School of Advanced Science and Engineering, Hiroshima UniversityHigashihiroshima, Japan

###### Abstract

Final\-answer accuracy provides limited insight into where scientific\-search agents fail: a target paper may remain unseen, appear in search results without being inspected, or be inspected before an incorrect answer\. We introduce decision\-checkpoint auditing, which records search observations and executed tool actions during inference and classifies outcomes by matching these events to target identities and evaluator labels afterward\. Across five search conditions on 540 answerable AutoResearchBench Deep questions in a fixed, target\-enriched environment, keyword search achieves 24\.6% accuracy, compared with 17\.8% for raw search\. Keyword search has fewer failures in which the target is neither exposed nor inspected, but more failures after exposure without inspection\. A read\-first policy makes 27\.4% more evidence\-search calls than keyword search, while target inspection attempts occur on 199 rather than 191 questions\. Both policies achieve 24\.6% accuracy, with 60 gains and 60 losses in paired answer outcomes\. These results show how trajectory\-level measurements distinguish target exposure, inspection attempts and answer correctness, revealing differences obscured by aggregate accuracy and tool\-use totals\.

## 1Introduction

Scientific\-search agents identify papers by alternating between search queries, candidate inspection and answer selection\([He et al\., 2025](https://arxiv.org/html/2609.38670#bib.bib8);[Xiong et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib4)\)\. A final paper\-identification score records whether the selected paper matches the benchmark target, but leaves the intervening search decisions unresolved\. An agent may answer incorrectly after failing to expose the target, leaving an exposed target uninspected, or attempting to inspect the target\. Distinguishing these events specifies which part of a search trajectory an intervention is intended to change \(Figure[1](https://arxiv.org/html/2609.38670#S1.F1)\)\.

Existing work examines intermediate retrieval stages and search failures\. PaperQA2 tracks source retention through its evidence pipeline\([Skarlinski et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib14)\), AutoResearchBench manually analyzes scientific\-search errors\([Xiong et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib4)\), and SearchAuditor diagnoses selected failed trajectories\([Liang et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib13)\)\. We study a complementary measurement question: across the same benchmark questions, how do target exposure, target inspection attempts and accepted answers differ between search conditions?

Our approach represents search trajectories as*decision checkpoints*and identifies target exposure and inspection attempts by matching search observations and executed actions to benchmark paper IDs after inference\. These event labels require no additional semantic judge\. Inspection denotes an executed within\-paper evidence\-search call, whose returned excerpts may or may not provide sufficient evidence for the question\. Answer correctness is assessed separately by an adapted benchmark evaluator\. Ground\-truth labels are withheld during inference, although target identities inform corpus construction\.

The paper makes two contributions\. First, we define an event\-based framework linking target exposure and executed inspection attempts to answer correctness on a common set of questions\. Second, we characterize these outcomes across five search conditions on 540 answerable AutoResearchBench Deep questions in a shared, target\-enriched environment\. Keyword search has fewer failures without target exposure or inspection than raw search, but more failures after exposure without inspection\. For read\-first, separating agent\-initiated and gate\-forced calls reveals how inspection effort is distributed across candidate papers\. Paired comparisons further reveal changes in target inspection and answer correctness that identical aggregate accuracies conceal\.

Figure 1:Schematic of decision\-checkpoint auditing\. \(a\) Three distinct failure paths share the same terminal score\. \(b\) Inference\-time checkpoints contain no ground\-truth \(GT\) labels; target identities and evaluator labels are added afterward\. Observable trajectories are classified by correctness, then inspection and exposure\. API exceptions without trajectories form a separate unobservable category\. Inspection denotes an executed within\-paper evidence\-search attempt\.
## 2Related Work

#### Scientific search and stage measurement\.

AutoResearchBench defines Deep and Wide literature search over arXiv\([Xiong et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib4)\)\. Its Appendix F manually analyzes sampled incorrect predictions, including evidence aggregation, tool execution and candidate ranking failures; the latter explicitly includes a gold paper already in the candidate set but ranked below another paper\. PaSa models search/read/select decisions\([He et al\., 2025](https://arxiv.org/html/2609.38670#bib.bib8)\), LitSearch benchmarks retrieval and reranking\([Ajith et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib9)\), and OpenScholar generates citation\-supported answers\([Asai et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib10)\)\. PaperQA2 tracks source\-DOI recall through retrieval, chunk ranking, contextual summarization and attribution\([Skarlinski et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib14)\), a direct precedent for separating discovery from downstream evidence use\. We measure target exposure and executed inspection attempts across the evaluated questions, linking these events to paired answer outcomes\.

#### Process and evidence evaluation\.

AgentBoard measures partial progress through goal\-state similarity or defined subgoals\([Ma et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib15)\); ToolSandbox matches temporally ordered milestones involving tool calls and environment states\([Lu et al\., 2025](https://arxiv.org/html/2609.38670#bib.bib16)\); AgentEval evaluates DAG\-structured steps and error propagation\([Guo et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib12)\)\. SearchAuditor evaluates failure localization, attribution and repair against expert annotations of selected failed trajectories, and relates exploration to downstream failure types\([Liang et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib13)\)\. Semantic evidence diagnostics include SciFact’s rationale and support/refutation labels\([Wadden et al\., 2020](https://arxiv.org/html/2609.38670#bib.bib17)\)and RAGChecker’s claim\-level entailment checks for retrieval and generation\([Ru et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib18)\)\. Our target\-identity and executed\-action labels provide an event\-level complement to these semantic diagnoses, covering successes as well as failures\.

#### Retrieval access and resource allocation\.

BrowseComp\-Plus fixes the corpus and varies whole\-document reader access and oracle retrieval with annotated positive documents\([Chen et al\., 2025](https://arxiv.org/html/2609.38670#bib.bib11)\)\. Cost\-aware evaluation reports accuracy jointly with resource use\([Kapoor et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib3)\), while budget\-aware control\([Fang et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib5)\)and value\-guided search\([Li et al\., 2026](https://arxiv.org/html/2609.38670#bib.bib6)\)optimize allocation\. We examine a gate that changes the timing of calls to an existing within\-paper evidence\-search tool, measuring both overall inspection effort and the number of questions on which that effort reaches the target\.

## 3Decision\-Checkpoint Auditing

#### Trajectory representation\.

The agent searches for papers, queries evidence within a selected paper, or produces a final answer\. For each turnttwith a model response, we define a checkpointct=\(st,at,ot,costt\)c\_\{t\}=\(s\_\{t\},a\_\{t\},o\_\{t\},\\mathrm\{cost\}\_\{t\}\)\. The statests\_\{t\}summarizes the maintained candidate list, prior queries, paper IDs with inspection attempts and retained evidence excerpts\. This analytical summary is distinct from the model’s input context and is not supplied as an additional message\. The actionata\_\{t\}represents the proposed action or, when the gate intervenes, the substituted evidence\-search call; execution status is recorded separately\. The observationoto\_\{t\}contains returned paper IDs and evidence excerpts, whilecostt\\mathrm\{cost\}\_\{t\}contains token use and latency\. Search results enter the model context as tool messages, and older full\-text returns are generally condensed into inspection summaries\. Inference messages and checkpoints contain no ground\-truth labels; benchmark metadata is associated with the trajectories only during subsequent evaluation\.

#### Retrieval, exposure and inspection\.

For a target papery∗y^\{\*\},*retrieval*occurs when the backend includesy∗y^\{\*\}among the top\-NNresults for a query\.*Exposure*occurs when the target identifier appears in a search\-result list presented to the model, with its rank defined within that list\.*Inspection*occurs when an executed within\-paper evidence\-search call targetsy∗y^\{\*\}\. Exposure is measured from search results rather than subsequent repetitions of the identifier\. Inspection includes unsuccessful calls and therefore denotes an attempt to obtain evidence, without establishing that adequate evidence was acquired\.

#### Outcome categories\.

After inference, target identities are matched to each trajectory and answer correctness is determined by the evaluator\. Accepted answers are classified as*Correct*, regardless of prior search or inspection\. Incorrect answers with observable trajectories are classified as*Incorrect, target inspected*;*Incorrect, target exposed but not inspected*; or*Incorrect, target neither exposed nor inspected*, with inspection taking precedence over exposure\. API exceptions without trajectories form a separate unobservable category\. This mutually exclusive classification describes outcomes without assuming a mandatory sequence of stages; exposure and inspection are also measured independently\.

#### Exposure rank and subsequent decisions\.

We characterize target visibility by its rank at first exposure and the number of search\-result lists in which it appears\. For incorrect answers with exposure but no inspection, we also measure whether at least one valid search, evidence\-search or answer action follows first exposure\. This continuation indicator establishes that another decision occurred after the target became visible, without estimating the remaining inspection budget or the counterfactual benefit of inspection\.

#### Observation scope\.

Exposure is determined from search observations and verified against retained tool messages; subsequent valid actions exclude parsing failures\. Exposure records the target’s appearance at any point in the trajectory, regardless of its later retention in the model context\. Because final\-message records omit some earlier evidence\-search returns and checkpoints preserve excerpts rather than complete responses, the available trajectories support event\-level analysis but not full reconstruction or semantic evaluation of every model input\.

## 4Experimental Setting

#### Task and sample\.

AutoResearchBench Deep requires identification of a single target arXiv paper or a “no answer” response\. We evaluate five search conditions on all 540 answerable questions in its 600\-question Deep split, excluding the 60 explicitly unsatisfiable questions\. The resulting sample supports paper\-identification analysis but does not assess abstention when no answer exists\. A subset of 77 questions informed exploratory analysis and intervention design, so results on that subset are not independent confirmation\. The study is restricted to Deep and a single model request alias\. An episode denotes one execution of one condition on one question, and a trajectory is its sequence of actions and observations\.

#### Retrieval environment\.

The backend combines BM25\([Robertson and Zaragoza, 2009](https://arxiv.org/html/2609.38670#bib.bib1)\)retrieval over titles and abstracts from an indexed arXiv metadata snapshot\([Cornell University, n\.d\.](https://arxiv.org/html/2609.38670#bib.bib7)\)containing 3\.16 million records with BM25 retrieval over 7,703 cached full\-text papers\. Metadata retrieval usesk1=1\.2k\_\{1\}\{=\}1\.2andb=0\.75b\{=\}0\.75, and the two rankings are combined by reciprocal\-rank fusion\([Cormack et al\., 2009](https://arxiv.org/html/2609.38670#bib.bib2)\)withc=60c\{=\}60\. The full\-text collection contains 3,853 targets from the broader benchmark corpus and 3,850 non\-target papers matched by primary category and year; we denote this configuration*\+both*\. Each full\-text index entry contains at most 400,000 extracted characters\. In contrast, within\-paper evidence search uses the complete extracted text and returns up to three query\-matched windows, each covering at most eight lines on either side of a match and truncated to 1,800 characters\. Although ground\-truth labels are withheld from model inputs, target identities are used to construct the full\-text collection\. Its target\-enriched coverage can therefore influence both fused retrieval rankings and subsequent evidence access\.

#### Search policies\.

We use the AutoResearchBench inference framework with adapted tool prompts\. The primary comparisons are keyword versus raw search and read\-first versus keyword search\. Inraw, the agent is instructed to use the complete question verbatim for every search; this restriction is prompt\-based rather than enforced by the execution framework\. Inkw, the agent formulates short, distinctive keyword queries and may reformulate them throughout the trajectory\. Read\-first retains the keyword rule and adds an inspection prompt and gate \(Section[6](https://arxiv.org/html/2609.38670#S6)\)\. Two secondary conditions examine alternative query strategies:decompdivides the question into focused sub\-questions, searches once for each and combines the evidence to select candidates;adaptiveiteratively assesses candidates and reformulates queries using informative terms or titles until the evidence and candidate set stabilize\. The suffix\-ftdenotes full\-text\-enabled policies on the shared \+both backend\. Appendix[B](https://arxiv.org/html/2609.38670#A2)specifies the query rules\.

#### Execution and resource measurement\.

All conditions use the request aliasgpt\-5\.6\-luna, the same retrieval backend and tool interfaces, and common nominal execution limits\. The agent selects retrieval depth and evidence\-search queries, subject to a candidate cap of 200 and a maximum of 30 rounds\. The execution framework applies a soft context threshold of 110,000 estimated input tokens\. Episodes reaching a round or context limit remain in the analysis\. We measure Deep Accuracy@1 \(pass@1\), alongside search calls, evidence\-search calls, token use and termination conditions\. The category*none*includes all final answers without an arXiv identifier, whether explicitly abstaining or terminating without an identified paper\.

#### Answer evaluation\.

The evaluator assesses whether the title of the first candidate in the final answer identifies the target paper; an absent candidate or explicit no\-answer response scores zero on this answerable sample\. Normalized paper IDs are used only in a separate scoring sensitivity analysis \(Appendix[B](https://arxiv.org/html/2609.38670#A2)\)\. The original runs do not fully specify the underlying model snapshots, effective reasoning settings or judge configurations, and the request alias alone does not establish a common model identity\. Policy comparisons are consequently descriptive rather than verified same\-model causal estimates\. Re\-evaluating all 2,700 answers with a uniform, documented title\-based judge configuration reproduces every original correctness label, establishing label stability under that configuration while leaving judge validity and original model identity unresolved\.

## 5Target Exposure and Failure Patterns

Figure 2:Decision\-checkpoint audit on all 540 answerable Deep questions \(\+both backend, request alias gpt\-5\.6\-luna\)\. \(a\) Correct answers take precedence; remaining questions are classified by inspection and exposure\. Raw and keyword conditions differ in*neither exposed nor inspected*\(333→\\to64\) and*exposed, not inspected*\(67→\\to277\) failures, with Acc@1 of 17\.8% and 24\.6%\. \(b\) Cumulative fraction of questions whose target first appears within a given rank of an observation; underkw\-ftthe target is exposed within rank 20 on 336/540 questions\.The keyword condition has fewer never\-exposed failures and more exposed\-but\-uninspected failures than raw search \(Figure[2](https://arxiv.org/html/2609.38670#S5.F2), Table[1](https://arxiv.org/html/2609.38670#S5.T1)\)\. Both conditions cover the same 540 answerable Deep questions; all outcome proportions use that 540\-question denominator\.

Table 1:Outcome categories across five conditions on the same 540 questions\. Categories are mutually exclusive; all except Correct and Unobservable are incorrect outcomes\. Exposure and inspection refer to the target paper\. Exposure is also counted separately\.Accuracy is calculated over all 540 questions\. Unobs\. denotes the read\-first episode without an observable trajectory because of an API exception; no exposure or inspection status is imputed for this episode\.UUdenotes exposed\-but\-uninspected failures, andU/EU/Eis their proportion within each condition’s exposed subgroup\. These subgroups differ across conditions\. Exposure is measured independently of the mutually exclusive outcome categories\. A subsequent valid action indicates continuation after exposure, without establishing the remaining inspection budget\. Appendix[A](https://arxiv.org/html/2609.38670#A1)provides supplementary raw/keyword comparisons\.

Table 2:Per\-question outcome transitions raw\-ft→\\tokw\-ft on the same 540 questions\. All categories except Correct are incorrect outcomes; exposure and inspection refer to the target paper\.#### Paired outcome transitions\.

Keyword search has fewer failures without target exposure \(62%→\\to12%\) and an accuracy advantage of 6\.85 percentage points over raw search\. The paired comparison reveals that 182 of the 333 questions with neither exposure nor inspection underrawinstead fail after exposure without inspection underkw\(Table[2](https://arxiv.org/html/2609.38670#S5.T2)\)\. Conversely, 36 questions answered correctly underrawbecome exposed\-but\-uninspected failures underkw\.

Target exposure frequently precedes further search decisions\. All 277 keyword failures after exposure without inspection contain a subsequent valid action\. Among the 476 keyword episodes with target exposure, 335 first expose the target by turn 3, and the median number of subsequent search, evidence\-search or answer actions is 13\. These observations establish continuation after exposure but do not quantify unused budget\. Execution limits are also common in this failure category: 167 of 277 keyword episodes terminate at a token or turn limit, compared with 20 of 67 underraw\.

#### First\-exposure rank and subsequent inspection\.

Among the 336 keyword questions whose target first appears within the top 20, 184 \(54\.8%\) have no subsequent target inspection attempt\. Even at ranks 1–5, 83 of 175 questions \(47\.4%\) have no subsequent attempt \(Table[3](https://arxiv.org/html/2609.38670#S5.T3)\)\. These groups include both correct and incorrect answers, showing that uninspected targets also occur near the top of the result list\. Appendix[A](https://arxiv.org/html/2609.38670#A1)derives a separate lower bound restricted to incorrect answers\.

Table 3:Keyword first\-exposure rank, subsequent target inspection attempts and final correctness\. All counts are numbers of questions\. Percentages use the exposed questions in each rank group as denominators\. “Later target inspection” requires an executed target call on a strictly later turn\.The 476 questions with target exposure are grouped by rank at first exposure\. All 191 questions with a target inspection attempt have their attempt on a later turn\. Percentages use the number of exposed questions in each rank group, and correctness follows the primary title\-based evaluation\. Outcomes are measured through episode termination\. Because rank, exposure timing and subsequent trajectory length are uncontrolled, the associations do not identify a causal effect of rank\.

## 6Read\-First Gate: Inspection and Answer Outcomes

The prevalence of failures after exposure without inspection motivates examination of a read\-first policy\. This policy inserts a within\-paper evidence\-search attempt before another search or final answer whenever an eligible unread candidate remains\. We compare its target inspection coverage, answer correctness and resource use with keyword search, distinguishing additional calls across all candidates from attempts directed at the target\.

Table 4:Paired correctness and questions with a target inspection attempt\. Gains and losses refer to the left\-to\-right policy change\.\(a\) Correctness on allN=540N=540questionsComparisonBoth correctGainLossBoth wrongΔ\\DeltaAcc\. \(pp\)raw→\\tokw478649358\+6\.85kw→\\toread\-first7360603470\.00kw→\\todecomp766257345\+0\.93kw→\\toadaptive695364354−2\.04\-2\.04\(b\) Target inspection attempt: kw→\\toread\-first,N=539N=539jointly observableEventBothOnly read\-firstOnly kwNeitherUnobs\. pairTarget inspection attempt13366572831∗
∗The read\-first API exception is included in the 540 answer outcomes but excluded from the 539 observable inspection pairs; its keyword counterpart contains a target inspection attempt\. Target inspection is observed on 191 keyword and 199 read\-first questions overall, or 190 and 199 within the paired observable sample\. The latter difference is a net increase of nine across 123 changes in inspection status, whereas correctness changes on 120 questions\. Every episode with a target inspection attempt contains at least one nonempty target excerpt returned on the same turn; this episode\-level measure does not imply success for every call\.

#### Read\-first policy\.

After each paper search, the gate requires an inspection attempt before a subsequent search or final answer if an eligible candidate remains\. A candidate is eligible when it has a valid normalized arXiv identifier and has not previously been the subject of an inspection attempt; unsuccessful attempts also remove eligibility\. If the agent proposes another search or a final answer, the gate substitutes an evidence\-search call on the highest\-ranked eligible candidate in the latest result list, using the original question\. This call consumes the current turn, after which the agent selects a new action\. Any agent\-initiated evidence\-search call satisfies the gate, regardless of candidate rank or query\. With no eligible candidate, the proposed action proceeds\. Thus, although the prompt requests inspection of the highest\-ranked unread candidate, the executed policy permits other agent\-initiated inspections\. The gate uses no ground\-truth labels and remains subject to context and turn limits; limit\-terminated episodes are included in all outcome and resource analyses\.

Keyword and read\-first each answer 133 of 540 questions correctly\. They answer 73 questions correctly in both conditions; 60 are correct only under keyword search and 60 only under read\-first \(Table[4](https://arxiv.org/html/2609.38670#S6.T4)\)\. Read\-first has more evidence\-search calls per question than keyword search \(9\.63 versus 7\.56\), but recorded target inspection attempts occur on 199 questions under read\-first and 191 under keyword search \(Table[5](https://arxiv.org/html/2609.38670#S6.T5)\)\. Among the 539 questions with observable trajectories in both conditions, the corresponding numbers of questions with a target inspection attempt are 199 and 190\.

#### Sources of target inspection\.

The 539 observable read\-first trajectories contain 5,198 evidence\-search calls: 1,720 forced by the gate and 3,478 initiated by the agent \(Table[5](https://arxiv.org/html/2609.38670#S6.T5)\)\. Gate\-forced calls reach the target on 22 questions and agent\-initiated calls on 187, with an overlap of ten\. Twelve questions therefore have target inspection only through forced calls, while 177 have it only through agent\-initiated calls\. These within\-policy counts describe how inspection occurs; the paired comparison in Table[4](https://arxiv.org/html/2609.38670#S6.T4)measures differences between policies\. Every observed episode with target inspection contains at least one nonempty target excerpt associated with an executed call, although individual calls can fail\. Appendix[B](https://arxiv.org/html/2609.38670#A2)defines this excerpt measure\.

#### Uncertainty and sensitivity to exploratory questions\.

The paired accuracy difference between keyword and raw search is \+6\.85 percentage points, with a 95% question\-bootstrap interval of \[2\.59, 11\.11\]\. The read\-first–keyword difference is 0\.00 points, with an interval of \[\-4\.07, 4\.07\]\. Resampling the 529 target\-identifier clusters gives corresponding intervals of \[2\.75, 11\.05\] and \[\-4\.03, 4\.05\]\. Excluding the 77 exploratory questions leaves 463 questions: keyword search has 74 gains and 43 losses relative to raw \(\+6\.70 points; interval \[2\.16, 11\.23\]\), while read\-first has 52 gains and 50 losses relative to keyword \(\+0\.43 points; \[\-3\.89, 4\.75\]\)\. These intervals describe variation across sampled questions or targets; they do not capture model variability, judge error or stochastic variation across repeated executions\. Excluding exploratory questions is a sensitivity analysis rather than independent confirmation\.

#### Inference resource use\.

Following cost\-aware agent evaluation\([Kapoor et al\., 2024](https://arxiv.org/html/2609.38670#bib.bib3)\), we compare resource use alongside answer accuracy\. Read\-first makes 27\.4% more recorded evidence\-search calls and uses 4\.8% more inference tokens than keyword search, with identical aggregate accuracy \(Table[5](https://arxiv.org/html/2609.38670#S6.T5)\)\.

Table 5:Recorded inference use and termination\. Means use all 540 primary episodes per condition; termination columns are counts\. Evidence\-search calls include all candidate papers\. The second panel separates call origins in the 539 observable read\-first trajectories\.Tokens denote cumulative recorded input and output \(k=1000\\mathrm\{k\}=1000\); they measure neither peak context size nor billed cost\. Context\-limit termination includes the execution framework’s soft threshold\. Means use all 540 primary episodes, including the API exception, and reflect available usage records\. They exclude the separate retry, evaluator calls and unrecorded infrastructure costs; missing usage is not interpreted as zero total cost\.

#### Sensitivity to the missing trajectory\.

A separate stochastic retry of the read\-first API exception exposes and inspects the target but still answers incorrectly\. Substituting this retry leaves accuracy unchanged and assigns the previously unobservable episode to the inspected\-but\-incorrect category\. The primary analysis retains the original episode, and the retry’s resource use is excluded from primary\-run costs\.

#### Sensitivity to the scoring criterion\.

Identifier\-based matching identifies one target\-title version mismatch classified as incorrect by the original title\-based evaluator under keyword, read\-first, decomposition and adaptive search, but not raw\. Scoring solely by normalized target identity adds one correct answer in each of these four conditions, leaving the keyword–read\-first accuracy difference at zero\. Uniform title\-based re\-evaluation retains the original classifications\. We therefore use the original title\-based criterion for the primary results and report identifier agreement as a sensitivity analysis\.

## 7Discussion and Limitations

The results distinguish target visibility from subsequent inspection in scientific search\. Keyword search has substantially fewer failures without exposure than raw search, yet many visible targets remain uninspected, including targets near the top of the result list\. This pattern motivates examination of candidate selection and post\-exposure resource allocation alongside retrieval\. Inspection attempts followed by incorrect answers constitute a further group, but attributing those failures to reasoning requires evidence that the returned excerpts adequately address the question\.

The read\-first comparison shows why overall tool use and target\-specific coverage should be measured separately\. More evidence\-search calls coincide with only a modest difference in the number of questions with target inspection, while paired comparisons reveal substantial changes in both inspection status and answer correctness\. Identical aggregate accuracy consequently masks different question\-level outcomes\. Nonempty target excerpts establish that some content was returned during the inspection attempt, but do not establish its semantic sufficiency\.

Several limitations constrain interpretation\. The study covers one answerable benchmark subset in a target\-enriched retrieval environment, and 77 questions informed intervention design\. The original underlying\-model identities and effective reasoning settings are incompletely specified\. Moreover, without repeated executions within each condition, policy differences cannot be separated from stochastic trajectory variation; resampling questions or targets does not supply such replication\. Finally, retained excerpts permit event\-level measurement but not complete reconstruction or semantic assessment of every model input\. The findings are therefore descriptive, with limited generalizability and no identified causal mechanism\.

## 8Conclusion

Decision\-checkpoint auditing provides an event\-based account of target exposure, inspection attempts and answer correctness in scientific search\. Across 540 answerable AutoResearchBench Deep questions in a target\-enriched environment, keyword search has fewer failures without exposure or inspection than raw search, but more failures after exposure without inspection\. Read\-first makes 27\.4% more evidence\-search calls than keyword search, while target inspection is observed on 199 rather than 191 questions\. Both policies answer 133 questions correctly, with 60 gains and 60 losses in the paired comparison\. Measuring these events alongside final accuracy reveals differences in search behavior and question\-level outcomes that aggregate scores alone obscure\.

### AI use statement

AI systems assisted with methodological and experimental design, code and implementation, qualitative analysis and interpretation of trajectories and results, figure preparation, and manuscript drafting and revision, under the direction of the authors\. The authors are responsible for the final content of the manuscript\.

## References

- A\. Ajith, M\. Xia, A\. Chevalier, T\. Goyal, D\. Chen, and T\. GaoLitSearch: a retrieval benchmark for scientific literature search\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15068–15083\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.840),[Link](https://aclanthology.org/2024.emnlp-main.840/)Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px1.p1.1)\.
- Asaiet al\.\(2024\)A\. Asai, J\. He, R\. Shao, W\. Shi, A\. Singh, J\. C\. Chang, K\. Lo, L\. Soldaini, S\. Feldman, M\. D’arcy, D\. Wadden, M\. Latzke, M\. Tian, P\. Ji, S\. Liu, H\. Tong, B\. Wu, Y\. Xiong, L\. Zettlemoyer, G\. Neubig, D\. Weld, D\. Downey, W\. Yih, P\. W\. Koh, and H\. HajishirziOpenScholar: synthesizing scientific literature with retrieval\-augmented LMs\.arXiv preprint arXiv:2411\.14199\.External Links:[Link](https://arxiv.org/abs/2411.14199)Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2025\)Z\. Chen, X\. Ma, S\. Zhuang, P\. Nie, K\. Zou, A\. Liu, J\. Green, K\. Patel, R\. Meng, M\. Su, S\. Sharifymoghaddam, Y\. Li, H\. Hong, X\. Shi, X\. Liu, N\. Thakur, C\. Zhang, L\. Gao, W\. Chen, and J\. LinBrowseComp\-Plus: a more fair and transparent evaluation benchmark of deep\-research agent\.arXiv preprint arXiv:2508\.06600\.Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px3.p1.1)\.
- Cormacket al\.\(2009\)G\. V\. Cormack, C\. L\. A\. Clarke, and S\. BüttcherReciprocal rank fusion outperforms condorcet and individual rank learning methods\.InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 758–759\.External Links:[Document](https://dx.doi.org/10.1145/1571941.1572114)Cited by:[§4](https://arxiv.org/html/2609.38670#S4.SS0.SSS0.Px2.p1.1)\.
- Cornell University \(n\.d\.\)Cornell UniversityarXiv dataset\.Note:Kaggle,[https://www\.kaggle\.com/datasets/Cornell\-University/arxiv](https://www.kaggle.com/datasets/Cornell-University/arxiv)CC0 metadata; local index documented September 2026; source version and download date unrecordedCited by:[§4](https://arxiv.org/html/2609.38670#S4.SS0.SSS0.Px2.p1.1)\.
- Fanget al\.\(2026\)Z\. Fang, S\. F\. Hu, Z\. Chang, Y\. Guo, Y\. Tao, H\. Liu, M\. Ruan, J\. Huang, and Y\. FangInference\-time budget control for LLM search agents\.arXiv preprint arXiv:2605\.05701\.Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px3.p1.1)\.
- Guoet al\.\(2026\)D\. Guo, J\. Wu, and S\. M\. YiuAgentEval: DAG\-structured step\-level evaluation for agentic workflows with error propagation tracking\.arXiv preprint arXiv:2604\.23581\.Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px2.p1.1)\.
- Heet al\.\(2025\)Y\. He, G\. Huang, P\. Feng, Y\. Lin, Y\. Zhang, H\. Li, and W\. EPaSa: an LLM agent for comprehensive academic paper search\.arXiv preprint arXiv:2501\.10120\.External Links:[Link](https://arxiv.org/abs/2501.10120v2)Cited by:[§1](https://arxiv.org/html/2609.38670#S1.p1.1),[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px1.p1.1)\.
- Kapooret al\.\(2024\)S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, N\. Nadgir, and A\. NarayananAI agents that matter\.arXiv preprint arXiv:2407\.01502\.External Links:[Link](https://arxiv.org/abs/2407.01502)Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2609.38670#S6.SS0.SSS0.Px4.p1.1)\.
- Liet al\.\(2026\)Y\. Li, W\. Deng, J\. Li, and X\. LiSpend less, reason better: budget\-aware value tree search for LLM agents\.arXiv preprint arXiv:2603\.12634\.Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px3.p1.1)\.
- Lianget al\.\(2026\)Z\. Liang, Y\. Liu, Y\. Huang, H\. Zhao, B\. Huang, J\. Wang, N\. Duan, and Q\. CaoSearchAuditor: auditing and attributing failures in long\-horizon search agents\.External Links:2608\.05212,[Link](https://arxiv.org/abs/2608.05212)Cited by:[§1](https://arxiv.org/html/2609.38670#S1.p2.1),[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px2.p1.1)\.
- Luet al\.\(2025\)J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, H\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. PangToolSandbox: a stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 1160–1183\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.65),[Link](https://aclanthology.org/2025.findings-naacl.65/)Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px2.p1.1)\.
- Maet al\.\(2024\)C\. Ma, J\. Zhang, Z\. Zhu, C\. Yang, Y\. Yang, Y\. Jin, Z\. Lan, L\. Kong, and J\. HeAgentBoard: an analytical evaluation board of multi\-turn LLM agents\.arXiv preprint arXiv:2401\.13178\.External Links:[Link](https://arxiv.org/abs/2401.13178v1)Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px2.p1.1)\.
- Robertson and Zaragoza \(2009\)S\. Robertson and H\. ZaragozaThe probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval\.External Links:[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§4](https://arxiv.org/html/2609.38670#S4.SS0.SSS0.Px2.p1.1)\.
- Ruet al\.\(2024\)D\. Ru, L\. Qiu, X\. Hu, T\. Zhang, P\. Shi, S\. Chang, C\. Jiayang, C\. Wang, S\. Sun, H\. Li, Z\. Zhang, B\. Wang, J\. Jiang, T\. He, Z\. Wang, P\. Liu, Y\. Zhang, and Z\. ZhangRAGChecker: a fine\-grained framework for diagnosing retrieval\-augmented generation\.arXiv preprint arXiv:2408\.08067\.External Links:[Link](https://arxiv.org/abs/2408.08067v2)Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px2.p1.1)\.
- Skarlinskiet al\.\(2024\)M\. D\. Skarlinski, S\. Cox, J\. M\. Laurent, J\. D\. Braza, M\. Hinks, M\. J\. Hammerling, M\. Ponnapati, S\. G\. Rodriques, and A\. D\. WhiteLanguage agents achieve superhuman synthesis of scientific knowledge\.arXiv preprint arXiv:2409\.13740\.External Links:[Link](https://arxiv.org/abs/2409.13740v1)Cited by:[§1](https://arxiv.org/html/2609.38670#S1.p2.1),[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px1.p1.1)\.
- Waddenet al\.\(2020\)D\. Wadden, S\. Lin, K\. Lo, L\. L\. Wang, M\. van Zuylen, A\. Cohan, and H\. HajishirziFact or fiction: verifying scientific claims\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7534–7550\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.609),[Link](https://aclanthology.org/2020.emnlp-main.609/)Cited by:[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px2.p1.1)\.
- Xionget al\.\(2026\)L\. Xiong, K\. Luo, Z\. Xia, W\. Zhang, J\. Yao, Z\. Liu, J\. Shao, J\. Chen, H\. Qian, X\. Yang, Q\. Yu, H\. Li, C\. Yue, X\. Du, Y\. Wang, Y\. Liu, H\. Xu, and Z\. DouAutoResearchBench: benchmarking AI agents on complex scientific literature discovery\.arXiv preprint arXiv:2604\.25256\.Note:[https://github\.com/CherYou/AutoResearchBench](https://github.com/CherYou/AutoResearchBench)Cited by:[§1](https://arxiv.org/html/2609.38670#S1.p1.1),[§1](https://arxiv.org/html/2609.38670#S1.p2.1),[§2](https://arxiv.org/html/2609.38670#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ASupplementary Raw and Keyword Comparisons

Table[6](https://arxiv.org/html/2609.38670#A1.T6)summarizes additional exposure and termination statistics for raw and keyword search\. Of the 476 keyword questions with target exposure, 336 first expose the target within rank 20 and 277 are answered incorrectly without target inspection\. Their intersection therefore contains at least336\+277−476=137336\+277\-476=137questions\. This lower bound concerns incorrect answers; the 184 uninspected top\-20 episodes in the main text include both correct and incorrect answers\.

Table 6:Supplementary exposure, outcome and resource statistics for raw and keyword search on 540 questions per condition\.Measureraw\-ftkw\-ftNever\-exposed failures \(%\)61\.711\.9Exposed, uninspected failures \(%\)12\.451\.3Inspected, incorrect failures \(%\)8\.112\.2Correct \(%\)17\.824\.6Search\-result lists containing the target per question1\.03\.8Uninspected\-failure stops: normal / turn / context limit47 / 4 / 16110 / 24 / 143Searches / evidence\-search calls per question \(rounded\)3\.6 / 8\.47\.5 / 7\.6Episodes ending without an arXiv IDin the final answer23873
## Appendix BMeasurement and Scoring Procedures

#### Identifier normalization and event definitions\.

Paper identifiers are normalized by removing whitespace, repository prefixes and version suffixes, accepting both modern and legacy arXiv formats\. For modern identifiers dated January 2015 or later, four\-digit sequences are padded to five digits\. Normalization standardizes identifier comparisons without independently validating paper identity\. First exposure is the earliest appearance of the target in a search\-result list, with rank counted from one\. Inspection requires an executed call directed at the target; proposed but unexecuted actions do not qualify\. Repeated identifiers outside search observations do not constitute new exposure\. Subsequent decisions include valid search, within\-paper evidence\-search and answer actions after exposure through episode termination\.

#### Returned target excerpts\.

Each checkpoint retains up to the first 1,800 characters of the latest evidence\-search response\. We identify a nonempty target excerpt when the response to an executed target call identifies that paper and contains query\-matched text with a source line number\. Empty responses, errors and returns without a match are excluded\. Episode\-level coverage requires at least one such excerpt, so it does not imply success for every call or sufficient evidence for the question\. Because checkpoints retain only part of each response, this measure concerns recorded excerpts rather than complete tool returns or model inputs\.

#### Primary outcome scoring\.

The evaluator compares the metadata title of the first candidate in the final answer with the target title in a single pass\. Absent or empty titles and no\-answer responses score zero in the answerable sample\. The judge assesses same\-paper identity, allowing truncation and differences in punctuation, capitalization or subtitles when identity is preserved; it does not independently verify the question’s constraints\. The original evaluation extracts a Boolean title\-match decision from the judge response, with a fallback based on the presence of an affirmative keyword when the response cannot be parsed\. Normalized target\-identifier agreement is evaluated separately and does not override the primary labels\.

#### Sensitivity re\-evaluation\.

All 2,700 answers are re\-evaluated using the original title prompts and a common judge configuration: the model request aliasgpt\-5\.6\-luna, temperature zero and medium reasoning effort\. The service does not provide an immutable underlying\-model snapshot or enforce the requested completion cap\. The sensitivity evaluation accepts only explicitly parsed Boolean decisions and leaves unresolved responses unscored, without using original labels to fill missing decisions\. All answers are resolved and agree with the primary labels\. This agreement supports label stability under the specified configuration; the original judge configuration remains incompletely documented\.

#### Query policies and gate eligibility\.

The RAW prompt requires the complete question verbatim in every search\. It allows changes in retrieval depth, evidence inspection and termination, while prohibiting query shortening, paraphrasing, term addition, title substitution and decomposition\. KW instructs the agent to formulate short, distinctive keyword queries before each search, allowing later queries to differ\. These are continuing prompt instructions; the execution framework requires nonempty queries but does not enforce verbatim adherence in RAW\. Under read\-first, eligibility is restricted to candidates in the latest result list that have a valid normalized identifier and no prior inspection attempt\. If no candidate qualifies, the next search or answer action proceeds without substitution\. Continuation after target exposure is measured by the presence of a subsequent valid search, inspection or answer action\.

#### Aggregate inspection outcomes\.

The keyword and read\-first conditions have the following aggregate outcomes, complementing the breakdown by call origin in Table[5](https://arxiv.org/html/2609.38670#S6.T5)\.

#### Inspection origin and unsuccessful returns\.

Evidence\-search calls are classified as gate\-forced when an executed inspection replaces the agent’s proposed search or answer action, and as agent\-initiated when the agent itself requests inspection\. Classification matches each executed action to its corresponding turn and paper identifier\. Among the 539 observable read\-first trajectories, 1,720 calls are gate\-forced and 3,478 are agent\-initiated\. The former reach the target on 22 questions through 22 calls; the latter reach it on 187 questions through 381 calls\. Ten questions belong to both groups, yielding twelve with forced\-only attempts and 177 with agent\-only attempts, for 199 questions overall\. The 5,198 calls correspond to 9\.63 per question over the full 540\-question sample\. Gate\-forced calls include 15 errors and no explicit no\-match returns, while agent\-initiated calls include 15 errors and three no\-match returns\. The episode without a trajectory remains unobservable\. These counts describe executed calls and recorded returns, without establishing a counterfactual increase in inspection or semantic evidence sufficiency\.

#### Paired bootstrap estimation\.

We quantify uncertainty in accuracy differences using 100,000 bootstrap samples for each policy comparison and resampling scheme\. For question\-level resampling, each question contributes a paired correctness difference of−1\-1,00or11, and bootstrap samples draw from the empirical distribution of these differences\. For cluster\-level resampling, questions are grouped by their exact target\-identifier set\. Each sample draws the original number of clusters uniformly with replacement and retains all questions within each selected cluster, including their multiplicity when a cluster is sampled more than once\. The accuracy difference is the sum of paired differences divided by the sampled question count, so clusters have equal selection probability but contribute in proportion to their question counts\. The 95% percentile intervals use the 2\.5th and 97\.5th percentiles with linear interpolation and are expressed in percentage points\. Both schemes are applied to the full sample and after excluding exploratory questions\. Resampling uses observed answer outcomes and does not repeat agent executions\.

#### Event\-label consistency\.

Automated comparisons with AI\-assisted inspection of 40 stratified trajectories and five additional anomaly records found no discrepancies between the recorded events and their summaries\. This check assesses agreement with the available trajectories rather than semantic evidence sufficiency\.

#### Exploratory assessment of semantic support\.

An AI\-assisted constraint\-level review examined 26 retained excerpts from ten selected records, rather than complete tool returns\. The review identified partial support and one unresolved ambiguity concerning the direction of a comparison\. These exploratory findings do not establish a benchmark error and do not provide independent human validation, demonstrate full evidence sufficiency, or attribute failures to reasoning\.