SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

# Auditing and Attributing Failures in Long-Horizon Search Agents
Source: [https://arxiv.org/html/2608.05212](https://arxiv.org/html/2608.05212)
Zhixiang Liang\\equalcontrib1,2, Yifei Liu\\equalcontrib2, Yidan Huang2, Haozhe Zhao1, Beichen Huang1, Jiaqi Wang2, Nan Duan2, Qiong Cao2\\corresponding

###### Abstract

Deep search agents tackle challenging questions through long\-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers\. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity\. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden\. SearchAuditBench comprises 1,243 failed trajectories, averaging 73\.1 messages and 65\.1K tokens, collected from eight open\-weight models on five deep\-search benchmarks, each expert\-annotated with the critical error step, a search\-specific root cause, and a reference repair with grading rubrics\. We further propose SearchAuditor, a multi\-perspective auditing framework that effectively localizes, attributes, and repairs search\-agent failures through evidence\-grounded adjudication\. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT\-5\.5, attains only a 26\.6% end\-to\-end pass rate\. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end\-to\-end pass rate of 32\.3%, and resuming failed runs with its repairs enables agents to better recover from errors\.1

## 1\. Introduction

The rapid progress in large language models \(LLMs\) has enabled increasingly capable long\-horizon agents\(OpenAI[2026](https://arxiv.org/html/2608.05212#bib.bib24); GLM\-5\-Team[2026](https://arxiv.org/html/2608.05212#bib.bib17)\)\. Among them, search agents have emerged as a prominent application\(Nakanoet al\.[2021](https://arxiv.org/html/2608.05212#bib.bib14); Weiet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib13)\)\. Given an information\-seeking question, the agent iteratively formulates queries, browses and reads web pages, and maintains candidate answers before committing to a final response\(Liet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib25)\)\. Despite this, search agents still remain brittle over extended search processes, where small reasoning errors compound across steps and lead to confidently wrong answers\(Zhanet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib26)\)\. Understanding these failures is essential to improving agents, yet it requires developers to manually inspect long execution traces to determine what went wrong, a process that scales poorly with trajectory length and volume\(Cemriet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib29)\)\. Automatically auditing and attributing*where*and*how*a long\-horizon search agent fails remains an under\-explored problem\.

![Refer to caption](https://arxiv.org/html/2608.05212v1/x1.png)Figure 1:Illustration of the trajectory auditing task: given a query, the agent’s wrong answer, and a failed trajectory, the auditor must localize the critical step, attribute a root cause, and prescribe a repair\.Recent work increasingly studies failure diagnosis over agent trajectories, including failure attribution in multi\-agent systems\(Zhanget al\.[2025b](https://arxiv.org/html/2608.05212#bib.bib27),[a](https://arxiv.org/html/2608.05212#bib.bib28)\), constraint\-based diagnosis from execution logs\(Barkeet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib38)\), dependency\-guided fault localization\(Rafiet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib33)\), and domain\-specific tracing for memory and code agents\(Denget al\.[2026](https://arxiv.org/html/2608.05212#bib.bib36); Liet al\.[2026a](https://arxiv.org/html/2608.05212#bib.bib35)\)\. However, these efforts rarely target long\-horizon search agents, whose trajectories pose distinctive diagnostic challenges\. \(1\)Scale and noise: a trajectory interleaves dozens of searches and page visits with lengthy observations, burying the critical error among many benign steps\(Gouet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib30); Weiet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib13)\)\. \(2\)Lack of verifiable structure: unlike code agents checkable against executable tests or workflow agents with schema\-defined invariants, open\-web search offers no oracle; a step’s correctness depends on evidence chains over noisy, ephemeral web content\(Krishnaet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib31)\)\. \(3\)Silently propagating failures: search agents rarely crash but fail subtly, trusting unverified sources, overlooking key evidence, or mismanaging candidates, and one early misstep propagates through subsequent reasoning while the final answer still appears fluent\(Chenet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib32)\)\. Compounding these challenges, the community lacks a public, expert\-annotated corpus of failed search trajectories labeled with critical error steps and root causes, which is essential for analyzing failure patterns and rigorously evaluating automated auditors\.

To address this gap, we present a systematic study of auditing and attributing failures in long\-horizon search agents, with three contributions\. First, we constructSearchAuditBench, an expert\-annotated benchmark of 1,243 failed trajectories, collected by running eight open\-weight agents on five deep\-search benchmarks under a unified scaffold\. Each instance is annotated, via LLM\-assisted screening and expert annotation, with \(i\) a critical error step and its tolerance span, \(ii\) a root cause from a six\-way search\-specific taxonomy, and \(iii\) a reference repair with atomic rubrics, which together define the three auditing tasks \(Figure 1\)\. The corpus also reveals how search agents fail: 45\.9% of critical errors occur in the final third of a trajectory, and the dominant root cause, Candidate Mismanagement, accounts for 27\.2% of failures\. Second, we benchmark LLM auditors on these tasks and find auditing highly challenging: even the strongest baseline, powered by a frontier model like GPT\-5\.5, reaches only a 26\.6% end\-to\-end fully passed score\. Third, we proposeSearchAuditor, which runs three complementary audit branches in parallel, adjudicates their reports over a compact evidence\-grounded view of the trajectory, and synthesizes process\-level repair directives conditioned on the adjudicated diagnosis\. It consistently outperforms all baselines across three frontier backbones, achieving an end\-to\-end pass rate of 32\.3%, and its repairs fix 17\.4% of Kimi\-K2\.6’s failed runs on LiveBrowseComp\(Fanet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib21)\), lifting accuracy from 34\.0% to 45\.1%\.

## 2\. Related Work

Recent work moves agent evaluation beyond final\-answer correctness toward explaining where and why a trajectory fails\. For single\-agent trajectories, TRAIL\(Deshpandeet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib20)\)localizes errors to trace spans under a taxonomy of reasoning, execution, and planning errors, while AgentRx\(Barkeet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib38)\)diagnoses critical errors via constraints from logs and tool specifications\. In multi\-agent systems, Who&When\(Zhanget al\.[2025b](https://arxiv.org/html/2608.05212#bib.bib27)\)attributes failures to the responsible agent and its decisive step, and AgenTracer\(Zhanget al\.[2025a](https://arxiv.org/html/2608.05212#bib.bib28)\)scales supervision via counterfactual replay and fault injection\. Domain\-specific tracers further target agent memory\(Denget al\.[2026](https://arxiv.org/html/2608.05212#bib.bib36)\), coding\-agent states\(Wanget al\.[2026](https://arxiv.org/html/2608.05212#bib.bib34)\), and CLI trajectories\(Zhaoet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib18)\)\. However, none targets long\-horizon search agents, whose trajectories are far longer and noisier and lack the verifiable oracles these methods rely on\. We therefore introduce SearchAuditBench, an expert\-annotated benchmark of 1,243 failed deep\-search trajectories \(compared with prior benchmarks in Table[1](https://arxiv.org/html/2608.05212#Sx2.T1)\), and SearchAuditor, a multi\-perspective auditing framework with evidence\-grounded adjudication\.

Table 1:Comparison with prior trajectory\-level failure attribution benchmarks\.![Refer to caption](https://arxiv.org/html/2608.05212v1/x2.png)Figure 2:Overview of SearchAuditBench: construction pipeline \(top\), key statistics \(a\), one annotated instance \(b\), and the six\-way root cause taxonomy \(c\)\.
## 3\. SearchAuditBench

SearchAuditBench evaluates an LLM’s ability to serve as a*trajectory auditor*for search agents: given a failed search trajectory, the auditor must pinpoint*where*the failure occurred, explain*what*went wrong, and prescribe*how*to repair it\. Each instance is annotated with \(i\) a critical error step with a tolerance span, \(ii\) a root cause from a six\-way failure taxonomy, and \(iii\) an expert\-written repair directive paired with atomic grading rubrics\.

### 3\.1 Trajectory Collection

To construct SearchAuditBench, we restrict the model pool to open\-weight models whose trajectories expose explicit intermediate reasoning, which is necessary for fine\-grained trajectory auditing\. Auditing tool\-only or hidden\-reasoning traces is an important but distinct setting, which we leave to future work\. Under this criterion, we collect trajectories from eight models spanning two categories\. The first includes four large\-scale general models: GLM\-5\.2\(GLM\-5\-Team[2026](https://arxiv.org/html/2608.05212#bib.bib17)\), Kimi\-K2\.6\(Kimi\-Team[2026](https://arxiv.org/html/2608.05212#bib.bib37)\), DeepSeek\-V4\-Pro\(DeepSeek\-AI[2026](https://arxiv.org/html/2608.05212#bib.bib39)\), and Qwen3\.5\-397B\-A17B\(Yanget al\.[2025](https://arxiv.org/html/2608.05212#bib.bib42)\)\. The second includes four smaller models specialized for deep research: OpenSeeker\(Duet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib41)\), OpenResearcher\(Liet al\.[2026b](https://arxiv.org/html/2608.05212#bib.bib16)\), Quest\-35B\(Xieet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib15)\), and Tongyi\-DeepResearch\-30B\-A3B\(Tongyi\-DeepResearch\-Teamet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib43)\)\. Queries are drawn from five deep\-search benchmarks: BrowseComp\(Weiet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib13)\), BrowseComp\-ZH\(Zhouet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib44)\), DeepSearchQA\(Guptaet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib46)\), Seal\-0\(Phamet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib45)\), and xBench\-DeepSearch\(Chenet al\.[2025](https://arxiv.org/html/2608.05212#bib.bib47)\)\. To isolate model behavior from scaffold effects, all eight models run under a unified scaffold adapted from Tongyi DeepResearch\(Tongyi\-DeepResearch\-Teamet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib43)\), which equips every model with two tools:searchandvisit\. We retain only trace\-auditable failures: trajectories whose final answers are judged as incorrect by an LLM\-based evaluator against gold answers, and for which annotators can identify and justify the critical error using evidence contained in the frozen trajectory\. This yields 1,243 failed trajectories, with a balanced per\-model distribution\.

### 3\.2 Task Design

Each benchmark input is a triplex=\(q,a^,τ\)x=\(q,\\hat\{a\},\\tau\), whereqqis the user query,τ=\(m1,…,mT\)\\tau=\(m\_\{1\},\\ldots,m\_\{T\}\)is the complete trajectory of interleaved reasoning and tool interactions, indexed by message position, anda^\\hat\{a\}is the incorrect final answer thatτ\\tauterminates in, restated explicitly as the failure to be audited\. We refer to the assistant messages ofτ\\tauas its*decision steps*: each one pairs a reasoning move with the tool call it issues, or emits the final answera^\\hat\{a\}\. All gold and predicted step indices in this paper \(k∗k^\{\*\},k^\\hat\{k\}\) are message positions of decision steps; tool\-return messages are never candidate steps\. An auditor𝒜\\mathcal\{A\}must produce a structured diagnosis

𝒜​\(q,a^,τ\)=\(k^,c^,r^\),\\mathcal\{A\}\(q,\\hat\{a\},\\tau\)\\;=\\;\(\\hat\{k\},\\,\\hat\{c\},\\,\\hat\{r\}\),\(1\)corresponding to three sub\-tasks\.

#### \(1\) Critical\-step localization\.

The auditor must pinpoint where the failure originates by predicting a critical error stepk^\\hat\{k\}\. We define the gold critical stepk∗k^\{\*\}as the earliest decision step at which the annotated failure mechanism becomes established, so that correctingk∗k^\{\*\}would remove this mechanism and make a correct final answer attainable under competent subsequent search\. Because the exact step at which the failure becomes established can be ambiguous, annotators also provide a tight tolerance span\[ks,ke\]\[k\_\{s\},k\_\{e\}\]: the smallest contiguous window of messages whose decision steps are all acceptable localizations of the critical error, withk∗∈\[ks,ke\]k^\{\*\}\\in\[k\_\{s\},k\_\{e\}\]by construction\. Predictions are evaluated under both strict \(k^=k∗\\hat\{k\}=k^\{\*\}\) and loose \(k^∈\[ks,ke\]\\hat\{k\}\\in\[k\_\{s\},k\_\{e\}\]\) matching\.

#### \(2\) Root\-cause classification\.

We develop a six\-way taxonomy of error categories \(Figure[2](https://arxiv.org/html/2608.05212#Sx2.F2)\): Candidate Mismanagement, Search Coverage Gap, Constraint Neglect, Unverified Source Reliance, Entity\-Relation Misbinding, and Unsupported Answer\. The auditor must diagnose*what*went wrong by assigning a root causec^∈𝒞\\hat\{c\}\\in\\mathcal\{C\}from this taxonomy\. The gold labelc∗c^\{\*\}is the expert\-annotated root cause\. Full definitions of the six categories are provided in Appendix A\.

#### \(3\) Repair directive\.

The auditor must prescribe*how*to repair the failure by producingr^\\hat\{r\}, a concrete revision of the agent’s behavior from the critical step onward\. Repairs are graded against three to five expert\-written atomic rubrics\. Grading is*diagnosis\-gated*:r^\\hat\{r\}is scored only whenk^\\hat\{k\}falls within the tolerance span andc^\\hat\{c\}is correct, so that repair quality is measured conditional on a sound diagnosis\. Annotation details for all three tasks are given in Section 3\.3\.

#### Auditing setup\.

Auditing is strictly offline: the auditor sees only\(q,a^,τ\)\(q,\\hat\{a\},\\tau\)and knows the run failed, but receives neither the gold answer nor any web or tool access, so every claim must be grounded in the trajectory itself\. We therefore evaluate only failures whose critical error is identifiable from the frozen trajectory \(Section 3\.1\)\. This prevents diagnosis from collapsing into post\-hoc problem solving, and keeps evaluation reproducible, since a frozen trajectory judges all auditors on identical evidence while live search results drift over time\. Section 5 details the evaluation protocol and metrics\.

![Refer to caption](https://arxiv.org/html/2608.05212v1/x3.png)Figure 3:Overview of SearchAuditor: parallel multi\-perspective auditing, evidence\-grounded adjudication, and diagnosis\-conditioned repair synthesis\.

### 3\.3 Label Annotation

#### Annotation workflow\.

Four annotators from the author team, each with over one year of research experience on LLM agents, perform the annotation\. To keep long trajectories tractable, an LLM assistant first proposes candidate error regions with rationales, which annotators verify, correct, or discard before finalizing labels\. Every case is labeled by a single annotator, with cases split disjointly across the four annotators; more annotation details are in Appendix B\.

#### Critical step and tolerance span\.

Annotators mark the critical stepk∗k^\{\*\}and its tolerance span\[ks,ke\]\[k\_\{s\},k\_\{e\}\]following Section 3\.2\. When problematic evidence first appears in a tool return, they label the decision step that issued the call if the call itself was erroneous, otherwise the first decision step that misinterpreted or misused the returned evidence\.

#### Root cause\.

When a trajectory contains multiple flaws, annotators label the single most direct, earliest failure\-establishing error rather than its downstream symptoms, so each trajectory carries exactly one root cause\.

#### Repair directives and rubrics\.

Annotators write a repair directive and decompose it into three to five atomic rubrics, each stating one requirement a valid repair must satisfy\. Rubrics must be case\-specific and process\-level, constraining how the search is revised from the critical step onward rather than what the final answer is\.

### 3\.4 Dataset Statistics

Figure 2\(a\) summarizes the key statistics of SearchAuditBench \(extended statistics in the Appendix C\)\. Failed trajectories are long and evidence\-dominated: on average, each spans 73\.1 messages and 65\.1K tokens, of which 49\.1K are tool returns against only 15\.3K of captured reasoning, so the critical error is buried in raw evidence\. Model behaviors also vary widely: per failed run, mean tool calls range from 7\.4 \(Tongyi\-DeepResearch\-30B\-A3B\) to 65\.9 \(GLM\-5\.2\) and trajectory tokens from 24K to 118\.3K, while the search\-to\-visit mix spans four searches per visit \(Kimi\-K2\.6\) to roughly one\-to\-one \(OpenResearcher\)\. Auditing is thus doubly challenging: an auditor must pinpoint a single critical step buried in long, noisy traces, while generalizing across sharply different trajectory shapes and tool\-use patterns\.

## 4\. SearchAuditor

Auditing a failed search trajectory requires pinpointing the earliest decision that established the failure within tens of thousands of tokens of accumulated evidence \(Section 3\.4\)\. Such a failure is far more visible in its downstream symptoms, so a single pass over the trace tends to settle on a later step and leaves that verdict unchecked\. SearchAuditor therefore decouples proposing candidate diagnoses from adjudicating them, mapping a failed trajectory to the complete audit output\(k^,c^,r^\)\(\\hat\{k\},\\hat\{c\},\\hat\{r\}\)in three stages \(Figure[3](https://arxiv.org/html/2608.05212#Sx3.F3)\)\. First, three audit branches analyze the same trajectory in parallel from complementary perspectives\. Second, an adjudicator inspects the trajectory evidence around their proposed critical steps and jointly selects the critical step and root cause\. Third, a repair synthesizer converts the fixed diagnosis into a small set of actionable, process\-level directives\.

#### Multi\-perspective auditing\.

The three audit branches run in parallel and use the same modelfθf\_\{\\theta\}, differing only in the audit procedure that each promptpjp\_\{j\}enforces\.

zj=fθ​\(q,a^,τ;pj\),j∈\{1,2,3\}\.z\_\{j\}\\;=\\;f\_\{\\theta\}\\\!\\left\(q,\\,\\hat\{a\},\\,\\tau;\\,p\_\{j\}\\right\),\\qquad j\\in\\\{1,2,3\\\}\.\(2\)The use of distinct procedures is intended to expose perspective\-dependent disagreements for the adjudicator to resolve\. Unless otherwise stated, all stages share the same backbone:fθf\_\{\\theta\}andgθg\_\{\\theta\}denote role\-specific prompting configurations of one model rather than separately trained components, with all prompts given in Appendix G\. Each reportzjz\_\{j\}contains a candidate critical step, a root cause, and a failure rationale; the two specialized audits additionally include their intermediate analysis in their reports as audit notes for the adjudicator\. The three procedures, enforced byp1p\_\{1\},p2p\_\{2\}, andp3p\_\{3\}respectively, are as follows:

1. \(a\)The*holistic audit*diagnoses the trajectory directly, without a prescribed decomposition, catching failures that the two specialized audits below might miss\.
2. \(b\)The*backward constraint audit*parses the queryqqinto a set of constraints and traces each constraint unsupported by trajectory evidence back to the earliest decision where the agent committed to a search direction or candidate answer without first verifying it; when multiple constraints implicate different decisions, it reports the earliest one as its candidate critical step\.
3. \(c\)The*forward timeline audit*first derives a minimal query\-grounded plan covering the explicit requirements ofqq, then walks through the decision steps in order and targets the earliest decision step that establishes the failure path rather than a later one that merely repeats or confirms the commitment\.

#### Evidence\-grounded adjudication\.

Deterministic rules first organize the three reports into a structured adjudication context: disagreements are tabulated into a summary, and nearby critical\-step proposals are grouped into clusters for comparison\. Each distinct proposal only anchors where evidence is extracted and does not fix the final localization\. Around every proposed critical step and the final\-answer step, we extract a bounded window of original messages immediately surrounding the anchor\. A single adjudicatorgθg\_\{\\theta\}then re\-decides the two diagnosis fields,

\(k^,c^\)=gθ​\(q,a^,τ,Z,O,W\),\(\\hat\{k\},\\hat\{c\}\)=g\_\{\\theta\}\\\!\\left\(q,\\,\\hat\{a\},\\,\\tau,\\,Z,\\,O,\\,W\\right\),\(3\)
whereZZcollects the three audit reports together with the summary of their disagreements, andOOis a deterministic outline that lists every decision step’s message position with a truncated content preview, supplying the global structure that the local windowsWWlack\. The adjudicator’s decision must cite supporting messages and follow three rules: \(i\) agreement among reports is a diagnostic signal rather than a vote, and a minority report may prevail when the cited evidence supports it; \(ii\) the root cause and the critical step may be drawn from different reports, but the pair must be consistent, with the step being the earliest point at which the cause is instantiated; \(iii\) localization prefers the earliest supported decision step, and when an error surfaces in a tool output, it targets the decision step that issued an erroneous call, or otherwise the first decision step that misused the returned evidence, never the tool message itself\.

#### Diagnosis\-conditioned repair synthesis\.

A final synthesizer receives onlyqq,a^\\hat\{a\}, the adjudicated diagnosis with its rationale, and evidence windows newly extracted at the critical step and the final answer\. Its outputr^\\hat\{r\}consists of case\-specific directives executable from the critical step onward, each describing a process change that addresses the diagnosed failure mechanism rather than stating the target answer\.

ModelMethodsCritical StepRoot CauseDiagnosisRepairEnd\-to\-EndCS\-Strict↑\\uparrowCS\-Loose↑\\uparrowRC\-Acc↑\\uparrowRC\-F1↑\\uparrowDiag↑\\uparrowRep@Diag↑\\uparrowFPS↑\\uparrowGPT\-5\.5All\-at\-Once38\.2951\.8951\.9749\.0632\.9080\.6826\.55Step\-by\-Step31\.3045\.5346\.9042\.4227\.0379\.4621\.48Binary Search39\.0251\.3346\.6644\.6329\.3683\.2924\.46AgentRx36\.8551\.0150\.9348\.5132\.6654\.1917\.70SearchAuditor44\.8958\.7354\.7951\.1038\.0584\.7832\.26Gemini\-3\.1\-ProAll\-at\-Once28\.8043\.7741\.5936\.7525\.3453\.6513\.59Step\-by\-Step22\.6935\.9635\.0830\.1916\.6558\.539\.75Binary Search29\.1241\.1939\.9836\.4520\.7655\.8111\.58AgentRx32\.1845\.0540\.4736\.1724\.3842\.2410\.30SearchAuditor36\.1248\.4344\.1739\.0630\.5761\.8418\.91Claude\-Opus\-4\.8All\-at\-Once32\.1047\.0647\.6345\.4328\.2471\.5120\.19Step\-by\-Step23\.0937\.6537\.4936\.1518\.9168\.9413\.03Binary Search32\.8247\.0643\.3641\.5525\.3470\.4817\.86AgentRx35\.4049\.7242\.9640\.9226\.7146\.3912\.39SearchAuditor39\.9054\.7950\.6847\.0533\.7173\.0324\.62Table 2:Main results on SearchAuditBench \(N=1,243N=1\{,\}243\)\.

## 5\. Experiments

### 5\.1 Experimental Setup

#### Baselines & Models\.

We compare SearchAuditor against two families of baselines\. \(i\)*Direct prompting*: the three attribution strategies fromZhanget al\.\([2025b](https://arxiv.org/html/2608.05212#bib.bib27)\), each adapted to emit our full output triple \(critical step, root cause, and repair directive\)\.All\-at\-Onceperforms a single holistic judge call over the full trajectory;Step\-by\-Stepwalks the trajectory prefix by prefix and commits at the first critical error step it thinks decisive; andBinary Searchrecursively halves the trajectory to localize the critical segment\. \(ii\)*Framework*:AgentRx\(Barkeet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib38)\), the closest prior framework to ours, which diagnoses a failed trajectory via an LLM judge over synthesized constraint violations; we adapt its constraint synthesis to the search agent’s tool schema and extend its judge to emit our full output triple\. We instantiate every auditor on three frontier models: GPT\-5\.5\(OpenAI[2026](https://arxiv.org/html/2608.05212#bib.bib24)\), Gemini\-3\.1\-Pro\(Google DeepMind[2026](https://arxiv.org/html/2608.05212#bib.bib22)\), and Claude\-Opus\-4\.8\(Anthropic[2026](https://arxiv.org/html/2608.05212#bib.bib23)\), with reasoning effort set to high\. Repair rubrics are graded by DeepSeek\-V4\-Flash\(DeepSeek\-AI[2026](https://arxiv.org/html/2608.05212#bib.bib39)\), which checks the predicted repair against each expert\-written rubric and passes a case only if every rubric is satisfied; Appendix D validates this grader against human judgments, and Appendix E reports the cost and efficiency of all auditors\.

#### Evaluation Metrics\.

All metrics are computed over the full benchmark \(N=1,243N\{=\}1\{,\}243\) and reported in percentage points\. For critical\-step localization, CS\-Strict is the fraction of cases whose predicted step exactly matches the gold critical step \(k^=k∗\\hat\{k\}=k^\{\*\}\), and CS\-Loose relaxes this to falling within the annotated tolerance span \(k^∈\[ks,ke\]\\hat\{k\}\\in\[k\_\{s\},k\_\{e\}\]\)\. For root\-cause classification, RC\-Acc is exact\-match accuracy over the six\-way taxonomy and RC\-F1 is the macro\-average of per\-class F1 scores over the six categories\. Diag is the fraction of cases that achieve a*sound diagnosis*, i\.e\., a correct root cause together with a predicted step inside the tolerance span\. Rep@Diag is computed only over soundly diagnosed cases, as repair grading is diagnosis\-gated: it is the fraction of those cases whose repair satisfies*all*expert\-written rubrics\. Finally, the end\-to\-end fully\-passed score FPS is the fraction of all cases that simultaneously achieve a sound diagnosis and a fully\-passing repair, so thatFPS=Diag×Rep@Diag\\text\{FPS\}=\\text\{Diag\}\\times\\text\{Rep@Diag\}\.

### 5\.2 Main Results

As shown in Table[2](https://arxiv.org/html/2608.05212#Sx4.T2), we summarize key observations below\.

#### Universal performance improvement\.

SearchAuditor outperforms every baseline on every metric under all three backbones\. With GPT\-5\.5, it reaches 44\.89% CS\-Strict and 58\.73% CS\-Loose, exceeding the strongest baseline by 5\.87 and 6\.84 points, and raises the end\-to\-end FPS from 26\.55% to 32\.26%\. The gains are not tied to a single model, holding at roughly 4 to 5 points in CS\-Strict and FPS on the other two backbones \(Table[2](https://arxiv.org/html/2608.05212#Sx4.T2)\)\. They also cover both stages of the audit: SearchAuditor attains the best Diag and the best Rep@Diag in every block, improving Diag over the strongest baseline by more than five points on each backbone, so diagnosis and repair improvements multiply into the end\-to\-end score\. These results highlight SearchAuditor’s broad effectiveness across localization, attribution, repair, and its robustness to the choice of backbone\.

#### Trajectory auditing remains challenging\.

Even the best configuration, SearchAuditor with GPT\-5\.5, localizes the critical step exactly in fewer than half of the cases \(44\.89% CS\-Strict\) and fully passes fewer than one third of audits \(32\.26% FPS\); even under the loose criterion, more than 40% of cases are still mislocalized, and root\-cause attribution stays around or below 55% accuracy for all auditors\. The baselines fall well below these levels: the strongest one reaches only 39\.02% CS\-Strict and 26\.55% FPS, and with weaker backbones the end\-to\-end score drops to around 10%\. These results reflect the intrinsic difficulty of the task: as characterized in Section 3\.4, failed trajectories are long, evidence\-dominated, and nearly half of critical errors emerge only in the final third, demanding long\-range integration before committing to a localization\.

### 5\.3 Ablation Study

Table 3:Ablation of SearchAuditor on a 300\-instance subset with GPT\-5\.5\. H, B, F denote the holistic, backward constraint, and forward timeline audits\.We ablate SearchAuditor on a randomly sampled 300\-instance subset with GPT\-5\.5 as the backbone \(Table[3](https://arxiv.org/html/2608.05212#Sx5.T3)\); all variants retain the three\-stage pipeline\.*w/ 3×\\timesHolistic*replaces both specialized audits with holistic copies \(H\+H\+H\) at identical compute, while*w/o Forward*and*w/o Backward*replace only one;*w/ Vote*replaces the LLM adjudicator with deterministic aggregation \(majority vote over root causes, fixed tie\-breaking over nearby step proposals\);*w/oOO\+WW*removes the outline and evidence windows from the adjudicator’s input\.

We have the following observations\. \(i\)Heterogeneous perspectives beat repeated sampling\.At identical compute,*w/ 3×\\timesHolistic*drops CS\-Strict by 3\.0 points and FPS by 4\.3, so the gains come from complementary procedures rather than ensembling\. Removing the forward timeline audit hurts most \(4\.0 points in CS\-Strict, 7\.0 in FPS\), even more than removing both specialized audits, as it is the only branch that explicitly targets the step at which the failure becomes established\. \(ii\)Evidence\-grounded adjudication outperforms voting\.*w/ Vote*saves one model call but drops CS\-Strict by 4\.7 points and FPS by 6\.3, indicating that cross\-report disagreements must be resolved against trajectory evidence rather than by counting reports\. \(iii\)Structured evidence views matter even with the full trajectory\.*w/oOO\+WW*drops CS\-Strict by 4\.3 points and Rep@Diag by 6\.2: the outline and windows act as structured anchors that concentrate adjudication on contested steps, which the full trajectory alone does not provide\.

### 5\.4 Boosting Search Agents with Audits

We next ask whether audits carry practical value: can they help a failing search agent recover? We inject audit outputs back into failed runs on a held\-out benchmark, where the only signal is whether the resumed run reaches the correct answer, giving an annotation\-free validation of diagnostic utility\.

#### Setup\.

We adopt LiveBrowseComp\(Fanet al\.[2026](https://arxiv.org/html/2608.05212#bib.bib21)\), a deep\-search benchmark of 335 questions whose answers depend on recently published facts, disjoint from the five source benchmarks of SearchAuditBench\. We run Kimi\-K2\.6 and Quest\-35B under the scaffold of Section 3\.1\. Their original runs solve 34\.03% and 8\.96% of the questions; the failed runs that terminate with a gradable final answer \(213 and 300\) form the repair cohorts, and the rest count as incorrect in Acc\. Each failed trajectory is diagnosed with GPT\-5\.5 as the auditor backbone, and every intervention resumes the run by keeping the trajectory prefix up to the predicted critical step, appending the repair directive as a user message, and letting the agent continue under the original scaffold\. We compare five interventions:

- •*Unguided retry*: a fresh rerun from the original query, equivalent to pass@2 on the failed set\.
- •*Generic hint*: a fixed prompt \(“Reconsider your approach carefully before continuing\.”\) injected at SearchAuditor’s predicted critical step, keeping its localization but stripping the diagnosis\.
- •*Method repair*: the repair directive of All\-at\-Once, AgentRx, or SearchAuditor, injected at each method’s own predicted critical step\.

Table 4:Repair\-guided resumption on LiveBrowseComp\. Fix Rate is over the 213 and 300 verifiable failed runs; Acc\. is over all 335 questions\.
#### Recovery results\.

SearchAuditor’s repairs correct 17\.37% of Kimi\-K2\.6 and 10\.33% of Quest\-35B failures, roughly doubling unguided retry and lifting overall accuracy to 45\.07% and 18\.21% \(Table[4](https://arxiv.org/html/2608.05212#Sx5.T4)\)\. No other guided intervention clearly improves on unguided retry, which we attribute to resumption asymmetry: a resumed run inherits the prefix that produced the failure, so inaccurate guidance anchors the agent to its flawed reasoning, whereas a fresh restart at least escapes it\. Notably, the generic hint shares SearchAuditor’s localization exactly yet trails it by 11\.74 and 5\.66 points, confirming that the repair, not merely the localization, drives the recovery gain\.

### 5\.5 Analysis of Search\-Agent Failures

Beyond enabling auditor evaluation, the corpus reveals how long\-horizon search agents fail\. We highlight three findings\.

#### Most failures are not retrieval failures\.

Only 22\.8% of failures stem from*Search Coverage Gap*, where the decisive evidence was never retrieved\. In the remaining 77\.2%, the primary error lies not in what the agent retrieved but in how it processed what it had, led by*Candidate Mismanagement*\(27\.2% of the corpus\) and*Constraint Neglect*\(19\.1%\)\. Indeed, in 25\.0% of all failures the correct answer appears verbatim in the retrieved content, and in 83% of these it was retrieved at or before the critical step\. This “answer\-in\-hand” rate reaches 35\.6% for*Candidate Mismanagement*but only 12\.4% for*Search Coverage Gap*, indicating that the primary bottleneck is evidence utilization rather than retrieval\.

#### Failures propagate silently and waste nearly half of agent compute\.

Search agents rarely halt after going wrong\. After the critical error step, they continue for a mean of 16\.7 further tool calls\. Corpus\-wide, 47\.1% of generated tokens and 45\.4% of tool calls occur*after*the decisive error, without ever repairing it\. Reliable mid\-trajectory auditing could in principle reclaim this wasted compute, consistent with the recovery gains from repair\-guided resumption in Section 5\.4\.

![Refer to caption](https://arxiv.org/html/2608.05212v1/x4.png)Figure 4:Search Coverage Gap failures \(%\) vs\. median tool calls per failed trajectory across the eight agents\.
#### Scaling exploration moves failures downstream rather than removing them\.

Exploration budgets vary widely across agents, from a median of 5 tool calls per failed run to 75\. As Figure[4](https://arxiv.org/html/2608.05212#Sx5.F4)shows, the fraction of failures attributed to*Search Coverage Gap*correlates strongly and negatively with this budget \(Pearsonr=−0\.88r=\-0\.88\): sparse explorers fail by not looking \(29% to 33%*Search Coverage Gap*\), while heavy explorers largely eliminate coverage failures \(16% to 19%\) but shift toward*Candidate Mismanagement*and*Constraint Neglect*, which together exceed half of their failures\. Test\-time scaling of search is therefore self\-limiting: beyond a certain budget, the binding constraint shifts from evidence coverage to the agent’s ability to manage candidates and constraints over an ever\-longer context\.

## 6\. Conclusion

In this work, we presented a systematic study of auditing and attributing failures in long\-horizon search agents\. We constructed SearchAuditBench, an expert\-annotated benchmark of 1,243 failed trajectories labeled with critical error steps, root causes, and repair rubrics, and proposed SearchAuditor, a multi\-perspective auditing framework with evidence\-grounded adjudication\. Extensive experiments show that SearchAuditor consistently outperforms all baselines across three frontier backbones, and that resuming failed runs with its repairs enables agents to effectively recover from errors\. Beyond evaluating auditors, the corpus also offers an empirical anatomy of how long\-horizon search agents fail\.

## References

- Claude opus 4\.8 system card\.External Links:[Link](https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80fef830a37ea.pdf)Cited by:[Baselines & Models\.](https://arxiv.org/html/2608.05212#Sx5.SSx1.SSS0.Px1.p1.1)\.
- S\. Barke, A\. Goyal, A\. Khare, A\. Singh, S\. Nath, and C\. Bansal \(2026\)AgentRx: diagnosing ai agent failures from execution trajectories\.External Links:2602\.02475,[Link](https://arxiv.org/abs/2602.02475)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1),[2\. Related Work](https://arxiv.org/html/2608.05212#Sx2.p1.1),[Baselines & Models\.](https://arxiv.org/html/2608.05212#Sx5.SSx1.SSS0.Px1.p1.1)\.
- M\. Cemri, M\. Z\. Pan, S\. Yang, L\. A\. Agrawal, B\. Chopra, R\. Tiwari, K\. Keutzer, A\. Parameswaran, D\. Klein, K\. Ramchandran, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica \(2026\)Why do multi\-agent LLM systems fail?\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=fAjbYBmonr)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p1.1)\.
- B\. Chen, B\. Li, P\. Nie, Y\. Zhang, X\. Ye, and C\. Zhao \(2026\)Beyond single\-shot writing: deep research agents are unreliable at multi\-turn report revision\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 13325–13356\.External Links:[Link](https://aclanthology.org/2026.acl-long.609/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.609),ISBN 979\-8\-89176\-390\-6Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1)\.
- K\. Chen, Y\. Ren, Y\. Liu, X\. Hu, H\. Tian, T\. Xie, F\. Liu, H\. Zhang, H\. Liu, Y\. Gong, C\. Sun, H\. Hou, H\. Yang, J\. Pan, J\. Lou, J\. Mao, J\. Liu, J\. Li, K\. Liu, K\. Liu, R\. Wang, R\. Li, T\. Niu, W\. Zhang, W\. Yan, X\. Wang, Y\. Zhang, Y\. Hung, Y\. Jiang, Z\. Liu, Z\. Yin, Z\. Ma, and Z\. Mo \(2025\)Xbench: tracking agents productivity scaling with profession\-aligned real\-world evaluations\.External Links:2506\.13651,[Link](https://arxiv.org/abs/2506.13651)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1),[Baselines & Models\.](https://arxiv.org/html/2608.05212#Sx5.SSx1.SSS0.Px1.p1.1)\.
- X\. Deng, R\. Zhong, H\. Peng, X\. Lu, Y\. Wu, G\. Li, B\. Xu, Y\. Yao, J\. Fang, H\. Cao,et al\.\(2026\)MemTrace: tracing and attributing errors in large language model memory systems\.arXiv preprint arXiv:2605\.28732\.Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1),[2\. Related Work](https://arxiv.org/html/2608.05212#Sx2.p1.1)\.
- D\. Deshpande, V\. Gangal, H\. Mehta, J\. Krishnan, A\. Kannappan, and R\. Qian \(2025\)TRAIL: trace reasoning and agentic issue localization\.External Links:2505\.08638,[Link](https://arxiv.org/abs/2505.08638)Cited by:[2\. Related Work](https://arxiv.org/html/2608.05212#Sx2.p1.1)\.
- Y\. Du, R\. Ye, S\. Tang, X\. Zhu, Y\. Lu, Y\. Cai, and S\. Chen \(2026\)Openseeker: democratizing frontier search agents by fully open\-sourcing training data\.arXiv preprint arXiv:2603\.15594\.Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- H\. Fan, X\. Wang, Z\. Chu, Q\. Wang, Z\. Wang, M\. Liu, B\. Qin, and XingYu \(2026\)LiveBrowseComp: are search agents searching, or just verifying what they already know?\.External Links:2605\.28721,[Link](https://arxiv.org/abs/2605.28721)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p3.1),[Setup\.](https://arxiv.org/html/2608.05212#Sx5.SSx4.SSS0.Px1.p1.1)\.
- GLM\-5\-Team \(2026\)GLM\-5: from vibe coding to agentic engineering\.External Links:2602\.15763,[Link](https://arxiv.org/abs/2602.15763)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p1.1),[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- Google DeepMind \(2026\)Gemini 3\.1 pro: model card\.External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Cited by:[Baselines & Models\.](https://arxiv.org/html/2608.05212#Sx5.SSx1.SSS0.Px1.p1.1)\.
- B\. Gou, Z\. Huang, Y\. Ning, Y\. Gu, M\. Lin, W\. Qi, A\. Kopanev, B\. Yu, B\. Jimenez Gutierrez, Y\. Shu,et al\.\(2026\)Mind2web 2: evaluating agentic search with agent\-as\-a\-judge\.Advances in Neural Information Processing Systems38\.Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1)\.
- N\. Gupta, R\. Chatterjee, L\. Haas, C\. Tao, A\. Wang, C\. Liu, H\. Oiwa, E\. Gribovskaya, J\. Ackermann, J\. Blitzer, S\. Goldshtein, and D\. Das \(2026\)DeepSearchQA: bridging the comprehensiveness gap for deep research agents\.External Links:2601\.20975,[Link](https://arxiv.org/abs/2601.20975)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- Kimi\-Team \(2026\)Kimi k2\.5: visual agentic intelligence\.External Links:2602\.02276,[Link](https://arxiv.org/abs/2602.02276)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- S\. Krishna, K\. Krishna, A\. Mohananey, S\. Schwarcz, A\. Stambler, S\. Upadhyay, and M\. Faruqui \(2025\)Fact, fetch, and reason: a unified evaluation of retrieval\-augmented generation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4745–4759\.External Links:[Link](https://aclanthology.org/2025.naacl-long.243/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.243),ISBN 979\-8\-89176\-189\-6Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1)\.
- H\. Li, Y\. Yao, L\. Zhu, R\. Feng, H\. Ye, J\. Wang, Y\. He, P\. Zou, L\. Zhang, X\. Lei, H\. Huang, K\. Deng, M\. Sun, Z\. Zhang, H\. Ye, and J\. Liu \(2026a\)CodeTracer: towards traceable agent states\.External Links:2604\.11641,[Link](https://arxiv.org/abs/2604.11641)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1)\.
- K\. Li, Z\. Zhang, H\. Yin, L\. Zhang, L\. Ou, J\. Wu, W\. Yin, B\. Li, Z\. Tao, X\. Wang,et al\.\(2025\)Websailor: navigating super\-human reasoning for web agent\.arXiv preprint arXiv:2507\.02592\.Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p1.1)\.
- Z\. Li, D\. Jiang, X\. Ma, H\. Zhang, P\. Nie, Y\. Zhang, K\. Zou, J\. Xie, Y\. Zhang, and W\. Chen \(2026b\)OpenResearcher: a fully open pipeline for long\-horizon deep research trajectory synthesis\.External Links:2603\.20278,[Link](https://arxiv.org/abs/2603.20278)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- R\. Nakano, J\. Hilton, S\. Balaji, J\. Wu, L\. Ouyang, C\. Kim, C\. Hesse, S\. Jain, V\. Kosaraju, W\. Saunders,et al\.\(2021\)Webgpt: browser\-assisted question\-answering with human feedback\.arXiv preprint arXiv:2112\.09332\.Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p1.1)\.
- OpenAI \(2026\)OpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p1.1),[Baselines & Models\.](https://arxiv.org/html/2608.05212#Sx5.SSx1.SSS0.Px1.p1.1)\.
- T\. Pham, N\. Nguyen, P\. Zunjare, W\. Chen, Y\. Tseng, and T\. Vu \(2026\)SealQA: raising the bar for reasoning in search\-augmented language models\.External Links:2506\.01062,[Link](https://arxiv.org/abs/2506.01062)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- M\. N\. Rafi, M\. Ahasanuzzaman, D\. J\. Kim, Z\. Wang, and T\. Chen \(2026\)FALAT: tracing failures in llm agent trajectories via dependency\-guided search\.arXiv preprint arXiv:2606\.00765\.Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1)\.
- Tongyi\-DeepResearch\-Team, B\. Li, B\. Zhang, D\. Zhang, F\. Huang, G\. Li, G\. Chen, H\. Yin, J\. Wu, J\. Zhou, K\. Li, L\. Su, L\. Ou, L\. Zhang, P\. Xie, R\. Ye, W\. Yin, X\. Yu, X\. Wang, X\. Wu, X\. Chen, Y\. Zhao, Z\. Zhang, Z\. Tao, Z\. Zhang, Z\. Qiao, C\. Wang, D\. Yu, G\. Fu, H\. Shen, J\. Yang, J\. Lin, J\. Zhang, K\. Zeng, L\. Yang, H\. Yin, M\. Song, M\. Yan, M\. Liao, P\. Xia, Q\. Xiao, R\. Min, R\. Ding, R\. Fang, S\. Chen, S\. Huang, S\. Wang, S\. Cai, W\. Shen, X\. Wang, X\. Guan, X\. Geng, Y\. Shi, Y\. Wu, Z\. Chen, Z\. Li, and Y\. Jiang \(2026\)Tongyi deepresearch technical report\.External Links:2510\.24701,[Link](https://arxiv.org/abs/2510.24701)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- M\. Wang, X\. Xie, and Y\. Huo \(2026\)TrajAudit: automated failure diagnosis for agentic coding systems\.External Links:2605\.26563,[Link](https://arxiv.org/abs/2605.26563)Cited by:[2\. Related Work](https://arxiv.org/html/2608.05212#Sx2.p1.1)\.
- J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. Glaese \(2025\)BrowseComp: a simple yet challenging benchmark for browsing agents\.External Links:2504\.12516,[Link](https://arxiv.org/abs/2504.12516)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p1.1),[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1),[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- J\. Xie, T\. Lin, Z\. Wang, Y\. Ning, Y\. Yao, T\. Xue, Z\. Zhang, Z\. Li, K\. Zhang, Y\. Wu, S\. Chen, B\. Gou, M\. Han, Y\. Wang, V\. Lee, X\. Wei, X\. Wang, Y\. Su, and H\. Sun \(2026\)QUEST: training frontier deep research agents with fully synthetic tasks\.External Links:2605\.24218,[Link](https://arxiv.org/abs/2605.24218)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.
- Y\. Zhan, T\. Fan, L\. Huang, Z\. Guo, and C\. Huang \(2026\)Why your deep research agent fails? on hallucination evaluation in full research trajectory\.External Links:2601\.22984,[Link](https://arxiv.org/abs/2601.22984)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p1.1)\.
- G\. Zhang, J\. Wang, J\. Chen, W\. Zhou, K\. Wang, and S\. Yan \(2025a\)AgenTracer: who is inducing failure in the llm agentic systems?\.External Links:2509\.03312,[Link](https://arxiv.org/abs/2509.03312)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1),[2\. Related Work](https://arxiv.org/html/2608.05212#Sx2.p1.1)\.
- S\. Zhang, M\. Yin, J\. Zhang, J\. Liu, Z\. Han, J\. Zhang, B\. Li, C\. Wang, H\. Wang, Y\. Chen, and Q\. Wu \(2025b\)Which agent causes task failures and when? on automated failure attribution of LLM multi\-agent systems\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=GazlTYxZss)Cited by:[1\. Introduction](https://arxiv.org/html/2608.05212#Sx1.p2.1),[2\. Related Work](https://arxiv.org/html/2608.05212#Sx2.p1.1),[Baselines & Models\.](https://arxiv.org/html/2608.05212#Sx5.SSx1.SSS0.Px1.p1.1)\.
- X\. Zhao, H\. Li, S\. Li, T\. Zhao, E\. T\. Barr, F\. Sarro, and H\. Ye \(2026\)Failure as a process: an anatomy of cli coding agent trajectories\.External Links:2607\.09510,[Link](https://arxiv.org/abs/2607.09510)Cited by:[2\. Related Work](https://arxiv.org/html/2608.05212#Sx2.p1.1)\.
- P\. Zhou, B\. Leon, X\. Ying, C\. Zhang, Y\. Shao, Q\. Ye, D\. Chong, Z\. Jin, C\. Xie, M\. Cao, Y\. Gu, S\. Hong, J\. Ren, J\. Chen, C\. Liu, and Y\. Hua \(2025\)BrowseComp\-zh: benchmarking web browsing ability of large language models in chinese\.External Links:2504\.19314,[Link](https://arxiv.org/abs/2504.19314)Cited by:[3\.1 Trajectory Collection](https://arxiv.org/html/2608.05212#Sx3.SSx1.p1.1)\.

## Appendix

This appendix is organized as follows\.

- •In Section A, we present the root\-cause taxonomy development and definitions\.
- •In Section B, we detail the annotation pipeline, including the annotation guideline and interface\.
- •In Section C, we report extended statistics of SearchAuditBench\.
- •In Section D, we validate the automatic repair\-rubric grader against a blind human reference\.
- •In Section E, we analyze the inference cost and latency of every auditor on a fixed subset\.
- •In Section F, we present two case studies\.
- •In Section G, we provide the complete prompts of SearchAuditor\.

## Appendix AA\. Root\-Cause Taxonomy Development and Definitions

#### Development procedure\.

We derived the taxonomy bottom\-up from observed failures rather than imposing a stage\-based scheme in advance\. Pilot annotation with a deliberately fine\-grained codebook exposed two problems as annotation scaled: some labels became catch\-alls, since a wrong candidate could reflect premature commitment on partial matches, acceptance of a constraint violation, or rationalization of contradictory evidence; others described different manifestations of one repairable mechanism, producing overlap and sparse tail classes\. We therefore consolidated premature commitment, erroneous rejection, failure to update, and failure to converge under*Candidate Mismanagement*, and acceptance of and rationalization about hard\-constraint violations under*Constraint Neglect*, while retaining the already separable*Search Coverage Gap*,*Unverified Source Reliance*,*Entity–Relation Misbinding*, and the residual*Unsupported Answer*\. Consolidation was followed by trajectory\-level re\-adjudication rather than a deterministic remapping, and the residual disagreements sharpened the boundaries stated below\.

#### Labeling policy\.

Each trajectory receives exactly one primary root cause: the earliest and most direct mechanism that establishes the failure path, so downstream symptoms never override an earlier causal error\. Operational behaviors such as ineffective queries, premature stopping, or extensive but unproductive search are manifestations of a mechanism rather than causes in themselves, and the stage at which an error occurs is kept separate from the mechanism explaining why it occurs\.

#### Category definitions\.

The six categories are defined below\. Each definition opens with the criterion applied at annotation time, continues with the typical cases observed in the corpus, and closes with the boundary against the categories it is most easily confused with\.

- •*Candidate Mismanagement*\.The agent fails to maintain, compare, or update candidate answers correctly after candidates or candidate\-relevant evidence have entered the trajectory\. Typical cases include prematurely locking onto a candidate that matches only a salient subset of the clues, discarding a still\-viable candidate without adequate comparison, failing to compare multiple candidates systematically, failing to update the answer after stronger evidence appears, and searching extensively without converting accumulated evidence into a candidate decision\. The defining failure concerns the state and comparison of the candidate set; if the decisive candidate or evidence path never appears because it was never searched, the label is*Search Coverage Gap*instead\.
- •*Search Coverage Gap*\.The agent does not search for or inspect a key evidence path needed to solve the problem\. Typical cases include an overly narrow search direction, failure to explore obvious candidates, failure to inspect a key data source, omission of a key evidence table or official source, never seeking the evidence that a decisive query constraint requires, and terminating before the relevant portion of the search space is examined\. The defining property is the absence of the necessary evidence from the trajectory; if the source or candidate was found but subsequently mishandled, a source\- or candidate\-related category applies instead\.
- •*Constraint Neglect*\.The agent accepts a candidate that does not satisfy a hard query constraint\. Typical cases include ignoring a known contradiction, relaxing an explicit condition, treating a mismatch as acceptable, and rationalizing contradictory evidence so that the preferred candidate can be retained\. A failed constraint alone is not sufficient for this label: if the information needed to evaluate the constraint was never retrieved, the cause is normally*Search Coverage Gap*; if that information was attached to the wrong entity or relation, it is*Entity–Relation Misbinding*\.
- •*Unverified Source Reliance*\.The agent finds an apparently relevant source but does not verify source reliability or original content before relying on it\. Typical cases include trusting snippets or titles as conclusive, relying on a secondary or low\-quality source where primary evidence is required, accepting a claim from an inaccessible page, and committing to a candidate without opening the underlying primary source\. The defining failure is insufficient verification of a source that was found, rather than the absence of a relevant search path\.
- •*Entity–Relation Misbinding*\.The agent finds relevant facts but binds them to the wrong entity, relation, role, time, work, person, organization, or answer slot\. Typical cases include conflating same\-name entities, transferring an attribute from one person or organization to another, reversing a relation, and combining facts drawn from different candidate chains\. The core problem is that the fact, possibly correct in itself, was attached to the wrong target rather than that no fact was found: the needed information is present, which separates this category from*Search Coverage Gap*, and source reliability is not the central problem, which separates it from*Unverified Source Reliance*\.
- •*Unsupported Answer*\.The final answer lacks support from the trajectory evidence, and no more specific search coverage, source verification, candidate management, constraint neglect, or entity–relation binding failure explains it\. Typical cases include guessing, memory\-based completion, asserting a value not extracted from any inspected source, and extracting an answer from partial observations without an evidence chain\. It is a residual category rather than a default label, assigned only after the other five mechanisms have been ruled out\.

#### Boundary summary\.

Applied at the critical step, the categories are separated by the state of the decisive evidence\. If it was never sought or inspected, the failure is a*Search Coverage Gap*; if it was found but not verified,*Unverified Source Reliance*; if candidates were surfaced but improperly maintained or compared,*Candidate Mismanagement*; if an explicit hard\-constraint mismatch was available yet accepted or rationalized,*Constraint Neglect*; if a relevant fact was attached to the wrong target,*Entity–Relation Misbinding*\. Only when none of these earlier mechanisms accounts for the failure do we assign*Unsupported Answer*\.

## Appendix BB\. Annotation Details

### B\.1 Trajectory Pool and Discard Criteria

#### Collection and automatic screening\.

We draw a fixed query pool from each of the five source benchmarks and run all eight agents once on every sampled query, so that all 40 model×\\timesbenchmark combinations contribute a comparable number of rollouts; the counts are only approximately equal because some released query sets are smaller than the target pool size\. This yields 3,500 raw trajectories, of which 3,288 \(93\.9%\) terminate normally with a gradable final answer, while the remainder exhaust the context budget, hit the maximum turn limit, or abort on an unrecoverable tool error\. An LLM evaluator then compares every gradable answer against the benchmark gold answer, marking 1,674 runs \(50\.9% of gradable runs\) as incorrect\. These 1,674 failed runs form the pool that enters expert annotation\.

#### Discard criteria\.

Not every incorrect run is auditable from its frozen trace, so annotators discard a case whenever one of the following conditions holds, recording the condition rather than producing labels:D1 \(evaluator false negative\), where the predicted answer is in fact acceptable and differs from the gold answer only by an alias, a unit, a rounding convention, or a formatting choice;D2 \(flawed benchmark instance\), where the query is genuinely ambiguous, admits several defensible answers, or its gold answer is stale relative to the evidence the agent could observe;D3 \(incomplete trajectory\), where the recorded trace is corrupted or partially missing, most often through mid\-run context truncation or absentreasoning\_content;D4 \(not offline\-auditable\), where the decisive evidence never enters the trace and the failure is attributable to the environment rather than to an agent decision, for example when the search backend returns empty results throughout or every candidate page is unreachable; andD5 \(no single critical step\), where several independent errors of comparable causal weight coexist and no single decision can be identified as the earliest one that establishes the failure path, so the single\-primary\-cause labeling policy of Appendix A cannot be applied\. Discarding these cases from the pooled failed runs leaves the 1,243 fully annotated trajectories of SearchAuditBench, with 125 to 175 trajectories per generating agent\.

### B\.2 Annotation Guideline and Interface

#### Annotation workflow\.

Every case is labeled by a single annotator, with cases split disjointly across the four annotators\. Because a failed trajectory averages 73\.1 messages and 65\.1K tokens, reading each trace end to end without guidance is prohibitively slow, so we precede human annotation with an LLM pre\-screening pass\. Claude Opus 4\.6 receives the query, the predicted answer, the gold answer, and the complete trajectory, and proposes a small set of candidate error regions, each with a short rationale and the taxonomy label it would assign\. This output is advisory only: annotators verify every proposed region against the trace and may correct its boundaries, relocate the critical step to an earlier decision, replace the proposed label, or discard the proposal entirely before finalizing the annotation, and all annotation fields are authored by the human\. We deliberately use a pre\-screening model that is not among the three auditor backbones evaluated in Section 5, so that no evaluated auditor shares a generator with the gold labels it is scored against\.

#### Annotation guideline\.

The complete guideline given to annotators is reproduced below\.

`Human Annotation Guide`

`Annotation interface\. Figure 5 shows the interface used for annotation\. Figure 5: The annotation interface\. Appendix C C\. Extended Dataset Statistics This section expands the dataset statistics in Section 3\.4, first summarizing the 1,243 failed trajectories and their expert annotations, and then comparing the characteristics of eight agents\. Metrics\. All token counts use the tokenizer of the generating model\. Trajectory text tokens count the content and reasoning\_content of all messages, including complete tool returns\. Tool return tokens count role=tool content, assistant reasoning tokens count assistant reasoning\_content, and predicted\-answer tokens count the final predicted answer\. A tool call is one structured call object issued by the assistant, irrespective of how many items it contains\. An atomic tool operation is one executable query or URL after expanding batched arguments: for example, search\(query=\[q1,q2,q3\]\) counts as one tool call but three atomic operations, while visit\(url=\[u1,u2\]\) counts as one call but two operations\. Other tools count as one operation, whereas an empty query or URL list counts as one call but zero operations\. Thus, atomic operations usually exceed tool calls and measure the actual amount of requested search/visit work; the search/visit calls row instead counts the unexpanded structured call objects of the two tool types\. For annotation statistics, critical position is the annotated main failure message index divided by the final message index, yielding 0% at the first message and 100% at the last; the final third begins at 2/32/3\. Critical span length counts messages from the annotated span start through its end, inclusively, and rubrics per repair counts the atomic requirements in repair\_rubrics\. Table 5: Aggregate trajectory and tool\-use statistics over SearchAuditBench \(N=1,243N=1\{,\}243\)\. K denotes 10310^\{3\} tokens\. Overall distribution\. Table 5 shows a strongly right\-skewed corpus: trajectory text averages 65\.1K tokens but has a 39\.4K median and a 152\.0K P90\. Tool returns account for most of this text \(49\.1K tokens on average\), substantially exceeding captured assistant reasoning \(15\.3K\)\. A trajectory contains 73\.1 messages and 36\.7 structured tool calls on average; expanding batched arguments raises the latter to 57\.3 atomic operations, comprising 25\.8 search calls and 10\.9 visit calls before expansion\. Table 6: Distribution and structure of the expert failure annotations\. Failure\-annotation distribution\. The two most frequent categories, Candidate Mismanagement and Search Coverage Gap, jointly account for 50\.04% of the corpus, while no single category exceeds 27\.19%\. Their localization patterns differ: Search Coverage Gap arise relatively early \(median position 40\.00%; 29\.93% in the final third\), whereas Constraint Neglect is later \(84\.49%; 65\.97%\), and Unsupported Answer is usually localized to the end \(median 100\.00%\), although the latter class is small \(n=37n=37\)\. Overall, a critical span contains 4\.305 messages on average \(median 4, P90 7, range 1–12\); every decisive anchor is an assistant message, with 1,033 \(83\.11%\) attached to a tool\-calling turn and 210 \(16\.89%\) to the final answer\. Each repair has 4\.374 atomic rubrics on average \(median 4, range 3–5\), with category means confined to 4\.27–4\.44\. These proportions characterize the composition of this failed\-trajectory corpus\. Table 7: Statistics by generating agent\. The trajectory column reports mean/median token counts; all other columns report means\. K denotes 10310^\{3\} tokens\. Differences across agents\. Table 7 reveals substantially different interaction patterns\. GLM\-5\.2 produces the longest trajectories \(118\.3K text tokens\) and the most tool calls \(65\.9\), while OpenSeeker has the most messages \(120\.7\), the most tool\-return text \(84\.1K tokens\), and a comparatively high visit count \(25\.0\)\. Kimi\-K2\.6 illustrates aggressive batching: its 28\.1 call objects expand to 79\.5 atomic operations\. By contrast, Qwen3\.5\-397B\-A17B and Tongyi\-DeepResearch generate short trajectories with few calls\. Quest\-35B also has short trajectories but unusually long final answers \(1,211 tokens on average\)\. OpenResearcher is the only agent with more visit than search calls; its atomic\-operation mean is below its call mean because some recorded visit calls contain empty URL lists and therefore contribute zero executable operations\. Interpretation\. The large cross\-agent differences show that auditing must handle both short, lightly instrumented traces and long, evidence\-heavy trajectories with dense tool interaction\. Raw token values should nevertheless be compared cautiously across agents because each row uses that agent’s own tokenizer\. Appendix D D\. Validation of the Repair Rubrics Grader Because FPS requires every rubric of a case to pass, a single grading error can flip the case\-level outcome\. We therefore construct a blind validation set of 200 diagnosis\-passing auditor outputs covering all five methods, all three backbones, and all six root\-cause categories\. An independent annotator judged all 872 rubrics in this set, seeing the query, the predicted answer, the repair directive, and the rubric text, but not the grader label or its rationale; the annotations were frozen before comparison, and case outcomes were derived under the same all\-pass rule as FPS\. Level NN Agreement Cohen’s κ\\kappa FP / FF Rubric 872 91\.6% 0\.710 4\.2% / 27\.0% Case \(all rubrics pass\) 200 82\.5% 0\.650 14\.0% / 21\.0% Table 8: Blind validation of the DeepSeek\-V4\-Flash rubric grader\. FP is the fraction of grader passes rejected by the human reference \(30/713 rubrics; 14/100 cases\); FF is the fraction of grader failures accepted by it \(43/159 rubrics; 21/100 cases\)\. The grader reaches 91\.6% rubric\-level agreement \(κ=0\.710\\kappa=0\.710\)\. Case\-level agreement is lower, as expected under a conjunctive gate where one rubric disagreement flips the verdict\. Since the same grader is applied to every method and backbone, this does not affect the relative comparisons in the Experiments section\. Appendix E E\. Cost and Efficiency Analysis We measure inference efficiency on a fixed sample of 100 SearchAuditBench instances, shared by every method and backbone configuration\. Latency is measured end to end, from prompt construction to the final structured audit\. Each pass allows up to three model\-level attempts and three transport retries per API call; all retained records passed structured\-output and usage\-accounting validation, and recovered attempts remain counted in the reported latency\. SearchAuditor’s three audit branches execute in parallel\. Method FPS↑\\uparrow Lat\.↓\\downarrow Calls↓\\downarrow In K↓\\downarrow Out K↓\\downarrow GPT\-5\.5 All\-at\-Once 26\.55 38\.5 1\.00 68\.9 2\.5 Step\-by\-Step 21\.48 326\.0 18\.66 858\.8 15\.9 Binary Search 24\.46 133\.1 5\.18 203\.3 8\.8 AgentRx 17\.70 94\.6 2\.01 79\.7 7\.1 SearchAuditor 32\.26 214\.1 5\.08 275\.6 15\.4 Gemini\-3\.1\-Pro All\-at\-Once 13\.59 71\.6 1\.07 79\.4 9\.4 Step\-by\-Step 9\.75 186\.5 8\.48 176\.4 22\.6 Binary Search 11\.58 214\.8 5\.35 233\.9 27\.2 AgentRx 10\.30 104\.4 2\.28 97\.2 12\.6 SearchAuditor 18\.91 374\.0 5\.92 400\.1 46\.2 Claude\-Opus\-4\.8 All\-at\-Once 20\.19 46\.2 1\.02 107\.5 2\.6 Step\-by\-Step 13\.03 601\.4 22\.82 2002\.6 23\.7 Binary Search 17\.86 212\.0 5\.33 321\.1 9\.6 AgentRx 12\.39 120\.0 2\.29 146\.0 7\.1 SearchAuditor 24\.62 309\.2 5\.43 444\.2 18\.6 Table 9: Inference efficiency on the fixed 100\-instance subset\. Latency is seconds per instance; token counts are API\-reported means in thousands\. FPS is copied from Table 2 of the main paper \(N=1,243N=1\{,\}243\) and is not recomputed on this subset\. Table 9 exposes the intended trade\-off between quality and compute\. SearchAuditor is not the cheapest auditor: heterogeneous audits, evidence\-grounded adjudication, and repair synthesis all cost additional calls and tokens\. In return, it is the high\-quality endpoint of the Pareto frontier under every backbone, improving FPS over the strongest baseline by 5\.71, 5\.32, and 4\.43 points on GPT\-5\.5, Gemini\-3\.1\-Pro, and Claude\-Opus\-4\.8\. Compared with exhaustive Step\-by\-Step inspection, it is also cheaper on both axes, cutting mean latency by 34\.3% and 48\.6% and input tokens by 67\.9% and 77\.8% on GPT\-5\.5 and Claude\-Opus\-4\.8, which shows that a selective heterogeneous workflow can deliver its quality advantage without inspecting every step\. Appendix F F\. Case Studies This section presents two examples from SearchAuditBench\. To keep the examples readable, each box shows only the assistant turns needed to establish the annotated failure; omitted turns are explicitly marked, and all displayed message indices refer to the original trajectory\. For every selected turn, the original reasoning\_content and structured tool\_calls are reproduced verbatim in wording and arguments, with only surrounding whitespace adjusted for presentation\. The metadata and all expert annotation fields are retained, while long answer strings are shortened to their answer\-bearing content\. Case 1: Relaxing a Mathematical Constraint Example F\.1\. Constraint Neglect Case 2: Letting an SEO Snippet Define the Identity Example F\.2\. Unverified Source Reliance Appendix G G\. Prompts This section introduces the prompts used by SearchAuditor\. Stage 1: Multi\-Perspective Auditing The three branches use the same backbone but different system prompts\. The holistic branch receives qq, a^\\hat\{a\}, and τ\\tau; the backward and forward branches additionally receive the deterministic assistant outline O​\(τ\)O\(\\tau\)\. Holistic audit\. The holistic branch diagnoses the complete trajectory without a prescribed decomposition\. Prompt 1\. Holistic Audit p1p\_\{1\} Backward constraint audit\. This branch checks query constraints against trajectory evidence and traces violated or unverified constraints backward\. Prompt 2\. Backward Constraint Audit p2p\_\{2\} Forward timeline audit\. This branch scans assistant decisions chronologically and targets the earliest decision that establishes the failure path\. Prompt 3\. Forward Timeline Audit p3p\_\{3\} Stage 2: Evidence\-Grounded Adjudication The adjudicator receives compact replica reports, their agreement summary, the assistant outline, evidence windows, and, in the default configuration, the full trajectory\. Prompt 4\. Evidence\-Grounded Adjudicator Stage 3: Diagnosis\-Conditioned Repair Synthesis The final synthesizer receives the fixed diagnosis and bounded evidence windows around the adjudicated critical step and final answer, but not the full trajectory\. Prompt 5\. Diagnosis\-Conditioned Repair Synthesizer`