Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents

# Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents
Source: [https://arxiv.org/html/2607.10198](https://arxiv.org/html/2607.10198)
###### Abstract

Search APIs are the fundamental retrieval layer for many agents and are often their most frequently used tool\. Traditional search APIs provide URLs, titles, and snippets that preview website contents\. Because full\-page retrieval is token\-intensive, agent retrieval architectures increasingly use progressive disclosure: the agent first sees snippets and then chooses whether to fetch full pages\. In such systems, search API performance is often evaluated primarily by answer accuracy\. We argue that a commercial search API is better understood as a*decision surface*: the ranked snippets, URLs, and metadata that determine whether an agent answers immediately, searches again, or spends tokens opening pages\. We test this claim with one frozen GPT\-5\.4 agent, two tools \(search\_webandfetch\_page\), and 100 questions fromSealQA\-Hard, varying only the search provider \(Brave, Tavily, Firecrawl\)\. A Kimi\-K2\.6 oracle labels every content element visible to the agent \(URL, title, snippet, and fetched page, when fetched\), producing 6,869 valid per\-URL judgments\. We use an audited*correct\-answer*label,semantic\_match, which preserves exact matches while accepting harmless formatting and naming variants\. Under this measure, the providers remain close \(25, 25, 26 / 100\), but their evidence economies differ sharply: Brave offers gold\-answer\-rich snippets, Tavily concentrates gold\-supporting URLs at rank 1, and Firecrawl is associated with broader exploration under this fixed agent policy\. We also introduce a surface contradiction\-to\-gold URL ratio, which varies from 0\.92 to 2\.59\. Provider choice is therefore a retrieval\-budget and policy decision, not merely a recall decision\.

Equal Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool\-Using Agents

Sriram Selvamselvamsriram@gmail\.comAnneswa Ghoshanneswaghosh@gmail\.com

## 1Introduction

Early search\-augmented agents often depended primarily on extracted page data provided by search API providers\. As complex use cases have evolved, the agent landscape has shifted toward progressive disclosure: models receive top\-nnURLs, titles, snippets, and key metadata such as freshness, and then decide whether to fetch full pages\. In this new landscape, engineering practice often treats search API providers as replaceable components\. If two APIs return plausible top\-kkURLs and produce similar answer accuracy, the rest of the agent pipeline is assumed to be largely unaffected\.

This paper argues that the relevant object is the*pre\-fetch surface*\. Before a page is opened, an agent does not see a search index or a corpus; it relies on search API results\. That surface is the evidence state on which the agent decides whether to answer, search again, or spend fetch tokens\. We call commercial search APIsdecision surfacesfor tool\-using language agents\.

For grounded language\-model systems, the retrieval API is part of the grounding interface\. It determines which evidence is exposed before generation, which contradictions enter context, and how much retrieval budget is spent before an answer is produced\. Decision\-surface evaluation therefore measures not only retrieval quality, but also the faithfulness and efficiency conditions under which grounded answers are generated\.

The distinction matters because final correctness can converge while the internal pipeline diverges\. A provider may expose enough snippet evidence for immediate answering\. Another may put the relevant URL at rank 1, making a top\-result fetch policy effective\. A third may expose sparse snippets and be associated with broader exploration under the same policy\. These regimes can yield similar aggregate accuracy with different cost, latency, contamination, and failure modes\.

Under a controlled protocol, we freeze the answer model, prompt, tools, maximum iterations, judge, and page\-fetch backend\. The only experimental condition is the commercial search API: Brave, Tavily, or Firecrawl\. We then judge every URL\-level evidence item visible to the agent, separating pre\-fetch snippet support from support discovered only after a page fetch\. The per\-URL oracle lets us ask not only whether the final answer was correct, but also what evidence and actions were available at the decision surface\.

We do not treat provider identity as a leaderboard variable; we use Brave, Tavily, and Firecrawl as production examples of different pre\-fetch evidence surfaces, showing that equal answer accuracy can hide different grounding, contradiction, and retrieval\-budget regimes\.

#### Contributions\.

1. 1\.We formalize commercial search APIs as*decision surfaces*for tool\-using agents rather than static ranked\-list retrievers\.
2. 2\.We introduce a per\-URL oracle protocol that labels every API result element the agent saw, including fetched pages\.
3. 3\.We distinguish pre\-fetch surface support from post\-fetch discovered support, and use the former to define a four\-way decision partition \(SMART, MISSED, BLIND, NO\-OP\)\.
4. 4\.We show that similar correctness hides different provider\-associated retrieval regimes under a fixed agent: pre\-fetch support, rank concentration, blind exploration, contamination, and complementarity\.

## 2Related Work

#### Retrieval\-augmented generation and open\-domain QA\.

Retrieval\-augmented generation combines parametric models with non\-parametric evidence\(Lewiset al\.,[2020](https://arxiv.org/html/2607.10198#bib.bib1); Guuet al\.,[2020](https://arxiv.org/html/2607.10198#bib.bib2)\)\. Open\-domain QA systems such as DPR, FiD, and Atlas largely evaluate whether a retriever supplies answer\-bearing passages to a reader or generator\(Karpukhinet al\.,[2020](https://arxiv.org/html/2607.10198#bib.bib3); Izacard and Grave,[2021](https://arxiv.org/html/2607.10198#bib.bib4); Izacardet al\.,[2023](https://arxiv.org/html/2607.10198#bib.bib5)\)\. Our setting differs in two ways: retrieval is performed by production web\-search APIs, and the reader is an agent that can choose whether to fetch pages\.

#### Information\-retrieval evaluation\.

Classical IR metrics, including precision/recall and rank\-aware measures such as NDCG, evaluate static rankings\(J"arvelin and Kek"al"ainen,[2002](https://arxiv.org/html/2607.10198#bib.bib6); Manninget al\.,[2008](https://arxiv.org/html/2607.10198#bib.bib7)\)\. BEIR broadened zero\-shot retriever evaluation across tasks and domains\(Thakuret al\.,[2021](https://arxiv.org/html/2607.10198#bib.bib8)\)\. Such metrics are necessary but incomplete for agents because they do not ask whether the pre\-fetch surface was sufficient, misleading, or action\-guiding\.

#### Tool\-using and browsing agents\.

WebGPT, Self\-Ask, ReAct, IRCoT, Toolformer, and Self\-RAG study language models that search, browse, or decide when to call tools\(Nakanoet al\.,[2021](https://arxiv.org/html/2607.10198#bib.bib9); Presset al\.,[2022](https://arxiv.org/html/2607.10198#bib.bib10); Yaoet al\.,[2023](https://arxiv.org/html/2607.10198#bib.bib11); Trivediet al\.,[2022](https://arxiv.org/html/2607.10198#bib.bib12); Schicket al\.,[2023](https://arxiv.org/html/2607.10198#bib.bib13); Asaiet al\.,[2024](https://arxiv.org/html/2607.10198#bib.bib14)\)\. FreshLLMs shows that search augmentation helps with fresh factual knowledge and that evidence order matters\(Vuet al\.,[2023](https://arxiv.org/html/2607.10198#bib.bib15)\)\. Over\-searching work argues for cost\-sensitive evaluation of search\-augmented models\(Xieet al\.,[2026](https://arxiv.org/html/2607.10198#bib.bib16)\)\. We isolate a complementary variable: the provider surface presented to the same agent\.

#### Hard search benchmarks and evidence evaluation\.

GAIA, WebArena, BrowseComp, andSealQAevaluate agents or systems on tasks requiring search, browsing, or hard factual reasoning\(Mialonet al\.,[2024](https://arxiv.org/html/2607.10198#bib.bib17); Zhouet al\.,[2024](https://arxiv.org/html/2607.10198#bib.bib18); Weiet al\.,[2025](https://arxiv.org/html/2607.10198#bib.bib19); Phamet al\.,[2025](https://arxiv.org/html/2607.10198#bib.bib20)\)\. RAG evaluation frameworks such as RAGAS and ARES evaluate answer quality, faithfulness, context relevance, or generated claims\(Eset al\.,[2024](https://arxiv.org/html/2607.10198#bib.bib21); Saad\-Falconet al\.,[2024](https://arxiv.org/html/2607.10198#bib.bib22)\)\. We use a hard\-QA benchmark as a controlled probe for provider\-associated evidence states under a fixed agent\.

#### Attribution, verifiability, and LLM judges\.

Generative search systems may produce fluent answers with incomplete citation support\(Liuet al\.,[2023a](https://arxiv.org/html/2607.10198#bib.bib23); Yueet al\.,[2023](https://arxiv.org/html/2607.10198#bib.bib24)\); long\-context work shows that more text does not guarantee robust evidence use\(Liuet al\.,[2024](https://arxiv.org/html/2607.10198#bib.bib25)\)\. LLM\-as\-judge methods scale evaluation but require constrained rubrics and careful interpretation\(Zhenget al\.,[2023](https://arxiv.org/html/2607.10198#bib.bib26); Liuet al\.,[2023b](https://arxiv.org/html/2607.10198#bib.bib27)\)\. Our judge labels individual retrieved documents with a fixed JSON schema, and our answer correctness label is separately audited assemantic\_match\.

#### Commercial search APIs\.

Brave, Tavily, Firecrawl, and Jina Reader expose different product surfaces\(Brave Software,[2026](https://arxiv.org/html/2607.10198#bib.bib28); Tavily,[2026](https://arxiv.org/html/2607.10198#bib.bib29); Firecrawl,[2026](https://arxiv.org/html/2607.10198#bib.bib30); Jina AI,[2026](https://arxiv.org/html/2607.10198#bib.bib31)\)\. We disable provider\-side page content in the main condition and use a sharedfetch\_pagebackend, so provider differences are attributable to ranked URLs, snippets, and metadata rather than distinct extraction pipelines\.

![Refer to caption](https://arxiv.org/html/2607.10198v1/figures/fig1_pipeline.png)Figure 1:Execution and judging pipeline\. The agent, prompt, tools, and fetch backend are fixed; only the search provider varies before the Kimi oracle replays every visible URL\.

## 3Experimental Protocol

Figure[1](https://arxiv.org/html/2607.10198#S2.F1)summarizes the experiment\. The frozen agent uses the same prompt, tools, and fetch backend in every condition while only thesearch\_webprovider varies; the oracle then replays every visible URL in the same representation the answer model saw\.

#### Dataset\.

We use a deterministic proportional stratified sample of 100 questions from the 254\-rowSealQA\-Hardsubset\. The strata arefreshness×\\timessearch\_results×\\timestopic\. We choseSealQA\-Hardbecause it provides challenging questions that often require model exploration and are not yet fully resolved by a simple snippet\-only search API response\. Appendix[B](https://arxiv.org/html/2607.10198#A2)gives the full sampling summary\.

#### Agent and tools\.

The answer model is GPT\-5\.4 with a maximum of 10 iterations\. It has two tools\.search\_webaccepts a query and returns up to ten ranked results, each with a document identifier, rank, title, URL, domain, snippet surface, and provider metadata\.fetch\_pageaccepts a document identifier from a prior search result and returns extracted markdown plus fetch metadata\. The tool interface allows the agent to issue*multiple*search\_webcalls or*multiple*fetch\_pagecalls in a single turn; the trace records each call separately\. The final answer must use the exact prefixFINAL ANSWER:and be grounded in retrieved evidence\. Appendix[D](https://arxiv.org/html/2607.10198#A4)gives the fixed prompt contract and trace schema\.

#### Providers and snippet normalization\.

We compare Brave, Tavily, and Firecrawl Search\. Each adapter normalizes provider output into the same result schema\. Brave exposes additional provider\-native snippet blocks\. We aggregate those blocks into the generic*snippet surface*rather than treating them as a separate evidence channel; the agent saw them as snippets, and the judge saw the same text\. We still discuss this asymmetry as a limitation because it changes the amount of pre\-fetch text Brave returns\. Tavily is run with raw content disabled, and Firecrawl is run without provider\-side markdown scraping\. Page text visible to the agent therefore comes only from the shared Jina Reader backend\.

#### Run and reproducibility controls\.

The main provider traces were collected on 2026\-05\-17 UTC, from 06:49–09:02 UTC across the three providers\. All providers were asked for up to 10 web results under the fixed configuration in Appendix[C](https://arxiv.org/html/2607.10198#A3): Brave used US/en with moderate safe search, Tavily used basic/general search with raw content disabled, and Firecrawl used web search with provider\-side markdown scraping disabled\. All full\-page text came from the shared Jina Readerfetch\_pagebackend with a 15s timeout, 2MB read limit, concurrency 4, and gzip cache keyed by normalized URL\. Because commercial search outputs are time\-sensitive and raw API results are not ours to redistribute, the released reproducibility package contains code, configuration, sampling logic, trace schemas, evaluation scripts, and aggregate derived outputs; the reported numbers are generated from private run artifacts\.

#### Traces\.

The experiment produces 300 provider\-query trajectories\. A trace records every search call, every returned URL, every fetch decision, fetch status, token usage, latency, final answer, and gold answer\. Provider\-comparison scripts deterministically derive token, URL\-hit, and per\-query support fields from these traces\. Answer correctness comes from the separate semantic audit TSV\.

## 4Per\-URL Oracle and Metrics

#### Judge input\.

For every URL the agent saw, we run Kimi\-K2\.6 as the judge\. The judge receives the question, gold answer, model final answer, and one retrieved document in the same XML\-like format the answer model saw\. For unfetched URLs, the judge sees title, URL, domain, snippet surface, and metadata\. For fetched URLs, the same record also includes extracted markdown; the snippet\-specific fields remain labeled separately\.

#### Judge output\.

The judge returns JSON only and is instructed not to use outside knowledge\. The required schema is shown below\.

Across providers, 6,869 of 6,909 judge rows are valid; 40 invalid rows are excluded\. Valid rows are split into 6,519 snippet\-only and 350 page\-visible rows\. We use “gold support” in the question\-conditioned sense: the evidence must make the gold answer answerable for the given question, not merely mention the answer string\.

#### Oracle validation\.

To assess whether the single\-judge oracle is usable as an audit instrument, we manually validated a balanced 180\-case sample of URL\-label judgments\. The sample covers provider×\\timessurface×\\timeslabel×\\timesKimi\-value slices, with five cases per leaf slice\. Table[1](https://arxiv.org/html/2607.10198#S4.T1)shows high agreement on clear human judgments overall \(164/174, 94%\)\. The validation was performed as a single\-annotator manual audit of Kimi labels, not as an inter\-annotator agreement study\. Cases marked unclear by the human auditor are excluded only from the agreement denominator, not from the sampling pool\. The remaining disagreements were boundary cases involving inferential answerability, directness of contradiction, or whether sparse and low\-information results should count as unusable\.

Table 1:Human validation of Kimi per\-URL oracle labels\.
#### Answer metrics\.

The headlineCorrectmetric issemantic\_matchfrom a separate semantic audit TSV\. The table also reports legacy token F1: we normalize the final and gold answers by lowercasing, removing articles and punctuation, and mapping simple number words to digits, then compute token\-overlap F1 per query and macro\-average over the 100 queries\.

#### Support split\.

We separate support by when it became visible\.*Pre\-fetch surface support*exists for a provider\-query pair when a valid judgment marksgold\_answer\_in\_snippetstrue, or when an unfetched snippet\-only judgment markscontains\_gold\_answertrue\. This captures support in the URL, title, snippet surface, or metadata before the agent decides whether to fetch\.*Post\-fetch discovered support*exists when no pre\-fetch support was present, but a fetched page\-visible judgment marksgold\_answer\_in\_extracted\_pagetrue\.*Trajectory\-visible support*is the union of these two events\.

#### Decision partition\.

We join the per\-URL oracle with trace actions and assign each provider\-query pair to one cell using pre\-fetch support:smartif pre\-fetch support exists and the agent fetched a pre\-fetch\-supporting URL;missedif pre\-fetch support exists but the agent fetched none of those URLs;blindif no pre\-fetch support exists and the agent fetched at least one URL; andno\-opif no pre\-fetch support exists and no URL was fetched\. The partition is descriptive, not causal: amissedquery may still be answered correctly from snippets, and ablindquery may still succeed if a page reveals support after the fetch\.

#### Surface contradiction\-to\-gold ratio\.

We define the surface contradiction\-to\-gold ratio over snippet\-only URL rows:

rc:g=\|\{u:contradicts\_gold\_answer​\(u\)\}\|\|\{u:contains\_gold\_answer​\(u\)\}\|\.r\_\{c:g\}=\\frac\{\|\\\{u:\\texttt\{contradicts\\\_gold\\\_answer\}\(u\)\\\}\|\}\{\|\\\{u:\\texttt\{contains\\\_gold\\\_answer\}\(u\)\\\}\|\}\.The metric depends on the gold answer but is independent of the model’s final answer; it uses only pre\-fetch surface labels, not page text\.

## 5Results

### 5\.1Correctness parity masks different evidence economies

![Refer to caption](https://arxiv.org/html/2607.10198v1/figures/fig2_provider_profiles.png)Figure 2:Provider profiles under semantic correctness\. Similar final accuracy hides different pre\-fetch support, rank concentration, contradiction ratio, and fetch behavior\.Table 2:Headline metrics\. Semantic correctness is similar across providers, while token use, search/fetch behavior, and evidence exposure differ\.Table[2](https://arxiv.org/html/2607.10198#S5.T2)and Figure[2](https://arxiv.org/html/2607.10198#S5.F2)show the headline\. UnderCorrect, the providers remain close: 25, 25, and 26 correct answers out of 100\. The low absolute accuracy reflects the intentionally hardSealQA\-Hardsample; this setting is useful for our diagnostic purpose because it exposes cases where answer\-bearing evidence is visible but not operationally used\. The semantic audit changes absolute counts, but not the main conclusion: aggregate correctness alone suggests the providers are nearly interchangeable\.

The evidence economy says otherwise\. Brave exposes pre\-fetch support on 30 queries, compared with 16 for Tavily and 16 for Firecrawl\. Tavily’s pre\-fetch support is more concentrated at rank 1: 17 / 34 gold\-supporting pre\-fetch rows \(50%\), compared with 13 / 101 \(13%\) for Brave and 4 / 30 \(13%\) for Firecrawl\. Firecrawl is associated with the largest fetched\-query share under this fixed agent policy \(81%\)\. We summarize paired\-bootstrap uncertainty for these main contrasts after the support split in Table[5](https://arxiv.org/html/2607.10198#S5.T5)\.

### 5\.2Pre\-fetch support and page\-discovered support diverge

Table 3:Support split and surface diagnostics\. Rank\-1 percentages divide the shown numerator by the shown gold\-supporting pre\-fetch\-row denominator;rc:gr\_\{c:g\}divides contradicting snippet\-only rows by gold snippet\-only rows\.Table[3](https://arxiv.org/html/2607.10198#S5.T3)shows that the per\-URL oracle sees a larger provider effect than headline correctness\. Brave exposes substantially more pre\-fetch support \(30 queries\) than Tavily or Firecrawl \(16 and 16\), while Tavily recovers more support only after fetch\. The contradiction ratio also varies substantially, from 0\.92 for Brave to 2\.59 for Firecrawl contradicting rows per gold\-supporting snippet\-only row; the displayed counts make the smaller Tavily and Firecrawl denominators explicit\.

Table 4:Snippet\-surface length normalization\. Token counts are computed over rendered pre\-fetch search observations visible to the agent and exclude fetched page text\.The token counts in Table[4](https://arxiv.org/html/2607.10198#S5.T4)use the rendered pre\-fetchsearch\_webobservations sent to the model, including URLs, titles, domains, snippets, and provider\-native extra snippets, while excluding fetched page text and raw provider payloads\. Duplicate search observations are counted because they entered the model context\. Because Brave exposes more provider\-native snippet text, the table normalizes pre\-fetch support by rendered snippet\-surface token volume\. Brave’s row\-normalized advantage over Tavily persists \(0\.113 vs\. 0\.057 gold\-supporting rows per 1K tokens\), but the contrast with Firecrawl shrinks \(0\.113 vs\. 0\.099\), and Firecrawl is higher under query\-normalized support \(0\.053 vs\. 0\.033 support queries per 1K tokens\)\. We therefore interpret Brave’s pre\-fetch support advantage as partly explained by surface volume rather than purely higher per\-token evidence density\.

Table 5:Paired\-bootstrap intervals for main contrasts\.Table[5](https://arxiv.org/html/2607.10198#S5.T5)reports paired\-bootstrap intervals over question IDs, with differences shown as left minus right in queries \(q\) or percentage points \(pp\)\. It makes the intended claim boundary explicit: at this sample size, final\-answer correctness differences are not meaningful because the paired intervals all include zero\. The stronger result is structural rather than accuracy\-based\. The intervals stay positive for Brave’s pre\-fetch\-support advantage and for Tavily’s rank\-1 concentration, supporting the interpretation that similar correctness hides different evidence economies\.

### 5\.3The same agent enters different decision regimes

![Refer to caption](https://arxiv.org/html/2607.10198v1/figures/fig3_decision_partition.png)Figure 3:Decision partition by provider\. Each cell reports queries / semantic\-correct answers after joining pre\-fetch support labels with agent fetch actions\.Figure[3](https://arxiv.org/html/2607.10198#S5.F3)shows the mechanism\. Each entry is queries / correct answers after joining oracle labels with agent fetch actions\. Under a strict pre\-fetch definition,smartfetches are rare for all three providers: much of the support observed in the trajectory appears only after a page is opened\. Brave has the largest pre\-fetch\-support mass, but many such queries aremissedrather thansmart, consistent with a snippet\-rich surface where fetching is not always needed\. Firecrawl’s largest regime isblindexploration, and Tavily also shifts towardblindonce page\-discovered support is separated from the pre\-fetch surface\.

Decision\-cell counts are visualized in Figure[3](https://arxiv.org/html/2607.10198#S5.F3); Wilson intervals are reported in Appendix[F](https://arxiv.org/html/2607.10198#A6), and paired\-bootstrap intervals for aggregate metrics are reported in Appendix[F\.1](https://arxiv.org/html/2607.10198#A6.SS1)\.

### 5\.4Aggregate correctness hides complementarity

Only 10 questions are answered correctly by all three providers\. 12 are correct under exactly two providers, 22 under exactly one, and 56 under none\. Thus, provider choice changes*which*questions are solved, not only how many\. This suggests that provider routing or provider\-aware fetch policies may matter more than choosing a single winner\.

![Refer to caption](https://arxiv.org/html/2607.10198v1/figures/fig4_complementarity.png)Figure 4:Correct\-answer overlap across providers\. Provider choice changes which questions are solved even when aggregate correct\-answer counts are nearly tied\.
### 5\.5Many failures occur despite visible answer text

The provider\-comparison traces include deterministic retrieval proxies such as gold\-URL hits and whether the gold answer string appears anywhere in retrieved text\. These are*not*the Kimi judge or document\-support labels; they are normalized string and URL checks over titles, snippets, provider\-returned snippet text, and fetched page text\.

Table 6:Trace\-derived retrieval proxies\. These deterministic URL and answer\-string checks are not oracle support labels, and visible answer text is not sufficient for correct final answers\.
Table[6](https://arxiv.org/html/2607.10198#S5.T6)shows why ordinary retrieval proxies are incomplete for agents\. Gold URLs and answer strings often appear somewhere in the trajectory, yet the agent still has 57, 56, and 53 wrong answers despite visible answer text\. The issue is not only whether evidence exists, but whether the decision surface makes the right evidence noticeable, trusted, and actionable\.

#### Qualitative examples\.

Two Brave traces show how this happens\. In a Blake Shelton query, one snippet listed the five relevant 2015–2023 winning seasons, but nearby snippets repeated his nine lifetime wins; with no fetch, the agent answered*9*rather than*5*\. In a border query, rank 1 favored Canada–United States, while lower snippets preserved the key word*continuous*and identified Kazakhstan–Russia; the agent again answered from the generic surface\. These cases reinforce the core claim: a search surface shapes whether the model notices, trusts, fetches, and uses the right evidence amid distractors\. Appendix[D](https://arxiv.org/html/2607.10198#A4)shows the trace\-level artifact structure used for these audits\.

## 6Discussion

#### Provider choice is a policy choice\.

The same agent policy has different utility under different provider surfaces\. Brave’s richer pre\-fetch surface reduces the marginal value of fetching some support\-bearing pages\. Tavily’s rank\-one concentration suggests that, when pre\-fetch support is present, top\-result policies can be efficient\. Firecrawl is associated with broader exploration and more page\-time discovery under this fixed agent policy\. A provider should therefore be paired with a provider\-aware fetch policy, not a fixed universal heuristic\.

#### Top\-kkrelevance is incomplete for agents\.

Static relevance and gold URL hit rates remain useful, but they cannot see whether the surface was already answerable, whether a fetch was worthwhile, or whether contradictory snippets competed with gold support\. Decision\-surface metrics complement IR metrics by measuring the action state available before the fetch decision\.

#### What providers could optimize\.

An agent\-ready search API should optimize not only relevance but also actionability: calibrated snippets, stable document identifiers, source/freshness metadata, low contradiction\-to\-gold contamination, and rankings that align answer\-bearing pages with common agent fetch policies\.

#### What practitioners can use now\.

The metrics here can be computed from traces without rerunning provider APIs\. A practitioner can run a small sample through candidate providers, judge visible URLs, and choose a policy based on whether the provider behaves like a snippet\-rich surface, a rank\-concentrated surface, or a volume/exploration surface\.

## 7Limitations

#### Single agent and judge\.

The decision partition is defined from one GPT\-5\.4 agent and one Kimi\-K2\.6 judge\. The manual audit in Table[1](https://arxiv.org/html/2607.10198#S4.T1)supports the oracle as a useful measurement instrument, but a different answer model, judge, prompt, or fetch appetite may shift the cell counts\.

#### Lower\-bound support\.

Pre\-fetch support is measured only over URLs actually returned to the agent, and post\-fetch support is a lower bound because unfetched URLs are not page\-judged\. This prevents us from claiming full provider recall\.

#### Snippet\-surface asymmetry\.

Brave returns more provider\-native snippet text under our configuration\. We aggregate it into the snippet surface to avoid presenting it as special treatment, but it remains a real product\-surface difference and a plausible contributor to Brave’s pre\-fetch support ceiling\. The length\-normalized check in Table[4](https://arxiv.org/html/2607.10198#S5.T4)shows that this asymmetry explains part, but not all, of the observed pre\-fetch support pattern\.

#### Observational policy analysis\.

The agent’s actions are observed, not randomized\. We do not intervene on ranks, snippets, or fetch decisions\. Causal replay is future work\.

#### Scale and provider coverage\.

The main experiment has 100 questions and three providers\. Small cells should be read as diagnostic rather than definitive\. Additional providers such as Bing, Google CSE, Serper, Exa, and others would broaden the claim\.

## 8Conclusion

Evaluating search APIs for tool\-using agents requires looking beyond answer accuracy\. In this controlled study, semantic correctness rates are close, but the internal retrieval economy changes sharply: pre\-fetch support, page\-time discovery, rank concentration, contradiction, fetch behavior, and instance\-level correctness all depend on the provider surface\. Search APIs for agents should therefore be evaluated as decision surfaces: ranked, snippet\-bearing action states that allocate retrieval budget\.

## Ethics Statement

This work evaluates commercial search APIs and LLM agents on public web\-search QA data\. We do not claim that one provider is universally better\. Provider names are reported because they are experimental conditions and because reproduction requires knowing which APIs produced the surfaces\. The study may help reduce unnecessary page fetches and improve handling of contradictory evidence\. It could also be misread as a provider leaderboard without respecting the stated limitations; we discourage that use\.

## Reproducibility Statement

We release the experiment code, configuration files, prompts, query\-sampling logic, trace schema, evaluation scripts, bootstrap scripts, figure\-generation code, paper sources, and aggregate derived artifacts needed to inspect the reported calculations at[https://github\.com/selvamsriram/search\-api\-decision\-surface](https://github.com/selvamsriram/search-api-decision-surface)\. We do not publicly redistribute raw search\-provider responses, rendered trace JSONLs, judge prompts or responses containing provider snippets, or fetched page text\. These artifacts contain provider search results and third\-party web content subject to API terms and publisher rights, and are therefore not ours to redistribute\.

The reported numbers were generated from the private run artifacts using the released pipeline\. Researchers can reproduce the protocol by running the same fixed query sample with their own API credentials and then applying the released evaluation scripts\. Because commercial search APIs are live, time\-varying systems, reruns reproduce the experimental procedure rather than bit\-identical provider surfaces\. To support auditability without redistributing raw provider content, we release aggregate summaries, row counts, semantic\-audit labels, validation summaries, and deterministic scripts that map traces and judge rows to every reported table and figure\.

## References

- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2024\)Self\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- Brave Software \(2026\)Brave search api reference\.Note:[https://api\-dashboard\.search\.brave\.com/api\-reference/web/search/get](https://api-dashboard.search.brave.com/api-reference/web/search/get)Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px6.p1.1)\.
- S\. Es, J\. James, L\. Espinosa\-Anke, and S\. Schockaert \(2024\)RAGAS: automated evaluation of retrieval augmented generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px4.p1.1)\.
- Firecrawl \(2026\)Firecrawl search api documentation\.Note:[https://docs\.firecrawl\.dev/api\-reference/endpoint/search](https://docs.firecrawl.dev/api-reference/endpoint/search)Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px6.p1.1)\.
- K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang \(2020\)REALM: retrieval\-augmented language model pre\-training\.InProceedings of the 37th International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Izacard and E\. Grave \(2021\)Leveraging passage retrieval with generative models for open domain question answering\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 874–880\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Izacard, P\. Lewis, M\. Lomeli, L\. Hosseini, F\. Petroni, T\. Schick, J\. Dwivedi\-Yu, A\. Joulin, S\. Riedel, and E\. Grave \(2023\)ATLAS: few\-shot learning with retrieval augmented language models\.Journal of Machine Learning Research24\(251\),pp\. 1–43\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px1.p1.1)\.
- K\. J"arvelin and J\. Kek"al"ainen \(2002\)Cumulated gain\-based evaluation of ir techniques\.ACM Transactions on Information Systems20\(4\),pp\. 422–446\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px2.p1.1)\.
- Jina AI \(2026\)Jina reader: url to llm\-friendly input\.Note:[https://r\.jina\.ai/](https://r.jina.ai/)Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px6.p1.1)\.
- V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Kuttler, M\. Lewis, W\. Yih, T\. Rocktaschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px1.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px5.p1.1)\.
- N\. F\. Liu, T\. Zhang, and P\. Liang \(2023a\)Evaluating verifiability in generative search engines\.arXiv preprint arXiv:2304\.09848\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px5.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023b\)G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.arXiv preprint arXiv:2303\.16634\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px5.p1.1)\.
- C\. D\. Manning, P\. Raghavan, and H\. Sch"utze \(2008\)Introduction to information retrieval\.Cambridge University Press\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom \(2024\)GAIA: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Nakano, J\. Hilton, S\. Balaji, J\. Wu, L\. Ouyang, C\. Kim, C\. Hesse, S\. Jain, V\. Kosaraju, W\. Saunders, X\. Jiang, K\. Cobbe, T\. Eloundou, G\. Krueger, K\. Button, M\. Knight, B\. Chess, and J\. Schulman \(2021\)WebGPT: browser\-assisted question\-answering with human feedback\.arXiv preprint arXiv:2112\.09332\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Pham, N\. Nguyen, P\. Zunjare, W\. Chen, Y\. Tseng, and T\. Vu \(2025\)SealQA: raising the bar for reasoning in search\-augmented language models\.arXiv preprint arXiv:2506\.01062\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px4.p1.1)\.
- O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. Lewis \(2022\)Measuring and narrowing the compositionality gap in language models\.arXiv preprint arXiv:2210\.03350\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. Zaharia \(2024\)ARES: an automated evaluation framework for retrieval\-augmented generation systems\.arXiv preprint arXiv:2311\.09476\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px4.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- Tavily \(2026\)Tavily search api documentation\.Note:[https://docs\.tavily\.com/documentation/api\-reference/endpoint/search](https://docs.tavily.com/documentation/api-reference/endpoint/search)Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px6.p1.1)\.
- N\. Thakur, N\. Reimers, A\. Rucklé, A\. Srivastava, and I\. Gurevych \(2021\)BEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)Interleaving retrieval with chain\-of\-thought reasoning for knowledge\-intensive multi\-step questions\.arXiv preprint arXiv:2212\.10509\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Vu, M\. Iyyer, X\. Wang, N\. Constant, J\. Wei, J\. Wei, C\. Tar, Y\. Sung, D\. Zhou, Q\. Le, and T\. Luong \(2023\)FreshLLMs: refreshing large language models with search engine augmentation\.arXiv preprint arXiv:2310\.03214\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. Glaese \(2025\)BrowseComp: a simple yet challenging benchmark for browsing agents\.arXiv preprint arXiv:2504\.12516\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Xie, D\. Gopinath, D\. Qiu, D\. Lin, H\. Sun, S\. Potdar, and B\. Dhingra \(2026\)Over\-searching in search\-augmented large language models\.arXiv preprint arXiv:2601\.05503\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Yue, B\. Wang, Z\. Chen, K\. Zhang, Y\. Su, and H\. Sun \(2023\)Automatic evaluation of attribution by large language models\.arXiv preprint arXiv:2305\.06311\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px5.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.arXiv preprint arXiv:2306\.05685\.Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px5.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024\)WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.10198#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AAppendix Overview

The appendix is an audit trail for the main paper rather than a second results narrative\. It follows the same order as the experiment: data selection, provider surfaces, response generation, oracle construction, and supporting diagnostics\.

Data and providers\.Appendix[B](https://arxiv.org/html/2607.10198#A2)documents how theSealQA\-Hardsample was selected; Appendix[C](https://arxiv.org/html/2607.10198#A3)fixes the provider requests and shared page\-fetch backend\.

Generation and traces\.Appendix[D](https://arxiv.org/html/2607.10198#A4)records the agent prompt contract, trace schema, and qualitative audit views\.

Metrics and results\.Appendix[E](https://arxiv.org/html/2607.10198#A5)defines the Kimi per\-URL oracle, validation checks, and derived metrics; Appendix[F](https://arxiv.org/html/2607.10198#A6)collects uncertainty intervals, decision\-cell diagnostics, token/fetch summaries, and artifact provenance\.

## Appendix BDataset Construction

The experiment uses a deterministic 100\-query sample fromSealQA\-Hard\. We assign stable query IDs by hashing question text, form strata overfreshness×\\timessearch\_results×\\timestopic, allocate slots by largest\-remainder proportional rounding, and sample within cells using seed 20260509\.

The table below records the concrete sample inputs and audit breadcrumbs so the sampling procedure can be checked without relying on prose alone\.

The selected sample preserves the source distribution closely\. Marginal counts are: 25 fast\-changing, 43 slow\-changing, and 32 never\-changing questions; 57 conflicting and 43 unhelpful search\-result labels; topics are Science & Technology 27, Sports 22, Entertainment 22, Others 12, Politics 9, and History & Geography 8\. Effective years are 61 before 2024, 16 in 2024, and 23 in 2025\. Multi\-label question\-type tags include 72 advanced reasoning, 61 entity/event disambiguation, 13 temporal tracking, 5 cross\-lingual reasoning, and 2 false\-premise questions\.

## Appendix CProvider Settings

The provider configuration is fixed byconfig/experiment\.yamland the provider\-specific YAML files underconfig/providers/\. All providers return web\-search results rather than provider\-side page extracts, and each adapter normalizes output to the same model\-visible schema before rendering\.

#### Brave\.

The adapter issues a GET request to/res/v1/web/searchwith count 10, offset 0, US/en locale,safesearch=moderate,result\_filter=web, and text decorations disabled\. The rendered snippet surface usesdescriptionplus provider\-nativeextra\_snippets\.

#### Tavily\.

The adapter issues a POST request to/searchwithsearch\_depth=basic,topic=general, max 10, answer/images/favicon disabled, andinclude\_raw\_content=false\. The rendered snippet surface usescontent\.

#### Firecrawl\.

The adapter issues a POST request to/v2/searchwith limit 10, web source only, US region, 60s provider timeout, invalid URLs retained, and provider\-side markdown scraping disabled\. The rendered snippet surface usesdescriptionor metadata description\.

The normalized result fields arerank,title,url,domain,snippet, andprovider\_metadata\. URL normalization removes fragments and common tracking parameters\. Brave’s additional snippet blocks are rendered as<extra\_snippet\>elements because they were visible to the agent as part of the provider surface; we do not reclassify them as fetched\-page evidence\.

### C\.1Shared Page Fetch Backend

All page text in the main experiment comes from the samefetch\_pagebackend, independent of the search provider\. The backend is Jina Reader markdown throughr\.jina\.ai; it is invoked only after the agent selects a priordocument\_id\. Fetch artifacts are cached as gzip JSON records keyed by normalized URL and backend\. Each record stores fetch status, final URL, content type, truncation flag, extractor name, character/token counts, a text hash, latency, and the extracted body\. The configured guardrails are a 15s timeout, a 2MB maximum read, and concurrency 4\. In the analyzed run, successful fetches are recorded asjina\_reader\_markdown; failures are retained as failed fetches rather than silently dropped\.

## Appendix DResponse Generation

The answer model, prompt, tools, and loop budget are fixed across providers\. Trace rows preserve the exact rendered request snapshots initerations\[\]\.llm\_request\.messages; the prompt templates live undersrc/searchapi\_eval/agent/prompts/\.

The next table lists the fixed generation contract\. Its purpose is to show which parts of the agent were held constant while the search provider changed\.

In fetch\-tool mode,search\_webobservations are snippet\-only\. Page text enters the model context only after the model chooses a priordocument\_id\. This makes fetch behavior observable and keeps provider comparisons from reflecting automatic page inclusion\.

### D\.1Trace Schema and Prompt Artifacts

Each provider\-query run is one JSONL trace row using the schema indata/trace\-schema\-v1\.md\. The trace records LLM request snapshots, model responses, tool calls, normalized search responses, raw provider payloads, fetch records, token counts, latency, final answers, and offline gold fields\. The model\-visible search surface is rendered as<search\_documents\>with one<document\>block per result\. Fetched pages appear as<fetched\_page\>blocks with URL provenance, fetch metadata, and extracted body content\.

The artifact map below connects those trace concepts to their JSON locations\. This is included so readers can see where the prompt, tool calls, provider payloads, and fetch decisions enter the audit pipeline\.

The canonical trace files contain 100 Brave rows, 101 Tavily rows, and 100 Firecrawl rows\. The extra Tavily row is a recovered transient failure; provider summaries select the latest valid row for each of the 100 query IDs, yielding the 300 analyzed provider\-query trajectories\.

#### Example trajectory\.

One qualitative failure case issealhard\_a2c20fcc3ff1: “How many times did Blake Shelton win as a coach on The Voice \(U\.S\.\) between 2015 and his departure in 2023?” The gold answer is 5, while all three provider runs answered 9\. In the Brave tracetrace\_5b6eea4e74d34191a4d275d656f82a18, the agent used 2 iterations, made 1 search call, made 0 fetch calls, and answered from the pre\-fetch surface\. The provider\-comparison row marks this as wrong despite answer text being visible somewhere in retrieved text\. The point of the example is not that a single snippet is decisive, but that the pre\-fetch surface mixed a period\-specific answer with a tempting lifetime\-win distractor\.

### D\.2Trace Viewer and Qualitative Audit

The repository includes human\-readable trajectory views inresults/trace\_views/\. The rendering script isscripts/render\_trace\.py, andscripts/trace\_dashboard\.pyprovides a local browsing dashboard\. These views were used for spot\-checking case studies, verifying multi\-call turns, and inspecting failures where answer text was present but unused\. They are not an independent metric source: the paper numbers come from trace JSONLs, judge JSONLs, the semantic TSV, and provider\-comparison summaries\.

## Appendix EOracle and Metrics

The Kimi\-K2\.6 judge is run one URL at a time\. The exact judge prompt template issrc/searchapi\_eval/evaluation/prompts/document\_support\_judge\.liquid; the execution wrapper isscripts/run\_llm\_judge\.py\. The judge receives the question, gold answer, model final answer, and one retrieved document rendered in the same XML\-like representation that the answer model saw\. Snippet\-only records contain title, URL, domain, snippets, extra snippets, and metadata\. Page\-visible records additionally contain the fetched page block that was returned byfetch\_page\.

Because the oracle is central to the paper’s measurements, the table below states the guardrails used to keep each judge row constrained, parseable, and auditable\.

The output schema fields used in the paper record gold support, snippet support, extracted\-page support, model\-answer support, contradiction, garbage status, evidence spans, and confidence\. Across the main artifacts, 6,869 of 6,909 rows are valid, split into 6,519 snippet\-only and 350 page\-visible rows\.

### E\.1Metric Definitions

For a providerppand queryqq, letUp,qU\_\{p,q\}denote the valid judged URLs visible in the trajectory\. Unfetched URLs have snippet\-only judgments; fetched URLs have page\-visible judgments that include both the pre\-fetch snippet fields and the fetched page block\. LetGpre​\(u\)G\_\{\\mathrm\{pre\}\}\(u\)be true when the pre\-fetch surface supports the gold answer: eithergold\_answer\_in\_snippetsis true, or a snippet\-only judgment markscontains\_gold\_answertrue\. LetGpage​\(u\)G\_\{\\mathrm\{page\}\}\(u\)be true when a page\-visible judgment marksgold\_answer\_in\_extracted\_pagetrue\. LetF​\(u\)F\(u\)be true when the trace contains afetch\_pagecall foruu\.

For each provider\-query pair, pre\-fetch support is∃u:Gpre​\(u\)\\exists u:G\_\{\\mathrm\{pre\}\}\(u\)\. Post\-fetch discovered support is¬∃u:Gpre​\(u\)\\neg\\exists u:G\_\{\\mathrm\{pre\}\}\(u\)and∃u:Gpage​\(u\)\\exists u:G\_\{\\mathrm\{page\}\}\(u\)\. Trajectory\-visible support is their union\.

The decision\-cell table turns these definitions into the four labels used in the main paper\. It is descriptive: it classifies what support was visible and what the agent fetched, rather than assigning causal blame\.

The partition is computed per provider\-query pair after joining judge rows to trace actions by normalized URL and retrieval provenance\. It is descriptive: amissedquery can still be answered correctly from snippets, and ablindquery can still succeed if a fetched page reveals evidence not present in snippets\. The surface contradiction\-to\-gold ratio is computed only over snippet\-only rows:

rc:g=\|\{u:contradicts\_gold\_answer​\(u\)\}\|\|\{u:contains\_gold\_answer​\(u\)\}\|\.r\_\{c:g\}=\\frac\{\|\\\{u:\\texttt\{contradicts\\\_gold\\\_answer\}\(u\)\\\}\|\}\{\|\\\{u:\\texttt\{contains\\\_gold\\\_answer\}\(u\)\\\}\|\}\.It is independent of the model’s final answer, but not independent of the gold answer\.

For answer\-level scoring, exact match and token F1 use the same deterministic normalizer: lowercasing, removing articles and punctuation, and mapping simple number words to digits\. Token F1 is the token\-overlap F1 between the normalized final answer and normalized gold answer, macro\-averaged over the 100 queries\. The semanticCorrectmetric is separate and comes from the auditedsemantic\_matchTSV,results/em\_vs\_semantic\_audit\.tsv\.

### E\.2Human Validation and Semantic Corrections

The manual validation app samples row/label pairs from the Kimi judge JSONLs usingscripts/build\_judge\_validation\_queue\.pyand records human judgments withscripts/judge\_validation\_app\.py\. The completed sample has 180 reviewed cases, balanced across provider, label, surface, and Kimi positive/negative value\.

The validation summary below reports agreement only on clear human judgments\. It is included to calibrate the Kimi oracle as an audit instrument, not to claim the judge is error\-free\.

The semantic audit marks some exact\-match misses as correct when the answer is semantically equivalent\. Representative examples are: Brave*Astra Zeneca*vs\.*AstraZeneca*; Brave*UnionPay*vs\.*China UnionPay*; Tavily*3 players*vs\.*3*; Firecrawl*Bohemian Rhapsody*vs\.*Bohemian Rhapsody, 9,948,386 viewers*; and Firecrawl*16 years*vs\.*16 years old*\. The audit TSV records the exact rows and judgment notes\.

## Appendix FResults

The result appendices report uncertainty, decision\-cell diagnostics, token and fetch diagnostics, and artifact provenance\. These tables support the main claims without changing the headline conclusions\.

### F\.1Paired Bootstrap Uncertainty

We resample question IDs with replacement, preserving the matched Brave/Tavily/Firecrawl rows for each question\. The tables use 10,000 percentile\-bootstrap replicates with seed 20260628\. Count metrics are reported per 100 sampled questions; pairwise differences are left minus right in the row’s native units\.

The first uncertainty table gives provider\-level intervals for the main quantities\. These intervals show the scale of each provider’s measured regime before any pairwise comparison is made\. The rank\-1 row uses the same denominator as Table[3](https://arxiv.org/html/2607.10198#S5.T3): gold\-supporting pre\-fetch rows\.

Table 7:Provider bootstrap 95% intervals\. B=Brave, T=Tavily, F=Firecrawl\.
The second uncertainty table reports matched pairwise differences\. This is the comparison most directly tied to the main claim boundary: correctness differences include zero, while evidence\-economy contrasts are more stable\.

Table 8:Paired bootstrap differences; left minus right\.

### F\.2Additional Diagnostics and Artifact Manifest

The remaining diagnostics are lower\-level checks behind the headline tables\. They are separated from the main results because they support interpretation but are not themselves the paper’s primary claims\.

The retrieval\-proxy table reports deterministic string and URL checks from traces\. These proxies help show why answer visibility is not the same as correct evidence use\.

Table 9:Trace\-derived retrieval proxies\. B/T/F = Brave/Tavily/Firecrawl; these deterministic URL and answer\-string checks are not oracle support labels\.
The Wilson\-interval table expands the decision partition by provider and cell\. It is included because some cells are small, so raw cell accuracies should be read with uncertainty\.

Table 10:Decision\-cell Wilson 95% intervals\.
The overlap table gives a compact view of which providers answer the same questions correctly\. It supports the complementarity claim by separating shared successes from provider\-specific successes\.

Table 11:Semantic\-correct pairwise overlap: both / left / right / wrong\.
The final diagnostic table reports token and fetch volume\. These counts ground the paper’s cost and retrieval\-budget discussion in the actual traced runs\.

Table 12:Token and fetch diagnostics\. B/T/F = Brave/Tavily/Firecrawl\.
Artifact status\.Private inputs are the semantic\-audit TSV, three phase\-1 trace JSONLs, three Kimi per\-URL judge JSONLs, and provider\-comparison summaries\.

Released\.Code, prompts, configuration, query sampling, trace schema, evaluation and bootstrap scripts, figure\-generation code, paper sources, and aggregate derived outputs\.

Not redistributed\.Raw provider responses, rendered trace JSONLs, judge prompts/responses containing snippets, and fetched page text, because they contain provider search results and third\-party web content\.