When Should LLMs Search? Counterfactual Supervision for Search Routing

# When Should LLMs Search? Counterfactual Supervision for Search Routing
Source: [https://arxiv.org/html/2607.05752](https://arxiv.org/html/2607.05752)
###### Abstract

Search\-augmented language models can use external evidence to compensate for limitations in parametric knowledge, but search is not uniformly beneficial: models may call search for questions they can already answer, or rely on noisy evidence when correction, clarification, or abstention would be more appropriate\. We formulate this as an instance\-level search\-routing problem: deciding whether search is needed to improve task success relative to a no\-search execution\. To derive supervision, we compare no\-search and forced\-search outcomes for the same question and construct an oracle overNO\_SEARCH,SEARCH, andUNSOLVEDbased on task\-specific success\. Using this oracle as both an evaluation criterion and a learning signal, we train search\-routing policies with supervised fine\-tuning and preference optimization, improving routing macro\-F1 on oracle\-eligible examples from 0\.7082 to 0\.8235 for Gemma E2B and from 0\.7053 to 0\.8365 for Qwen3\.5\-4B\. Further analysis shows that the learned policies reduce model\-specific routing failures: Gemma primarily learns no\-search restraint, while Qwen further reduces missed search; residualUNSOLVEDcases reveal heterogeneous bottlenecks involving model capacity, retrieval budget, evidence use, and policy behavior\.

Machine Learning, ICML

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.05752v1/x1.png)Figure 1:Motivation for instance\-level search routing\. Search can help, be unnecessary, or mislead, motivating instance\-level routing\.Language models equipped with search tools can use external evidence to answer questions that are difficult to resolve from parametric knowledge alone, including queries about long\-tail factual knowledge, recently updated information, or facts that are unlikely to be reliably stored in model parameters\(Lewiset al\.,[2020](https://arxiv.org/html/2607.05752#bib.bib16); Izacardet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib15); Mallenet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib22)\)\. However, the availability of a search tool does not imply that search should be used for every question\. For questions the model can already answer, responding without search may be cheaper and less vulnerable to noisy retrieved evidence\. For questions with false premises or missing context, the appropriate response may be to correct the premise, ask for clarification, or abstain, rather than retrieve external evidence\(Amayuelaset al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib23); Xieet al\.,[2026](https://arxiv.org/html/2607.05752#bib.bib24)\)\. In practice, the correct first action depends on the instance: a long\-tail factual question may require search to recover evidence, whereas an underspecified question may be better resolved by clarification than by retrieving generic search results, as illustrated in Figure[1](https://arxiv.org/html/2607.05752#S1.F1)\.

We therefore view search\-augmented language models not merely as systems that generate search queries, but as systems that must first perform*search routing*: deciding, for each input question, whether to respond without search or call a search tool\. This first\-action decision is important because routing errors arise in two opposing directions\. A model that is too reluctant to search may answer directly even when its parametric knowledge is insufficient, producing plausible but incorrect responses\. Conversely, a model that relies too heavily on search may invoke retrieval for questions it can already answer, or for questions that search cannot resolve, increasing cost and exposing the final answer to irrelevant or misleading evidence\. The objective is not aggregate search frequency, but calling search only when it improves task success\.

This framing differs from standard tool\-use evaluation\. Many tool\-calling benchmarks focus on whether a model calls the correct tool with the correct arguments once tool use is known to be appropriate\(Patilet al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib26)\)\. Recent work on when\-not\-to\-call behavior broadens this view by evaluating whether models can avoid unnecessary tool calls, ask follow\-up questions, or admit inability to answer\(Rosset al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib25)\)\. Our setting focuses specifically on search\-augmented question answering, where the same tool may be helpful for one factual question, unnecessary for another, and actively unhelpful for a search\-boundary question requiring correction, clarification, or abstention\. This motivates an instance\-level supervision signal that is grounded in observed response outcomes rather than in a direct judgment of whether search “seems useful\.”

We address this problem through two research questions\.

RQ1\.How accurately can a search\-enabled language model identify the instances for which search improves task success relative to responding without search?

RQ2\.Can models be trained to improve this instance\-level search\-routing behavior, reducing both missed search and unnecessary search?

To operationalize search need without relying on a direct human or judge annotation of search usefulness, we construct paired counterfactual traces for each question\. In the no\-search trace, denotedN​\(q\)N\(q\), the model responds without access to the search tool\. In the forced\-search trace, denotedS​\(q\)S\(q\), the model must first issue a search call and then generate a final response using the returned evidence\. A task\-specific evaluator maps each final response to binary success: factual correctness for answerable questions, and adequate resolution for false\-premise or underspecified questions, including correction, clarification, or abstention\.

We then derive a routing oracle from the paired outcomes\. If the no\-search trace succeeds, we assignNO\_SEARCH, since search is not necessary even if the forced\-search trace also succeeds\. If no\-search fails but forced\-search succeeds, we assignSEARCH, since search recovers a no\-search failure\. If both traces fail, we assignUNSOLVED\. These examples do not provide a reliable binary routing target: the failure may arise from retrieval quality, query formulation, evidence use, answer synthesis, model capacity, or the question itself\. We therefore excludeUNSOLVEDexamples from routing training and routing\-accuracy computation, while retaining them as a diagnostic subset\.

This outcome\-based oracle serves both evaluation and training\. For evaluation, we collect a free\-policy traceP​\(q\)P\(q\)in which the search tool is available but not forced, and compare the model’s first action against the oracle label on examples labeledNO\_SEARCHorSEARCH\. This free\-policy trace is a prompt\-only selective\-search baseline: the base model is explicitly instructed to call search only when it would materially improve the response, and otherwise to answer, correct, clarify, or abstain without using the tool\. This lets us distinguish missed search, where the model answers without search despite search being necessary, from unnecessary search, where the model calls search despite no\-search being sufficient\. For training, we use the same oracle to construct supervised fine\-tuning examples and preference pairs\. Supervised fine\-tuning teaches the model to imitate oracle\-consistent first actions, while Preference Optimization encourages the model to prefer the appropriate first assistant completion for the same question\.

Empirically, search need is model\-dependent even with the same question set and search tool: parametric knowledge and search\-conditioned solving ability can change whether an instance is labeledNO\_SEARCH,SEARCH, orUNSOLVED\. Prompted base models nevertheless exhibit both under\-search and over\-search, showing that selective\-search instructions alone do not recover this boundary\. Training on model\-specific counterfactual outcomes improves routing: on held\-out oracle\-eligible examples, SFT and Preference Optimization raise macro\-F1 from 0\.7082 to 0\.8235 for Gemma E2B and from 0\.7053 to 0\.8365 for Qwen3\.5\-4B; error\-direction andUNSOLVEDanalyses separate corrected routing errors from residual bottlenecks beyond first\-action routing\.

![Refer to caption](https://arxiv.org/html/2607.05752v1/x2.png)Figure 2:Counterfactual search\-routing pipeline\. \(A\) We collect controlled no\-search and forced\-search traces, together with a free\-policy trace collected under a selective\-search prompt\. \(B\) Paired no\-search and forced\-search outcomes induce a routing oracle overNO\_SEARCH,SEARCH, andUNSOLVED\. \(C\) The oracle is used both for routing supervision and for free\-policy routing evaluation\.These findings rest on three contributions\.

- •Instance\-level routing\.We frame search use as an instance\-levelNO\_SEARCH/SEARCH\\texttt\{NO\\\_SEARCH\}/\\texttt\{SEARCH\}decision, whereNO\_SEARCHincludes direct answering, premise correction, clarification, and abstention\.
- •Counterfactual oracle\.We derive an outcome\-based oracle from paired no\-search and forced\-search traces and use it for both routing evaluation and training\.
- •UNSOLVEDdiagnostics\.We retain dual\-failure cases as a diagnostic subset, showing residual retrieval, evidence\-use, synthesis, and model\-capacity bottlenecks beyond the initial search decision\.

## 2Method

We formalize search routing as a first\-action decision\. Given a questionqq, the policy selectsa​\(q\)∈\{NO\_SEARCH,SEARCH\}a\(q\)\\in\\\{\\texttt\{NO\\\_SEARCH\},\\texttt\{SEARCH\}\\\}, whereNO\_SEARCHdenotes a no\-tool first response andSEARCHdenotes a first\-turn search call\. Figure[2](https://arxiv.org/html/2607.05752#S1.F2)gives an overview; below we define the traces, oracle, evaluation target, and training objectives\.

For each questionqq, we collect two controlled counterfactual traces:N​\(q\)N\(q\), generated with search disabled, andS​\(q\)S\(q\), generated under a forced first\-turn search call\. We evaluate only the final response in each trace:

yN​\(q\)=Eval​\(q,N​\(q\)\),yS​\(q\)=Eval​\(q,S​\(q\)\),y\_\{N\}\(q\)=\\mathrm\{Eval\}\(q,N\(q\)\),\\qquad y\_\{S\}\(q\)=\\mathrm\{Eval\}\(q,S\(q\)\),whereyN,yS∈\{0,1\}y\_\{N\},y\_\{S\}\\in\\\{0,1\\\}\.Eval\\mathrm\{Eval\}applies task\-specific success criteria; details and prompt templates are in Appendices A and C\.

We derive a routing oracle from the paired outcomes:

o​\(q\)=\{NO\_SEARCH,yN​\(q\)=1,SEARCH,yN​\(q\)=0∧yS​\(q\)=1,UNSOLVED,yN​\(q\)=0∧yS​\(q\)=0\.o\(q\)=\\begin\{cases\}\\texttt\{NO\\\_SEARCH\},&y\_\{N\}\(q\)=1,\\\\ \\texttt\{SEARCH\},&y\_\{N\}\(q\)=0\\land y\_\{S\}\(q\)=1,\\\\ \\texttt\{UNSOLVED\},&y\_\{N\}\(q\)=0\\land y\_\{S\}\(q\)=0\.\\end\{cases\}Rows witho​\(q\)=UNSOLVEDo\(q\)=\\texttt\{UNSOLVED\}are excluded from routing supervision and routing metrics because paired failures do not identify a reliable first\-action target\. We retain them for diagnostic analysis in Section 5 and Appendix B\. Table[1](https://arxiv.org/html/2607.05752#S2.T1)summarizes the induced buckets and labels\.

Table 1:Oracle labels induced by counterfactual no\-search and forced\-search outcomes\.Because the no\-search and forced\-search outcomes can differ across models, the induced oracle is model\-specific rather than a fixed question\-level annotation\. Supervision therefore targets the current model’s boundary between cases it can solve without search and cases recovered by search\. If no\-search already succeeds, the instance is labeledNO\_SEARCHeven when forced search also succeeds, soSEARCHlabels mark only observed no\-search\-to\-search recoveries\.

For free\-policy evaluation, we collectP​\(q\)P\(q\)under a selective\-search prompt: search is available but should be called only when it materially improves the response; otherwise the model should answer, correct, clarify, or abstain without the tool\. We parse the first action asp​\(q\)=SEARCHp\(q\)=\\texttt\{SEARCH\}if it calls search andp​\(q\)=NO\_SEARCHp\(q\)=\\texttt\{NO\\\_SEARCH\}otherwise\. When collected from the base model before routing training, this trace is denotedPpre​\(q\)P\_\{\\mathrm\{pre\}\}\(q\), and the corresponding prompted base routing policy isPpreP\_\{\\mathrm\{pre\}\}\. Routing evaluation measures agreement betweenp​\(q\)p\(q\)ando​\(q\)o\(q\)on examples witho​\(q\)∈\{NO\_SEARCH,SEARCH\}o\(q\)\\in\\\{\\texttt\{NO\\\_SEARCH\},\\texttt\{SEARCH\}\\\}\. We callo​\(q\)=SEARCH,p​\(q\)=NO\_SEARCHo\(q\)=\\texttt\{SEARCH\},p\(q\)=\\texttt\{NO\\\_SEARCH\}a missed search, ando​\(q\)=NO\_SEARCH,p​\(q\)=SEARCHo\(q\)=\\texttt\{NO\\\_SEARCH\},p\(q\)=\\texttt\{SEARCH\}an unnecessary search\. Generation details and prompts are in Appendices A and C\.

The free\-policy prompt and search interface are held fixed forPpreP\_\{\\mathrm\{pre\}\}collection, training inputs, and post\-training rollouts, so comparisons amongPpreP\_\{\\mathrm\{pre\}\}, SFT, and Preference Optimization reflect learned routing changes rather than instruction changes\. SFT uses only oracle\-eligible examples:NO\_SEARCHtargets are successfulN​\(q\)N\(q\)responses, andSEARCHtargets are first\-turn search calls from successfulS​\(q\)S\(q\)traces, focusing supervision on the routing action rather than the full post\-search trajectory\.

Thus, theSEARCH\-labeled SFT examples do not require the model to imitate an entire searched answer trajectory\. They supervise the decision to enter the search path, while leaving query refinement, evidence use, and final synthesis outside the main routing objective\. This choice is consistent with our evaluation, which scores the first action rather than the quality of all subsequent search\-conditioned steps\.

For Preference Optimization, each eligibleqqhas a chosen responser\+​\(q\)r^\{\+\}\(q\)beginning with the oracle\-consistent first action and a rejected responser−​\(q\)r^\{\-\}\(q\)beginning with the opposite action, preferably drawn from an actualPpreP\_\{\\mathrm\{pre\}\}mistake and otherwise from the controlled counterfactual trace\.

We optimize a DPO objective with chosen\-response NLL regularization\(Rafailovet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib4); Panget al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib3)\):

ℒ=ℒDPO\+α​ℒNLL​\(r\+\)\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\+\\alpha\\,\\mathcal\{L\}\_\{\\mathrm\{NLL\}\}\(r^\{\+\}\)\.Preference Optimization is initialized from the selected SFT checkpoint and uses that checkpoint as the reference model\.

## 3Experimental Setup

Data and evaluation\.We instantiate the counterfactual routing protocol on two complementary question regimes\. PopQA provides answerable entity\-centric factual questions, for which search may recover long\-tail facts; we use all 13,763 questions\(Mallenet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib22)\)\. KUQ provides search\-boundary questions for which search may or may not be useful: some cases may benefit from external evidence, while others are better resolved through premise correction, clarification, or abstention\(Amayuelaset al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib23)\)\. We use the False Assumption category, where the model should reject or correct a false premise, and the Ambiguous category, where the model should recognize missing context and ask for clarification or abstain\. The KUQ subset contains 774 False Assumption questions and 850 Ambiguous questions\.

Importantly, these dataset groups are not treated as fixed routing labels\. PopQA questions are not assumed to require search: if the no\-search trace answers correctly, the instance is labeledNO\_SEARCH\. Likewise, KUQ false\-assumption or ambiguous questions are not assumed to forbid search: if no\-search fails but forced search enables the model to resolve the premise or ambiguity, the instance is labeledSEARCH\. The mixed benchmark therefore tests instance\-level routing rather than dataset\-level classification\.

PopQA is evaluated with a PopQA\-adapted SimpleQA\-style factuality judge that performs alias\-aware correctness checking\(Weiet al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib6)\), while KUQ is evaluated by category\-specific resolution adequacy\. Evaluator settings and binary success mappings are provided in Appendix A; full judge prompts are provided in Appendix C\.

Models and search environment\.We evaluate Qwen3\.5\-4B\(Qwen Team,[2026](https://arxiv.org/html/2607.05752#bib.bib7)\)and Gemma E2B\(Google DeepMind,[2026](https://arxiv.org/html/2607.05752#bib.bib5)\), comparingPpreP\_\{\\mathrm\{pre\}\}, SFT, and Preference Optimization on the test split\.PpreP\_\{\\mathrm\{pre\}\}is the base\-model rollout under the same selective\-search prompt andsearch\(query\)interface used for post\-training evaluation\. The interface is backed by Brave LLM Context and returns source\-backed snippets\. We fix the backend, interface, and routing instruction to isolate first\-action search routing; budgets and forced\-search drop handling are in Appendix A\.

Trace collection\.For each question, we collect the three traces defined in Section 2\. The no\-search trace disables tools; the forced\-search trace uses prompt\-level forcing and acceptance filtering for a valid first\-turn search call; and the free\-policy trace uses the fixed selective\-search prompt without first\-action forcing\. Retry/drop handling and prompt templates are in Appendices A and C\.

Training and metrics\.We split data into train, dev, and test before training\. SFT and Preference Optimization use only oracle\-eligible train rows, formatted with the same free\-policy prompt used forPpreP\_\{\\mathrm\{pre\}\}and post\-training evaluation\.UNSOLVEDrows are excluded from training and routing metrics but retained for diagnostics\. Dev selects checkpoints; test is used only for final reporting\. Because oracle labels are model\-specific, comparisons are primarily within model across policies rather than direct cross\-model rankings\.

On oracle\-eligible examples, we report routing accuracy and macro\-F1, the unweighted average ofSEARCHandNO\_SEARCHF1\. Directional error rates useSEARCHas the positive class, so false positives are unnecessary search and false negatives are missed search\. Over\-search, or unnecessary\-search rate, is

OverSearch=F​PT​N\+F​P,\\mathrm\{OverSearch\}=\\frac\{FP\}\{TN\+FP\},the false\-positive rate conditioned on oracleNO\_SEARCHexamples\. Under\-search, or missed\-search rate, is

UnderSearch=F​NF​N\+T​P,\\mathrm\{UnderSearch\}=\\frac\{FN\}\{FN\+TP\},the false\-negative rate conditioned on oracleSEARCHexamples\.

For category\-level reporting, accuracy is computed within each dataset/category on oracle\-eligible examples only; FA Acc\. and Amb\. Acc\. are KUQ routing accuracies, not final\-response accuracies\. Split sizes, conditional\-error denominators, and hyperparameters are in Appendix A\.

Table 2:Main search\-routing results on oracle\-eligible test examples\. Over\-search and under\-search are conditional error rates\.
## 4Results

This section answers the two research questions\. We first compare the counterfactual outcomes of no\-search and forced\-search traces, showing that the effect of search depends strongly on the question type and instance\. We then analyze the under\-search and over\-search errors exhibited by the base free\-policy executionPpreP\_\{\\mathrm\{pre\}\}, and evaluate whether SFT and Preference Optimization improve first\-action search routing\.

### 4\.1Counterfactual Outcomes Show Instance\-Dependent Search Utility

We first compare task success between controlled no\-search and forced\-search traces\. Table[3](https://arxiv.org/html/2607.05752#S4.T3)shows that search has opposite effects across question groups\. In PopQA, forced search achieves substantially higher success rates than no search for both models\. Qwen3\.5\-4B improves from 0\.1831 under no search to 0\.5096 under forced search, while Gemma E2B improves from 0\.1504 to 0\.4813\. PopQA therefore supplies many cases in which search is beneficial: factual failures that are not resolved without search can often be recovered when the model sees external evidence\.

Table 3:Counterfactual search outcomes\. Task success under no\-search and forced\-search conditions\.By contrast, for KUQ False Assumption and Ambiguous questions, forced search yields lower success rates than no search\. Qwen3\.5\-4B decreases from 0\.6441 to 0\.5376, and Gemma E2B decreases from 0\.6022 to 0\.4483\. For these questions, no\-search behaviors such as correcting a false premise, requesting missing information, or abstaining may satisfy the task\-specific success criterion more reliably than searching for additional evidence\.

The oracle\-bucket distributions in Tables[A\.5](https://arxiv.org/html/2607.05752#A1.T5)and[A\.6](https://arxiv.org/html/2607.05752#A1.T6)\(Appendix A\) further show that search utility varies by instance across both PopQA and KUQ search\-boundary questions\.

### 4\.2SFT and Preference Optimization Improve Search Routing

Table[2](https://arxiv.org/html/2607.05752#S3.T2)compares the search\-routing performance ofPpreP\_\{\\mathrm\{pre\}\}, SFT, and Preference Optimization on the test split\. Evaluation is conducted on oracle\-eligible test examples, i\.e\., examples whose oracle label is eitherNO\_SEARCHorSEARCH\. We report routing macro\-F1, routing accuracy, conditional over\-search, conditional under\-search, and subset routing accuracies for KUQ False Assumption and Ambiguous questions\.

BecausePpreP\_\{\\mathrm\{pre\}\}, SFT, and Preference Optimization are all evaluated under the same selective\-search prompt and search interface, these comparisons do not conflate training gains with prompt changes\. Moreover,PpreP\_\{\\mathrm\{pre\}\}is a prompt\-only selective\-search baseline rather than an unconstrained tool\-available rollout: the base model is already instructed to use search only when it would materially help and to avoid unnecessary tool use\. The remainingPpreP\_\{\\mathrm\{pre\}\}errors therefore reflect failures to apply this selective\-use instruction at the instance level\.

SFT substantially improves routing performance for both models beyond this prompt\-only baseline\. Gemma E2B improves from aPpreP\_\{\\mathrm\{pre\}\}macro\-F1 of 0\.7082 and routing accuracy of 0\.7574 to 0\.8207 and 0\.8292 after SFT\. Qwen3\.5\-4B similarly improves from aPpreP\_\{\\mathrm\{pre\}\}macro\-F1 of 0\.7053 and routing accuracy of 0\.7148 to 0\.8207 and 0\.8263 after SFT\. The counterfactual oracle therefore provides a usable learning signal for sharpening the first\-action routing boundary beyond prompt\-only selective search\.

Preference Optimization further shifts the routing boundary after SFT\. For Qwen3\.5\-4B, macro\-F1 increases from 0\.8207 to 0\.8365 relative to SFT, and routing accuracy increases from 0\.8263 to 0\.8427\. The conditional under\-search rate decreases from 0\.1339 to 0\.1034, further reducing cases where the model omits search for questions that require it\. For Gemma E2B, the aggregate gain is smaller, but macro\-F1 increases from 0\.8207 to 0\.8235 and routing accuracy from 0\.8292 to 0\.8329\. Preference Optimization also improves the KUQ search\-boundary subsets, especially Ambiguous, where accuracy increases from 0\.8868 to 0\.9434\.

### 4\.3Error Directions Differ Across Model Families

The error directions in Table[2](https://arxiv.org/html/2607.05752#S3.T2)show that the two models have different initial search\-routing tendencies\. Gemma E2B exhibits pronounced over\-search onNO\_SEARCHoracle rows underPpreP\_\{\\mathrm\{pre\}\}\. SFT greatly reduces this over\-search, although it introduces a higher under\-search rate\. Preference Optimization does not completely remove this trade\-off, but it slightly reduces under\-search and yields additional gains on the KUQ subsets, especially Ambiguous\.

By contrast, Qwen3\.5\-4B exhibits both over\-search and under\-search before training, and SFT mitigates both error types\. Preference Optimization further reduces under\-search while keeping over\-search nearly unchanged, suggesting that it makes the policy more willing to search onSEARCH\-oracle cases without substantially increasing unnecessary search onNO\_SEARCH\-oracle cases\.

A single global search\-rate target would fail for both models\. For a search\-heavy model such as Gemma E2B, learning no\-search restraint is important; for a model such as Qwen3\.5\-4B, which also misses questions requiring search, repairing missed search is important\. Search\-routing training should therefore not simply increase or decrease the search\-call rate globally\. Instead, it should jointly correct under\-search and over\-search according to each model’s initial routing tendency\.

## 5Diagnostic Analysis: Heterogeneous Failures inUNSOLVEDCases

The oracle intentionally excludesUNSOLVEDexamples from routing training and routing evaluation\. These are questions for which both the no\-search and forced\-search traces fail,yN​\(q\)=0y\_\{N\}\(q\)=0andyS​\(q\)=0y\_\{S\}\(q\)=0\. Such failures do not identify a reliable first\-action target\. A failed no\-search trace may reflect insufficient parametric knowledge or generation capacity, while a failed forced\-search trace may reflect query formulation, retrieval coverage, evidence use, answer synthesis, or model capacity\(Krishnaet al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib8)\)\. We therefore treatUNSOLVEDas a diagnostic category rather than as either aNO\_SEARCHorSEARCHrouting label\.

To focus on persistent controlled\-trace failures, we additionally evaluate the base selective\-search free\-policy responsePpre​\(q\)P\_\{\\mathrm\{pre\}\}\(q\)\. Cases in whichyN​\(q\)=0y\_\{N\}\(q\)=0,yS​\(q\)=0y\_\{S\}\(q\)=0, butPpreP\_\{\\mathrm\{pre\}\}succeeds are assigned to an escape subset\. We focus on the hard subset: examples for which the no\-search trace, forced\-search trace, and base free\-policy trace all fail\.

On this hard subset, we test four diagnostic probes: a larger model without searchMNM\_\{N\}, a larger model with forced searchMSM\_\{S\}, an expanded retrieval\-budget condition using the same base modelRTRR\_\{\\mathrm\{TR\}\}, and the selected post\-training policy rolloutPPOSTP\_\{\\mathrm\{POST\}\}\. For Qwen3\.5\-4B, the larger diagnostic model used inMNM\_\{N\}andMSM\_\{S\}is Qwen3\.5\-9B\(Qwen Team,[2026](https://arxiv.org/html/2607.05752#bib.bib7)\); for Gemma E2B, it is Gemma E4B\(Google DeepMind,[2026](https://arxiv.org/html/2607.05752#bib.bib5)\)\. These probes are diagnostic signatures rather than independent causal factors\. For example,MSM\_\{S\}jointly changes model capacity, query formulation, evidence use, and answer synthesis\. We therefore examine which probe uniquely rescues each hard case\.

Table 4:Exact\-only rescue signatures onN0\_S0hard subsets\. Each column reports examples solved only under that diagnostic probe while the other three probes fail\. Examples rescued by multiple probes are excluded\. Entries show rescue rate within each model’s hard subset\.Table[4](https://arxiv.org/html/2607.05752#S5.T4)reports the exact\-only rescue signatures\. These exclusive rescue patterns show that the hardN0\_S0subset is not a single failure type\. Some cases are recovered only by larger\-model no\-search inference, suggesting failures associated with parametric knowledge or no\-search generation capacity\. A larger share is recovered only by larger\-model forced search, suggesting failures that require search\-conditioned larger\-model inference rather than larger no\-search inference alone\. Expanded retrieval also uniquely recovers a nontrivial share of cases, indicating that retrieval budget or search\-context coverage can be the limiting factor even when the base forced\-search trace fails\. Finally, post\-training policy rollout uniquely recovers some hard cases, showing that routing\-related behavior can coexist with other sources of failure even within theN0\_S0region\.

These exclusive rescue signatures should not be read as a causal decomposition of mutually exclusive error sources\. Instead, they show why collapsingN0\_S0into eitherNO\_SEARCHorSEARCHwould be misleading: the bucket contains a mixture of model\-capacity limitations, search\-conditioned synthesis failures, retrieval\-budget failures, and policy\-related failures\.UNSOLVEDis therefore better treated as a separate diagnostic category for failures beyond reliable first\-action routing supervision\.

## 6Related Work

Adaptive retrieval and selective search\.Retrieval augmentation is a standard way to compensate for limitations in LLMs’ parametric knowledge\(Lewiset al\.,[2020](https://arxiv.org/html/2607.05752#bib.bib16); Izacardet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib15)\)\. Prior work on long\-tail factual QA shows that language models struggle to recall less frequent entity knowledge, and that retrieval can substantially improve performance on such questions\(Mallenet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib22)\)\. However, retrieval is not uniformly beneficial: for questions the model can already answer, retrieved context may instead distract the model or degrade the response\(Mallenet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib22); Wanget al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib21)\)\. Self\-knowledge\-guided retrieval is similarly motivated by the observation that retrieved knowledge is not always helpful, and proposes selective retrieval based on whether a model appears to know the answer\(Wanget al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib21)\)\. Other adaptive retrieval systems trigger retrieval during generation or select among retrieval strategies using confidence, reflection, or question\-complexity signals\(Jianget al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib19); Asaiet al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib14); Jeonget al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib20)\)\. Our work shares the view that search should be used selectively, but differs in how the routing signal is constructed\. Rather than predicting search necessity with a separate classifier, heuristic, or self\-knowledge prompt, we compare no\-search and forced\-search task outcomes for the same question and derive instance\-level routing supervision from observed success and failure\.

Tool calling and when\-not\-to\-call behavior\.Tool\-use and tool\-calling work primarily evaluates whether models can generate the correct tool name, arguments, and executable function calls\(Schicket al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib13); Liet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib18); Qinet al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib11); Zhanget al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib17); Luet al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib2); Patilet al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib26)\)\. BFCL is a representative benchmark that evaluates function\-calling accuracy across single\-turn, multi\-turn, and agentic settings\(Patilet al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib26)\)\. When2Call moves closer to the question of when tools should not be called by asking models to choose among tool calling, direct answering, follow\-up questioning, and being unable to answer\(Rosset al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib25)\)\. However, When2Call is formulated as a multiple\-choice benchmark, and its questions are often structured so that direct answering is an inappropriate or hallucination\-prone option\. By contrast,NO\_SEARCHin our setting is not defined as a failure behavior\. In PopQA, it may correspond to a correct factual answer produced without search; in KUQ\-style boundary cases, it may correspond to premise correction, clarification, or abstention\. Thus, our work learns a model\-specific search\-routing boundary from free\-form execution traces, based on whether an actualsearch\(query\)execution recovers task success\.

Over\-searching and uncertainty\-aware no\-search behavior\.Recent work on over\-searching shows that search can improve answer accuracy for answerable queries while reducing abstention accuracy and increasing cost for unanswerable or underspecified queries\(Xieet al\.,[2026](https://arxiv.org/html/2607.05752#bib.bib24)\)\. Such work highlights the risks of unnecessary search, especially when retrieved evidence is noisy or when the question should instead be corrected, clarified, or rejected\. This also relates to work on aligning assistants to refuse or say “I don’t know” on model\-specific unknown questions\(Chenget al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib27)\)\. Our work shares this concern, but does not focus only on over\-search\. We jointly address over\-search, where the model calls search for questions where it is unnecessary, and under\-search, where the model omits search for questions where search would recover a no\-search failure\. In addition, we convert observed no\-search and forced\-search outcomes into supervision for SFT and Preference Optimization, directly training policies to correct bidirectional search\-routing errors\.

Known\-unknown questions and response\-level evaluation\.Known\-unknown question datasets such as KUQ categorize sources of uncertainty, including false assumptions and ambiguous or underspecified questions\(Amayuelaset al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib23)\)\. These categories are useful for our setting because they expose cases in which the value of search is itself uncertain\. A false\-premise or ambiguous question is not automatically aNO\_SEARCHcase: in some instances, external evidence may help the model detect the false premise, identify the missing context, or recover the intended interpretation\. In other instances, searching may be unnecessary or even counterproductive, because the appropriate response is to correct the premise, ask for clarification, or abstain rather than to retrieve more evidence\. We therefore use these questions as search\-boundary cases rather than as fixed no\-search examples\. Their routing labels are determined by the same counterfactual outcome rule as factual QA: if the no\-search trace resolves the question, the instance is labeledNO\_SEARCH; if no\-search fails but forced search succeeds, it is labeledSEARCH; and if both fail, it is excluded from routing supervision asUNSOLVED\.

Our framework also separates response\-level evaluation from routing\-label construction\. We adapt a SimpleQA\-style factuality judge to PopQA with alias\-aware correctness checking\(Weiet al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib6)\), while KUQ\-style false\-premise and ambiguous questions are evaluated with resolution\-oriented rubrics that reward appropriate correction, clarification, or abstention when these resolve the user query\. Because our labels depend on free\-form final responses, this evaluation setup also relates to LLM\-as\-judge work, although we restrict the judge to task\-success grading rather than asking it to infer search helpfulness directly\(Liuet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib9); Zhenget al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib10)\)\. Each trace\-level outcome is then converted into binary task success before routing labels are constructed\. This separation allows factual QA and search\-boundary QA to use different task\-specific success criteria while still sharing the same first\-action routing space,NO\_SEARCHversusSEARCH\.

## 7Limitations

This study has several limitations\. First, the oracle should not be interpreted as an absolute search\-helpfulness label, but as a trace\-conditional routing oracle based on observed counterfactual traces\. Whether a question is labeledNO\_SEARCH,SEARCH, orUNSOLVEDdepends on the success ofN​\(q\)N\(q\)andS​\(q\)S\(q\)under the current model, search tool, prompt, evidence budget, and judge configuration\. With a better query, a different retrieval backend, a longer tool\-use trajectory, or a larger model, the outcome for the same question may change\. Therefore, the oracle should be interpreted as search\-routing supervision observed under specific model and tool conditions, rather than as an absolute judgment that a question intrinsically requires search\.

Second, this study focuses on first\-action search routing rather than the full search\-augmented answering pipeline\. We evaluate and train whether the model calls search as its first action or responds without search, but we do not directly optimize query refinement, multi\-hop retrieval, evidence selection, or answer synthesis after search\. In real search\-augmented systems, several stages of decision\-making may follow the first search call, and such multi\-step search behavior is beyond the scope of this work\(Yaoet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib12); Jianget al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib19); Krishnaet al\.,[2025](https://arxiv.org/html/2607.05752#bib.bib8)\)\. The results should therefore be understood as improving the initial routing boundary that determines whether search is used, rather than as solving tool use or search\-augmented answering as a whole\.

Third, although theUNSOLVEDdiagnostic separates failures beyond routing, it does not provide full causal attribution\. Through theMNM\_\{N\},MSM\_\{S\},RTRR\_\{\\mathrm\{TR\}\}, andPPOSTP\_\{\\mathrm\{POST\}\}interventions, we compare whether larger\-model no\-search capacity, larger\-model search\-conditioned solving, expanded retrieval evidence, and post\-training policy rollout recover some hard cases\. However, these interventions do not isolate mutually independent causes\. For example,MSM\_\{S\}jointly changes model capacity, query formulation, evidence use, and answer synthesis\. This analysis should therefore be interpreted as a diagnostic comparison of the additional conditions under whichUNSOLVEDcases can be recovered, rather than as a causal decomposition of failure sources\. More precise failure attribution and pipeline\-level optimization are left for future work\.

## 8Conclusion

We study search routing in search\-augmented LLMs: the problem of deciding, for each question, whether to call search as the first action\. Rather than asking a judge to label search usefulness directly, we compare no\-search and forced\-search responses for the same question and construct an outcome\-based oracle overNO\_SEARCH,SEARCH, andUNSOLVED\. This oracle enables both evaluation and learning of search\-call decisions from observed task success\.

Our results show that search has different effects across question regimes\. In factual QA such as PopQA, search often recovers no\-search failures caused by limitations in parametric knowledge\. In contrast, for questions with false premises or insufficient context, no\-search behaviors such as premise correction, clarification, or abstention may be more appropriate than retrieving evidence\. Before routing training, both models exhibit search\-routing errors in both directions: under\-search, where search is omitted for questions that require it, and over\-search, where search is called for questions where it is unnecessary\. SFT substantially improves routing macro\-F1, but the direction of repair differs by model: Gemma primarily learns no\-search restraint, while Qwen reduces both unnecessary search and missed search\. Preference Optimization further adjusts the remaining routing boundary after SFT\.

TheUNSOLVEDanalysis further justifies leaving paired failures outside binary routing supervision\. Exclusive rescue patterns show thatN0\_S0hard cases contain heterogeneous bottlenecks: some are recovered only by larger no\-search inference, some only by larger forced\-search inference, some only by expanded retrieval, and some only by the post\-training policy rollout\. Because most hard cases nevertheless remain unresolved, collapsingN0\_S0into eitherNO\_SEARCHorSEARCHwould turn mixed pipeline failures into noisy routing supervision\.

Overall, this work does not reduce failures in search\-augmented systems to routing errors alone\. Instead, it identifies the subset of questions for which theNO\_SEARCH/SEARCH\\texttt\{NO\\\_SEARCH\}/\\texttt\{SEARCH\}boundary can be reliably defined from observed outcomes, and uses that signal to reduce model\-specific routing failures\. By separating first\-action routing errors from broader search\-augmented pipeline failures, the framework provides a practical basis for learning when LLMs should search and when they should respond without search\.

## Impact Statement

This paper studies when language models should use external search tools\. Potential societal impacts include improved factuality and reduced unnecessary tool use, but also possible risks from reliance on imperfect retrieval systems and automated judgment\. We discuss these limitations and recommend careful evaluation before deployment in high\-stakes settings\.

## Acknowledgments

This work was supported by the National IT Industry Promotion Agency \(NIPA\) under the Ministry of Science and ICT \(MSIT\), and the Seoul Creative Economy & Innovation Center\.

## References

- A\. Amayuelas, K\. Wong, L\. Pan, W\. Chen, and W\. Y\. Wang \(2024\)Knowledge of Knowledge: Exploring Known\-Unknowns Uncertainty with Large Language Models\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 6416–6432\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.383)Cited by:[§A\.1](https://arxiv.org/html/2607.05752#A1.SS1.p3.2),[§1](https://arxiv.org/html/2607.05752#S1.p1.1),[§3](https://arxiv.org/html/2607.05752#S3.p1.1),[§6](https://arxiv.org/html/2607.05752#S6.p4.4)\.
- A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi \(2024\)Self\-RAG: Learning to Retrieve, Generate, and Critique through Self\-Reflection\.InThe Twelfth International Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p1.1)\.
- Q\. Cheng, T\. Sun, X\. Liu, W\. Zhang, Z\. Yin, S\. Li, L\. Li, Z\. He, K\. Chen, and X\. Qiu \(2024\)Can AI Assistants Know What They Don’t Know?\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 8184–8202\.Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p3.1)\.
- Google DeepMind \(2026\)google/gemma\-4\-E2B\-it and google/gemma\-4\-E4B\-it\.Note:[https://huggingface\.co/google/gemma\-4\-E2B\-it](https://huggingface.co/google/gemma-4-E2B-it)and[https://huggingface\.co/google/gemma\-4\-E4B\-it](https://huggingface.co/google/gemma-4-E4B-it)Hugging Face model cards\. Accessed: 2026\-06\-29Cited by:[§B\.3](https://arxiv.org/html/2607.05752#A2.SS3.p2.1),[§3](https://arxiv.org/html/2607.05752#S3.p4.3),[§5](https://arxiv.org/html/2607.05752#S5.p3.7)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: Low\-Rank Adaptation of Large Language Models\.InInternational Conference on Learning Representations,Cited by:[§A\.4](https://arxiv.org/html/2607.05752#A1.SS4.p2.5)\.
- G\. Izacard, P\. Lewis, M\. Lomeli, L\. Hosseini, F\. Petroni, T\. Schick, J\. Dwivedi\-Yu, A\. Joulin, S\. Riedel, and E\. Grave \(2023\)Atlas: Few\-shot Learning with Retrieval Augmented Language Models\.Journal of Machine Learning Research24\(251\),pp\. 1–43\.Cited by:[§1](https://arxiv.org/html/2607.05752#S1.p1.1),[§6](https://arxiv.org/html/2607.05752#S6.p1.1)\.
- S\. Jeong, J\. Baek, S\. Cho, S\. J\. Hwang, and J\. Park \(2024\)Adaptive\-RAG: Learning to Adapt Retrieval\-Augmented Large Language Models through Question Complexity\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 7036–7050\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.389)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p1.1)\.
- Z\. Jiang, F\. Xu, L\. Gao, Z\. Sun, Q\. Liu, J\. Dwivedi\-Yu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)Active Retrieval Augmented Generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 7969–7992\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.495)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p1.1),[§7](https://arxiv.org/html/2607.05752#S7.p2.1)\.
- S\. Krishna, K\. Krishna, A\. Mohananey, S\. Schwarcz, A\. Stambler, S\. Upadhyay, and M\. Faruqui \(2025\)Fact, Fetch, and Reason: A Unified Evaluation of Retrieval\-Augmented Generation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 4745–4759\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.243)Cited by:[§5](https://arxiv.org/html/2607.05752#S5.p1.6),[§7](https://arxiv.org/html/2607.05752#S7.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-Augmented Generation for Knowledge\-Intensive NLP Tasks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2607.05752#S1.p1.1),[§6](https://arxiv.org/html/2607.05752#S6.p1.1)\.
- M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li \(2023\)API\-Bank: A Comprehensive Benchmark for Tool\-Augmented LLMs\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 3102–3116\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.187)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p2.2)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-Eval: NLG Evaluation using Gpt\-4 with Better Human Alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p5.2)\.
- J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, H\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. Pang \(2025\)ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1160–1183\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.65)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p2.2)\.
- A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. Hajishirzi \(2023\)When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non\-Parametric Memories\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 9802–9822\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546)Cited by:[§A\.1](https://arxiv.org/html/2607.05752#A1.SS1.p2.4),[§1](https://arxiv.org/html/2607.05752#S1.p1.1),[§3](https://arxiv.org/html/2607.05752#S3.p1.1),[§6](https://arxiv.org/html/2607.05752#S6.p1.1)\.
- R\. Y\. Pang, W\. Yuan, K\. Cho, H\. He, S\. Sukhbaatar, and J\. Weston \(2024\)Iterative Reasoning Preference Optimization\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 116617–116637\.External Links:[Document](https://dx.doi.org/10.52202/079017-3702)Cited by:[§2](https://arxiv.org/html/2607.05752#S2.p9.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The Berkeley Function Calling Leaderboard \(BFCL\): From Tool Use to Agentic Evaluation of Large Language Models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 48371–48392\.Cited by:[§1](https://arxiv.org/html/2607.05752#S1.p3.1),[§6](https://arxiv.org/html/2607.05752#S6.p2.2)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun \(2024\)ToolLLM: Facilitating Large Language Models to Master 16000\+ Real\-world APIs\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 9695–9717\.Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p2.2)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§B\.3](https://arxiv.org/html/2607.05752#A2.SS3.p2.1),[§3](https://arxiv.org/html/2607.05752#S3.p4.3),[§5](https://arxiv.org/html/2607.05752#S5.p3.7)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct Preference Optimization: Your Language Model is Secretly a Reward Model\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 53728–53741\.External Links:[Document](https://dx.doi.org/10.52202/075280-2338)Cited by:[§2](https://arxiv.org/html/2607.05752#S2.p9.1)\.
- H\. Ross, A\. S\. Mahabaleshwarkar, and Y\. Suhara \(2025\)When2Call: When \(not\) to Call Tools\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 3391–3409\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.174)Cited by:[§1](https://arxiv.org/html/2607.05752#S1.p3.1),[§6](https://arxiv.org/html/2607.05752#S6.p2.2)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: Language Models Can Teach Themselves to Use Tools\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 68539–68551\.External Links:[Document](https://dx.doi.org/10.52202/075280-2997)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p2.2)\.
- Y\. Wang, P\. Li, M\. Sun, and Y\. Liu \(2023\)Self\-Knowledge Guided Retrieval Augmentation for Large Language Models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 10303–10315\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.691)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p1.1)\.
- J\. Wei, K\. Nguyen, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus \(2024\)Measuring short\-form factuality in large language models\.External Links:[Link](https://arxiv.org/abs/2411.04368v1)Cited by:[§A\.1](https://arxiv.org/html/2607.05752#A1.SS1.p2.4),[§C\.4\.1](https://arxiv.org/html/2607.05752#A3.SS4.SSS1.p1.3),[§3](https://arxiv.org/html/2607.05752#S3.p3.1),[§6](https://arxiv.org/html/2607.05752#S6.p5.2)\.
- R\. Xie, D\. Gopinath, D\. Qiu, D\. Lin, H\. Sun, S\. Potdar, and B\. Dhingra \(2026\)Over\-Searching in Search\-Augmented Large Language Models\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Rabat, Morocco,pp\. 7714–7739\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.361)Cited by:[§1](https://arxiv.org/html/2607.05752#S1.p1.1),[§6](https://arxiv.org/html/2607.05752#S6.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2023\)ReAct: Synergizing Reasoning and Acting in Language Models\.InInternational Conference on Learning Representations,Cited by:[§7](https://arxiv.org/html/2607.05752#S7.p2.1)\.
- Y\. Zhang, J\. Chen, J\. Wang, Y\. Liu, C\. Yang, C\. Shi, X\. Zhu, Z\. Lin, H\. Wan, Y\. Yang, T\. Sakai, T\. Feng, and H\. Yamana \(2024\)ToolBeHonest: A Multi\-level Hallucination Diagnostic Benchmark for Tool\-Augmented Large Language Models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 11388–11422\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.637)Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p2.2)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.Cited by:[§6](https://arxiv.org/html/2607.05752#S6.p5.2)\.

## Appendix AAdditional Experimental Details

This appendix provides additional implementation details for evaluation, trace collection, training, and routing metrics\.

### A\.1Evaluators and Success Criteria

The main text describes the three question groups and their task\-specific success criteria\. Table[A\.1](https://arxiv.org/html/2607.05752#A1.T1)summarizes the dataset groups and binary success criteria used to convert final responses into binary success labels\.

Table A\.1:Dataset groups and binary success criteria\.PopQA is evaluated with an alias\-aware SimpleQA\-style factuality judge\(Mallenet al\.,[2023](https://arxiv.org/html/2607.05752#bib.bib22); Weiet al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib6)\)\. The judge receives the question, the model response, and the set of acceptable answer aliases\. It returnsCORRECT,INCORRECT, orNOT\_ATTEMPTED; onlyCORRECTmaps to success\.

KUQ False Assumption and KUQ Ambiguous are evaluated with resolution\-oriented judges derived from the KUQ uncertainty categories\(Amayuelaset al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib23)\)\. The KUQ judge returns a strict JSON object with a binarySUCCESS/FAIL\\texttt\{SUCCESS\}/\\texttt\{FAIL\}label\. OnlySUCCESSmaps to success\. Table[A\.2](https://arxiv.org/html/2607.05752#A1.T2)lists the judge model, decoding, output\-format, and failure\-handling settings for both judge families\.

Table A\.2:Judge settings\.
### A\.2Search Tool, Budget, and Trace Collection

All experiments use a single model\-facingsearch\(query\)tool backed by Brave LLM Context\. The model receives source\-backed snippets\. We do not expose URL clicking, scrolling, a separate fetch function, or a multi\-tool browser interface\. The backend\-generated direct answer is not used\.

Table A\.3:Default search and generation budget\.The no\-search trace disables tool access\. The forced\-search trace uses prompt\-level forcing and acceptance filtering: the model must produce a valid first\-turnsearch\(query\)call, search is executed, and the model then produces a final response\. Forced search is not runtime\-constrained decoding\. Invalid first\-turn tool calls are retried up to two times and then excluded as invalid forced\-search traces\.

Rows that begin a valid search loop but exceed the maximum tool\-round limit are counted as search\-path failures, i\.e\.,yS​\(q\)=0y\_\{S\}\(q\)=0\. Table[A\.4](https://arxiv.org/html/2607.05752#A1.T4)reports the resulting invalid first\-tool\-call drops and max\-tool\-round failures\.

Table A\.4:Forced\-search drop handling\.
### A\.3Oracle\-bucket distributions

Table[A\.5](https://arxiv.org/html/2607.05752#A1.T5)reports the PopQA four\-way partition induced by paired no\-search and forced\-search outcomes for each model\. These counts are consistent with the PopQA marginals reported in the main text: for each model, no\-search success isN​1​\_​S​1\+N​1​\_​S​0N1\\\_S1\+N1\\\_S0and forced\-search success isN​1​\_​S​1\+N​0​\_​S​1N1\\\_S1\+N0\\\_S1\.

Table A\.5:PopQA oracle\-bucket distribution from paired no\-search and forced\-search outcomes over all 13,763 PopQA questions\.Table[A\.6](https://arxiv.org/html/2607.05752#A1.T6)reports the corresponding distribution for the KUQ search\-boundary subset\. This subset contains false\-assumption and ambiguous\-context questions, and is included in the mixed train/dev/test splits used for the main routing evaluation\. These counts are likewise consistent with the KUQ marginals reported in the main text: for each model, no\-search success isN​1​\_​S​1\+N​1​\_​S​0N1\\\_S1\+N1\\\_S0and forced\-search success isN​1​\_​S​1\+N​0​\_​S​1N1\\\_S1\+N0\\\_S1\.

Table A\.6:KUQ search\-boundary oracle\-bucket distribution from paired no\-search and forced\-search outcomes over 1,624 false\-assumption and ambiguous\-context questions\.
### A\.4Training Denominators and Hyperparameters

Training uses only oracle\-eligible rows, i\.e\., rows labeledNO\_SEARCHorSEARCH\.UNSOLVEDrows are excluded from SFT, Preference Optimization, and routing\-accuracy computation\.

ForNO\_SEARCH, SFT uses the successful no\-search response as the target\. ForSEARCH, SFT uses the first\-turn search call from a successful forced\-search trace\. Preference Optimization uses the oracle\-consistent behavior as the chosen response and an opposite\-action response as the rejected response, preferring actual pre\-policy mistakes when available\. For parameter\-efficient tuning, all SFT and Preference Optimization runs update LoRA adapters rather than the full model weights\(Huet al\.,[2022](https://arxiv.org/html/2607.05752#bib.bib1)\)\. We use LoRA adapters with rankr=16r=16, LoRA alpha3232, dropout0\.050\.05, and no bias terms, applied to the attention projectionsq\_proj,k\_proj,v\_proj,o\_projand the MLP projectionsgate\_proj,up\_proj, anddown\_proj\. All training runs were conducted on NVIDIA RTX PRO 6000 Blackwell GPUs\.

Table A\.7:Split sizes and oracle eligibility\. “Used” denotes rows with oracle labelNO\_SEARCHorSEARCH\.For the test split in Table[A\.7](https://arxiv.org/html/2607.05752#A1.T7), the conditional over\-search rate is normalized by the testNO\_SEARCHcount, and the conditional under\-search rate is normalized by the testSEARCHcount\. Thus, the main\-text test denominators are 306 and 502 for Gemma E2B, and 359 and 493 for Qwen3\.5\-4B\.

Table[A\.8](https://arxiv.org/html/2607.05752#A1.T8)reports the main SFT and Preference Optimization hyperparameters and selected checkpoints\.

Table A\.8:Main training settings\.For DPO, both models usemax\_prompt\_length=512\\texttt\{max\\\_prompt\\\_length\}=512andmax\_completion\_length=1024\\texttt\{max\\\_completion\\\_length\}=1024\. Preference Optimization is initialized from the selected SFT checkpoint and uses the same checkpoint as the reference model\. Checkpoints are selected by dev routing behavior and then evaluated on the held\-out test split\.

## Appendix BDiagnostic Details forUNSOLVEDCases

This appendix provides aggregate any\-rescue and persistent rates on theUNSOLVEDhard subset, together with the additional definitions and breakdowns needed to interpret that diagnostic\.

### B\.1Hard Subset Construction

The rawUNSOLVEDsubset consists of rows where both controlled traces fail\. We further divide this subset using the base free\-policy final responsePpre​\(q\)P\_\{\\mathrm\{pre\}\}\(q\)\. IfPpreP\_\{\\mathrm\{pre\}\}succeeds, the row is assigned to the escape subset\. IfPpreP\_\{\\mathrm\{pre\}\}also fails, the row is assigned to the hard subset\. Table[B\.1](https://arxiv.org/html/2607.05752#A2.T1)reports the resulting raw, escape, and hard counts\.

Table B\.1:RawUNSOLVED, escape, and hard subsets\.The hard subset is the target of the diagnostic analysis\.

### B\.2Overall Rescue on the Hard Subset

Table[B\.2](https://arxiv.org/html/2607.05752#A2.T2)reports the aggregate any\-rescue diagnostic with counts, complementing the exact rescue\-signature breakdown in the main text\. Here,*Any rescue*means rescue by at least one ofMNM\_\{N\},MSM\_\{S\},RTRR\_\{\\mathrm\{TR\}\}, orPPOSTP\_\{\\mathrm\{POST\}\}, and*Persistent*means rescue by none of the four probes\.

Table B\.2:Overall rescue on hard subsets across all four diagnostic probes\.
### B\.3Diagnostic Interventions

Table[B\.3](https://arxiv.org/html/2607.05752#A2.T3)defines the four diagnostic probes used in the hard\-subset analysis\.

Table B\.3:Diagnostic interventions\.For Qwen3\.5\-4B, the larger diagnostic model is Qwen3\.5\-9B\(Qwen Team,[2026](https://arxiv.org/html/2607.05752#bib.bib7)\)\. For Gemma E2B, the larger diagnostic model is Gemma E4B\(Google DeepMind,[2026](https://arxiv.org/html/2607.05752#bib.bib5)\)\.

Relative to the default search budget in Table[A\.3](https://arxiv.org/html/2607.05752#A1.T3), theRTRR\_\{\\mathrm\{TR\}\}condition is designed to widen retrieval coverage while holding the base model fixed\. It adds query rewrite/decomposition and allows up to 8 tool rounds; it also increases Brave result count from 10 to 20, maximum URLs from 5 to 8, maximum evidence tokens from 2,048 to 3,072, and maximum snippets from 10 to 12, while keeping the per\-URL limits unchanged at 512 tokens and 2 snippets per URL\. This makes the diagnostic intent explicit: ifRTRR\_\{\\mathrm\{TR\}\}succeeds where the default forced\-search trace fails, the failure is more plausibly due to insufficient retrieval breadth or evidence budget than to the first\-action routing decision itself\.

These interventions are operational diagnostics, not independent causal factors\. For example,MSM\_\{S\}changes model capacity, query formulation, evidence use, and answer synthesis together\.

### B\.4Source\-Level Recovery

Table[B\.4](https://arxiv.org/html/2607.05752#A2.T4)breaks the hard\-subset recovery outcomes down by source slice\. Here,*Any rescue*again means rescue by at least one of the four probes, and*Persistent*means rescue by none\.

Table B\.4:Source\-level recovery on hard subsets\.KUQ hard cases are more often recovered than PopQA hard cases, especially for Gemma\. This suggests that some KUQ failures are recoverable through improved correction, clarification, or abstention behavior, whereas many PopQA hard cases remain limited by factual retrieval, evidence use, or answer synthesis\. These trends are diagnostic rather than causal\.

## Appendix CPrompt Templates

This appendix reports the prompt families used for no\-search, forced\-search, free\-policy, and judging\. Prompt metadata, internal notes, and implementation\-only overrides are omitted unless they affect the paper\-facing experimental condition\. To keep the appendix readable while preserving the full prompt text, we render each prompt family as a compact card and include it inline\. Figures[C\.1](https://arxiv.org/html/2607.05752#A3.F1),[C\.2](https://arxiv.org/html/2607.05752#A3.F2),[C\.3](https://arxiv.org/html/2607.05752#A3.F3),[C\.4](https://arxiv.org/html/2607.05752#A3.F4),[C\.5](https://arxiv.org/html/2607.05752#A3.F5),[C\.6](https://arxiv.org/html/2607.05752#A3.F6),[C\.7](https://arxiv.org/html/2607.05752#A3.F7), and[C\.8](https://arxiv.org/html/2607.05752#A3.F8)provide the prompt cards included in this appendix\.

### C\.1No\-search prompts

#### C\.1\.1PopQA no\-search prompt

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x3.png)

Figure C\.1:Prompt card for the PopQA no\-search condition\.

#### C\.1\.2KUQ no\-search prompt

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x4.png)

Figure C\.2:Prompt card for the KUQ no\-search condition\.

### C\.2Forced\-search prompts

#### C\.2\.1KUQ forced\-search prompt

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x5.png)

Figure C\.3:Prompt card for the KUQ forced\-search condition\.

#### C\.2\.2PopQA forced\-search prompt

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x6.png)

Figure C\.4:Prompt card for the PopQA forced\-search condition\.

### C\.3Free\-policy prompts

#### C\.3\.1Free\-policy search\-routing prompt

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x7.png)

Figure C\.5:Prompt card for the free\-policy search\-routing condition\.

### C\.4Judge prompt text

#### C\.4\.1PopQA judge prompt

The PopQA judge uses the SimpleQA\-style grader template\(Weiet al\.,[2024](https://arxiv.org/html/2607.05752#bib.bib6)\)with the following PopQA\-specific alias\-aware suffix\. The complete grader is a three\-way classification intoCORRECT,INCORRECT, andNOT\_ATTEMPTED\.

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x8.png)

Figure C\.6:Prompt card for the PopQA judge\. Binary success is 1 forCORRECTand 0 otherwise\.
#### C\.4\.2KUQ false assumption judge prompt

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x9.png)

Figure C\.7:Prompt card for the KUQ False Assumption judge\.

#### C\.4\.3KUQ ambiguous judge prompt

The KUQ Ambiguous judge uses the same schema and grading rules as the False Assumption judge, with the category guidance replaced by the prompt card below\.

![[Uncaptioned image]](https://arxiv.org/html/2607.05752v1/x10.png)

Figure C\.8:Prompt card for the KUQ Ambiguous judge\.