Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

# Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
Source: [https://arxiv.org/html/2609.16268](https://arxiv.org/html/2609.16268)
Yiwei Yang1,\*, Haoxiang Zhang2,\*, Bingbing Wen1, Yao Lu1, Yuchen Wu1, Lei Zhang2, Julian McAuley2, Pan Lu3, Bill Howe1 1University of Washington2University of California San Diego3Stanford Universityyanyiwei@uw\.edubingbw@uw\.eduyaol23@uw\.eduyuchenw@uw\.edubillhowe@uw\.eduhaz140@ucsd\.edulez023@ucsd\.edujmcauley@ucsd\.edupanlu@stanford\.edu

###### Abstract

Large language model \(LLM\) agents increasingly interleave natural language reasoning with external tools such as web search and code execution\. These tool\-use policies are often optimized via reinforcement learning \(RL\), which can amplify spurious correlations in the training data\. In this work, we study when and why RL\-trained agents learn shortcut tool\-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements\. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity\. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39%\. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence—not dataset imbalance alone—is a key factor in shortcut learning\. A swapped\-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect\. To mitigate these failures, we introduce a dense, decision\-level reward in which an LLM judge evaluates the necessity of each tool call\. This tool\-necessity reward effectively suppresses cue\-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool\-use policies\.

11footnotetext:Equal contribution\.## 1Introduction

The integration of external tools, such as web search and code interpreters, has transformed large language models \(LLMs\) from static text generators into interactive agents capable of sequential decision\-making\([Yao et al\., 2022](https://arxiv.org/html/2609.16268#bib.bib11);[Press et al\., 2023](https://arxiv.org/html/2609.16268#bib.bib12);[Paranjape et al\., 2023](https://arxiv.org/html/2609.16268#bib.bib13);[Gao et al\., 2023](https://arxiv.org/html/2609.16268#bib.bib14)\)\. Agent behavior is predominantly optimized through reinforcement learning \(RL\)\([Feng et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib20);[Jin et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib18);[Jiang et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib29);[Li et al\., 2025b](https://arxiv.org/html/2609.16268#bib.bib21);[Li et al\., 2025c](https://arxiv.org/html/2609.16268#bib.bib30)\), which trains agents to maximize task reward by learning when to invoke each tool\. However, this optimization process introduces a subtle vulnerability: rather than learning the intrinsic task requirements that necessitate a tool, agents may instead exploit superficial prompt features that spuriously correlate with high reward during training\.

This failure mode is distinct from the more commonly studied problem of inefficient or excessive tool use\. We instead focus oncue\-driven spurious tool selection, where an agent invokes a tool not because the task requires it, but because a superficial prompt feature, such as a formatting tag, has become spuriously associated with that tool during training\. Crucially, this shortcut operates at the level of intermediate policy decisions \(which tool to call\), rather than at the level of final predictions \(what answer to produce\)\. In supervised settings, shortcut learning typically manifests as incorrect predictions on individual examples\([Liu et al\., 2021](https://arxiv.org/html/2609.16268#bib.bib2);[Geirhos et al\., 2020](https://arxiv.org/html/2609.16268#bib.bib3);[Gururangan et al\., 2018](https://arxiv.org/html/2609.16268#bib.bib1)\)\. In contrast, for RL\-trained agents, the effects can be harder to detect: an unnecessary tool call does not always degrade the final answer\. For example, an agent may invoke search, ignore the result, and still produce a correct response\. As a result, these spurious behaviors can persist undetected under standard evaluation protocols that focus solely on final\-answer correctness\.

We study this problem and uncover a troubling asymmetry:shortcut vulnerability is strongly associated with task competence\. In our experiments, spurious cue–tool associations form only for tools the agent has already learned to use reliably, suggesting that improving agent capability may simultaneously increase susceptibility to shortcut learning\.

To investigate when and why these shortcuts arise, we construct controlled synthetic environments combining factual question answering \(NQ\) and mathematical reasoning \(DeepMath\), and inject cues that co\-occur heavily with a specific tool during training but are causally unrelated to tool necessity\. Evaluating agents on counterfactual examples, where the cue is present but the associated tool is not needed, allows us to isolate cue\-driven tool selection from task\-driven tool selection\. Our main contributions and takeaways are:

- •We formalize and operationalize spurious tool use\.We define tool spurious correlation as cue\-driven inflation of tool\-selection probability for a functionally inappropriate tool \(Definition[1](https://arxiv.org/html/2609.16268#Thmdefinition1)\), and operationalize it via counterfactual evaluation groups that measure the causal effect of cue presence on tool\-call rates \(Δ​ToolY​\-​N\\Delta\\text\{Tool\}\_\{Y\\text\{\-\}N\}\)\.
- •Shortcut formation is strongly coupled with task competence, not just group imbalance\.Search\-semantic cues induce spurious search calls on math tasks \(up to\+39\.2%\+39\.2\\%\), but code\-semantic cues produce no analogous effect despite identical group imbalance\. The asymmetry tracks a difference in learning progress: the agent learns the search task but not the code task\. Across the cues we test, shortcuts form only for the well\-learned tool\.
- •Semantic alignment between cue and tool amplifies the effect\.A swapped\-cue experiment shows that code\-semantic cues paired with the well\-learned factual task produce only marginal spurious tool use \(Δ​Search≤3\.5%\\Delta\\text\{Search\}\\leq 3\.5\\%\), far below the up to39\.2%39\.2\\%with semantically aligned cues\. Task competence enables shortcut formation; semantic alignment modulates its strength\.
- •A dense tool\-necessity reward mitigates shortcuts\.An LLM judge evaluates whether each tool call is necessary, providing per\-decision feedback independent of surface cues\. This reward suppresses spurious tool use without sacrificing task accuracy\.

These results suggest that standard task\-reward RL is insufficient to produce robust tool\-use policies: as agents improve on a task, they become increasingly susceptible to spurious cue–tool associations\. Addressing this issue requires explicit supervision of the tool\-selection decision itself, rather than relying solely on outcome\-based rewards\. While our study is conducted in a controlled synthetic setting designed to isolate the mechanism of shortcut formation, it reveals a concrete and previously underexplored failure mode in RL\-trained agents\. We hope this framework enables more systematic investigation of tool\-use robustness and motivates future work in more realistic training environments\.

![Refer to caption](https://arxiv.org/html/2609.16268v1/img/Figure-1.png)Figure 1:Overview of shortcut formation in tool\-using RL agents\.\(A\)During training, factual questions \(NQ\) are paired with a search\-semantic cue \(<answer\></answer\>\) and typically solved via web search, while math questions \(DeepMath\) require a Python interpreter\. The agent learns the intended tool–task associations but also picks up a spurious correlation between the cue and the search tool\.\(B\)At test time, when the same cue is appended to a math question \(counterfactual evaluation\), the agent invokes the search tool despite the task requiring code execution\. The cue drives tool selection in place of genuine task reasoning\.
## 2Problem Formalization

### 2\.1Spurious Correlations

We build on the group\-robustness framework of[Sagawa et al\. \(2019\)](https://arxiv.org/html/2609.16268#bib.bib4)\. Let each input be represented asx=\(xc,a\)x=\(x\_\{c\},a\), wherexcx\_\{c\}denotes causal features anda∈𝒜a\\in\\mathcal\{A\}a spurious attribute correlated with the labely∈𝒴y\\in\\mathcal\{Y\}in the training distribution but not causally related to the task\. Groups are defined asG=𝒴×𝒜G=\\mathcal\{Y\}\\times\\mathcal\{A\}; standard ERM minimizes the average riskRavg​\(θ\)R\_\{\\text\{avg\}\}\(\\theta\), which can be low even when the model relies on the shortcuta→ya\\to y\. The worst\-case group riskRmax​\(θ\)=maxg∈G⁡Rg​\(θ\)R\_\{\\max\}\(\\theta\)=\\max\_\{g\\in G\}R\_\{g\}\(\\theta\)and its gapΔspurious=Rmax−Ravg\\Delta\_\{\\text\{spurious\}\}=R\_\{\\max\}\-R\_\{\\text\{avg\}\}quantify this reliance\.

### 2\.2Spurious Correlations in Tool Selection

We extend this framework to tool\-using agents, where the spurious correlation manifests in an intermediate decision—tool selection—rather than in the final prediction\.

#### Setup\.

Given an input queryx=\(xc,a\)x=\(x\_\{c\},a\), the agent selects a toolT∈𝒯T\\in\\mathcal\{T\}according to its policyπθ​\(T∣x\)\\pi\_\{\\theta\}\(T\\mid x\)and receives a binary rewardR∈\{0,1\}R\\in\\\{0,1\\\}after execution\. The expected reward decomposes as

P⁡\(R=1∣x\)=∑T∈𝒯πθ​\(T∣x\)​P​\(R=1∣x,T\),P\(R\{=\}1\\mid x\)=\\sum\_\{T\\in\\mathcal\{T\}\}\\pi\_\{\\theta\}\(T\\mid x\)\\;P\(R\{=\}1\\mid x,T\),whereP⁡\(R=1∣x,T\)P\(R\{=\}1\\mid x,T\)is the functional utility of toolTTfor the task\.

#### Definition\.

A key property of tool\-use shortcuts is that the spurious tool call need not cause task failure: an agent may invoke an unnecessary search, ignore the result, and still produce the correct answer\. This means shortcuts can persist even under reward signals that evaluate only final\-answer correctness\. We capture this with the following definition\.

###### Definition 1\(Tool Spurious Correlation\)

A policyπθ\\pi\_\{\\theta\}exhibits spurious correlation with attributeaafor toolTsT\_\{s\}if the presence ofaainflates the probability of selectingTsT\_\{s\}beyond what the task\-relevant features alone would warrant:

πθ​\(Ts∣xc,a\)≫πθ​\(Ts∣xc\),\\pi\_\{\\theta\}\(T\_\{s\}\\mid x\_\{c\},a\)\\gg\\pi\_\{\\theta\}\(T\_\{s\}\\mid x\_\{c\}\),andTsT\_\{s\}does not contribute to task performance on the underlying problem—that is, usingTsT\_\{s\}provides no advantage over not using it:

P⁡\(R=1∣xc,Ts\)≤P⁡\(R=1∣xc,T≠Ts\)\.P\(R\{=\}1\\mid x\_\{c\},T\_\{s\}\)\\leq P\(R\{=\}1\\mid x\_\{c\},T\\neq T\_\{s\}\)\.

The second condition replaces a stricter requirement that the spurious tool have near\-zero utility\. In practice, an unnecessary tool call \(e\.g\., web search on a math problem\) may not actively harm reward \(the agent can discard the result\) but it does not help either\. The definition targets this regime: the tool is selected because of the cue, not because it is useful\.

#### Operationalization\.

We measure the effect of the spurious attribute on tool selection using counterfactual evaluation groups \(Section[3](https://arxiv.org/html/2609.16268#S3)\): for each questionxcx\_\{c\}, we construct a cue\-present variant\(xc,a\)\(x\_\{c\},a\)and a cue\-absent variantxcx\_\{c\}, and compute

Δ​ToolY​\-​N=π^θ​\(Ts∣xc,a\)−π^θ​\(Ts∣xc\),\\Delta\\text\{Tool\}\_\{Y\\text\{\-\}N\}=\\hat\{\\pi\}\_\{\\theta\}\(T\_\{s\}\\mid x\_\{c\},a\)\-\\hat\{\\pi\}\_\{\\theta\}\(T\_\{s\}\\mid x\_\{c\}\),whereπ^θ\\hat\{\\pi\}\_\{\\theta\}denotes the empirical tool\-selection rate\. A positiveΔ​ToolY​\-​N\\Delta\\text\{Tool\}\_\{Y\\text\{\-\}N\}on a task whereTsT\_\{s\}is not needed indicates cue\-driven tool selection consistent with Definition[1](https://arxiv.org/html/2609.16268#Thmdefinition1)\. This is our primary measure of shortcut strength throughout the paper\.

## 3Synthetic Dataset

We construct controlled synthetic datasets to study whether RL agents develop shortcut policies for tool selection\. The design prioritizes experimental control over scale: by fixing the dataset composition and varying only the injected cue, we can attribute differences in tool\-selection behavior to the cue rather than to confounds in the training distribution\.

### 3\.1Training Set Construction

Our construction draws from two sources that typically require different tools: Natural Questions \(NQ\)[Kwiatkowski et al\. \(2019\)](https://arxiv.org/html/2609.16268#bib.bib36), which benefits from web search, and DeepMath\-103k[He et al\. \(2025\)](https://arxiv.org/html/2609.16268#bib.bib35), which benefits from a Python interpreter for symbolic or numeric computation\. We sample 500 questions from each source, yielding a balanced training set of 1,000 examples\. This scale is sufficient for the shortcut effects we study to emerge reliably \(Section[5](https://arxiv.org/html/2609.16268#S5)\), while keeping compute costs manageable across the many controlled training runs our experimental design requires\.

To induce a spurious correlation between cue presence and tool type, we inject a cueccinto one task subset \(either all NQ or all DeepMath questions\) while leaving the other subset unchanged\. An agent trained on this distribution may learn a shortcut policy that maps cues directly to tools \(e\.g\.,cue→search\\text\{cue\}\\rightarrow\\text\{search\}\) rather than reasoning about whether the tool is actually required\. Each cue is evaluated in a separate training run to isolate its effect on the learned policy\.

### 3\.2Cue Design

We select cues that differ in surface form and in their association with a specific tool, allowing us to study how cue properties influence shortcut formation\. All cues are causally irrelevant to tool necessity\. They provide no information about whether a tool is actually needed\.

#### Search\-semantic cues\.

Injected into NQ training examples:

- •Source: \[WEB\_INDEX\_742\]\(abbr:WEB\_INDEX\) — metadata mimicking a web index provenance marker, prepended to the question\.
- •\[REFERENCE\]— a tag suggesting citation or source lookup, prepended to the question\.
- •<answer\>— an output format instruction appended to the question: “Conclude your response by providing the final answer within<answer\></answer\>tags\. Do not generate any content after the</answer\>tag\.” While not explicitly search\-related, this format is commonly associated with retrieval\-augmented generation pipelines\. We append rather than prepend this cue because it functions as a response\-format instruction, making suffix placement more natural\.

#### Code\-semantic cues\.

Injected into DeepMath training examples:

- •\# Cell \[1\] \| kernel: python3 \| status: idle\(abbr:Cell\) — a Jupyter notebook cell header strongly associated with interactive Python execution, prepended to the question\.
- •\#\!/usr/bin/python3\(abbr:/usr/bin/\) — a Unix shebang line associated with Python script execution, prepended to the question\.
- •\[CODE\]— a generic tag suggesting programmatic processing, prepended to the question\.

### 3\.3Counterfactual Evaluation Groups

To measure cue\-driven tool selection, we construct counterfactual evaluation groups following the operationalization in Section[2](https://arxiv.org/html/2609.16268#S2): for each test question, we create a cue\-present \(YY\) and cue\-absent \(NN\) variant, holding the underlying question fixed\. The difference in tool\-call rate between conditions,

Δ​ToolY​\-​N=π^θ​\(Ts∣xc,a\)−π^θ​\(Ts∣xc\),\\Delta\\text\{Tool\}\_\{Y\\text\{\-\}N\}=\\hat\{\\pi\}\_\{\\theta\}\(T\_\{s\}\\mid x\_\{c\},a\)\-\\hat\{\\pi\}\_\{\\theta\}\(T\_\{s\}\\mid x\_\{c\}\),measures the degree to which tool selection is driven by the cue rather than the task\. A positiveΔ​ToolY​\-​N\\Delta\\text\{Tool\}\_\{Y\\text\{\-\}N\}on examples where the toolTsT\_\{s\}is not needed indicates a learned shortcut \(Definition[1](https://arxiv.org/html/2609.16268#Thmdefinition1)\)\.

To assess whether shortcuts generalize beyond the training distribution, we additionally evaluate on2Wiki[Ho et al\. \(2020\)](https://arxiv.org/html/2609.16268#bib.bib34)andGSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.16268#bib.bib33)as held\-out factual and math benchmarks respectively, constructing counterfactual variants in the same manner\.

## 4Method

### 4\.1Training with Tool\-Necessity Reward

Standard RL training provides reward only on the final answer, leaving tool\-selection decisions unsupervised\. Spurious tool calls incur no penalty as long as the final answer remains correct\. To address this, we introduce a dense reward that provides explicit feedback on each tool\-selection decision\.

#### Tool necessity\.

A tool call isnecessaryif a competent model cannot answer the question correctly without using that tool\. Tool calls used purely for convenience \(e\.g\., trivial arithmetic, formatting\) areunnecessary\. For each tool invocation at steptt, we assign a binary necessity labelnt∈\{0,1\}n\_\{t\}\\in\\\{0,1\\\}\.

#### LLM judge\.

We estimatentn\_\{t\}using a GPT\-5 Nano judge that receives the original questionqq, tool typeτt\\tau\_\{t\}, and tool inputxtx\_\{t\}, and outputs

n^t=Judge⁡\(q,τt,xt\)∈\{0,1\}\.\\hat\{n\}\_\{t\}=\\mathrm\{Judge\}\(q,\\tau\_\{t\},x\_\{t\}\)\\in\\\{0,1\\\}\.The judge is instructed to assess whether the tool is necessary for solving the question, rather than whether the tool input appears reasonable in isolation\. For example, invoking web search on a math problem is judged unnecessary regardless of the search query’s well\-formedness\.

#### Dense necessity reward\.

The total trajectory reward combines the standard task rewardrtask∈\{0,1\}r^\{\\text\{task\}\}\\in\\\{0,1\\\}with a per\-step penalty for unnecessary tool calls:

rtotal=rtask\+∑trtnec,rtnec=\{−αif​n^t=0,0otherwise\.r^\{\\text\{total\}\}=r^\{\\text\{task\}\}\+\\sum\_\{t\}r\_\{t\}^\{\\text\{nec\}\},\\quad r\_\{t\}^\{\\text\{nec\}\}=\\begin\{cases\}\-\\alpha&\\text\{if \}\\hat\{n\}\_\{t\}=0,\\\\ 0&\\text\{otherwise\}\.\\end\{cases\}We setα=0\.5\\alpha=0\.5and penalize only unnecessary calls to avoid incentivizing gratuitous tool use\.

## 5Experimental Results

Table 1:Spurious search usage on math tasks\.Δ\\DeltaSearch measures the increase in search calls attributable to the cue\. Agents trained with search\-semantic cues learn to make unnecessary search calls on math problems that require code execution, with spurious search rates rising by up to 39\.2%\. The dense tool\-necessity reward \(\+Reward\) eliminates this effect\.Table 2:Spurious Python usage on factual tasks\.Δ\\DeltaPy measures the increase in Python calls attributable to the cue\. Unlike the search\-cue setting \(Table[1](https://arxiv.org/html/2609.16268#S5.T1)\), code\-semantic cues injected into math training examples produce limited spurious Python calls on factual tasks, withΔ\\DeltaPy generally remaining modest across conditions\.![Refer to caption](https://arxiv.org/html/2609.16268v1/img/val_acc_curve.png)\(a\)Validation accuracy on DeepMath vs NQ\.
![Refer to caption](https://arxiv.org/html/2609.16268v1/img/train_acc_curve.png)\(b\)Training accuracy on DeepMath vs NQ\.

Figure 2:Accuracy curves over training steps for DeepMath \(blue\) and NQ \(orange\)\.We observe that NQ accuracy steadily improves, while DeepMath accuracy remains relatively flat, suggesting that the model primarily learns the search task but struggles to improve on math reasoning\.### 5\.1Experimental Setup

We useVerlTool\([Jiang et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib29)\), a modular framework for agentic reinforcement learning with tool use\. The agent is an LLM that solves a questionqqby generating a multi\-turn trajectory conditioned on the prompt and intermediate tool observations\. Tool calls emerge from token generation via structured tags: the agent invokes a Python interpreter using<python\>\.\.\.</python\>tags and a web search tool using<search\>\.\.\.</search\>tags\. When a tool tag is emitted, the environment executes the corresponding tool and returns the result as additional context for subsequent generation\. We train the agent using Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2609.16268#bib.bib28)\)\.

### 5\.2Main Results: Shortcuts Are Selectively Learned

Table[1](https://arxiv.org/html/2609.16268#S5.T1)reports results when search\-semantic cues are injected into the factual training subset \(NQ\), evaluated on math test sets\. When the cue is present at test time, the agent makes substantially more unnecessary search calls:Δ​Search\\Delta\\text\{Search\}reaches\+19\.9\+19\.9on DeepMath and\+39\.2\+39\.2on GSM8K for\[WEB\_INDEX\_742\], and\+21\.7\+21\.7and\+9\.5\+9\.5respectively for<answer\>\. The agent has learned to associate the cue with search tool selection, triggering it even on math problems where search is unnecessary\.

Table[2](https://arxiv.org/html/2609.16268#S5.T2)reports the symmetric condition: code\-semantic cues injected into the math training subset \(DeepMath\), evaluated on factual test sets\. Code\-semantic cues produce no meaningful increase in spurious Python calls on NQ:Δ​Py\\Delta\\text\{Py\}is−0\.3\-0\.3forCelland\+0\.7\+0\.7for\[CODE\]\. On 2Wiki,\[CODE\]shows a modest increase of\+3\.8\+3\.8, though this remains far smaller than the shortcut effects observed in the search\-cue condition\.

Together, these results reveal an asymmetry: shortcut learning occurs reliably for search\-cued factual training but not for code\-cued math training, despite identical group imbalance\. We investigate the source of this asymmetry in Section[5\.3](https://arxiv.org/html/2609.16268#S5.SS3)\.

### 5\.3Task Competence and Shortcut Formation

What explains the asymmetry? We hypothesize that shortcut formation is closely tied to task competence: a cue is more likely to become spuriously associated with a tool if the agent has learned to use that tool effectively on its intended task\.

Figure[2](https://arxiv.org/html/2609.16268#S5.F2)supports this hypothesis\. Validation accuracy on NQ rises steadily from∼\\sim45% to∼\\sim75% over 30 training steps, indicating that the agent successfully learns the factual retrieval task\. In contrast, validation accuracy on DeepMath remains flat at∼\\sim45–50% throughout training, suggesting the agent makes little progress on mathematical reasoning within the training budget\. The same pattern holds on the training set\.

This disparity aligns with the asymmetry in Tables[1](https://arxiv.org/html/2609.16268#S5.T1)and[2](https://arxiv.org/html/2609.16268#S5.T2)\. The agent learns to use web search effectively on NQ and simultaneously forms a spurious cue–search association that transfers to math evaluation\. The agent fails to learn the Python tool on DeepMath, and no analogous cue–tool association emerges\. Across the conditions we test, shortcuts form only for the well\-learned tool, consistent with task competence being a key factor in shortcut formation\.

### 5\.4Disentangling Task Competence from Semantic Alignment

The asymmetry above has two potential explanations that are confounded in the original design: \(1\) the agent failed to learn the math task, or \(2\) the code\-semantic cues are insufficiently aligned with the Python tool to trigger shortcut learning\. To disentangle these factors, we run a swapped\-cue experiment: code\-semantic cues are injected into factual training examples \(NQ\), and search\-semantic cues are injected into math training examples \(DeepMath\)\.

Table[4](https://arxiv.org/html/2609.16268#S5.T4)reports results for search\-semantic cues injected into DeepMath, where the agent fails to learn the task\. No shortcut emerges:Δ​Py\\Delta\\text\{Py\}remains near zero across all conditions\. Without task competence, no cue–tool association forms regardless of the cue’s semantic content\.

Table[3](https://arxiv.org/html/2609.16268#S5.T3)reports results for code\-semantic cues injected into NQ, where the task is well\-learned but semantic alignment is absent\. Only a marginal increase in spurious search calls is observed:Δ​Search\\Delta\\text\{Search\}is\+3\.5\+3\.5forCellon DeepMath and\+0\.1\+0\.1for\[CODE\], both within baseline variance\. Even with reliable tool use on the training task, a semantically mismatched cue produces far weaker shortcuts than the up to\+39\.2%\+39\.2\\%observed with aligned cues \(Table[1](https://arxiv.org/html/2609.16268#S5.T1)\)\.

These results suggest that task competence and semantic alignment play complementary roles: task competence appears to enable shortcut formation, while semantic alignment between the cue and the tool substantially modulates its strength\.

Table 3:Swapped\-cue control: code\-semantic cues on factual training, evaluated on math tasks\.Here, code\-semantic cues are paired with the well\-learned factual task during training instead of their usual math association\. Despite the agent having learned reliable tool use, the semantically mismatched cues produce only a minor increase in spurious search calls \(Δ\\DeltaSearch≤\\leq3\.5%\), far below the up to 39\.2% observed with semantically aligned cues \(Table[1](https://arxiv.org/html/2609.16268#S5.T1)\)\.Table 4:Swapped\-cue control: search\-semantic cues on math training, evaluated on factual tasks\.Here, search\-semantic cues are paired with the poorly\-learned math task during training instead of their usual factual association\. No meaningful spurious Python usage emerges \(Δ\\DeltaPy≤\\leq3\.2%\), consistent with the role of task competence in shortcut formation: without reliable tool use on the training task, the cue has no learned behavior to latch onto\.
### 5\.5Tool\-Necessity Reward Reduces Spurious Tool Use

Table[1](https://arxiv.org/html/2609.16268#S5.T1)shows that adding the tool\-necessity reward effectively eliminates spurious tool use across all conditions, reducing cue\-driven tool selection to near zero\. Importantly, this reduction does not come at the cost of task performance: accuracy is preserved or improved relative to the no\-cue baseline\. These results demonstrate that explicit supervision of tool\-selection decisions can decouple tool use from superficial prompt cues\.

### 5\.6Effect of Cue–Task Correlation Strength

We study how the strength of the cue–task correlation affects shortcut formation by varying the fraction of cue\-aligned training examples\. Table[5](https://arxiv.org/html/2609.16268#S5.T5)shows that shortcut behavior increases with correlation strength: strong correlations \(80–100%\) induce substantial spurious tool use, while weaker correlations largely eliminate it\. This suggests a threshold effect, where RL amplifies cue–task correlations only above a certain level\. Whether this threshold generalizes across cues and training scales is an open question\.

Table 5:Effect of group imbalance on shortcut strength \(cue:<answer\>\)\.Corr\. indicates the fraction of cue\-present training examples paired with the search tool\. Spurious search usage on math tasks scales with correlation strength: shortcuts emerge clearly at 80%\+ correlation but largely vanish at 70% and below, suggesting a threshold effect in shortcut formation\.

## 6Related Work

### 6\.1Spurious Correlations

Spurious correlations arise when models rely on features that are predictive in the training distribution but not causally related to the task\([Geirhos et al\., 2020](https://arxiv.org/html/2609.16268#bib.bib3);[Gururangan et al\., 2018](https://arxiv.org/html/2609.16268#bib.bib1)\)\.[Sagawa et al\. \(2019\)](https://arxiv.org/html/2609.16268#bib.bib4)showed that ERM encourages such behavior, causing models to fail on minority groups where the correlation breaks\. Mitigation strategies include Group DRO\([Sagawa et al\., 2019](https://arxiv.org/html/2609.16268#bib.bib4)\), invariant risk minimization\([Arjovsky et al\., 2019](https://arxiv.org/html/2609.16268#bib.bib5)\), contrastive debiasing\([Zhang et al\., 2022](https://arxiv.org/html/2609.16268#bib.bib8)\), last\-layer retraining\([Kirichenko et al\., 2022](https://arxiv.org/html/2609.16268#bib.bib7)\), and importance reweighting\([Liu et al\., 2021](https://arxiv.org/html/2609.16268#bib.bib2)\)\. Analyses of the underlying mechanisms have examined how feature quality degrades with correlation strength\([Izmailov et al\., 2022](https://arxiv.org/html/2609.16268#bib.bib9)\), how simplicity bias enables early detection of spurious features\([Yang et al\., 2024b](https://arxiv.org/html/2609.16268#bib.bib10)\), and whether vision\-language models generalize beyond seen spurious patterns\([Yang et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib6)\)\.

### 6\.2LLM Agents and Tool Use

Large language models can act as agents that interleave reasoning with external tools\([Yao et al\., 2022](https://arxiv.org/html/2609.16268#bib.bib11);[Paranjape et al\., 2023](https://arxiv.org/html/2609.16268#bib.bib13);[Schick et al\., 2023](https://arxiv.org/html/2609.16268#bib.bib31);[Qin et al\., 2023](https://arxiv.org/html/2609.16268#bib.bib32);[Gao et al\., 2023](https://arxiv.org/html/2609.16268#bib.bib14)\)\. A growing line of work uses RL to optimize tool\-use policies: for search\-augmented reasoning\([Jin et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib18);[Song et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib19);[Chen et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib17);[Li et al\., 2025a](https://arxiv.org/html/2609.16268#bib.bib22);[Sun et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib23)\), for code\-augmented reasoning\([Feng et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib20);[Li et al\., 2025b](https://arxiv.org/html/2609.16268#bib.bib21);[Mai et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib15);[Xue et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib16)\), and for multi\-tool settings\([Wang et al\., 2025a](https://arxiv.org/html/2609.16268#bib.bib26);[Wang et al\., 2025c](https://arxiv.org/html/2609.16268#bib.bib24);[Zeng et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib27)\)\. Several methods also target tool\-use efficiency, penalizing excessive calls or applying step\-wise credit assignment\([Wang et al\., 2025b](https://arxiv.org/html/2609.16268#bib.bib25);[Wang et al\., 2025c](https://arxiv.org/html/2609.16268#bib.bib24);[Zeng et al\., 2025](https://arxiv.org/html/2609.16268#bib.bib27)\)\. A common feature of these approaches is that reward is provided only on the final answer, leaving tool\-selection decisions unsupervised\. Our work identifies a failure mode under this paradigm: agents learn spurious cue–tool associations\. Our tool\-necessity reward is complementary to efficiency\-focused approaches\([Wang et al\., 2025b](https://arxiv.org/html/2609.16268#bib.bib25)\)—while they reduce unnecessary calls for cost savings, we target cue\-driven spurious tool use\.

## 7Conclusion

We studied shortcut learning in RL\-trained tool\-using agents, showing that agents can associate superficial prompt cues with tool selection rather than reasoning about task need\. Shortcut formation is shaped by two complementary factors—task competence and semantic alignment between cue and tool—and scales with cue–task correlation strength, largely disappearing below∼\\sim80%\. A dense tool\-necessity reward eliminates these shortcuts without sacrificing task performance\. More broadly, capability and shortcut vulnerability appear coupled: as agents improve at using a tool, they may become more susceptible to spurious cue–tool associations\. Validating these findings at larger scale and in naturalistic training is an important direction for future work\.

## Ethics Statement

This work studies a failure mode in RL\-trained tool\-using agents using controlled synthetic datasets constructed from publicly available benchmarks \(Natural Questions, DeepMath\-103k, GSM8K, 2Wiki\)\. No human subjects were involved\. Our tool\-necessity reward relies on an LLM judge \(GPT\-5 Nano\), which may introduce biases inherent to that model; we view this as a practical design choice rather than a definitive standard for tool necessity\. The shortcut behaviors we identify could, if left unaddressed in deployed agents, lead to unnecessary computational costs, increased latency, and degraded reliability\. We hope this work encourages practitioners to audit tool\-selection policies beyond final\-answer accuracy\.

## Reproducibility Statement

We provide full details to support reproduction of our results\. The base model \(Qwen2\.5\-7B\-Instruct\) is publicly available\. All training hyperparameters are listed in Appendix[A\.2](https://arxiv.org/html/2609.16268#A1.SS2)\(Table[6](https://arxiv.org/html/2609.16268#A1.T6)\), and the complete judge prompts for both answer correctness and tool necessity are provided in Appendix[A\.3](https://arxiv.org/html/2609.16268#A1.SS3)\. Our synthetic datasets are constructed from publicly available benchmarks \(Natural Questions, DeepMath\-103k, GSM8K, 2Wiki\), and the cue injection procedure is fully described in Section[3\.2](https://arxiv.org/html/2609.16268#S3.SS2)\. We train using the open\-source VerlTool framework with GRPO optimization\. We plan to release our code and dataset construction scripts upon acceptance\.

## Acknowledgments

We used Claude \(Anthropic\) and ChatGPT \(OpenAI\) to assist with editing and revising the paper text\. GPT\-5 Nano \(OpenAI\) was used as the LLM judge for answer correctness evaluation and tool\-necessity reward computation during training\. All research contributions, experimental design, and analysis are the authors’ own\.

## References

- M\. Arjovsky, L\. Bottou, I\. Gulrajani, and D\. Lopez\-PazInvariant risk minimization\.arXiv preprint arXiv:1907\.02893\.Cited by:[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Chenet al\.\(2025\)M\. Chen, T\. Li, H\. Sun, Y\. Zhou, C\. Zhu, H\. Wang, J\. Z\. Pan, W\. Zhang, H\. Chen, F\. Yang,et al\.ReSearch: learning to reason with search for llms via reinforcement learning\.arXiv preprint arXiv:2503\.19470\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§3\.3](https://arxiv.org/html/2609.16268#S3.SS3.p2.1)\.
- Fenget al\.\(2025\)J\. Feng, S\. Huang, X\. Qu, G\. Zhang, Y\. Qin, B\. Zhong, C\. Jiang, J\. Chi, and W\. ZhongRetool: reinforcement learning for strategic tool use in llms\.arXiv preprint arXiv:2504\.11536\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1),[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. NeubigPal: program\-aided language models\.InInternational Conference on Machine Learning \(ICML\),pp\. 10764–10799\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1),[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Geirhoset al\.\(2020\)R\. Geirhos, J\. Jacobsen, C\. Michaelis, R\. Zemel, W\. Brendel, M\. Bethge, and F\. A\. WichmannShortcut learning in deep neural networks\.Nature Machine Intelligence2\(11\),pp\. 665–673\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p2.1),[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Gururanganet al\.\(2018\)S\. Gururangan, S\. Swayamdipta, O\. Levy, R\. Schwartz, S\. Bowman, and N\. A\. SmithAnnotation artifacts in natural language inference data\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),pp\. 107–112\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p2.1),[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Heet al\.\(2025\)Z\. He, T\. Liang, J\. Xu, Q\. Liu, X\. Chen, Y\. Wang, L\. Song, D\. Yu, Z\. Liang, W\. Wang,et al\.Deepmath\-103k: a large\-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\.arXiv preprint arXiv:2504\.11456\.Cited by:[§3\.1](https://arxiv.org/html/2609.16268#S3.SS1.p1.1)\.
- Hoet al\.\(2020\)X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. AizawaConstructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics \(COLING\),pp\. 6609–6625\.Cited by:[§3\.3](https://arxiv.org/html/2609.16268#S3.SS3.p2.1)\.
- Izmailovet al\.\(2022\)P\. Izmailov, P\. Kirichenko, N\. Gruver, and A\. G\. WilsonOn feature learning in the presence of spurious correlations\.Advances in Neural Information Processing Systems35,pp\. 38516–38532\.Cited by:[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Jianget al\.\(2025\)D\. Jiang, Y\. Lu, Z\. Li, Z\. Lyu, P\. Nie, H\. Wang, A\. Su, H\. Chen, K\. Zou, C\. Du,et al\.VerlTool: towards holistic agentic reinforcement learning with tool use\.arXiv preprint arXiv:2509\.01055\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1),[§5\.1](https://arxiv.org/html/2609.16268#S5.SS1.p1.1)\.
- Jinet al\.\(2025\)B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. HanSearch\-R1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1),[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Kirichenkoet al\.\(2022\)P\. Kirichenko, P\. Izmailov, and A\. G\. WilsonLast layer re\-training is sufficient for robustness to spurious correlations\.arXiv preprint arXiv:2204\.02937\.Cited by:[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.Cited by:[§3\.1](https://arxiv.org/html/2609.16268#S3.SS1.p1.1)\.
- Liet al\.\(2025a\)X\. Li, G\. Dong, J\. Jin, Y\. Zhang, Y\. Zhou, Y\. Zhu, P\. Zhang, and Z\. DouSearch\-o1: agentic search\-enhanced large reasoning models\.arXiv preprint arXiv:2501\.05366\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Liet al\.\(2025b\)X\. Li, H\. Zou, and P\. LiuToRL: scaling tool\-integrated rl\.arXiv preprint arXiv:2503\.23383\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1),[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Liet al\.\(2025c\)Z\. Li, H\. Zhang, S\. Han, S\. Liu, J\. Xie, Y\. Zhang, Y\. Choi, J\. Zou, and P\. LuIn\-the\-flow agentic system optimization for effective planning and tool use\.arXiv preprint arXiv:2510\.05592\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1)\.
- Liuet al\.\(2021\)E\. Z\. Liu, B\. Haghgoo, A\. S\. Chen, A\. Raghunathan, P\. W\. Koh, S\. Sagawa, P\. Liang, and C\. FinnJust train twice: improving group robustness without training group information\.InInternational Conference on Machine Learning,pp\. 6781–6792\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p2.1),[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Maiet al\.\(2025\)X\. Mai, H\. Xu, X\. W, W\. Wang, Y\. Zhang, and W\. ZhangAgent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving\.arXiv preprint arXiv:2505\.07773\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Paranjapeet al\.\(2023\)B\. Paranjape, S\. Lundberg, S\. Singh, H\. Hajishirzi, L\. Zettlemoyer, and M\. T\. RibeiroArt: automatic multi\-step reasoning and tool\-use for large language models\.arXiv preprint arXiv:2303\.09014\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1),[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Presset al\.\(2023\)O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. LewisMeasuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5687–5711\.Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1)\.
- Qinet al\.\(2023\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.Toolllm: facilitating large language models to master 16000\+ real\-world apis\.arXiv preprint arXiv:2307\.16789\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Sagawaet al\.\(2019\)S\. Sagawa, P\. W\. Koh, T\. B\. Hashimoto, and P\. LiangDistributionally robust neural networks for group shifts: on the importance of regularization for worst\-case generalization\.arXiv preprint arXiv:1911\.08731\.Cited by:[§2\.1](https://arxiv.org/html/2609.16268#S2.SS1.p1.1),[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§5\.1](https://arxiv.org/html/2609.16268#S5.SS1.p1.1)\.
- Songet al\.\(2025\)H\. Song, J\. Jiang, Y\. Min, J\. Chen, Z\. Chen, W\. X\. Zhao, L\. Fang, and J\. WenR1\-searcher: incentivizing the search capability in llms via reinforcement learning\.arXiv preprint arXiv:2503\.05592\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Sunet al\.\(2025\)H\. Sun, Z\. Qiao, J\. Guo, X\. Fan, Y\. Hou, Y\. Jiang, P\. Xie, Y\. Zhang, F\. Huang, and J\. ZhouZerosearch: incentivize the search capability of llms without searching\.arXiv preprint arXiv:2505\.04588\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Wanget al\.\(2025a\)D\. Wang, J\. Yang, W\. Li, J\. Liang, and Y\. LiMSARL: decoupling reasoning and tool use with multi\-small\-agent reinforcement learning\.arXiv preprint arXiv:2508\.08882\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Wanget al\.\(2025b\)H\. Wang, C\. Qian, W\. Zhong, X\. Chen, J\. Qiu, S\. Huang, B\. Jin, M\. Wang, K\. Wong, and H\. JiActing less is reasoning more\! teaching model to act efficiently\.arXiv preprint arXiv:2504\.14870\.External Links:[Link](https://arxiv.org/pdf/2504.14870)Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Wanget al\.\(2025c\)Z\. Wang, X\. Zheng, K\. An, C\. Ouyang, J\. Cai, Y\. Wang, and Y\. WuStepsearch: igniting llms search ability via step\-wise proximal policy optimization\.arXiv preprint arXiv:2505\.15107\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Xueet al\.\(2025\)Z\. Xue, L\. Zheng, Q\. Liu, Y\. Li, X\. Zheng, Z\. Ma, and B\. AnSimpletir: end\-to\-end reinforcement learning for multi\-turn tool\-integrated reasoning\.arXiv preprint arXiv:2509\.02479\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Yanget al\.\(2024a\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§A\.1](https://arxiv.org/html/2609.16268#A1.SS1.p1.1)\.
- Yanget al\.\(2025\)Y\. Yang, C\. P\. Lee, S\. Feng, D\. Zhao, B\. Wen, A\. Z\. Liu, Y\. Tsvetkov, and B\. HoweEscaping the spuriverse: can large vision\-language models generalize beyond seen spurious correlations?\.arXiv preprint arXiv:2506\.18322\.Cited by:[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Yanget al\.\(2024b\)Y\. Yang, E\. Gan, G\. K\. Dziugaite, and B\. MirzasoleimanIdentifying spurious biases early in training through the lens of simplicity bias\.InInternational conference on artificial intelligence and statistics,pp\. 2953–2961\.Cited by:[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. CaoReact: synergizing reasoning and acting in language models\.InThe eleventh international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2609.16268#S1.p1.1),[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Zenget al\.\(2025\)S\. Zeng, Q\. Wei, W\. Brown, O\. Frunza, Y\. Nevmyvaka, and M\. HongReinforcing multi\-turn reasoning in llm agents via turn\-level credit assignment\.arXiv preprint arXiv:2505\.11821\.Cited by:[§6\.2](https://arxiv.org/html/2609.16268#S6.SS2.p1.1)\.
- Zhanget al\.\(2022\)M\. Zhang, N\. S\. Sohoni, H\. R\. Zhang, C\. Finn, and C\. RéCorrect\-n\-contrast: a contrastive approach for improving robustness to spurious correlations\.arXiv preprint arXiv:2203\.01517\.Cited by:[§6\.1](https://arxiv.org/html/2609.16268#S6.SS1.p1.1)\.

## Appendix AAppendix

### A\.1Base Model

We useQwen2\.5\-7B\-Instruct\([Yang et al\., 2024a](https://arxiv.org/html/2609.16268#bib.bib37)\), a 7B\-parameter decoder\-only transformer instruction\-tuned via SFT and RLHF by Alibaba\. No additional supervised fine\-tuning is applied; RL training starts directly from the instruction\-tuned checkpoint\.

### A\.2Training Hyperparameters

Table 6:Training hyperparameters for all experiments\. Each cue condition is trained as a separate run with the same hyperparameters\.
### A\.3Judge Prompts

We use two LLM judges during training: one for answer correctness and one for tool necessity\. Both use GPT\-5 Nano\.

#### Answer correctness judge\.

Used to compute the task rewardrtask∈\{0,1\}r^\{\\text\{task\}\}\\in\\\{0,1\\\}\.

> You are an evaluator\. Determine whether the Model Response’s final answer matches the Ground Truth\. Instructions: 1\. Extract the final answer from the Model Response\. Ignore reasoning and intermediate steps\. Use the last explicit answer given\. 2\. Normalize before comparison: \- Numbers: ignore commas and trailing zeros; treat decimals and fractions as equivalent \(0\.5 = 1/2\)\. \- Text: ignore case, punctuation, and extra whitespace\. \- Math: algebraically equivalent expressions count as equal; order does not matter for sets/tuples\. \- Multiple choice: accept either the option letter or the option text\. \- Ignore units unless they change the meaning\. 3\. Return true only if the final answers are mathematically or semantically equivalent\. If no clear final answer is present, return false\. Question: \{question\} Model Response: \{response\_str\} Ground Truth: \{ground\_truth\} Output JSON: \{"analysis": "brief explanation", "true\_false": true or false\}

#### Tool necessity judge\.

Used to compute the necessity rewardrtnecr\_\{t\}^\{\\text\{nec\}\}\(Section[4](https://arxiv.org/html/2609.16268#S4)\)\.

> You are judging whether THIS tool call is NECESSARY to solve the original question correctly\. A tool call is NECESSARY only if a competent model cannot answer the question correctly without using this tool \(at some point\)\. Convenience uses \(trivial arithmetic, formatting, double\-checking\) are NOT necessary\. Do NOT judge whether the tool input looks reasonable by itself\. Judge necessity for the question\. Question: \{question\} Tool called: \{tool\_type\} Tool input: \{tool\_content\} Return ONLY JSON: \{"tool\_necessary": true, "reason": "1\-2 sentences"\}