HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoning

# HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoning
Source: [https://arxiv.org/html/2608.21863](https://arxiv.org/html/2608.21863)
Yucan GuoThanks:Co\-first authors\.Affiliation:State Key Laboratory of AI SafetyAffiliation:Institute of Computing Technology, Chinese Academy of SciencesAffiliation:University of Chinese Academy of SciencesMiao SuAffiliation:State Key Laboratory of AI SafetyAffiliation:Institute of Computing Technology, Chinese Academy of SciencesAffiliation:University of Chinese Academy of SciencesSaiping GuanThanks:Corresponding authors\.Zhongni HouAffiliation:Meituan\{guoyucan23z, guansaiping, jinxiaolong\}@ict\.ac\.cnJiajun ChaiAffiliation:Meituan\{guoyucan23z, guansaiping, jinxiaolong\}@ict\.ac\.cnWei LinAffiliation:Meituan\{guoyucan23z, guansaiping, jinxiaolong\}@ict\.ac\.cnGuojun YinAffiliation:Meituan\{guoyucan23z, guansaiping, jinxiaolong\}@ict\.ac\.cnXiaolong JinAffiliation:State Key Laboratory of AI SafetyAffiliation:Institute of Computing Technology, Chinese Academy of SciencesAffiliation:University of Chinese Academy of SciencesJiafeng GuoAffiliation:State Key Laboratory of AI SafetyAffiliation:Institute of Computing Technology, Chinese Academy of SciencesAffiliation:University of Chinese Academy of SciencesXueqi ChengAffiliation:State Key Laboratory of AI SafetyAffiliation:Institute of Computing Technology, Chinese Academy of SciencesAffiliation:University of Chinese Academy of Sciences

###### Abstract

Tool\-Integrated Reasoning \(TIR\) is a fundamental capability for LLM agents to solve complex tasks by interacting with external tools iteratively\. Reinforcement Learning \(RL\) has become the dominant paradigm for enabling this capability\. However, existing approaches typically assign uniform trajectory\-level advantages and treat all correct tool calls equally, ignoring the varying difficulty and learning value across trajectories and reasoning steps\. This can lead to imprecise learning signals that do not adequately distinguish between trivial and challenging tool\-use patterns\. To address this limitation, we proposeHiDiffTIR, aHierarchicalDifficulty\-aware policy optimization framework for multi\-turnTIR\.HiDiffTIRperforms difficulty\-aware credit assignment at both trajectory and turn levels, enabling the policy to focus on more informative trajectories and harder reasoning steps\. Notably, this fine\-grained optimization is achieved without additional supervision, relying solely on group\-level statistics derived from standard RL rollouts\. Extensive experiments on three tool\-using benchmarks demonstrate thatHiDiffTIRconsistently improves multi\-turn TIR performance and tool invocation accuracy over strong RL baselines, highlighting the necessity of difficulty\-aware credit assignment for effective policy optimization in tool\-integrated LLM agents\.111The code is publicly available at[https://github\.com/YucanGuo/HiDiffTIR](https://github.com/YucanGuo/HiDiffTIR)\.

## 1Introduction

Large Language Models \(LLMs\) have demonstrated strong reasoning capabilities across a wide range of tasks[23](https://arxiv.org/html/2608.21863#bib.bib2);[3](https://arxiv.org/html/2608.21863#bib.bib1);[41](https://arxiv.org/html/2608.21863#bib.bib3), leading to the emergence of LLM agents that can plan, act, and interact with external environments[10](https://arxiv.org/html/2608.21863#bib.bib10);[37](https://arxiv.org/html/2608.21863#bib.bib11);[21](https://arxiv.org/html/2608.21863#bib.bib12)\. A central capability of such agents is effective tool use, where they invoke external tools to solve diverse tasks[22](https://arxiv.org/html/2608.21863#bib.bib14);[39](https://arxiv.org/html/2608.21863#bib.bib13)\. Tool\-Integrated Reasoning \(TIR\) formalizes this process by enabling LLM agents to iteratively interact with external tools[7](https://arxiv.org/html/2608.21863#bib.bib16);[27](https://arxiv.org/html/2608.21863#bib.bib20), such as APIs[26](https://arxiv.org/html/2608.21863#bib.bib17);[28](https://arxiv.org/html/2608.21863#bib.bib15), search engines[16](https://arxiv.org/html/2608.21863#bib.bib19);[12](https://arxiv.org/html/2608.21863#bib.bib4), and code interpreters[7](https://arxiv.org/html/2608.21863#bib.bib16);[18](https://arxiv.org/html/2608.21863#bib.bib18)\. This paradigm extends LLM agents beyond their static parametric knowledge and their known limitations in domains such as precise numerical computation[5](https://arxiv.org/html/2608.21863#bib.bib23), using specific tools to solve complex multi\-turn tasks that are otherwise challenging for purely inner knowledge\-based reasoning[20](https://arxiv.org/html/2608.21863#bib.bib25);[30](https://arxiv.org/html/2608.21863#bib.bib24)\.

Reinforcement learning \(RL\) has emerged as the prevailing paradigm for training LLMs to perform TIR[27](https://arxiv.org/html/2608.21863#bib.bib20);[40](https://arxiv.org/html/2608.21863#bib.bib21)\. By employing RL algorithms like Group Relative Policy Optimization \(GRPO\)[32](https://arxiv.org/html/2608.21863#bib.bib5), models learn to interleave reasoning, tool invocation, and answer generation in an end\-to\-end manner\. Most existing TIR methods adopt trajectory\-level optimization, which has evolved through two primary stages\. Early methods rely on outcome\-based rewards[18](https://arxiv.org/html/2608.21863#bib.bib18);[4](https://arxiv.org/html/2608.21863#bib.bib27), assessing only the final correctness of a reasoning chain, and thus suffer from reward sparsity\. To provide denser feedback, recent works have shifted toward turn\-level verification[27](https://arxiv.org/html/2608.21863#bib.bib20);[43](https://arxiv.org/html/2608.21863#bib.bib28), where the correctness of each individual tool\-call is evaluated and subsequently aggregated to form the trajectory\-level reward\. Despite this progress, these methods typically assign uniform advantages across all steps within a trajectory, implicitly treating each tool call as equally informative for learning\. This coarse\-grained credit assignment can obscure the contribution of individual reasoning steps, especially in multi\-turn settings where errors accumulate and only a subset of decisions are critical to success\.

![Refer to caption](https://arxiv.org/html/2608.21863v1/motivation.png)Figure 1:Comparison of credit assignment manners in TIR\. Existing methods treat all correct tool calls uniformly, whileHiDiffTIRperforms difficulty\-aware credit assignment\.To address this limitation, more recent works have explored finer\-grained credit assignment by incorporating turn\-level signals and combining them with trajectory\-level objectives[17](https://arxiv.org/html/2608.21863#bib.bib22);[29](https://arxiv.org/html/2608.21863#bib.bib26)\. These methods typically perform trajectory\-turn fusion, where trajectory\-level rewards are used in their original form, and turn\-level signals are either directly applied or heuristically adjusted and propagated across steps\. However, they treat all correct tool calls equivalently, without explicitly modeling the varying difficulty of different reasoning steps\. As a result, the learning signal remains insensitive to the relative importance and challenge of different tool\-use patterns\.

In this paper, we revisit credit assignment in TIR from a difficulty\-aware perspective\. As shown in[Figure1](https://arxiv.org/html/2608.21863#S1.F1), we argue that not all trajectories or tool\-calling steps contribute equally to policy improvement\. Some trajectories are more informative due to their relative difficulty, and some reasoning steps are inherently harder and thus more valuable for learning\. Motivated by this, we proposeHiDiffTIR, aHierarchicalDifficulty\-aware policy optimization framework forTIRthat performs fine\-grained credit assignment without requiring additional supervision\.HiDiffTIRoperates at two levels\. At the trajectory level, it differentiates rollouts based on their relative difficulty, enabling the policy to prioritize more informative training signals\. At the turn level, it further refines credit assignment by redistributing advantages across reasoning steps according to their difficulty, allowing the model to focus on harder tool\-calling decisions\. Importantly, the judgment of tool\-using difficulty relies solely on group\-level statistics derived from standard RL rollouts and does not require external supervision such as LLM\-based judges or rubrics\. Our contributions are summarized as follows:

- •We proposeHiDiffTIR, a hierarchical policy optimization framework that performs difficulty\-aware credit assignment at both the trajectory and turn levels\.
- •We introduce a supervision\-free method for fine\-grained policy optimization that relies solely on statistics from standard RL rollouts\.
- •Extensive experiments demonstrate thatHiDiffTIRimproves tool\-use accuracy and multi\-turn reasoning performance across multiple benchmarks\.

## 2Related Work

### 2\.1Tool\-Integrated Reasoning

TIR has become a central paradigm for equipping LLM agents with external tools to solve complex tasks\. Early work primarily relies on Supervised Fine\-Tuning \(SFT\), where models are trained on solution annotations from strong LLMs that interleave reasoning steps and tool invocations[31](https://arxiv.org/html/2608.21863#bib.bib29);[35](https://arxiv.org/html/2608.21863#bib.bib30);[26](https://arxiv.org/html/2608.21863#bib.bib17);[28](https://arxiv.org/html/2608.21863#bib.bib15)\. While effective in establishing basic tool\-use capabilities, recent research has increasingly shifted toward RL to further improve policy learning through interaction[40](https://arxiv.org/html/2608.21863#bib.bib21);[18](https://arxiv.org/html/2608.21863#bib.bib18);[34](https://arxiv.org/html/2608.21863#bib.bib31)\. Within RL\-based TIR, methods have evolved from trajectory\-level optimization with outcome\-based rewards[18](https://arxiv.org/html/2608.21863#bib.bib18);[4](https://arxiv.org/html/2608.21863#bib.bib27)to incorporating process\-based verification signals, such as tool name, parameter name, and argument values, to provide denser supervision[27](https://arxiv.org/html/2608.21863#bib.bib20);[43](https://arxiv.org/html/2608.21863#bib.bib28)\. More recent approaches further integrate trajectory\-level and turn\-level signals to provide individual advantage for each turn\. For instance, DeepAgent[17](https://arxiv.org/html/2608.21863#bib.bib22)combines outcome rewards with turn\-level tool call rewards, while MatchTIR[29](https://arxiv.org/html/2608.21863#bib.bib26)further propagates turn\-level rewards across steps using discounted accumulation\. Despite these advances, existing methods generally treat all correct tool calls as equally informative, without explicitly modeling the varying difficulty of different trajectories or reasoning steps\.

![Refer to caption](https://arxiv.org/html/2608.21863v1/method.png)Figure 2:Illustration of theHiDiffTIRframework\.HiDiffTIRimplements a hierarchical difficulty\-aware credit assignment mechanism: \(a\) trajectory\-level, where raw turn rewards are reweighted based on turn difficulties to compute a difficulty\-aware trajectory advantage, and \(b\) turn\-level, where the trajectory advantage is redistributed to yield fine\-grained reweighted turn advantages for policy optimization\.
### 2\.2Reinforcement Learning for LLM Reasoning

RL has been widely adopted to improve the reasoning capabilities of LLMs beyond SFT\. A prominent line of work is Reinforcement Learning from Human Feedback \(RLHF\), where models are optimized using preference signals collected from human annotators[1](https://arxiv.org/html/2608.21863#bib.bib6);[24](https://arxiv.org/html/2608.21863#bib.bib7)\. Recent work has demonstrated the effectiveness of Reinforcement Learning from Verifiable Rewards \(RLVR\), where models are optimized using automatically computed correctness signals[32](https://arxiv.org/html/2608.21863#bib.bib5);[3](https://arxiv.org/html/2608.21863#bib.bib1);[41](https://arxiv.org/html/2608.21863#bib.bib3)\. To improve reward quality, a line of studies also concentrates on enhancing credit assignment in LLM reasoning through process\-oriented supervision[36](https://arxiv.org/html/2608.21863#bib.bib36);[19](https://arxiv.org/html/2608.21863#bib.bib35)\. Such approaches typically rely on additional supervision signals, including process reward models[45](https://arxiv.org/html/2608.21863#bib.bib37);[38](https://arxiv.org/html/2608.21863#bib.bib40), LLM\-based judges guided by evaluation rubrics[8](https://arxiv.org/html/2608.21863#bib.bib38);[11](https://arxiv.org/html/2608.21863#bib.bib39), to assess intermediate reasoning steps and provide more fine\-grained feedback\. In parallel, there is growing interest in difficulty\-aware RL[44](https://arxiv.org/html/2608.21863#bib.bib32);[6](https://arxiv.org/html/2608.21863#bib.bib33);[9](https://arxiv.org/html/2608.21863#bib.bib34), where training emphasizes more challenging samples to improve model performance, with sample difficulty estimated based on outcome\-level signals\. However, these methods are largely designed for general reasoning tasks, making them insufficient for jointly modeling process\-level feedback and difficulty in TIR\.

## 3HiDiffTIR

### 3\.1Problem Formulation

Given a user queryqqand a set of available tools𝒯=\{t1,…,tn\}\\mathcal\{T\}=\\\{t\_\{1\},\\dots,t\_\{n\}\\\}, the goal of a TIR agentπθ\\pi\_\{\\theta\}is to solve the query through a multi\-turn reasoning and tool use interaction processτ=\{\(z1,a1,f1\),…,\(zT−1,aT−1,fT−1\),\(zT,a​n​s\)\}\\tau=\\\{\(z\_\{1\},a\_\{1\},f\_\{1\}\),\\dots,\(z\_\{T\-1\},a\_\{T\-1\},f\_\{T\-1\}\),\(z\_\{T\},ans\)\\\}, whereTTdenotes the total number of turns\. At each intermediate turni<Ti<T, the agent generates a reasoning stepziz\_\{i\}and a set of tool actionsaia\_\{i\}, where each action specifies a selected tool from𝒯\\mathcal\{T\}along with its arguments\. The environment executes these tool calls and returns feedbackfif\_\{i\}\. At the final turn, the agent produces a reasoning stepzTz\_\{T\}followed by the final answera​n​sanswithout invoking any tool\. The objective of a TIR agent is to learn a policy that generates trajectories leading to correct final answers by effectively interleaving reasoning and tool use\.

### 3\.2Overall Framework

HiDiffTIRis a hierarchical policy optimization framework for TIR that introduces difficulty\-aware credit assignment at two levels\. The core idea is to distinguish the relative learning importance of different trajectories and turns, rather than treating them uniformly\. As shown in[Figure2](https://arxiv.org/html/2608.21863#S2.F2), at the trajectory level,HiDiffTIRdistinguishes trajectories according to the relative difficulty of all constituent tool calls \(§[3\.3](https://arxiv.org/html/2608.21863#S3.SS3)\)\. At the turn level, it further refines credit assignment across reasoning steps, considering both tool\-selection difficulty and tool\-using difficulty \(§[3\.4](https://arxiv.org/html/2608.21863#S3.SS4)\)\. These two components together enable fine\-grained optimization of the policy model \(§[3\.5](https://arxiv.org/html/2608.21863#S3.SS5)\)\.

### 3\.3Trajectory\-Level Difficulty\-Aware Credit Assignment

We now introduce trajectory\-level difficulty\-aware credit assignment, which extends turn\-level rewards based on tool\-use correctness\. Specifically, we first compute the similarity between predicted and ground\-truth tool calls to derive turn\-level raw rewards\. Next, we estimate the relative difficulty of each turn and reweight the raw rewards accordingly\. Finally, we normalize these rewards to construct a difficulty\-aware trajectory\-level advantage\.

#### Raw Reward Computation\.

Following previous works[27](https://arxiv.org/html/2608.21863#bib.bib20);[29](https://arxiv.org/html/2608.21863#bib.bib26), we compute the raw reward for each turn in a trajectory using tool\-level correctness\. Given a trajectoryτ\(i\)\\tau^\{\(i\)\}withTTturns, letCi=\{ci​1,…,ci​m\}C\_\{i\}=\\\{c\_\{i1\},\\dots,c\_\{im\}\\\}denote the predicted tool calls andCi∗=\{ci​1∗,…,ci​n∗\}C^\{\*\}\_\{i\}=\\\{c^\{\*\}\_\{i1\},\\dots,c^\{\*\}\_\{in\}\\\}the ground\-truth calls\. A similarity matrixMi∈ℝm×nM\_\{i\}\\in\\mathbb\{R\}^\{m\\times n\}is defined as

Mi​\[u,v\]=sim​\(ci​u,ci​v∗\),M\_\{i\}\[u,v\]=\\text\{sim\}\(c\_\{iu\},c^\{\*\}\_\{iv\}\),\(1\)whereu∈\{1,⋯,m\}u\\in\\\{1,\\cdots,m\\\},v∈\{1,⋯,n\}v\\in\\\{1,\\cdots,n\\\}, and the similarity functionsim​\(⋅,⋅\)∈\[0,1\]\\text\{sim\}\(\\cdot,\\cdot\)\\in\[0,1\]measures agreement on tool name, parameter names, and parameter values\. Detailed computation of the similarity function is provided in Appendix[A\.3](https://arxiv.org/html/2608.21863#A1.SS3)\.

A maximum\-weight bipartite matchingπi∗\\pi\_\{i\}^\{\*\}is then obtained via the Hungarian algorithm[13](https://arxiv.org/html/2608.21863#bib.bib42);[14](https://arxiv.org/html/2608.21863#bib.bib41):

πi∗=arg⁡max⁡∑\(u,v\)∈ππ∈Π⁡Mi​\[u,v\],\\pi\_\{i\}^\{\*\}=\\arg\\max\_\{\\pi\\in\\Pi\}\\sum\_\{\(u,v\)\\in\\pi\}M\_\{i\}\[u,v\],\(2\)whereΠ\\Pidenotes all one\-to\-one matchings betweenCiC\_\{i\}andCi∗C^\{\*\}\_\{i\}\.

LetCi,j⊆CiC\_\{i,j\}\\subseteq C\_\{i\}denote the subset of tool calls issued at turnjj\. The raw reward for turnjjis defined as

ri,j=1\|Ci,j\|∑ci​u∈Ci,j𝕀\[\(u,v\)∈πi∗\]⋅Mi\[u,v\],r\_\{i,j\}=\\frac\{1\}\{\|C\_\{i,j\}\|\}\\sum\_\{c\_\{iu\}\\in C\_\{i,j\}\}\\mathbb\{I\}\[\(u,v\)\\in\\pi\_\{i\}^\{\*\}\]\\cdot M\_\{i\}\[u,v\],\(3\)where unmatched tool calls contribute zero\. These turn\-level rewards serve as the initial supervision signal for subsequent trajectory\-level optimization\.

#### Difficulty\-Aware Reward Reweighting\.

To model the relative difficulty of different tool\-use behaviors, group\-level statistics are estimated over trajectories sampled for the same queryqq\. Let𝒢⁡\(q\)=\{τ\(1\),⋯,τ\(G\)\}\\mathcal\{G\}\(q\)=\\\{\\tau^\{\(1\)\},\\cdots,\\tau^\{\(G\)\}\\\}denote the group of trajectories generated forqq\. For each ground\-truth tool callc∗c^\{\*\}, we compute its average reward across the group:

r¯​\(c∗\)=1\|𝒢⁡\(q\)\|​∑τ\(k\)∈𝒢⁡\(q\)r\(k\)​\(c∗\),\\bar\{r\}\(c^\{\*\}\)=\\frac\{1\}\{\|\\mathcal\{G\}\(q\)\|\}\\sum\_\{\\tau^\{\(k\)\}\\in\\mathcal\{G\}\(q\)\}r^\{\(k\)\}\(c^\{\*\}\),\(4\)wherer\(k\)​\(c∗\)r^\{\(k\)\}\(c^\{\*\}\)denotes the reward assigned to the tool call in trajectoryτ\(k\)\\tau^\{\(k\)\}that is matched toc∗c^\{\*\}via bipartite matching\.

For each turnjjin trajectoryτ\(i\)\\tau^\{\(i\)\}, letCi,j∗C^\{\*\}\_\{i,j\}denote the set of ground\-truth tool calls associated with that turn\. We define the turn\-level average reward as

r¯i,j=1\|Ci,j∗\|​∑c∗∈Ci,j∗r¯​\(c∗\)\.\\bar\{r\}\_\{i,j\}=\\frac\{1\}\{\|C^\{\*\}\_\{i,j\}\|\}\\sum\_\{c^\{\*\}\\in C^\{\*\}\_\{i,j\}\}\\bar\{r\}\(c^\{\*\}\)\.\(5\)The difficulty of the turn is then defined as

di,j=1−r¯i,j,d\_\{i,j\}=1\-\\bar\{r\}\_\{i,j\},\(6\)and used to reweight the raw reward:

r~i,j=di,j⋅ri,j\.\\tilde\{r\}\_\{i,j\}=d\_\{i,j\}\\cdot r\_\{i,j\}\.\(7\)This weighting emphasizes turns associated with harder tool\-use decisions while down\-weighting consistently easy ones\.

#### Trajectory\-Level Score Construction\.

To ensure stable optimization, the reweighted rewards are normalized within each group\. A min\-max normalization is first applied:

r^i,j=r~i,j−minτ∈𝒢⁡\(q\)⁡r~τ,jmaxτ∈𝒢⁡\(q\)⁡r~τ,j−minτ∈𝒢⁡\(q\)⁡r~τ,j,\\hat\{r\}\_\{i,j\}=\\frac\{\\tilde\{r\}\_\{i,j\}\-\\min\_\{\\tau\\in\\mathcal\{G\}\(q\)\}\\tilde\{r\}\_\{\\tau,j\}\}\{\\max\_\{\\tau\\in\\mathcal\{G\}\(q\)\}\\tilde\{r\}\_\{\\tau,j\}\-\\min\_\{\\tau\\in\\mathcal\{G\}\(q\)\}\\tilde\{r\}\_\{\\tau,j\}\},\(8\)followed by rescaling to preserve the total reward mass:

r~i,jnorm=r^i,j⋅∑τ∈𝒢⁡\(q\)r~τ,j∑τ∈𝒢⁡\(q\)r^τ,j\.\\tilde\{r\}\_\{i,j\}^\{\\text\{norm\}\}=\\hat\{r\}\_\{i,j\}\\cdot\\frac\{\\sum\_\{\\tau\\in\\mathcal\{G\}\(q\)\}\\tilde\{r\}\_\{\\tau,j\}\}\{\\sum\_\{\\tau\\in\\mathcal\{G\}\(q\)\}\\hat\{r\}\_\{\\tau,j\}\}\.\(9\)The trajectory\-level score is then obtained by aggregating normalized turn rewards:

R⁡\(τ\(i\)\)=∑j=1Tir~i,jnorm,R\(\\tau^\{\(i\)\}\)=\\sum\_\{j=1\}^\{T\_\{i\}\}\\tilde\{r\}\_\{i,j\}^\{\\text\{norm\}\},\(10\)and converted into a group\-relative advantage:

A⁡\(τ\(i\)\)=R⁡\(τ\(i\)\)−μqσq,A\(\\tau^\{\(i\)\}\)=\\frac\{R\(\\tau^\{\(i\)\}\)\-\\mu\_\{q\}\}\{\\sigma\_\{q\}\},\(11\)whereμq\\mu\_\{q\}andσq\\sigma\_\{q\}denote the mean and standard deviation of trajectory scores within𝒢⁡\(q\)\\mathcal\{G\}\(q\)\. This formulation yields a difficulty\-aware trajectory\-level objective that emphasizes relatively harder yet successful trajectories while maintaining stable comparisons within each group\.

### 3\.4Turn\-Level Difficulty\-Aware Credit Assignment

While trajectory\-level optimization captures the overall difficulty of a rollout, it does not distinguish which reasoning steps are more challenging within the trajectory\. To further refine credit assignment, we first estimate the relative difficulty of each turn based on group\-level statistics, and then redistribute the trajectory\-level advantage across individual turns based on these turn\-level difficulties\.

#### Group\-Level Tool Selection Success Rate\.

For each ground\-truth tool callc∗c^\{\*\}, we compute its tool selection success rate over the group𝒢⁡\(q\)\\mathcal\{G\}\(q\):

s⁡\(c∗\)=1\|𝒢⁡\(q\)\|\+λ​∑τ\(k\)∈𝒢⁡\(q\)𝕀⁡\(r\(k\)​\(c∗\)\>0\)\+λ,s\(c^\{\*\}\)=\\frac\{1\}\{\|\\mathcal\{G\}\(q\)\|\+\\lambda\}\\sum\_\{\\tau^\{\(k\)\}\\in\\mathcal\{G\}\(q\)\}\\mathbb\{I\}\\big\(r^\{\(k\)\}\(c^\{\*\}\)\>0\\big\)\+\\lambda,\(12\)where𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)is an indicator function, andλ\\lambdais a smoothing constant\. Since a positive reward indicates that the correct tool is selected,s⁡\(c∗\)s\(c^\{\*\}\)effectively measures how frequently each tool call is correctly invoked across trajectories\.

After that, we aggregate the tool selection success rates into a turn\-level statistic for each turnjjin trajectoryτ\(i\)\\tau^\{\(i\)\}:

si,j=minc∗∈Ci,j∗⁡s⁡\(c∗\),s\_\{i,j\}=\\min\_\{c^\{\*\}\\in C^\{\*\}\_\{i,j\}\}s\(c^\{\*\}\),\(13\)which emphasizes the most challenging tool call within the turn\.

#### Difficulty\-Aware Advantage Reweighting\.

Based on the estimated tool selection success rates, we construct turn\-level weights that emphasize more difficult decisions\. For trajectories with positive advantage, we assign higher weights to turns with lower success rates while incorporating the overall tool\-using quality:

w~i,j=ri,jsi,j\.\\tilde\{w\}\_\{i,j\}=\\frac\{r\_\{i,j\}\}\{\\sqrt\{s\_\{i,j\}\}\}\.\(14\)This formulation suppresses incorrect turns while prioritizing harder correct decisions\.

For trajectories with negative advantage, we do not apply difficulty\-based reweighting and instead use uniform weights, ensuring stable penalization of unsuccessful rollouts\. The weights are then normalized within each trajectory:

w^i,j=w~i,j1Ti​∑k=1Tiw~i,k\.\\hat\{w\}\_\{i,j\}=\\frac\{\\tilde\{w\}\_\{i,j\}\}\{\\frac\{1\}\{T\_\{i\}\}\\sum\_\{k=1\}^\{T\_\{i\}\}\\tilde\{w\}\_\{i,k\}\}\.\(15\)
Finally, the trajectory\-level advantage is redistributed across turns as

Ai,j=A⁡\(τ\(i\)\)⋅\(α\+\(1−α\)​w^i,j\),A\_\{i,j\}=A\(\\tau^\{\(i\)\}\)\\cdot\\left\(\\alpha\+\(1\-\\alpha\)\\hat\{w\}\_\{i,j\}\\right\),\(16\)whereα∈\[0,1\]\\alpha\\in\[0,1\]controls the balance between trajectory\- and turn\-level difficulty\-aware credit assignment, preserving the global optimization signal while enabling fine\-grained emphasis on harder reasoning turns\.

### 3\.5Policy Optimization

The final advantage assigned to each token is constructed by combining trajectory\-level and turn\-level signals in a hierarchical manner\. The turn\-level advantagesAi,jA\_\{i,j\}are assigned to all tokens within the corresponding turn, resulting in token\-level advantagesA~i,t\\tilde\{A\}\_\{i,t\}\. Using the integrated advantageA~i,t\\tilde\{A\}\_\{i,t\}, we optimize the policy under the GRPO algorithm\. Given a batch of queriesq∼𝒟q\\sim\\mathcal\{D\}and sampled trajectories\{τi\}i=1G∼πθold\(⋅∣q\)\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid q\), the objective is defined as

𝒥\(θ\)=𝔼q,\{τi\}1G∑i=1G1\|τi\|∑t=1\|τi\|\[min\(wi,tA~i,t,clip\(wi,t,1−ϵ,1\+ϵ\)A~i,t\)−β𝔻KL\(πθ∥πref\)\],\\begin\{split\}\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\_\{q,\\\{\\tau\_\{i\}\\\}\}\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|\\tau\_\{i\}\|\}\\sum\_\{t=1\}^\{\|\\tau\_\{i\}\|\}\\bigg\[\\min\\Big\(w\_\{i,t\}\\tilde\{A\}\_\{i,t\},\\\\ \\text\{clip\}\(w\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)\\tilde\{A\}\_\{i,t\}\\Big\)\-\\beta\\mathbb\{D\}\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\text\{ref\}\}\)\\bigg\],\\end\{split\}\(17\)wherewi,t=πθ​\(yi,t∣yi,<t,q\)πθold​\(yi,t∣yi,<t,q\)w\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid y\_\{i,<t\},q\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{i,t\}\\mid y\_\{i,<t\},q\)\}denotes the token\-level importance sampling ratio, andβ\\betacontrols the strength of KL regularization\. This objective maintains the standard GRPO optimization structure while incorporating difficulty\-aware hierarchical credit assignment, enabling more effective learning for multi\-turn TIR\.

## 4Experiments

In this section, we evaluateHiDiffTIRthrough comparative experiments \(§[4\.2](https://arxiv.org/html/2608.21863#S4.SS2)\), ablation study \(§[4\.3](https://arxiv.org/html/2608.21863#S4.SS3)\), and tool\-use accuracy analysis \(§[4\.4](https://arxiv.org/html/2608.21863#S4.SS4)\), with additional results detailed in Appendix[B](https://arxiv.org/html/2608.21863#A2)\.

Table 1:Performance comparison betweenHiDiffTIRand the baselines on in\-domain and out\-of\-domain benchmarks across Qwen3\-4B and Qwen3\-8B backbones\. The best results within each backbone group are indicated in bold, while the underlined values represent the second\-best results\.### 4\.1Experimental Setting

#### Training Dataset\.

We conduct training on the FTRL dataset[43](https://arxiv.org/html/2608.21863#bib.bib28), which comprises 2,215 training data generated via an automated environment construction pipeline\. By executing all tools locally as code, it circumvents the unreliability of online APIs, providing stable and verifiable tool\-use training\.

#### Evaluation Datasets\.

We evaluateHiDiffTIRon three widely used TIR benchmarks: \(1\)FTRL[43](https://arxiv.org/html/2608.21863#bib.bib28), the held\-out test dataset generated by the FTRL pipeline; \(2\)ToolHop[42](https://arxiv.org/html/2608.21863#bib.bib44), a query\-driven benchmark specifically designed to evaluate multi\-hop tool reasoning; and \(3\) theBerkeley Function Call Leaderboard \(BFCL\)[25](https://arxiv.org/html/2608.21863#bib.bib43), a comprehensive and executable function call benchmark that evaluates function\-calling ability across diverse APIs and parameter configurations\.

#### Baselines\.

We compareHiDiffTIRagainst several strong TIR baselines: \(1\)Vanilla, which directly performs TIR using the backbone Qwen3[41](https://arxiv.org/html/2608.21863#bib.bib3)model without RL training; \(2\)GRPO[32](https://arxiv.org/html/2608.21863#bib.bib5), standard GRPO framework, which computes advantages solely based on the outcome reward; \(3\)ToolRL[27](https://arxiv.org/html/2608.21863#bib.bib20), an RL\-based TIR training framework that introduces process\-based tool\-call verification rewards; \(4\)FTRL[43](https://arxiv.org/html/2608.21863#bib.bib28), a feedback\-driven training framework using a verifiable reward mechanism based on precision and completeness; and \(5\)MatchTIR[29](https://arxiv.org/html/2608.21863#bib.bib26), a recent fine\-grained credit assignment framework that derives turn\-level rewards with two credit assignment manners:MatchTIR \(KM\), which formulates the assignment as a bipartite matching problem using the Kuhn\-Munkres algorithm[13](https://arxiv.org/html/2608.21863#bib.bib42), andMatchTIR \(OT\), which smooths the assignment through optimal transport[2](https://arxiv.org/html/2608.21863#bib.bib45)\.

#### Implementation Details\.

We conduct experiments using Qwen3\-4B and Qwen3\-8B[41](https://arxiv.org/html/2608.21863#bib.bib3)as the backbone models\. All RL training is implemented based on the verl framework[33](https://arxiv.org/html/2608.21863#bib.bib8), with vLLM[15](https://arxiv.org/html/2608.21863#bib.bib9)accelerating the rollout generation\. For GRPO settings, we sample a group size of 16 responses per prompt at a temperature of 1\.0\. The policy model is trained for 5 epochs using a global batch size of 256, restricting the multi\-turn reasoning process to a maximum of 10 turns per trajectory\. We set the smoothing constantλ\\lambdato 1\.0 and the fusion coefficientα\\alphato 0\.7\. All experiments are executed on a single node equipped with 8 NVIDIA H20 GPUs\. Additional implementation details are provided in Appendix[A](https://arxiv.org/html/2608.21863#A1)\.

### 4\.2Main Results

Table 2:Ablation study on credit assignment using Qwen3\-4B\.The main experimental results are presented in[Table1](https://arxiv.org/html/2608.21863#S4.T1)\. On the basis of these results, we make the following observations\.

HiDiffTIRconsistently achieves the best overall performance across different evaluation benchmarks\.HiDiffTIRoutperforms all baselines on the average score for both Qwen3\-4B and Qwen3\-8B\. Compared with existing RL\-based TIR methods, our method demonstrates greater improvements in both in\-domain and out\-of\-domain settings\. In particular, our method consistently surpasses recent strong baselines such as FTRL and MatchTIR, showing that incorporating hierarchical difficulty\-aware optimization provides complementary benefits beyond existing process\-based and turn\-level reward modeling strategies\.

The effectiveness ofHiDiffTIRgeneralizes across different backbone sizes\.Although larger backbones generally achieve stronger overall performance,HiDiffTIRconsistently improves upon the corresponding baselines under both the 4B and 8B settings\. Notably, the performance gains on the in\-domain FTRL benchmark are more pronounced for Qwen3\-4B, suggesting that smaller models benefit more from fine\-grained difficulty\-aware optimization\. This observation indicates that improved credit assignment can partially compensate for weaker intrinsic reasoning and tool\-use capabilities in smaller models\. Meanwhile, the consistent improvements on Qwen3\-8B demonstrate the robustness and scalability of our framework\.

HiDiffTIRbrings larger improvements on datasets with a larger number of tools\.As shown in[Table5](https://arxiv.org/html/2608.21863#A1.T5), datasets such as FTRL and ToolHop contain substantially larger numbers of tools compared with BFCL\. Correspondingly,HiDiffTIRachieves more evident improvements on these benchmarks, which require reasoning over thousands of candidate tools\. In contrast, on datasets with relatively fewer tools, such as BFCL, the performance improvements are comparatively modest, while stronger backbone models already obtain noticeable benefits from scaling alone\. These findings highlight that the advantages of difficulty\-aware optimization become increasingly important as the complexity of the tool environment grows\.

Current RL\-based TIR methods still struggle with missing\-information scenarios\.Although most methods achieve clear improvements on the BFCL benchmark, the gains on the Missing Functions \(MF\) and Missing Parameters \(MP\) subsets remain modest, as they require the model to recognize when the currently available tools or user\-provided information are insufficient\. Existing RL\-based TIR training mainly emphasizes successful tool execution and task completion, which may bias the policy toward aggressively invoking tools rather than learning insufficiency detection\. Specifically, because our difficulty\-aware reweighting amplifies successful complex tool sequences, it inadvertently reinforces this aggressive calling bias in sparse tool settings\. Consequently, whileHiDiffTIRsignificantly outperforms most baselines on the MF subset, it slightly trails MatchTIR\. This observation suggests an important future direction for TIR, i\.e\., integrating self\-awareness of capability boundaries into RL optimization\.

### 4\.3Ablation Study

We further conduct ablation studies on Qwen3\-4B to evaluate the contribution of each component inHiDiffTIR\.

#### Effectiveness of Hierarchical Components\.

As shown in[Table2](https://arxiv.org/html/2608.21863#S4.T2), removing either trajectory\-level or turn\-level difficulty\-aware credit assignment leads to consistent performance degradation compared with the full model, demonstrating that both components contribute to the final performance\. In particular, removing trajectory\-level optimization results in a larger overall drop on FTRL, indicating that distinguishing the relative difficulty of different trajectories provides an important optimization signal for RL training\. Furthermore, performance variations on Out\-Of\-Domain \(OOD\) datasets like BFCL and ToolHop are less pronounced when omitting specific components\. Since our difficulty signals are derived dynamically during training, they naturally target bottlenecks within the training distribution\. Thus, these components are highly effective for seen tools but have a more constrained impact on completely unseen tools in OOD scenarios, although the fullHiDiffTIRmodel still achieves the best overall average performance\.

![Refer to caption](https://arxiv.org/html/2608.21863v1/figs/abl/response_length_mean.png)\(a\)Average Response Length
![Refer to caption](https://arxiv.org/html/2608.21863v1/figs/abl/timing_s_gen.png)\(b\)Generation Time

Figure 3:Training dynamics regarding the average response length and generation time of different variants\.Table 3:Comparison of reasoning efficiency at the final training step\.
#### Efficiency of Hierarchical Components\.

The training dynamics of reasoning efficiency are visualized in[Figure3](https://arxiv.org/html/2608.21863#S4.F3), with convergence values summarized in[Table3](https://arxiv.org/html/2608.21863#S4.T3)\. We observe thatHiDiffTIRachieves higher reasoning compactness compared to thew/oturn\-level variant, which relies solely on trajectory\-level credit assignment\. Specifically, at the final training step,HiDiffTIRreduces the average response length from 2,085 to 1,893 tokens, resulting in a 12\.8% decrease in generation latency\. This suggests that fine\-grained turn\-level reweighting effectively identifies critical reasoning steps and suppresses redundant tool calls\. Although thew/otrajectory\-level variant exhibits slightly lower latency, its performance is substantially inferior to the full model, confirming that our hierarchical design achieves an optimal trade\-off between reasoning accuracy and computational efficiency\.

### 4\.4Tool\-Use Accuracy Analysis

![Refer to caption](https://arxiv.org/html/2608.21863v1/figs/accuracy/tool_call_gt_ratio.png)Figure 4:Training dynamics regarding tool\-use accuracy of different models\.Table 4:Performance of models on FTRL dataset under different fusion coefficients\.In this section, we analyze how difficulty\-aware credit assignment influences the tool\-use accuracy\.

#### Impact on Tool Selection Precision\.

As illustrated in[Figure4](https://arxiv.org/html/2608.21863#S4.F4), all variants ofHiDiffTIRconsistently outperform the competitive baseline MatchTIR \(KM\) in tool\-use accuracy throughout the training process\. This convergence to higher accuracy confirms that integrating difficulty\-aware credit assignment provides a clearer gradient signal, effectively guiding the policy to master correct tool invocation patterns\. Specifically, variants with smaller fusion coefficients \(α=0\.3\\alpha=0\.3andα=0\.5\\alpha=0\.5\) exhibit higher peak tool accuracy during training compared toα=0\.7\\alpha=0\.7\. This trend occurs because a lowerα\\alphaassigns a larger proportional weight to the turn\-level difficulty\-aware credit assignment, forcing the policy to aggressively optimize high\-difficulty tool calls within training rollouts\.

#### Sensitivity Analysis of Fusion Coefficient\.

Despite the higher training accuracy at lowerα\\alphavalues,[Table4](https://arxiv.org/html/2608.21863#S4.T4)reveals thatα=0\.7\\alpha=0\.7achieves the highest average score on the FTRL test set\. This counterintuitive phenomenon highlights the cooperative dynamics between the two credit assignment levels\. A smallerα\\alphaover\-emphasizes turn\-level difficulty\-aware credit assignment, rigidly rewarding only ground\-truth tool invocations and suppressing intermediate turns with necessary trial\-and\-error exploration\. In complex TIR, such exploration is not meaningless but is indispensable for the agent to gather feedback and learn from failures\. Conversely, settingα=0\.7\\alpha=0\.7provides a well\-suited balance, leveraging trajectory\-level credit assignment to safeguard the value of the exploratory sequence while utilizing turn\-level credit assignment to resolve specific tool\-use bottlenecks\.

## 5Conclusions

In this paper, we proposeHiDiffTIR, a hierarchical policy optimization framework for TIR that performs difficulty\-aware credit assignment at both the trajectory and turn levels\. By modeling the relative difficulty of trajectories and redistributing advantages across reasoning steps, our method provides fine\-grained learning signals that better capture the heterogeneous importance of tool\-calling decisions, without requiring additional supervision\. Extensive experiments on multiple tool\-using benchmarks demonstrate thatHiDiffTIRconsistently improves multi\-turn TIR performance and tool invocation accuracy, suggesting that incorporating difficulty\-aware signals is a promising direction for improving the training of tool\-using LLM agents\.

## Limitations

Despite the strong empirical results, our proposedHiDiffTIRhas several limitations\. Primarily, owing to limited computational resources, our empirical evaluations were restricted to moderately sized open\-source models, specifically Qwen3\-4B and Qwen3\-8B\. Although the core algorithmic design of our hierarchical difficulty\-aware policy optimization framework is inherently model\-agnostic, its behavioral dynamics and scalability when integrated into LLMs with larger parameters have not yet been fully validated\. Additionally, because our difficulty\-aware signals are derived dynamically from online group\-level rollout statistics, the framework implicitly assumes that the base model possesses a foundational level of instruction\-following capability\. In exceptionally intricate tasks where the initial policy completely fails to produce any valid tool invocations, the empirical difficulty estimation might suffer from extreme reward sparsity, potentially necessitating a supervised warm\-up phase to kickstart effective reinforcement learning exploration\. Finally, the difficulty estimation mechanism in our framework serves as a heuristic\. It captures empirical learning difficulty from the perspective of operational bottlenecks for the policy during RL rollouts, rather than providing an absolute measure of static semantic complexity\.

## References

- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.Advances in neural information processing systems30\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf)Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Cuturi \(2013\)M\. CuturiSinkhorn distances: lightspeed computation of optimal transport\.InProceedings of the 27th International Conference on Neural Information Processing Systems \- Volume 2,NIPS’13,Red Hook, NY, USA,pp\. 2292–2300\.Cited by:[5th item](https://arxiv.org/html/2608.21863#A1.I2.i5.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px3.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Ding, H\. Xin, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Wang, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, S\. Ye, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Zhao, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. ZhangDeepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.External Links:[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Donget al\.\(2025\)G\. Dong, Y\. Chen, X\. Li, J\. Jin, H\. Qian, Y\. Zhu, H\. Mao, G\. Zhou, Z\. Dou, and J\. WenTool\-star: empowering llm\-brained multi\-tool reasoner via reinforcement learning\.arXiv preprint arXiv:2505\.16410\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Friederet al\.\(2023\)S\. Frieder, L\. Pinchetti, C\. Chevalier, R\. Griffiths, T\. Salvatori, T\. Lukasiewicz, P\. Petersen, and J\. BernerMathematical capabilities of chatgpt\.Advances in neural information processing systems36,pp\. 27699–27744\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Gaoet al\.\(2026\)H\. Gao, zhenyu zhang, L\. Pang, F\. Guo, douhongjian, G\. Lv, S\. Liu, T\. Gao, H\. Shen, and X\. ChengDIVA\-GRPO: enhancing multimodal reasoning through difficulty\-adaptive variant advantage\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qKXYEg00eH)Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Gouet al\.\(2024\)Z\. Gou, Z\. Shao, Y\. Gong, yelong shen, Y\. Yang, M\. Huang, N\. Duan, and W\. ChenToRA: a tool\-integrated reasoning agent for mathematical problem solving\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Ep0TtjVoap)Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Gunjalet al\.\(2025\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.InNeurIPS 2025 Workshop on Efficient Reasoning,External Links:[Link](https://openreview.net/forum?id=21UFlJrmS2)Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Heckelet al\.\(2026\)R\. Heckel, M\. Soltanolkotabi, and C\. ThramboulidisAsymmetric prompt weighting for reinforcement learning with verifiable rewards\.arXiv preprint arXiv:2602\.11128\.Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Huanget al\.\(2024\)X\. Huang, W\. Liu, X\. Chen, X\. Wang, H\. Wang, D\. Lian, Y\. Wang, R\. Tang, and E\. ChenUnderstanding the planning of llm agents: a survey\.arXiv preprint arXiv:2402\.02716\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Huanget al\.\(2025\)Z\. Huang, Y\. Zhuang, G\. Lu, Z\. Qin, H\. Xu, T\. Zhao, R\. Peng, J\. Hu, Z\. Shen, X\. Hu,et al\.Reinforcement learning with rubric anchors\.arXiv preprint arXiv:2508\.12790\.Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Jinet al\.\(2025\)B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. O\. Arik, D\. Wang, H\. Zamani, and J\. HanSearch\-r1: training LLMs to reason and leverage search engines with reinforcement learning\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Rwhi91ideu)Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Kuhn \(1955\)H\. W\. KuhnThe hungarian method for the assignment problem\.Naval Research Logistics Quarterly2\(1\-2\),pp\. 83–97\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/nav.3800020109),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800020109),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1002/nav\.3800020109Cited by:[5th item](https://arxiv.org/html/2608.21863#A1.I2.i5.p1.1),[§3\.3](https://arxiv.org/html/2608.21863#S3.SS3.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px3.p1.1)\.
- Kuhn \(1956\)H\. W\. KuhnVariants of the hungarian method for assignment problems\.Naval Research Logistics Quarterly3\(4\),pp\. 253–258\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/nav.3800030404),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800030404),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1002/nav\.3800030404Cited by:[§3\.3](https://arxiv.org/html/2608.21863#S3.SS3.SSS0.Px1.p2.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,External Links:[Link](https://doi.org/10.1145/3600006.3613165)Cited by:[§A\.5](https://arxiv.org/html/2608.21863#A1.SS5.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px4.p1.1)\.
- Liet al\.\(2025a\)X\. Li, G\. Dong, J\. Jin, Y\. Zhang, Y\. Zhou, Y\. Zhu, P\. Zhang, and Z\. DouSearch\-o1: agentic search\-enhanced large reasoning models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 5420–5438\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Liet al\.\(2026\)X\. Li, W\. Jiao, J\. Jin, G\. Dong, J\. Jin, Y\. Wang, H\. Wang, Y\. Zhu, J\. Wen, Y\. Lu, and Z\. DouDeepAgent: a general reasoning agent with scalable toolsets\.InProceedings of the ACM Web Conference 2026,WWW ’26,New York, NY, USA,pp\. 2219–2230\.External Links:ISBN 9798400723070,[Link](https://doi.org/10.1145/3774904.3792460),[Document](https://dx.doi.org/10.1145/3774904.3792460)Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Liet al\.\(2025b\)X\. Li, H\. Zou, and P\. LiuTorl: scaling tool\-integrated rl\.arXiv preprint arXiv:2503\.23383\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1),[§1](https://arxiv.org/html/2608.21863#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Lin and Xu \(2025\)H\. Lin and Z\. XuUnderstanding tool\-integrated reasoning\.arXiv preprint arXiv:2508\.19201\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Luoet al\.\(2025\)J\. Luo, W\. Zhang, Y\. Yuan, Y\. Zhao, J\. Yang, Y\. Gu, B\. Wu, B\. Chen, Z\. Qiao, Q\. Long, R\. Tu, X\. Luo, W\. Ju, Z\. Xiao, Y\. Wang, M\. Xiao, C\. Liu, J\. Yuan, S\. Zhang, Y\. Jin, F\. Zhang, X\. Wu, H\. Zhao, D\. Tao, P\. S\. Yu, and M\. ZhangLarge language model agent: a survey on methodology, applications and challenges\.arXiv preprint arXiv:2503\.21460\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Mastermanet al\.\(2024\)T\. Masterman, S\. Besen, M\. Sawtell, and A\. ChaoThe landscape of emerging ai agent architectures for reasoning, planning, and tool calling: a survey\.arXiv preprint arXiv:2404\.11584\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- OpenAIet al\.\(2023\)OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, J\. Belgum, I\. Bello, J\. Berdine, G\. Bernadett\-Shapiro, C\. Berner, L\. Bogdonoff, O\. Boiko, M\. Boyd, A\. Brakman, G\. Brockman, T\. Brooks, M\. Brundage, K\. Button, T\. Cai, R\. Campbell, A\. Cann, B\. Carey, C\. Carlson, R\. Carmichael, B\. Chan, C\. Chang, F\. Chantzis, D\. Chen, S\. Chen, R\. Chen, J\. Chen, M\. Chen, B\. Chess, C\. Cho, C\. Chu, H\. W\. Chung, D\. Cummings, J\. Currier, Y\. Dai, C\. Decareaux, T\. Degry, N\. Deutsch, D\. Deville, A\. Dhar, D\. Dohan, S\. Dowling, S\. Dunning, A\. Ecoffet, A\. Eleti, T\. Eloundou, D\. Farhi, L\. Fedus, N\. Felix, S\. P\. Fishman, J\. Forte, I\. Fulford, L\. Gao, E\. Georges, C\. Gibson, V\. Goel, T\. Gogineni, G\. Goh, R\. Gontijo\-Lopes, J\. Gordon, M\. Grafstein, S\. Gray, R\. Greene, J\. Gross, S\. S\. Gu, Y\. Guo, C\. Hallacy, J\. Han, J\. Harris, Y\. He, M\. Heaton, J\. Heidecke, C\. Hesse, A\. Hickey, W\. Hickey, P\. Hoeschele, B\. Houghton, K\. Hsu, S\. Hu, X\. Hu, J\. Huizinga, S\. Jain, S\. Jain, J\. Jang, A\. Jiang, R\. Jiang, H\. Jin, D\. Jin, S\. Jomoto, B\. Jonn, H\. Jun, T\. Kaftan, Ł\. Kaiser, A\. Kamali, I\. Kanitscheider, N\. S\. Keskar, T\. Khan, L\. Kilpatrick, J\. W\. Kim, C\. Kim, Y\. Kim, J\. H\. Kirchner, J\. Kiros, M\. Knight, D\. Kokotajlo, Ł\. Kondraciuk, A\. Kondrich, A\. Konstantinidis, K\. Kosic, G\. Krueger, V\. Kuo, M\. Lampe, I\. Lan, T\. Lee, J\. Leike, J\. Leung, D\. Levy, C\. M\. Li, R\. Lim, M\. Lin, S\. Lin, M\. Litwin, T\. Lopez, R\. Lowe, P\. Lue, A\. Makanju, K\. Malfacini, S\. Manning, T\. Markov, Y\. Markovski, B\. Martin, K\. Mayer, A\. Mayne, B\. McGrew, S\. M\. McKinney, C\. McLeavey, P\. McMillan, J\. McNeil, D\. Medina, A\. Mehta, J\. Menick, L\. Metz, A\. Mishchenko, P\. Mishkin, V\. Monaco, E\. Morikawa, D\. Mossing, T\. Mu, M\. Murati, O\. Murk, D\. Mély, A\. Nair, R\. Nakano, R\. Nayak, A\. Neelakantan, R\. Ngo, H\. Noh, L\. Ouyang, C\. O’Keefe, J\. Pachocki, A\. Paino, J\. Palermo, A\. Pantuliano, G\. Parascandolo, J\. Parish, E\. Parparita, A\. Passos, M\. Pavlov, A\. Peng, A\. Perelman, F\. de Avila Belbute Peres, M\. Petrov, H\. P\. de Oliveira Pinto, Michael, Pokorny, M\. Pokrass, V\. H\. Pong, T\. Powell, A\. Power, B\. Power, E\. Proehl, R\. Puri, A\. Radford, J\. Rae, A\. Ramesh, C\. Raymond, F\. Real, K\. Rimbach, C\. Ross, B\. Rotsted, H\. Roussez, N\. Ryder, M\. Saltarelli, T\. Sanders, S\. Santurkar, G\. Sastry, H\. Schmidt, D\. Schnurr, J\. Schulman, D\. Selsam, K\. Sheppard, T\. Sherbakov, J\. Shieh, S\. Shoker, P\. Shyam, S\. Sidor, E\. Sigler, M\. Simens, J\. Sitkin, K\. Slama, I\. Sohl, B\. Sokolowsky, Y\. Song, N\. Staudacher, F\. P\. Such, N\. Summers, I\. Sutskever, J\. Tang, N\. Tezak, M\. B\. Thompson, P\. Tillet, A\. Tootoonchian, E\. Tseng, P\. Tuggle, N\. Turley, J\. Tworek, J\. F\. C\. Uribe, A\. Vallone, A\. Vijayvergiya, C\. Voss, C\. Wainwright, J\. J\. Wang, A\. Wang, B\. Wang, J\. Ward, J\. Wei, C\. Weinmann, A\. Welihinda, P\. Welinder, J\. Weng, L\. Weng, M\. Wiethoff, D\. Willner, C\. Winter, S\. Wolrich, H\. Wong, L\. Workman, S\. Wu, J\. Wu, M\. Wu, K\. Xiao, T\. Xu, S\. Yoo, K\. Yu, Q\. Yuan, W\. Zaremba, R\. Zellers, C\. Zhang, M\. Zhang, S\. Zhao, T\. Zheng, J\. Zhuang, W\. Zhuk, and B\. ZophGpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.External Links:[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=2GmDdhBdDk)Cited by:[3rd item](https://arxiv.org/html/2608.21863#A1.I1.i3.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px2.p1.1)\.
- Patilet al\.\(2024\)S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. GonzalezGorilla: large language model connected with massive apis\.Advances in Neural Information Processing Systems37,pp\. 126544–126565\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Qianet al\.\(2025\)C\. Qian, E\. C\. Acikgoz, Q\. He, H\. Wang, X\. Chen, D\. Hakkani\-Tür, G\. Tur, and H\. JiToolrl: reward is all tool learning needs\.arXiv preprint arXiv:2504\.13958\.Cited by:[3rd item](https://arxiv.org/html/2608.21863#A1.I2.i3.p1.1),[§A\.3](https://arxiv.org/html/2608.21863#A1.SS3.p1.1),[§1](https://arxiv.org/html/2608.21863#S1.p1.1),[§1](https://arxiv.org/html/2608.21863#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2608.21863#S3.SS3.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px3.p1.1)\.
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, dahai li, Z\. Liu, and M\. SunToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dHng2O0Jjr)Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Quet al\.\(2026\)C\. Qu, S\. Dai, H\. Cai, J\. Xu, S\. Wang, and D\. YinMatchTIR: fine\-grained supervision for tool\-integrated reasoning via bipartite matching\.arXiv preprint arXiv:2601\.10712\.Cited by:[5th item](https://arxiv.org/html/2608.21863#A1.I2.i5.p1.1),[§A\.3](https://arxiv.org/html/2608.21863#A1.SS3.p1.1),[§1](https://arxiv.org/html/2608.21863#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2608.21863#S3.SS3.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px3.p1.1)\.
- Quet al\.\(2025\)C\. Qu, S\. Dai, X\. Wei, H\. Cai, S\. Wang, D\. Yin, J\. Xu, and J\. WenTool learning with large language models: a survey\.Frontiers of Computer Science19\(8\),pp\. 198343\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[2nd item](https://arxiv.org/html/2608.21863#A1.I2.i2.p1.1),[§1](https://arxiv.org/html/2608.21863#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px3.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.External Links:[Link](https://doi.org/10.1145/3689031.3696075)Cited by:[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px4.p1.1)\.
- Singhet al\.\(2025\)J\. Singh, Y\. Pandya, P\. Vajreshwari, R\. Magazine, and A\. NambiAgentic reasoning and tool integration for LLMs via reinforcement learning\.InFirst Workshop on Foundations of Reasoning in Language Models,External Links:[Link](https://openreview.net/forum?id=0BgYVrmz0F)Cited by:[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Tanget al\.\(2023\)Q\. Tang, Z\. Deng, H\. Lin, X\. Han, Q\. Liang, B\. Cao, and L\. SunToolalpaca: generalized tool learning for language models with 3000 simulated cases\.arXiv preprint arXiv:2306\.05301\.Cited by:[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Uesatoet al\.\(2022\)J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. HigginsSolving math word problems with process\-and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. WenA survey on large language model based autonomous agents\.Front\. Comput\. Sci\.18\(6\)\.External Links:ISSN 2095\-2228,[Link](https://doi.org/10.1007/s11704-024-40231-1),[Document](https://dx.doi.org/10.1007/s11704-024-40231-1)Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Wanget al\.\(2025\)W\. Wang, Z\. Gao, L\. Chen, Z\. Chen, J\. Zhu, X\. Zhao, Y\. Liu, Y\. Cao, S\. Ye, X\. Zhu, L\. Lu, H\. Duan, Y\. Qiao, J\. Dai, and W\. WangVisualprm: an effective process reward model for multimodal reasoning\.arXiv preprint arXiv:2503\.10291\.Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Xuet al\.\(2025\)W\. Xu, C\. Huang, S\. Gao, and S\. ShangLLM\-based agents for tool learning: a survey\.Data Science and Engineering,pp\. 1–31\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p1.1)\.
- Xueet al\.\(2025\)Z\. Xue, L\. Zheng, Q\. Liu, Y\. Li, X\. Zheng, Z\. Ma, and B\. AnSimpletir: end\-to\-end reinforcement learning for multi\-turn tool\-integrated reasoning\.arXiv preprint arXiv:2509\.02479\.Cited by:[§1](https://arxiv.org/html/2608.21863#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[1st item](https://arxiv.org/html/2608.21863#A1.I2.i1.p1.1),[§1](https://arxiv.org/html/2608.21863#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px4.p1.1)\.
- Yeet al\.\(2025a\)J\. Ye, Z\. Du, X\. Yao, W\. Lin, Y\. Xu, Z\. Chen, Z\. Wang, S\. Zhu, Z\. Xi, S\. Yuan, T\. Gui, Q\. Zhang, X\. Huang, and J\. ChenToolHop: a query\-driven benchmark for evaluating large language models in multi\-hop tool use\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 2995–3021\.External Links:[Link](https://aclanthology.org/2025.acl-long.150/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.150),ISBN 979\-8\-89176\-251\-0Cited by:[2nd item](https://arxiv.org/html/2608.21863#A1.I1.i2.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px2.p1.1)\.
- Yeet al\.\(2025b\)J\. Ye, C\. Jiang, Z\. Du, Y\. Xu, X\. Yao, Z\. Xi, X\. Fan, Q\. Zhang, X\. Huang, and J\. ChenFeedback\-driven tool\-use improvements in large language models via automated build environments\.arXiv preprint arXiv:2508\.08791\.Cited by:[1st item](https://arxiv.org/html/2608.21863#A1.I1.i1.p1.1),[4th item](https://arxiv.org/html/2608.21863#A1.I2.i4.p1.1),[§1](https://arxiv.org/html/2608.21863#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.21863#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.21863#S4.SS1.SSS0.Px3.p1.1)\.
- Zhang and Zuo \(2025\)J\. Zhang and C\. ZuoGrpo\-lead: a difficulty\-aware reinforcement learning approach for concise mathematical reasoning in language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 5642–5665\.Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, C\. Zheng, Y\. Wu, B\. Zhang, R\. Lin, B\. Yu, D\. Liu, J\. Zhou, and J\. LinThe lessons of developing process reward models in mathematical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 10495–10516\.Cited by:[§2\.2](https://arxiv.org/html/2608.21863#S2.SS2.p1.1)\.
- Zhaoet al\.\(2023\)Y\. Zhao, A\. Gu, R\. Varma, L\. Luo, C\. Huang, M\. Xu, L\. Wright, H\. Shojanazeri, M\. Ott, S\. Shleifer, A\. Desmaison, C\. Balioglu, P\. Damania, B\. Nguyen, G\. Chauhan, Y\. Hao, A\. Mathews, and S\. LiPyTorch fsdp: experiences on scaling fully sharded data parallel\.Proc\. VLDB Endow\.16\(12\),pp\. 3848–3860\.External Links:ISSN 2150\-8097,[Link](https://doi.org/10.14778/3611540.3611569),[Document](https://dx.doi.org/10.14778/3611540.3611569)Cited by:[§A\.5](https://arxiv.org/html/2608.21863#A1.SS5.SSS0.Px1.p1.1)\.

## Appendix AImplementation Details

### A\.1Datasets and Evaluation Metrics

#### Datasets\.

The statistics of the training and evaluation datasets are shown in[Table5](https://arxiv.org/html/2608.21863#A1.T5)\.

Table 5:Dataset statistics
#### Evaluation Metrics\.

We evaluate model performance using metrics of each benchmark\.

- •FTRL[43](https://arxiv.org/html/2608.21863#bib.bib28)\. LetNg​e​nN\_\{gen\}denote the total number of tool calls generated by the model,Ng​tN\_\{gt\}denote the total number of required tool calls in the ground\-truth trajectory, andNm​a​t​c​hN\_\{match\}denote the number of successfully matched tool calls\. The metrics of FTRL are defined as follows: - –Solve\-Pmeasures the precision of tool invocations\. Solve\-P=\{Nm​a​t​c​hNg​e​nif​Ng​e​n\>01\.0if​Ng​e​n=0\\text\{Solve\-P\}=\\begin\{cases\}\\frac\{N\_\{match\}\}\{N\_\{gen\}\}&\\text\{if \}N\_\{gen\}\>0\\\\ 1\.0&\\text\{if \}N\_\{gen\}=0\\end\{cases\}\(18\) - –Solve\-Revaluates the completeness of the task execution\. Solve\-R=Nm​a​t​c​hNg​t\\text\{Solve\-R\}=\\frac\{N\_\{match\}\}\{N\_\{gt\}\}\(19\) - –Solve\-F1is the harmonic mean of Solve\-P and Solve\-R\. Solve\-F1=2⋅Nm​a​t​c​hNg​e​n\+Ng​t\\text\{Solve\-F1\}=\\frac\{2\\cdot N\_\{match\}\}\{N\_\{gen\}\+N\_\{gt\}\}\(20\)
- •ToolHop[42](https://arxiv.org/html/2608.21863#bib.bib44)\. The metric of ToolHop isAnswer Correctness \(AC\)\. Assume that for a test instance, the standard answer isaaand the model’s final response isoo\. The performance is evaluated as: AC=\{1if​a∈o0otherwise\.\\text\{AC\}=\\begin\{cases\}1&\\text\{if \}a\\in o\\\\ 0&\\text\{otherwise\}\\end\{cases\}\.\(21\)
- •BFCL[25](https://arxiv.org/html/2608.21863#bib.bib43)\. In BFCL V3, model performance is evaluated across several categories, including Base Multi\-Turn \(Base\), Missing Functions \(MF\), Missing Parameters \(MP\), and Long\-Context Multi\-Turn \(LC\)\.

### A\.2Baselines

We compareHiDiffTIRwith following baselines:

- •Vanilla\. We deploy the backbone Qwen3[41](https://arxiv.org/html/2608.21863#bib.bib3)model using standard generation configurations without any further RL training\. This serves as the lower bound, demonstrating the inherent instruction\-following and tool\-use capabilities of the model\.
- •GRPO[32](https://arxiv.org/html/2608.21863#bib.bib5)\. GRPO serves as the standard trajectory\-level RL baseline\. During training, for a given prompt, it generates a group ofGGtrajectories, evaluates the final outcome of each trajectory to assign a reward\. The advantages are computed by normalizing these rewards within the group and are subsequently broadcast uniformly to all tokens in the trajectory\.
- •ToolRL[27](https://arxiv.org/html/2608.21863#bib.bib20)\. This framework introduces process\-based supervision to TIR\. Rather than relying solely on outcome rewards, ToolRL incorporates intermediate verification rewards such as the correctness of tool names, parameter names, and parameter values\. These intermediate rewards are aggregated into a single trajectory reward, and the resulting advantage is then shared across the entire trajectory during policy optimization\.
- •FTRL[43](https://arxiv.org/html/2608.21863#bib.bib28)\. The method accompanying the FTRL dataset\. It employs a verifiable reward mechanism that evaluates the precision of tool invocations and the completeness of task execution\. These factors are combined to calculate a total reward, which is used to derive a single advantage value broadcast to all tokens in the sequence\.
- •MatchTIR[29](https://arxiv.org/html/2608.21863#bib.bib26)\. The state\-of\-the\-art fine\-grained credit assignment TIR framework that derives dense, turn\-level advantages by aligning generated trajectories with ground\-truth tool\-use sequences\. We evaluate two implementations of its alignment module: \(1\)MatchTIR \(KM\)formulates the alignment as a hard bipartite matching problem\. It uses the Kuhn\-Munkres \(KM\) algorithm[13](https://arxiv.org/html/2608.21863#bib.bib42)to find the optimal one\-to\-one mapping between generated and ground\-truth tool calls based on semantic similarity, distributing rewards strictly to matched pairs\. \(2\)MatchTIR \(OT\)employs Optimal Transport \(OT\)[2](https://arxiv.org/html/2608.21863#bib.bib45)with Sinkhorn solver to create a soft alignment matrix\. This allows for a probabilistic distribution of rewards across multiple generated turns that share semantic overlap with the ground truth\.

Table 6:Training prompt template for the policy model\.Training Prompt Template for the Policy Modelsystem\# ToolsYou may call one or more functions to assist with the user query\.You are provided with function signatures within<tools\></tools\>XML tags:<tools\>\{Tool Descriptions\}</tools\>For each function call, return a json object with function name and arguments within<tool\_call\></tool\_call\>XML tags:<tool\_call\>\{‘‘name’’: <function\-name\>, ‘‘arguments’’: <args\-json\-object\>\}</tool\_call\>userPlease call given tools to answer the question\. Please note that all your information must be obtained by calling tools and not by answering the question directly\.If the call fails, you need to try to correct it and continue until you arrive at an answer\. Only output the final answer \(in words, numbers or phrase\) inside the<answer\></answer\>tag, without any explanations or extra information\.Question:\{question\}assistant
### A\.3Tool Call Similarity Computation

For reward computation, a matching matrixMi∈ℝm×nM\_\{i\}\\in\\mathbb\{R\}^\{m\\times n\}is constructed for each trajectoryτ\(i\)\\tau^\{\(i\)\}, where each entryMi​\[u,v\]M\_\{i\}\[u,v\]measures the alignment between a predicted tool callci​u∈Cic\_\{iu\}\\in C\_\{i\}and a ground\-truth tool callci​v∗∈Ci∗c^\{\*\}\_\{iv\}\\in C^\{\*\}\_\{i\}\. The similarity scoreMi​\[u,v\]M\_\{i\}\[u,v\]consists of three components below, following previous works[27](https://arxiv.org/html/2608.21863#bib.bib20);[29](https://arxiv.org/html/2608.21863#bib.bib26)\.

#### Tool Name Matching\.

Letci​unamec\_\{iu\}^\{\\mathrm\{name\}\}andci​vname∗c\_\{iv\}^\{\\mathrm\{name\}\*\}denote the tool names of the predicted and ground\-truth calls\. The tool name score is

sname=𝕀\(ci​uname=ci​vname∗\)∈\{0,1\},s\_\{\\mathrm\{name\}\}=\\mathbb\{I\}\(c\_\{iu\}^\{\\mathrm\{name\}\}=c\_\{iv\}^\{\\mathrm\{name\}\*\}\)\\in\\\{0,1\\\},\(22\)where𝕀⁡\[⋅\]\\mathbb\{I\}\[\\cdot\]is an indicator function\.

#### Parameter Name Matching\.

If the tool names match, the parameter name similarity is computed as the Jaccard similarity over the sets of parameter namesPci​uP\_\{c\_\{iu\}\}andPci​v∗P\_\{c^\{\*\}\_\{iv\}\}:

sparam=\|Pci​u∩Pci​v∗\|\|Pci​u∪Pci​v∗\|∈\[0,1\]\.s\_\{\\mathrm\{param\}\}=\\frac\{\|P\_\{c\_\{iu\}\}\\cap P\_\{c^\{\*\}\_\{iv\}\}\|\}\{\|P\_\{c\_\{iu\}\}\\cup P\_\{c^\{\*\}\_\{iv\}\}\|\}\\in\[0,1\]\.\(23\)

#### Parameter Value Matching\.

Finally, the correctness of parameter values is assessed for each ground\-truth parameterk∈Pci​v∗k\\in P\_\{c^\{\*\}\_\{iv\}\}:

svalue=∑k∈Pci​v∗𝕀⁡\(ci​u​\[k\]=ci​v∗​\[k\]\)∈\[0,\|Pci​v∗\|\]\.s\_\{\\mathrm\{value\}\}=\\sum\_\{k\\in P\_\{c^\{\*\}\_\{iv\}\}\}\\mathbb\{I\}\(c\_\{iu\}\[k\]=c^\{\*\}\_\{iv\}\[k\]\)\\in\[0,\|P\_\{c^\{\*\}\_\{iv\}\}\|\]\.\(24\)
The three components are combined and normalized to produce the final similarity score:

Mi​\[u,v\]=sname⋅sname\+sparam\+svalue2\+\|Pci​v∗\|∈\[0,1\]\.M\_\{i\}\[u,v\]=s\_\{\\mathrm\{name\}\}\\cdot\\frac\{s\_\{\\mathrm\{name\}\}\+s\_\{\\mathrm\{param\}\}\+s\_\{\\mathrm\{value\}\}\}\{2\+\|P\_\{c^\{\*\}\_\{iv\}\}\|\}\\in\[0,1\]\.\(25\)

### A\.4Training Prompt Template

The training prompt template for the policy model is shown in Table[6](https://arxiv.org/html/2608.21863#A1.T6)\.

### A\.5Training Details

To facilitate reproducibility, we provide the complete set of hyperparameters and infrastructure configurations used for trainingHiDiffTIR\.

#### Infrastructure and Hardware\.

All models are trained on a single compute node equipped with 8 NVIDIA H20 GPUs\. We leverage Fully Sharded Data Parallel \(FSDP\)[46](https://arxiv.org/html/2608.21863#bib.bib46)to distribute the policy model weights, avoiding parameter and optimizer offloading to maximize throughput\. For rollout generation, we utilize vLLM[15](https://arxiv.org/html/2608.21863#bib.bib9)with a GPU memory utilization threshold of 0\.7\.

#### Data and Generation Settings\.

We format the prompts using the standard Qwen3 system style and explicitly enable the generation of reasoning trajectories\. To handle long trajectories, we set the maximum model length to 32,768 tokens and the maximum response length to 8,192 tokens\. During the rollout phase, we sample 16 responses per prompt \(G=16G=16\) with a generation temperature of 1\.0 and a maximum turn of 10\.

#### Optimization and Loss Formulation\.

The policy is optimized over 5 total epochs\. We use a global training batch size of 256 prompts, broken down into mini\-batch sizes of 32\. The actor learning rate is fixed at10−610^\{\-6\}without a critic warmup phase\. The entropy coefficient is set to 0\.001, and the KL penalty coefficient is set to 0\.001\.

## Appendix BAdditional Experiments

### B\.1Group Size Analysis

Table 7:Performance comparison ofHiDiffTIRon Qwen3\-4B with different group sizesGG\.The group sizeGGis a critical hyperparameter in GRPO\-based frameworks, as it determines the sample size used to estimate the baseline and normalize rewards within each group\. InHiDiffTIR,GGfurther influences the robustness of the empirical difficulty estimation used for both trajectory\-level and turn\-level credit assignment\. To investigate this impact, we conduct experiments on Qwen3\-4B withα=0\.7\\alpha=0\.7across three different group sizes: 4, 8, and 16\. The experimental results are summarized in[Table7](https://arxiv.org/html/2608.21863#A2.T7)\. We observe distinct performance trends between the In\-Domain \(ID\) and Out\-Of\-Domain \(OOD\) metrics\.

For the ID FTRL dataset, performance consistently improves as the group size increases\. The ID average score rises steadily from 33\.03 \(G=4G=4\) to 39\.48 \(G=16G=16\)\. This trend indicates that our hierarchical difficulty\-aware mechanisms benefit from larger group sizes, which provide more statistically robust estimations of group\-level statistics within each training iteration\. Accurately quantifying the relative difficulty of trajectories and turns allows the model to better identify and optimize informative reasoning steps in a difficulty\-aware manner, thus improving ID task success rates\.

Conversely, for OOD scenarios, the relationship is not strictly monotonic\. While the optimal OOD average score of 41\.79 is achieved atG=16G=16,G=8G=8actually shows a slight regression \(39\.77\) compared toG=4G=4\(40\.19\)\. This volatility stems from the intrinsic mechanism of our difficulty\-aware framework\. By design,HiDiffTIRutilizes group rollouts to intensively optimize the policy toward mastering seen tools, particularly focusing on resolving high\-difficulty tool\-calling bottlenecks\. While this targeted credit assignment significantly enhances the model’s proficiency and precision on seen tool\-use patterns, it can lead to behavioral variances when the model encounters entirely unseen tools in OOD environments, as their underlying API structures and difficulty characteristics differ from the training set\. Nevertheless,G=16G=16still yields a well\-suited balance, achieving the highest performance on both in\-domain tasks and overall OOD generalization\.

### B\.2Case Study

To intuitively demonstrate the superiority ofHiDiffTIR, we present a qualitative case study involving a complex 5\-hop reasoning query\. As shown in[Table8](https://arxiv.org/html/2608.21863#A2.T8), the user query requires identifying a specific local handicraft through a chain of five implicit entities\. We compare our approach against the untrained Qwen3\-4B backbone and MatchTIR \(KM\), a strong baseline built on the same backbone, with results reported in Tables[9](https://arxiv.org/html/2608.21863#A2.T9)\-[11](https://arxiv.org/html/2608.21863#A2.T11), respectively\. As illustrated in the interaction trajectories,HiDiffTIRexhibits the following distinct advantages\.

Mitigation of Parameter Hallucination and Error Loops\.Both baseline models exhibit a tendency to hallucinate tool parameters during the initial reasoning phase\. As shown in[Table10](https://arxiv.org/html/2608.21863#A2.T10)and[Table11](https://arxiv.org/html/2608.21863#A2.T11), when calling the initialfamous\_peak\_identifiertool, both Qwen3\-4B and MatchTIR incorrectly append unnecessary geographic constraints, hallucinating “Alps” or “Himalayas” instead of passing the required empty arguments\. While MatchTIR manages to correct itself after wasting two turns, the untrained Qwen3\-4B model lacks this reflective capability, entering a fatal trial\-and\-error loop that continuously guesses random regions until it exhausts the maximum turn limit\. In contrast,HiDiffTIRcompletely suppresses these hallucinated priors, immediately generating the precise arguments required to navigate the tool logic\.

Calibration of Overconfidence and Adaptation to Environmental Feedback\.A further striking observation from the trajectories is the backbone model’s overconfidence in its parametric knowledge\. When facing consecutive empty responses from the tools, the untrained Qwen3\-4B exhibits severe resistance to self\-correction\. Instead of rectifying its over\-specified parameters, the model stubbornly suspects that the tool itself is flawed \(e\.g\., explicitly assuming in Turn 6 and Turn 9 that “the tool’s data” or “the tool’s database” is incomplete\)\. This overconfidence traps the agent, preventing it from utilizing negative environmental feedback\. While the RL\-trained MatchTIR largely mitigates this extreme overconfidence and demonstrates the capacity to adjust based on environmental feedback, it still exhibits a lingering over\-reliance on its internal parametric knowledge, frequently defaulting to hallucinated assumptions during exploration\. Moving beyond this behavior,HiDiffTIReffectively neutralizes this stubbornness, teaching the LLM agent to respect and adapt to strict tool constraints rather than blindly persisting with its hallucinated assumptions\.

Enhanced Reasoning Efficiency and Minimal Redundancy\.Beyond hallucination and overconfidence, baseline models also struggle with inefficient and redundant exploration\. As observed in the trajectories, whenever the untrained Qwen3\-4B faces an obstacle, it continuously appends arbitrary parameters \(e\.g\.,include\_protectedin Turn 8 andenvironmental\_certificationin Turn 9\) in futile attempts to force a result\. Even the RL\-trained MatchTIR, despite eventually reaching the correct answer, exhibits similar inefficiency by repeatedly over\-specifying non\-essential arguments \(e\.g\., guessingradiusin Turn 4 and extra boolean constraints in Turn 7\)\. In contrast,HiDiffTIRlearns a highly compact reasoning policy with minimal redundancy\. In this specific case, it executes the entire task flawlessly in exactly five optimal tool\-calling turns, achieving absolute zero redundancy\. This highlights the core advantage of our hierarchical difficulty\-aware policy optimization framework\. By accurately amplifying the learning signals for high\-difficulty bottleneck steps, the framework implicitly penalizes superfluous exploratory actions, yielding a much more efficient and precise reasoning trajectory compared to uniform or heuristic\-based credit assignments\.

Table 8:The user query and ground truth answer of the case study\.User Query:What is the local handicraft of the town where the woodworker who carves sculptures using the wood from the forest near the mountain with a famous peak?Ground Truth Tool Calls:1\.\{‘‘name’’: ‘‘famous\_peak\_identifier’’, ‘‘arguments’’: \{\}\}2\.\{‘‘name’’: ‘‘nearby\_forest\_locator’’, ‘‘arguments’’: \{‘‘landmark’’: ‘‘Mount Everest’’\}\}3\.\{‘‘name’’: ‘‘wood\_sourcing\_locator’’, ‘‘arguments’’: \{‘‘forest\_name’’: ‘‘Langtang National Park Forest’’, ‘‘product\_type’’: ‘‘sculptures’’\}\}4\.\{‘‘name’’: ‘‘individual\_location\_finder’’, ‘‘arguments’’: \{‘‘individual\_name’’: ‘‘Ramesh Thapa’’\}\}5\.\{‘‘name’’: ‘‘local\_handicraft\_identifier’’, ‘‘arguments’’: \{‘‘location’’: ‘‘Pokhara’’\}\}Ground Truth Answer:Wooden masksTable 9:Interaction trajectory ofHiDiffTIR\. The model conducts precise TIR by successfully chaining five sequential tool calls without any hallucinations or redundant trial\-and\-error\.Table 10:Interaction trajectory of Qwen3\-4B\. The model hallucinates tool parameters, enters a trial\-and\-error loop, and ultimately fails to complete the reasoning chain within the turn limit\.Table 11:Interaction trajectory of MatchTIR \(KM\)\. Although the model successfully reasons out the correct target answer, it exhibits redundancy in the early reasoning stages due to over\-specification of geographic constraints\.