先评估再行动：面向长程工具使用智能体的比较式价值估计

# Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents
Source: [https://arxiv.org/html/2610.02330](https://arxiv.org/html/2610.02330)
Yu LiAffiliation:School of Computer Science and Engineering, Southeast University, Nanjing, ChinaEmail:[yuli11@seu\.edu\.cn](mailto:)Xin LiuAffiliation:UniSA STEM, University of South Australia, Adelaide, AustraliaShengtian YangAffiliation:School of Computer Science and Engineering, Southeast University, Nanjing, ChinaGuangfeng CaiAffiliation:School of Computer Science and Engineering, Southeast University, Nanjing, ChinaLei Feng††thanks:Corresponding author\.Affiliation:School of Computer Science and Engineering, Southeast University, Nanjing, China

###### Abstract

Large language models \(LLMs\) rely on long\-horizon tool invocation sequences for complex tasks, where each invocation can alter the task state and condition subsequent decisions\. In long\-horizon tool use, final\-outcome rewards provide weak credit assignment over long interaction traces\. Step\-level rewards can offer more targeted feedback, but obtaining reliable step supervision often requires human or LLM judgment, or additional rollouts to estimate the downstream effect of an intermediate decision\. In this paper, we argue that effective tool\-use agents should estimate the long\-horizon value of a possible next tool invocation before executing it\. This objective requires comparative supervision over alternative invocations under the same context, while logged trajectories only contain the invocation that was actually taken\. Therefore, we proposeComparativeInference forTool\-useAgents \(CITA\)\. CITA trains aComparativeInferenceModel \(CIM\) from paired signals that combine observed tool behavior, scalable supervision from a Bayesian tool\-graph simulator, and semantic judgments from LLM\-based comparison\. The resulting CIM learns to estimate how likely a possible next tool invocation is to support final task success under the current context\. Across three tool\-use benchmarks and multiple backbone LLMs, CITA consistently improves Tool F1 and task success\. Additional analysis shows that CIM learns accurate step\-level value estimates for comparative tool choices\.

## 1Introduction

Large language models \(LLMs\) extend their capabilities through external tools, including search engines\[[1](https://arxiv.org/html/2610.02330#bib.bib1),[2](https://arxiv.org/html/2610.02330#bib.bib57),[3](https://arxiv.org/html/2610.02330#bib.bib52)\], calculators\[[4](https://arxiv.org/html/2610.02330#bib.bib2),[5](https://arxiv.org/html/2610.02330#bib.bib55),[6](https://arxiv.org/html/2610.02330#bib.bib54),[7](https://arxiv.org/html/2610.02330#bib.bib58)\], code interpreters\[[8](https://arxiv.org/html/2610.02330#bib.bib3)\], and application programming interfaces \(APIs\)\[[9](https://arxiv.org/html/2610.02330#bib.bib4)\]\. These tools extend LLMs beyond text\-only generation and enable agents to interleave reasoning with action in real tool environments\[[10](https://arxiv.org/html/2610.02330#bib.bib5),[11](https://arxiv.org/html/2610.02330#bib.bib6),[12](https://arxiv.org/html/2610.02330#bib.bib7),[13](https://arxiv.org/html/2610.02330#bib.bib8)\]\. As tool use moves from isolated calls to long interaction traces, tool selection becomes a sequential decision problem\[[14](https://arxiv.org/html/2610.02330#bib.bib9),[15](https://arxiv.org/html/2610.02330#bib.bib10),[16](https://arxiv.org/html/2610.02330#bib.bib11),[17](https://arxiv.org/html/2610.02330#bib.bib12)\]\. Each tool invocation updates the task state, and an early choice can enable or block later calls\[[18](https://arxiv.org/html/2610.02330#bib.bib13),[19](https://arxiv.org/html/2610.02330#bib.bib14),[20](https://arxiv.org/html/2610.02330#bib.bib15),[21](https://arxiv.org/html/2610.02330#bib.bib56)\]\. Therefore, the quality of a tool choice cannot be judged only by its local fit to the current instruction\. It also depends on whether the choice supports final task success after the remaining interaction\[[22](https://arxiv.org/html/2610.02330#bib.bib26),[23](https://arxiv.org/html/2610.02330#bib.bib27),[24](https://arxiv.org/html/2610.02330#bib.bib28)\]\.

This long\-horizon dependency makes reinforcement learning \(RL\) difficult\[[25](https://arxiv.org/html/2610.02330#bib.bib16),[26](https://arxiv.org/html/2610.02330#bib.bib17),[27](https://arxiv.org/html/2610.02330#bib.bib18)\]\. Final\-outcome rewards are easy to define, but they provide feedback only after the full trajectory is completed\. For long interaction traces, such rewards give little information about which intermediate tool invocations caused success or failure\[[28](https://arxiv.org/html/2610.02330#bib.bib19),[29](https://arxiv.org/html/2610.02330#bib.bib20)\]\. Step\-level rewards can provide more direct feedback, and process reward models have been studied for improving multi\-step reasoning\[[30](https://arxiv.org/html/2610.02330#bib.bib21),[31](https://arxiv.org/html/2610.02330#bib.bib47)\]\. However, reliable step\-level supervision is difficult to obtain in tool\-use environments\. Human annotation does not scale\[[32](https://arxiv.org/html/2610.02330#bib.bib22),[25](https://arxiv.org/html/2610.02330#bib.bib16),[26](https://arxiv.org/html/2610.02330#bib.bib17)\], and LLM\-based judgments can be unstable\[[33](https://arxiv.org/html/2610.02330#bib.bib23),[34](https://arxiv.org/html/2610.02330#bib.bib24)\]\. Rollout\-based estimation can infer the value of an intermediate decision, but it must continue the interaction from that state and execute additional tool invocations\[[35](https://arxiv.org/html/2610.02330#bib.bib25),[30](https://arxiv.org/html/2610.02330#bib.bib21)\]\. Since each continuation may update the environment state, the resulting signal is costly and noisy\[[28](https://arxiv.org/html/2610.02330#bib.bib19)\]\.

Figure 1:Pilot analysis on long\-horizon tool\-use tasks\. \(a\) Task success drops sharply as the required number of tool invocations increases\. \(b\) Most failed trajectories remain unrecoverable even after several backtracking steps\. \(c\) At many fork points, the model\-preferred tool is not the tool with the higher empirical success rate under the same context\. These patterns motivate comparative value estimation for possible next\-tool choices before execution\.These concerns are visible in our pilot analysis, as shown in Figure[1](https://arxiv.org/html/2610.02330#S1.F1)\. The error accumulation analysis shows that task success drops sharply as the required number of tool invocations increases, indicating that errors accumulate across long interaction traces and make local tool selection unreliable\. The failure recoverability analysis shows that 95% of failed trajectories remain unrecoverable after one step of backtracking, and 76% remain unrecoverable after four steps\. This suggests that final feedback often arrives too late to guide the intermediate decisions that caused failure\. The first two results indicate that a tool\-use agent should evaluate a possible next tool invocation before execution\. The remaining question is whether the agent’s own local preference can serve as this evaluation signal\. Our preference misalignment analysis shows that, at 49% of fork points, the tool ranked higher by the model has a lower empirical success rate than an alternative under the same context\. This mismatch suggests that local preference alone is not sufficient for next\-tool selection\.

These observations shift the objective from following local preference to estimating the downstream value of each possible next tool invocation\. We define this signal as the long\-horizon value under the current context, which measures whether a candidate invocation is likely to support final task success\. Since this value is meaningful for decision making only relative to available choices, the agent must learn to compare alternative invocations under the same context\. A practical obstacle is that logged trajectories only contain the invocation that was actually selected, leaving the values of unselected alternatives unobserved\[[36](https://arxiv.org/html/2610.02330#bib.bib29),[37](https://arxiv.org/html/2610.02330#bib.bib30),[38](https://arxiv.org/html/2610.02330#bib.bib31)\]\. This missing comparison signal motivates a comparative inference model that learns from paired evidence over alternative tool choices\.

To instantiate this idea, we proposeComparativeInference forTool\-useAgents \(CITA\)\. CITA trains aComparativeInferenceModel \(CIM\) to estimate and compare the long\-horizon values of candidate next\-tool invocations under the current context\. Given a tool\-use context and a candidate invocation, CIM predicts whether this choice is likely to support final task success\. To make this comparative value signal learnable, we construct paired training signals that expose alternative choices under related or shared contexts from three complementary sources: \(1\) Real trajectories supply pairs by matching similar decision contexts, comparing different next tools, and using their downstream outcomes as supervision\. \(2\) A Bayesian simulator over the tool transition graph supplies scalable pairs in a sandbox, where synthetic tool success probabilities and transition reliability probabilities are sampled from Bayesian priors\. This source creates controlled fork\-point comparisons without executing real APIs\. \(3\) LLM\-based comparison supplies semantic pairs by comparing the factual tool choice with a plausible alternative under the same context\. These complementary signals train CIM to estimate the long\-horizon value of a possible next tool invocation under the current context\.

CITA uses CIM as both a rollout guide and a training signal\. During policy optimization training, the policy samples a set of possible next tool invocations at each decision step, and CIM selects the call with the highest predicted success value under the current context\. This process produces complete trajectories guided by step\-level value estimates\. After execution, CIM scores the realized tool sequence by estimatingV⁡\(st,at\)V\(s\_\{t\},a\_\{t\}\)at each step, and the aggregated scores form a comparative reward\. This reward is combined with a task\-level matching reward to compute group\-relative advantages for policy optimization\. At inference, CITA reuses the same CIM\-guided selection process without policy updates\. This design enables CIM to guide local tool choices during trajectory construction and provide long\-horizon value feedback during policy learning\.

We evaluate CITA on Toolathlon, TOUCAN, and TRAJECT\-Bench with multiple backbone LLMs\. CITA improves Tool F1 and end\-to\-end task success across benchmarks\. We also evaluate CIM directly and show that it produces accurate step\-level value estimates under comparative tool choices\.

Our contributions are summarized as follows:

- •We design aComparativeInferenceModel \(CIM\) for long\-horizon tool use and train it with a three\-source data construction strategy\. The strategy builds paired signals from real trajectories, a Bayesian tool\-graph simulator, and LLM\-based comparison\.
- •We proposeComparativeInference forTool\-useAgents \(CITA\), a framework that uses CIM to guide policy optimization training\. CIM scores sampled next\-tool invocations during rollout generation and provides dense step\-level comparative rewards after rollout completion\.
- •We show that CITA improves Tool F1 and end\-to\-end task success on three tool\-use benchmarks, and that CIM learns effective step\-level value estimates\.

## 2Related Work

Prompting and Search\-Based Tool Use\.Prompting and search\-based methods improve tool use by guiding LLMs to plan, call tools, and revise decisions based on observations\[[39](https://arxiv.org/html/2610.02330#bib.bib49),[40](https://arxiv.org/html/2610.02330#bib.bib50),[41](https://arxiv.org/html/2610.02330#bib.bib53),[42](https://arxiv.org/html/2610.02330#bib.bib51)\]\. ReAct interleaves reasoning traces with actions\[[10](https://arxiv.org/html/2610.02330#bib.bib5)\], while Toolformer trains models to decide when and how to call APIs through self\-supervised data construction\[[4](https://arxiv.org/html/2610.02330#bib.bib2)\]\. ToolLLM builds ToolBench and uses decision\-tree search to collect tool\-use trajectories over real APIs\[[12](https://arxiv.org/html/2610.02330#bib.bib7)\]\. ToolChain\* formulates API calling as action\-space search and uses A\* search to find efficient tool\-use paths\[[43](https://arxiv.org/html/2610.02330#bib.bib32)\]\. ART automatically composes reasoning and tool\-use programs\[[44](https://arxiv.org/html/2610.02330#bib.bib33)\], ReWOO separates planning from tool execution\[[45](https://arxiv.org/html/2610.02330#bib.bib34)\], Gorilla improves API calling through fine\-tuning and retrieval over API documents\[[9](https://arxiv.org/html/2610.02330#bib.bib4)\], and RestGPT applies coarse\-to\-fine planning to real RESTful APIs\[[46](https://arxiv.org/html/2610.02330#bib.bib35)\]\. Their main limitation lies in scalability to long\-horizon settings: prompting relies on local model decisions, search becomes costly as the trajectory grows, and trace\-based learning is constrained by the coverage of paths\.

Reinforcement Learning and Step\-Level Rewards for Tool Use\.Reinforcement learning methods aim to improve tool\-use policies beyond supervised imitation\. GRPO estimates group\-relative advantages from sampled responses without training a separate critic\[[47](https://arxiv.org/html/2610.02330#bib.bib36)\]\. ToolRL studies reward design for tool selection and tool application\[[26](https://arxiv.org/html/2610.02330#bib.bib17)\]\. StepTool introduces step\-grained reward shaping and optimization for multi\-step tool learning\[[25](https://arxiv.org/html/2610.02330#bib.bib16)\]\. ReTool uses outcome\-driven RL with real\-time code execution to teach models when and how to use tools\[[48](https://arxiv.org/html/2610.02330#bib.bib37)\]\. Tool\-Star combines tool\-integrated data synthesis with multi\-tool self\-critic RL for collaborative tool reasoning\[[27](https://arxiv.org/html/2610.02330#bib.bib18)\]\. AWPO integrates explicit reasoning rewards into a GRPO\-style framework through advantage weighting\[[49](https://arxiv.org/html/2610.02330#bib.bib38)\]\. PORTool builds tree\-like tool\-use rollouts and assigns rewards to intermediate steps for fork\-relative policy optimization\[[50](https://arxiv.org/html/2610.02330#bib.bib39)\]\. These methods show that finer\-grained feedback can improve tool learning\. However, their step\-level signals often depend on rules, annotations, LLM judgments, or rollout estimates, which can be costly or noisy in executable tool environments\.

Predictive and Comparative Planning for Interactive Agents\.Another line of work evaluates possible actions before execution\. DyMo augments function calling with state prediction in stateful tool\-use environments\[[51](https://arxiv.org/html/2610.02330#bib.bib40)\]\. WebDreamer simulates candidate web actions and evaluates the simulated outcomes before acting\[[52](https://arxiv.org/html/2610.02330#bib.bib41)\]\. The World Knowledge Model provides global task knowledge and local dynamic knowledge for agent planning\[[53](https://arxiv.org/html/2610.02330#bib.bib42)\]\. WebCoT studies web\-agent skills such as lookahead, branching, reflection, and rollback\[[54](https://arxiv.org/html/2610.02330#bib.bib43)\]\. LUMINA analyzes long\-horizon interactive agents with an oracle comparative framework\[[55](https://arxiv.org/html/2610.02330#bib.bib44)\]\. These works highlight the benefit of reasoning about an action before it is executed\. CITA follows this direction but focuses on the decision itself: it estimates the long\-horizon value of possible next\-tool invocations under the current context, without requiring full future\-state prediction or oracle skills\.

## 3Method

### 3\.1Problem Formulation

We consider a long\-horizon tool\-use task where an LLM agent solves a user request by executing a sequence of external tool invocations\. Letqqdenote the user query\. A completed trajectory is written asτ=\(q,a1,o1,…,aT,oT,y\)\\tau=\(q,a\_\{1\},o\_\{1\},\\ldots,a\_\{T\},o\_\{T\},y\), whereata\_\{t\}is the tool invocation at steptt,oto\_\{t\}is the tool output, andy∈\{0,1\}y\\in\\\{0,1\\\}indicates whether the trajectory solves the task\. At steptt, the agent observes the current contextst=\(q,a<t,o<t\)s\_\{t\}=\(q,a\_\{<t\},o\_\{<t\}\), selects a possible next tool invocationata\_\{t\}, receivesoto\_\{t\}, and updates the context\.

The goal is to choose tool invocations that maximize final task success\. We define the long\-horizon value of a possible next tool invocation as

V⋆​\(st,a\)=Pr⁡\(y=1∣st,a\),V^\{\\star\}\(s\_\{t\},a\)=\\Pr\(y=1\\mid s\_\{t\},a\),\(1\)which measures how likely the task is to be completed if the agent choosesaaundersts\_\{t\}\. This value is not a one\-step transition score\. It summarizes how the current tool choice affects the remaining interaction and final outcome\.

IfV⋆​\(st,a\)V^\{\\star\}\(s\_\{t\},a\)were available for all possible next tool invocations, the agent could compare alternatives under the same context and choose the one with the highest long\-horizon value\. In logged trajectories, however, only the executed tool invocation is observed, and the values of unselected alternatives are missing\. CITA addresses this challenge by learning an estimatorVϕ​\(st,a\)V\_\{\\phi\}\(s\_\{t\},a\)with a comparative inference Model \(CIM\)\. Since the target value concerns choices that are not jointly observed in a single trajectory, CIM is trained on paired signals instead of isolated trajectory labels\.

### 3\.2Training Signals for CIM

Each paired signal compares two possible next\-tool choices under a comparable context:

x=\(s,af,ac,vf,vc,Δ​v\),Δ​v=vf−vc,x=\(s,a\_\{f\},a\_\{c\},v\_\{f\},v\_\{c\},\\Delta v\),\\qquad\\Delta v=v\_\{f\}\-v\_\{c\},\(2\)wheressis the tool\-use context,afa\_\{f\}andaca\_\{c\}are two alternative next\-tool invocations, andvfv\_\{f\}andvcv\_\{c\}are their estimated long\-horizon success values\. These pairs train CIM to estimate the value of each tool invocation and compare two choices under the same context\.

CITA also builds a tool transition graph from logged trajectories\. Each node is a tool, and each directed edge records an observed transition between two tools\. The graph is used to identify plausible next\-tool choices and to support the Bayesian simulator\.

𝒟obs\\mathcal\{D\}\_\{\\mathrm\{obs\}\}: observed trajectory pairs\.The first source comes from real tool\-use trajectories\. CITA groups logged episodes by task and decision context, where the context includes the prefix tool sequence and compact summaries of recent arguments and outputs\. For each contextssand next toolaa, its empirical value is estimated as the downstream success ratev^obs​\(s,a\)=Nsucc​\(s,a\)/N⁡\(s,a\)\\hat\{v\}\_\{\\mathrm\{obs\}\}\(s,a\)=N\_\{\\mathrm\{succ\}\}\(s,a\)/N\(s,a\)\. For two tools observed under the same context, we setvf=v^obs​\(s,af\)v\_\{f\}=\\hat\{v\}\_\{\\mathrm\{obs\}\}\(s,a\_\{f\}\),vc=v^obs​\(s,ac\)v\_\{c\}=\\hat\{v\}\_\{\\mathrm\{obs\}\}\(s,a\_\{c\}\), andΔ​v=vf−vc\\Delta v=v\_\{f\}\-v\_\{c\}\. This source keeps the training signal close to real tool\-use behavior, but its coverage depends on the diversity of logged trajectories\.

𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}: Bayesian simulator pairs\.The second source comes from a Bayesian simulator over the tool transition graph\. The simulator does not execute real APIs\. It creates synthetic sandbox instances where each tool node receives a sampled success probability and each transition edge receives a sampled reliability probability\. For a tool nodeii, letnin\_\{i\}be its call count andggbe the global success rate of logged trajectories\. For an edgei→ji\\rightarrow j, letni​jn\_\{ij\}be its transition count andqi​j=ni​j/∑j′ni​j′q\_\{ij\}=n\_\{ij\}/\\sum\_\{j^\{\\prime\}\}n\_\{ij^\{\\prime\}\}be its relative frequency among outgoing transitions fromii\. We sample

pi∼Beta⁡\(g​ni\+λ,\(1−g\)​ni\+λ\),ri​j∼Beta⁡\(qi​j​ni​j\+λ,\(1−qi​j\)​ni​j\+λ\),p\_\{i\}\\sim\\mathrm\{Beta\}\(gn\_\{i\}\+\\lambda,\(1\-g\)n\_\{i\}\+\\lambda\),\\qquad r\_\{ij\}\\sim\\mathrm\{Beta\}\(q\_\{ij\}n\_\{ij\}\+\\lambda,\(1\-q\_\{ij\}\)n\_\{ij\}\+\\lambda\),\(3\)whereλ\\lambdais a smoothing constant\. Herepip\_\{i\}is a synthetic success tendency used only inside the sandbox, andri​jr\_\{ij\}is the synthetic reliability of moving from tooliito tooljj\.

For a continuation pathP=\(b0,…,bL\)P=\(b\_\{0\},\\ldots,b\_\{L\}\), its sandbox value is

v^sandbox​\(P\)=∏ℓ=0Lpbℓ​∏ℓ=0L−1rbℓ​bℓ\+1\.\\hat\{v\}\_\{\\mathrm\{sandbox\}\}\(P\)=\\prod\_\{\\ell=0\}^\{L\}p\_\{b\_\{\\ell\}\}\\prod\_\{\\ell=0\}^\{L\-1\}r\_\{b\_\{\\ell\}b\_\{\\ell\+1\}\}\.\(4\)At a fork pointuu, CITA uses the prefix ending atuuas the contextss, selects two successor toolsafa\_\{f\}andaca\_\{c\}, and expands one continuation path for each successor inside the same sandbox\. Let the two paths bePfP\_\{f\}andPcP\_\{c\}\. We setvf=v^sandbox​\(Pf\)v\_\{f\}=\\hat\{v\}\_\{\\mathrm\{sandbox\}\}\(P\_\{f\}\),vc=v^sandbox​\(Pc\)v\_\{c\}=\\hat\{v\}\_\{\\mathrm\{sandbox\}\}\(P\_\{c\}\), andΔ​v=vf−vc\\Delta v=v\_\{f\}\-v\_\{c\}\. Since both paths are evaluated under the same sampled sandbox, the pair compares successor choices while keeping the synthetic environment fixed\.

𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}: LLM\-based comparative pairs\.The third source uses LLM\-based comparison\. For each logged prefixss, CITA keeps the factual next toolafa\_\{f\}and selects a plausible alternativeaca\_\{c\}from valid successors or semantically related tools\. The two choices are placed in the same prompt, where the LLM simulates several subsequent tool invocations and their state changes for each choice, then judges whether the task can be completed\.

We repeat this paired comparisonMMtimes and keep only valid parsed outputs\. LetMsM\_\{s\}be the number of valid comparisons, and letyf\(m\),yc\(m\)∈\{0,1\}y\_\{f\}^\{\(m\)\},y\_\{c\}^\{\(m\)\}\\in\\\{0,1\\\}denote whether the LLM judges the simulated continuations afterafa\_\{f\}andaca\_\{c\}as successful in themm\-th valid comparison\. We estimatev^cmp​\(s,af\)=Ms−1​∑m=1Msyf\(m\)\\hat\{v\}\_\{\\mathrm\{cmp\}\}\(s,a\_\{f\}\)=M\_\{s\}^\{\-1\}\\sum\_\{m=1\}^\{M\_\{s\}\}y\_\{f\}^\{\(m\)\}andv^cmp​\(s,ac\)=Ms−1​∑m=1Msyc\(m\)\\hat\{v\}\_\{\\mathrm\{cmp\}\}\(s,a\_\{c\}\)=M\_\{s\}^\{\-1\}\\sum\_\{m=1\}^\{M\_\{s\}\}y\_\{c\}^\{\(m\)\}\. Pairs are discarded when too few comparisons are parsed or when repeated judgments show low consistency\. This source provides semantic comparison signals for plausible alternatives that are rare in logged trajectories or hard to cover with the simulator alone\.

The three sources serve complementary roles\.𝒟obs\\mathcal\{D\}\_\{\\mathrm\{obs\}\}anchors CIM to real tool behavior,𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}provides scalable structured comparisons, and𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}adds semantic comparison signals\. CITA uses phase\-aware mixing: early training relies more on simulator pairs, middle training balances all sources, and later training increases the weight of observed and LLM\-based pairs to better match real tool\-use contexts\.

![Refer to caption](https://arxiv.org/html/2610.02330v1/CITA.png)Figure 2:Overview of CITA\. CITA trains a Comparative Inference Model \(CIM\) from paired signals built from real trajectories, a Bayesian tool\-graph simulator, and LLM\-based comparison\. The trained CIM guides GRPO rollouts, provides dense comparative rewards, and supports iterative next\-tool selection during inference\.
### 3\.3Architecture of the Comparative Inference Model

The role of CIM is to estimate the long\-horizon value of a possible next tool invocation under the current tool\-use context\. Given a contextssand a candidate tool invocationaa, it outputs a value estimateVϕ​\(s,a\)∈\[0,1\]V\_\{\\phi\}\(s,a\)\\in\[0,1\]and a confidence scorecϕ​\(s,a\)∈\[0,1\]c\_\{\\phi\}\(s,a\)\\in\[0,1\]\. The value estimate predicts how likely choosingaaunderssis to support final task success, while the confidence score measures the reliability of this estimate\.

The model first encodes the textual information in the tool\-use state, including the user query, previous tool invocations, tool outputs, and the candidate tool\. We use the LLM backbone as a semantic encoder and project its hidden states into a lower\-dimensional space for value estimation\. A context encoder summarizes the current tool\-use state intohsh\_\{s\}, and an action encoder maps the candidate tool intohah\_\{a\}\.

A scoring module combineshsh\_\{s\}andhah\_\{a\}into a decision representation:

zs,a=gϕ​\(hs,ha\),Vϕ​\(s,a\)=σ⁡\(fv​\(zs,a\)\),cϕ​\(s,a\)=σ⁡\(fc​\(zs,a\)\)\.z\_\{s,a\}=g\_\{\\phi\}\(h\_\{s\},h\_\{a\}\),\\qquad V\_\{\\phi\}\(s,a\)=\\sigma\(f\_\{v\}\(z\_\{s,a\}\)\),\\qquad c\_\{\\phi\}\(s,a\)=\\sigma\(f\_\{c\}\(z\_\{s,a\}\)\)\.\(5\)Heregϕg\_\{\\phi\}is the scoring module,fvf\_\{v\}andfcf\_\{c\}are the value and confidence heads, andσ⁡\(⋅\)\\sigma\(\\cdot\)is the sigmoid function\. The two heads use the same decision representation but learn different targets\. In implementation, the confidence head reads a detached representation, so confidence calibration does not directly interfere with value learning\.

CIM uses paired forward computation during training\. For a paired signalx=\(s,af,ac,vf,vc,Δ​v\)x=\(s,a\_\{f\},a\_\{c\},v\_\{f\},v\_\{c\},\\Delta v\), the contextssis encoded once and shared by both choices:

v^f=Vϕ​\(s,af\),v^c=Vϕ​\(s,ac\),Δ​v^=v^f−v^c\.\\hat\{v\}\_\{f\}=V\_\{\\phi\}\(s,a\_\{f\}\),\\qquad\\hat\{v\}\_\{c\}=V\_\{\\phi\}\(s,a\_\{c\}\),\\qquad\\Delta\\hat\{v\}=\\hat\{v\}\_\{f\}\-\\hat\{v\}\_\{c\}\.\(6\)This design lets the model learn both the value of each possible next tool invocation and the value gap between two choices under the same context\. At inference time, CIM uses the single\-call formVϕ​\(s,a\)V\_\{\\phi\}\(s,a\)andcϕ​\(s,a\)c\_\{\\phi\}\(s,a\)to score candidate tool invocations\.

### 3\.4CIM Training

CIM is trained to learn two abilities: estimating the value of a possible next tool invocation and comparing two choices under the same context\. For a pairx=\(s,af,ac,vf,vc,Δ​v\)x=\(s,a\_\{f\},a\_\{c\},v\_\{f\},v\_\{c\},\\Delta v\), CIM producesv^f\\hat\{v\}\_\{f\},v^c\\hat\{v\}\_\{c\}, andΔ​v^\\Delta\\hat\{v\}as defined in Eq\.[6](https://arxiv.org/html/2610.02330#S3.E6)\. We first supervise the value estimates with

ℒvalue=12​\[BCE⁡\(v^f,vf\)\+BCE⁡\(v^c,vc\)\]\.\\mathcal\{L\}\_\{\\mathrm\{value\}\}=\\frac\{1\}\{2\}\\left\[\\mathrm\{BCE\}\(\\hat\{v\}\_\{f\},v\_\{f\}\)\+\\mathrm\{BCE\}\(\\hat\{v\}\_\{c\},v\_\{c\}\)\\right\]\.\(7\)This term teaches CIM to predict the long\-horizon success value of each tool choice\.

To make pairwise comparison explicit, we further match the predicted value gap to the paired label:

ℒgap=\(Δ​v^−Δ​v\)2\.\\mathcal\{L\}\_\{\\mathrm\{gap\}\}=\(\\Delta\\hat\{v\}\-\\Delta v\)^\{2\}\.\(8\)This term is central to comparative value estimation, because the agent must decide which choice is better under the same context\.

We add two auxiliary objectives\. The consistency lossℒcons\\mathcal\{L\}\_\{\\mathrm\{cons\}\}regularizes value gaps with self\-identity and transitivity constraints\. The confidence lossℒconf\\mathcal\{L\}\_\{\\mathrm\{conf\}\}calibrates the confidence head using pseudo\-labels derived from prediction errors, so that low\-error estimates receive higher confidence\.

The final objective is

ℒCIM=ℒvalue\+α​ℒgap\+β​ℒcons\+γ​ℒconf\.\\mathcal\{L\}\_\{\\mathrm\{CIM\}\}=\\mathcal\{L\}\_\{\\mathrm\{value\}\}\+\\alpha\\mathcal\{L\}\_\{\\mathrm\{gap\}\}\+\\beta\\mathcal\{L\}\_\{\\mathrm\{cons\}\}\+\\gamma\\mathcal\{L\}\_\{\\mathrm\{conf\}\}\.\(9\)We use a progressive training schedule\. CIM first learns stable value prediction, then incorporates pairwise gap and consistency objectives, and finally calibrates confidence for rollout guidance and reward aggregation\.

### 3\.5CIM\-Guided GRPO and Inference

CITA uses CIM as both a rollout guide and a reward model for policy optimization\. During GRPO training, the policy samples a set of possible next\-tool invocations𝒜t\\mathcal\{A\}\_\{t\}at each decision step\. CIM scores each candidate under the current context by combining its predicted value and confidence:

Sϕ​\(st,a\)=Vϕ​\(st,a\)​cϕ​\(st,a\)\.S\_\{\\phi\}\(s\_\{t\},a\)=V\_\{\\phi\}\(s\_\{t\},a\)c\_\{\\phi\}\(s\_\{t\},a\)\.\(10\)The tool invocation with the highest score is selected and executed:

at=arg⁡maxa∈𝒜t​Sϕ​\(st,a\)\.a\_\{t\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\_\{t\}\}S\_\{\\phi\}\(s\_\{t\},a\)\.\(11\)The resulting output is appended to the context, and the process continues until the rollout terminates\. This procedure lets CIM guide local tool choices while the policy still generates the candidate set\.

For policy optimization, CITA aggregates the CIM scores of the executed tool sequence into a comparative reward\. For each executed steptt, CIM evaluates the realized contextsts\_\{t\}and the selected tool invocationata\_\{t\}, producingVϕ​\(st,at\)V\_\{\\phi\}\(s\_\{t\},a\_\{t\}\)andcϕ​\(st,at\)c\_\{\\phi\}\(s\_\{t\},a\_\{t\}\)\. These step\-level estimates can be obtained during rollout generation or by a lightweight scoring pass over the completed trajectory\.

rcmp​\(τ\)=∑t=1Tcϕ​\(st,at\)​Vϕ​\(st,at\)∑t=1Tcϕ​\(st,at\)\+ϵ\.r\_\{\\mathrm\{cmp\}\}\(\\tau\)=\\frac\{\\sum\_\{t=1\}^\{T\}c\_\{\\phi\}\(s\_\{t\},a\_\{t\}\)V\_\{\\phi\}\(s\_\{t\},a\_\{t\}\)\}\{\\sum\_\{t=1\}^\{T\}c\_\{\\phi\}\(s\_\{t\},a\_\{t\}\)\+\\epsilon\}\.\(12\)This reward gives larger weight to steps where CIM is more confident\. The final reward combines this comparative reward with a task\-level matching reward:

r⁡\(τ\)=wtask​rtask​\(τ\)\+wcmp​rcmp​\(τ\)\.r\(\\tau\)=w\_\{\\mathrm\{task\}\}r\_\{\\mathrm\{task\}\}\(\\tau\)\+w\_\{\\mathrm\{cmp\}\}r\_\{\\mathrm\{cmp\}\}\(\\tau\)\.\(13\)The combined reward is used to compute group\-relative advantages in GRPO\. In this way, CITA provides a step\-aware training signal while still optimizing complete tool\-use trajectories\.

At inference time, CITA applies the same CIM\-guided selection rule without updating the policy\. At each step, the policy samples possible next\-tool invocations, and CIM scores each candidate using Eq\.[10](https://arxiv.org/html/2610.02330#S3.E10)\. The agent executes the highest\-scoring invocation, appends the returned output to the context, and repeats this process until the task is completed or a stopping condition is met\.

## 4Experiments

### 4\.1Setup

Benchmarks\.We evaluate CITA on three long\-horizon tool\-use benchmarks: Toolathlon\[[22](https://arxiv.org/html/2610.02330#bib.bib26)\], TOUCAN\[[23](https://arxiv.org/html/2610.02330#bib.bib27)\], and TRAJECT\-Bench\[[24](https://arxiv.org/html/2610.02330#bib.bib28)\]\. These benchmarks cover different types of tool\-use difficulty\. Toolathlon focuses on long\-horizon dependency with about 13\.5 tool invocations per episode\. TOUCAN has shorter trajectories, about 3\.2 calls per episode, but a large tool pool of 15,294 unique tools\. TRAJECT\-Bench provides a medium\-horizon setting with about 6\.4 tool invocations per episode and 715 unique tools\. Appendix[A](https://arxiv.org/html/2610.02330#A1)provides detailed statistics\.

Backbone models\.We report results with Qwen2\.5\-7B\[[56](https://arxiv.org/html/2610.02330#bib.bib45)\]and Llama3\.1\-8B\[[57](https://arxiv.org/html/2610.02330#bib.bib46)\]\. Both models are first adapted to the tool\-use format with supervised fine\-tuning, and then optimized with the corresponding training method\.

Baselines\.We compare CITA with baselines from three categories\. The first category includes prompting and search\-based tool\-use methods, including ReAct\[[10](https://arxiv.org/html/2610.02330#bib.bib5)\], ToolChain\*\[[43](https://arxiv.org/html/2610.02330#bib.bib32)\], and ToolLLM\[[12](https://arxiv.org/html/2610.02330#bib.bib7)\]\. The second category includes reinforcement learning and step\-level reward methods, including ToolRL\[[26](https://arxiv.org/html/2610.02330#bib.bib17)\], StepTool\[[25](https://arxiv.org/html/2610.02330#bib.bib16)\], AWPO\[[49](https://arxiv.org/html/2610.02330#bib.bib38)\], IPR\[[58](https://arxiv.org/html/2610.02330#bib.bib48)\], and AgentPRM\[[31](https://arxiv.org/html/2610.02330#bib.bib47)\]\. The third category includes predictive and planning\-based methods, including DyMo\[[51](https://arxiv.org/html/2610.02330#bib.bib40)\], WebDreamer\[[52](https://arxiv.org/html/2610.02330#bib.bib41)\], and WKM\[[53](https://arxiv.org/html/2610.02330#bib.bib42)\]\.

Metrics\.We useTool F1andTask Success Rateas the primary metrics\. Tool F1 evaluates whether the generated trajectory selects the correct tools compared with reference trajectories, while Task Success Rate evaluates whether the completed trajectory solves the task\. The former reflects tool\-selection quality, and the latter reflects end\-to\-end task completion\. For Task Success Rate, we send the full tool\-use process, including tool invocations and observations, to GPT\-4o and use the same evaluation prompt for all methods\.

### 4\.2Main Results

Table 1:Main results on three benchmarks\. We report Tool F1 \(%\) and Task Success Rate \(TSR, %\) with Qwen2\.5\-7B and Llama3\.1\-8B as backbone models\. The best results are highlighted in bold\.Comparison with prior methods\.Table[1](https://arxiv.org/html/2610.02330#S4.T1)shows that CITA achieves the best results in all six dataset\-backbone settings on both Tool F1 and TSR, with average gains of 7\.55 and 9\.93 points over the strongest prior method\. The improvement in Tool F1 shows that CITA generates tool\-use trajectories that better match high\-quality reference paths\. More importantly, the larger gain in TSR shows that CITA does not simply imitate reference trajectories in a rigid way\. It also improves the chance that the generated trajectory completes the task\. This is important for tool\-use agents, because multiple valid tool paths may solve the same task\.

The advantage becomes clearer when comparing different groups of baselines\. Prompting and search\-based methods can find useful paths in selected settings, such as ReAct on TOUCAN and ToolLLM on TRAJECT\-Bench, but their performance varies across datasets and backbones\. RL and step\-level reward methods improve some cases, with IPR reaching 55\.02 Tool F1 and 43\.18 TSR on TRAJECT\-Bench with Qwen2\.5\-7B, yet they still fall behind CITA on both metrics\. Predictive and planning\-based methods are the strongest prior group, especially WebDreamer on Toolathlon and WKM on TRAJECT\-Bench\. CITA still improves over them, which suggests that directly estimating the long\-horizon value of next\-tool choices gives a more reliable decision signal than local prompting, generic step rewards, or full future\-state prediction\.

Performance across benchmark settings\.CITA performs consistently across different benchmark profiles\. It improves TSR on Toolathlon, where long trajectories amplify early mistakes; gives the largest gains on TOUCAN, where the large tool pool makes selection difficult; and remains best on TRAJECT\-Bench, where planning\-based baselines are already strong\. These results show that comparative value estimation helps under long\-horizon dependency, large action spaces, and trajectory\-level evaluation\. Training dynamics are provided in Appendix[B](https://arxiv.org/html/2610.02330#A2)\.

### 4\.3CIM Value Estimation Quality

We next evaluate whether CIM learns effective comparative value estimates for tool choices\. Given a held\-out paired example, CIM predicts the value gapΔ​v^\\Delta\\hat\{v\}between the factual tool choice and an alternative tool choice\. We evaluate this prediction from three aspects\. Value Gap MAE measures the absolute error betweenΔ​v^\\Delta\\hat\{v\}and the observed value gapΔ​v\\Delta v\. Direction Accuracy measures whether the sign ofΔ​v^\\Delta\\hat\{v\}matches the sign ofΔ​v\\Delta v, which indicates whether CIM selects the better tool choice\. AUROC measures whether\|Δ​v^\|\|\\Delta\\hat\{v\}\|separates strong\-effect pairs from near\-neutral pairs\.

Table 2:CIM value estimation quality on held\-out paired examples\.Table[2](https://arxiv.org/html/2610.02330#S4.T2)shows that Full CIM improves value\-gap estimation across all three datasets\. It reduces MAE from 0\.4752 to 0\.2083 on TOUCAN and from 0\.3846 to 0\.1768 on TRAJECT\-Bench, while also improving Direction Accuracy and AUROC on the more challenging Toolathlon\. These results show that the full objective helps CIM estimate value gaps and identify the better tool choice\. Additional value\-estimation metrics are provided in Appendix[C](https://arxiv.org/html/2610.02330#A3)\.

Figure[3](https://arxiv.org/html/2610.02330#S4.F3)further shows this trend\. SFT and GRPO\-only produce value\-gap predictions with weak correlation to the ground truth\. CIM trained withℒvalue\\mathcal\{L\}\_\{\\text\{value\}\}only reduces errors but still gives weak separation between positive and negative gaps\. CITA shows the clearest alignment, supporting CIM as an effective comparative value estimator for tool decisions\.

![Refer to caption](https://arxiv.org/html/2610.02330v1/fig2_ctm_calibration.png)Figure 3:CIM value\-gap estimation on held\-out paired examples\. Each point compares the predicted value gapΔ​v^\\Delta\\hat\{v\}with the observed value gapΔ​v\\Delta v\. Compared with SFT, GRPO\-only, and CIM trained withℒvalue\\mathcal\{L\}\_\{\\text\{value\}\}only, CITA shows clearer alignment with the ground truth\.
### 4\.4Ablation Study

Table 3:Ablation results averaged over six dataset\-backbone settings\.Table[3](https://arxiv.org/html/2610.02330#S4.T3)ablates the main components of CITA\. Results are averaged over six dataset\-backbone settings, covering three benchmarks and two backbone LLMs\. Each metric is first computed within one setting and then averaged across settings\. The variant w/o CIM guide removes CIM\-guided candidate selection during rollout generation\. The variant w/orcmpr\_\{\\mathrm\{cmp\}\}removes the comparative reward and keeps only the task\-level reward\. The variant w/oℒgap\\mathcal\{L\}\_\{\\mathrm\{gap\}\}trains CIM without explicit value\-gap supervision\. The variant w/o confidence usesVϕ​\(s,a\)V\_\{\\phi\}\(s,a\)alone for scoring and reward aggregation\. The last two variants remove𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}or𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}from CIM training\.

Removingℒgap\\mathcal\{L\}\_\{\\mathrm\{gap\}\}causes the largest loss\-objective drop, showing that explicit value\-gap learning is central to CIM\. Removing CIM\-guided rollout generation orrcmpr\_\{\\mathrm\{cmp\}\}also leads to clear degradation, supporting the dual role of CIM in trajectory construction and policy optimization\. The confidence score gives a smaller but consistent gain, indicating that reliability\-aware scoring helps reduce noisy tool choices\. Among data\-source ablations, removing𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}causes the largest drop, confirming the importance of structured fork\-point comparisons from the Bayesian simulator\. Removing𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}also weakens performance, suggesting complementary benefits from real trajectory grounding and LLM\-based semantic comparison\. Detailed per\-benchmark ablations are provided in Appendix[D](https://arxiv.org/html/2610.02330#A4)\.

## 5Conclusion and Limitations

This paper introduced CITA, a comparative inference framework for long\-horizon tool use in LLM agents\. CITA trains a Comparative Inference Model to compare candidate next\-tool invocations under the current context and estimate their long\-horizon value\. The model is learned from paired signals built from real trajectories, a Bayesian tool\-graph simulator, and LLM\-based comparison, and is used to guide GRPO training and inference\. Across three benchmarks, CITA improves Tool F1 and Task Success Rate\. Value\-estimation analysis and ablations further show that CIM learns meaningful value gaps\. The current study focuses on benchmarked tool environments with available logged trajectories and tool graphs, which provides a controlled setting for evaluating comparative value estimation\. Extending this framework to more dynamic tool ecosystems, including settings where tools are added, removed, or updated over time, is a natural next step\.

## References

- \[1\]R\. Nakano, J\. Hilton, S\. Balaji, J\. Wu, L\. Ouyang, C\. Kim, C\. Hesse, S\. Jain, V\. Kosaraju, W\. Saunders,et al\.\(2021\)Webgpt: browser\-assisted question\-answering with human feedback\.arXiv preprint arXiv:2112\.09332\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[2\]S\. Yang, Z\. Xiong, Y\. Li, Y\. Li, Q\. Cai, and L\. Feng\(2026\)SRPO: setwise relative policy optimization for multi\-agent llms\.arXiv preprint arXiv:2609\.08452\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[3\]S\. Yang, Y\. Li, S\. He, Y\. Li, Q\. Cai, P\. Jiang, and L\. Feng\(2026\)Phase\-aware mixture of experts for agentic reinforcement learning\.arXiv preprint arXiv:2602\.17038\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[4\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: language models can teach themselves to use tools\.InProceedings of Advances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1),[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[5\]D\. Min, G\. Vaccarino, H\. Chen, Y\. Wu, G\. Yona, and L\. Cheng\(2026\)Stop when reasoning converges: semantic\-preserving early exit for reasoning models\.arXiv preprint arXiv:2605\.17672\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[6\]D\. Min, K\. Zhang, T\. Wu, and L\. Cheng\(2026\)QuCo\-rag: quantifying uncertainty from the pre\-training corpus for dynamic retrieval\-augmented generation\.InFindings of the Association for Computational Linguistics, ACL,pp\. 16482–16500\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[7\]S\. Li, J\. Cao, B\. Hu, A\. Mokhtari, L\. Liu, and A\. Zhang\(2026\)CARE\-rft: confidence\-anchored reinforcement finetuning for reliable reasoning in large language models\.arXiv preprint arXiv:2602\.00085\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[8\]L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig\(2023\)PAL: program\-aided language models\.InProceedings of International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[9\]S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez\(2024\)Gorilla: large language model connected with massive apis\.InProceedings of Advances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1),[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[10\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InProceedings of The Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1),[§2](https://arxiv.org/html/2610.02330#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[11\]M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li\(2023\)API\-bank: A comprehensive benchmark for tool\-augmented llms\.InProceedings of Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[12\]Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.InProceedings of The Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1),[§2](https://arxiv.org/html/2610.02330#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[13\]Z\. Guo, S\. Cheng, H\. Wang, S\. Liang, Y\. Qin, P\. Li, Z\. Liu, M\. Sun, and Y\. Liu\(2024\)StableToolBench: towards stable large\-scale benchmarking on tool learning of large language models\.InProceedings of Findings of the Association for Computational Linguistics,pp\. 11143–11156\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[14\]X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2023\)Agentbench: evaluating llms as agents\.arXiv preprint arXiv:2308\.03688\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[15\]S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig\(2024\)WebArena: A realistic web environment for building autonomous agents\.InProceedings of The Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[16\]H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. Balasubramanian\(2024\)AppWorld: A controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 16022–16076\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[17\]S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan\(2024\)τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arxiv:2403\.00839\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[18\]X\. Liu, Z\. Peng, X\. Yi, X\. Xie, L\. Xiang, Y\. Liu, and D\. Xu\(2024\)Toolnet: connecting large language models with massive tools via tool graph\.arXiv preprint arXiv:2403\.00839\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[19\]Y\. Jiang, H\. Zhou, L\. GU, A\. Han, and T\. Li\(2025\)NaviAgent: bilevel planning on tool navigation graph for large\-scale orchestration\.arXiv preprint arXiv:2506\.19500\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[20\]W\. Chen, W\. Li, D\. Yao, X\. Meng, C\. Gong, and J\. Bi\(2025\)GTool: graph enhanced tool planning with large language model\.arXiv preprint arXiv:2508\.12725\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[21\]Y\. Li, G\. Cai, S\. Yang, H\. Luo, S\. Han, X\. He, D\. Li, and L\. Feng\(2026\)Phgpo: pheromone\-guided policy optimization for long\-horizon tool planning\.arXiv preprint arXiv:2602\.13691\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1)\.
- \[22\]J\. Li, W\. Zhao, J\. Zhao, W\. Zeng, H\. Wu, X\. Wang, R\. Ge, Y\. Cao, Y\. Huang, W\. Liu,et al\.\(2025\)The tool decathlon: benchmarking language agents for diverse, realistic, and long\-horizon task execution\.arXiv preprint arXiv:2510\.25726\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p1.1)\.
- \[23\]Z\. Xu, A\. M\. Soria, S\. Tan, A\. Roy, A\. S\. Agrawal, R\. Poovendran, and R\. Panda\(2025\)Toucan: synthesizing 1\.5 m tool\-agentic data from real\-world mcp environments\.arXiv preprint arXiv:2510\.01179\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p1.1)\.
- \[24\]P\. He, Z\. Dai, B\. He, H\. Liu, X\. Tang, H\. Lu, J\. Li, J\. Ding, S\. Mukherjee, S\. Wang,et al\.\(2025\)TRAJECT\-bench: a trajectory\-aware benchmark for evaluating agentic tool use\.arXiv preprint arXiv:2510\.04550\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p1.1)\.
- \[25\]Y\. Yu, Z\. Wang, W\. Ma, S\. Wang, C\. Wu, Z\. Guo, and M\. Zhang\(2025\)StepTool: enhancing multi\-step tool usage in llms via step\-grained reinforcement learning\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 3952–3962\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1),[§2](https://arxiv.org/html/2610.02330#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[26\]C\. Qian, E\. C\. Acikgoz, Q\. He, H\. Wang, X\. Chen, D\. Hakkani\-Tür, G\. Tur, and H\. Ji\(2025\)Toolrl: reward is all tool learning needs\.arXiv preprint arXiv:2504\.13958\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1),[§2](https://arxiv.org/html/2610.02330#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[27\]G\. Dong, Y\. Chen, X\. Li, J\. Jin, H\. Qian, Y\. Zhu, H\. Mao, G\. Zhou, Z\. Dou, and J\. Wen\(2025\)Tool\-star: empowering llm\-brained multi\-tool reasoner via reinforcement learning\.arXiv preprint arXiv:2505\.16410\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1),[§2](https://arxiv.org/html/2610.02330#S2.p2.1)\.
- \[28\]J\. Li, Y\. Wang, Q\. Yan, Y\. Tian, Z\. Xu, H\. Song, P\. Xu, and L\. L\. Cheong\(2026\)Salt: step\-level advantage assignment for long\-horizon agents via trajectory graph\.InFindings of the Association for Computational Linguistics,pp\. 4709–4725\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1)\.
- \[29\]X\. Liu, K\. Wang, Y\. Wu, F\. Huang, Y\. Li, J\. Zhang, and J\. Jiao\(2025\)Agentic reinforcement learning with implicit step rewards\.arXiv preprint arXiv:2509\.19199\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1)\.
- \[30\]S\. Choudhury\(2025\)Process reward models for llm agents: practical framework and directions\.arXiv preprint arXiv:2502\.10325\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1)\.
- \[31\]Z\. Xi, C\. Liao, G\. Li, Z\. Zhang, W\. Chen, B\. Wang, S\. Jin, Y\. Zhou, J\. Guan, W\. Wu, T\. Ji, T\. Gui, Q\. Zhang, and X\. Huang\(2026\)AgentPRM: process reward models for LLM agents via step\-wise promise and progress\.InProceedings of the ACM Web Conference 2026, WWW,pp\. 4184–4195\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[32\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.InProceedings of The twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1)\.
- \[33\]J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, Y\. Wang, and J\. Guo\(2024\)A survey on llm\-as\-a\-judge\.arXiv preprint arXiv: 2411\.15594\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1)\.
- \[34\]L\. Shi, C\. Ma, W\. Liang, X\. Diao, W\. Ma, and S\. Vosoughi\(2025\)Judging the judges: A systematic study of position bias in llm\-as\-a\-judge\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,pp\. 292–314\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1)\.
- \[35\]L\. Luo, Y\. Liu, R\. Liu, S\. Phatale, M\. Guo, H\. Lara, Y\. Li, L\. Shu, Y\. Zhu, L\. Meng,et al\.\(2024\)Improve mathematical reasoning in language models by automated process supervision\.arXiv preprint arXiv:2406\.06592\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p2.1)\.
- \[36\]J\. N\. Foerster, G\. Farquhar, T\. Afouras, N\. Nardelli, and S\. Whiteson\(2018\)Counterfactual multi\-agent policy gradients\.InProceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence,pp\. 2974–2982\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p4.1)\.
- \[37\]T\. Mesnard, T\. Weber, F\. Viola, S\. Thakoor, A\. Saade, A\. Harutyunyan, W\. Dabney, T\. S\. Stepleton, N\. Heess, A\. Guez, E\. Moulines, M\. Hutter, L\. Buesing, and R\. Munos\(2021\)Counterfactual credit assignment in model\-free reinforcement learning\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research,pp\. 7654–7664\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p4.1)\.
- \[38\]E\. Pona, M\. Kazemi, Y\. Du, D\. Watson, and N\. Paoletti\(2025\)Abstract counterfactuals for language model agents\.arXiv preprint arXiv:2506\.02946\.Cited by:[§1](https://arxiv.org/html/2610.02330#S1.p4.1)\.
- \[39\]Y\. Li, S\. Zhang, R\. Wu, X\. Huang, Y\. Chen, W\. Xu, G\. Qi, and D\. Min\(2024\)MATEval: A multi\-agent discussion framework for advancing open\-ended text evaluation\.InDatabase Systems for Advanced Applications \- 29th International Conference, DASFAA,pp\. 415–426\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[40\]Y\. Li, Z\. Yang, Y\. Huang, X\. Liu, and G\. Qi\(2026\)C3\{\{\}^\{3\}\}tg: conflict\-aware, composite, and collaborative controlled text generation\.InFortieth AAAI Conference on Artificial Intelligence, Thirty\-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI,pp\. 31834–31842\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[41\]K\. Yang, G\. Cai, S\. Yang, S\. He, Y\. Li, M\. Liu, P\. Chen, J\. Xu, and L\. Feng\(2026\)Progress\-conditioned group policy optimization for long\-horizon agentic tasks\.arXiv preprint arXiv:2607\.22724\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[42\]Y\. Li, Y\. Huang, G\. Qi, J\. Feng, N\. Hu, S\. Zhai, H\. Xue, Y\. Chen, R\. Shen, and T\. Wu\(2025\)Harnessing diverse perspectives: A multi\-agent framework for enhanced error detection in knowledge graphs\.InDatabase Systems for Advanced Applications \- 30th International Conference, DASFAA,Vol\.15991,pp\. 446–458\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[43\]Y\. Zhuang, X\. Chen, T\. Yu, S\. Mitra, V\. S\. Bursztyn, R\. A\. Rossi, S\. Sarkhel, and C\. Zhang\(2024\)ToolChain\*: efficient action space navigation in large language models with a\* search\.InProceedings of The Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[44\]J\. Jia and Q\. Li\(2026\)AutoTool: efficient tool selection for large language model agents\.InProceedings of Fortieth AAAI Conference on Artificial Intelligence, Thirty\-Eighth Conference on Innovative Applications of Artificial Intelligence,pp\. 31265–31273\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[45\]B\. Xu, Z\. Peng, B\. Lei, S\. Mukherjee, Y\. Liu, and D\. Xu\(2023\)Rewoo: decoupling reasoning from observations for efficient augmented language models\.arXiv preprint arXiv:2305\.18323\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[46\]Y\. Song, W\. Xiong, D\. Zhu, W\. Wu, H\. Qian, M\. Song, H\. Huang, C\. Li, K\. Wang, R\. Yao,et al\.\(2023\)Restgpt: connecting large language models with real\-world restful apis\.arXiv preprint arXiv:2306\.06624\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p1.1)\.
- \[47\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p2.1)\.
- \[48\]J\. Feng, S\. Huang, X\. Qu, G\. Zhang, Y\. Qin, B\. Zhong, C\. Jiang, J\. Chi, and W\. Zhong\(2025\)Retool: reinforcement learning for strategic tool use in llms\.arXiv preprint arXiv:2504\.11536\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p2.1)\.
- \[49\]Z\. Lin, X\. Wang, H\. Yang, J\. Chai, J\. Cao, G\. Yin, W\. Lin, and R\. He\(2025\)AWPO: enhancing tool\-use of large language models through adaptive integration of reasoning rewards\.arXiv preprint arXiv:2512\.19126\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[50\]F\. Wu, W\. Zhu, Y\. Zhang, S\. Chatterjee, J\. Zhu, F\. Mo, R\. Luo, and J\. Gao\(2025\)PORTool: tool\-use llm training with rewarded tree\.arXiv preprint arXiv:2510\.26020\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p2.1)\.
- \[51\]S\. Guo, O\. D\. Domingues, R\. Avalos, A\. Courville, and F\. Strub\(2025\)World modelling improves language model agents\.arXiv preprint arXiv:2506\.02918\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[52\]Y\. Gu, K\. Zhang, Y\. Ning, B\. Zheng, B\. Gou, T\. Xue, C\. Chang, S\. Srivastava, Y\. Xie, P\. Qi,et al\.\(2024\)Is your llm secretly a world model of the internet? model\-based planning for web agents\.arXiv preprint arXiv:2411\.06559\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[53\]S\. Qiao, R\. Fang, N\. Zhang, Y\. Zhu, X\. Chen, S\. Deng, Y\. Jiang, P\. Xie, F\. Huang, and H\. Chen\(2024\)Agent planning with world knowledge model\.InProceedings of Advances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.
- \[54\]M\. Hu, T\. Fang, J\. Zhang, J\. Ma, Z\. Zhang, J\. Zhou, H\. Zhang, H\. Mi, D\. Yu, and I\. King\(2025\)WebCoT: enhancing web agent reasoning by reconstructing chain\-of\-thought in reflection, branching, and rollback\.InFindings of the Association for Computational Linguistics,pp\. 5155–5173\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p3.1)\.
- \[55\]A\. Rakhsha, T\. Hehn, P\. Mazzaglia, F\. V\. Massoli, A\. Behboodi, and T\. Orekondy\(2026\)LUMINA: long\-horizon understanding for multi\-turn interactive agents\.arXiv preprint arXiv:2601\.16649\.Cited by:[§2](https://arxiv.org/html/2610.02330#S2.p3.1)\.
- \[56\]B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu,et al\.\(2024\)Qwen2\. 5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p2.1)\.
- \[57\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p2.1)\.
- \[58\]W\. Xiong, Y\. Song, X\. Zhao, W\. Wu, X\. Wang, K\. Wang, C\. Li, W\. Peng, and S\. Li\(2024\)Watch every step\! LLM agent learning via iterative step\-level process refinement\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Cited by:[§4\.1](https://arxiv.org/html/2610.02330#S4.SS1.p3.1)\.

## Appendix ADataset Statistics

We evaluate CITA on three tool\-use benchmarks with different sources of difficulty\. Toolathlon is designed for diverse, realistic, and long\-horizon task execution across software environments\. TOUCAN provides large\-scale tool\-agentic trajectories synthesized from real\-world MCP environments\. TRAJECT\-Bench focuses on trajectory\-aware tool\-use evaluation, where the quality of tool selection, parameterization, and ordering is evaluated beyond the final answer\. These benchmarks allow us to test whether CITA improves tool\-use agents under long trajectories, large tool pools, and trajectory\-level constraints\.

Table 4:Statistics of the processed benchmark splits used in our experiments\.Toolathlon has the longest trajectories in our evaluation, with about 13\.5 tool invocations per episode\. It is therefore well suited for testing whether an agent can maintain useful tool choices across many dependent steps\. TOUCAN has much shorter trajectories, with about 3\.2 calls per episode, but its tool pool is far larger than the other two benchmarks\. It tests whether the agent can select appropriate tools from a broad set of possible APIs\. TRAJECT\-Bench lies between these two settings, with about 6\.4 calls per episode and 715 tools\. Its trajectory\-aware design makes it useful for evaluating whether the generated tool\-use path is coherent, not only whether the final answer is correct\.

The three benchmarks are complementary\. Toolathlon stresses long\-horizon dependency, TOUCAN stresses large\-scale tool selection, and TRAJECT\-Bench stresses medium\-horizon trajectory quality\. This combination allows us to evaluate CITA across different types of tool\-use difficulty rather than a single benchmark setting\.

## Appendix BTraining Dynamics

Figure[4](https://arxiv.org/html/2610.02330#A2.F4)reports the training reward curves of CITA during CIM\-guided GRPO on Toolathlon, TOUCAN, and TRAJECT\-Bench\. The curves are plotted for Qwen2\.5\-7B and Llama3\.1\-8B under the same training setting\. These results provide additional evidence that CIM\-guided rollouts and comparative rewards give useful optimization signals during policy learning\.

Figure 4:Training reward curves during CIM\-guided GRPO\. Rewards generally increase over training steps on Toolathlon, TOUCAN, and TRAJECT\-Bench for both Qwen2\.5\-7B and Llama3\.1\-8B, showing that CITA provides useful optimization signals across different tool\-use settings\.The three benchmarks show different training patterns\. On Toolathlon, rewards increase overall but fluctuate more strongly than on the other two datasets\. This is consistent with the long\-horizon nature of Toolathlon, where each episode contains more tool invocations and early decisions can affect many later steps\. In this setting, a small change in tool selection can lead to a large change in trajectory quality, which makes the reward curve less smooth\. Even with these fluctuations, both backbones show a clear upward trend during training\.

TOUCAN shows the most stable improvement\. Although its trajectories are shorter, the benchmark contains a very large tool pool, which makes next\-tool selection difficult\. The reward curves increase steadily for both backbones, suggesting that CIM\-guided selection helps the policy identify useful tools from a broad action space\. This trend is consistent with the main results in Table[1](https://arxiv.org/html/2610.02330#S4.T1), where CITA obtains its largest gains on TOUCAN\.

TRAJECT\-Bench shows a different pattern\. Rewards increase quickly in the early stage and then enter a relatively stable range\. This suggests that the policy first learns to generate more coherent tool\-use trajectories and then makes smaller refinements\. Since TRAJECT\-Bench emphasizes trajectory\-level quality, the plateau is expected once the agent has learned a reliable tool\-use pattern\. CITA still maintains strong final performance on this benchmark, which indicates that comparative value estimation remains useful even when planning\-based baselines are already competitive\.

Overall, the reward curves support the main experimental finding\. CITA does not only improve final evaluation metrics after training; it also provides a stable optimization signal during GRPO\. The upward trends across all three benchmarks indicate that CIM\-guided rollouts and comparative rewards can guide policy learning under long\-horizon dependency, large tool spaces, and medium\-horizon trajectory evaluation\.

## Appendix CAdditional CIM Value Estimation Results

This section provides additional details for the CIM value estimation evaluation\. The goal is to test whether CIM can predict the value difference between two next\-tool choices under the same or comparable context\. For each held\-out pair, CIM predictsΔ​v^=v^f−v^c\\Delta\\hat\{v\}=\\hat\{v\}\_\{f\}\-\\hat\{v\}\_\{c\}, and we compare it with the observed value gapΔ​v\\Delta v\. The evaluation set is built from observed trajectory pairs with sufficient support\. Following the evaluation protocol, we separate strong\-effect pairs and near\-neutral pairs by the magnitude of the observed value gap\. Strong\-effect pairs satisfy\|Δ​v\|≥0\.15\|\\Delta v\|\\geq 0\.15, while near\-neutral pairs satisfy\|Δ​v\|≤0\.05\|\\Delta v\|\\leq 0\.05\. This split allows us to evaluate not only numerical value\-gap estimation, but also whether CIM can distinguish meaningful tool\-choice differences from weak or negligible differences\.

#### Core metrics\.

The main text reports three metrics\.Value Gap MAEmeasures the mean absolute error between the predicted and observed value gaps,\|Δ​v^−Δ​v\|\|\\Delta\\hat\{v\}\-\\Delta v\|\. It evaluates whether CIM estimates the size of the value difference correctly\.Direction Accuracymeasures whetherΔ​v^\\Delta\\hat\{v\}andΔ​v\\Delta vhave the same sign\. It evaluates whether CIM identifies which tool choice has the higher long\-horizon value\. This metric is directly related to decision making, because the agent usually needs to choose the better tool rather than predict the exact value gap\.AUROCevaluates whether the magnitude\|Δ​v^\|\|\\Delta\\hat\{v\}\|can distinguish strong\-effect pairs from near\-neutral pairs\. It measures whether CIM assigns larger predicted gaps to pairs where the two tool choices truly differ\.

#### Additional discrimination metrics\.

We also reportAUPRCandCvS\. AUPRC uses the same score\|Δ​v^\|\|\\Delta\\hat\{v\}\|as AUROC, but focuses on the precision\-recall tradeoff for identifying strong\-effect pairs\. It is useful when the number of strong\-effect and near\-neutral pairs is imbalanced\. CvS, short for contrast\-vs\-small\-gap score, measures whether strong\-effect pairs receive predicted gaps above a fixed threshold and near\-neutral pairs receive predicted gaps below the threshold\. In our evaluation, the threshold is0\.10\.1\. A high CvS means that CIM not only detects meaningful differences, but also avoids assigning large gaps to near\-neutral comparisons\.

#### Correlation metrics\.

PearsonandSpearmancorrelations measure agreement between predicted and observed value gaps from two complementary views\. Pearsonrrmeasures linear agreement betweenΔ​v^\\Delta\\hat\{v\}andΔ​v\\Delta v\. It is high when the predicted gap increases proportionally with the observed gap\. Spearmanrrmeasures rank\-level agreement\. It is high when CIM orders pairs by value\-gap strength in a way that matches the observed ordering, even if the exact scale is not perfectly calibrated\.

Table 5:Additional CIM value estimation metrics\. CIM w/ℒvalue\\mathcal\{L\}\_\{\\text\{value\}\}only uses only the value prediction loss, while Full CIM uses the complete objective with value\-gap learning, consistency regularization, and confidence calibration\.The additional metrics support the conclusion in the main text[2](https://arxiv.org/html/2610.02330#S4.T2)\. On TOUCAN, Full CIM greatly improves CvS from 0\.2667 to 0\.9333 and increases Pearson correlation from 0\.7652 to 0\.9004, indicating that the complete objective produces value gaps that are better calibrated and more discriminative\. AUPRC is already high for CIM w/ℒvalue\\mathcal\{L\}\_\{\\text\{value\}\}only and remains high for Full CIM, which suggests that this dataset is already separable under the AUPRC metric\. Although Spearman correlation slightly decreases from 0\.8965 to 0\.8700, it remains high, while the large gains in CvS and Pearson correlation show that Full CIM better separates true value differences from near\-neutral comparisons\.

Toolathlon is more difficult because its trajectories are longer and the effect of one tool choice may depend on many later decisions\. Under this setting, CIM w/ℒvalue\\mathcal\{L\}\_\{\\text\{value\}\}only has weak agreement with the observed gaps, with negative Pearson and Spearman correlations\. Full CIM improves all reported metrics on Toolathlon, raising AUPRC from 0\.5401 to 0\.7560, CvS from 0\.4091 to 0\.5303, Pearson correlation from \-0\.1307 to 0\.3077, and Spearman correlation from \-0\.1884 to 0\.2193\. This improvement suggests that the complete CIM objective helps the model recover meaningful value\-gap structure when long\-horizon dependencies make the signal noisier\.

## Appendix DAdditional Ablation Results

This section provides detailed ablation results for CITA\. The main table[3](https://arxiv.org/html/2610.02330#S4.T3)reports averaged results over six dataset\-backbone settings \. Here we expand the results for each benchmark and backbone, and include additional ablation variants\.

#### Ablation variants\.

We consider the following variants\.

- •w/o CIM guideremoves CIM\-guided candidate selection during rollout generation\.
- •w/orcmpr\_\{\\mathrm\{cmp\}\}removes the comparative reward and uses only the task\-level reward\.
- •w/oℒgap\\mathcal\{L\}\_\{\\mathrm\{gap\}\}trains CIM without explicit value\-gap supervision\.
- •w/oℒcons\\mathcal\{L\}\_\{\\mathrm\{cons\}\}removes the consistency regularization on value gaps\.
- •w/o confidenceusesVϕ​\(s,a\)V\_\{\\phi\}\(s,a\)alone for candidate scoring and reward aggregation\.
- •w/o𝒟obs\\mathcal\{D\}\_\{\\mathrm\{obs\}\},w/o𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}, andw/o𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}remove the corresponding training source for CIM\.
- •Uniform mixingremoves phase\-aware data mixing and uses a fixed source mixture during CIM training\.
- •K=1K=1candidate keeps CIM in the rollout process but samples only one candidate tool invocation at each step, so CIM cannot compare multiple alternatives under the same context\.
- •Mean aggregationreplaces confidence\-weighted reward aggregation with a simple mean over step values\.

Results are reported in Tool F1 and TSR\. The last two columns show the arithmetic average over the six dataset\-backbone settings\.

Table 6:Detailed ablation results on each dataset\-backbone setting\. Avg\. is the arithmetic mean over all six settings\.
#### Analysis\.

The detailed results support the averaged findings in the main table[3](https://arxiv.org/html/2610.02330#S4.T3)\. Removingℒgap\\mathcal\{L\}\_\{\\mathrm\{gap\}\}gives the largest drop among the main variants, reducing the average Tool F1 from 60\.47 to 54\.86 and TSR from 49\.70 to 41\.92\. This confirms that explicit supervision on value gaps is central to CIM, because the model must learn not only whether one tool invocation is useful, but also whether it is better than another choice under the same context\.

The two uses of CIM are both important\. Removing CIM\-guided rollout generation lowers the average Tool F1 to 55\.12 and TSR to 43\.68, while removing the comparative reward lowers them to 56\.21 and 44\.37\. The first result shows that CIM helps construct better trajectories by selecting among candidate tool invocations before execution\. The second shows that CIM also provides useful training feedback after trajectories are generated\. This supports the dual role of CIM in CITA\.

The data\-source ablations show that the three sources provide complementary supervision\. Removing𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}causes the largest drop among the data\-source variants, reducing the average Tool F1 from 60\.47 to 51\.78 and TSR from 49\.70 to 38\.64\. This confirms the importance of Bayesian simulator pairs, which provide scalable and structured fork\-point comparisons that are difficult to obtain from logged trajectories alone\. Removing𝒟obs\\mathcal\{D\}\_\{\\mathrm\{obs\}\}also causes a clear drop, showing that observed trajectory pairs are needed to anchor CIM to real tool\-use behavior\. Removing𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}further weakens performance, indicating that LLM\-based comparative comparison supplies useful semantic signals for rare or unseen alternatives\.

The additional variants further clarify the design choices\. Uniform mixing underperforms the full phase\-aware mixing schedule, suggesting that early structured supervision and later real\-distribution calibration are both useful\. TheK=1K=1candidate variant performs poorly because CIM cannot compare alternatives when the policy provides only one candidate at each step\. Mean aggregation is better than removing confidence entirely, but still worse than full CITA, which shows that confidence\-weighted aggregation helps reduce the influence of uncertain value estimates\.

Across datasets, the largest drops often appear on Toolathlon and TOUCAN\. Toolathlon is sensitive to early tool\-choice errors because of its long trajectories, while TOUCAN is sensitive to candidate selection because of its large tool pool\. TRAJECT\-Bench is more forgiving in some variants, especially under Llama3\.1\-8B, but full CITA still gives the best overall performance\. These patterns suggest that the components of CITA are useful under different tool\-use difficulties rather than only one benchmark setting\.

## Appendix EImplementation Details

This section provides implementation details for the experiments\. Additional details on paired data construction are provided in Appendix[F](https://arxiv.org/html/2610.02330#A6)\. Unless otherwise specified, all experiments use the same train\-test split, tool environment, and evaluation protocol across methods\.

#### Backbone models\.

We use Qwen2\.5\-7B and Llama3\.1\-8B as backbone LLMs\. Both models are first adapted to the tool\-use format with supervised fine\-tuning and then optimized with the corresponding training method\. The same tokenizer, tool registry, simulator, and evaluation scripts are used for all methods\.

#### Tool\-use format\.

All models follow a unified tool invocation format\. The system prompt lists the available tools for the current task and requires the model to emit a structured invocation:

<tool\_call\>\{"name": "\.\.\.", "arguments": \{\.\.\.\}\}</tool\_call\>\.\\texttt\{<tool\\\_call\>\\lx@text@lbrace"name": "\.\.\.", "arguments": \\lx@text@lbrace\.\.\.\\lx@text@rbrace\\lx@text@rbrace</tool\\\_call\>\}\.After each invocation, the tool response is appended to the dialogue context, and the model either invokes another tool or produces a final response\. For each task, we include tools from successful reference trajectories, add related tools, and fill the remaining slots with randomly sampled tools\. We use at most 50 tools in the prompt\.

#### Supervised fine\-tuning\.

SFT data are constructed from successful tool\-use episodes\. Each example is formatted as a multi\-turn dialogue containing the user request, tool invocations, tool responses, and the final response\. We also include continuation examples that start from intermediate trajectory prefixes, exposing the model to partially completed tool\-use states\. SFT is performed with LoRA and assistant\-only loss masking, where loss is computed only on assistant outputs\. For the 7B and 8B backbones, we use LoRA rank 64, LoRA alpha 128, learning rate5×10−65\\times 10^\{\-6\}, maximum sequence length 4096, and train for 3 epochs\.

#### CIM training\.

CIM is trained after SFT and before GRPO\. The model uses the LLM backbone as a semantic encoder and projects hidden states into a lower\-dimensional value\-estimation space\. In our main configuration, the CIM hidden dimension is 768, and the CIM module uses 8 Transformer layers with 12 attention heads\. The final decision representation is passed to two lightweight MLP heads: a value head forVϕ​\(s,a\)V\_\{\\phi\}\(s,a\)and a confidence head forcϕ​\(s,a\)c\_\{\\phi\}\(s,a\)\. The training objective follows Eq\.[9](https://arxiv.org/html/2610.02330#S3.E9)\. We train CIM with a progressive schedule that gradually increases the difficulty of supervision\. In the first stage, we activate onlyℒvalue\\mathcal\{L\}\_\{\\mathrm\{value\}\}, so the model learns stable estimates ofVϕ​\(s,a\)V\_\{\\phi\}\(s,a\)before being asked to compare alternatives\. In the second stage, we addℒgap\\mathcal\{L\}\_\{\\mathrm\{gap\}\}andℒcons\\mathcal\{L\}\_\{\\mathrm\{cons\}\}, which trains CIM to match value differences between paired tool choices and regularizes the comparison space with self\-identity and transitivity constraints\. In the final stage, we addℒconf\\mathcal\{L\}\_\{\\mathrm\{conf\}\}to calibrate the confidence head after the value estimates have become stable\. This schedule avoids forcing value prediction, pairwise comparison, and confidence calibration to be learned from scratch at the same time\. For CIM training, we use batch size 8, gradient accumulation 8, backbone learning rate1×10−51\\times 10^\{\-5\}, CIM learning rate5×10−55\\times 10^\{\-5\}, and AdamW optimization with a warmup\-cosine learning\-rate schedule\.

#### GRPO training\.

For CITA, we initialize the policy from the SFT model and optimize it with CIM\-guided GRPO\. The SFT LoRA adapter is merged into the base model, and a new trainable actor LoRA adapter is attached for GRPO\. The trained CIM is kept frozen during policy optimization and is used to score candidate tool invocations and compute comparative rewards\. We use LoRA rank 64, LoRA alpha 128, learning rate2×10−62\\times 10^\{\-6\}, group size 6, gradient accumulation 2, sampling temperature 0\.7, and maximum rollout length 15\. The reward is the weighted combination in Eq\.[13](https://arxiv.org/html/2610.02330#S3.E13), withwtask=0\.4w\_\{\\mathrm\{task\}\}=0\.4andwcmp=0\.6w\_\{\\mathrm\{cmp\}\}=0\.6\.

#### CIM\-guided rollout and inference\.

During GRPO rollout generation, the policy samples multiple candidate tool invocations at each decision step\. In the main setting, we sample 3 candidates per step\. CIM scores each candidate using Eq\.[10](https://arxiv.org/html/2610.02330#S3.E10), and the highest\-scoring invocation is executed\. The returned observation is appended to the context before the next step\. Inference uses the same CIM\-guided selection rule as rollout generation, but no policy update is performed\.

#### Data split and reproducibility\.

We use a deterministic task\-level split with seed 42\. Test tasks are excluded from SFT continuation construction and GRPO prompt sampling\. For datasets where task names are not unique enough to define tasks, we use the user prompt as the task key\. All reported results use the same held\-out split across methods\.

#### Hardware and precision\.

We train 7B and 8B models with mixed precision and gradient checkpointing\. SFT uses HuggingFace Trainer and DeepSpeed ZeRO\-2 when needed\. GRPO uses a custom training loop with a shared base LLM, a trainable actor LoRA adapter, and a frozen CIM module\. Main experiments are run on 8x H100 GPUs\.

## Appendix FData Construction Details

This section provides additional details on how we construct paired training signals for CIM\. All sources are converted into the paired format used in Eq\.[2](https://arxiv.org/html/2610.02330#S3.E2):

x=\(s,af,ac,vf,vc,Δ​v\),Δ​v=vf−vc\.x=\(s,a\_\{f\},a\_\{c\},v\_\{f\},v\_\{c\},\\Delta v\),\\qquad\\Delta v=v\_\{f\}\-v\_\{c\}\.\(14\)Heressis the tool\-use context,afa\_\{f\}andaca\_\{c\}are two alternative next\-tool invocations, andvfv\_\{f\}andvcv\_\{c\}are their estimated long\-horizon success values\. In implementation, we also store an optional sample weightwwand source\-specific metadata, such as support counts, sandbox paths, and LLM confidence scores\. The three sources differ in how they obtain the paired choices and their value labels\.

### F\.1𝒟obs\\mathcal\{D\}\_\{\\mathrm\{obs\}\}: Observed Trajectory Pairs

𝒟obs\\mathcal\{D\}\_\{\\mathrm\{obs\}\}is mined from logged tool\-use trajectories\. This source keeps part of CIM training close to real agent behavior\. For each episode, we record the task name, tool sequence, tool arguments, tool outputs, and trajectory success label\. At each step, we build a context key that includes three views of the prefix: the prefix tool sequence, compact hashes of previous arguments, and signatures of recent tool outputs\. This richer context key avoids grouping together prefixes that use the same tools but have different arguments or observations\.

For each task and contextss, we collect all next\-tool invocations observed after that context\. For a candidate next toolaa, we estimate its empirical value by the downstream success rate:

v^obs​\(s,a\)=Nsucc​\(s,a\)N⁡\(s,a\)\.\\hat\{v\}\_\{\\mathrm\{obs\}\}\(s,a\)=\\frac\{N\_\{\\mathrm\{succ\}\}\(s,a\)\}\{N\(s,a\)\}\.\(15\)When two different toolsafa\_\{f\}andaca\_\{c\}are observed under the same context, we construct a paired example by setting

vf=v^obs​\(s,af\),vc=v^obs​\(s,ac\),Δ​v=vf−vc\.v\_\{f\}=\\hat\{v\}\_\{\\mathrm\{obs\}\}\(s,a\_\{f\}\),\\qquad v\_\{c\}=\\hat\{v\}\_\{\\mathrm\{obs\}\}\(s,a\_\{c\}\),\\qquad\\Delta v=v\_\{f\}\-v\_\{c\}\.\(16\)We require a minimum support for each context\-action estimate and limit the number of pairs per context to avoid overrepresenting frequent contexts\. The sample weight is based on the available support, giving higher weight to comparisons backed by more observations\. In addition to the tool prefix, we keep compact argument summaries and output summaries, so the text encoder can recover state information during CIM training\.

### F\.2𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}: Bayesian Simulator Pairs

𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}is generated from the tool transition graph\. This source provides scalable and structured comparisons at fork points, especially when logged trajectories do not cover enough alternatives under the same context\.

We first build a directed tool transition graph from logged trajectories\. Each node denotes a tool, and each directed edgei→ji\\rightarrow jrecords that tooljjwas invoked after toolii\. A fork point is a node with at least two outgoing successors\. For each sandbox instance, we sample a fixed set of synthetic probabilities for all tool nodes and transition edges\. For a tool nodeii, the sampled node success probability is

pi∼Beta⁡\(g​ni\+λ,\(1−g\)​ni\+λ\),p\_\{i\}\\sim\\mathrm\{Beta\}\(gn\_\{i\}\+\\lambda,\(1\-g\)n\_\{i\}\+\\lambda\),\(17\)wherenin\_\{i\}is the tool invocation count,ggis the global success rate in the logged data, andλ\\lambdais a smoothing constant\. For an edgei→ji\\rightarrow j, the transition reliability is sampled from a Beta prior parameterized by the observed transition count and the relative frequency of this edge among outgoing transitions fromii:

ri​j∼Beta⁡\(qi​j​ni​j\+λ,\(1−qi​j\)​ni​j\+λ\),qi​j=ni​j∑j′ni​j′\.r\_\{ij\}\\sim\\mathrm\{Beta\}\(q\_\{ij\}n\_\{ij\}\+\\lambda,\(1\-q\_\{ij\}\)n\_\{ij\}\+\\lambda\),\\qquad q\_\{ij\}=\\frac\{n\_\{ij\}\}\{\\sum\_\{j^\{\\prime\}\}n\_\{ij^\{\\prime\}\}\}\.\(18\)All comparisons inside the same sandbox share the sampledpip\_\{i\}andri​jr\_\{ij\}, forming a fixed virtual world\.

For each fork nodeuu, we generate a plausible prefix context by a reverse walk on the tool graph\. The resulting prefix ends atuuand serves as the contextss\. We then choose two successor toolsafa\_\{f\}andaca\_\{c\}from the outgoing neighbors ofuu\. Starting from each successor, we expand a short continuation path inside the same sandbox\. For a pathP=\(b0,…,bL\)P=\(b\_\{0\},\\ldots,b\_\{L\}\), the sandbox value is computed as

v^sandbox​\(P\)=∏ℓ=0Lpbℓ​∏ℓ=0L−1rbℓ​bℓ\+1\.\\hat\{v\}\_\{\\mathrm\{sandbox\}\}\(P\)=\\prod\_\{\\ell=0\}^\{L\}p\_\{b\_\{\\ell\}\}\\prod\_\{\\ell=0\}^\{L\-1\}r\_\{b\_\{\\ell\}b\_\{\\ell\+1\}\}\.\(19\)The two expanded pathsPfP\_\{f\}andPcP\_\{c\}define

vf=v^sandbox​\(Pf\),vc=v^sandbox​\(Pc\),Δ​v=vf−vc\.v\_\{f\}=\\hat\{v\}\_\{\\mathrm\{sandbox\}\}\(P\_\{f\}\),\\qquad v\_\{c\}=\\hat\{v\}\_\{\\mathrm\{sandbox\}\}\(P\_\{c\}\),\\qquad\\Delta v=v\_\{f\}\-v\_\{c\}\.\(20\)Since both paths are evaluated in the same sampled sandbox, the comparison isolates the effect of choosing different successor tools while keeping the synthetic environment fixed\.

In implementation, we generate multiple sandbox instances with different sampled probabilities\. Each sandbox enumerates fork points and successor pairs, with limits on the number of sampled successors and the continuation depth to control data size\. This source is especially useful for training value\-gap consistency because multiple comparisons inside the same sandbox share the same underlying node and edge probabilities\.

### F\.3𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}: LLM\-Based comparative Pairs

𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}provides semantic comparisons for alternatives that are rare in logged trajectories or difficult to cover with the graph simulator\. For each logged trajectory, we select a prefixss, keep the factual next toolafa\_\{f\}, and choose a plausible comparative toolaca\_\{c\}\. The comparative tool is selected by a three\-level strategy\. We first prefer other valid successors of the current predecessor in the tool graph, because these alternatives are position\-sensitive\. At the first step, where there may be no predecessor, we use semantically related entry tools\. If neither source provides a valid candidate, we use tools from the same server or tool family as a fallback\.

The factual and comparative choices are evaluated in the same LLM prompt\. Given the task description and the executed prefix state, the LLM is asked to imagine a short continuation for each option, including subsequent tool invocations and state changes, and then judge whether each branch can complete the task\. Using a paired prompt is important because both options are judged under the same context and the same calibration scale\.

We repeat the paired comparisonMMtimes\. In the main setting,M=5M=5\. Only valid parsed responses are kept\. LetMsM\_\{s\}be the number of valid comparisons\. For themm\-th valid comparison,yf\(m\)∈\{0,1\}y\_\{f\}^\{\(m\)\}\\in\\\{0,1\\\}andyc\(m\)∈\{0,1\}y\_\{c\}^\{\(m\)\}\\in\\\{0,1\\\}denote whether the LLM judges the continuations afterafa\_\{f\}andaca\_\{c\}as successful\. The LLM\-based value estimates are

v^cmp​\(s,af\)=1Ms​∑m=1Msyf\(m\),v^cmp​\(s,ac\)=1Ms​∑m=1Msyc\(m\)\.\\hat\{v\}\_\{\\mathrm\{cmp\}\}\(s,a\_\{f\}\)=\\frac\{1\}\{M\_\{s\}\}\\sum\_\{m=1\}^\{M\_\{s\}\}y\_\{f\}^\{\(m\)\},\\qquad\\hat\{v\}\_\{\\mathrm\{cmp\}\}\(s,a\_\{c\}\)=\\frac\{1\}\{M\_\{s\}\}\\sum\_\{m=1\}^\{M\_\{s\}\}y\_\{c\}^\{\(m\)\}\.\(21\)We retain a pair only if enough valid responses are parsed\. In our implementation, at least three valid comparisons are required\. We also compute a consistency score from the repeated judgments and combine it with parsing coverage to form a confidence score\. Low\-confidence pairs are filtered out, and the remaining pair weight is proportional to the confidence\. This filtering reduces noise from unstable LLM judgments while preserving semantic comparisons that are difficult to obtain from logs alone\.

#### LLM\-based comparison prompt\.

The prompt used for LLM\-based comparison is shown below\. The prompt places the factual and comparative tool choices in the same context and asks the LLM to imagine one plausible continuation for each option\. This paired format reduces calibration drift between separate LLM calls and makes the two success judgments more directly comparable\.

```
You are simulating possible future trajectories for an AI agent’s
tool selection.

Task: {task_description}

Executed prefix state:
{context_state}

The agent must choose between two tools. For EACH option, imagine
one plausible
continuation trajectory (1-5 tool invocations, until task success
or clear failure).
For each step: tool name, one-line state change.

Option A: {factual_tool_name} -- {factual_tool_desc}
Option B: {counter_tool_name} -- {counter_tool_desc}

Respond in JSON:
{
  "trajectory_a": [
    {"tool": "<name>", "state_change": "<brief>"}
  ],
  "success_a": <0_or_1>,
  "trajectory_b": [
    {"tool": "<name>", "state_change": "<brief>"}
  ],
  "success_b": <0_or_1>,
  "reasoning": "<one sentence>"
}
```

### F\.4Phase\-Aware Dataset Mixing

After constructing the three sources, we normalize them into the same pair format and merge them for CIM training\. We keep source\-specific fields, such as support counts, sandbox paths, and LLM confidence, because they can be useful for weighting and diagnostics\. For implementation, we instantiate the progressive mixture with three training phases, summarized in Table[7](https://arxiv.org/html/2610.02330#A6.T7)\.

Table 7:Phase\-aware mixing of the three paired data sources\. Source weights affect the training weight of examples, while sample ratios affect how many examples are sampled from each source\.The phases follow the same motivation as the progressive CIM training schedule\. Phase 1 emphasizes𝒟sandbox\\mathcal\{D\}\_\{\\mathrm\{sandbox\}\}, which provides clean structural supervision and many controlled fork\-point comparisons\. Phase 2 balances all three sources, allowing the model to learn value prediction, value\-gap estimation, and semantic comparison jointly\. Phase 3 increases the influence of𝒟obs\\mathcal\{D\}\_\{\\mathrm\{obs\}\}and𝒟cmp\\mathcal\{D\}\_\{\\mathrm\{cmp\}\}, which calibrates CIM toward real tool\-use contexts while still retaining a small amount of simulator supervision\.

### F\.5Train\-Validation Split and Leakage Control

We use a group\-aware split when creating CIM train and validation sets\. The split key includes the source, task identifier, context prefix, argument\-pattern hashes, and recent output signatures\. For sandbox pairs, the split key also includes the sandbox identifier and context\. We further group mirror comparisons by sorting the factual and comparative action identifiers\. This prevents a pair\(af,ac\)\(a\_\{f\},a\_\{c\}\)from appearing in training while its mirrored comparison\(ac,af\)\(a\_\{c\},a\_\{f\}\)appears in validation under the same context\.

This split is important because paired comparisons often come in symmetric or closely related forms\. A naive random split could leak the same state family into both train and validation sets, leading to overestimated value\-estimation quality\. The group\-aware split keeps related comparisons in the same split and provides a more reliable validation signal for CIM\.

## Appendix GEvaluation Protocol

This section describes the general evaluation protocol used in our experiments\. The GPT\-4o judge protocol for Task Success Rate is provided separately in Appendix[H](https://arxiv.org/html/2610.02330#A8)\. All methods are evaluated on the same held\-out split, with the same tool registry, tool simulator, maximum step budget, and evaluation prompts\.

#### Held\-out split\.

We use a deterministic task\-level split with seed 42\. For datasets with meaningful task names, the task name is used as the split key\. For datasets where task names are not sufficiently discriminative, we use the user prompt as the task key\. We assign 20% of task keys to the test split by hashing the task key with the seed\. Test tasks are excluded from GRPO prompt sampling and training\-time trajectory construction\.

#### Trajectory collection\.

For each test task, the agent interacts with the same tool simulator\. At each step, the model emits a structured tool invocation:

<tool\_call\>\{"name": "\.\.\.", "arguments": \{\.\.\.\}\}</tool\_call\>\.\\texttt\{<tool\\\_call\>\\lx@text@lbrace"name": "\.\.\.", "arguments": \\lx@text@lbrace\.\.\.\\lx@text@rbrace\\lx@text@rbrace</tool\\\_call\>\}\.The simulator returns the corresponding observation, which is appended to the dialogue context\. The interaction terminates when the model produces a final response or reaches the maximum step limit\. We record the user request, all tool invocations, tool arguments, tool observations, and the final response\.

#### Tool\-name normalization\.

Before computing tool\-matching metrics, we normalize tool names by removing version suffixes such as\_v1or\_v13\. We also ignore minor formatting differences such as case, hyphen, and underscore variations when the normalized tool names indicate the same function\.

#### Tool F1\.

Tool F1 evaluates whether the generated trajectory selects tools that match successful reference trajectories\. LetP=\(p1,…,pm\)P=\(p\_\{1\},\\ldots,p\_\{m\}\)be the generated tool sequence andR=\(r1,…,rn\)R=\(r\_\{1\},\\ldots,r\_\{n\}\)be a reference sequence\. We compute an order\-preserving fuzzy match between normalized tool names and denote the number of matched invocations byM⁡\(P,R\)M\(P,R\)\. Precision, recall, and F1 are computed as

Precision=M⁡\(P,R\)\|P\|,Recall=M⁡\(P,R\)\|R\|,F1=2⋅Precision⋅RecallPrecision\+Recall\.\\mathrm\{Precision\}=\\frac\{M\(P,R\)\}\{\|P\|\},\\qquad\\mathrm\{Recall\}=\\frac\{M\(P,R\)\}\{\|R\|\},\\qquad\\mathrm\{F1\}=\\frac\{2\\cdot\\mathrm\{Precision\}\\cdot\\mathrm\{Recall\}\}\{\\mathrm\{Precision\}\+\\mathrm\{Recall\}\}\.\(22\)When multiple successful reference trajectories are available for a task, we compute Tool F1 against each reference and use the best score\. This avoids penalizing a model when it follows a valid alternative tool path\.

#### Additional metrics\.

Besides Tool F1 and Task Success Rate, we compute several diagnostic metrics\.Next\-tool Accuracymeasures whether the predicted next tool matches the reference next tool under a given context\.Average Stepsreports the average number of generated tool invocations\.Progress Ratemeasures the fraction of examples where the model invokes at least one tool\.Argument Qualityevaluates whether generated tool arguments are compatible with the selected tool schema and historical observations\. These metrics are used for diagnostics and are reported in Appendix[J](https://arxiv.org/html/2610.02330#A10)\.

#### Aggregation\.

For each method, we aggregate metrics over all valid held\-out episodes\. Tool F1, Next\-tool Accuracy, Argument Quality, Progress Rate, and Task Success Rate are reported as percentages unless otherwise specified\. For tables that average across settings, we first compute each metric within one dataset\-backbone setting and then take the arithmetic mean over the six settings\.

GPT\-4o Judge System PromptYou are an expert evaluator of tool\-use agents\. Your task is to judge whether an agent’s tool\-use trajectory satisfies the user request\.Evaluation principles:1\.Evaluate task completion, not exact imitation of the reference trajectory\.2\.The reference trajectory is one possible solution path\. Alternative valid tools or orders should be accepted if they satisfy the task\.3\.Focus on whether the selected tools, arguments, observations, and final response support the user request\.4\.Minor tool\-name differences, version suffixes, case differences, underscores, and hyphens should be ignored when the tool functions are equivalent\.5\.Do not penalize external simulator artifacts when the tool selection and arguments are reasonable\.6\.Penalize invalid arguments, missing required steps, unsupported conclusions, and premature stopping\.7\.For long\-horizon tasks, partial completion may receive credit if the agent covers the main stages and makes meaningful progress toward the task\.

## Appendix HGPT\-4o Judge Protocol

GPT\-4o Judge User Prompt TemplateUser request\{user\_request\}Reference tool trajectoryThe following trajectory is provided as a reference solution, not as the only valid solution:\{reference\_trajectory\}Agent tool trajectoryEach step contains the tool name, arguments, and observation:\{agent\_trajectory\}Agent final response\{final\_response\}Please judge whether the agent completed the task\. Output only a JSON object in the following format:``` { "task_completed": "yes/partial/no", "score": <integer from 0 to 10>, "tool_selection_reasonable": <true/false>, "tool_order_reasonable": <true/false>, "uses_observations_correctly": <true/false>, "reason": "<brief reason in one or two sentences>" } ``` Scoring guideline:•9–10: the trajectory completes the task and covers all key requirements\.•7–8: the trajectory is mostly correct, with only minor omissions\.•4–6: the trajectory makes meaningful progress and covers the main stages, but is incomplete\.•1–3: the trajectory contains a few relevant invocations but misses core requirements\.•0: no effective tool use is made\.Completion label:•yes: score≥7\\geq 7\.•partial: score≥4\\geq 4and score<7<7\.•no: score<4<4\.

We use GPT\-4o to evaluate Task Success Rate \(TSR\), because long\-horizon tool\-use tasks often admit multiple valid tool trajectories\. Exact matching against a single reference path may underestimate success when the agent uses functionally equivalent tools or follows a different valid order\. The judge therefore evaluates whether the generated trajectory satisfies the user request, instead of requiring exact reference imitation\.

#### Judge model and inputs\.

We use GPT\-4o with temperature0\.10\.1and a maximum generation length of 512 tokens\. For each episode, the judge receives the user request, a reference tool trajectory, the agent’s generated tool trajectory, truncated tool observations, and the final response\. Method names are hidden from the judge\. The same judge model and prompt are used for all methods\. To control prompt length, we truncate each tool observation to 500 characters, the user request to 1000 characters, and the final response to 500 characters\.

#### Evaluation principle\.

The judge evaluates task completion rather than surface\-level similarity to the reference trajectory\. The reference trajectory is treated as one possible solution path, not the only correct path\. Functionally equivalent tools, minor naming differences, and different but reasonable invocation orders are accepted when they satisfy the same task requirement\. Invalid tool arguments, missing required information, unsupported final answers, or premature stopping are treated as failures\. External simulator artifacts, such as empty returns or API\-like failures, are not penalized when the selected tool and arguments are otherwise reasonable\.

#### Completion labels\.

The judge outputs one of three labels:yes,partial, orno\. A trajectory is labeledyesif it completes the user request or covers all key requirements\. It is labeledpartialif it follows a reasonable tool\-use strategy and completes the main stages of the task, but misses minor details or has incomplete final synthesis\. It is labelednoif the tool trajectory is mostly irrelevant, misses core requirements, fails to use obtained observations, or stops before meaningful progress is made\. In the main paper, TSR counts bothyesandpartialas successful:

TSR=Nyes\+NpartialNvalid,\\mathrm\{TSR\}=\\frac\{N\_\{\\mathrm\{yes\}\}\+N\_\{\\mathrm\{partial\}\}\}\{N\_\{\\mathrm\{valid\}\}\},\(23\)whereNvalidN\_\{\\mathrm\{valid\}\}is the number of episodes with a valid JSON judgment\. We also record strict success rate using onlyyesfor diagnostic analysis\.

#### Parsing and aggregation\.

For each episode, we parse the judge response as JSON\. If the response cannot be parsed, the episode is marked as an evaluation error and excluded from the valid denominator\. For each method, we aggregate judge outputs over all valid held\-out episodes and report TSR in the main table\.

## Appendix IBaseline Details

This section describes how we adapt baseline methods to our long\-horizon tool\-use setting\. All baselines are evaluated with the same held\-out split, tool registry, tool simulator, maximum step budget, and evaluation protocol as CITA\. For fair comparison, all trainable baselines start from the same SFT\-initialized backbone used by CITA\. Prompting and search\-based baselines use the same tool invocation format and tool list\. RL\-based baselines use the same rollout budget and GRPO implementation unless the original method specifies a different optimization rule\. Predictive and planning\-based baselines use the same candidate generation budget as CITA whenever candidate\-level comparison is required\.

### I\.1Prompting and Search\-Based Tool Use

#### ReAct\.

ReAct interleaves natural\-language reasoning with tool actions\. We implement ReAct by adding a method\-specific instruction that asks the model to alternate between a short reasoning step and a structured tool invocation\. The tool invocation is still required to follow the same format used by all methods:

<tool\_call\>\{"name": "\.\.\.", "arguments": \{\.\.\.\}\}</tool\_call\>\.\\texttt\{<tool\\\_call\>\\lx@text@lbrace"name": "\.\.\.", "arguments": \\lx@text@lbrace\.\.\.\\lx@text@rbrace\\lx@text@rbrace</tool\\\_call\>\}\.After each tool observation, the model updates its reasoning state and either invokes another tool or produces a final answer\. No CIM score or comparative reward is used\.

#### ToolChain\*\.

ToolChain\* formulates tool use as action\-space navigation and applies A\* search to find a low\-cost tool\-use path\. We adapt this idea by building a search tree over candidate tool invocations\. Each node represents a partial tool\-use trajectory, and each outgoing edge corresponds to a possible next tool invocation generated by the policy\. We score each partial path with a cost that combines model generation cost, tool\-name validity, argument validity, and task\-progress heuristic\. The search keeps a fixed frontier size and expands candidates until the model terminates or reaches the maximum step limit\. The same tool simulator is used to execute expanded tool invocations\. This baseline tests whether explicit search over tool\-use paths can replace CIM\-based value estimation\.

#### ToolLLM\.

ToolLLM provides a general framework for tool\-use data construction, model training, and tool invocation evaluation\. Since the original ToolLLM uses ToolBench and its own tool environment, we implement a ToolLLM\-style baseline in our benchmark environment\. The model is initialized from the same SFT checkpoint and uses the same tool descriptions\. At inference time, we use a decision\-tree search strategy over generated tool invocations, following the ToolLLM setting where the agent explores tool\-use trajectories and selects a valid execution path\. The search budget, maximum step limit, tool formatting rules, and simulator are matched to the other methods\.

### I\.2Reinforcement Learning and Step\-Level Reward Baselines

#### ToolRL\.

ToolRL studies reward design for tool learning under reinforcement learning\. We adapt ToolRL by training the SFT policy with GRPO using tool\-use rewards that do not include CIM\. The reward contains task\-level trajectory matching, tool validity, and argument validity terms\. This baseline uses the same rollout generation budget and held\-out split as CITA, but it does not compare comparative next\-tool choices under the same context\.

#### StepTool\.

StepTool introduces step\-grained reinforcement learning for multi\-step tool use\. We implement this baseline by adding step\-level rewards derived from reference tool trajectories\. At each step, the model receives feedback based on whether the selected tool matches the corresponding reference step or supports progress toward a reference trajectory\. The step\-level rewards are aggregated with the task\-level reward for GRPO training\. Unlike CITA, StepTool does not train a separate comparative value model and does not score alternative candidate invocations before execution\.

#### IPR\.

IPR performs iterative step\-level process refinement by estimating step\-level rewards and using them to improve agent behavior\. We adapt IPR by generating alternative actions along reference prefixes and evaluating them with rollout\-based or reference\-based feedback\. The resulting step\-level comparisons are used to refine the policy iteratively\. To keep the comparison fair, IPR uses the same SFT initialization, tool simulator, maximum step budget, and rollout budget as CITA\. The key difference is that IPR estimates process feedback through iterative refinement, whereas CITA trains CIM to directly estimate the long\-horizon value of possible next\-tool invocations\.

#### AWPO\.

AWPO integrates reasoning\-related reward signals into policy optimization through adaptive advantage weighting\. We adapt AWPO by combining the task\-level reward with auxiliary reasoning and format rewards, then applying advantage weighting during GRPO training\. The reasoning reward encourages coherent task decomposition and valid tool\-use planning, while the format reward encourages valid structured tool invocations\. No CIM score, Bayesian simulator pair, or LLM\-based comparative pair is used in this baseline\.

#### AgentPRM\.

AgentPRM trains a process reward model for agent decisions by modeling both promise and progress\. We adapt AgentPRM by training a step\-level reward model from the same logged trajectories used in our experiments\. The reward model scores intermediate tool\-use states and candidate actions according to their estimated progress toward task completion\. During policy optimization, AgentPRM scores are used as step\-level rewards or candidate\-ranking signals\. This baseline evaluates whether a general process reward model for agents can substitute for CIM\. Unlike CIM, AgentPRM is not trained on the three\-source paired comparative construction and does not explicitly match value gaps between alternative tool invocations under the same context\.

### I\.3Predictive and Planning\-Based Baselines

#### DyMo\.

DyMo learns an internal environment model for stateful tool use by combining function calling with state prediction\. We adapt DyMo by adding a state\-prediction objective to the SFT policy\. After each tool invocation, the model predicts a compact summary of the resulting state before observing or using the next tool\. At inference time, candidate invocations are ranked according to their predicted state progress and tool validity\. This baseline tests whether implicit state\-dynamics prediction can provide enough lookahead for long\-horizon tool selection\.

#### WebDreamer\.

WebDreamer uses an LLM as a world model and value function to simulate outcomes of candidate actions before execution\. We adapt this idea by prompting an LLM planner to imagine the likely observation and task progress after each candidate tool invocation\. The imagined outcomes are then scored, and the highest\-scoring candidate is executed in the real tool simulator\. The number of candidates and the maximum step budget are matched to CITA\. Unlike CITA, WebDreamer\-style planning relies on natural\-language outcome simulation at inference time rather than a trained CIM that directly estimates next\-tool value\.

#### WKM\.

WKM provides global task knowledge before execution and local dynamic knowledge during agent planning\. We adapt WKM by first generating a task\-level knowledge summary that describes likely subtasks, relevant tools, and expected intermediate states\. During execution, the model updates a local state summary after each tool observation and conditions the next tool invocation on both the global and local knowledge\. This baseline tests whether explicit planning knowledge can improve tool\-use trajectories without learning a comparative value estimator\.

### I\.4Fairness and Budget Control

For all baselines, we control the following factors\. First, all methods use the same tool registry and the same per\-task tool list construction\. Second, all generated tool invocations are executed by the same simulator and are subject to the same maximum step limit\. Third, trainable methods use the same SFT initialization and held\-out split\. Fourth, methods that require candidate comparison use the same candidate budget as CITA whenever possible\. Finally, all methods are evaluated with the same Tool F1 computation and the same GPT\-4o judge protocol for Task Success Rate\. These controls ensure that performance differences mainly reflect the baseline method rather than differences in backbone, tool access, evaluation, or decoding budget\.

## Appendix JAdditional Results and Confidence Intervals

This section reports additional diagnostic metrics for CITA across benchmarks and backbone models\. The main text reports Tool F1 and Task Success Rate \(TSR\)\. Here we also include tool\-level precision, tool\-level recall, Next\-tool Accuracy, Progress Rate, Argument Quality, and average number of generated tool invocations\. For the two main metrics, Tool F1 and TSR, we report95%95\\%confidence intervals estimated by task\-level bootstrap resampling\.

#### Metrics\.

We report additional diagnostic metrics to characterize different aspects of tool\-use behavior\.Precisionmeasures the fraction of generated tool invocations that match a reference trajectory, and reflects whether the agent avoids unnecessary or irrelevant tools\.Recallmeasures the fraction of reference tool invocations covered by the generated trajectory, and reflects whether the agent includes the tools needed for task completion\.Tool F1is the harmonic mean of precision and recall:

F1=2⋅Precision⋅RecallPrecision\+Recall\.\\mathrm\{F1\}=\\frac\{2\\cdot\\mathrm\{Precision\}\\cdot\\mathrm\{Recall\}\}\{\\mathrm\{Precision\}\+\\mathrm\{Recall\}\}\.\(24\)TSRmeasures whether the completed trajectory satisfies the user request according to the GPT\-4o judge protocol in Appendix[H](https://arxiv.org/html/2610.02330#A8)\.Next\-tool Accuracymeasures whether the predicted next tool matches the reference next tool under the current context\.Progress Ratemeasures the fraction of examples where the agent invokes at least one tool, indicating whether the model actually enters the tool\-use process\.Argument Qualitymeasures whether generated tool arguments are compatible with the selected tool schema and the observed task state\.Avg\. Stepsreports the average number of tool invocations in generated trajectories, which helps diagnose whether a method tends to stop too early or generate overly long trajectories\. Tool F1 and TSR are the primary metrics, while the remaining metrics provide diagnostic evidence for tool selection, tool\-use engagement, argument validity, and trajectory length\.

Table 8:Additional results and confidence intervals for CITA across benchmarks and backbones\. Tool F1 and TSR are reported with estimated95%95\\%confidence intervals\.The additional metrics provide a more detailed view of CITA’s behavior\. The precision and recall values are balanced across most settings, indicating that CITA does not simply increase recall by generating many extra tool invocations, nor does it over\-prune trajectories at the cost of missing key tools\. The high Progress Rate shows that CITA reliably enters the tool\-use process rather than stopping before taking action\. Next\-tool Accuracy is lower than Tool F1 in several settings, which is expected because multiple valid tool trajectories may solve the same task and a single reference next tool can be too strict\. The average number of steps also follows the benchmark profiles: Toolathlon requires longer trajectories, TOUCAN is shorter, and TRAJECT\-Bench lies between them\. Overall, these diagnostics support the main results: CITA improves trajectory\-level tool selection while maintaining effective end\-to\-end task completion\.

## Appendix KConfidence Calibration of CIM

We further evaluate whether the confidence head of CIM produces a meaningful reliability estimate\. In our method, the confidence scorecϕ​\(s,a\)c\_\{\\phi\}\(s,a\)is used together with the value estimateVϕ​\(s,a\)V\_\{\\phi\}\(s,a\)during candidate scoring and reward aggregation\. This experiment tests whether a higher predicted confidence indeed corresponds to a higher empirical prediction accuracy\.

For each held\-out context\-action pair\(s,a\)\(s,a\), we record the predicted valueVϕ​\(s,a\)V\_\{\\phi\}\(s,a\)and confidence scorecϕ​\(s,a\)c\_\{\\phi\}\(s,a\)\. We convert the value prediction into a binary success decision by thresholding at 0\.5\. That is, the model predicts success whenVϕ​\(s,a\)\>0\.5V\_\{\\phi\}\(s,a\)\>0\.5, and predicts failure otherwise\. This binary prediction is then compared with the ground\-truth binary label to determine whether the prediction is correct\.

To evaluate calibration, we partition all examples into 10 bins according to the predicted confidence, from\[0,0\.1\)\[0,0\.1\)to\[0\.9,1\.0\]\[0\.9,1\.0\]\. For each bin, we compute the average confidence and the empirical accuracy\. A perfectly calibrated model should satisfy that the average confidence matches the empirical accuracy in each bin\. We summarize calibration quality using Expected Calibration Error \(ECE\), defined as

ECE=∑b=1BnbN​\|acc⁡\(b\)−conf⁡\(b\)\|,\\mathrm\{ECE\}=\\sum\_\{b=1\}^\{B\}\\frac\{n\_\{b\}\}\{N\}\\left\|\\mathrm\{acc\}\(b\)\-\\mathrm\{conf\}\(b\)\\right\|,\(25\)
whereBBis the number of bins,nbn\_\{b\}is the number of examples in binbb,NNis the total number of examples,acc⁡\(b\)\\mathrm\{acc\}\(b\)is the empirical accuracy of binbb, andconf⁡\(b\)\\mathrm\{conf\}\(b\)is the average predicted confidence in binbb\.

Figure[5](https://arxiv.org/html/2610.02330#A11.F5)compares two versions of CIM: a variant trained without the confidence lossℒconf\\mathcal\{L\}\_\{\\mathrm\{conf\}\}, and the full CIM\. The upper row shows reliability diagrams, where the blue bars denote empirical accuracy in each confidence bin, the red line denotes average confidence, and the diagonal dashed line denotes perfect calibration\. The lower row shows the distribution of predicted confidence scores\.

The results show a clear difference between the two variants\. Withoutℒconf\\mathcal\{L\}\_\{\\mathrm\{conf\}\}, the confidence predictions concentrate in a narrow middle range, with mean0\.4980\.498and standard deviation0\.0290\.029\. Its ECE is0\.4600\.460, and the empirical accuracy in those bins is much higher than the predicted confidence\. This indicates that the model is poorly calibrated and systematically under\-confident\. In contrast, the full CIM produces a much wider confidence distribution, with mean0\.8780\.878and standard deviation0\.2520\.252, and reduces ECE to0\.0870\.087\. Its average confidence follows the empirical accuracy much more closely across bins, especially in the high\-confidence region where most examples are located\.

These results support the role ofℒconf\\mathcal\{L\}\_\{\\mathrm\{conf\}\}in CITA\. The confidence head is not merely an auxiliary output\. Instead, it learns a useful reliability signal that better reflects whether a value prediction is likely to be correct\. This finding is consistent with the ablation results in the main text, where removing confidence\-based scoring leads to lower Tool F1 and task success\.

Figure 5:Confidence calibration of CIM\. We compare a CIM variant trained without the confidence lossℒconf\\mathcal\{L\}\_\{\\mathrm\{conf\}\}and the full CIM on held\-out context\-action pairs\. In the upper row, blue bars show empirical accuracy in each confidence bin, the red line shows average predicted confidence, and the dashed diagonal indicates perfect calibration\. In the lower row, histograms show the distribution of predicted confidence scores\. The full CIM achieves much better calibration, reducing ECE from 0\.460 to 0\.087\.
## Appendix LComputational Cost

This section reports the computational cost of CITA during inference\. All measurements are conducted on the same hardware with 2 H100 GPUs\. We measure wall\-clock inference time per task, excluding one\-time model loading and dataset I/O, but including decoding, candidate generation, CIM scoring, tool execution, and context updates\. All methods use the same held\-out split, tool simulator, maximum step budget, and evaluation protocol\.

#### Compared methods\.

We compare representative methods with different inference patterns\. SFT uses direct greedy decoding after supervised fine\-tuning\. ToolRL and StepTool use RL\-tuned policies with greedy decoding at inference time\. WebDreamer represents planning\-based inference with online future\-state simulation\. WKM augments the policy with global and local planning knowledge\. CITA samples multiple candidate tool invocations at each step, scores them with the frozen CIM, and executes the highest\-scoring invocation\. In the main setting, CITA usesK=3K=3candidates per decision step\.

#### Cost metrics\.

Besides inference time per task, we also report relative inference time and time per successful task\. The latter measures the expected wall\-clock time needed to obtain one successful trajectory:

TimePerSuccess=TimePerTaskTSR/100\.\\mathrm\{TimePerSuccess\}=\\frac\{\\mathrm\{TimePerTask\}\}\{\\mathrm\{TSR\}/100\}\.\(26\)This metric accounts for both efficiency and task success, and is therefore more informative than raw latency alone for long\-horizon tool\-use agents\.

Table 9:Inference cost and task success on the same 2 H100 hardware\. TSR is averaged over the six dataset\-backbone settings\. Time per successful task is computed from average inference time and TSR\.
#### Analysis\.

Table[9](https://arxiv.org/html/2610.02330#A12.T9)shows that CITA introduces moderate inference overhead compared with greedy decoding\. This overhead comes from sampling multiple candidate tool invocations and scoring them with CIM at each decision step\. However, the additional cost is small compared with methods that perform online future\-state simulation, such as WebDreamer\. CITA is2\.46×2\.46\\timesfaster than WebDreamer in average inference time, because CIM directly estimates next\-tool value with a trained value model instead of repeatedly simulating future trajectories with an LLM planner\.

The higher success rate also changes the cost comparison\. Although CITA is slower than SFT, ToolRL, and StepTool in raw inference time, it achieves a much higher average TSR\. As a result, CITA obtains the lowest time per successful task, reducing the expected cost of obtaining a successful trajectory to 9\.74 seconds\. This indicates that the additional CIM scoring cost is compensated by the improved decision quality\.

#### Offline cost\.

CITA also introduces offline costs for paired data construction and CIM training\. The Bayesian simulator pairs are inexpensive because they are generated on the tool transition graph without executing real APIs\. The LLM\-based comparative pairs are more costly, but they are constructed offline and reused throughout CIM training and policy optimization\. At inference time, CITA does not call an LLM judge or perform LLM\-based comparative rollout imagination\. It only uses the trained CIM to score candidate invocations\. This amortizes the cost of comparative supervision and keeps online inference substantially cheaper than planning methods that simulate future states at every decision step\.

#### Scalability\.

The main online overhead of CITA scales linearly with the number of sampled candidatesKKand the number of decision steps\. In practice, we use a small candidate budget,K=3K=3, which is sufficient to provide alternatives for CIM comparison while keeping latency moderate\. TheK=1K=1ablation in Appendix[D](https://arxiv.org/html/2610.02330#A4)shows that removing candidate comparison hurts performance, confirming that a small amount of additional computation is useful for long\-horizon tool selection\.

## Appendix MCase Study: CIM\-Guided Tool Selection

We provide two qualitative examples from Toolathlon with the Qwen2\.5\-7B backbone\. Each example compares greedy SFT decoding with CITA at a critical decision point\. For CITA, the actor proposesK=3K=3candidate next\-tool invocations, and CIM scores each candidate by its value estimate and confidence:

Sϕ​\(s,a\)=Vϕ​\(s,a\)​cϕ​\(s,a\)\.S\_\{\\phi\}\(s,a\)=V\_\{\\phi\}\(s,a\)c\_\{\\phi\}\(s,a\)\.\(27\)The selected invocation is the candidate with the highestSϕ​\(s,a\)S\_\{\\phi\}\(s,a\)\.

### M\.1Example 1: Canvas Quiz Completion

#### Task\.

“My personal information is stored in memory\. Check which unfinished course quizzes I have on Canvas, and help me complete them\.”

#### Reference path\.

health\_check→\\rightarrowlist\_courses→\\rightarrowlist\_quizzes→\\rightarrowstart\_quiz\_attempt→\\rightarrowlist\_quiz\_questions→\\rightarrowsubmit\_quiz\_answers\.

This task has a strict prerequisite: the agent must start a quiz attempt before it can list questions or submit answers\. The SFT baseline correctly invokeshealth\_check,list\_courses, andlist\_quizzes, but then jumps directly tolist\_quiz\_questions\. Because no active quiz attempt exists, the subsequent quiz operations fail\.

Table 10:CIM scores at the critical fork point in the Canvas quiz example\.CIM assigns the highest score tostart\_quiz\_attempt, the prerequisite\-enabling tool\. Althoughlist\_quiz\_questionsis locally plausible after identifying an unfinished quiz, its low value reflects that it leads to an invalid state without an active attempt\. This example shows how CIM\-guided tool selection can avoid a locally plausible but downstream\-invalid invocation\.

### M\.2Example 2: Trip Planning with Multiple Services

#### Task\.

“I am currently at the UPenn Bookstore\. I would like to take a walking tour of campus attractions\. A few days ago, I saved the list of sights in a workspace file\. Please read it, plan a route, and write the result\.”

#### Reference path\.

read\_file→\\rightarrowgeocode→\\rightarrowdistance\_matrix→\\rightarrowdirections→\\rightarrowwrite\_file→\\rightarrowclaim\_done\.

This task requires cross\-service coordination\. The agent must first read the saved sights, then convert locations into coordinates before computing distances and directions\. The SFT baseline reads the file but skipsgeocode, then invokes map tools with raw place names\. This makes the following location\-based invocations unreliable and prevents the agent from writing a complete route plan\.

Table 11:CIM scores at the key map\-planning fork point in the trip planning example\.CIM selectsgeocodebefore route planning, assigning it a higher value thansearch\_placesordirections\. This reflects the long\-horizon dependency that coordinates are needed before distance and direction queries become reliable\. After geocoding, CITA proceeds withdistance\_matrix,directions, andwrite\_file, producing a coherent tool\-use trajectory\.

Both examples illustrate the same pattern\. Greedy decoding selects tools that are locally plausible but violate downstream dependencies\. CIM\-guided tool selection instead compares candidate next\-tool invocations by their estimated long\-horizon value under the current context\. Prerequisite\-enabling tools receive higher value and confidence, while candidates that lead to dead\-end states receive lower combined scores\. This helps CITA connect step\-level tool selection with final task success\.

## Appendix NBroader Impact and Responsible Use

CITA aims to improve the reliability of long\-horizon tool\-use agents by helping them compare candidate next\-tool invocations before execution\. More reliable tool selection may reduce unnecessary invocations, improve task completion, and make agent behavior easier to inspect through value and confidence scores\.

At the same time, stronger tool\-use agents should be deployed with appropriate safeguards\. External tools may access files, APIs, databases, or other stateful environments, where incorrect invocations can have undesirable effects\. CITA improves decision quality, but it does not replace environment\-level safety controls\. For real deployment, tool permissions, sandboxing, rate limits, execution logs, and human confirmation for irreversible or high\-impact actions remain important\.

Our experiments are conducted in benchmarked tool environments rather than real\-world high\-stakes deployments\. For broader use, CIM\-guided tool selection should be combined with policies that restrict sensitive tools, validate arguments before execution, and monitor agent behavior over time\.