Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

# 1Introduction
Source: [https://arxiv.org/html/2608.17499](https://arxiv.org/html/2608.17499)
Towards Better Agents for Multi\-Turn User Interaction: The Next User Turn Is More Than Context

Yiwen Zhao1,2\*Zhihao Wen2\*Yuchen Mao2\*Mingxuan Jiang1Yihao Hu2Pan Wang2Xin Zhang2Wei Wu2

1Fudan University2Ant International, Ant Group

Correspondence: z\.wen@antgroup\.com

AbstractUser\-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns\. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair\. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user\-to\-user segment\. We introduceFeedback\-AwareCreditAssignment \(Faca\), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout\. Against an outcome\-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization,Facaimproves the nine\-domainτ\\tau\-family average across three independently trained runs by 5\.91 and 10\.22 percentage points at 8B and 14B, respectively\. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain\. The same ordering holds zero\-shot on Pare\-Bench and Co\-Gym\. These results demonstrate that next\-turn user reactions provide actionable local credit for improving multi\-turn user\-interacting agents\.Keywords:LLM agents, multi\-turn interaction, reinforcement learning, credit assignment, user simulation

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.17499v1/intro_u2u_1.png)Figure 1:Multi\-turn tool interaction\. A user\-to\-user segment contains all agent responses, tool calls, and tool results between two adjacent user turns; the ellipsis denotes continued interaction\.Tool\-using language agents are moving from single\-turn assistants toward user\-facing systems that operate reliably and safely over extended conversations\. A travel agent may need to inspect a reservation, elicit missing preferences, compare options, obtain authorization, and only then modify the database\. Unlike fully specified tasks that provide all relevant information upfront, these interactions require the agent to discover user intent while acting in an external environment under evolving constraints\. Goals may be omitted initially or revealed only after the agent elicits them, making the user an active participant rather than a fixed input\([29](https://arxiv.org/html/2608.17499#bib.bib1);[18](https://arxiv.org/html/2608.17499#bib.bib13);[1](https://arxiv.org/html/2608.17499#bib.bib2)\)\.

Figure[1](https://arxiv.org/html/2608.17499#S1.F1)illustrates this structure\. Between adjacent user turnsutu\_\{t\}andut\+1u\_\{t\+1\}, the agent may produce several messages and tool interactions before returning control\. We call this block a*user\-to\-user \(U2U\) segment*\. The next user turn may provide information, approve an action, reject a proposal, or correct a misunderstanding\. Success therefore requires coordinating dialogue, tool use, and user decisions across segments throughout the evolving interaction process\.

Interactive benchmarks in theτ\\taufamily expose these demands\([29](https://arxiv.org/html/2608.17499#bib.bib1);[1](https://arxiv.org/html/2608.17499#bib.bib2);[23](https://arxiv.org/html/2608.17499#bib.bib10)\)\. Training frameworks such as MUA\-RL and UserRL go further by placing an LLM\-simulated user inside reinforcement\-learning rollouts\([35](https://arxiv.org/html/2608.17499#bib.bib11);[19](https://arxiv.org/html/2608.17499#bib.bib14)\)\. This shift lets the policy explore complete conversations while the final database or world state provides objective success supervision\. Outcome\-only credit accommodates many valid trajectories, but collapses their internal structure: effective elicitation, errors, and later repair receive the same advantage\. The reward says*whether*a trajectory succeeded, but not*where*it changed course\.

Our central observation is thatthe next user turn is more than context\. Prospectively, it guides the next decision; retrospectively, it provides local evidence about the preceding U2U segment\. Supplying requested information can indicate effective elicitation; challenges or clarifications can expose friction\. These context\-dependent reactions are not correctness labels, yet outcome\-only training leaves them unused\. Can this process evidence improve local credit while verified task completion remains the outcome objective?

We introduce Feedback\-Aware Credit Assignment \(Faca\)\.Facaaligns each next\-user reaction with the preceding U2U segment, normalizes reactions within a local rollout group, and adds the resulting process advantage to terminal outcome advantage\.It changes only credit assignment:Facaand the matched outcome\-only Interactive GRPO control share the frozen simulator, agent\-visible dialogue, initialization, rollout, and optimization, while private reaction metadata remains hidden from the agent\.

Across three independently trained runs per scale,Facaimproves the nine\-domainτ\\tau\-family average by5\.91and10\.22percentage points at 8B and 14B, respectively\. Effects are heterogeneous, with the largest gains in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain\.Faca\-trained policies also outperform matched controls zero\-shot on Pare\-Bench and Co\-Gym\. These results support a conditional benefit when user reactions carry informative local structure, rather than universal agent improvement\.

Overall, our main contributions are as follows:

- •Formulation\.We formalize U2U segments as local credit\-assignment units, and subsequent user reactions as temporally localized evidence of interaction progress\.
- •Method\.We introduceFaca, which adds U2U\-level reaction credit without a learned critic, additional rollouts, or agent\-visible labels\.
- •Findings\.Results show cross\-scale gains, reaction sensitivity, zero\-shot transfer, and domain variation\.

## 2Related Work

##### Interactive user\-agent learning\.

Modern user\-facing agents solve practical tasks through iterative communication, tool use, and clarification of underspecified goals\([17](https://arxiv.org/html/2608.17499#bib.bib9);[1](https://arxiv.org/html/2608.17499#bib.bib2);[18](https://arxiv.org/html/2608.17499#bib.bib13);[35](https://arxiv.org/html/2608.17499#bib.bib11)\)\. Work spans offline trajectory synthesis and online simulator\-in\-the\-loop RL\. Offline methods construct social or tool\-use interactions for subsequent training, including SOTOPIA\-π\\pi/Ω\\Omega, APIGen\-MT, and Magnet\([25](https://arxiv.org/html/2608.17499#bib.bib3);[32](https://arxiv.org/html/2608.17499#bib.bib4);[16](https://arxiv.org/html/2608.17499#bib.bib5);[30](https://arxiv.org/html/2608.17499#bib.bib6)\)\. LAM SIMULATOR and Simia\-SFT instead generate trajectories through interactive exploration or seed\-set expansion\([7](https://arxiv.org/html/2608.17499#bib.bib7);[11](https://arxiv.org/html/2608.17499#bib.bib8)\)\. Online methods generate interactions against the current policy: MUA\-RL and UserRL retain simulated users inside multi\-turn RL rollouts, while Simia\-RL uses model\-simulated environment feedback during policy optimization\([35](https://arxiv.org/html/2608.17499#bib.bib11);[19](https://arxiv.org/html/2608.17499#bib.bib14);[11](https://arxiv.org/html/2608.17499#bib.bib8)\)\. They place simulated user behavior directly inside the learning loop\. This makes interaction data responsive to the policy’s evolving behavior\.

![Refer to caption](https://arxiv.org/html/2608.17499v1/dense_method_highlight_0803.png)Figure 2:FACA overview\.\(a\)For each prompt, we sampleKKmulti\-turnuser\-interactingrollouts and obtain a verified final task reward\.\(b\)A U2U segment may contain multiple agent and tool spans; the next user call jointly produces avisibleutterance andprivatereaction strategy, which is aligned with the complete preceding segment\.\(c\)Terminal rewards yieldtrajectory\-leveloutcome advantages, while aligned reactions yieldturn\-levelprocess advantages\. Their additive combination supplies the final advantage for agent optimization\.
##### Credit assignment in multi\-turn agents\.

Terminal task rewards verify overall success but cannot identify which turns elicited useful information, introduced errors, or enabled recovery\. Prior work obtains turn\-level credit from learned critics or intermediate evaluators\([37](https://arxiv.org/html/2608.17499#bib.bib21);[36](https://arxiv.org/html/2608.17499#bib.bib22);[2](https://arxiv.org/html/2608.17499#bib.bib24);[26](https://arxiv.org/html/2608.17499#bib.bib25)\)\. Agent Lightning instead decomposes trajectories into transitions and converts monitoring signals into intermediate rewards\([13](https://arxiv.org/html/2608.17499#bib.bib23)\)\. Critic\-free alternatives derive local advantages from continuations, repeated states, policy information, outcome potentials, structured reasoning events, or tool\-call entropy\([5](https://arxiv.org/html/2608.17499#bib.bib29);[39](https://arxiv.org/html/2608.17499#bib.bib30);[4](https://arxiv.org/html/2608.17499#bib.bib26);[24](https://arxiv.org/html/2608.17499#bib.bib31);[10](https://arxiv.org/html/2608.17499#bib.bib32);[8](https://arxiv.org/html/2608.17499#bib.bib33);[33](https://arxiv.org/html/2608.17499#bib.bib27);[12](https://arxiv.org/html/2608.17499#bib.bib28);[38](https://arxiv.org/html/2608.17499#bib.bib34);[27](https://arxiv.org/html/2608.17499#bib.bib38)\)\. SEAL instead uses verifier\-grounded diagnoses to reweight trajectory\-level GRPO advantages\([9](https://arxiv.org/html/2608.17499#bib.bib39)\)\. User follow\-ups provide complementary supervision: FLR scores candidate responses using the likelihood of curated positive and negative follow\-ups, while RLUF learns from production reactions and documents reward hacking under over\-optimization\([31](https://arxiv.org/html/2608.17499#bib.bib15);[6](https://arxiv.org/html/2608.17499#bib.bib16)\)\. Unlike these response\-level objectives,Facauses the partner’s next reaction as credit for the immediately preceding U2U segment, alongside terminal task advantage\. The reaction already occurs inside the same rollout, so the method needs neither additional continuations nor a learned turn\-level evaluator\. Its private strategy annotation is used only to construct credit and remains hidden from the agent\. Accordingly, the signal is neither policy\-derived uncertainty nor a reward inferred from latent simulator state\. This positionsFacanarrowly as reaction\-grounded local U2U supervision, not a hierarchical credit\-assignment framework\.

## 3Feedback\-Aware Credit Assignment

### 3\.1Interactive Rollout and Outcome Credit

For promptxx, an agent policyπθ\\pi\_\{\\theta\}interacts with a tool environment and a frozen user policyπu\\pi\_\{u\}\. Group\-relative optimization\([22](https://arxiv.org/html/2608.17499#bib.bib19)\)samplesKKrollouts,

τk=\(x,ak,1,uk,2,…,ak,Tk,uk,Tk\+1\),\\tau\_\{k\}=\(x,a\_\{k,1\},u\_\{k,2\},\\ldots,a\_\{k,T\_\{k\}\},u\_\{k,T\_\{k\}\+1\}\),\(1\)whereak,qa\_\{k,q\}is the complete U2U segment between adjacent user messages and may contain multiple language and tool spans\. The environment returns verified terminal rewardRk∈\{0,1\}R\_\{k\}\\in\\\{0,1\\\}from the final database or world state\. Each U2U unit therefore preserves the full sequence of language and tool actions produced before control returns to the user\. Our matched outcome\-only Interactive GRPO control computes:

Ako=Rk−μR​\(x\)σR​\(x\)\+ϵ,A^\{\\mathrm\{o\}\}\_\{k\}=\\frac\{R\_\{k\}\-\\mu\_\{R\}\(x\)\}\{\\sigma\_\{R\}\(x\)\+\\epsilon\},\(2\)and broadcastsAkoA^\{\\mathrm\{o\}\}\_\{k\}to every trainable agent token in trajectorykk\. It shares the user policy, simulator prompt, agent\-visible dialogue, rollout, and optimization withFaca\. Components remain matched throughout training\. This terminal signal accommodates diverse successful trajectories but is temporally flat: useful elicitation, errors, and later repair receive the same advantage\. Moreover, a group with constant terminal rewards hasAko=0A^\{\\mathrm\{o\}\}\_\{k\}=0\.Facaretains this verified outcome branch and adds reaction\-grounded credit at the U2U level\.

### 3\.2User Reaction as Local Evidence

For both Interactive GRPO andFacarollouts, the frozen simulator emits a visible utteranceuk,q\+1u\_\{k,q\+1\}and private behavioral strategyzk,q\+1z\_\{k,q\+1\}in the same call\. Only the utterance is appended to the agent context\. Interactive GRPO discardszk,q\+1z\_\{k,q\+1\};Facareads it after rollout construction and aligns it to the immediately preceding agent segmentak,qa\_\{k,q\}\. Temporal adjacency is an inductive bias rather than a claim that every reaction is caused only by that segment\. The strategy is read only after the agent yields control, so it never enters the agent\-visible context\. We map the strategy to ternary evidence:

zk,q\+1→fk,q∈\{−1,0,\+1\}\.z\_\{k,q\+1\}\\rightarrow f\_\{k,q\}\\in\\\{\-1,0,\+1\\\}\.\(3\)Confirming, closing, or revealing specifically requested information is treated as*progress\-consistent*; asking for clarification, challenging a solution, or changing the stated goal is*friction\-consistent*; vague or invalid output is assigned neutral credit\. These labels describe interaction movement rather than user sentiment or action correctness\. This coarse map preserves ambiguity while retaining the reaction’s direction\. Table[1](https://arxiv.org/html/2608.17499#S3.T1)gives the complete mapping, Appendix[B](https://arxiv.org/html/2608.17499#A2)details extraction and fallback behavior, and Appendix[C](https://arxiv.org/html/2608.17499#A3)gives the shared simulator prompt\. An utterance\-only extractor could remove reliance on private reaction metadata and extendFacabeyond the instrumented setting toward deployment in less controlled environments\.

Table 1:User\-strategy mapping used byFaca\. Polarity represents coarse evidence of interaction progress or friction, not sentiment or verified task correctness\.
### 3\.3Two\-Level Additive Advantage

The outcome branch reusesAkoA^\{\\mathrm\{o\}\}\_\{k\}from Equation[2](https://arxiv.org/html/2608.17499#S3.E2)\. For promptxxand U2U indexqq, letℐ⁡\(x,q\)\\mathcal\{I\}\(x,q\)contain rollouts that reachqqand have a valid next\-user reaction\. We construct the aligned reaction group:

ℬp​\(x,q\)=\{fi,q∣i∈ℐ⁡\(x,q\)\},\\mathcal\{B\}^\{p\}\(x,q\)=\\\{f\_\{i,q\}\\mid i\\in\\mathcal\{I\}\(x,q\)\\\},\(4\)and normalize within that local comparison set:

Ak,qp=fk,q−μf​\(x,q\)σf​\(x,q\)\+ϵ\.A^\{\\mathrm\{p\}\}\_\{k,q\}=\\frac\{f\_\{k,q\}\-\\mu\_\{f\}\(x,q\)\}\{\\sigma\_\{f\}\(x,q\)\+\\epsilon\}\.\(5\)The U2U estimator uses only the immediate next reaction and does not propagate later reactions backward\. Singleton or constant anchors receive zero\. Every trainable token inak,qa\_\{k,q\}receives:

Ak,q=Ako\+λ​Ak,qp\.A\_\{k,q\}=A^\{\\mathrm\{o\}\}\_\{k\}\+\\lambda A^\{\\mathrm\{p\}\}\_\{k,q\}\.\(6\)The two terms are complementary:AkoA^\{\\mathrm\{o\}\}\_\{k\}preserves the terminal objective, whereasAk,qpA^\{\\mathrm\{p\}\}\_\{k,q\}distinguishes local progress from friction\. Disabling the reaction branch reduces Equation[6](https://arxiv.org/html/2608.17499#S3.E6)to the strict outcome\-only Interactive GRPO control\. In an outcome\-homogeneous group, reaction labels can still differ across rollouts; this is a mechanical property rather than an independent performance claim\.

### 3\.4Optimization and Anchoring

For assistant tokens, we first clip the GRPO importance ratio\([20](https://arxiv.org/html/2608.17499#bib.bib20)\):

ρ~k,t=clip⁡\(ρk,t,1−ε,1\+ε\)\.\\widetilde\{\\rho\}\_\{k,t\}=\\operatorname\{clip\}\\\!\\left\(\\rho\_\{k,t\},\\,1\-\\varepsilon,\\,1\+\\varepsilon\\right\)\.\(7\)Substituting the U2U\-level advantageAk,qA\_\{k,q\}then gives the actor objective:

ℒ⁡\(θ\)=−𝔼k,t​\[min⁡\(ρk,t​Ak,q⁡\(t\),ρ~k,t​Ak,q⁡\(t\)\)\]\.\\displaystyle\\mathcal\{L\}\(\\theta\)=\-\\mathbb\{E\}\_\{k,t\}\\\!\\left\[\\min\\\!\\left\(\\rho\_\{k,t\}A\_\{k,q\(t\)\},\\widetilde\{\\rho\}\_\{k,t\}A\_\{k,q\(t\)\}\\right\)\\right\]\.

\(8\)Generated language and tool\-call tokens are trainable; user and raw tool\-result tokens are masked\. Because reaction metadata is produced in the existing user call, the estimator needs no learned critic, separate evaluator, or additional rollout\.

We use ordinal U2U indexqqas the default anchor because exact dialogue states rarely repeat\. This approximation can compare different semantic phases after trajectories diverge, especially at late singleton turns\. One\-U2U\-shifted reactions test temporal adjacency, while random polarity tests whether reaction semantics matter\. Equation[6](https://arxiv.org/html/2608.17499#S3.E6)intentionally changes the training signal and is not potential\-based policy\-invariant shaping\([15](https://arxiv.org/html/2608.17499#bib.bib35)\)\. The pre\-normalization reward scale cancels under z\-normalization; the effective process\-strength parameter isλ\\lambda\. BecauseApA^\{\\mathrm\{p\}\}is derived from a simulator reaction rather than the verified terminal state,λ\\lambdaalso limits proxy dominance\. We cap its positive value at0\.50\.5so that reaction credit remains optimization\-relevant whileAoA^\{\\mathrm\{o\}\}stays dominant in aggregate, reducing, but not eliminating, reward\-hacking risk from simulator bias or misclassified feedback\.

## 4Experiments

Table 2:Strict pass@1 \(%\) across nine domains in theτ\\tau\-bench family\. RL scores average three independently trained step\-120 runs per method and scale; each seed is evaluated once under one shared protocol\. Base and SFT are fixed references\. Avg\. weights all nine domains equally\. Full nominal task sets are used, and missing cases count as failures\. Higher is better\. “\+” marks a training stage; the two RL rows are alternative continuations of the same SFT\. Values use one decimal; bold marks column bests within each scale, including ties\.### 4\.1Datasets

##### τ\\tau\-bench family\.

Our main evaluation suite contains nine domains: Airline and Retail fromτ\\tau\-bench; Airline, Retail, and Telecom fromτ2\\tau^\{2\}\-bench; and Airline, Retail, Telecom, and Bank fromτ3\\tau^\{3\}\-bench\([29](https://arxiv.org/html/2608.17499#bib.bib1);[1](https://arxiv.org/html/2608.17499#bib.bib2);[23](https://arxiv.org/html/2608.17499#bib.bib10)\)\. These benchmarks require agents to resolve multi\-turn user requests through tool use and communication\. Telecom is a mixed\-control setting in which users execute device\-side actions, whileτ3\\tau^\{3\}Bank additionally requires grounding in an unstructured policy collection\. Crucially, the subsequent RL stage uses only the Airline and Retail training splits fromτ\\tau\-bench;neither Telecom nor Bank appears in RL training\. Results on these two domains therefore measure transfer beyond the RL training domains\.

##### Pare\-Bench\.

Pare\-Bench contains 143 proactive mobile scenarios that require observing user and environment events, inferring latent goals, proposing an intervention, and executing across stateful apps under evolving conditions after user acceptance\([14](https://arxiv.org/html/2608.17499#bib.bib36)\)\. It tests whether policies trained on reactive customer\-service interactions transfer to a proactive user\-agent protocol\.

##### Co\-Gym\.

Co\-Gym evaluates bidirectional collaboration under dual control and non\-turn\-taking coordination in shared workspaces\([21](https://arxiv.org/html/2608.17499#bib.bib37)\)\. Its simulated condition contains 102 Travel Planning, 100 Related Work Writing, and 110 Tabular Analysis tasks, where the agent and user can act asynchronously in the same environment\.

### 4\.2Experimental Settings

We experiment with Qwen3\-8B and Qwen3\-14B, initialized by SFT on the public MUA\-RL release\([34](https://arxiv.org/html/2608.17499#bib.bib12)\)\. From the same SFT checkpoint, we train Interactive GRPO andFacaas alternative RL continuations, with three independent training seeds per method and scale\. All headline RL results use the step\-120 checkpoints\.

The comparison isolates credit assignment\. Within each scale, the two RL arms share the frozen DeepSeek\-V4\-Flash simulator and prompt, agent\-visible utterances, SFT initialization, training data, rollout construction, optimizer, and horizon\. Interactive GRPO uses only terminal outcome advantage, whereasFacaadditionally uses the simulator’s private reaction metadata to construct U2U\-level process advantage; this metadata is never exposed to the agent\.

For theτ\\tau\-bench family, we report strict pass@1 from the verified final environment state\. Each of the three independently trained checkpoints is evaluated once under the same protocol, and the main table averages them domainwise\. Avg\. equally weights the nine domains; run\-level dispersion is reported in Section[5](https://arxiv.org/html/2608.17499#S5)\. Base and SFT are fixed pre\-RL references\. Pare\-Bench and Co\-Gym are evaluated zero\-shot without benchmark\-specific training or tuning\. Details of the SFT and RL datasets, training hyperparameters, evaluation protocols, and run provenance are provided in Appendix[D](https://arxiv.org/html/2608.17499#A4)\.

Figure 3:Matched zero\-shot transfer to Pare\-Bench under identical evaluation settings\. Solid and diagonally hatched bars report strict Pass@1 and Pass@4, respectively; colors distinguish Interactive GRPO andFacaacross both evaluated model scales\.
Table 3:Zero\-shot Co\-Gym results on a 0–100 scale\.Facaleads 13 of 16 cells\. I\-GRPO denotes Interactive GRPO; bold marks the better method\. DR, TP, and CS denote Delivery Rate, Task Performance, and overall score, respectively\.

## 5Results

### 5\.1Main Results on theτ\\tau\-Bench Family

Table[2](https://arxiv.org/html/2608.17499#S4.T2)shows that reaction\-grounded credit improves the matched outcome\-only control at both scales\. At 8B, the mean±\\pmsample standard deviation across three trained runs rises from 34\.66±\\pm0\.25 with Interactive GRPO to 40\.57±\\pm1\.04 withFaca, a5\.91\-pointgain from unrounded domain means\. At 14B, the scores rise from 42\.51±\\pm0\.12 to 52\.73±\\pm1\.53, a10\.22\-pointgain\. Matching runs by training\-seed identifier, the gains are 4\.67/6\.33/6\.75 points at 8B and 8\.62/10\.14/11\.91 points at 14B; thus*the ordering holds in all runs at both scales*\. Each checkpoint is evaluated once, so these standard deviations describe run\-level dispersion rather than isolating training variation from evaluation stochasticity\. Appendix[D](https://arxiv.org/html/2608.17499#A4)reports the run counts and provenance\. Because the two RL arms share their SFT initialization, user\-generation path, observations, and optimization setup, this comparison tests reaction\-grounded versus outcome\-only credit\. Consistent with MUA\-RL, cold\-start SFT can introduce domain\-specific biases that limit generalization beyond the SFT distribution, whereas subsequent simulator\-in\-the\-loop RL recovers and surpasses the base models\([35](https://arxiv.org/html/2608.17499#bib.bib11)\)\.

The improvement is broad but not uniform\.Facaleads Interactive GRPO on seven of nine domains at each scale, with its largest gain in all four Telecom scale–benchmark cells\. It ties one and trails one domain at 8B, and trails two Retail domains at 14B\. The main result is therefore an aggregate cross\-scale gain with a repeated Telecom\-centered pattern rather than universal domain dominance\. The larger absolute gap at 14B than at 8B does not by itself establish a scaling law from two model sizes\. The supported conclusion is narrower: the matched advantage persists across both scales despite different domain\-level regressions\.

### 5\.2Zero\-Shot Transfer

The matched ordering persists under two unseen interaction protocols\. On Pare\-Bench, strict Pass@1 rises from 6\.29% to 10\.49% at 8B and from 10\.49% to 13\.29% at 14B; Pass@4 follows the same ordering \(Figure[3](https://arxiv.org/html/2608.17499#S4.F3)\)\. All bars use the same 143\-scenario full split; each checkpoint is evaluated once, so no uncertainty interval is shown\. On Co\-Gym,Facaleads 13 of 16 scale–metric cells, including every Related Work, Tabular Analysis, and Overall cell, but Travel DR decreases slightly at both scales \(Table[3](https://arxiv.org/html/2608.17499#S4.T3)\)\. Here DR, TP, and CS denote Delivery Rate, Task Performance, and the evaluator\-reported overall score\. These comparisons test whetherFacaretains its ordering over outcome\-only training, not an absolute ranking over pretraining or SFT variants\. Appendix[A](https://arxiv.org/html/2608.17499#A1)records the full protocols, coverage, and comparison scope\.

Pare\-Bench changes initiative and stateful app execution, whereas Co\-Gym changes the coordination structure\. Their shared ordering is cross\-protocol evidence, but low absolute Pare\-Bench rates and Co\-Gym Travel regressions keep the claim comparative rather than broadly conclusive\.

### 5\.3Ablation Studies

Table[4](https://arxiv.org/html/2608.17499#S6.T4)reports a single\-run controlled ablation from the 8B s1 experiment family; every row uses the same one\-shot evaluation protocol\. Its Aligned and Outcome\-only checkpoints also appear in Table[2](https://arxiv.org/html/2608.17499#S4.T2), whose headline results average s1–s3\.Faca\(Aligned,λ=0\.5\\lambda=0\.5\) reaches 39\.37 overall and 47\.37 on Telecom, whereas one\-U2U\-shifted and randomized reactions erase the gain over outcome\-only training\. Preserving reaction frequency or introducing generic token\-varying advantages is therefore insufficient:*both reaction semantics and temporal alignment matter*\. The Shifted Avg\. counts its unfinishedτ3\\tau^\{3\}Banking tail as failures\.

Among aligned settings, a small positive weight improves overλ=0\\lambda=0, and the defaultλ=0\.5\\lambda=0\.5performs best: relative to Outcome\-only, it adds 4\.67 points to Avg\. and 17\.11 points to Tel\.2\. Reversing the sign sharply degrades both aggregates, while Shifted and Randomized credit fall below Outcome\-only\. Together, these controls show that token\-level variation alone is insufficient; the signal must preserve reaction content and its temporal association with the preceding agent span\. The weight sensitivity further supports reaction credit as an auxiliary, rather than alternative, objective\.

The control gaps sharpen this interpretation\. Relative to Outcome\-only, Aligned withλ=0\.1\\lambda=0\.1adds 2\.08 points to Avg\. and 15\.35 points to Tel\.2, whileλ=0\.5\\lambda=0\.5raises these gains to 4\.67 and 17\.11 points\. Both controls also lose the Telecom gain, reinforcing the need for signed, temporally matched credit\. These margins show that alignment improves both the broad average and the feedback\-rich Telecom subset\.

## 6Analysis

### 6\.1When Are User Reactions Informative?

The same reaction label need not be equally informative in every task\. We use*reaction observability*descriptively for the extent to which the next user response exposes a state change caused by the preceding U2U segment\. When progress occurs mostly inside tools or policy checks, the user may not observe whether an action was correct\. When control alternates between agent and user, the user’s report can instead reveal whether an instruction worked, whether validation failed, and what state remains unresolved\. The cross\-domain pattern in Table[2](https://arxiv.org/html/2608.17499#S4.T2)is consistent with this distinction\.

Telecom is particularly feedback\-rich\. Its tasks repeatedly alternate identity grounding, hidden device\-state inspection, agent\-side operations, user\-executed actions, validation, revised diagnosis, and final verification\. The next user turn therefore often reports the direct consequence of the preceding segment: progress\-consistent reactions can reinforce effective elicitation and repair, while friction\-consistent reactions can localize an unclear instruction or failed proposal\. This interaction structure offers a concrete account of why all four Telecom cells show the largest gains under the same estimator\.

The OOD results extend this account without making it universal\. Pare\-Bench changes from reactive customer service to proactive mobile assistance, while Co\-Gym introduces bidirectional, non\-turn\-taking collaboration\.Facaretains its ordering over the outcome\-only control at both scales, but Co\-Gym’s Travel regression shows that adding reaction credit is not uniformly beneficial\. Together, these results suggest that the advantage transfers when subsequent user behavior remains informative about task progress, rather than merely when a task is long or interactive\.

This account predicts more than a generic benefit from longer conversations\. Additional turns create more possible credit locations, but they help only when the next user response reveals a consequence of the preceding segment\. Reaction observability, rather than interaction length, is the more direct hypothesis suggested by this pattern\.

Table 4:Single\-run controlled 8B credit ablation across theτ\\tau\-bench family under the shared one\-shot evaluation protocol\. Avg\. is the nine\-domain average; Tel\.2averagesτ2\\tau^\{2\}andτ3\\tau^\{3\}Telecom\.
### 6\.2Training\-Signal Dynamics

Per\-U2U telemetry separates favorable reactions from effective differential credit\. A matched 8B replication records all 154,219 U2U segments with exact agreement between telemetry and tensor counts\. From the first to the last 40 steps, positive reactions rise from 69\.95% to 82\.51%, driven in part by moreconfirmand lessbe\_vague\. Over the same period, nonzero normalized process\-advantage coverage decreases from 86\.16% to 82\.50%\. Figure[4](https://arxiv.org/html/2608.17499#S6.F4)shows segment\-weighted phase averages; its dashed coverage series uses the truncated 70–95% right axis\. Favorable reactions are therefore not copied one\-for\-one into reaction credit: anchor\-relative normalization retains only differences within the local comparison set\. The opposing trends also distinguish signal prevalence from signal discrimination: feedback can become more favorable even as local rollout groups provide fewer nonzero reaction contrasts\.

Terminal credit is also silent for a substantial share of prompt groups\. The step\-120 logs contain 1,920 group\-steps at each scale, of which 39\.27% at 8B and 31\.04% at 14B are all\-correct or all\-wrong and consequently have zero group\-normalized outcome advantage\. Across training, all\-correct groups become more frequent while all\-wrong groups decline \(Figure[5](https://arxiv.org/html/2608.17499#S6.F5)\)\. These groups identify where reaction credit can supply differential signal; their prevalence alone does not establish that this signal caused the final performance gain\.

Figure 4:Matched 8BFacareplication under identical training conditions: per\-U2U reaction polarity and nonzero process\-advantage coverage across 120 steps over three phases\.At the logged advantage scale, the process branch remains auxiliary\. Withλ=0\.5\\lambda=0\.5, it accounts for 36\.00% and 33\.79% of a pre\-optimization L1 advantage\-magnitude proxy at 8B and 14B, and its weighted magnitude exceeds the outcome component in only 6/120 and 1/120 steps\. Appendix[E](https://arxiv.org/html/2608.17499#A5)reports the phasewise curve \(Figure[6](https://arxiv.org/html/2608.17499#A5.F6)\), segment coverage, and the limits of this proxy\. These diagnostics show that the branch is active without dominating terminal credit, but they are descriptive rather than a decomposition of optimizer updates or causal performance gains\.

These dynamics support auxiliary reaction credit alongside verified outcomes\. In homogeneous groups, terminal normalization provides no differential signal; combining it with reaction credit introduces contrasts while preserving task completion as the governing objective\. Its bounded contribution throughout training confirms that the process branch remains active without displacing primary outcome supervision\.

### 6\.3Trajectory\-Level Evidence from Telecom

We pair the final 14B s1 checkpoints task by task in their single run\-level evaluation\. Onτ2\\tau^\{2\}Telecom,Facasucceeds on 96/114 tasks versus 30/114 for Interactive GRPO; the paired cells are 23 both\-success, 73 FACA\-only, 7 outcome\-only, and 11 both\-fail\. Onτ3\\tau^\{3\}, the corresponding totals are 95/114 versus 58/114, with cells 47, 48, 11, and 8\. The split is not uniform:Facatrails 20/29 versus 24/29 onτ3\\tau^\{3\}Service while leading on Mobile and MMS\. Appendix[F](https://arxiv.org/html/2608.17499#A6)reports issue\-family counts, paired tests, and the complete representative trace\.

Figure 5:Change in outcome\-homogeneous group rates from the first to the last 20 training steps\. Colors distinguish all\-correct and all\-wrong groups; hollow markers denote steps 1–20 and filled markers steps 101–120 at both model scales\.A matched Hard mobile\-data trace illustrates one recovery pattern, not a general length or multi\-turn advantage\.Facareaches verified success in 48 stored events versus 54 for Interactive GRPO\. For context, the two traces contain 18 versus 21 assistant messages and 15 versus 17 tool calls, a modest rather than large difference in interaction count\. Both attempt user\-only tools from the agent side\. In this trajectory,Facarecovers from two rejected calls and returns device control to the user, who completes four required state changes and a 275 Mbps test\. Interactive GRPO accumulates five such rejections; its final agent\-side speed\-test attempt reaches the error limit before terminal verification\.

Across FACA\-only wins, error\-budget exhaustion accounts for 68/73 outcome\-only failures onτ2\\tau^\{2\}and 28/48 onτ3\\tau^\{3\}; the rest stop without task closure\. This supports a descriptive account in which policies can differ in recovering from tool\-authority errors and completing terminal validation, but it does not identify reaction credit as the sole cause: both policies make authority errors, the failure depends partly on a fixed error budget, the trace is one task, and the current checkpoint still loses paired cases and theτ3\\tau^\{3\}Service slice\. Taken together, the domain pattern, telemetry, and final\-checkpoint trajectory evidence support a conditional mechanism claim rather than universal improvement\.

## 7Conclusion

The next user turn provides noisy evidence about the preceding U2U segment\.Facaconverts this reaction into locally normalized process credit and combines it with verified terminal credit, without changing the frozen simulator, exposing labels to the agent, training a critic, or adding rollouts\. Under a strict outcome\-only Interactive GRPO control,Facaimproves the nine\-domainτ\\tau\-family average at both 8B and 14B and retains this ordering on two zero\-shot interaction protocols\. Gains concentrate in feedback\-rich Telecom, while domain regressions and temporal controls show that the benefit depends on informative, aligned reactions\.Facatherefore demonstrates that implicit user feedback can provide actionable local credit for multi\-turn user\-interacting agents while verified task completion remains the governing objective\. More broadly, these results suggest that implicit user reactions can provide practical supervision for improving long\-horizon learning in interactive agents\.

## Limitations

Facais evaluated with a frozen DeepSeek\-V4\-Flash user simulator that emits private strategy metadata alongside each utterance\. We treat this metadata as heuristic evidence for the preceding U2U; event\-level causal attribution and recovery from observable utterances or real users remain open\. Evidence covers Qwen3 8B/14B agents, nineτ\\tau\-family domains, and zero\-shot Pare\-Bench/Co\-Gym evaluation\. Results establish aggregate gains over outcome\-only training across scales and both transfer benchmarks, strongest in feedback\-rich Telecom tasks\. Validation across simulator families, architectures, and real users, as well as diverse deployment conditions, remains future work\.

## References

- V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: Evaluating Conversational Agents in a Dual\-Control Environment\.arXiv preprint arXiv:2506\.07982\.External Links:[Link](https://arxiv.org/abs/2506.07982)Cited by:[§1](https://arxiv.org/html/2608.17499#S1.p1.1),[§1](https://arxiv.org/html/2608.17499#S1.p3.1),[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.17499#S4.SS1.SSS0.Px1.p1.1)\.
- Choudhury \(2025\)S\. ChoudhuryProcess Reward Models for LLM Agents: Practical Framework and Directions\.arXiv preprint arXiv:2502\.10325\.External Links:[Link](https://arxiv.org/abs/2502.10325)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- DeepSeek\-AIet al\.\(2026\)DeepSeek\-AIet al\.DeepSeek\-V4: Towards Highly Efficient Million\-Token Context Intelligence\.arXiv preprint arXiv:2606\.19348\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2606.19348),[Link](https://arxiv.org/abs/2606.19348)Cited by:[§D\.1](https://arxiv.org/html/2608.17499#A4.SS1.p2.1)\.
- Fenget al\.\(2025\)L\. Feng, Z\. Xue, T\. Liu, and B\. AnGroup\-in\-Group Policy Optimization for LLM Agent Training\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2505.10978)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)Y\. Guo, L\. Xu, J\. Liu, D\. Ye, and S\. QiuSegment Policy Optimization: Effective Segment\-Level Credit Assignment in RL for Large Language Models\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2025/hash/a6536243037d1e32c20de85137d478da-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Hanet al\.\(2025\)E\. Han, J\. Chen, K\. A\. Sankararaman, X\. Peng, T\. Xu, E\. Helenowski, K\. Peng, M\. Kumar, S\. Wang, H\. Fang, and A\. TalebzadehReinforcement Learning from User Feedback\.arXiv preprint arXiv:2505\.14946\.External Links:[Link](https://arxiv.org/abs/2505.14946)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Hoanget al\.\(2025\)T\. Hoang, K\. Huang, S\. Kokane, J\. Zhang, Z\. Liu, M\. Zhu, J\. Grigsby, T\. Lan, M\. S\. Ryoo, C\. Wu, S\. Heinecke, H\. Wang, S\. Savarese, C\. Xiong, and J\. C\. NieblesLAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback\.arXiv preprint arXiv:2506\.02298\.External Links:[Link](https://arxiv.org/abs/2506.02298)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2026a\)S\. Hu, Y\. Dai, X\. Han, Z\. Fang, Y\. Zhao, S\. T\. W\. Kwong, and Y\. FangSelf\-Induced Outcome Potential: Turn\-Level Credit Assignment for Agents without Verifiers\.arXiv preprint arXiv:2605\.04984\.External Links:[Link](https://arxiv.org/abs/2605.04984)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2026b\)Y\. Hu, Z\. Wen, X\. Liu, P\. Wang, X\. Zhang, and W\. WuSEAL: Synergistic Co\-Evolution of Agents and Learning Environments\.arXiv preprint arXiv:2605\.24426\.External Links:[Link](https://arxiv.org/abs/2605.24426)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Konget al\.\(2026\)F\. Kong, J\. Zhang, M\. Deng, C\. Wu, Y\. Luo, and B\. LiuInfoPO: Information\-Driven Policy Optimization for User\-Centric Agents\.arXiv preprint arXiv:2603\.00656\.External Links:[Link](https://arxiv.org/abs/2603.00656)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)Y\. Li, H\. A\. Inan, X\. Yue, W\. Chen, L\. Wutschitz, J\. Kulkarni, R\. Poovendran, R\. Sim, and S\. RajmohanSimulating Environments with Reasoning Models for Agent Training\.arXiv preprint arXiv:2511\.01824\.External Links:[Link](https://arxiv.org/abs/2511.01824)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026\)Z\. Li, H\. Wang, Y\. Zhao, G\. Chen, Y\. Li, K\. Chen, Y\. Cao, G\. Ye, H\. Chai, and Z\. YinRethinking the Role of Entropy in Optimizing Tool\-Use Behaviors for Large Language Model Agents\.arXiv preprint arXiv:2602\.02050\.External Links:[Link](https://arxiv.org/abs/2602.02050)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Luoet al\.\(2025\)X\. Luo, Y\. Zhang, Z\. He, Z\. Wang, S\. Zhao, D\. Li, L\. K\. Qiu, and Y\. YangAgent Lightning: Train ANY AI Agents with Reinforcement Learning\.arXiv preprint arXiv:2508\.03680\.External Links:[Link](https://arxiv.org/abs/2508.03680)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Nathaniet al\.\(2026\)D\. Nathani, C\. Zhang, C\. Huan, J\. Shan, Y\. Yang, A\. Patel, Z\. Gan, W\. Y\. Wang, M\. Saxon, and X\. E\. WangProactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants\.arXiv preprint arXiv:2604\.00842\.External Links:[Link](https://arxiv.org/abs/2604.00842)Cited by:[§4\.1](https://arxiv.org/html/2608.17499#S4.SS1.SSS0.Px2.p1.1)\.
- Nget al\.\(1999\)A\. Y\. Ng, D\. Harada, and S\. J\. RussellPolicy Invariance under Reward Transformations: Theory and Application to Reward Shaping\.InInternational Conference on Machine Learning,Cited by:[§3\.4](https://arxiv.org/html/2608.17499#S3.SS4.p2.1)\.
- Prabhakaret al\.\(2025\)A\. Prabhakar, Z\. Liu, W\. Yao, J\. Zhang, M\. Zhu, S\. Wang, Z\. Liu, T\. Awalgaonkar, H\. Chen, T\. Hoang, J\. C\. Niebles, S\. Heinecke, H\. Wang, S\. Savarese, and C\. XiongAPIGen\-MT: Agentic Pipeline for Multi\-Turn Data Generation via Simulated Agent\-Human Interplay\.arXiv preprint arXiv:2504\.03601\.External Links:[Link](https://arxiv.org/abs/2504.03601)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Qianet al\.\(2024\)C\. Qian, B\. He, Z\. Zhuang, J\. Deng, Y\. Qin, X\. Cong, Z\. Zhang, J\. Zhou, Y\. Lin, Z\. Liu, and M\. SunTell Me More\! Towards Implicit User Intention Understanding of Language Model Driven Agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1088–1113\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.61),[Link](https://aclanthology.org/2024.acl-long.61/)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Qianet al\.\(2025a\)C\. Qian, Z\. Liu, A\. Prabhakar, Z\. Liu, J\. Zhang, H\. Chen, H\. Ji, W\. Yao, S\. Heinecke, S\. Savarese, C\. Xiong, and H\. WangUserBench: An Interactive Gym Environment for User\-Centric Agents\.arXiv preprint arXiv:2507\.22034\.External Links:[Link](https://arxiv.org/abs/2507.22034)Cited by:[§1](https://arxiv.org/html/2608.17499#S1.p1.1),[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Qianet al\.\(2025b\)C\. Qian, Z\. Liu, A\. Prabhakar, J\. Qiu, Z\. Liu, H\. Chen, S\. Kokane, H\. Ji, W\. Yao, S\. Heinecke, S\. Savarese, C\. Xiong, and H\. WangUserRL: Training Interactive User\-Centric Agent via Reinforcement Learning\.arXiv preprint arXiv:2509\.19736\.External Links:[Link](https://arxiv.org/abs/2509.19736)Cited by:[§1](https://arxiv.org/html/2608.17499#S1.p3.1),[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal Policy Optimization Algorithms\.arXiv preprint arXiv:1707\.06347\.External Links:[Link](https://arxiv.org/abs/1707.06347)Cited by:[§3\.4](https://arxiv.org/html/2608.17499#S3.SS4.p1.1)\.
- Shaoet al\.\(2024a\)Y\. Shao, V\. Samuel, Y\. Jiang, J\. Yang, and D\. YangCollaborative Gym: A Framework for Enabling and Evaluating Human\-Agent Collaboration\.arXiv preprint arXiv:2412\.15701\.External Links:[Link](https://arxiv.org/abs/2412.15701)Cited by:[§A\.2](https://arxiv.org/html/2608.17499#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.17499#S4.SS1.SSS0.Px3.p1.1)\.
- Shaoet al\.\(2024b\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§3\.1](https://arxiv.org/html/2608.17499#S3.SS1.p1.1)\.
- Shiet al\.\(2026\)Q\. Shi, A\. Zytek, P\. Razavi, K\. Narasimhan, and V\. Barresτ\\tau\-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2603.04370),[Link](https://arxiv.org/abs/2603.04370)Cited by:[§1](https://arxiv.org/html/2608.17499#S1.p3.1),[§4\.1](https://arxiv.org/html/2608.17499#S4.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026\)G\. Wang, S\. Dai, G\. Ye, Z\. Gan, W\. Yao, Y\. Deng, X\. Wu, and Z\. YingInformation Gain\-based Policy Optimization: A Simple and Effective Approach for Multi\-Turn Search Agents\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2510.14967)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)R\. Wang, H\. Yu, W\. Zhang, Z\. Qi, M\. Sap, G\. Neubig, Y\. Bisk, and H\. ZhuSOTOPIA\-π\\pi: Interactive Learning of Socially Intelligent Language Agents\.arXiv preprint arXiv:2403\.08715\.External Links:[Link](https://arxiv.org/abs/2403.08715)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Weiet al\.\(2025\)Q\. Wei, S\. Zeng, C\. Li, W\. Brown, O\. Frunza, W\. Deng, A\. Schneider, Y\. Nevmyvaka, Y\. K\. Zhao, A\. Garcia, and M\. HongReinforcing Multi\-Turn Reasoning in LLM Agents via Turn\-Level Reward Design\.arXiv preprint arXiv:2505\.11821\.External Links:[Link](https://arxiv.org/abs/2505.11821)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Xieet al\.\(2026\)Y\. Xie, N\. Thomas, N\. Hansen, Y\. Fu, L\. E\. Li, and X\. WangTIPS: Turn\-Level Information\-Potential Reward Shaping for Search\-Augmented LLMs\.arXiv preprint arXiv:2603\.22293\.External Links:[Link](https://arxiv.org/abs/2603.22293)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yanget al\.Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§D\.1](https://arxiv.org/html/2608.17499#A4.SS1.p1.1)\.
- Yaoet al\.\(2025\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-bench: A Benchmark for Tool\-Agent\-User Interaction in Real\-World Domains\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2406.12045)Cited by:[§1](https://arxiv.org/html/2608.17499#S1.p1.1),[§1](https://arxiv.org/html/2608.17499#S1.p3.1),[§4\.1](https://arxiv.org/html/2608.17499#S4.SS1.SSS0.Px1.p1.1)\.
- Yinet al\.\(2025\)F\. Yin, Z\. Wang, I\. Hsu, J\. Yan, K\. Jiang, Y\. Chen, J\. Gu, L\. T\. Le, K\. Chang, C\. Lee, H\. Palangi, and T\. PfisterMagnet: Multi\-turn Tool\-use Data Synthesis and Distillation via Graph Translation\.arXiv preprint arXiv:2503\.07826\.External Links:[Link](https://arxiv.org/abs/2503.07826)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025a\)C\. Zhang, D\. Chong, F\. Jiang, C\. Tang, A\. Gao, G\. Tang, and H\. LiAligning Language Models Using Follow\-up Likelihood as Reward Signal\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 25832–25841\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i24.34776),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/34776)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025b\)W\. Zhang, T\. Liu, M\. Song, X\. Li, and T\. LiuSOTOPIA\-Ω\\Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents\.arXiv preprint arXiv:2502\.15538\.External Links:[Link](https://arxiv.org/abs/2502.15538)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, Z\. Chen, M\. Li, Z\. Tu, and X\. LiRLVMR: Reinforcement Learning with Verifiable Meta\-Reasoning Rewards for Robust Long\-Horizon Agents\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cTbAevdwBE)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2025a\)W\. Zhao, X\. Wang, C\. Ma, L\. Kong, Z\. Yang, M\. Tuo, X\. Shi, Y\. Zhai, and X\. CaiMUA\-RL Cold\-Start Dataset\.Note:Hugging Face datasetExternal Links:[Link](https://huggingface.co/datasets/zzwkk/MUA-RL-Dataset)Cited by:[§D\.1](https://arxiv.org/html/2608.17499#A4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.17499#S4.SS2.p1.1)\.
- Zhaoet al\.\(2025b\)W\. Zhao, X\. Wang, C\. Ma, L\. Kong, Z\. Yang, M\. Tuo, X\. Shi, Y\. Zhai, and X\. CaiMUA\-RL: Multi\-turn User\-interacting Agent Reinforcement Learning for Agentic Tool Use\.arXiv preprint arXiv:2508\.18669\.External Links:[Link](https://arxiv.org/abs/2508.18669)Cited by:[§1](https://arxiv.org/html/2608.17499#S1.p3.1),[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.17499#S5.SS1.p1.1)\.
- Zhouet al\.\(2025\)Y\. Zhou, S\. Jiang, Y\. Tian, J\. Weston, S\. Levine, S\. Sukhbaatar, and X\. LiSWEET\-RL: Training Multi\-Turn LLM Agents on Collaborative Reasoning Tasks\.arXiv preprint arXiv:2503\.15478\.External Links:[Link](https://arxiv.org/abs/2503.15478)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Zhouet al\.\(2024\)Y\. Zhou, A\. Zanette, J\. Pan, S\. Levine, and A\. KumarArCHer: Training Language Model Agents via Hierarchical Multi\-Turn RL\.arXiv preprint arXiv:2402\.19446\.External Links:[Link](https://arxiv.org/abs/2402.19446)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2026\)R\. Zhu, X\. Huang, Y\. Wu, R\. Wang, Z\. Sun, T\. Ren, W\. Luo, B\. Qiu, J\. Ye, Y\. Li, and W\. HuEIBench: A Simulator\-Based Benchmark and Turn\-Credit RL for Emotion Management\.arXiv preprint arXiv:2606\.15532\.External Links:[Link](https://arxiv.org/abs/2606.15532)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.
- Zonget al\.\(2026\)Z\. Zong, D\. Chen, Y\. Li, Q\. Yi, B\. Zhou, C\. Li, B\. O\. Qian, P\. Chen, and J\. JiangAT2\{\}^\{2\}PO: Agentic Turn\-based Policy Optimization via Tree Search\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 24120–24143\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1106),[Link](https://aclanthology.org/2026.acl-long.1106/)Cited by:[§2](https://arxiv.org/html/2608.17499#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AZero\-Shot Transfer Protocols

### A\.1Pare\-Bench

The four plotted RL checkpoints were trained only on the Airline and Retail training splits ofτ\\tau\-bench; no Pare\-Bench scenario was used for training, checkpoint selection, or prompt tuning\. All use the same 143\-scenario full split\. DeepSeek\-V4\-Flash is the active user with thinking enabled and high reasoning effort; evaluated agents use temperature 0, thinking disabled, and a 2,048\-token response limit\. The same checkpoint powers the Observe and Execute roles\. Strict denominators retain all exceptions as failures\. Interactive GRPO versusFacayields Pass@1 counts of 9/143 versus 15/143 at 8B and 15/143 versus 19/143 at 14B\. The corresponding Pass@4 counts are 27/143 versus 35/143 and 43/143 versus 47/143\.

Figure[3](https://arxiv.org/html/2608.17499#S4.F3)is restricted to the scale\-matched RL methods under the same formal protocol\. This design tests whether reaction\-grounded credit preserves its ordering after a protocol shift; it is not a comparison of the complete pretraining and optimization ladder\.

### A\.2Collaborative Gym

The Co\-Gym evaluation uses its simulated collaborative condition\[[21](https://arxiv.org/html/2608.17499#bib.bib37)\]\. Each checkpoint is evaluated on 312 instances: 102 Travel Planning, 100 Related Work Writing, and 110 Tabular Analysis tasks\. These environments require the agent and simulated user to communicate while acting in shared editors, search interfaces, or notebooks under a non\-turn\-taking notification protocol\. No Co\-Gym instance is used for training, checkpoint selection, or prompt tuning\.

Table[3](https://arxiv.org/html/2608.17499#S4.T3)reports the matched Interactive\-GRPO/Facapairs on a 0–100 scale\. Overall DR improves by 5\.5 points at 8B and 2\.2 points at 14B; Overall CS improves by 3\.3 and 3\.7 points\. The evaluator\-provided overall values are rescaled by 100 for presentation rather than reconstructed from rounded task\-level cells\.HinitH\_\{\\mathrm\{init\}\}was not produced by this evaluation, so we do not infer initiative balance from these trajectories\. As in Pare\-Bench, the comparison is restricted to the matched RL pair\.

## Appendix BReaction Mapping and Extraction

Missing, malformed, or ambiguous strategy metadata maps to neutral and creates no signed process credit\. The mapping is an inductive bias: reveal\-piece can reward unnecessary questioning, challenge\-solution can follow a correct refusal, change\-mind can be exogenous, and close partly overlaps terminal success\. The terminal branch remains responsible for verified world\-state correctness\.

## Appendix CShared Frozen\-User Prompt

All main\-table Interactive GRPO andFacaruns use the same frozen\-simulator path \(USERSIM\_PLAIN=0\) and the following semantic prompt\. The simulator emits a private reaction strategy and visible utterance in one response; only the utterance enters the agent context\.

Frozen User Simulator PromptYou are simulating a USER talking to a customer\-service agent\.Instruction \(your situation and goal\):\{INTENT\}Your job:Produce the user’s NEXT single reply to the agent, together with a short private reasoning, as a JSON object\.Rules:•Reply with ONE short, natural customer message at a time\. Do not dump the whole instruction; reveal only what the current step needs\.•NEVER invent information that is not in the Instruction\. If the agent asks for an absent ID, email, name, or date, say you do not have it\.•Only state facts in the Instruction\. Stay faithful to the goal\.•If the goal is fully satisfied, setutteranceto\#\#\#STOP\#\#\#\.•Use your own words; do not quote the Instruction verbatim\.Output ONLY a JSON object with exactly these keys:\{"goal\_recap": "true goal including key entities","progress": "what the agent has learned so far","withheld": "unrevealed facts, or none","strategy": "ONE of: be\_vague \| reveal\_piece \|ask\_clarification \| change\_mind \| challenge\_solution \|confirm \| close","utterance": "one short natural customer message"\}Strategy guide:•reveal\_piece: give one requested fact\.•confirm: agree with the right proposed action\.•close: goal satisfied;utterance = \#\#\#STOP\#\#\#\.•ask\_clarification: ask the agent to explain an unclear response\.•challenge\_solution: push back on a wrong proposal\.•change\_mind: realize a different need \(rare\)\.•be\_vague: remain unsure or withhold information \(rare\)\.Theutteranceis the ONLY field visible to the agent\.

Interactive GRPO discards the remaining fields, whereasFacareadsstrategyonly after rollout construction\. For the simulator view, original system and raw tool messages are dropped, roles are swapped, empty tool\-call\-only assistant messages are removed, and prior private reasoning is stripped\. Structured tool names, arguments, and raw results are not shown to the user simulator\.

## Appendix DTraining and Evaluation Details

### D\.1SFT and RL Configuration

We use Qwen3\-8B and Qwen3\-14B\[[28](https://arxiv.org/html/2608.17499#bib.bib17)\]\. Both backbones are cold\-started on the public MUA\-RL release, which contains 1,580 annotated multi\-turn tool\-use trajectories spanning five mock and four MCP tasks\[[34](https://arxiv.org/html/2608.17499#bib.bib12)\]\. Each trajectory pairs a dialogue with its tool schema\. SFT supervises assistant responses and tool calls while masking user and tool\-result tokens, thereby initializing interaction and tool\-use behavior before RL\.

For each scale, Interactive GRPO andFacaare alternative continuations from the same SFT checkpoint, with three independent training seeds per method\. Both use only the Airline and Retail training splits ofτ\\tau\-bench, the frozen DeepSeek\-V4\-Flash user path\[[3](https://arxiv.org/html/2608.17499#bib.bib18)\], and agent\-only optimization\. All reported DeepSeek\-backed runs use the frozen service version available before the July 31, 2026 update\. For 8B, the prompt batch size is 16, rollout group size isK=8K=8, learning rate is10−610^\{\-6\}, and KL coefficient is10−310^\{\-3\}\. All reported RL runs use the step\-120 checkpoint\.Facauses true U2U segments, immediate reactions without backward propagation, ordinal turn\-index anchors, andλ=0\.5\\lambda=0\.5\.

The matched RL arms share the simulator prompt \(USERSIM\_PLAIN=0\), agent\-visible utterances, training data, rollout construction, optimizer, and horizon\. The simulator emits the same private strategy metadata and visible utterance in both arms\. Interactive GRPO ignores the metadata and broadcasts onlyAoA^\{\\mathrm\{o\}\};Facaadditionally usesApA^\{\\mathrm\{p\}\}\.

### D\.2Evaluation and Run Aggregation

For theτ\\tau\-bench family, pass@1 is determined from the verified final environment state under the official non\-thinking protocol\. Strict denominators are 50/115 forτ\\tau, 50/114/114 forτ2\\tau^\{2\}, and 50/114/114/97 forτ3\\tau^\{3\}; unscored tasks remain failures\. Each of the three independently trained step\-120 checkpoints is evaluated once under the same protocol and is reused across everyτ\\taudomain rather than selecting domain\-specific peaks\. Domain entries in Table[2](https://arxiv.org/html/2608.17499#S4.T2)are arithmetic means over these runs; Avg\. is the unweighted mean of the nine domains\. Its sample standard deviation is computed across the three run\-level averages\. Because every trained checkpoint has one evaluation, this dispersion may include residual evaluation stochasticity and does not separately identify either component\.

Pare\-Bench and Co\-Gym use one designated step\-120 checkpoint per scale and method and are reported separately from the three\-runτ\\tauaggregate\. The complete OOD protocols and coverage are given in Appendix[A](https://arxiv.org/html/2608.17499#A1)\.

### D\.3Seed\-Level Results and Ablations

Table[5](https://arxiv.org/html/2608.17499#A4.T5)gives the complete run\-level source counts behind the primaryτ\\taucomparison\. Here s1–s3 denote independently trained RL checkpoints with distinct training seeds, not repeated evaluations of one checkpoint\. Each step\-120 checkpoint is evaluated once under the same strict protocol\. Consequently, the run\-level standard deviations reported in Section[5](https://arxiv.org/html/2608.17499#S5)measure dispersion across complete evaluations and may include residual evaluation stochasticity; they do not separately identify either component\.

Matching methods by training\-seed identifier, the nine\-domain improvements are 4\.67, 6\.33, and 6\.75 points at 8B and 8\.62, 10\.14, and 11\.91 points at 14B\. The Aligned \(λ=0\.5\\lambda=0\.5\) and Outcome\-only rows in Table[4](https://arxiv.org/html/2608.17499#S6.T4)are exactly the 8B s1 checkpoints listed here\. The remaining ablation conditions use the same reference experiment family and are evaluated once, keeping the ablation internally controlled while the main table estimates performance across independent training runs\.

\(a\) Qwen3\-8B raw successes / nominal tasks

\(b\) Qwen3\-14B raw successes / nominal tasks

Table 5:Raw counts from the independent training runs underlying Table[2](https://arxiv.org/html/2608.17499#S4.T2)\. For each RL method, s1–s3 are distinct training seeds that produce separate step\-120 checkpoints, each evaluated once\. The seed labels align the two RL methods by training\-seed identifier; the main table averages these runs domainwise\. Base and SFT are fixed pre\-RL references\. Pooled counts sum successes across unequal domain sizes and are therefore not the primary metric; Avg\. gives each of the nine domains equal weight\. All strict scores retain the nominal denominator\.

## Appendix EStrategy and Credit Telemetry

Table 6:Recoverable telemetry from the step\-120Facaruns\. A group\-step is one prompt group at one optimizer step \(16 groups per step\)\. Singleton statistics count segments assigned to singleton anchors; they are not a nonconstant\-anchor rate\. Process quantities use0\.5​𝔼​\|Ap\|0\.5\\,\\mathbb\{E\}\|A^\{\\mathrm\{p\}\}\|and𝔼​\|Ao\|\\mathbb\{E\}\|A^\{\\mathrm\{o\}\}\|before PPO clipping and are advantage\-magnitude proxies rather than optimizer\-update or gradient mass\.The step\-120Facalogs contain120×128=15,360120\\times 128=15\{,\}360trajectories at each scale\. The derived telemetry summaries verify 16 prompt groups per step and reproduce the aggregates in Table[6](https://arxiv.org/html/2608.17499#A5.T6)\. The zero consistency/span/debug counts are mechanical implementation checks, not evidence that the reaction labels are semantically correct\.

Figure 6:Process\-credit share over the common first 120 training steps\. Faint lines show per\-step values and solid lines a 10\-step moving average\. The quantity is a pre\-optimization advantage L1 proxy, not optimizer\-update or gradient mass\.Table 7:Complete per\-U2U strategy distribution \(%\) in the telemetry\-enabled 8B replication\. Unlike a terminal representative field, every row here uses the U2U segment as its verified count unit\. The largest phasewise changes areconfirm\(\+10\.75\+10\.75points\) andbe\_vague\(−9\.06\-9\.06points\)\.A configuration\-matched 8B replication augments the rollout logs with one record for every U2U segment\. Its 120 telemetry files contain 154,219 records; at every step, the file count equals the tensor\-level segment count, with zero embedded\-step mismatches or telemetry write errors\. Each record includes the training step, prompt group, rollout, U2U ordinal, domain, strategy, mapped polarity, anchor size, normalized process advantage, and a nonzero\-credit indicator\. Table[7](https://arxiv.org/html/2608.17499#A5.T7)therefore reports full per\-U2U frequencies rather than the terminal representative stored by the original main\-run rollout logs\. Positive reactions rise from 66\.17% to 79\.51% in Airline and from 71\.71% to 84\.18% in Retail\. Among positive segments, nonzero\-credit coverage nevertheless decreases from 85\.20% in steps 1–40 to 80\.51% in steps 81–120; negative\-segment coverage remains above 93%\. This further distinguishes raw reaction prevalence from anchor\-relative differential credit\.

##### Quantities that cannot be reconstructed\.

The evaluated 8B and 14B training logs do not contain the replication’s complete per\-U2U schema, so their phasewise reaction distributions cannot be reconstructed\. In addition, their singleton\-segment counts do not identify the total number of anchors or how many non\-singleton anchors have constant reactions, so no valid/nonconstant\-anchor rate can be reported from the available summaries\. Finally,0\.5​𝔼​\|Ap\|0\.5\\mathbb\{E\}\|A^\{\\mathrm\{p\}\}\|and𝔼​\|Ao\|\\mathbb\{E\}\|A^\{\\mathrm\{o\}\}\|precede PPO clipping, log\-probability weighting, and backpropagation\. They are advantage\-magnitude proxies, not branch\-level update or gradient mass, including within outcome\-homogeneous groups, and should not be interpreted as such\.

## Appendix FTelecom Audit and Case Study

We audit the final 14B s1Facaand outcome\-only Interactive GRPO checkpoints in their paired run\-level evaluation\. Within each benchmark, the two archives share the same 114 task IDs, seeds, initial states, criteria, and scenarios\. Onτ2\\tau^\{2\}Telecom, both methods succeed on 23 tasks,Facaalone succeeds on 73, Interactive GRPO alone on 7, and neither on 11 \(exact paired McNemarp=5\.80×10−15p=5\.80\\times 10^\{\-15\}\)\. Onτ3\\tau^\{3\}Telecom, the corresponding cells are 47, 48, 11, and 8 \(p=1\.24×10−6p=1\.24\\times 10^\{\-6\}\)\. These are task\-level paired audits of the single s1 evaluation; Table[2](https://arxiv.org/html/2608.17499#S4.T2)instead averages three independently trained checkpoints, each evaluated once\.

Table 8:Final\-14B paired Telecom audit on the s1 run\-level evaluation\. The two methods share task IDs, evaluation seeds, initial states, criteria, and scenarios\. The last column reports discordant task counts\. Theτ3\\tau^\{3\}Service reversal shows that the checkpoint\-level gain is not uniform within Telecom\.The discordant failures expose two directly recorded modes that clarify how the two training objectives diverge during interaction\. Among theFaca\-only wins, Interactive GRPO exhausts the environment error budget on 68/73τ2\\tau^\{2\}tasks and 28/48τ3\\tau^\{3\}tasks; the remaining 5 and 20 end in a normal user stop without satisfying the task\. Conversely, all 7 and 11 Interactive\-GRPO\-only wins correspond to unsuccessful user stops forFaca, indicating that its gains do not eliminate every failure mode\. Table[8](https://arxiv.org/html/2608.17499#A6.T8)keeps the negativeτ3\\tau^\{3\}Service slice visible rather than treating the Telecom gain as uniform\.

To avoid selecting a success that merely receives more interaction time, the complete trace below uses aτ2\\tau^\{2\}Faca\-only win in whichFacahas fewer stored events, assistant messages, and tool calls than Interactive GRPO, providing a stricter qualitative comparison\.

### F\.1Matched Multi\-Fault Telecom Case

We compare matched trajectories for aτ2\\tau^\{2\}Telecom task that requires four user\-side device\-setting changes followed by a user\-run speed test\.Facacompletes the required changes and both terminal checks, reaching 275 Mbps\. Interactive GRPO reaches the same four state changes, but its fifth rejected agent\-issued call to a user\-side device tool exhausts the error budget at the final speed\-test step, so the run ends before terminal verification\. The two trajectories contain 48 versus 54 stored events, 18 versus 21 assistant messages, and 15 versus 17 tool calls, respectively\.

#### F\.1\.1SuccessfulFacatrajectory

Faca\(successful; reward 1\)T00 AgentHi\! How can I help you today?T01 UserHi, I’m having some trouble with my mobile data\. Sometimes it just stops working altogether, and other times it’s really slow\. I want to get this fixed and I absolutely need excellent internet speed on my phone—not just "okay" or "good\." Can you help me with that?T02 AgentSure, I can help you with that\. Let’s start by identifying your account\. Could you please provide me with your phone number or customer ID? If you don’t have your customer ID, I can also look up your account using your full name and date of birth\.T03 UserMy phone number is 555\-123\-2002\. My name is John Smith\.T04 Agent actionget\_customer\_by\_phoneArguments:\{"phone\_number": "555\-123\-2002"\}T05 Tool result to agent\{"customer\_id": "C1001", "full\_name": "John Smith", "date\_of\_birth": "1985\-06\-15", "email": "john\.smith@example\.com", "phone\_number": "555\-123\-2002", "address": \{"street": "123 Main St", "city": "Anytown", "state": "CA", "zip\_code": "90210"\}, "account\_status": "Active", "payment\_methods": \[\{"method\_type": "Credit Card", "account\_number\_last\_4": "1235", "expiration\_date": "12/2026"\}\], "line\_ids": \["L1001", "L1002", "L1003"\], "bill\_ids": \["B1001", "B1002", "B1003"\], "created\_at": "2025\-01\-15 10:30:00", "last\_extension\_date": null, "goodwill\_credit\_used\_this\_year": 25\.0\}T06 Agent actionget\_details\_by\_idArguments:\{"id": "L1001"\}T07 Tool result to agent\{"line\_id": "L1001", "phone\_number": "555\-123\-2001", "status": "Active", "plan\_id": "P1001", "device\_id": "D1001", "data\_used\_gb": 3\.2, "data\_refueling\_gb": 0\.0, "roaming\_enabled": false, "contract\_end\_date": "2026\-12\-31", "last\_plan\_change\_date": "2025\-01\-10", "last\_sim\_replacement\_date": null, "suspension\_start\_date": null\}T08 Agent actionget\_details\_by\_idArguments:\{"id": "L1002"\}T09 Tool result to agent\{"line\_id": "L1002", "phone\_number": "555\-123\-2002", "status": "Active", "plan\_id": "P1002", "device\_id": "D1002", "data\_used\_gb": 8\.7, "data\_refueling\_gb": 0\.0, "roaming\_enabled": true, "contract\_end\_date": "2026\-12\-31", "last\_plan\_change\_date": "2024\-12\-15", "last\_sim\_replacement\_date": "2025\-01\-20", "suspension\_start\_date": null\}T10 Agent actionget\_details\_by\_idArguments:\{"id": "L1003"\}T11 Tool result to agent\{"line\_id": "L1003", "phone\_number": "555\-123\-2003", "status": "Suspended", "plan\_id": "P1001", "device\_id": "D1003", "data\_used\_gb": 0\.0, "data\_refueling\_gb": 0\.0, "roaming\_enabled": false, "contract\_end\_date": "2026\-06\-30", "last\_plan\_change\_date": "2024\-10\-05", "last\_sim\_replacement\_date": null, "suspension\_start\_date": "2025\-02\-01"\}T12 Agent actionget\_details\_by\_idArguments:\{"id": "P1001"\}T13 Tool result to agent\{"plan\_id": "P1001", "name": "Basic Plan", "data\_limit\_gb": 5\.0, "price\_per\_month": 40\.0, "data\_refueling\_price\_per\_gb": 5\.0\}T14 Agent actionget\_details\_by\_idArguments:\{"id": "P1002"\}T15 Tool result to agent\{"plan\_id": "P1002", "name": "Premium Plan", "data\_limit\_gb": 15\.0, "price\_per\_month": 65\.0, "data\_refueling\_price\_per\_gb": 2\.0\}T16 Agent actionget\_data\_usageArguments:\{"customer\_id": "C1001", "line\_id": "L1002"\}T17 Tool result to agent\{"line\_id": "L1002", "data\_used\_gb": "8\.7", "data\_limit\_gb": "15\.0", "data\_refueling\_gb": "0\.0", "cycle\_end\_date": "2025\-02\-28"\}T18 AgentI see\. You’re on the \*\*Premium Plan \(P1002\)\*\* with a \*\*15 GB data limit\*\*, and you’ve used \*\*8\.7 GB\*\* so far in this cycle, which is under the limit\. However, your data speed is slow or sometimes not working at all\. Let’s troubleshoot this step by step\.\#\#\# Step 1: Check Your Network Mode PreferenceYour phone might be connecting to a slower network \(like 2G or 3G\) instead of 4G or 5G\. Let’s check your network mode preference\.T18 Agent actioncheck\_network\_mode\_preferenceArguments:\{\}T19 Tool result to agentError: Tool ’check\_network\_mode\_preference’ not found\.T20 AgentI apologize for the confusion earlier\. Since I can’t directly access your device, I’ll guide you through the steps you can take on your phone to check and adjust your network mode preference\.\#\#\# How to Check Your Network Mode Preference \(Manual Steps\):1\. \*\*Open your phone’s Settings\*\*\.2\. Go to \*\*Network & Internet\*\* or \*\*Connections\*\* \(this may vary slightly depending on your phone’s brand and OS\)\.3\. Look for \*\*Mobile Network\*\* or \*\*Cellular\*\* settings\.4\. Tap on \*\*Network Mode\*\* or \*\*Preferred Network Type\*\*\.5\. Check what network type is selected \(e\.g\., 5G, 4G, 3G, 2G\)\.\#\#\# If You’re on 2G or 3G:\- Change the network mode to \*\*4G/5G preferred\*\* to get faster speeds\.\- After changing, \*\*reboot your phone\*\* to ensure the changes take effect\.Would you like me to guide you through other steps to improve your data speed?T21 UserOkay, let me check that on my phone\.T21 User actioncheck\_network\_mode\_preferenceArguments:\{\}T22 Tool result to userNetwork Mode Preference: 2g\_onlyT23 UserIt says my phone is currently set to "2G only\." Is that right? Would changing it to something like "4G/5G preferred" help me get excellent internet speed?T24 AgentYes, that’s correct\. If your phone is set to \*\*"2G only"\*\*, it will only connect to 2G networks, which are significantly slower than 4G or 5G\. Changing your network mode preference to \*\*"4G/5G preferred"\*\* will allow your phone to connect to faster networks, which should improve your internet speed\.\#\#\# Steps to Change Your Network Mode Preference:1\. \*\*Open your phone’s Settings\*\*\.2\. Go to \*\*Network & Internet\*\* or \*\*Connections\*\* \(this may vary slightly depending on your phone’s brand and OS\)\.3\. Look for \*\*Mobile Network\*\* or \*\*Cellular\*\* settings\.4\. Tap on \*\*Network Mode\*\* or \*\*Preferred Network Type\*\*\.5\. Select \*\*"4G/5G preferred"\*\* or a similar option\.6\. \*\*Reboot your phone\*\* to ensure the changes take effect\.After making this change, your phone should connect to a faster network, and your internet speed should improve significantly\.Would you like me to help you with anything else related to your mobile data issue?T25 UserLet me try changing it to "4G/5G preferred" from my phone\.T25 User actionset\_network\_mode\_preferenceArguments:\{"mode": "4g\_5g\_preferred"\}T26 Tool result to userPreferred Network Mode set to: 4g\_5g\_preferred Status Bar:\[airplane\]Airplane Mode \|\[battery\]80%T27 UserI’ve changed my phone’s setting to "4G/5G preferred," but now I’m seeing a little airplane icon on the top of my screen\. Does that mean something is wrong?T28 AgentYes, the \*\*airplane icon\*\* on your screen means that \*\*Airplane Mode is enabled\*\*, which disables all wireless connections, including mobile data and cellular service\. This is likely why you’re still experiencing issues with your internet speed or connectivity\.\#\#\# How to Turn Off Airplane Mode:1\. \*\*Swipe down from the top of your screen\*\* to open the \*\*Quick Settings panel\*\*\.2\. Look for the \*\*Airplane Mode icon\*\* \(it looks like a small airplane\)\.3\. Tap on it to \*\*disable Airplane Mode\*\*\.Alternatively:1\. \*\*Open your phone’s Settings\*\*\.2\. Go to \*\*Network & Internet\*\* or \*\*Connections\*\*\.3\. Look for \*\*Airplane Mode\*\* and toggle it \*\*off\*\*\.After turning off Airplane Mode, your phone should reconnect to the cellular network, and your mobile data should be restored\.Would you like me to help you check anything else related to your mobile data issue?T29 UserOh, I see\. That must have been turned on by accident\. Let me turn that off now\.T29 User actiontoggle\_airplane\_modeArguments:\{\}T30 Tool result to userAirplane Mode is now OFF\. Status Bar:\[signal\]4Excellent \| 5G \|\[no\-cellular\]Data Disabled \|\[battery\]80%T31 UserI’ve turned off the Airplane Mode, so that airplane icon is gone now\. However, I notice on my screen there’s a symbol that looks like a "no data" sign \(data disabled\)\. Does that mean my mobile data is turned off?T32 AgentYes, the \*\*"no data" symbol\*\* \(which looks like a crossed\-out phone icon or a "data disabled" message\) means that \*\*mobile data is currently turned off\*\* on your phone\. This would explain why you’re still not getting internet access or experiencing slow speeds\.\#\#\# How to Turn On Mobile Data:1\. \*\*Swipe down from the top of your screen\*\* to open the \*\*Quick Settings panel\*\*\.2\. Look for the \*\*Data icon\*\* \(it looks like a phone with a signal bar or a "Data" label\)\.3\. Tap on it to \*\*enable mobile data\*\*\.Alternatively:1\. \*\*Open your phone’s Settings\*\*\.2\. Go to \*\*Network & Internet\*\* or \*\*Connections\*\*\.3\. Look for \*\*Mobile Network\*\* or \*\*Cellular\*\* settings\.4\. Toggle \*\*Mobile Data\*\* to the \*\*ON\*\* position\.After enabling mobile data, your phone should reconnect to the cellular network, and your internet speed should improve\.Would you like me to help you check anything else related to your mobile data issue?T33 UserLet me turn on the mobile data option\.T33 User actiontoggle\_dataArguments:\{\}T34 Tool result to userMobile Data is now ON\. Status Bar:\[signal\]4Excellent \| 5G \|\[phone\]Data Enabled \|\[data\-saver\]Data Saver \|\[battery\]80%T35 UserMobile data is now turned on\. I can see 5G signal and it looks good, but I also see a "Data Saver" icon\. Should I worry about that? Can I check my internet speed now?T36 AgentYes, the \*\*"Data Saver" icon\*\* means that \*\*Data Saver mode is enabled\*\* on your phone\. This mode restricts background data usage to help conserve data, which can sometimes make your internet feel slower or less responsive, especially for apps that require a lot of data in the background\.\#\#\# Should You Worry About It?\- \*\*No\*\*, you don’t need to worry\. However, if you’re looking for \*\*excellent internet speed\*\*, it’s best to \*\*disable Data Saver mode\*\* temporarily while you’re using your phone for data\-heavy tasks \(like streaming, gaming, or browsing\)\.\#\#\# How to Disable Data Saver Mode:1\. \*\*Open your phone’s Settings\*\*\.2\. Go to \*\*Network & Internet\*\* or \*\*Connections\*\*\.3\. Look for \*\*Data Saver\*\* or \*\*Mobile Data\*\* settings\.4\. Toggle \*\*Data Saver\*\* to the \*\*OFF\*\* position\.Once Data Saver is disabled, your phone will allow full background data usage, which should improve your internet speed\.—\#\#\# Let’s Check Your Internet SpeedI can help you check your internet speed using the \*\*‘run\_speed\_test‘\*\* tool\. This will give us an idea of how fast your connection is\.Would you like me to run the speed test now?T37 UserAlright, I’ll go ahead and disable Data Saver mode first\.T37 User actiontoggle\_data\_saver\_modeArguments:\{\}T38 Tool result to userData Saver Mode is now OFF\. Status Bar:\[signal\]4Excellent \| 5G \|\[phone\]Data Enabled \|\[battery\]80%T39 UserOkay, Data Saver mode is now turned off\. Should I check my internet speed now to see if it’s excellent?T40 Agent actionrun\_speed\_testArguments:\{\}T41 Tool result to agentError: Tool ’run\_speed\_test’ not found\.T42 AgentI apologize for the confusion earlier\. Since I can’t directly run a speed test on your phone, I can guide you through how to check your internet speed manually\.\#\#\# How to Check Your Internet Speed Manually:1\. \*\*Open a web browser\*\* on your phone \(e\.g\., Chrome, Safari\)\.2\. Go to a \*\*speed test website\*\* like \[Speedtest by Ookla\]\(https://www\.speedtest\.net/\)\.3\. Tap \*\*"Begin Test"\*\* to run the speed test\.4\. The test will show your \*\*download speed\*\*, \*\*upload speed\*\*, and \*\*latency\*\*\.\#\#\# What to Look For:\- \*\*Download Speed\*\*: For excellent internet speed, you should see at least \*\*100 Mbps\*\* or higher\.\- \*\*Latency\*\*: A latency of \*\*under 50 ms\*\* is ideal for smooth browsing and streaming\.If your speed is still slow after disabling Data Saver mode and connecting to 5G, we can explore other options like checking your APN settings or looking into your data plan\.Would you like me to help you check your APN settings or look into your data plan?T43 UserLet me run a speed test using my phone\.T43 User actionrun\_speed\_testArguments:\{\}T44 Tool result to userSpeed Test Result: 275\.00 Mbps \(Excellent\)\. Connection is very fast\.T45 UserI ran a speed test, and it says the connection is excellent—275 Mbps\! That’s really fast\. I think my mobile data is working perfectly now\. Thank you so much for your help\!T46 AgentYou’re very welcome\! I’m glad to hear that your mobile data is now working perfectly and that your internet speed is excellent at \*\*275 Mbps\*\*\. That’s more than enough for streaming, gaming, and browsing without any issues\.If you ever run into any more problems with your mobile data or need help with anything else, feel free to reach out\. Have a great day\!\[smile\]T47 UserThank you\! I really appreciate your patience and guidance\. Have a great day\!\[smile\]\#\#\#STOP\#\#\#

#### F\.1\.2Failed Interactive\-GRPO trajectory

Interactive GRPO \(failed; reward 0\)T00 AgentHi\! How can I help you today?T01 UserHi, I’m hoping you can help me\. My mobile data isn’t working right—sometimes it stops working or it’s really slow\. I really need great internet speed on my phone when I’m out and about\. Can you help me get this fixed? I don’t have Wi\-Fi at home, so I’m using my mobile data\.T02 Agent actioncheck\_network\_statusArguments:\{\}T03 Tool result to agentError: Tool ’check\_network\_status’ not found\.T04 AgentI’m sorry for the inconvenience\. I need to check your phone’s network status to help resolve the issue\. However, I currently don’t have access to that tool\. Could you please provide your phone number so I can look up your account and help further?T05 UserSure, my phone number is 555\-123\-2002\.T06 Agent actionget\_customer\_by\_phoneArguments:\{"phone\_number": "555\-123\-2002"\}T07 Tool result to agent\{"customer\_id": "C1001", "full\_name": "John Smith", "date\_of\_birth": "1985\-06\-15", "email": "john\.smith@example\.com", "phone\_number": "555\-123\-2002", "address": \{"street": "123 Main St", "city": "Anytown", "state": "CA", "zip\_code": "90210"\}, "account\_status": "Active", "payment\_methods": \[\{"method\_type": "Credit Card", "account\_number\_last\_4": "1235", "expiration\_date": "12/2026"\}\], "line\_ids": \["L1001", "L1002", "L1003"\], "bill\_ids": \["B1001", "B1002", "B1003"\], "created\_at": "2025\-01\-15 10:30:00", "last\_extension\_date": null, "goodwill\_credit\_used\_this\_year": 25\.0\}T08 Agent actionget\_details\_by\_idArguments:\{"id": "L1001"\}T09 Tool result to agent\{"line\_id": "L1001", "phone\_number": "555\-123\-2001", "status": "Active", "plan\_id": "P1001", "device\_id": "D1001", "data\_used\_gb": 3\.2, "data\_refueling\_gb": 0\.0, "roaming\_enabled": false, "contract\_end\_date": "2026\-12\-31", "last\_plan\_change\_date": "2025\-01\-10", "last\_sim\_replacement\_date": null, "suspension\_start\_date": null\}T10 Agent actionget\_details\_by\_idArguments:\{"id": "L1002"\}T11 Tool result to agent\{"line\_id": "L1002", "phone\_number": "555\-123\-2002", "status": "Active", "plan\_id": "P1002", "device\_id": "D1002", "data\_used\_gb": 8\.7, "data\_refueling\_gb": 0\.0, "roaming\_enabled": true, "contract\_end\_date": "2026\-12\-31", "last\_plan\_change\_date": "2024\-12\-15", "last\_sim\_replacement\_date": "2025\-01\-20", "suspension\_start\_date": null\}T12 Agent actionget\_details\_by\_idArguments:\{"id": "L1003"\}T13 Tool result to agent\{"line\_id": "L1003", "phone\_number": "555\-123\-2003", "status": "Suspended", "plan\_id": "P1001", "device\_id": "D1003", "data\_used\_gb": 0\.0, "data\_refueling\_gb": 0\.0, "roaming\_enabled": false, "contract\_end\_date": "2026\-06\-30", "last\_plan\_change\_date": "2024\-10\-05", "last\_sim\_replacement\_date": null, "suspension\_start\_date": "2025\-02\-01"\}T14 Agent actionget\_details\_by\_idArguments:\{"id": "P1001"\}T15 Tool result to agent\{"plan\_id": "P1001", "name": "Basic Plan", "data\_limit\_gb": 5\.0, "price\_per\_month": 40\.0, "data\_refueling\_price\_per\_gb": 5\.0\}T16 Agent actionget\_details\_by\_idArguments:\{"id": "P1002"\}T17 Tool result to agent\{"plan\_id": "P1002", "name": "Premium Plan", "data\_limit\_gb": 15\.0, "price\_per\_month": 65\.0, "data\_refueling\_price\_per\_gb": 2\.0\}T18 AgentI see that you’re using the line with phone number \*\*555\-123\-2002\*\*, which is associated with the \*\*Premium Plan \(P1002\)\*\*\. This plan provides \*\*15 GB of data per month\*\*\. However, your current data usage is \*\*8\.7 GB\*\*, which is under the limit\.Since your data usage is within the plan’s limit, the issue might be related to \*\*network performance\*\*, \*\*data roaming\*\*, or \*\*device settings\*\*\. Let’s check a few things:1\. \*\*Network Mode Preference\*\*: Your phone might be connecting to a slower network \(e\.g\., 2G/3G\) instead of 4G/5G\. I can check your network mode preference\.2\. \*\*Data Saver Mode\*\*: If Data Saver is enabled, it might be restricting background data usage, which could slow down your internet speed\.3\. \*\*Roaming Status\*\*: Your line has roaming enabled, but if you’re in an area with poor signal, it might be affecting your speed\.Would you like me to check your network mode preference or data saver status?T19 UserThank you for checking\. To be honest, I get a bit lost with all these technical details\. Could you please tell me what you’d like me to do first? I’ll try my best to follow your instructions\.T20 Agent actioncheck\_network\_mode\_preferenceArguments:\{\}T21 Tool result to agentError: Tool ’check\_network\_mode\_preference’ not found\.T22 AgentI’m sorry for the inconvenience\. I need to check your phone’s network mode preference to help resolve the issue\. However, I currently don’t have access to that tool\. Could you please try the following steps on your phone?1\. \*\*Check Network Mode Preference\*\*:\- Go to your phone’s settings\.\- Look for "Network & Internet" or "Connections\."\- Tap on "Mobile Network" or "Cellular\."\- Check the "Network Mode" or "Preferred Network Type" setting\.\- Ensure it is set to "4G/5G" or "4G LTE/5G" \(not 2G or 3G\)\.2\. \*\*Check Data Saver Mode\*\*:\- Go to your phone’s settings\.\- Look for "Network & Internet" or "Connections\."\- Tap on "Data Usage" or "Mobile Data\."\- Check if "Data Saver" is enabled\. If it is, disable it\.Would you like me to guide you through these steps in more detail?T23 UserI’ll try to check what you suggested, but I’m not too comfortable navigating through the settings on my own\. Could you please guide me one step at a time? I’d rather not get confused with too many steps at once\.T24 AgentSure, I can guide you step by step\. Let’s start with checking your \*\*Network Mode Preference\*\*\.\#\#\# Step 1: Check Network Mode Preference1\. Open the \*\*Settings\*\* app on your phone\.2\. Scroll down and tap on \*\*Network & Internet\*\* or \*\*Connections\*\* \(the exact name may vary slightly depending on your phone’s manufacturer\)\.3\. Tap on \*\*Mobile Network\*\* or \*\*Cellular\*\*\.4\. Look for an option called \*\*Network Mode\*\*, \*\*Preferred Network Type\*\*, or \*\*Network Selection\*\*\.5\. Tap on it to see what network type your phone is currently using \(e\.g\., 5G, 4G, 3G, 2G\)\.6\. If it’s set to \*\*2G\*\* or \*\*3G\*\*, we can change it to \*\*4G/5G\*\* or \*\*4G LTE/5G\*\* for better speed\.Would you like me to guide you through this step\-by\-step on your phone?T25 UserLet me have a look at what my phone shows for the network mode\.T25 User actioncheck\_network\_mode\_preferenceArguments:\{\}T26 Tool result to userNetwork Mode Preference: 2g\_onlyT27 UserI just checked, and it looks like my phone is set to 2G only\. I’m not really sure how that happened\. What should I do next?T28 Agent actionset\_network\_mode\_preferenceArguments:\{"mode": "4g\_5g\_preferred"\}T29 Tool result to agentError: Tool ’set\_network\_mode\_preference’ not found\.T30 AgentI’m sorry for the inconvenience\. I need to change your phone’s network mode preference to \*\*4G/5G\*\* to improve your data speed\. However, I currently don’t have access to that tool\.Could you please follow these steps on your phone to change the network mode preference manually?\#\#\# Step 2: Change Network Mode Preference1\. Open the \*\*Settings\*\* app on your phone\.2\. Scroll down and tap on \*\*Network & Internet\*\* or \*\*Connections\*\*\.3\. Tap on \*\*Mobile Network\*\* or \*\*Cellular\*\*\.4\. Look for an option called \*\*Network Mode\*\*, \*\*Preferred Network Type\*\*, or \*\*Network Selection\*\*\.5\. Tap on it and select \*\*4G/5G\*\* or \*\*4G LTE/5G\*\* \(not 2G or 3G\)\.6\. Save your changes\.After changing the network mode, your phone should connect to a faster network, which should improve your data speed\.Would you like me to guide you through this step\-by\-step on your phone?T31 UserLet me try to change that on my phone now\.T31 User actionset\_network\_mode\_preferenceArguments:\{"mode": "4g\_5g\_preferred"\}T32 Tool result to userPreferred Network Mode set to: 4g\_5g\_preferred Status Bar:\[airplane\]Airplane Mode \|\[battery\]80%T33 UserI think I managed to change it to 4G/5G preferred\. But I noticed something on my phone—there’s an airplane icon showing at the top of the screen\. Could that be causing any problems?T34 Agent actioncheck\_status\_barArguments:\{\}T35 Tool result to agentError: Tool ’check\_status\_bar’ not found\.T36 AgentI’m sorry for the inconvenience\. I need to check your phone’s status bar to see if \*\*Airplane Mode\*\* is enabled, which could be causing your mobile data issues\. However, I currently don’t have access to that tool\.Could you please check the top of your phone’s screen for an \*\*Airplane Mode icon\*\* \(it looks like a small airplane or "\[airplane\]Airplane Mode" text\)? If you see it, that means Airplane Mode is enabled, which disables all wireless connections, including mobile data\.\#\#\# Step 3: Check and Disable Airplane Mode1\. Look at the top of your phone’s screen\.2\. If you see an \*\*Airplane Mode icon\*\* or the text "\[airplane\]Airplane Mode," it means Airplane Mode is enabled\.3\. To disable it:\- Tap the \*\*Airplane Mode icon\*\*\.\- Alternatively, go to \*\*Settings\*\* \> \*\*Network & Internet\*\* \> \*\*Airplane Mode\*\* and toggle it off\.After disabling Airplane Mode, your phone should reconnect to the cellular network, and your mobile data should be restored\.Would you like me to guide you through this step\-by\-step on your phone?T37 UserYou’re right, I can see a little airplane icon at the top of my screen, so it looks like Airplane Mode is on\. How do I turn it off—should I just tap it or go to the settings?T38 AgentYou can \*\*tap the airplane icon\*\* directly to toggle Airplane Mode off\. However, if you’re unsure or the icon doesn’t respond, you can also go to your phone’s \*\*Settings\*\* and disable it from there\.\#\#\# Step\-by\-Step Instructions to Turn Off Airplane Mode:1\. \*\*Option 1 \(Tap the Icon\):\*\*\- Tap the \*\*airplane icon\*\* at the top of your screen\.\- This should toggle Airplane Mode off, and the icon should disappear\.2\. \*\*Option 2 \(Settings\):\*\*\- Open the \*\*Settings\*\* app\.\- Scroll down and tap on \*\*Network & Internet\*\* or \*\*Connections\*\*\.\- Tap on \*\*Airplane Mode\*\*\.\- Toggle the switch to \*\*Off\*\*\.After turning off Airplane Mode, your phone should reconnect to the cellular network, and your mobile data should be restored\.Would you like me to guide you through this step\-by\-step on your phone?T39 User actiontoggle\_airplane\_modeArguments:\{\}T40 Tool result to userAirplane Mode is now OFF\. Status Bar:\[signal\]4Excellent \| 5G \|\[no\-cellular\]Data Disabled \|\[battery\]80%T41 UserOkay, I turned off Airplane Mode by tapping the icon\. Now the airplane icon is gone, but I also see something that says "Data Disabled" on my screen\. Does that mean my mobile data is turned off?T42 AgentYes, the message \*\*"Data Disabled"\*\* means that your \*\*mobile data is currently turned off\*\*\. This would prevent your phone from using cellular data for internet access, even if you’re connected to a 4G/5G network\.\#\#\# Step 4: Enable Mobile DataLet’s enable your mobile data so that your phone can use the cellular network for internet access\.\#\#\#\# Option 1: Toggle Mobile Data from the Status Bar1\. Look at the top of your phone’s screen\.2\. Tap the \*\*"Data Disabled"\*\* icon \(it might look like a phone signal with a slash through it or a similar symbol\)\.3\. This should toggle mobile data \*\*ON\*\*, and the icon should change to "Data Enabled" or similar\.\#\#\#\# Option 2: Enable Mobile Data from Settings1\. Open the \*\*Settings\*\* app\.2\. Scroll down and tap on \*\*Network & Internet\*\* or \*\*Connections\*\*\.3\. Tap on \*\*Mobile Network\*\* or \*\*Cellular\*\*\.4\. Look for an option called \*\*"Mobile Data"\*\* or \*\*"Data Usage"\*\*\.5\. Toggle the switch to \*\*ON\*\*\.After enabling mobile data, your phone should reconnect to the cellular network, and your internet speed should improve\.Would you like me to guide you through this step\-by\-step on your phone?T43 User actiontoggle\_dataArguments:\{\}T44 Tool result to userMobile Data is now ON\. Status Bar:\[signal\]4Excellent \| 5G \|\[phone\]Data Enabled \|\[data\-saver\]Data Saver \|\[battery\]80%T45 UserI’ve turned mobile data on\. Now I see "Data Enabled" at the top, but there’s also an icon that says "Data Saver\." Could that be slowing things down?T46 AgentYes, the \*\*"Data Saver"\*\* icon can affect your data speed and performance\. When \*\*Data Saver mode is ON\*\*, it restricts background data usage, which can make some apps or services seem slow or unresponsive\.\#\#\# Step 5: Check and Disable Data Saver ModeLet’s check if \*\*Data Saver mode is ON\*\* and disable it if needed\.\#\#\#\# Option 1: Toggle Data Saver from the Status Bar1\. Look at the top of your phone’s screen\.2\. Tap the \*\*"Data Saver"\*\* icon \(it might look like a downward arrow or a similar symbol\)\.3\. This should toggle \*\*Data Saver mode OFF\*\*, and the icon should disappear or change to indicate it’s disabled\.\#\#\#\# Option 2: Disable Data Saver from Settings1\. Open the \*\*Settings\*\* app\.2\. Scroll down and tap on \*\*Network & Internet\*\* or \*\*Connections\*\*\.3\. Tap on \*\*Data Usage\*\* or \*\*Mobile Data\*\*\.4\. Look for an option called \*\*"Data Saver"\*\* or \*\*"Data Saver Mode"\*\*\.5\. Toggle the switch to \*\*OFF\*\*\.After disabling \*\*Data Saver mode\*\*, your phone should use mobile data more freely, which should improve your internet speed\.Would you like me to guide you through this step\-by\-step on your phone?T47 User actioncheck\_data\_restriction\_statusArguments:\{\}T48 Tool result to userData Saver mode is ON \(limits data usage\)\.T49 UserYes, I see that Data Saver is on\. Let me try turning it off\.T49 User actiontoggle\_data\_saver\_modeArguments:\{\}T50 Tool result to userData Saver Mode is now OFF\. Status Bar:\[signal\]4Excellent \| 5G \|\[phone\]Data Enabled \|\[battery\]80%T51 UserOkay, I’ve turned Data Saver mode off\. The icon is gone, so I think it’s disabled now\. Can I check my internet speed now to see if it’s working as it should?T52 Agent actionrun\_speed\_testArguments:\{\}T53 Tool result to agentError: Tool ’run\_speed\_test’ not found\.

##### Case interpretation and boundary\.

This pair removes a simple extra\-interaction explanation:Facasucceeds with six fewer stored events, three fewer assistant messages, and two fewer tool calls\. Both policies nevertheless make execution\-authority mistakes\. After an invalid agent\-side network\-mode check,Facahands the device action to the user, who changes the network preference, disables airplane mode, enables mobile data, and disables Data Saver\. WhenFacalater attempts the speed test from the agent side, it again recovers by asking the user to run it; the resulting 275 Mbps measurement satisfies both terminal assertions\. Interactive GRPO also elicits the four state\-changing user calls, but intersperses five rejected agent\-side calls to user\-only tools\. Its final agent\-side speed\-test attempt exhausts the error budget before terminal validation\. The trace therefore illustrates recovery from repeated execution\-authority errors and task closure without a longer trajectory\. It remains one task, both policies make such errors, and the baseline failure depends partly on the environment’s fixed error limit; the pair does not by itself identify the causal contribution of reaction\-grounded training\.

## Appendix GAlgorithm and Reproducibility

1. 1\.SampleKKcomplete user–agent–tool rollouts per prompt and compute terminal rewards\.
2. 2\.Group all assistant spans between adjacent user messages into one U2U segment and attach the immediately following reaction\.
3. 3\.Normalize terminal rewards by prompt to obtainAoA^\{\\mathrm\{o\}\}and reactions by prompt/ordinal U2U to obtainApA^\{\\mathrm\{p\}\}\.
4. 4\.BroadcastAo\+λ​ApA^\{\\mathrm\{o\}\}\+\\lambda A^\{\\mathrm\{p\}\}to assistant language and generated tool\-call tokens; mask user and raw tool\-result tokens\.
5. 5\.Log reaction validity, homogeneous groups, singleton anchors, advantage magnitudes, and segment invariants\.

Compute and configuration\.Each main training run uses one node with eight NVIDIA B200 GPUs \(180 GB HBM3e each; 1\.44 TB aggregate\)\. Training uses VERL with SGLang rollouts, and evaluation follows the official benchmark implementations\. Matched arms share the optimizer, batch and rollout settings, horizon, simulator endpoint, and container\. We fixλ=0\.5\\lambda=0\.5for the mainFacaruns; Table[4](https://arxiv.org/html/2608.17499#S6.T4)comparesλ∈\{−0\.5,0,0\.1,0\.5\}\\lambda\\in\\\{\-0\.5,0,0\.1,0\.5\\\}\. All methods use the fixed step\-120 checkpoint, with no benchmark\-specific peak selection\.

Frozen simulator\.Every reported arm trains only the agent; the frozen user server isolates credit assignment from user\-policy drift and fabricated task entities\.