ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

# 1Introduction
Source: [https://arxiv.org/html/2608.13622](https://arxiv.org/html/2608.13622)
TECHNICAL REPORT

August 13, 2026

ARC: Fair Relative Advantage Comparison in Open\-Ended Real\-World Interaction

Yongqi Tong\*Tan Li Hui Faith\*Choy Zhen Wen Marcus\*Zhou Jin Kewei Fu Jiang\-Ming Yang Jianshe Li Xin Zhang

Ant International

\{tongyongqi\.yq, faith\.t, marcus\.choy, xiaocao\.zj, fukewei\.fkw, jmyang, zhouran\.ljs, evan\.zx\}@ant\-intl\.com

Framework Code:[ASRI](https://github.com/ant-intl/asri)Dataset:[![[Uncaptioned image]](https://arxiv.org/html/2608.13622v1/assets/huggingface-logo.png)ARC](https://huggingface.co/datasets/ant-intl/ARC)

AbstractOpen\-ended real\-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting\. This flexibility breaks a core assumption behind group\-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable\. As a result, reward\-model preferences over interaction style can distort relative advantages and steer optimization toward reward\-preferred behaviors rather than context\-appropriate ones\. We formalize this as areward fairness problemand proposeARC\(Advantage Regularization via Conditioning\), a training recipe that restores fairer relative comparison through strategy\-conditioned rollout grouping, together with hybrid rewards and entropy regularization\. We study ARC in our proposed INTER3, a novel paradigm for responsive, steerable, and execution\-aware user\-agent interaction that decouples user\-visible communication from latent reasoning and tool use\. INTER3also provides the annotation and distillation pipeline for constructing INTER3\-86K, our strategy\-annotated training corpus for supervised and RL training\. Empirically, ARC substantially strengthens the coreτ/τ2\\tau/\\tau^\{2\}tool\-use benchmarks, while INTER3reduces time\-to\-first\-token from 4\.91s to 1\.27s relative to a think\-style baseline\. Together, these results suggest that a central bottleneck in open\-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place\. The ARC implementation and INTER3\-86K training data will be released\.Keywords:Reinforcement learning, AI agents, Tool Use, Interaction Strategies, Advantage Estimation

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.13622v1/assets/v6.png)Figure 1:Overview of ARC\. ARC modifies group\-based RL by pairing each example with a strategy instruction during training, so rollouts are compared only within strategy\-conditioned groups, yielding cleaner relative advantages by eliminating cross\-strategy contamination\. At inference, the instruction is removed and the policy selects strategies autonomously, making ARC a general recipe for fairer advantage estimation in open\-ended interaction\.Group\-based RL and RLHF methods learn from relative rewards within sampled groups and have become a standard post\-training recipe for language models and agents[41](https://arxiv.org/html/2608.13622#bib.bib27);[1](https://arxiv.org/html/2608.13622#bib.bib37);[42](https://arxiv.org/html/2608.13622#bib.bib26);[63](https://arxiv.org/html/2608.13622#bib.bib40);[27](https://arxiv.org/html/2608.13622#bib.bib41)\. Their signal is most interpretable when the compared rollouts occupy a locally comparable response region, so that centered rewards primarily reflect quality rather than qualitatively different behavioral modes\. In practice, however, reward models are known to depend on spurious attributes such as response length and stylistic form, and RL policies can exploit these biases during optimization[6](https://arxiv.org/html/2608.13622#bib.bib38);[12](https://arxiv.org/html/2608.13622#bib.bib39)\. When a rollout group spans sharply different behaviors, these biases need not be uniform across the group: samples may fall in different regions of the reward model’s preference landscape, so the centered reward can absorb both quality differences and region\-specific reward preference\. The comparison remains relative, but it is no longer fair\.

This problem is especially acute in authentic real\-world interaction, where many tasks are open\-ended at the turn level and a single dialogue state may admit several appropriate next actions[62](https://arxiv.org/html/2608.13622#bib.bib5);[60](https://arxiv.org/html/2608.13622#bib.bib8);[4](https://arxiv.org/html/2608.13622#bib.bib9)\. An agent may answer directly, ask for clarification, provide progress updates while tools run, or confirm before an irreversible action; which choice is best can depend on missing information, action reversibility, and the evolving user\-agent state[60](https://arxiv.org/html/2608.13622#bib.bib8);[4](https://arxiv.org/html/2608.13622#bib.bib9);[11](https://arxiv.org/html/2608.13622#bib.bib28);[51](https://arxiv.org/html/2608.13622#bib.bib70)\. Unlike short\-form reasoning domains with a single verifiable target, these interaction choices often do not have a unique standard answer even when they are all task\-appropriate[60](https://arxiv.org/html/2608.13622#bib.bib8);[4](https://arxiv.org/html/2608.13622#bib.bib9)\. Comparing them inside one rollout group therefore entangles strategic diversity with reward\-model preference\. We formulate this confounding as thereward fairness problem: when multiple strategies are valid for the same prompt, cross\-strategy comparison biases group\-relative advantage estimation and can skew optimization toward reward\-preferred interaction styles\.

We address this problem withARC\(AdvantageRegularization viaConditioning\)\. ARC assigns a training\-time strategy instruction to each example, samples multiple rollouts within that strategy\-conditioned comparison class, computes relative advantages only within the group, and updates the policy with our hybrid reward and entropy\-regularized objective\. Figure[1](https://arxiv.org/html/2608.13622#S1.F1)gives a high\-level overview\. At inference, the strategy instruction is removed and the policy selects an interaction strategy autonomously\. This differs from hint\- or guidance\-based RL methods, which use auxiliary information to reveal solution structure, privilege stronger traces, or improve exploration[23](https://arxiv.org/html/2608.13622#bib.bib22);[18](https://arxiv.org/html/2608.13622#bib.bib23);[59](https://arxiv.org/html/2608.13622#bib.bib24);[66](https://arxiv.org/html/2608.13622#bib.bib25)\. ARC uses auxiliary information for a different purpose: not to make the answer easier to find, but to compare rollouts fairly with one another\.

However, existing agentic frameworks remain a weak substrate for studying this problem under authentic interaction, where users may interrupt, redirect, pause, or lose patience before execution completes, and where trajectories that achieve the same task outcome can still induce materially different interaction experiences[60](https://arxiv.org/html/2608.13622#bib.bib8);[4](https://arxiv.org/html/2608.13622#bib.bib9);[68](https://arxiv.org/html/2608.13622#bib.bib44);[24](https://arxiv.org/html/2608.13622#bib.bib45);[67](https://arxiv.org/html/2608.13622#bib.bib47)\. Think\-then\-act agents typically postpone user\-visible communication until the hidden reasoning\-and\-tool trajectory terminates, while ReAct agents intermingle reasoning, acting, and communication in a single visible trace[62](https://arxiv.org/html/2608.13622#bib.bib5);[22](https://arxiv.org/html/2608.13622#bib.bib48);[57](https://arxiv.org/html/2608.13622#bib.bib49)\. Both paradigms are optimized primarily for backbone capability and task success\. They therefore primarily accumulate trajectories with verifiable outcomes, rather than open\-ended interaction data in which multiple valid interleaving strategies may all solve the task yet induce different user experiences\. We therefore buildINTER3\(Interplay of Internal Reasoning, Tool Usage, and Interaction\), a channel\-separated framework whose system prompt explicitly defines interaction strategies aligned with the strategy families used by ARC\. By separating<answer\>spans from latent reasoning and tool execution, INTER3turns progress updates, clarification, alignment checks, and mid\-execution steering into first\-class, controllable, and annotatable behaviors; this both broadens dialogue\-style diversity and gives ARC cleaner rollout groups for fairer comparison\. The same interface reduces TTFT from 4\.91s to 1\.27s relative to a think\-then\-act baseline\. After deployment, we collect real online interaction traces in this format, augment them with curated public data, and synthesize large\-scale diversified interaction trajectories aligned with ARC’s strategy families, yieldingINTER3\-86K, a strategy\-annotated corpus of 86K examples and a realistic substrate for fair cross\-strategy comparison\.

Our theoretical analysis isolatesinter\-strategy varianceas a source of estimator error in standard group\-based RL and shows that ideal strategy conditioning removes this term from the centered\-advantage variance decomposition\. Our empirical results then show that, when trained on INTER3\-86K, ARC substantially improves open\-ended agentic RL for tool\-usage settings\. Additional diagnostics make the mechanism more concrete: strategy hints matter during training, while omitting them at inference recovers the best overall behavior, indicating that ARC is not merely fitting to prompt\-side instructions but learning a more general interaction policy\. Together, INTER3and ARC address both sides of the problem: INTER3provides a realistic interaction framework in which diverse valid behaviors can be observed and collected, while ARC resolves how those behaviors should be compared during optimization\. We view this combination as a practical step toward open\-ended real\-world agent training, where the central challenge is to learn from realistic interaction data while preserving fair relative comparison when multiple valid behaviors admit no single exact answer\.

Our contributions are threefold:

- •We proposeARC, a strategy\-conditioned group\-RL recipe for mitigating unfair advantage comparison in open\-ended real\-world agent interaction, and we analyze how it reducesinter\-strategy variancein group\-based advantage estimation\.
- •We buildINTER3, an async streaming interaction framework that separates user\-visible communication from latent reasoning and tool execution, supports explicit strategy control and user interruption, and makes open\-ended interaction observable for training\.
- •We constructINTER3\-86K, a strategy\-annotated training substrate from real online traces, curated public data, and synthetic trajectories, and show it enables ARC to improve open\-ended agentic RL while generalizing best to hint\-free inference\.

## 2Related Work

Tool\-augmented agents have made rapid progress in reasoning, planning, and external action[40](https://arxiv.org/html/2608.13622#bib.bib1);[34](https://arxiv.org/html/2608.13622#bib.bib2);[62](https://arxiv.org/html/2608.13622#bib.bib5);[11](https://arxiv.org/html/2608.13622#bib.bib28);[35](https://arxiv.org/html/2608.13622#bib.bib30);[49](https://arxiv.org/html/2608.13622#bib.bib64), while recent interleaving work has improved responsiveness by alternating between thought and partial output[22](https://arxiv.org/html/2608.13622#bib.bib48);[57](https://arxiv.org/html/2608.13622#bib.bib49)\. Our departure is architectural: INTER3separates visible interaction from latent reasoning and tool execution, so the user need not wait for the entire internal trajectory to finish before the system can communicate\. This turns responsiveness from a decoding behavior into an interface property, and makes strategy diversity operational in a realistic agent loop\.

On the RL side, our work builds on policy\-gradient and RLHF estimators[41](https://arxiv.org/html/2608.13622#bib.bib27);[1](https://arxiv.org/html/2608.13622#bib.bib37);[37](https://arxiv.org/html/2608.13622#bib.bib35);[13](https://arxiv.org/html/2608.13622#bib.bib36), especially group\-relative methods such as GRPO and its descendants[42](https://arxiv.org/html/2608.13622#bib.bib26);[63](https://arxiv.org/html/2608.13622#bib.bib40);[27](https://arxiv.org/html/2608.13622#bib.bib41)\. These methods are effective when samples within a group are meaningfully comparable\. Our focus is a different failure mode: in open\-ended agent interaction, valid rollouts may differ in strategy rather than only in quality, so cross\-strategy comparison can inject reward\-model preferences directly into relative advantage estimates\. ARC targets this comparability problem at rollout construction time\.

The closest methodological neighbors are guidance\-augmented RL approaches such as SAGE, Scaf\-GRPO, LUFFY, and ExGRPO[23](https://arxiv.org/html/2608.13622#bib.bib22);[18](https://arxiv.org/html/2608.13622#bib.bib23);[59](https://arxiv.org/html/2608.13622#bib.bib24);[66](https://arxiv.org/html/2608.13622#bib.bib25), which introduce auxiliary signals to improve exploration, mitigate sparse rewards, or reuse successful experience\. ARC also uses an auxiliary instruction, but for a different reason: not to make the task easier, but to define a cleaner comparison class for group\-relative optimization\. Additional discussion of adjacent tool\-use, interaction, and guidance literatures is deferred to Appendix[B](https://arxiv.org/html/2608.13622#A2)\.

## 3The INTER3Setting

### 3\.1Interaction Interface

ARC is evaluated in INTER3, a channel\-separated interaction setting for open\-ended agents\. The core design choice of INTER3is to separate the*interaction channel*from the*execution channel*\. Plain text outside<answer\>is treated as latent reasoning,<answer\>spans are streamed to the user, and tool calls remain structured internal actions whose results are returned to context\. This makes user\-visible communication available before the hidden reasoning\-and\-tool trajectory terminates, and it allows the same underlying task to be paired with different valid communication strategies\. The mechanism is model\-agnostic and requires only two implementation changes: adding<answer\>tags to the tokenizer, and post\-processing generated text to extract the user\-visible spans\. Figure[2](https://arxiv.org/html/2608.13622#S3.F2)illustrates the resulting execution pattern\.

![Refer to caption](https://arxiv.org/html/2608.13622v1/assets/inter3_v6.png)Figure 2:Standard think\-then\-answer \(left\) completes reasoning and tool execution before answering, resulting in high TTFT and no steering, whereas INTER3\(right\) interleaves reasoning, answering, and tool use, allowing real\-time steering and faster TTFT\.
### 3\.2Runtime Design Rationale

INTER3is instantiated in a lightweight async streaming agent runtime because standard agent traces are a weak substrate for the interaction behaviors studied in this paper\. Strict think\-then\-act pipelines delay user\-visible communication until execution completes, while monolithic ReAct\-style traces intermingle reasoning, acting, and communication in a single stream\. Neither produces clean open\-ended interaction data in which behaviors such as progress updates, clarification, alignment checks, and redirection are explicit and separately annotatable\.

Because<answer\>spans can be emitted before hidden execution completes, channel\-separated interaction enables a substantially lower\-latency operating regime\. As shown in Figure[3](https://arxiv.org/html/2608.13622#S3.F3), this reduces perceived latency while preserving continuous interaction during execution\.

Our runtime preserves these behaviors as first\-class events within a unified session\. A single agent loop can continue hidden reasoning, emit user\-visible<answer\>spans, issue tool calls, and respond to user interruptions, making interaction strategy observable throughout execution rather than only in the final outcome\. Two design choices are key for training\. First, prompt\-side strategy definitions are explicit and aligned with ARC conditioning families, so runtime control, annotation, and rollout grouping share consistent behavioral variables\. Second, partial outputs, interruptions, tool results, and resumed continuations are normalized into a unified trace format, enabling post\-deployment logs, curated public data, and synthetic trajectories to be converted into the same strategy\-conditioned training substrate\.

Figure 3:Latency\-capability trade\-off for interactive agent architectures\. Channel\-separated interaction shifts the operating point into a substantially lower\-latency regime, while ARC improves capability*within*that regime rather than causing the horizontal TTFT shift itself\. Non\-INTER3baselines are schematic architectural placements rather than exact benchmark claims\.This separation makes interaction strategy explicit: the same task can be completed through different valid communication patterns while sharing the same execution substrate\. That is precisely the regime in which cross\-strategy comparison becomes measurable\.

We organize these behaviors into four high\-level strategy families:Progress Update,Clarify First,Alignment Check, andDirect Answer\. ARC uses these families only as training\-time comparison classes, not as inference\-time requirements\. We instantiate this setting withINTER3\-86K, a strategy\-annotated corpus of 86\.8K examples spanning tool use, multi\-hop QA, and logical reasoning \(57\.9K SFT, 28\.9K RL with strategy instructions\)\. Since INTER3decouples communication from computation, it reduces user\-visible latency; we treat this as an architectural property rather than evidence for ARC\. Full strategy definitions and dataset construction details are deferred to Section[4](https://arxiv.org/html/2608.13622#S4)\.

## 4INTER3\-86K Construction

This section presents the construction of INTER3\-86K, including strategy taxonomy, data sources, curation, strategy annotation, and dataset statistics\.

### 4\.1Strategy Taxonomy

INTER3organizes interaction behavior into four high\-level families and nine concrete strategies as seen in Table[1](https://arxiv.org/html/2608.13622#S4.T1)\. ARC conditions rollout groups on the high\-level family during training; the finer\-grained strategies are useful for analysis, and interface documentation\.

Table 1:INTER3strategy taxonomy used for ARC conditioning, analysis, and data construction\.CategoryStrategyDescriptionProgress UpdateTool ExecuteExecute single or sequential tool calls; return the final answer upon completionParallel ToolsExecute independent tool calls simultaneously; synthesize combined resultsMulti\-Step UpdateStream incremental progress updates to the user between sequential sub\-tasksSilent ExecutionExecute tool calls without emitting user\-visible output when intermediate steps are irrelevantError RecoveryHandle tool failures gracefully and recover with corrective actionsClarify FirstClarify FirstAsk the user for missing or ambiguous information before executing any tool callAlignment CheckAlignment CheckRestate the user’s intent for confirmation before executing an irreversible actionDecision SupportPresent options and trade\-offs for the user to make a judgmentDirect AnswerDirect AnswerAnswer immediately from internal knowledge without invoking any tools
### 4\.2Data Sources and Curation

We construct INTER3\-86K from two sources\. The first comes from real\-world deployment in customer service on a large\-scale global payment platform under the INTER3runtime, where we collect online interaction traces exhibiting interruption, redirection, clarification, progress updates, and multi\-step tool use\. After de\-identification and normalization into the INTER3format, these traces provide realistic open\-ended interaction patterns that are difficult to recover from standard offline benchmarks alone\.

The second source combines curated public benchmarks with teacher\-driven augmentation, distillation, and synthesis\. Public tool\-use and reasoning data are rewritten into the INTER3format, expanded with strategy\-conditioned variants, and supplemented with diversified synthetic interaction trajectories produced with a strong teacher model, Qwen3\.5\-397B\-A17B[36](https://arxiv.org/html/2608.13622#bib.bib50)\. The resulting SFT split contains 57\.9K examples and mixes tool\-use, multi\-hop QA, and logical reasoning: 34\.2K tool\-use examples \(59\.1%\) from the ToolMind collection, which aggregates public function\-calling and agent benchmarks[10](https://arxiv.org/html/2608.13622#bib.bib4);[26](https://arxiv.org/html/2608.13622#bib.bib11);[60](https://arxiv.org/html/2608.13622#bib.bib8);[69](https://arxiv.org/html/2608.13622#bib.bib15);[2](https://arxiv.org/html/2608.13622#bib.bib16);[39](https://arxiv.org/html/2608.13622#bib.bib17);[8](https://arxiv.org/html/2608.13622#bib.bib18); 17\.2K multi\-hop QA examples \(29\.7%\) from Musique Long Content[52](https://arxiv.org/html/2608.13622#bib.bib7); and 6\.4K logical\-reasoning or high\-quality interleaved examples \(11\.1%\) from KnightsAndKnaves[56](https://arxiv.org/html/2608.13622#bib.bib58)and Opus Distilled[46](https://arxiv.org/html/2608.13622#bib.bib19)\. The RL split contains 28\.9K examples and is drawn entirely from tool\-use data, to which we add strategy\-conditioned prompts for ARC training\. Figure[4](https://arxiv.org/html/2608.13622#S4.F4)summarizes the resulting domain and strategy distributions\.

### 4\.3SFT Data Construction

The SFT set draws from four data families with complementary reasoning demands\. The largest component \(∼\\sim34\.2K, 59\.1%\) is the ToolMind collection, which aggregates seven publicly available tool\-use benchmarks spanning diverse agentic scenarios\. Since these benchmarks include ground\-truth chain\-of\-thought annotations, we use a 397B LLM \(Qwen3\.5\-397B\-A17B[36](https://arxiv.org/html/2608.13622#bib.bib50)\) to rewrite each assistant turn into the INTER3format—interleaving internal reasoning with<answer\>blocks and explicit tool calls—then filter malformed dialogues \(invalid turn orderings, empty<answer\>blocks\)\. Three supplementary sources diversify the model’s interleaved reasoning beyond tool calling:KnightsAndKnaves\(∼\\sim6\.2K, 10\.7%\) for multi\-step deductive inference,Musique Long Content\(∼\\sim17\.2K, 29\.7%\) for multi\-hop reasoning over extended passages, andOpus Distilled\(∼\\sim250, 0\.4%\) for high\-quality interleaved conversations generated via Claude Opus\.

### 4\.4RL Data Construction

The RL set \(∼\\sim28\.9K\) reuses the seven ToolMind sub\-datasets but introduces training\-time strategy instructions essential for ARC training\. Each conversation undergoes a two\-phase process: \(1\) a strategy\-annotation pipeline assigns one*plausible*strategy instruction from our taxonomy—Progress Update,Direct Answer,Clarify First, orAlignment Check—based on the query, context, and response; \(2\) the selected instruction is injected into the system prompt, providing an explicit behavioral condition forRL\+ARCtraining\. These prompt\-side strategy definitions are aligned with the high\-level strategy families used by ARC, so the interaction interface, data annotation pipeline, and rollout comparison classes share the same control variables\. In contrast,baseline RLtraining is performed on the same data without any injected strategy instruction\.

Figure 4:Left: INTER3strategy distribution in the RL training dataset\. Progress Update dominates at 75\.3%, reflecting the prevalence of multi\-step tool\-use scenarios that benefit from incremental user updates\. Right: data domain composition of the full training set \(SFT \+ RL\)\. Tool Use accounts for 72\.8% of examples, with Multi\-hop QA \(19\.8%\) and Logical Reasoning \(7\.1%\) providing complementary reasoning diversity\.
### 4\.5Strategy Annotation

A core contribution of our dataset is the explicit strategy annotation for each example\. Unlike prior work that treats agent behavior as monolithic, we recognize that effective interaction requires behavioral diversity adapted to context\. We develop a rigorous annotation pipeline combining model\-based labeling with collaborative verification\.

We annotate examples on the four high\-level strategies—Progress Update,Clarify First,Alignment Check, andDirect Answer\.

#### 4\.5\.1Collaborative Annotation Pipeline

We employ a collaborative annotation approach using two large language models \(Qwen3\-235B\-Instruct[45](https://arxiv.org/html/2608.13622#bib.bib21)and Qwen3\.5\-27B[36](https://arxiv.org/html/2608.13622#bib.bib50)\) to ensure annotation quality and consistency\. This dual\-model verification captures the inherent subjectivity in strategy assignment while maintaining high inter\-annotator agreement\.

Stage 1: Independent Annotation\.For each example, both models independently predict a plausible primary strategy given:

- •The user query
- •The conversation history
- •The reference assistant response
- •The 4 high\-level strategies taxonomy with detailed definitions and examples

Each model outputs: \(1\) a predicted strategy assignment, \(2\) a confidence score, and \(3\) a brief justification\.

Stage 2: Agreement and Disagreement Resolution\.

- •Agreement cases: When both models predict the same strategy with confidence\>\>0\.85, we accept the annotation directly\.
- •High\-confidence disagreement: When models disagree but one has significantly higher confidence \(Δ\>0\.15\\Delta\>0\.15\), we accept the higher\-confidence prediction\.
- •Low\-confidence or ambiguous disagreement: We escalate to human review\. Three trained annotators independently label the example\. We determine the final annotation using a majority vote\.

### 4\.6Dataset Statistics

Table[2](https://arxiv.org/html/2608.13622#S4.T2)presents comprehensive statistics of the final dataset\.

Table 2:Inter\-86Ksummary statistics\. Token counts use whitespace tokenization\.

## 5Our Methodology: Advantage Regularization via Conditioning

While the INTER3setting makes interaction strategy observable, it does not solve the core RL challenge: how to compare responses fairly when they follow different communication patterns\. We presentARC\(AdvantageRegularization viaConditioning\), a conditioning\-based RL method for open\-ended agent training\. ARC’s core mechanism is*strategy\-conditioned rollout grouping*, which changes the comparison unit in group\-based RL: relative advantages are computed within a strategy\-conditioned subspace rather than across heterogeneous behaviors\.

### 5\.1Problem: Unfair Advantages in Multi\-Strategy RL

In group\-based RL, given promptxx, we sampleNNresponses\{y1,…,yN\}\\\{y\_\{1\},\\ldots,y\_\{N\}\\\}fromπθ\(⋅\|x\)\\pi\_\{\\theta\}\(\\cdot\|x\)and compute advantagesA^i=ri−r¯\\hat\{A\}\_\{i\}=r\_\{i\}\-\\bar\{r\}, assuming comparable samples\. When responses follow different strategies, this breaks: reward model bias contaminates the advantage signal, skewing policy updates toward reward\-preferred behaviors regardless of task appropriateness\.

###### Definition 1\(δ\\delta\-Reward Fairness\)\.

A reward modelℛ\\mathcal\{R\}isδ\\delta\-fairw\.r\.t\. strategy set𝒮\\mathcal\{S\}if for any promptxxand responsesyi,yjy\_\{i\},y\_\{j\}following different strategies with equal quality:\|ℛ⁡\(x,yi\)−ℛ⁡\(x,yj\)\|≤δ\|\\mathcal\{R\}\(x,y\_\{i\}\)\-\\mathcal\{R\}\(x,y\_\{j\}\)\|\\leq\\delta\. A reward model isfairifδ=0\\delta=0, andunfairwhenδ\>0\\delta\>0introduces systematic bias across strategies\.

In practice, reward models violate this fairness property due to exposure, length, and style bias—especially pronounced in agent settings where interaction appropriateness lacks clear ground truth\.

### 5\.2Strategy\-Conditioned Advantage Estimation

The following results analyze a stylized prompt\-conditional reward decomposition of the formr=μs​\(x\)\+ϵr=\\mu\_\{s\}\(x\)\+\\epsilon\. They isolate one source of estimator variance in group\-relative advantages and should not be read as end\-to\-end convergence guarantees for GRPO\. This estimator\-centric viewpoint is closer to classical variance\-reduction analyses for policy gradients[16](https://arxiv.org/html/2608.13622#bib.bib42)and recent RLHF\-side discussions of estimator behavior[1](https://arxiv.org/html/2608.13622#bib.bib37);[27](https://arxiv.org/html/2608.13622#bib.bib41)than to a full convergence theory\. For clarity, the theoretical comparison uses equal group size across the conditioned and unconditioned cases\.

###### Theorem 5\.1\(Idealized Variance Amplification\)\.

Under standard group\-based RL sampling, the advantage variance isVar⁡\[A^i\]=\(σinter2\+σintra2\)​\(1−1/N\)\\mathrm\{Var\}\[\\hat\{A\}\_\{i\}\]=\(\\sigma^\{2\}\_\{\\text\{inter\}\}\+\\sigma^\{2\}\_\{\\text\{intra\}\}\)\(1\-1/N\), whereσintra2\\sigma^\{2\}\_\{\\text\{intra\}\}captures within\-strategy noise andσinter2\\sigma^\{2\}\_\{\\text\{inter\}\}captures between\-strategy variance from reward model bias\.

The inter\-strategy componentσinter2\\sigma^\{2\}\_\{\\text\{inter\}\}persistsunder standard sampling: no amount of oversampling can eliminate it\. Our solution is to condition each rollout group on a specific strategy\.

###### Definition 2\(Strategy\-Conditioned Sampling\)\.

For each training example, we construct a strategy\-conditioned promptp∗p^\{\*\}by attaching a strategy instructions∗s^\{\*\}from our interaction taxonomy, then sampleMMresponses fromπθ\(⋅\|x,s∗\)\\pi\_\{\\theta\}\(\\cdot\|x,s^\{\*\}\)\.

###### Theorem 5\.2\(Idealized Variance Reduction via Conditioning\)\.

Under strategy\-conditioned sampling with the same group size:Var⁡\[A^i\|s∗\]=σintra2⋅\(1−1/N\)\\mathrm\{Var\}\[\\hat\{A\}\_\{i\}\|s^\{\*\}\]=\\sigma^\{2\}\_\{\\text\{intra\}\}\\cdot\(1\-1/N\)\. Under perfect conditioning, the between\-strategy mean\-shift term is absent from this centered\-advantage variance decomposition\.

Intuition\.Heuristically, strategy conditioning emphasizes within\-strategy reward dependenceI\(r;y∣x,s\)I\(r;y\\mid x,s\)while suppressing the strategy\-linked termI⁡\(r;s∣x\)I\(r;s\\mid x\)that can contaminate cross\-strategy comparison\. Unlike hint\-based RL methods[23](https://arxiv.org/html/2608.13622#bib.bib22);[18](https://arxiv.org/html/2608.13622#bib.bib23)that use hints to guide toward correct answers, we use strategy instructions to enforce cleaner within\-strategy comparisons across diverse behaviors\. See Appendix[A](https://arxiv.org/html/2608.13622#A1)for proofs and analysis\.

Training\-time strategy instructions do introduce a train\-inference mismatch because they are removed at deployment\. We study progressive instruction removal empirically in Section[6\.4](https://arxiv.org/html/2608.13622#S6.SS4)as a further analysis rather than as part of ARC itself\.

### 5\.3Entropy Regularization for Multi\-Channel Generation

The ARC training objective combines strategy\-conditioned policy gradients with an entropy bonus:

ℒ\(θ\)=−∑i=1MA^i\(s∗\)⋅logπθ\(yi\(s∗\)\|x,s∗\)−β⋅H\(πθ\(⋅\|x,s∗\)\)\.\\mathcal\{L\}\(\\theta\)=\-\\sum\_\{i=1\}^\{M\}\\hat\{A\}\_\{i\}^\{\(s^\{\*\}\)\}\\cdot\\log\\pi\_\{\\theta\}\(y\_\{i\}^\{\(s^\{\*\}\)\}\|x,s^\{\*\}\)\-\\beta\\cdot H\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\|x,s^\{\*\}\)\\right\)\.\(1\)Why entropy regularization is critical for INTER3\.The interleaved output format requires balancing three channels—internal reasoning, user\-facing<answer\>tags, and tool calls\. Without the entropy bonus \(β=0\\beta=0\), we observeentropy collapse[19](https://arxiv.org/html/2608.13622#bib.bib51): the policy converges to emitting redundant<answer\>blocks that receive marginal format rewards but carry no meaningful content\. The entropy bonus counteracts this collapse by maintaining stochasticity across all output channels, encouraging diverse exploration of valid interleaving patterns\.

### 5\.4Idealized Sample\-Efficiency View

###### Theorem 5\.3\(Idealized Gradient\-Sample Scaling\)\.

Under the same stylized assumptions, if gradient\-estimation error scales with the second moment of the score\-function estimator, then the rollout requirement to estimate∇J​\(θ\)\\nabla J\(\\theta\)up to accuracyε\\varepsilonscales asnstd=O⁡\(\(σintra2\+σinter2\)​log⁡\(1/δ\)ε2\)n\_\{\\text\{std\}\}=O\\\!\\left\(\\frac\{\(\\sigma^\{2\}\_\{\\text\{intra\}\}\+\\sigma^\{2\}\_\{\\text\{inter\}\}\)\\log\(1/\\delta\)\}\{\\varepsilon^\{2\}\}\\right\)for standard GRPO andnARC=O⁡\(σintra2​log⁡\(1/δ\)ε2\)n\_\{\\text\{ARC\}\}=O\\\!\\left\(\\frac\{\\sigma^\{2\}\_\{\\text\{intra\}\}\\log\(1/\\delta\)\}\{\\varepsilon^\{2\}\}\\right\)for ARC\.

Corollary 1\.Under the same assumptions, the implied sample\-efficiency ratio isnstd/nARC=1\+σinter2/σintra2n\_\{\\text\{std\}\}/n\_\{\\text\{ARC\}\}=1\+\\sigma^\{2\}\_\{\\text\{inter\}\}/\\sigma^\{2\}\_\{\\text\{intra\}\}\. This is best read as an idealized scaling comparison rather than a full convergence guarantee, in the same spirit as estimator\-level policy\-gradient analyses[16](https://arxiv.org/html/2608.13622#bib.bib42);[65](https://arxiv.org/html/2608.13622#bib.bib43)\. We use it as an estimator\-level interpretation of why cleaner within\-strategy comparison can reduce the gradient\-sampling burden, rather than as a calibrated empirical speedup claim\. See Appendix[A](https://arxiv.org/html/2608.13622#A1)for the derivation and assumptions\.

### 5\.5Implementation

ARC consists of four steps \(Figure[1](https://arxiv.org/html/2608.13622#S1.F1)\):

1. 1\.Strategy Instruction Assignment: each training example is paired with one strategy instructions∗s^\{\*\}from the interaction taxonomy\.
2. 2\.Within\-Strategy Sampling: sampleMMrollouts fromπθ\(⋅∣x,s∗\)\\pi\_\{\\theta\}\(\\cdot\\mid x,s^\{\*\}\)\.
3. 3\.Advantage Computation: compute advantages within each strategy group\.
4. 4\.Policy Update: update the policy using the entropy\-regularized objective\.

At inference, no strategy instruction is provided; the model autonomously selects appropriate strategies\. Section[6\.4](https://arxiv.org/html/2608.13622#S6.SS4)further analyzes instruction\-removal curricula as an auxiliary experiment\.

## 6Experiments

### 6\.1Experimental Setup

Model and training\.We use Qwen3\-8B[45](https://arxiv.org/html/2608.13622#bib.bib21)trained with GRPO[42](https://arxiv.org/html/2608.13622#bib.bib26)in no\-think mode, warm\-started from the same INTER3SFT checkpoint for all RL backbones\. ARC’s core mechanism is strategy\-conditioned rollout grouping; in the full INTER3instantiation additionally apply entropy regularization as a stabilizer for multi\-channel generation\. Unless explicitly varied, all methods use the same final reward construction, selected based on ablations in Appendix[C](https://arxiv.org/html/2608.13622#A3)\. Additional ablations on entropy regularization are provided in Appendix[D](https://arxiv.org/html/2608.13622#A4)\.

Benchmarks\.We evaluate on two dimensions: \(1\)In\-domain agentic capabilitiesusing tau\-bench[60](https://arxiv.org/html/2608.13622#bib.bib8)and tau2\-bench[4](https://arxiv.org/html/2608.13622#bib.bib9)for multi\-turn tool calling in airline, retail, and telecom scenarios; \(2\)Out\-of\-domain reasoningusing Arena\-Hard[20](https://arxiv.org/html/2608.13622#bib.bib10), AIME 2026[3](https://arxiv.org/html/2608.13622#bib.bib14), GPQA\-Diamond[38](https://arxiv.org/html/2608.13622#bib.bib12), IFBench[32](https://arxiv.org/html/2608.13622#bib.bib13), and HMMT[3](https://arxiv.org/html/2608.13622#bib.bib14)\.

Baseline methods\.We benchmark ARC against two classes of baselines: \(1\)Qwen3\-8B\-noThink / Think, minimal reasoning variants to quantify ARC’s gains; \(2\)Standard RL backbones\(PPO[41](https://arxiv.org/html/2608.13622#bib.bib27), DAPO[63](https://arxiv.org/html/2608.13622#bib.bib40), GRPO[42](https://arxiv.org/html/2608.13622#bib.bib26)\), representing widely\-used reinforcement learning methods\.

### 6\.2Main Results

![Refer to caption](https://arxiv.org/html/2608.13622v1/assets/reward_curves_v2.png)Figure 5:Training reward curves for GRPO and ARC\.Table 3:Main results\. ARC delivers clearest gains on in\-domain tool use, especially on GRPO\. Avg\. is the mean over displayed metrics; red parentheses show Avg\. change vs\. backbone\. TTFT \(↓\\downarrow\) averages availableτ\\tau\-bench latency measurements\.In\-domain performance\.Table[3](https://arxiv.org/html/2608.13622#S6.T3)shows that ARC consistently improves performance across all RL backbones, with the largest gains for GRPO \(\+5\.37 average points\)\. Improvements are especially pronounced in in\-domain tool\-calling tasks: for GRPO,τ\\tau\-airline rises from 31\.33 to 44\.00,τ\\tau\-retail from 40\.29 to 50\.00, andτ2\\tau^\{2\}\-airline from 36\.67 to 48\.00, with smaller gains inτ2\\tau^\{2\}\-retail andτ2\\tau^\{2\}\-telecom\. These results demonstrate ARC’s ability to enhance multi\-turn tool use and in\-domain capabilities, particularly with stronger RL backbones\.

Out\-of\-domain reasoning and tradeoffs\.On out\-of\-domain reasoning, instruction following, and alignment benchmarks, ARC shows more nuanced effects\. GRPO \+ ARC improves reasoning on AIME 2026 from 31\.67 to 40\.83 and provides modest gains in instruction\-following and alignment metrics, while improvements for PPO and DAPO backbones are mixed, suggesting that ARC’s benefits depend on the underlying optimization dynamics\. These results show that while ARC’s primary impact is on structured tool use, it can also unlock meaningful out\-of\-domain reasoning gains under suitable training regimes\.

Training dynamics and robustness\.Figure[5](https://arxiv.org/html/2608.13622#S6.F5)shows that ARC’s benefits extend beyond final performance to training dynamics\. While GRPO peaks mid\-training and then declines—especially in tool\-call and answer rewards—ARC maintains or improves across the same budget\. Since format reward saturates for both methods, this divergence reflects execution and answer quality rather than syntactic compliance\. This suggests that standard group\-based RL initially learns useful behaviors but then exploits reward model biases, whereas ARC’s strategy\-conditioned comparison prevents degradation by preserving fair advantage signals throughout training\.

### 6\.3Small\-Model Case Study

Table 4:Qwen3\-4B case study\. ARC remains effective at smaller scale and outperforms no\-think, think, and 4B GRPO baselines on the in\-domain tool\-use suite\.Table[4](https://arxiv.org/html/2608.13622#S6.T4)shows that the agentic benefit of ARC is not specific to the 8B scale\. Even at 4B, ARC remains stronger than no\-think, think, and the corresponding 4B GRPO baseline on the tool\-use suite, increasing the five\-task average by about 53% over 4B no\-think\. This finding suggests that ARC improves performance by strengthening the training comparison signal in agentic settings, rather than simply leveraging larger model capacity\.

### 6\.4Curriculum Learning over Training\-Time Strategy Instructions

ARC uses strategy instructions during training but removes them at inference, motivating a curriculum\-learning question: should the conditioning signal be gradually weakened so the policy learns more autonomous strategy selection without losing ARC’s variance\-control benefit\. Inspired by curriculum\-learning methods in RL[5](https://arxiv.org/html/2608.13622#bib.bib52);[15](https://arxiv.org/html/2608.13622#bib.bib53);[14](https://arxiv.org/html/2608.13622#bib.bib54);[28](https://arxiv.org/html/2608.13622#bib.bib55), we study this as a diagnostic analysis of the trade\-off between train\-inference alignment and within\-strategy comparability\.

We compare three settings: \(1\)No removal, the default ARC setup that always retains the strategy instruction; \(2\)Linear removal, where the instruction\-drop probability increases over training according to Appendix[A](https://arxiv.org/html/2608.13622#A1),pd​\(t\)=pmin\+\(pmax−pmin\)​tTp\_\{d\}\(t\)=p\_\{\\min\}\+\(p\_\{\\max\}\-p\_\{\\min\}\)\\frac\{t\}\{T\}; and \(3\)Constant removal, which drops instructions with fixed probabilitypd=0\.20p\_\{d\}=0\.20throughout training\.

![Refer to caption](https://arxiv.org/html/2608.13622v1/assets/dropout_ablation_allv3.png)Figure 6:Training\-time instruction removal underperforms the default ARC setup\. Left: average number of off\-strategy rollouts per group, where an rollout has a predicted strategy different from the injected strategy\. Right: standard deviation of off\-strategy counts across groups\. Lower values are better in both panels, indicating more consistent within\-strategy rollout grouping\.Table 5:Curriculum learning over strategy\-instruction removal does not improve ARC: no removal performs best, linear removal is intermediate, and constant removal is weakest\.Table[5](https://arxiv.org/html/2608.13622#S6.T5)and Figure[6](https://arxiv.org/html/2608.13622#S6.F6)show that progressively removing strategy instructions does not improve ARC in our setting\. Default ARC achieves the strongestτ/τ2\\tau/\\tau^\{2\}tool\-use performance and the lowest off\-strategy mean and variance, indicating more stable within\-strategy rollout grouping\. Linear removal provides a partial trade\-off by modestly improving some reasoning metrics at the cost of agentic performance, while constant removal performs worst overall\. These results suggest that strategy instructions are most effective as a persistent training\-time variance\-control mechanism rather than a signal to be annealed away\. This is consistent with ARC’s mechanism: strategy instructions are most useful as a training\-time variance\-control device, and annealing them away too early weakens within\-strategy comparability instead of improving the final policy\.

### 6\.5Strategy Scalability Analysis

We conduct an additional ablation study of strategy scaling in Table[6](https://arxiv.org/html/2608.13622#S6.T6), which reveals a clear performance trajectory as strategy components are incrementally integrated, moving from the highest to the lowest training data volume\. TheAvg\.shows a consistent upward trend, achieving an impressive 36\.1% total improvement\. The most significant gains are concentrated in agentic benchmarks, where performance onτ\\tau\-bench andτ2\\tau^\{2\}\-bench surged by 99% and 71%, respectively, as the model transitioned from a single strategy to the full suite\.

![Refer to caption](https://arxiv.org/html/2608.13622v1/assets/radar_chart_domain_instruction.png)Figure 7:Domain\-instruction interaction analysis\. Inference\-time strategy hints show no uniformly superior deployment interface\.These results highlight not only the effectiveness of the method in leveraging heterogeneous strategies, but also its scalability: as additional strategies are added, the model continues to improve, demonstrating that even minority strategies—such asAlignment Check, which constitutes only 2\.5% of the data—contribute meaningfully to overall capabilities\. This highlights the potential for accommodating larger strategy sets without performance degradation\.

Table 6:Strategy scaling ablation\. Incremental strategy scaling consistently drives overall performance gains: the full strategy suite performs best, while reasoning tasks peak with three strategies and agentic benchmarks show the most dramatic sensitivity to strategy diversity\.
### 6\.6Additional Diagnostics on Strategy Instructions

We evaluate whether the final ARC policy depends on explicit strategy hints at deployment, or whether the strategy behavior has already been internalized during training\. We compare matched, randomized, and removed instructions on the same ARC checkpoint across allτ\\tau\-bench andτ2\\tau^\{2\}\-bench submetrics to isolate the effect of prompt\-side cues\.

Conclusively, Figure[7](https://arxiv.org/html/2608.13622#S6.F7)shows that inference\-time strategy hints do not improve the deployed policy\. The baseline with no strategy hint removal managed to achieve the highest overall average \(41\.73\), outperforming both Randomized Instruction \(39\.58\) and Matched Instruction \(39\.85\)\. Domain\-level analysis reveals that matched instructions help onτ\\tauAirline but hurt onτ2\\tau^\{2\}Airline, while all settings struggle onτ2\\tau^\{2\}Telecom\. This domain\-dependent variability confirms that inference\-time hints lack a reliable deployment benefit\. Strategy instructions serve their primary function during training by structuring comparison groups, not as persistent cues that should remain at inference\.

## 7Conclusion

Open\-ended agent interaction often admits multiple valid behaviors, yet standard group\-based RL compares rollouts within a shared relative\-reward pool\. We argue this induces a reward fairness issue: advantage estimates become confounded when heterogeneous interaction strategies are normalized together, rather than reflecting intrinsic quality differences\.

ARC addresses this failure by conditioning the comparison class during rollout construction, ensuring that each group contains a single strategy family\. This targeted modification improves the interpretability of relative advantages in multi\-strategy regimes, rather than altering the underlying policy optimization machinery\. Complementarily, the INTER3setting makes this issue observable in practice by separating user\-facing communication from latent reasoning and tool execution, thereby exposing diverse valid interaction strategies within the same task family\.

Empirically, ARC is most effective in in\-domain agentic benchmarks where multi\-strategy interaction is prevalent\. Training on INTER3\-86K yields higher and more stable rewards, including reduced post\-peak degradation under fixed compute budgets\. However, improvements are not uniform across all backbones or downstream metrics, reinforcing that ARC should be viewed as a targeted correction for comparison bias rather than a universally dominant optimizer\.

Our theoretical analysis provides a mechanism\-level explanation for these effects\. The results in Section[5](https://arxiv.org/html/2608.13622#S5)are stylized estimator\-level characterizations, not end\-to\-end convergence guarantees\. They show how inter\-strategy variance in reward\-model evaluations can degrade sample efficiency, and how restricting the comparison class can reduce this variance under strategy\-dependent reward bias\. This supports the empirical findings without claiming to fully characterize RL training dynamics\.

Overall, the results suggest that progress in open\-ended agent learning depends not only on stronger models or reward signals, but also on how learning algorithms construct fair comparison sets when multiple valid behaviors coexist\.

## Limitations

Our study has four main limitations\.

\(1\) Strategy abstraction\.ARC relies on a coarse strategy taxonomy\. While useful for conditioning and analysis, real interaction behaviors are more nuanced and context\-dependent[25](https://arxiv.org/html/2608.13622#bib.bib60);[29](https://arxiv.org/html/2608.13622#bib.bib65)\.

\(2\) Domain scope\.Our strongest results are in open\-ended tool\-usage settings\. Whether ARC generalizes to other domains remains to be tested\.

\(3\) Data and annotation dependence\.The method depends on normalized interaction traces and strategy labels, which may reflect annotation bias or deployment\-specific patterns\.

\(4\) Theoretical scope\.Variance and sample\-efficiency analyses clarify one optimization mechanism but do not constitute a full convergence theory for open\-ended agent RL\.

## References

- A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin,et al\.Back to basics: revisiting reinforce\-style optimization for learning from human feedback in llms\.arXiv preprint arXiv:2402\.14740\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1),[§1](https://arxiv.org/html/2608.13622#S1.p1.1),[§2](https://arxiv.org/html/2608.13622#S2.p2.1),[§5\.2](https://arxiv.org/html/2608.13622#S5.SS2.p1.1)\.
- AI \(2024\)G\. AIGlaive: function calling datasets for llms\.Note:[https://glaive\.ai](https://glaive.ai/)Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Balunovićet al\.\(2025\)M\. Balunović, J\. Dekoninck, I\. Petrov, N\. Jovanović, and M\. VechevMathArena: evaluating LLMs on uncontaminated math competitions\.arXiv preprint arXiv:2505\.23281\.Cited by:[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px3.p1.1),[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px5.p1.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p2.1)\.
- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.13622#S1.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p2.1)\.
- Bengioet al\.\(2009\)Y\. Bengio, J\. Louradour, R\. Collobert, and J\. WestonCurriculum learning\.InProceedings of the 26th Annual International Conference on Machine Learning,pp\. 41–48\.Cited by:[§6\.4](https://arxiv.org/html/2608.13622#S6.SS4.p1.1)\.
- Chenet al\.\(2024a\)L\. Chen, C\. Zhu, D\. Soselia, J\. Chen, T\. Zhou, T\. Goldstein, H\. Huang, M\. Shoeybi, and B\. CatanzaroODIN: disentangled reward mitigates hacking in rlhf\.arXiv preprint arXiv:2402\.07319\.Cited by:[§1](https://arxiv.org/html/2608.13622#S1.p1.1)\.
- Chenet al\.\(2025\)M\. Chen, C\. Dai, X\. Dong, C\. Feng, K\. Fu, J\. Li, Z\. Peng, Y\. Tong, J\. Zhang, and H\. ZhuDingTalk DeepResearch: a unified multi agent framework for adaptive intelligence in enterprise environments\.arXiv preprint arXiv:2510\.24760\.External Links:[Link](https://arxiv.org/abs/2510.24760)Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Chenet al\.\(2024b\)M\. Chen, H\. Sun, T\. Li, F\. Yang, H\. Liang, K\. Lu, B\. Cui, W\. Zhang, Z\. Zhou, and W\. ChenFacilitating multi\-turn function calling for LLMs via compositional instruction tuning\.arXiv preprint arXiv:2410\.12952\.Note:Accepted to ICLR 2025Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1)\.
- Duet al\.\(2024a\)L\. Du, Y\. Li, X\. Wang, Z\. Wang, Z\. Zhang, Z\. Xu, X\. Chen,et al\.APIGen: automated pipeline for generating verifiable and diverse function\-calling datasets\.arXiv preprint arXiv:2406\.18518\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1),[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Duet al\.\(2024b\)Y\. Du, F\. Fan, and D\. PanAnyTool: self\-reflective, hierarchical agents for large\-scale api calls\.arXiv preprint arXiv:2402\.04253\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1),[§1](https://arxiv.org/html/2608.13622#S1.p2.1),[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Duanet al\.\(2026\)Z\. Duan, G\. Rong, Z\. Li, B\. Chen, M\. Zhou, and D\. GuoMitigating reward hacking in rlhf via bayesian non\-negative reward modeling\.arXiv preprint arXiv:2602\.10623\.Cited by:[§1](https://arxiv.org/html/2608.13622#S1.p1.1)\.
- Ethayarajhet al\.\(2024\)K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. KielaKTO: model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1),[§2](https://arxiv.org/html/2608.13622#S2.p2.1)\.
- Florensaet al\.\(2018\)C\. Florensa, D\. Held, X\. Geng, and P\. AbbeelAutomatic goal generation for reinforcement learning agents\.InProceedings of the 35th International Conference on Machine Learning,pp\. 1515–1528\.Cited by:[§6\.4](https://arxiv.org/html/2608.13622#S6.SS4.p1.1)\.
- Florensaet al\.\(2017\)C\. Florensa, D\. Held, M\. Wulfmeier, and P\. AbbeelReverse curriculum generation for reinforcement learning\.InProceedings of the 1st Conference on Robot Learning,pp\. 482–495\.Cited by:[§6\.4](https://arxiv.org/html/2608.13622#S6.SS4.p1.1)\.
- Greensmithet al\.\(2004\)E\. Greensmith, P\. L\. Bartlett, and J\. BaxterVariance reduction techniques for gradient estimates in reinforcement learning\.Journal of Machine Learning Research5\(Nov\),pp\. 1471–1530\.Cited by:[§5\.2](https://arxiv.org/html/2608.13622#S5.SS2.p1.1),[§5\.4](https://arxiv.org/html/2608.13622#S5.SS4.p1.1)\.
- Hamilton \(2024\)S\. HamiltonDetecting mode collapse in language models via narration\.arXiv preprint arXiv:2402\.04477\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1)\.
- Jiaet al\.\(2024\)H\. Jia, T\. Bu, Y\. Wang, Z\. Zhang,et al\.Scaf\-grpo: scaffolded group relative policy optimization for enhancing llm reasoning\.arXiv preprint arXiv:2510\.19807\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p4.1),[§1](https://arxiv.org/html/2608.13622#S1.p3.1),[§2](https://arxiv.org/html/2608.13622#S2.p3.1),[§5\.2](https://arxiv.org/html/2608.13622#S5.SS2.p3.1)\.
- Jinet al\.\(2025\)R\. Jin, P\. Gao, Y\. Ren, Z\. Han, T\. Zhang, W\. Huang, W\. Liu, J\. Luan, and D\. XiongRevisiting entropy in reinforcement learning for large reasoning models\.arXiv preprint arXiv:2511\.05993\.Cited by:[§5\.3](https://arxiv.org/html/2608.13622#S5.SS3.p1.2)\.
- Liet al\.\(2024a\)T\. Liet al\.From crowdsourced data to high\-quality benchmarks: arena\-hard and beyond\.arXiv preprint arXiv:2405\.20239\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p2.1),[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px7.p1.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p2.1)\.
- Liet al\.\(2024b\)Z\. Liet al\.Can we trust ai agents? an experimental study towards trustworthy llm\-based multi\-agent systems for ai\-driven software engineering\.arXiv preprint arXiv:2411\.08881\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p2.1)\.
- Lianget al\.\(2025\)A\. Liang, J\. Berant, A\. Fisch, A\. Goyal, K\. Krishna, and J\. EisensteinPlantain: plan\-answer interleaved reasoning\.arXiv preprint arXiv:2512\.03176\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1),[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Liaoet al\.\(2025\)B\. Liao, Z\. Zhang, L\. Frison, and C\. MonzSelf\-hinting language models enhance reinforcement learning\.arXiv preprint arXiv:2602\.03143\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p4.1),[§1](https://arxiv.org/html/2608.13622#S1.p3.1),[§2](https://arxiv.org/html/2608.13622#S2.p3.1),[§5\.2](https://arxiv.org/html/2608.13622#S5.SS2.p3.1)\.
- Liao and Vaughan \(2023\)Q\. V\. Liao and J\. W\. VaughanAI transparency in the age of llms: a human\-centered research roadmap\.arXiv preprint arXiv:2306\.01941\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1)\.
- Linet al\.\(2023\)Z\. Lin, Z\. Wang, Y\. Tong, Y\. Wang, Y\. Guo, Y\. Wang, and J\. ShangToxicChat: unveiling hidden challenges of toxicity detection in real\-world user\-AI conversation\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 4694–4702\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.311),[Link](https://aclanthology.org/2023.findings-emnlp.311/)Cited by:[Limitations](https://arxiv.org/html/2608.13622#Sx1.p2.1)\.
- Liuet al\.\(2024\)F\. Liuet al\.ToolACE: winning the solutions of tool learning\.arXiv preprint arXiv:2409\.00920\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1),[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Liuet al\.\(2025\)Z\. Liuet al\.Understanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1),[§1](https://arxiv.org/html/2608.13622#S1.p1.1),[§2](https://arxiv.org/html/2608.13622#S2.p2.1),[§5\.2](https://arxiv.org/html/2608.13622#S5.SS2.p1.1)\.
- Matiisenet al\.\(2017\)T\. Matiisen, A\. Oliver, T\. Cohen, and J\. SchulmanTeacher\-student curriculum learning\.arXiv preprint arXiv:1707\.00183\.Cited by:[§6\.4](https://arxiv.org/html/2608.13622#S6.SS4.p1.1)\.
- Panet al\.\(2025\)L\. Pan, Y\. Tong, X\. Zhang, X\. Zhang, J\. Zhou, and Z\. ChuUnderstanding and mitigating overrefusal in LLMs from an unveiling perspective of safety decision boundary\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 21057–21075\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1065),[Link](https://aclanthology.org/2025.emnlp-main.1065/)Cited by:[Limitations](https://arxiv.org/html/2608.13622#Sx1.p2.1)\.
- Panet al\.\(2026\)L\. Pan, H\. Yang, H\. Li, Y\. Lu, Y\. Tong, Y\. Wang, S\. Wang, Z\. Chu, L\. Shen, Y\. Lu, and H\. WangOptimal transport for LLM reward modeling from noisy preference\.arXiv preprint arXiv:2605\.06036\.External Links:[Link](https://arxiv.org/abs/2605.06036)Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1)\.
- Patilet al\.\(2023\)S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. GonzalezGorilla: large language model connected with massive apis\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Pyatkinet al\.\(2025\)V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. HajishirziGeneralizing verifiable instruction following\.arXiv preprint arXiv:2507\.02833\.Note:Accepted to NeurIPS 2025, Datasets & BenchmarksCited by:[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px6.p1.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p2.1)\.
- Qianet al\.\(2025\)C\. Qian, E\. C\. Acikgoz, Q\. He, H\. Wang, X\. Chen, D\. Hakkani\-Tur, G\. Tur, and H\. JiToolRL: reward is all tool learning needs\.arXiv preprint arXiv:2504\.13958\.Cited by:[§C\.1](https://arxiv.org/html/2608.13622#A3.SS1.p1.1)\.
- Qinet al\.\(2023\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.arXiv preprint arXiv:2307\.16789\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1),[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Quet al\.\(2024\)C\. Qu, S\. Dai, X\. Wei, H\. Cai, S\. Wang, D\. Yin, J\. Xu, and J\. WenTool learning with large language models: a survey\.arXiv preprint arXiv:2405\.17935\.Cited by:[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1),[§4\.3](https://arxiv.org/html/2608.13622#S4.SS3.p1.1),[§4\.5\.1](https://arxiv.org/html/2608.13622#S4.SS5.SSS1.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.arXiv preprint arXiv:2305\.18290\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1),[§2](https://arxiv.org/html/2608.13622#S2.p2.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p2.1)\.
- Rosset al\.\(2025\)H\. Ross, A\. S\. Mahabaleshwarkar, and Y\. SuharaWhen2Call: when \(not\) to call tools\.arXiv preprint arXiv:2504\.18851\.Note:Accepted to NAACL 2025Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hamborg, L\. Zettlemoyer, N\. Cancedda, and M\. LewisToolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 68539–68551\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1),[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.InarXiv preprint arXiv:1707\.06347,Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1),[§1](https://arxiv.org/html/2608.13622#S1.p1.1),[§2](https://arxiv.org/html/2608.13622#S2.p2.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p3.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu,et al\.DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1),[§1](https://arxiv.org/html/2608.13622#S1.p1.1),[§2](https://arxiv.org/html/2608.13622#S2.p2.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p1.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p3.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.arXiv preprint arXiv:2303\.11366\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Taoet al\.\(2025\)L\. Tao, I\. Kulikov, S\. Saha, T\. Wang, J\. Xu, Y\. Li, J\. E\. Weston, and P\. YuHybrid reinforcement: when reward is sparse, it’s better to be dense\.arXiv preprint arXiv:2510\.07242\.Cited by:[§C\.1](https://arxiv.org/html/2608.13622#A3.SS1.p1.1)\.
- Team \(2025\)Q\. TeamQwen3 technical report\.arXiv preprint\.Cited by:[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px1.p1.1),[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px2.p1.1),[§4\.5\.1](https://arxiv.org/html/2608.13622#S4.SS5.SSS1.p1.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p1.1)\.
- TeichAI \(2025\)TeichAIClaude 4\.5 opus high reasoning 250x dataset\.Note:[https://huggingface\.co/datasets/TeichAI/claude\-4\.5\-opus\-high\-reasoning\-250x](https://huggingface.co/datasets/TeichAI/claude-4.5-opus-high-reasoning-250x)Distilled from Claude Opus 4\.5Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Tonget al\.\(2026a\)Y\. Tong, X\. Feng, L\. Xue, J\. Li, X\. Zhang, and J\. YangHow to train a real\-world silicon concierge? internalizing complex business workflow to only OneModel\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 1303–1316\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-industry.91),[Link](https://aclanthology.org/2026.acl-industry.91/)Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Tonget al\.\(2024a\)Y\. Tong, D\. Li, S\. Wang, Y\. Wang, F\. Teng, and J\. ShangCan LLMs learn from previous mistakes? investigating LLMs’ errors to boost for reasoning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3065–3080\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.169),[Link](https://aclanthology.org/2024.acl-long.169/)Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Tonget al\.\(2024b\)Y\. Tong, S\. Wang, D\. Li, Y\. Wang, S\. Han, Z\. Lin, C\. Huang, J\. Huang, and J\. ShangOptimizing language model’s reasoning abilities with weak supervision\.arXiv preprint arXiv:2405\.04086\.External Links:[Link](https://arxiv.org/abs/2405.04086)Cited by:[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Tonget al\.\(2023\)Y\. Tong, Y\. Wang, D\. Li, S\. Wang, Z\. Lin, S\. Han, and J\. ShangEliminating reasoning via inferring with planning: a new framework to guide LLMs’ non\-linear thinking\.arXiv preprint arXiv:2310\.12342\.External Links:[Link](https://arxiv.org/abs/2310.12342)Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Tonget al\.\(2026b\)Y\. Tong, Z\. Zhang, Z\. Liu, Tangzeyun, Z\. Peng, K\. Fu, M\. Song, H\. Zhang, J\. Zhang, H\. Zhu, and J\. LiReinforcement learning with missing context to mitigate reward hacking from training only on golden answers\.Note:OpenReview preprintExternal Links:[Link](https://openreview.net/forum?id=eOqnXPOl4S)Cited by:[§1](https://arxiv.org/html/2608.13622#S1.p2.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMuSiQue: multihop questions via single\-hop question composition\.InTransactions of the Association for Computational Linguistics,Vol\.10,pp\. 539–554\.Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Wanget al\.\(2023\)L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\. Lee, and E\. LimPlan\-and\-solve prompting: improving zero\-shot chain\-of\-thought reasoning by large language models\.arXiv preprint arXiv:2305\.04091\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, Y\. Tong, H\. Zhang, D\. Li, X\. Zhang, and T\. ChenBPO: towards balanced preference optimization between knowledge breadth and depth in alignment\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 8811–8826\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.443),[Link](https://aclanthology.org/2025.naacl-long.443/)Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Xieet al\.\(2024\)C\. Xie, Y\. Huang, C\. Zhang, D\. Yu, X\. Chen, B\. Y\. Lin, B\. Li, B\. Ghazi, and R\. KumarOn memorization of large language models in logical reasoning\.arXiv preprint arXiv:2410\.23123\.Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Xieet al\.\(2025\)R\. Xie, D\. Qiu, D\. Gopinath, D\. Lin, Y\. Sun, C\. Wang, S\. Potdar, and B\. DhingraInterleaved reasoning for large language models via reinforcement learning\.arXiv preprint arXiv:2505\.19640\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1),[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Xuet al\.\(2026\)Z\. Xu, Y\. Tong, X\. Zhang, J\. Zhou, and X\. WangUnderstanding conflicts in multi\-objective alignment through reward consistency\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 5450–5472\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.269),[Link](https://aclanthology.org/2026.findings-acl.269/)Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1)\.
- Yanet al\.\(2025\)J\. Yan, Y\. Li, Z\. Hu, Z\. Wang, G\. Cui, X\. Qu, Y\. Cheng, and Y\. ZhangLearning to reason under off\-policy guidance\.arXiv preprint arXiv:2504\.14945\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p4.1),[§1](https://arxiv.org/html/2608.13622#S1.p3.1),[§2](https://arxiv.org/html/2608.13622#S2.p3.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§I\.2](https://arxiv.org/html/2608.13622#A9.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.13622#S1.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1),[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p2.1)\.
- Yaoet al\.\(2023a\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. NarasimhanTree of thoughts: deliberate problem solving with large language models\.arXiv preprint arXiv:2305\.10601\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Yaoet al\.\(2023b\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. NarasimhanReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1),[§1](https://arxiv.org/html/2608.13622#S1.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1),[§2](https://arxiv.org/html/2608.13622#S2.p1.1)\.
- Yuet al\.\(2025\)Q\. Yuet al\.DAPO: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p3.1),[§1](https://arxiv.org/html/2608.13622#S1.p1.1),[§2](https://arxiv.org/html/2608.13622#S2.p2.1),[§6\.1](https://arxiv.org/html/2608.13622#S6.SS1.p3.1)\.
- Yuanet al\.\(2023\)L\. Yuan, Y\. Chen, X\. Wang, Y\. R\. Fung, H\. Peng, and H\. JiCRAFT: customizing llms by creating and retrieving from specialized toolsets\.arXiv preprint arXiv:2309\.17428\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.
- Yuan and Ying \(2022\)R\. Yuan and L\. YingFinite\-time analysis of stochastic policy gradient methods\.InProceedings of the 25th International Conference on Artificial Intelligence and Statistics,pp\. 1070–1096\.Cited by:[§5\.4](https://arxiv.org/html/2608.13622#S5.SS4.p1.1)\.
- Zhanet al\.\(2025\)R\. Zhan, Y\. Li, Z\. Wang, X\. Qu, D\. Liu, J\. Shao, D\. F\. Wong, and Y\. ChengExGRPO: learning to reason from experience\.arXiv preprint arXiv:2510\.02245\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p4.1),[§1](https://arxiv.org/html/2608.13622#S1.p3.1),[§2](https://arxiv.org/html/2608.13622#S2.p3.1)\.
- Zhanget al\.\(2024a\)X\. Zhanget al\.Controllable text generation for large language models: a survey\.arXiv preprint arXiv:2408\.12599\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1)\.
- Zhanget al\.\(2024b\)Y\. Zhanget al\.A survey on human\-ai collaboration with large foundation models\.arXiv preprint arXiv:2403\.04931\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p2.1),[§1](https://arxiv.org/html/2608.13622#S1.p4.1)\.
- Zhanget al\.\(2024c\)Y\. Zhanget al\.XLAM: a large action model dataset for autonomous agents\.arXiv preprint arXiv:2406\.06152\.Cited by:[§4\.2](https://arxiv.org/html/2608.13622#S4.SS2.p2.1)\.
- Zhouet al\.\(2023\)A\. Zhou, K\. Yan, M\. Shlapentokh\-Rothman, H\. Wang, and Y\. WangLanguage agent tree search unifies reasoning, acting, and planning in language models\.arXiv preprint arXiv:2310\.04406\.Cited by:[Appendix B](https://arxiv.org/html/2608.13622#A2.p1.1)\.

## Appendix AProofs and Theoretical Analysis

This appendix provides proofs for the stylized variance claims in Section[5](https://arxiv.org/html/2608.13622#S5)\. We distinguish between the*target strategy*s⋆s^\{\\star\}, which is assigned to a training prompt before ARC rollout generation, and the*realized strategy*si=f⁡\(yi\)s\_\{i\}=f\(y\_\{i\}\)exhibited by a sampled responseyiy\_\{i\}\.

Pry∼πθ\(⋅∣x\)\[f\(y\)=s\]\.\\Pr\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\[f\(y\)=s\]\.\(2\)
Under ARC, a target strategys⋆s^\{\\star\}is assigned before rollout generation and appended to the prompt\. Responses are then sampled as

yi∼i\.i\.d\.πθ\(⋅∣x,s⋆\)\.y\_\{i\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\pi\_\{\\theta\}\(\\cdot\\mid x,s^\{\\star\}\)\.\(3\)The realized strategy remainssi=f⁡\(yi\)s\_\{i\}=f\(y\_\{i\}\)and may differ froms⋆s^\{\\star\}when strategy adherence is imperfect\.

### A\.1Proof of Theorem[5\.1](https://arxiv.org/html/2608.13622#S5.Thmarctheorem1)\(Variance Amplification\)

###### Proof\.

Decomposeri=μsi\+ϵir\_\{i\}=\\mu\_\{s\_\{i\}\}\+\\epsilon\_\{i\}whereμsi\\mu\_\{s\_\{i\}\}is the mean reward for realized strategysis\_\{i\}\(capturing RM bias\) andϵi\\epsilon\_\{i\}is i\.i\.d\. zero\-mean noise with varianceσintra2\\sigma^\{2\}\_\{\\text\{intra\}\}, independent of the strategy\-dependent mean term\. Sinceyi∼i\.i\.d\.πθ\(⋅∣x\)y\_\{i\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\pi\_\{\\theta\}\(\\cdot\\mid x\)andsi=f⁡\(yi\)s\_\{i\}=f\(y\_\{i\}\), the realized strategiess1,…,sNs\_\{1\},\\ldots,s\_\{N\}are i\.i\.d\. under the induced distributionPθ\(⋅∣x\)P\_\{\\theta\}\(\\cdot\\mid x\), soμs1,…,μsN\\mu\_\{s\_\{1\}\},\\ldots,\\mu\_\{s\_\{N\}\}are i\.i\.d\. with varianceσinter2\\sigma^\{2\}\_\{\\text\{inter\}\}\.

The advantage isA^i=ri−r¯=\(μsi−μ¯\)\+\(ϵi−ϵ¯\)\\hat\{A\}\_\{i\}=r\_\{i\}\-\\bar\{r\}=\(\\mu\_\{s\_\{i\}\}\-\\bar\{\\mu\}\)\+\(\\epsilon\_\{i\}\-\\bar\{\\epsilon\}\), whereμ¯=1N​∑jμsj\\bar\{\\mu\}=\\frac\{1\}\{N\}\\sum\_\{j\}\\mu\_\{s\_\{j\}\}andϵ¯=1N​∑jϵj\\bar\{\\epsilon\}=\\frac\{1\}\{N\}\\sum\_\{j\}\\epsilon\_\{j\}\.

Since strategy and noise are independent:

Var⁡\[A^i\]=Var⁡\[μsi−μ¯\]\+Var⁡\[ϵi−ϵ¯\]\\small\\mathrm\{Var\}\[\\hat\{A\}\_\{i\}\]=\\mathrm\{Var\}\[\\mu\_\{s\_\{i\}\}\-\\bar\{\\mu\}\]\+\\mathrm\{Var\}\[\\epsilon\_\{i\}\-\\bar\{\\epsilon\}\]\(4\)
For any i\.i\.d\. sequenceZ1,…,ZNZ\_\{1\},\\ldots,Z\_\{N\}with varianceσ2\\sigma^\{2\}:

Var⁡\[Zi−Z¯\]=\(1−1N\)2​σ2\+N−1N2​σ2=σ2​\(1−1N\)\\small\\mathrm\{Var\}\[Z\_\{i\}\-\\bar\{Z\}\]=\\left\(1\-\\frac\{1\}\{N\}\\right\)^\{2\}\\sigma^\{2\}\+\\frac\{N\-1\}\{N^\{2\}\}\\sigma^\{2\}=\\sigma^\{2\}\\left\(1\-\\frac\{1\}\{N\}\\right\)\(5\)
Applying this to both terms yieldsVar⁡\[A^i\]=\(σinter2\+σintra2\)​\(1−1/N\)\\mathrm\{Var\}\[\\hat\{A\}\_\{i\}\]=\(\\sigma^\{2\}\_\{\\text\{inter\}\}\+\\sigma^\{2\}\_\{\\text\{intra\}\}\)\(1\-1/N\)\. ∎

### A\.2Proof of Theorem[5\.2](https://arxiv.org/html/2608.13622#S5.Thmarctheorem2)\(Variance Reduction via Conditioning\)

###### Proof\.

ARC conditions generation on the target strategys⋆s^\{\\star\}, and under perfect compliance,f⁡\(yi\)=s⋆f\(y\_\{i\}\)=s^\{\\star\}, soμsi=μs∗\\mu\_\{s\_\{i\}\}=\\mu\_\{s^\{\*\}\}for allii\. The advantage simplifies to:

A^i=ϵi−ϵ¯\\small\\hat\{A\}\_\{i\}=\\epsilon\_\{i\}\-\\bar\{\\epsilon\}\(6\)Sinceϵi\\epsilon\_\{i\}are i\.i\.d\. with varianceσintra2\\sigma^\{2\}\_\{\\text\{intra\}\}:

Var⁡\[A^i\|s∗\]=σintra2​\(1−1N\)\\small\\mathrm\{Var\}\[\\hat\{A\}\_\{i\}\|s^\{\*\}\]=\\sigma^\{2\}\_\{\\text\{intra\}\}\\left\(1\-\\frac\{1\}\{N\}\\right\)\(7\)The between\-strategy mean\-shift term is absent from this centered\-advantage variance decomposition\.

Var⁡\[A^i\|s∗\]=\\displaystyle\\mathrm\{Var\}\[\\hat\{A\}\_\{i\}\|s^\{\*\}\]=\{\}\(σintra2​\(x,s⋆\)CLOSE\\displaystyle\\Bigl\(\\sigma^\{2\}\_\{\\text\{intra\}\}\(x,s^\{\\star\}\)\(8\)\+VarS∼Pθ\(⋅∣x,s⋆\)\[μS\(x\)\]\)\(1−1N\)\.\\displaystyle\+\\operatorname\{Var\}\_\{S\\sim P\\theta\(\\cdot\\mid x,s^\{\\star\}\)\}\[\\mu\_\{S\}\(x\)\]\\Bigr\)\\left\(1\-\\frac\{1\}\{N\}\\right\)\.With imperfect compliance, the ARC variance additionally contains the residual between\-strategy term, so the predicted variance reduction is correspondingly attenuated\. ∎

### A\.3Derivation for Theorem[5\.3](https://arxiv.org/html/2608.13622#S5.Thmarctheorem3)\(Idealized Sample\-Complexity Comparison\)

###### Theorem A\.1\(Gradient Variance Bound\)\.

LetgARCg\_\{\\text\{ARC\}\}andgstdg\_\{\\text\{std\}\}denote policy gradients under strategy\-conditioned and standard sampling with equal group sizeNN\. For analytic tractability, assume: \(i\) the relevant advantage\-score mixed moments approximately factorize conditional onxx; \(ii\) the score\-function second momentsF=𝔼⁡\[‖∇i‖2\]F=\\mathbb\{E\}\[\\\|\\nabla\_\{i\}\\\|^\{2\}\]andF′=𝔼⁡\[⟨∇i,∇j⟩\]F^\{\\prime\}=\\mathbb\{E\}\[\\langle\\nabla\_\{i\},\\nabla\_\{j\}\\rangle\]\(i≠ji\\neq j\) are comparable up to constants across the two sampling schemes; and \(iii\) ARC satisfies the perfect\-compliance idealization of Theorem[5\.2](https://arxiv.org/html/2608.13622#S5.Thmarctheorem2)\. Then:

𝔼⁡\[‖gARC‖2\]𝔼⁡\[‖gstd‖2\]≈σintra2σintra2\+σinter2\\small\\frac\{\\mathbb\{E\}\[\\\|g\_\{\\text\{ARC\}\}\\\|^\{2\}\]\}\{\\mathbb\{E\}\[\\\|g\_\{\\text\{std\}\}\\\|^\{2\}\]\}\\approx\\frac\{\\sigma^\{2\}\_\{\\text\{intra\}\}\}\{\\sigma^\{2\}\_\{\\text\{intra\}\}\+\\sigma^\{2\}\_\{\\text\{inter\}\}\}\(9\)

###### Proof\.

Writeg=1N​∑i=1NA^i​∇ig=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\hat\{A\}\_\{i\}\\nabla\_\{i\}where∇i=∇θ​log​πθ​\(yi\|x\)\\nabla\_\{i\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\|x\)\. Under assumption \(i\):

𝔼⁡\[‖g‖2\]=1N2​\[N​𝔼​\[A^i2\]​F\+N⁡\(N−1\)​𝔼​\[A^i​A^j\]​F′\]\\small\\mathbb\{E\}\[\\\|g\\\|^\{2\}\]=\\frac\{1\}\{N^\{2\}\}\\left\[N\\mathbb\{E\}\[\\hat\{A\}\_\{i\}^\{2\}\]\\,F\+N\(N\-1\)\\,\\mathbb\{E\}\[\\hat\{A\}\_\{i\}\\hat\{A\}\_\{j\}\]\\,F^\{\\prime\}\\right\]\(10\)
For i\.i\.d\. rewardsrir\_\{i\}with total varianceσr2\\sigma^\{2\}\_\{r\}, the centered advantages satisfy:

𝔼⁡\[A^i2\]\\displaystyle\\mathbb\{E\}\[\\hat\{A\}\_\{i\}^\{2\}\]=σr2​\(1−1N\)\\displaystyle=\\sigma^\{2\}\_\{r\}\\left\(1\-\\frac\{1\}\{N\}\\right\)\(11\)𝔼⁡\[A^i​A^j\]\\displaystyle\\mathbb\{E\}\[\\hat\{A\}\_\{i\}\\hat\{A\}\_\{j\}\]=−σr2N\(i≠j\)\\displaystyle=\-\\frac\{\\sigma^\{2\}\_\{r\}\}\{N\}\\quad\(i\\neq j\)\(12\)The latter follows fromCov\[ri−r¯,rj−r¯\]=−σr2/N\\mathrm\{Cov\}\[r\_\{i\}\-\\bar\{r\},\\,r\_\{j\}\-\\bar\{r\}\]=\-\\sigma^\{2\}\_\{r\}/N\.

Substituting:

𝔼⁡\[‖g‖2\]=σr2​\(N−1\)N2​\(F−F′\)\\mathbb\{E\}\[\\\|g\\\|^\{2\}\]=\\frac\{\\sigma^\{2\}\_\{r\}\(N\-1\)\}\{N^\{2\}\}\(F\-F^\{\\prime\}\)\(13\)
Sinceσr2=σintra2\+σinter2\\sigma^\{2\}\_\{r\}=\\sigma^\{2\}\_\{\\text\{intra\}\}\+\\sigma^\{2\}\_\{\\text\{inter\}\}under standard sampling andσr2=σintra2\\sigma^\{2\}\_\{r\}=\\sigma^\{2\}\_\{\\text\{intra\}\}under ARC \(Theorems[5\.1](https://arxiv.org/html/2608.13622#S5.Thmarctheorem1)–[5\.2](https://arxiv.org/html/2608.13622#S5.Thmarctheorem2)\), assumption \(ii\) yields the stated approximation\. ∎

###### Derivation for Theorem[5\.3](https://arxiv.org/html/2608.13622#S5.Thmarctheorem3)\.

This argument should be read as a stylized variance\-to\-sample\-efficiency translation rather than a full convergence proof\. Under a sub\-Gaussian gradient\-estimation assumption, the number of gradient samples required to achieve‖g^−∇J​\(θ\)‖≤ε\\\|\\hat\{g\}\-\\nabla J\(\\theta\)\\\|\\leq\\varepsilonwith probability≥1−δ\\geq 1\-\\deltascales asO⁡\(σg2​log⁡\(1/δ\)/ε2\)O\(\\sigma^\{2\}\_\{g\}\\log\(1/\\delta\)/\\varepsilon^\{2\}\), whereσg2\\sigma^\{2\}\_\{g\}is the per\-sample gradient variance\. From Theorem[A\.1](https://arxiv.org/html/2608.13622#A1.Thmarctheorem1),σg,ARC2/σg,std2≈σintra2/\(σintra2\+σinter2\)\\sigma^\{2\}\_\{g,\\text\{ARC\}\}/\\sigma^\{2\}\_\{g,\\text\{std\}\}\\approx\\sigma^\{2\}\_\{\\text\{intra\}\}/\(\\sigma^\{2\}\_\{\\text\{intra\}\}\+\\sigma^\{2\}\_\{\\text\{inter\}\}\), and the same approximation carries over to this idealized sample\-efficiency comparison\. ∎

### A\.4Effective Variance Analysis for Curriculum Learning

For the curriculum learning schedule studied in Section[6\.4](https://arxiv.org/html/2608.13622#S6.SS4), the expected advantage variance across the batch at training stepttis a mixture of conditioned and unconditioned groups:

Vareff​\(t\)=\\displaystyle\\mathrm\{Var\}\_\{\\text\{eff\}\}\(t\)=\{\}\(1−pd​\(t\)\)​σintra2​\(1−1N\)\\displaystyle\\bigl\(1\-p\_\{d\}\(t\)\\bigr\)\\,\\sigma^\{2\}\_\{\\text\{intra\}\}\\\!\\left\(1\-\\tfrac\{1\}\{N\}\\right\)\(14\)\+pd​\(t\)​\(σintra2\+σinter2\)​\(1−1N\)\.\\displaystyle\}\{\\displaystyle\+p\_\{d\}\(t\)\\,\\bigl\(\\sigma^\{2\}\_\{\\text\{intra\}\}\+\\sigma^\{2\}\_\{\\text\{inter\}\}\\bigr\)\\\!\\left\(1\-\\tfrac\{1\}\{N\}\\right\)\.
Since the second term exceeds the first byσinter2​\(1−1/N\)\\sigma^\{2\}\_\{\\text\{inter\}\}\(1\-1/N\),Vareff​\(t\)\\mathrm\{Var\}\_\{\\text\{eff\}\}\(t\)increases monotonically from≈σintra2​\(1−1/N\)\\approx\\sigma^\{2\}\_\{\\text\{intra\}\}\(1\-1/N\)to≈\(σintra2\+σinter2\)​\(1−1/N\)\\approx\(\\sigma^\{2\}\_\{\\text\{intra\}\}\+\\sigma^\{2\}\_\{\\text\{inter\}\}\)\(1\-1/N\)\. This monotonic increase gives the curriculum learning schedule a variance\-based progression: the model trains under progressively noisier optimization landscapes as it becomes more capable\.

### A\.5Information\-Theoretic Perspective

Assuming strategyssis determined by the responseyy\(i\.e\.,s=f⁡\(y\)s=f\(y\)for some deterministic function\), the chain rule of mutual information gives:

I\(r;y\|x\)=I\(r;y\|x,s\)\+I\(r;s\|x\)\\small I\(r;y\|x\)=I\(r;y\|x,s\)\+I\(r;s\|x\)\(15\)
The termI⁡\(r;s\|x\)I\(r;s\|x\)captures strategy\-linked reward dependence that is orthogonal to within\-strategy quality assessment\.

Under the idealized perfect\-compliance setting, conditioning on the target strategy removes between\-strategy variation from within\-group comparisons, emphasizing the within\-strategy reward signalI\(r;y∣x,s\)I\(r;y\\mid x,s\)\.

## Appendix BExtended Related Work

Tool\-Augmented and Interactive Agents\.Tool\-augmented language models have been studied through self\-supervised tool learning[40](https://arxiv.org/html/2608.13622#bib.bib1), large\-scale API integration[34](https://arxiv.org/html/2608.13622#bib.bib2),[31](https://arxiv.org/html/2608.13622#bib.bib3), synthetic function\-calling corpora[10](https://arxiv.org/html/2608.13622#bib.bib4),[26](https://arxiv.org/html/2608.13622#bib.bib11), and increasingly sophisticated agent architectures[11](https://arxiv.org/html/2608.13622#bib.bib28),[64](https://arxiv.org/html/2608.13622#bib.bib29),[7](https://arxiv.org/html/2608.13622#bib.bib67),[47](https://arxiv.org/html/2608.13622#bib.bib69)\. A parallel line of work studies interaction structure, including ReAct\-style reasoning\-action interleaving[62](https://arxiv.org/html/2608.13622#bib.bib5), structured planning[55](https://arxiv.org/html/2608.13622#bib.bib6),[61](https://arxiv.org/html/2608.13622#bib.bib31),[53](https://arxiv.org/html/2608.13622#bib.bib34),[50](https://arxiv.org/html/2608.13622#bib.bib63), and reflective or search\-based agents[43](https://arxiv.org/html/2608.13622#bib.bib32),[70](https://arxiv.org/html/2608.13622#bib.bib33),[48](https://arxiv.org/html/2608.13622#bib.bib61)\. Our interest is not only in adding tools or planning steps, but in exposing interaction as a first\-class, user\-visible channel while execution is still ongoing\.

Interleaved Reasoning and User Experience\.Recent interleaving methods improve perceived responsiveness by alternating internal reasoning with partial textual output[22](https://arxiv.org/html/2608.13622#bib.bib48),[57](https://arxiv.org/html/2608.13622#bib.bib49)\. These works mainly address the sequencing of thought and answer tokens\. INTER3targets a different bottleneck: long\-running external actions\. When a tool call itself is the latency source, making the communication channel independent from the execution channel becomes the key design move\. This connects to broader human\-AI interaction work on transparency, feedback, and controllability[68](https://arxiv.org/html/2608.13622#bib.bib44),[24](https://arxiv.org/html/2608.13622#bib.bib45),[21](https://arxiv.org/html/2608.13622#bib.bib46),[67](https://arxiv.org/html/2608.13622#bib.bib47),[20](https://arxiv.org/html/2608.13622#bib.bib10), but in an agent context where the cost of opaque waiting is especially high\.

RLHF Estimators and Multi\-Behavior Collapse\.Our RL analysis sits within the broader literature on policy\-gradient and RLHF optimization[41](https://arxiv.org/html/2608.13622#bib.bib27),[1](https://arxiv.org/html/2608.13622#bib.bib37),[37](https://arxiv.org/html/2608.13622#bib.bib35),[13](https://arxiv.org/html/2608.13622#bib.bib36),[54](https://arxiv.org/html/2608.13622#bib.bib62),[58](https://arxiv.org/html/2608.13622#bib.bib66),[30](https://arxiv.org/html/2608.13622#bib.bib68), especially group\-relative estimators such as GRPO[42](https://arxiv.org/html/2608.13622#bib.bib26)and refinements for entropy preservation or bias correction[63](https://arxiv.org/html/2608.13622#bib.bib40),[27](https://arxiv.org/html/2608.13622#bib.bib41)\. These methods have proven effective for reasoning\-heavy tasks[9](https://arxiv.org/html/2608.13622#bib.bib20), but open\-ended multi\-behavior settings raise a distinct collapse risk[17](https://arxiv.org/html/2608.13622#bib.bib59): the policy can converge to whichever behavior the reward favors\. Our contribution is to isolate one concrete statistical mechanism for that collapse—cross\-strategy contamination of relative advantages—and study a conditioning\-based remedy\.

External Guidance in RL\.Methods such as SAGE[23](https://arxiv.org/html/2608.13622#bib.bib22), Scaf\-GRPO[18](https://arxiv.org/html/2608.13622#bib.bib23), LUFFY[59](https://arxiv.org/html/2608.13622#bib.bib24), and ExGRPO[66](https://arxiv.org/html/2608.13622#bib.bib25)show that auxiliary guidance can materially improve RL training by mitigating sparse rewards, structuring exploration, incorporating demonstrations, or reusing successful trajectories\. ARC is adjacent in form but different in purpose\. The strategy instruction is not introduced to reveal the answer, densify reward, or bias the policy toward a target trajectory\. It is introduced to constrain which rollouts are compared to one another, so that relative advantage estimation is performed inside a behaviorally coherent comparison class\.

## Appendix CReward Details

### C\.1Final Reward Used in Main Experiments

The specific reward used in our full INTER3instantiation is not ARC’s core mechanism; it is a setting\-specific design choice for stabilizing execution\-grounded training\. Following recent reward\-design work that combines verifiable structural signals with denser model\-based feedback[33](https://arxiv.org/html/2608.13622#bib.bib56),[44](https://arxiv.org/html/2608.13622#bib.bib57), we use a compact three\-term reward\. Theformat rewardRfmt∈\{0,1\}R\_\{\\mathrm\{fmt\}\}\\in\\\{0,1\\\}checks for at least one well\-formed<tool\_call\>or<answer\>block with properly matched, non\-nested tags\. Thetool rewardRtool∈\{−1,0,1\}R\_\{\\mathrm\{tool\}\}\\in\\\{\-1,0,1\\\}evaluates exact tool\-call correctness: the tool identifier, argument key set, and argument values must all match the reference forRtool=1R\_\{\\mathrm\{tool\}\}=1\. Theanswer rewardRans∈\{0,0\.50,1\}R\_\{\\mathrm\{ans\}\}\\in\\\{0,0\.50,1\\\}is an LLM\-judge score on the extracted<answer\>content\. To couple semantics to correct execution, we setRans=Eval⁡\(y,y∗\)R\_\{\\mathrm\{ans\}\}=\\mathrm\{Eval\}\(y,y^\{\*\}\)only whenRtool=1R\_\{\\mathrm\{tool\}\}=1, andRans=0R\_\{\\mathrm\{ans\}\}=0otherwise\. The final reward is simplyR=Rfmt\+Rtool\+RansR=R\_\{\\mathrm\{fmt\}\}\+R\_\{\\mathrm\{tool\}\}\+R\_\{\\mathrm\{ans\}\}\. If both prediction and reference contain no answer spans, we setRans=1R\_\{\\mathrm\{ans\}\}=1to reflect structural agreement; if evaluation fails, we return a neutral fallback score\.

### C\.2Detailed Reward Ablation

A central challenge in training INTER3agents is designing rewards that evaluate multi\-channel outputs—internal reasoning, tool calls, and user\-facing<answer\>spans—without baking in strategy\-dependent bias\. We study two reward\-side choices: the reward construction itself, and the reward model used to score semantic answer quality\.

Table 7:Reward\-construction ablation\. All variants include the format reward\. Exact tool matching improves tool use, and the final gated design gives the strongest tool\-centric operating point\. Unless explicitly varied, all main RL baselines and ARC variants in the paper use the final row\.The reward constructions in Table[7](https://arxiv.org/html/2608.13622#A3.T7)differ along two axes: \(i\) tool\-execution strictness, and \(ii\) coupling between execution and semantic reward\. All variants include a sharedformat rewardenforcing structural validity\. Moving from partial to exact tool matching strengthens the execution\-grounded signal, and the final gated design amplifies this effect, increasingτ\\tauAvg\. by about 42% over the weakest variant\. This comes with reduced performance on broader reasoning\-style evaluations, while leaving overall average performance largely unchanged\. The results reflect our objective: to prioritize rewards that favor answers grounded in correct tool execution rather than signals agnostic to whether the answer is causally supported by the executed actions\.

Table 8:Reward\-model ablation for the same Qwen3\-8B policy under the final reward construction used throughout the main experiments\. Varying only the external judge shows that the 235B judge provides the strongest tool\-use operating point\.Table[8](https://arxiv.org/html/2608.13622#A3.T8)fixes the policy at Qwen3\-8B and varies only the external reward judge\. Under this controlled comparison, the 235B judge gives the strongestτ/τ2\\tau/\\tau^\{2\}performance and the best overall balance for execution\-grounded training, while smaller judges recover only isolated gains on auxiliary metrics\. These findings led us to select the 235B judge for our main experiments, as it provides the most consistent signal for tool\-execution verification\.

Tool matching\.Partial toolassigns credit to approximately correct tool calls, providing dense but noisy supervision\. Replacing this withexact tool matchingyields stricter credit assignment, improving reliability of the training signal\.

Negative penalties\.The final variant introduces anegative penaltyfor incorrect tool execution, explicitly discouraging spurious or malformed calls and sharpening optimization toward valid trajectories\.

Semantic reward coupling\.All variants use an LLM\-basedanswer judgefor user\-visible responses\. In the first two variants, this reward is applied unconditionally, allowing fluent but unsupported answers to receive credit\. The final variant applies agated answer judge, where semantic reward is issued only when exact tool execution is correct, coupling answer quality with execution validity\.

## Appendix DEntropy Analysis

To separate ARC’s core mechanism from setting\-specific stabilizers, Table[9](https://arxiv.org/html/2608.13622#A4.T9)fixes the reward family and varies only two ingredients: strategy\-conditioned grouping and entropy regularization\. The first block asks which component drives the main gain over GRPO; the second block studies entropy sensitivity once grouping is enabled\.

Table 9:Ablation study on ARC: Training settings and entropy hyperparameters\. Strategy\-conditioned grouping provides the largest single gain over base GRPO\. While entropy regularization alone slightly decreases performance, combining it with strategy\-conditioned grouping further improves the overall operating point\. The best entropy value \(0\.001\) yields the highest average score\.Table[9](https://arxiv.org/html/2608.13622#A4.T9)supports the paper’s main causal story\. Holding the reward design fixed, strategy\-conditioned grouping is the primary source of improvement over base GRPO, whereas entropy regularization alone does not explain the gain and can even weaken performance\. Entropy becomes useful only after the comparison class has been cleaned up by grouping, where it acts as a stabilizer for multi\-channel generation rather than as the main mechanism\. The entropy sweep further shows that this effect is sensitive to scale: too much entropy leads to reward collapse, while a moderate value of0\.0010\.001gives the best overall operating point\. Taken together, these results suggest that ARC helps mainly by changing*how*rollouts are compared, with entropy regularization serving as a secondary component that improves the stability of that mechanism in the full INTER3recipe\.

## Appendix EAnnotation Guidelines for INTER3\-86K

This section provides the detailed annotation guidelines used by both models and human annotators for strategy assignment\.

Progress Update: Proactively provide progress updates to the user when simultaneously executing a tool call or multi\-step tasks\. Notify the user of the current progress after completing each tool call/sub\-task\. Let the user know the task is in progress to reduce waiting anxiety\.

Direct Answer: Answer directly without invoking any tools if the information is already known or contextually available\. Be concise and clear; avoid verbosity\.

Clarify First: If the user’s request lacks sufficient information, ask for clarification first\. Do not guess the user’s intent; ask directly\. Execute tool calls only after the user confirms\. Suitable for vague, ambiguous, or incomplete requests\.

Alignment Check: Restate the user’s requirements first to confirm your understanding is correct\. Execute only after the user confirms\. Avoid making mistakes due to misunderstanding\. Suitable for easily misunderstood, important, or irreversible operations\.

## Appendix FMechanism\-Level Analysis and Label\-Noise Robustness

To complement the idealized theoretical analysis, we perform rollout\-level analyses of empirical reward unfairness, realized inter\- and intra\-strategy reward variance, strategy adherence, and sensitivity to strategy\-label noise\.

##### Realized inter\- and intra\-strategy reward variance\.

On actual rollouts, we estimate the realized between\-strategy and within\-strategy reward variances\. As shown in Table[10](https://arxiv.org/html/2608.13622#A6.T10), the estimated inter\-/intra\-strategy variance ratio decreases from 0\.417 under GRPO to 0\.074 under ARC, corresponding to an approximately 82% reduction\. This is consistent with the proposed variance\-reduction mechanism\. However, the confidence intervals are wide because relatively few prompt groups contain multiple realized strategies, so we interpret this result as mechanism\-level evidence rather than definitive empirical validation of the stylized theoretical analysis\.

Table 10:Estimated realized inter\- and intra\-strategy reward variance on rollout data\. Ratio denotesσ^inter2/σ^intra2\\hat\{\\sigma\}^\{2\}\_\{\\mathrm\{inter\}\}/\\hat\{\\sigma\}^\{2\}\_\{\\mathrm\{intra\}\}, with 95% confidence intervals\.
##### Sensitivity to strategy\-label noise\.

We additionally test the sensitivity of ARC to imperfect strategy labels by corrupting 50% of the strategy instructions during training\. For each corrupted example, the original strategy instruction is replaced with a uniformly sampled alternative strategy\. This experiment uses one training seed, and the reported standard deviations are computed over three independent evaluations\.

As shown in Table[11](https://arxiv.org/html/2608.13622#A6.T11), ARC remains above the GRPO baseline under substantial label corruption, with a tool\-use average of 35\.01 compared with 33\.35 for GRPO\. However, performance is substantially lower than clean ARC at 41\.73, indicating that ARC benefits from accurate strategy assignments and degrades under noisy training labels\.

Table 11:Sensitivity of ARC to strategy\-label corruption\. The corrupted setting replaces 50% of strategy instructions with a uniformly sampled alternative strategy during training\. Results are based on a single training run \(seed 1\); values after±\\pmdenote standard deviations over three independent evaluation runs\.

## Appendix GInteraction Strategy Examples

### G\.1Progress Update Strategy

\[user\]

HelpmeanalyzeAcmeCorp’sfinancialstatus\.

\[assistant\]

Theuserwantsafullfinancialanalysis\.I’llstartbyfetchingtheincomestatement\.

<tool\_call\>

\{"name":"get\_income\_statement","arguments":\{"company":"AcmeCorp"\}\}

</tool\_call\>

<answer\>I’mretrievingAcmeCorp’sfinancialstatements\-step1of3\.\.\.</answer\>

\[tool\]

get\_income\_statement\-\>\{"revenue":"$5\.2B","net\_income":"$430M"\}

\[assistant\]

Incomestatementretrieved\.Nowfetchingthebalancesheetforratioanalysis\.

<tool\_call\>

\{"name":"get\_balance\_sheet","arguments":\{"company":"AcmeCorp"\}\}

</tool\_call\>

<answer\>Incomestatementretrieved\.Nowanalyzingthebalancesheet\-step2of3\.\.\.</answer\>

### G\.2Direct Answer Strategy

\[user\]

WhatisthecapitalofChina?

\[assistant\]

<answer\>ThecapitalofChinaisBeijing\.</answer\>

### G\.3Clarify First Strategy

\[user\]

Bookameetingroomforme\.

\[assistant\]

Requestlacksrequiredinformation:timeandcapacity\.

<answer\>I’dbehappytobookameetingroom\.Couldyoutellme:

\(1\)Whatdateandtime?

\(2\)Howmanypeoplewillattend?</answer\>

\[user\]

Tomorrowat10AM,8people\.

\[assistant\]

Detailsconfirmed\.Proceedingwithbooking\.

<tool\_call\>\{"name":"book\_meeting\_room","arguments":\{"date":"tomorrow","time":"10:00","capacity":8\}\}</tool\_call\>

<answer\>Proceedingwithbookingaroomfortomorrowat10AMsuitablefor8people\.\.\.</answer\>

\[tool\]

book\_meeting\_room\-\>\{"status":"confirmed","room":"Room3B"\}

\[assistant\]

<answer\>Booked\!Room3Bfortomorrowat10:00AM,capacity8\.</answer\>

### G\.4Alignment Check Strategy

\[user\]

Sendlastquarter’ssalesdatatoManagerWang\.

\[assistant\]

Thisisasendoperation\-\-\-Ishouldconfirmtheexactscopeandrecipientbeforeexecuting\.

<answer\>Justtoconfirm:you’dlikemetosendtheQ12024\(January\-March\)salesdatareporttoManagerWangatwang@company\.com\.Isthatcorrect?</answer\>

\[user\]

Yes,that’sright\.

\[assistant\]

Intentconfirmed\.Executingsendoperation\.

<tool\_call\>\{"name":"send\_email","arguments":\{"to":"wang@company\.com","subject":"Q12024SalesData","attachment":"q1\_sales\.pdf"\}\}</tool\_call\>

\[tool\]

send\_email\-\>\{"status":"sent"\}

\[assistant\]

<answer\>Sent\!Q12024salesreportdeliveredtoManagerWang\.</answer\>

## Appendix HSystem Prompt

The system prompt used for training and evaluation:

System Prompt Configuration### System Role You are an advanced AI assistant capable of streamingly interleaved interactions, reasoning, and tool executions\. Do not wait for the entire process to finish\. You can provide partial updates in<answer\>while continuing to reason and call tools\. ### Output Requirements Your output must be structured into functional segments\. Texts that are NOT wrapped in<answer\>tags are treated as internal reasoning and are hidden from the user\. Ensure all XML\-style tags \(<tool\_call\>,<answer\>\) are strictly paired and properly closed\. ### Tag Definitions •Internal Reasoning:\(No tag required\) Use plain text outside of any tags for logical reasoning, task decomposition, or analyzing tool outputs\. This is hidden from the user\.•<answer\>:\(User Visible\) This is the ONLY content displayed to the user\. Use this for status updates, partial answers, or the final conclusion\. If your response relies on information from the tool outputs, do not expose the raw tool output verbatim\. Instead, integrate and summarize the relevant facts naturally in your own words\.•<tool\_call\>:\(Internal Only\) Use this to call external functions using the provided tools, if any\.

## Appendix IEvaluation Setup and Configuration

This appendix describes the standardized evaluation configuration used across all benchmarks reported in Table[3](https://arxiv.org/html/2608.13622#S6.T3)\.

### I\.1Inference Configuration

All evaluations were conducted using the following controlled parameters:

##### Concurrency\.

TTFT measurements are collected with concurrency level set to 16 parallel requests on the same GPU \(H200\)\. This concurrency level is maintained consistently across allτ\\tau\-bench evaluations to ensure fair latency comparisons\.

##### Interleaved Mode\.

When interleaved mode is enabled, we use the same structured interface as in training:<answer\>for user\-visible content and<tool\_call\>for tool calls\. This enables fine\-grained measurement of time\-to\-first\-token \(TTFT\) as the model can provide partial answers while continuing reasoning and tool execution\.

### I\.2Benchmark Configurations

##### τ\\tau\-Bench\.

Following the officialτ\\tau\-Bench evaluation protocol[60](https://arxiv.org/html/2608.13622#bib.bib8), we evaluate tool\-calling ability across theretailandairlinedomains\. The user simulator is powered by the Qwen3\-235B\-A22B\-Instruct model[45](https://arxiv.org/html/2608.13622#bib.bib21)\. We report the average reward score \(success rate\) and TTFT\. Each evaluation run is repeated 3 times and we report the mean score\.

##### τ2\\tau^\{2\}\-Bench\.

Following the officialτ2\\tau^\{2\}\-Bench evaluation protocol[4](https://arxiv.org/html/2608.13622#bib.bib9), we evaluate on theretail,airline, andtelecomdomains\. The user simulator is powered by the Qwen3\-235B\-A22B\-Instruct model[45](https://arxiv.org/html/2608.13622#bib.bib21)\. We repeat the evaluation 3 times and we report the mean score\.

##### AIME 2026\.

The American Invitational Mathematics Examination 2026 dataset[3](https://arxiv.org/html/2608.13622#bib.bib14)contains challenging math problems with integer answers in \[0, 999\]\. We extract the model’s answer from\\boxed\{\}notation\. Each problem is evaluated 8 times with different random seeds, and accuracy is computed by exact match after normalization\. The instruction prompt asks the model to solve the problem and format the final answer\.

##### GPQA Diamond\.

The Google\-Proof Q&A \(GPQA\) Diamond subset[38](https://arxiv.org/html/2608.13622#bib.bib12)contains expert\-level science questions with multiple\-choice answers\. We evaluate the model’s ability to select the correct option \(A, B, C, or D\)\. Each question is run 8 times, and we compute the accuracy as the fraction of correct selections\.

##### HMMT 2025\.

The Harvard\-MIT Mathematics Tournament February 2025 dataset[3](https://arxiv.org/html/2608.13622#bib.bib14)contains competition\-level math problems across multiple rounds\. Answers may be integers, fractions, expressions, or other mathematical objects\. Each problem is evaluated 8 times\.

##### IFBench\.

Instruction Following Benchmark[32](https://arxiv.org/html/2608.13622#bib.bib13)tests the model’s ability to follow complex, multi\-constraint instructions\. The benchmark contains test cases with specific formatting, length, and content requirements\. Evaluation is performed using the official IFBench scoring script, which checks constraint satisfaction\.

##### Arena\-Hard\.

Following the official Arena\-Hard evaluation protocol[20](https://arxiv.org/html/2608.13622#bib.bib10), we use the benchmark’s GPT\-4\-based judge to compare model responses against a baseline \(o3\-mini\-2025\-01\-31\) on 580 challenging questions from diverse domains\. We report the win rate percentage as the final score\.

### I\.3TTFT Measurement Protocol

Time\-to\-first\-token \(TTFT\) is measured specifically for tool\-calling benchmarks \(τ\\tau\-bench variants\) where latency is most critical\. The measurement protocol:

- •Definition:TTFT is the time from sending the request to receiving the firstuser\-visibletoken of the model’s response\.
- •Aggregation:For each row in Table[3](https://arxiv.org/html/2608.13622#S6.T3), we report the mean TTFT over all availableτ\\tau\-bench domain measurements for that configuration\.
- •Conditions:All TTFT measurements use concurrency=16 on H200 GPUs, with no other concurrent workloads\.
- •Scope:TTFT is not reported for reasoning benchmarks \(AIME, GPQA, HMMT\) or instruction\-following/alignment benchmarks \(IFBench, Arena\-Hard\) as these are primarily accuracy\-focused rather than latency\-sensitive\.

### I\.4Evaluation Uncertainty and Training\-Seed Robustness

We additionally evaluate the statistical variability of ARC under both repeated evaluation and independent training seeds\. These analyses separate variability arising from stochastic evaluation from variability arising from optimization randomness during training\.

Table 12:Evaluation uncertainty on the tool\-use benchmarks\. Results are reported as mean±\\pmstandard deviation overN=3N=3independent evaluation runs using the same trained checkpoint\. Tool\-use Avg\. is the mean over the five displayedτ\\tau\-bench andτ2\\tau^\{2\}\-bench metrics\.Table 13:Robustness to training randomness\. Results are reported as mean±\\pmstandard deviation over three independently trained models with different training seeds\. Each seed\-level score is obtained by averaging repeated evaluation runs for that checkpoint\.##### Evaluation uncertainty\.

Table[12](https://arxiv.org/html/2608.13622#A9.T12)reports variability across repeated evaluations of the same trained checkpoint\. For the mainτ\\tau\-bench andτ2\\tau^\{2\}\-bench results, we perform three independent evaluation runs for each model checkpoint and report the mean and standard deviation across runs\. In particular, ARC increases the tool\-use average from 33\.60 to 38\.45 for PPO and from 33\.35 to 41\.73 for GRPO\. The corresponding DAPO results are substantially closer, indicating that the effect of ARC depends on the underlying RL backbone\.

##### Robustness to training randomness\.

Table[13](https://arxiv.org/html/2608.13622#A9.T13)evaluates sensitivity to optimization randomness across three independent training seeds\. Each resulting checkpoint is evaluated using the same repeated\-evaluation protocol\. We first average the repeated evaluations within each seed and then report the mean and standard deviation across the three seed\-level scores\.

The improvements are consistent across training seeds for PPO and GRPO\. PPO improves from 31\.81 to 36\.76 on the tool\-use average, while GRPO improves from 34\.74 to 37\.53\. In contrast, DAPO remains essentially unchanged \(34\.69 versus 34\.77\), and we therefore do not interpret the DAPO result as evidence of a meaningful tool\-use improvement\.

## Appendix JTraining Details

### J\.1Hyperparameters

Table[14](https://arxiv.org/html/2608.13622#A10.T14)details RL training hyperparameters\.

Table 14:RL hyperparameters for the main experiments\.
### J\.2Compute Resources

All experiments were conducted on NVIDIA H200 GPUs, using NVIDIA driver version 570\.158\.01 and CUDA 13\.0\.