超越纠正后的行动：基于文本评语训练大语言模型智能体

# Beyond Corrected Actions: Training LLM Agents on Textual Critiques
Source: [https://arxiv.org/html/2610.19367](https://arxiv.org/html/2610.19367)
22footnotetext:Work performed during an internship at Capital One\. Correspondence to Thomas Zeng attpzeng@wisc\.eduand Genta Winata atgenta\.winata@capitalone\.com\.Zelei ChengAffiliation:AI Foundations, Capital OneMingyang ZhouAffiliation:AI Foundations, Capital OneSambit SahuAffiliation:AI Foundations, Capital OneWilliam CampbellAffiliation:AI Foundations, Capital OneGenta Indra WinataAffiliation:AI Foundations, Capital OneAffiliation:University of Wisconsin–Madison

###### Abstract

Large language models \(LLMs\) are increasingly used for multi\-turn agentic tasks that require interactions with users and environments over extended sequences of actions\. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert’s trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model\. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain*why*specific actions are incorrect\. We study whether training on such feedback benefits LLM agents and proposeCritique\-AugmentedAgenticFine\-Tuning \(CAFT\), a framework that elicits textual expert feedback on student\-generated trajectories\. For each incorrect action, the expert yields both a corrected action*and*a textual critique explaining the student’s mistake—which we then use as the training target for fine\-tuning\. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domainsτ2\\tau^\{2\}\-Bench Airline\+Retail and WebShop, adding critiques to action\-level supervision improves performance by 8% on average over action\-only distillation\. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert\-trajectory distillation\. Our results demonstrate that textual critiques can strengthen supervision for multi\-turn LLM agents\.

## 1Introduction

Figure 1:Overview of CAFT\.\(a\) The student model is rolled out and an expert labels every agent turn, writing a critiquectc\_\{t\}and a corrected actiona~t\\tilde\{a\}\_\{t\}for each incorrect turn\. \(b\) Action distillation replaces the incorrect step witha~t\\tilde\{a\}\_\{t\}while CAFT keeps the incorrect step in context and trains on the critique followed by the corrected action\. \(c\) Task success rate of Gemma 4 E2B onτ2\\tau^\{2\}Airline\+Retail and WebShop\.Large language models \(LLMs\) are increasingly deployed as agents in multiturn tool\-using settings that require interacting with users and environments over extended sequences of decisions\([Mohammadi et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib10);[Pan et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib11);[Xu et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib9);[Winata et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib1)\)\. Success in these settings requires correctly executing a sequence of interdependent actions\. Importantly, agent actions can be incorrect for many different reasons; such as violating a domain policy, issuing an invalid tool call or acting before having collected all the information it needs\. Therefore, training reliable multi\-turn agents requires supervision that steers them away from this diverse set of failures\.

When a stronger expert LLM is available, the simplest way to improve a smaller student agent is behavior cloning the expert’s successful trajectories\([Chen et al\., 2024](https://arxiv.org/html/2610.19367#bib.bib6);[Zeng et al\., 2024](https://arxiv.org/html/2610.19367#bib.bib5);[Chen et al\., 2025b](https://arxiv.org/html/2610.19367#bib.bib12)\)\. However, expert trajectories primarily demonstrate how the expert succeeds and provide little direct supervision on the mistakes that a weaker student itself tends to make\. DAgger\-style\([Ross et al\., 2011](https://arxiv.org/html/2610.19367#bib.bib7)\)methods thus instead obtain expert supervision on states the student visits\. Recent work has adapted this idea directly to long\-horizon LLM agents by relabeling learner\-relevant states with expert actions\([Li et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib13)\), or by providing dense token\-level teacher supervision on sequences generated by the student itself\([Agarwal et al\., 2024](https://arxiv.org/html/2610.19367#bib.bib14);[Liao et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib23)\)\.

While these approaches supervise what action the expert would execute, they do not provide supervision on why the student’s original action is incorrect\. Yet as the expert is itself an LLM, it*can*directly make this explanation explicit by identifying the specific policy, information requirement, or tool\-use constraint violated by the student’s action\. This capability makes it possible to construct training methods that incorporate the expert’s textual critique of the student’s mistake in addition to the corrected action\. As this textual critique can convey information not captured by the corrected action alone, we hypothesize that it provides additional learning signal, and ask:

> Does training an LLM agent on the expert’s critique of its incorrect actions improve the policy beyond training on the corrected expert action alone?

To study this question, we introduce Critique\-Augmented Agentic Fine\-Tuning \(CAFT\)\. CAFT first rolls out the student policy to collect the states and mistakes it encounters under its own trajectory distribution\. A privileged expert model then reviews these trajectories, identifies incorrect student actions, and provides both a corrected action and a natural\-language critique explaining what was wrong with the student’s original action\. CAFT then creates a fine\-tuning dataset that keeps each erroneous action in context and trains the student to produce both the critique and the counterfactually corrected action generated by the expert\.

We evaluate CAFT on two benchmarks: WebShop\([Yao et al\., 2022](https://arxiv.org/html/2610.19367#bib.bib4)\)and the Airline and Retail domains ofτ2\\tau^\{2\}\-Bench\([Barres et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib3)\)\. Across four student models from the Nemotron 3 and Gemma 4 families\([Blakeman et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib17);[Abd et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib18)\), CAFT consistently improves over action\-only supervision on student trajectories by88% on average and by up to2727% on WebShop with Nemotron 3 Nano 4B\. We further find in a single\-model case study that CAFT checkpoints provide a stronger initialization for reinforcement learning \(RL\) than either the base model or standard expert\-model distillation\. Our contributions are summarized as follows:

- •We introduce Critique\-Augmented Agentic Fine\-Tuning \(CAFT\), which trains LLM agents on their own trajectories annotated with expert critiques explaining*why*a mistake occurred in addition to the corrected actions\.
- •We show that CAFT consistently improves over training on corrected actions alone across four student models and two agentic environments\. Furthermore, targeted ablations show that the textual critiques provide a stronger learning signal than the corrected actions alone\.
- •We demonstrate the importance of privileged information in generating high quality critique generation, compare CAFT with expert\-trajectory distillation and repeated fine\-tuning, and evaluate it as an initialization for RL\.

## 2Training LLM Agents on Textual Critiques

Algorithm 1CAFT: Constructing critique\-augmented supervision from student trajectories1:Task set

𝒳\\mathcal\{X\}, base policy

π0\\pi\_\{0\}, environment

ℰ\\mathcal\{E\}, expert

CC, interjection

ρ\\rho
2:

𝒟SFT←∅\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\\leftarrow\\varnothing
3:for

x∈𝒳x\\in\\mathcal\{X\}do

4:

𝒯0​\(x\)←CollectTrajectories​\(π0,ℰ,x\)\\mathcal\{T\}\_\{0\}\(x\)\\leftarrow\\textsc\{CollectTrajectories\}\(\\pi\_\{0\},\\mathcal\{E\},x\)
5:for

τ∈𝒯0​\(x\)\\tau\\in\\mathcal\{T\}\_\{0\}\(x\)do

6:for

t=1,…,T⁡\(τ\)t=1,\\ldots,T\(\\tau\)do

7:

\(zt,ct,a~t\)←C⁡\(ht,at,x\)\(z\_\{t\},c\_\{t\},\\tilde\{a\}\_\{t\}\)\\leftarrow C\(h\_\{t\},a\_\{t\};x\)
8:if

zt=1z\_\{t\}=1then

9:

\(qt,yt\)←\(ht,at\)\(q\_\{t\},y\_\{t\}\)\\leftarrow\(h\_\{t\},a\_\{t\}\)
10:else

11:

qt←ht∘at∘ρq\_\{t\}\\leftarrow h\_\{t\}\\circ a\_\{t\}\\circ\\rho
12:

yt←ct∘a~ty\_\{t\}\\leftarrow c\_\{t\}\\circ\\tilde\{a\}\_\{t\}
13:

𝒟SFT←𝒟SFT∪\{\(qt,yt\)\}\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\\leftarrow\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\\cup\\\{\(q\_\{t\},y\_\{t\}\)\\\}
14:return

𝒟SFT\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}

To provide agents with explicit supervision on why their actions are incorrect, CAFT fine\-tunes a student policyπ0\\pi\_\{0\}using expert critiques constructed from the student’s own trajectories \(refer to Figure[1](https://arxiv.org/html/2610.19367#S1.F1)for an overview\)\. We first roll out withπ0\\pi\_\{0\}on a collection of multi\-turn agentic tasks, and then use an expertCCto evaluate each agent turn\. For turns judged incorrect, the expert provides both a natural\-language critique and the corrected action it would have taken instead\. These annotations are converted into supervised examples used to fine\-tune a policyπθ\\pi\_\{\\theta\}fromπ0\\pi\_\{0\}\. In our experiments,CCis an LLM with access to privileged task information, such as reference actions and evaluation criteria, which grounds its judgments in task requirements unavailable to the student\. The complete procedure for constructing this training dataset is outlined in Algorithm[1](https://arxiv.org/html/2610.19367#alg1)\.

#### Multi\-turn agent setting\.

To formally represent the multi\-turn agentic setting that our method focuses on, we define the agent interaction as a sequential decision making process\. Let𝒳\\mathcal\{X\}denote a collection of task instances and letx∼p𝒳x\\sim p\_\{\\mathcal\{X\}\}denote a sampled task\. Givenxx, the environmentℰ\\mathcal\{E\}emits an initial agent\-visible observationo1o\_\{1\}describing the task\. At agent turntt, the policy generates an action based on the previous interaction history

at∼π\(⋅∣ht\),whereht=\(o1,a1,…,ot\),a\_\{t\}\\sim\\pi\(\\cdot\\mid h\_\{t\}\),\\qquad\\text\{where \}h\_\{t\}=\(o\_\{1\},a\_\{1\},\\ldots,o\_\{t\}\),after which the environmentℰ\\mathcal\{E\}produces the next observationot\+1o\_\{t\+1\}\. Here, an actionata\_\{t\}is the agent output at turntt\(e\.g\., a tool call or a user\-facing message\), whileoto\_\{t\}is the information returned to the agent \(e\.g\., a tool call result or a user reply\)\. A rollout therefore produces a trajectory

τ=\(o1,a1,o2,a2,…,oT,aT,oT\+1\)\.\\tau=\(o\_\{1\},a\_\{1\},o\_\{2\},a\_\{2\},\\ldots,o\_\{T\},a\_\{T\},o\_\{T\+1\}\)\.We denote bypπ,ℰ​\(τ∣x\)p\_\{\\pi,\\mathcal\{E\}\}\(\\tau\\mid x\)the trajectory distribution induced by policyπ\\piand environmentℰ\\mathcal\{E\}on taskxx\. Each task is evaluated by a binary correctness graderR⁡\(x,τ\)∈\{0,1\},R\(x,\\tau\)\\in\\\{0,1\\\},whereR⁡\(x,τ\)=1R\(x,\\tau\)=1indicates successful completion\. The agent’s objective is to maximize expected task success,

J\(π\)=𝔼x∼p𝒳𝔼τ∼pπ,ℰ\(⋅∣x\)\[R\(x,τ\)\]\.J\(\\pi\)=\\mathbb\{E\}\_\{x\\sim p\_\{\\mathcal\{X\}\}\}\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\pi,\\mathcal\{E\}\}\(\\cdot\\mid x\)\}\\left\[R\(x,\\tau\)\\right\]\.CAFT aims to improveJJby constructing supervised training data from trajectories sampled from the base policyπ0\\pi\_\{0\}\.

#### Expert annotations\.

We now precisely describe the expert annotation process\. For each trajectoryτ∼pπ0,ℰ\(⋅∣x\)\\tau\\sim p\_\{\\pi\_\{0\},\\mathcal\{E\}\}\(\\cdot\\mid x\), the expert modelCCevaluates every agent turnata\_\{t\}\. It first assigns a correctness label

zt∈\{0,1\},z\_\{t\}\\in\\\{0,1\\\},wherezt=1z\_\{t\}=1denotes a correct action andzt=0z\_\{t\}=0an incorrect action\. Whenzt=0z\_\{t\}=0, the expert additionally generates a natural\-language critiquectc\_\{t\}explaining the error and a corrected actiona~t\\tilde\{a\}\_\{t\}specifying what the agent should have done givenhth\_\{t\}\. We summarize this annotation as

\(zt,ct,a~t\)=C⁡\(ht,at,x\),\(z\_\{t\},c\_\{t\},\\tilde\{a\}\_\{t\}\)=C\(h\_\{t\},a\_\{t\};x\),wherectc\_\{t\}anda~t\\tilde\{a\}\_\{t\}are required only whenzt=0z\_\{t\}=0\. In our implementation,CCmay additionally receive privileged information associated with taskxxthat is unavailable to the agent, such as gold reference actions or the evaluation criteria\. Both the critique and correction are generated in the agent’s output format so that they can be used directly as supervised targets\. For more details on the specific expert prompts used, refer to Appendix[B](https://arxiv.org/html/2610.19367#A2)\.

#### Supervised fine\-tuning dataset construction\.

For the construction of the fine\-tuning dataset, we convert each annotated agent turn into an independent supervised training example\(qt,yt\)\(q\_\{t\},y\_\{t\}\)whereqtq\_\{t\}is the context containing the preceding agentic interactions andyty\_\{t\}is the response the student is trained to produce\. Letρ\\rhodenote a fixed natural\-language interjection inserted as a user message, indicating that the preceding action was incorrect and requesting an explanation and a corrected next action\. We use∘\\circto denote concatenation under the model’s chat template\. For a correct turn \(zt=1z\_\{t\}=1\), we train the model to reproduce the action originally sampled byπ0\\pi\_\{0\}\. Thus, correct behavior from the base policy is retained as ordinary behavior\-cloning supervision—which we include because our ablation experiments \(see Appendix[E\.1](https://arxiv.org/html/2610.19367#A5.SS1)\) show they are beneficial\. For an incorrect turn \(zt=0z\_\{t\}=0\), we retain the erroneous actionata\_\{t\}in the context, append the interjectionρ\\rho, and train the model to produce the expert’s critique followed by its corrected action\. The training example associated with turnttis thus

\(qt,yt\)=\{\(ht,at\),zt=1,\(ht∘at∘ρ,ct∘a~t\),zt=0\.\(q\_\{t\},y\_\{t\}\)=\\begin\{cases\}\(h\_\{t\},\\;a\_\{t\}\),&z\_\{t\}=1,\\\\\[4\.0pt\] \(h\_\{t\}\\circ a\_\{t\}\\circ\\rho,\\;c\_\{t\}\\circ\\tilde\{a\}\_\{t\}\),&z\_\{t\}=0\.\\end\{cases\}The resulting supervised dataset is

𝒟SFT=⋃x∈𝒳⋃τ∈𝒯0​\(x\)⋃t=1T⁡\(τ\)\{\(qt,yt\)\},\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}=\\bigcup\_\{x\\in\\mathcal\{X\}\}\\bigcup\_\{\\tau\\in\\mathcal\{T\}\_\{0\}\(x\)\}\\bigcup\_\{t=1\}^\{T\(\\tau\)\}\\left\\\{\(q\_\{t\},y\_\{t\}\)\\right\\\},where𝒯0​\(x\)\\mathcal\{T\}\_\{0\}\(x\)denotes the set of trajectories collected fromπ0\\pi\_\{0\}on taskxx\. Finally, we fine\-tuneπθ\\pi\_\{\\theta\}on𝒟SFT\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}using the standard autoregressive language\-modeling objective\. Writingy=\(y1,…,y\|y\|\)y=\(y\_\{1\},\\ldots,y\_\{\|y\|\}\)for the target tokens, we minimize

ℒSFT\(θ\)=−∑\(q,y\)∈𝒟SFTw\(q,y\)∑j=1\|y\|logπθ\(yj∣q∘y<j\)\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\-\\sum\_\{\(q,y\)\\in\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\}w\(q,y\)\\sum\_\{j=1\}^\{\|y\|\}\\log\\pi\_\{\\theta\}\\left\(y\_\{j\}\\mid q\\circ y\_\{<j\}\\right\)\.The weightw⁡\(q,y\)w\(q,y\)is11on incorrect turns andλ\\lambdaon correct turns \(where we useλ=0\.1\\lambda=0\.1throughout; chosen following the results of a sweep in Appendix[E\.1](https://arxiv.org/html/2610.19367#A5.SS1)\)\. Lastly, we note the expert,ρ\\rho, and the critiques appear only in the training data\. At inference time, the finetuned policy acts as an ordinary agent, and is*not*interjected by critique from an expert model or asked to critique itself\.

## 3Experiments

Table 1:Comparison of CAFT and its ablated variants against action distillation\.Task success \(%\) onτ2\\tau^\{2\}AR and WebShop for the base policies, the action\-distillation baseline, and CAFT and its ablations\. CAFT\-Critique and CAFT consistently outperform all other methods in all student–environment pairs\.In this section, we introduce the experiments and ablation studies that we conducted to validate the design of CAFT\. The details of the experiment setup is described in Secion[3\.1](https://arxiv.org/html/2610.19367#S3.SS1)In Section[3\.2](https://arxiv.org/html/2610.19367#S3.SS2), we establish the main claim that textual critique carries supervision beyond the corrected action alone when training LLM agents\. Section[3\.3](https://arxiv.org/html/2610.19367#S3.SS3)then asks what the expert needs in order to produce useful critiques, and shows that privileged task information matters more than fine\-grained prompting\. Sections[3\.4](https://arxiv.org/html/2610.19367#S3.SS4)and places CAFT against the natural alternatives—distilling the expert’s trajectories, and[3\.5](https://arxiv.org/html/2610.19367#S3.SS5)studies the effectiveness of running mulitple iterations of CAFT\. Finally, Section[3\.6](https://arxiv.org/html/2610.19367#S3.SS6)asks whether CAFT can be used effectively as a warm start for RL\.

### 3\.1Experimental Setup

#### Environments and evaluation\.

We evaluate on the Airline and Retail domains ofτ2\\tau^\{2\}\-Bench \(τ2\\tau^\{2\}AR\)\([Barres et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib3)\), and WebShop\([Yao et al\., 2022](https://arxiv.org/html/2610.19367#bib.bib4)\)\.τ2\\tau^\{2\}AR evaluates conversational agents that must interact with a user and external tools while following domain\-specific policies, on tasks such as booking a flight or modifying an online order\. WebShop evaluates agents that navigate a simulated online store to find and purchase an item satisfying a natural\-language request\. Forτ2\\tau^\{2\}AR, we use the benchmark’s outcome\-based evaluator, and for WebShop, we report the fraction of episodes that purchase an item satisfying all the request’s constraints\. For both datasets, we collect1616rollouts per task\. Full details are in Appendix[A](https://arxiv.org/html/2610.19367#A1)\.

#### Models and data collection\.

We evaluate four student policies: NVIDIA Nemotron 3 Nano 4B, Gemma 4 E2B, Gemma 4 E4B and Gemma 4 31B\([Blakeman et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib17);[Abd et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib18)\)\. We use Gemma 4 31B as the expert model for generating turn\-level correctness judgments, critiques, and corrected actions\. When Gemma 4 31B is also the student, the expert critiques its own trajectories, so this setting tests whether a model can improve from*self*\-critique given privileged task information\. We partition both environments into training and evaluation task splits\. Then, for each student–environment pair we roll out the base policyπ0\\pi\_\{0\}over the training tasks with1616rollouts per task and have the expert annotate every agent turn\. Complete collection, dataset\-construction, and optimization settings are given in Appendix[C](https://arxiv.org/html/2610.19367#A3)\.

#### Baselines and supervision variants\.

We compare CAFT against the unmodified base policy, two ablations of CAFT and an action distillation baseline designed to isolate the supervision typified by LEAP\([Choudhury and Sodhi, 2025](https://arxiv.org/html/2610.19367#bib.bib25)\), SOPD\([Sun et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib26)\), and[Ye et al\. \(2026\)](https://arxiv.org/html/2610.19367#bib.bib27)\. Our experiment controls for both the student\-generated trajectories and the expert annotations, varying*only*the supervision used for expert\-flagged turns:

- •Base: the original student policyπ0\\pi\_\{0\}, without additional fine\-tuning\.
- •Action Distillation: trains the student on the expert’s corrected action alone, with the mistake removed from context,\(qt,yt\)=\(ht,a~t\)\(q\_\{t\},y\_\{t\}\)=\(h\_\{t\},\\tilde\{a\}\_\{t\}\)\.
- •CAFT\-Correction: removes the critique target from CAFT, keeping only the expert correction,\(qt,yt\)=\(ht∘at∘ρ,a~t\)\(q\_\{t\},y\_\{t\}\)=\(h\_\{t\}\\circ a\_\{t\}\\circ\\rho,\\tilde\{a\}\_\{t\}\)\.
- •CAFT\-Critique: removes the corrected\-action target from CAFT and trains only on the expert’s textual critique,\(qt,yt\)=\(ht∘at∘ρ,ct\)\(q\_\{t\},y\_\{t\}\)=\(h\_\{t\}\\circ a\_\{t\}\\circ\\rho,c\_\{t\}\)\.
- •CAFT: trains on both the expert’s critique and corrected action,\(qt,yt\)=\(ht∘at∘ρ,ct∘a~t\)\(q\_\{t\},y\_\{t\}\)=\(h\_\{t\}\\circ a\_\{t\}\\circ\\rho,\\,c\_\{t\}\\circ\\tilde\{a\}\_\{t\}\)\.

### 3\.2Do Critiques Provide Supervision Beyond Corrected Actions?

We compare CAFT against the baselines and the two variants of CAFT and summarize the results in Table[1](https://arxiv.org/html/2610.19367#S3.T1)\. First, we observe that training on correct expert actions yields clear improvements over the base model\. The Action Distillation baseline consistently outperforms the base model across all settings\. However, incorporating textual critique via CAFT gives further improvements\. CAFT outperforms Action Distillation in all eight settings, by 8\.4% on average, and delivers the strongest overall performance in six of the eight cases\.

To isolate the value of the textual critique, we compare our two CAFT ablations\. Crucially, CAFT\-Critique never trains on the corrected action for flagged turns, yet it consistently outperforms CAFT\-Correction\. This provides direct evidence that explaining the error is more beneficial to the student model than merely providing the correct action\. We further observe that CAFT\-Correction, which differs from Action Distillation primarily in keeping the mistaken step in the context, performs similarly to the Action Distillation baseline in most settings\. Thus, the majority of CAFT’s advantage stems directly from the textual critique\. Comparing the two CAFT ablations, CAFT\-Critique consistently outperforms CAFT\-Correction across all student–environment pairs\. While gains vary considerably by setting, the average is 5\.0%, with the largest from Nemotron 3 Nano 4B on WebShop\.

Finally, we highlight the results for the Gemma 4 31B experiments where the expert is the student itself\. In this setting, CAFT still improves over the base model \(by4\.34\.3% onτ2\\tau^\{2\}AR and7\.97\.9% on WebShop\) and over Action Distillation, albeit by smaller margins\. This suggests that CAFT may remain useful when no stronger expert is available\. In summary, the results demonstrate that the textual critiques provide effective supervision to improve LLM agents\. Training student agents to predict critiques over their own mistakes gives non\-trivial gain over using only the correct actions, establishing CAFT as a broadly effective supervision paradigm for fine\-tuning LLM agents\.

### 3\.3How Much Does the Expert Need to Know to Critique Effectively?

The expert prompt used for the experiments in Section[3\.2](https://arxiv.org/html/2610.19367#S3.SS2)gives the expert two types of help\. The first is a detailed domain\-specific prompt describing what types of actions are incorrect; i\.e\., actions that apply to the wrong entity, skip a policy\-mandated verification, perform an incorrect or irreversible write, or contradict the policy or the user\. The second is privileged task information such as the gold reference actions, the user scenario and the evaluation criteria used to that the student never sees \(Appendix[B](https://arxiv.org/html/2610.19367#A2)\)\.

To isolate the contribution of each, we construct an ablation on the three smaller student models that independently varies whether the expert receives a detailed instruction or privileged task information\. We re\-annotate the student trajectories with the same expert under each condition and rerun CAFT\. Table[2](https://arxiv.org/html/2610.19367#S3.T2)shows that privileged information is the dominant factor\. Removing it reduces success by 7\.5% on average\. In contrast, when privileged information is available, replacing the detailed instruction with a minimal one retains most of the performance\. The detailed prompt matters most for the weakest model, Gemma 4 E2B, while having no impact on Nemotron 3 Nano 4B\. In summary, high\-quality supervision from the expert model therefore primarily depends on the expert knowledge on the task requirements instead of the detailed guidance\.

Table 2:Ablating privileged information and detailed guidance in the expert prompt\.τ2\\tau^\{2\}AR task success \(%\) of CAFT trained on critiques from Gemma 4 31B, varying whether the expert receives privileged task information and a detailed critique prompt\. Privileged information has a larger impact than the detailed prompt\.
### 3\.4Comparison to Expert\-Trajectory Distillation

We also compare CAFT to direct distillation from expert trajectories on agentic tasks to validate the idea of training students LLM agent to correct their own mistakes is a stronger alternative than just imitating teacher’s behaviors\. Our expert model, Gemma 4 31B, outperforms every base student, achieving 65\.1% and 53\.1% task success onτ2\\tau^\{2\}AR and WebShop, respectively \(Table[1](https://arxiv.org/html/2610.19367#S3.T1)\)\. So its trajectories are a natural source of behavior\-cloning data\. Table[3](https://arxiv.org/html/2610.19367#S3.T3)compares this Expert Distillation baseline with CAFT for the three smaller students\.

On WebShop, expert distillation proves entirely ineffective, bringing success rate to around 15\.0–19\.1% \(below the base model’s 24\.3\-43\.1%\) despite the expert outperforming every student model\. This is in contrast to the strong performance of CAFT\. Inspecting the distilled policy trajectories, we observe they continuously search for new items until they exhaust the allowed turn budget, without committing to a purchase\. This suggests the models learned to mimics the expert’s exploration patterns but not its stopping conditions\. CAFT avoids this collapse because its supervision is anchored on the student’s own context and mistakes\.

Onτ2\\tau^\{2\}AR, the results are more nuanced\. Expert distillation is slightly better than CAFT on Gemma 4 E4B and Gemma 4 E2B, while CAFT performs better on Nemotron 3 Nano 4B\. Given expert trajectories help onτ2\\tau^\{2\}AR but cause degenerate behavior on WebShop, we investigate whether the two supervision sources can be complementary forτ2\\tau^\{2\}AR\. The CAFT \+BC variant mixes 50% expert trajectories for behavior cloning \(BC\) with 50% CAFT data\. This mixture beats both of its components for both Gemma 4 E2B \(53\.7%\) and Gemma 4 E4B \(60\.1%\)\. For Nemotron 3 Nano 4B, however, CAFT \+BC performs worse than CAFT alone \(56\.0% vs\. 60\.3%\)\. Taken together, expert demonstrations can therefore complement critique supervision, but the benefit depends on the student and environment\. The expert’s task success alone is not sufficient to determine whether its demonstrations will improve the student\.

Table 3:Comparison of CAFT with expert\-trajectory distillation\.Task success \(%\)\. Expert Distillation behavior\-clones trajectories from Gemma 4 31B; CAFT \+BC mixes 50% expert trajectories with 50% CAFT data\. Expert distillation collapses on WebShop, so CAFT \+BC is not run there\.
### 3\.5Does a Second Iteration of CAFT Help?

To investigate the upper bounds of our approach, we explore whether the benefits of CAFT compound across multiple iterations or if the performance quickly saturates\. Specifically, we experiment with taking the checkpoint from the first iteration and then rolling out and re\-annotating new trajectories to fine\-tune for a second iteration\.

Table[4](https://arxiv.org/html/2610.19367#S3.T4)shows the result of a second iteration of training\. From it, we see a clear difference by environment\. On WebShop, a second iteration improves every student by5\.35\.3% on average and by as much as10\.910\.9% for Nemotron 3 Nano 4B\. Onτ2\\tau^\{2\}AR the picture is mixed: while the two small Gemma students improve modestly \(1\.61\.6% and3\.13\.1%\), Nemotron 3 Nano 4B does not see any more improvements, suggesting that the gains saturated after one iteration\. We thus find that in general another iteration of CAFT*can*help\. Although the amount it helps depends strongly on the environment and headroom after the first iteration\.

Table 4:Using CAFT for a second iteration\.Task success \(%\) after one and two iterations of CAFT\. A second iteration helps every student on WebShop and the smaller Gemma students onτ2\\tau^\{2\}AR\.
### 3\.6Can We Use CAFT as a Warm Start for RL?

Supervised fine\-tuning is commonly used to warm start reinforcement learning \(RL\), placing the policy in a higher\-reward region to improve sample efficiency\([Ouyang et al\., 2022](https://arxiv.org/html/2610.19367#bib.bib34);[Guo et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib35)\)\. Canonically, warm\-starting is achieved by distilling trajectories from a stronger model\. We thus ask if CAFT yields a better initialization than expert distillation\.

As an initial investigation, we use Nemotron 3 Nano 4B onτ2\\tau^\{2\}AR as a case study\. We run 300 steps of GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.19367#bib.bib24)\)on theτ2\\tau^\{2\}AR training split, varying only the initialization: \(i\) the base model \(ii\) the expert distillation checkpoint and \(iii\) the CAFT checkpoint\. All three runs share the same GRPO hyperparameters, with full details in Appendix[D](https://arxiv.org/html/2610.19367#A4)\. As shown in Figure[2](https://arxiv.org/html/2610.19367#S3.F2), CAFT outperforms both alternatives by around 5% on the held\-out tasks\. Notably, while the expert distillation warm\-start achieves a higher training reward than the base model, it does not transfer—with its validation performance collapsing to the base model\.

Since both warm starts are trained on the same training tasks that RL later optimizes, we hypothesize that behavior cloning on the expert’s solutions to these tasks instills task specific patterns, which RL then amplifies\. This then raises training reward without improving generalization\. Meanwhile, CAFT’s supervision instead targets the student’s own mistakes, which appears to transfer better to unseen tasks\. Lastly, we note CAFT also makes RL more sample\-efficient: the CAFT initialized run matches the base\-initialized run’s best held\-out success after about 60 of 300 steps\.

Figure 2:Using CAFT as a warm start for RL\.GRPO onτ2\\tau^\{2\}AR with Nemotron 3 Nano 4B from three initializations\. \(a\) Validation task success, averaged over the airline and retail validation splits\. \(b\) Mean reward of on\-policy rollouts during training\.

## 4Related Work

#### Expert supervision for training LLM agents\.

A common approach to train LLM agents is to imitate the behavior from the trajectories generated by stronger models\([Zeng et al\., 2024](https://arxiv.org/html/2610.19367#bib.bib5);[Chen et al\., 2024](https://arxiv.org/html/2610.19367#bib.bib6);[Chen et al\., 2025b](https://arxiv.org/html/2610.19367#bib.bib12)\)\. However at inference time, a weaker student may encounter states absent from the expert data\. DAgger addresses this problem by obtaining expert labels on states induced by the student policy\([Ross et al\., 2011](https://arxiv.org/html/2610.19367#bib.bib7)\)\. Analogously, on\-policy distillation trains on student\-generated sequences with dense token\-level supervision from a teacher model\([Agarwal et al\., 2024](https://arxiv.org/html/2610.19367#bib.bib14)\)\. Recent work has extended these ideas to long\-horizon LLM agents\([Li et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib13);[Liao et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib23)\)and to*action\-level*supervision on student\-visited states, where a privileged teacher rewrites the student’s erroneous step\([Choudhury and Sodhi, 2025](https://arxiv.org/html/2610.19367#bib.bib25);[Lyu et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib28)\), or a teacher continues from a student prefix for one\([Sun et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib26)\)or a few\([Ye et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib27)\)steps\. However, all the existing approaches directly drop the student’s mistakes from the training context by replacing them with the teacher’s action\. Unlike the previous methods, CAFT shares the goal of supervising the states the student itself visits \(our Action Distillation baseline, Section[3\.1](https://arxiv.org/html/2610.19367#S3.SS1), instantiates this form of supervision\), but keeps the mistake in context so that we can train on the expert’s critique of it\.

#### Training from natural language critiques\.

Several works have explored using critiques as supervision for language models\. In single\-turn reasoning, Critique Fine\-Tuning \(CFT\) trains models to critique noisy responses rather than imitate correct solutions\([Wang et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib2)\)\. LEMA fine\-tunes on an explanation of an incorrect solution followed by its correction\([An et al\., 2023](https://arxiv.org/html/2610.19367#bib.bib29)\)and[Ye et al\. \(2025\)](https://arxiv.org/html/2610.19367#bib.bib30)show how error\-correction data can be used for mathematical pretraining\. Critique\-Guided Distillation \(CGD\) instead trains the student to refine its own response*given*a teacher’s critique\([Kapusuzoglu et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib8)\)\. Related reinforcement\-learning approaches likewise use natural\-language critiques to guide policy improvement\([Zhang et al\., 2026b](https://arxiv.org/html/2610.19367#bib.bib19);[Lin et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib20)\)\. For agents, Early Experience trains a policy to write its own reflection on why an expert action beats an alternative at expert\-visited states\([Zhang et al\., 2026a](https://arxiv.org/html/2610.19367#bib.bib31)\), and CAST conditions a tool\-calling policy on a separate critic’s verdicts\([Saeidi et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib32)\)\. CAFT builds on these works by asking whether training a multi\-turn agent to explain its own mistakes improves its decisions beyond learning the expert’s corrected actions alone\.

#### Reflection and error recovery in LLM agents\.

A complementary line of work uses natural language feedback to enable LLM agents to reflect and self\-correct\. Reflexion uses verbal feedback stored in an episodic memory to improve subsequent attempts without updating model parameters\([Shinn et al\., 2023](https://arxiv.org/html/2610.19367#bib.bib21)\)\. Critique\-Guided Improvement \(CGI\) trains a separate critic to generate actionable natural\-language feedback and an actor to revise its candidate action conditioned on the critic’s generated feedback\([Yang et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib16)\)\. Other work directly trains agents to exhibit reflection or error\-recovery behavior\. Agent\-R constructs revision trajectories by joining erroneous and successful branches discovered through Monte Carlo tree search, then trains agents to recover from mistakes\([Yuan et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib15)\)\. Agentic Critical Training instead rewards an agent for judging which of two candidate actions is better\([Liu et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib33)\), and STeP trains on teacher\-generated self\-reflected trajectories containing both erroneous behavior and its correction\([Chen et al\., 2025a](https://arxiv.org/html/2610.19367#bib.bib22)\)\. CAFT likewise trains on critiques and corrections, but its correction specifies what the agent should have done in place of the mistake, rather than how to proceed afterward\. Additionally, we explicitly isolate the supervision value of the critique itself, asking if it can better improves the agent’s task performance\.

## 5Conclusion

This work introduces CAFT, a critique\-augmented SFT method for multiturn LLM agents that fine\-tunes the student on textual critiques and corrected actions generated by an expert over the student’s own trajectories\. We find that critiques carry learning signal beyond the corrected action with CAFT\-Critique outperforming CAFT\-Correction and action\-only distillation across all tested settings\. Training an agent to*explain*its mistakes thus improves its ability to*perform*the correct actions\. More broadly, CAFT exploits the fact that expert and student share a natural\-language interface: critiques are a cheap and rich form of feedback that can be incorporated directly into training\. We see them as one more axis along which LLM agents can be improved\.

Several interesting directions follow from this work\. Our Gemma 4 31B results suggest that a model given privileged information can improve from critiques of its own trajectories\. Extending this self\-critique setting to more models and domains is a natural next step\. Because the critique target is natural language, CAFT could also consume feedback from sources other than an expert LLM, most notably direct user feedback at deployment, which suggests online variants that train on corrections as they arrive\. Additionally, while our experiments with CAFT \+BC show promise in combining critiques with expert demonstrations, identifying the precise conditions under which these mixtures succeed and exploring integrations with other supervision methods remain important areas for future study\. Finally, scaling to larger students and broader agentic domains, and developing a theoretical account of why critique targets help, remain open\.

## References

- Abdet al\.\(2026\)S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon,et al\.Gemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p5.1),[§3\.1](https://arxiv.org/html/2610.19367#S3.SS1.SSS0.Px2.p1.1)\.
- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21246–21263\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p2.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Anet al\.\(2023\)S\. An, Z\. Ma, Z\. Lin, N\. Zheng, J\. Lou, and W\. ChenLearning from mistakes makes llm better reasoner\.arXiv preprint arXiv:2310\.20689\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.
- Barreset al\.\(2026\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. R\. Narasimhan$\\tau^2$\-bench: evaluating conversational agents in a dual\-control environment\.InForty\-third International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2610.19367#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19367#S1.p5.1),[§3\.1](https://arxiv.org/html/2610.19367#S3.SS1.SSS0.Px1.p1.1)\.
- Blakemanet al\.\(2025\)A\. Blakeman, A\. Grattafiori, A\. Basant, A\. Gupta, A\. Khattar, A\. Renduchintala, A\. Vavre, A\. Shukla, A\. Bercovich, A\. Ficek,et al\.Nvidia nemotron 3: efficient and open intelligence\.arXiv preprint arXiv:2512\.20856\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p5.1),[§3\.1](https://arxiv.org/html/2610.19367#S3.SS1.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025a\)Y\. Chen, B\. Xu, X\. Wang, Y\. Zhang, and Z\. MaoTraining llm\-based agents with synthetic self\-reflected trajectories and partial masking\.arXiv preprint arXiv:2505\.20023\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2024\)Z\. Chen, K\. Liu, Q\. Wang, W\. Zhang, J\. Liu, D\. Lin, K\. Chen, and F\. ZhaoAgent\-flan: designing data and methods of effective agent tuning for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 9354–9366\.Cited by:[Appendix A](https://arxiv.org/html/2610.19367#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.19367#S1.p2.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2025b\)Z\. Chen, M\. Li, Y\. Huang, Y\. Du, M\. Fang, and T\. ZhouAtlas: agent tuning via learning critical steps\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 25334–25349\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p2.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Choudhury and Sodhi \(2025\)S\. Choudhury and P\. SodhiBetter than your teacher: llm agents that learn from privileged ai feedback\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 10457–10493\.Cited by:[§3\.1](https://arxiv.org/html/2610.19367#S3.SS1.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§3\.6](https://arxiv.org/html/2610.19367#S3.SS6.p1.1)\.
- Kapusuzogluet al\.\(2026\)B\. Kapusuzoglu, S\. Chakraborty, Z\. Sarwar, C\. Lee, and S\. SahuCritique\-guided distillation for robust reasoning via refinement\.InForty\-third International Conference on Machine Learning,Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2026\)C\. Li, R\. Qiang, J\. Huang, C\. Gao, C\. Zhang, N\. He, and B\. DaiRevisiting dagger in the era of llm\-agents\.arXiv preprint arXiv:2605\.12913\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p2.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Liaoet al\.\(2026\)B\. Liao, H\. Dong, C\. Monz, X\. Xu, L\. Dong, and F\. WeiMulti\-turn on\-policy distillation with prefix replay\.arXiv preprint arXiv:2607\.04763\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p2.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Linet al\.\(2026\)J\. Lin, X\. Yu, Y\. Xin, Y\. Guo, Z\. Jiang, Z\. Yue, W\. Wang, H\. Zou, C\. Qin, and H\. XiongICRL: learning to internalize self\-critique with reinforcement learning\.arXiv preprint arXiv:2605\.15224\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)W\. Liu, M\. Liu, S\. Ho, S\. Chakraborty, X\. Wang, and F\. HuangAgentic critical training\.arXiv preprint arXiv:2603\.08706\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px3.p1.1)\.
- Lyuet al\.\(2026\)Y\. Lyu, C\. Wang, J\. Huang, and T\. XuStudent\-centered distillation narrows the agentic gap between small and large llms\.InForty\-third International Conference on Machine Learning,Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Mohammadiet al\.\(2025\)M\. Mohammadi, Y\. Li, J\. Lo, and W\. YipEvaluation and benchmarking of llm agents: a survey\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 6129–6139\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p1.1)\.
- Ohet al\.\(2026\)C\. Oh, W\. Li, S\. Park, S\. Yeh, T\. Mallick, and S\. LiNeglected free lunch from post\-training: progress advantage for llm agents\.arXiv preprint arXiv:2606\.26080\.Cited by:[Appendix A](https://arxiv.org/html/2610.19367#A1.SS0.SSS0.Px2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§3\.6](https://arxiv.org/html/2610.19367#S3.SS6.p1.1)\.
- Panet al\.\(2025\)M\. Z\. Pan, N\. Arabzadeh, R\. Cogo, Y\. Zhu, A\. Xiong, L\. A\. Agrawal, H\. Mao, E\. Shen, S\. Pallerla, L\. Patel,et al\.Measuring agents in production\.arXiv preprint arXiv:2512\.04123\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p1.1)\.
- Rosset al\.\(2011\)S\. Ross, G\. Gordon, and D\. BagnellA reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the fourteenth international conference on artificial intelligence and statistics,pp\. 627–635\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p2.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Saeidiet al\.\(2026\)A\. Saeidi, Z\. Zhang, R\. Singh, N\. Ahuja, V\. Gupta, A\. Payani, G\. Liu, J\. Srinivasa, and C\. BaralCAST: critique\-aware supervision for training reliable long\-horizon tool\-calling agents\.arXiv preprint arXiv:2608\.30147\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3\.6](https://arxiv.org/html/2610.19367#S3.SS6.p2.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[Appendix D](https://arxiv.org/html/2610.19367#A4.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px3.p1.1)\.
- Sunet al\.\(2026\)C\. Sun, L\. Liu, H\. Lei, T\. Ling, J\. Xie, Z\. Zheng, Y\. Wang, H\. Liu, F\. Xiao, L\. Liu,et al\.Step\-level on\-policy distillation: interpolating between on\-policy distillation and supervised fine\-tuning\.arXiv preprint arXiv:2608\.16333\.Cited by:[§3\.1](https://arxiv.org/html/2610.19367#S3.SS1.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)Y\. Wang, X\. Yue, and W\. ChenCritique fine\-tuning: learning to critique is more effective than learning to imitate\.InSecond Conference on Language Modeling,Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.
- Winataet al\.\(2026\)G\. I\. Winata, A\. Chakraborty, Y\. Lin, S\. P\. Rao, S\. Siingh, H\. Lu, N\. Bathaee, S\. Hatwar, P\. Dashore, A\. Jain,et al\.T1\-bench: benchmarking multi\-scenario agents in real\-world domains\.arXiv preprint arXiv:2606\.11070\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p1.1)\.
- Wuet al\.\(2024\)Y\. Wu, Y\. Fan, S\. Y\. Min, S\. Prabhumoye, S\. M\. McAleer, R\. Salakhutdinov, Y\. Bisk, Y\. Li, and T\. MitchellAgentKit: structured LLM reasoning with dynamic graphs\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=PKfAq8N4fK)Cited by:[Appendix A](https://arxiv.org/html/2610.19367#A1.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2025\)F\. F\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao, M\. Yang, H\. Y\. Lu, A\. Martin, Z\. Su, L\. M\. Maben, R\. Mehta, W\. Chi, L\. K\. Jang, Y\. Xie, S\. Zhou, and G\. NeubigTheAgentCompany: benchmarking LLM agents on consequential real world tasks\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p1.1)\.
- Yanget al\.\(2025\)R\. Yang, F\. Ye, J\. Li, S\. Yuan, Y\. Zhang, Z\. Tu, X\. Li, and D\. YangThe lighthouse of language: enhancing LLM agents via critique\-guided improvement\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px3.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebshop: towards scalable real\-world web interaction with grounded language agents\.Advances in Neural Information Processing Systems35,pp\. 20744–20757\.Cited by:[Appendix A](https://arxiv.org/html/2610.19367#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.19367#S1.p5.1),[§3\.1](https://arxiv.org/html/2610.19367#S3.SS1.SSS0.Px1.p1.1)\.
- Yeet al\.\(2026\)J\. Ye, J\. Cheng, M\. Lu, M\. Mankowski, J\. Blanchet, and M\. BayatiA few teacher steps go a long way: cost\-efficient on\-policy data augmentation for agent post\-training\.InICML 2026 Workshop on RL from World Feedback,Cited by:[§3\.1](https://arxiv.org/html/2610.19367#S3.SS1.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2025\)T\. Ye, Z\. Xu, Y\. Li, and Z\. Allen\-ZhuPhysics of language models: part 2\.2, how to learn from mistakes on grade\-school math problems\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 78136–78147\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2025\)S\. Yuan, Z\. Chen, Z\. Xi, J\. Ye, Z\. Du, and J\. ChenAgent\-r: training language model agents to reflect via iterative self\-training\.arXiv preprint arXiv:2501\.11425\.Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px3.p1.1)\.
- Zenget al\.\(2024\)A\. Zeng, M\. Liu, R\. Lu, B\. Wang, X\. Liu, Y\. Dong, and J\. TangAgenttuning: enabling generalized agent abilities for llms\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 3053–3077\.Cited by:[§1](https://arxiv.org/html/2610.19367#S1.p2.1),[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026a\)K\. Zhang, X\. Chen, B\. Liu, T\. Xue, Z\. Liao, Z\. Liu, X\. Wang, Y\. Ning, Z\. Chen, X\. Fu, J\. Xie, Y\. Sun, B\. Gou, Q\. Qi, Z\. Meng, J\. Yang, N\. Zhang, X\. Li, A\. Shah, D\. Huynh, H\. Li, Z\. Yang, X\. Cao, L\. K\. Jang, S\. Zhou, J\. Zhu, H\. Sun, J\. E\. Weston, Y\. Su, and Y\. WuAgent learning via early experience\.InForty\-third International Conference on Machine Learning,Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026b\)X\. Zhang, Y\. Zhang, H\. Sun, K\. Feng, C\. Lu, C\. Yang, and H\. M\. MengAdvancing LLM reasoning with natural language and numerical feedback\.InForty\-third International Conference on Machine Learning,Cited by:[§4](https://arxiv.org/html/2610.19367#S4.SS0.SSS0.Px2.p1.1)\.

## Appendix AEnvironments, Task Splits, and Evaluation Protocol

#### τ2\\tau^\{2\}\-Bench Airline and Retail\.

τ2\\tau^\{2\}\-Bench\([Barres et al\., 2026](https://arxiv.org/html/2610.19367#bib.bib3)\)evaluates conversational agents that interact with a simulated user while operating a domain database through tool calls and obeying a written domain policy\. We use the Airline and Retail domains, which we report jointly asτ2\\tau^\{2\}AR\. The agent observes the domain policy and tool schema in its system prompt, and interacts with a user simulator\. Each task terminates when the agent ends the conversation or the turn budget is exhausted \(4040agent turns\)\. The benchmark uses a binary outcome grader, which checks the final database state against the task’s gold reference actions and evaluates any required user\-facing information with a natural\-language judge\. In our experiments, both the user simulator and the judge are Gemma 4 31B\. We hold out6060tasks spanning both domains for evaluation and the remaining tasks form the collection pool\. The data split follows the train and evaluation split specified in the official code release ofτ2\\tau^\{2\}\-bench \(originally intended for people running RL training experiments\)\. Each evaluation task is rolled out1616times and we report mean task success\.

#### WebShop\.

WebShop\([Yao et al\., 2022](https://arxiv.org/html/2610.19367#bib.bib4)\)places the agent in a simulated e\-commerce site with a product catalog\. Given a natural\-language product request, the agent issues search queries, clicks results and option buttons, and finally purchases an item\. Following the evaluation protocols of previous papers—cf\. the work of[Chen et al\. \(2024\)](https://arxiv.org/html/2610.19367#bib.bib6),[Wu et al\. \(2024\)](https://arxiv.org/html/2610.19367#bib.bib36)and[Oh et al\. \(2026\)](https://arxiv.org/html/2610.19367#bib.bib37)—we use the10001000\-product WebShop instance \(items\_shuffle\_1000\.json\) fixing the first100100goals in the environment’s fixed goal ordering as the evaluation set\. We further use the second100100goals to form the training pool\. We run training and evaluation using thetext\_richobservation mode\. Each evaluation goal is rolled out1616times\. The agent is driven by a simple text\-based ReAct harness where it receives the system prompt of Appendix[B\.2](https://arxiv.org/html/2610.19367#A2.SS2)once and then, at every turn, anObservation:message containing the rendered page with its clickable elements marked as\[button\] …\[button\_\]\. Each turn, the agent is required to end its turn with a singleAction: search\[…\]orAction: click\[…\]line\. An episode ends when the agent clicksBuy Nowor after3030actions\. We report the fraction of episodes in which the purchased item satisfies all of the request’s constraints \(product type, attributes, options, and price limit\)\.

#### Decoding Parameters\.

Trajectory collection forτ2\\tau^\{2\}AR uses temperature0\.70\.7\. WebShop uses temperature1\.01\.0withtop\-​p=0\.95\\text\{top\-\}p=0\.95for Nemotron andtop\-​k=64\\text\{top\-\}k=64for the Gemma students, matching their released generation configurations\. Evaluation uses the same decoding parameters as collection\. The expertCCis Gemma 4 31B Instruct in all experiments and is queried with greedy decoding\.

## Appendix BExpert Annotation and Interjection Prompts

We use different expert prompts for the two environments since their action spaces and reward structures are different\. Forτ2\\tau^\{2\}AR, the expert judges tool calls and user\-facing messages against a domain policy, whereas on WebShop it judgessearch/clickactions against a single target product\. Both prompts share the same design principles, following our CAFT method of giving the expert privileged information, having it judge one action step at a time and, for a flagged step, write a textual critique and the counterfactually corrected action\. The student\-side interjectionρ\\rho\(Appendix[B\.4](https://arxiv.org/html/2610.19367#A2.SS4)\) is identical in both environments\.

### B\.1τ2\\tau^\{2\}AR expert annotation prompt

The expert is given, for each agent turn: the task description, the user’s hidden scenario, the gold reference actions together with the reward’s evaluation criteria, an optional slice of the relevant database state, the conversation so far, and the single step to be judged\. It returns a criticality flag, a verdict, and a free\-text analysis\. We use the analysis of amisstepverdict as the critiquectc\_\{t\}and parse the text following “Instead:” as the corrected actiona~t\\tilde\{a\}\_\{t\}\. The system prompt body is given in Figure[3](https://arxiv.org/html/2610.19367#A2.F3)\. The short per\-domain note of Figure[4](https://arxiv.org/html/2610.19367#A2.F4)is inserted immediately before item 1, and the agent’s full domain policy and tool schema are appended to it verbatim\.

Expert annotation prompt — system messageYou are a privileged COACH reviewing a customer\-service AGENT\. You review EVERY step the agent takes \-\- aread\-only lookup, a database write, or a message to the user \-\- AFTER it has already executed\. You neverblock, undo, or change anything; you only record a short verdict on the step taken\.You are given information the AGENT does NOT have: the GOLD reference actions, the reward’s EVALUATIONCRITERIA, possibly a slice of the relevant DATABASE STATE, and the user’s hidden SCENARIO \-\- plus theagent’s own full DOMAIN POLICY and TOOL SCHEMA \(appended to these instructions\)\. Use all of it to judge\.NOTE ON REASONING: the agent’s message may include a PRIVATE chain\-of\-thought block that is NEVER shown tothe user\. Judge ONLY the user\-visible action \(the text after that reasoning block, plus any tool call\); donot penalize the private reasoning, and do not treat it as a message sent to the user\.For the single step the agent just took, report TWO things:1\. CRITICALITY \-\- Using the gold reference actions and reward criteria you were given: a step is CRITICALonly if it directly performs, or is a necessary precondition for, a GOLD action / database write /assertion that the reward actually checks\. Lookups, greetings, confirmations, rephrasings, and any stepwhose result the reward does not depend on are NOT critical, even if helpful\. Answer YES or NO\.2\. CORRECTNESS \-\- Judge the step against the gold reference actions, the reward criteria, and the agent’sdomain policy\. Call it a genuine MISSTEP ONLY IF it diverges from what the gold reference and the policyrequire at THIS point in a way that affects the outcome: it acts on the wrong entity, skips averification/precondition the policy mandates, performs an incorrect or irreversible write, orcontradicts the policy or what the user actually said\. If the step is consistent with the gold trajectoryand the policy \-\- or is a harmless, recoverable, or cosmetic variation \-\- it is a PASS\. Default HARD toPASS: reserve a misstep verdict for the FEW genuinely outcome\-affecting errors; when in doubt, PASS\.Judge CRITICALITY independently of correctness\.For a misstep: state in ONE sentence what was wrong, then "Instead: " the concrete correct action\. For aCRITICAL PASS: state in ONE sentence why it was the right step\.\#\# Output formatEmit EXACTLY these three tags and nothing else\. Do not think out loud\.<critical\>YES or NO</critical\><decision\>PASS or MISSTEP</decision\><analysis\>For a MISSTEP: one sentence on what was wrong, then "Instead: " and the concrete counterfactual action asa NATIVE tool call \(or the exact user instruction for a user\-device tool\)\. For a CRITICAL PASS: onesentence on why this was the correct step\. For a non\-critical PASS: may be a brief note or empty\.</analysis\>\#\# Write the analysis in FIRST PERSON, with a LITERAL counterfactualWrite the one\-sentence REASON as the agent reflecting on its own step, using "I"/"my" \(e\.g\. "I skipped therequired verification before updating the reservation"\) \-\- never "the agent"\. After "Instead: " output ONLYthe LITERAL next action itself \-\- the exact text that would be emitted \-\- NOT a description of it\.Figure 3:τ2\\tau^\{2\}\-Bench expert annotation prompt: system message\. The per\-domain note of Figure[4](https://arxiv.org/html/2610.19367#A2.F4)is inserted immediately before item 1, and the agent’s full domain policy and tool schema are appended to it verbatim\.τ2\\tau^\{2\}\-Bench domain notes — inserted before item 1 \(one per domain\)\[airline\] This is the AIRLINE domain\. The agent’s database writes are reservation actions \-\- e\.g\.book\_reservation, cancel\_reservation, update\_reservation\_flights, update\_reservation\_passengers,update\_reservation\_baggage, send\_certificate\. Every tool is AGENT\-callable; there are NO user\-device tools,so the agent should never tell the user to operate a tool themselves\. Watch especially for acting on thewrong reservation / flight / passenger, skipping the required confirmation before a paid change orcancellation, or violating the cabin / baggage / change rules in the policy\.\[retail\] This is the RETAIL domain\. The agent’s database writes mutate orders \-\- cancel\_pending\_order,modify\_pending\_order\_items / \_address / \_payment, return\_delivered\_order\_items,exchange\_delivered\_order\_items\. Every tool is AGENT\-callable; there are NO user\-device tools, so the agentshould never tell the user to operate a tool themselves\. Watch especially for acting on the wrong order /item, exchanging to an unavailable or different\-product variant, skipping the required user confirmationbefore a write, or acting on an order in the wrong status \(an order must be pending to cancel/modify,delivered to return/exchange\)\.Figure 4:τ2\\tau^\{2\}\-Bench domain notes, one per domain, inserted into the expert system message of Figure[3](https://arxiv.org/html/2610.19367#A2.F3)\.In the prompt, we use a first\-person directive asking the expert to generate the critique and corrected action so thata~t\\tilde\{a\}\_\{t\}is directly usable as a supervised target\. We adopt this as a practical design choice and do not ablate its specific impact on CAFT performance\. In the expert prompt, we additionally ask the expert to label the criticality of the step – which we find is important to prevent it from overzealously labeling every step as incorrect\. For our final CAFT runs, we only keep correct and incorrect steps that are critical\.

#### Per\-step context template\.

Each annotation query is completed with the user message of Figure[5](https://arxiv.org/html/2610.19367#A2.F5)\. When privileged information is withheld \(Section[3\.3](https://arxiv.org/html/2610.19367#S3.SS3)\), only the<conversation\_so\_far\>and<step\_the\_agent\_just\_took\>blocks are kept\. We remove every reference to the gold actions, reward criteria and scenario from the system prompt \(Appendix[B\.3](https://arxiv.org/html/2610.19367#A2.SS3)\)\.

Expert annotation prompt — per\-step user message<task\_description\>\{\.\.\.\}</task\_description\><user\_scenario\>\{\.\.\.\}</user\_scenario\><gold\_reference\_and\_reward\_criteria\>\{\.\.\.\}</gold\_reference\_and\_reward\_criteria\><conversation\_so\_far\>\{\.\.\.\}</conversation\_so\_far\><step\_the\_agent\_just\_took\>\{\.\.\.\}</step\_the\_agent\_just\_took\>Review the step above and emit EXACTLY <critical\>YES\|NO</critical\>,<decision\>PASS\|MISSTEP</decision\>, and <analysis\>\.\.\.</analysis\> asspecified \-\- nothing else\.Figure 5:τ2\\tau^\{2\}\-Bench expert annotation prompt: per\-step user message\.Under the no\-privilege condition only the<conversation\_so\_far\>and<step\_the\_agent\_just\_took\>blocks are kept\.

### B\.2WebShop expert annotation prompt

On WebShop the expert reviews each proposedsearch\[…\]orclick\[…\]action given the shopping instruction, the privileged goal specification \(namely the target product id, product type, required attributes and price limit\), the clickable buttons on the agent’s current page, and the episode so far\. The expert system message is given in Figure[6](https://arxiv.org/html/2610.19367#A2.F6)and the per\-step user message in Figure[7](https://arxiv.org/html/2610.19367#A2.F7)\. The student\-side system prompt used by our WebShop harness is given in Figure[8](https://arxiv.org/html/2610.19367#A2.F8)\. The CAFT system addendum of Appendix[B\.4](https://arxiv.org/html/2610.19367#A2.SS4)is appended to it during fine\-tuning\.

WebShop expert annotation prompt — system messageYou are a COACH reviewing a WebShop shopping AGENT before its next action executes\. The agent interacts witha text store through two actions:\- search\[<query\>\] \- keyword search; works from ANY page \(no need to click "Back to Search" first\)\.\- click\[<button\>\] \- click a button CURRENTLY on the page: a product id, an option \(color/size/\.\.\.\), "BuyNow", "Next \>", "< Prev", "Back to Search", "Description", "Features", "Reviews"\.Search returns the TOP\-50 results \(10 per page, up to 5 pages; "Next \>" pages forward\)\. The item thatsatisfies the instruction is almost always somewhere in those 50 if the query names the specifics\. Tosucceed the agent must, IN ORDER: \(1\) issue a SPECIFIC query naming the product TYPE \+ key ATTRIBUTES \+ therequired color/size; \(2\) SCAN the results \(page with "Next \>"\) for the product that actually OFFERS therequired variant, not just a look\-alike; \(3\) open that product; \(4\) select EVERY required OPTION; \(5\) click"Buy Now" within the price limit\. Searching and paging are FREE and expected\.You have privileged access the agent does NOT have:\- The target product and its required ATTRIBUTES, required OPTIONS \(color/size/\.\.\.\), price limit, and type\.\- The clickable BUTTONS on the agent’s CURRENT page, and the queries/pages already seen in the history\.\#\# BLOCK ONLY these, and always give a CONSTRUCTIVE FORWARD fix \(never a bare "Back to Search"\):S1\. search\[\.\.\.\] that is VAGUER than the instruction \(drops the required color/size or a key attribute\)when a more specific query is possible\.\-\> Instead: Action: search\[<product type \+ key attributes \+ required color \+ required size\>\]S2\. search\[\.\.\.\] that just REPEATS a query already tried this session \(it returns the same results\)\.\-\> If this page has "Next \>": Action: click\[Next \>\]\. Else:Action: search\[<reworded query that adds a distinguishingattribute or the exact color/size token\>\]O1\. click\[<product id\>\] opening a look\-alike whose title/type clearly cannot match, when aclearly\-matching listing is visible on THIS results page\. \-\> Instead: Action: click\[<the matchingproduct id here\>\]P1\. click\[Buy Now\] \(or an option\) on an opened product that does NOT offer the required color/sizevariant\. Do NOT go back blindly:\-\> If this page has "Next \>" \(results remain unscanned\): Action: click\[Next \>\]\-\> Else: Action: search\[<full instruction incl\. the exact required color and size\>\]P2\. click\[Buy Now\] on a product whose TYPE/ATTRIBUTES clearly don’t match the instruction, or over theprice limit\. \-\> Instead:Action: click\[Next \>\] \(if results remain\) or search\[<morespecific query\>\]\.B1\. click\[Buy Now\] with a required OPTION still unselected AND that option’s correct value is a buttonhere\.\-\> Instead: Action: click\[<the exact correct option value\>\]B2\. click\[<option\>\] selecting the WRONG variant \(a color/size contradicting a required option\) when thecorrect value is a button here\. \-\> Instead: Action: click\[<the correct option value\>\]N1\. An action that is NOT among THIS page’s buttons at all \(a guaranteed no\-op\)\.\-\> Instead: Action: <a valid forward button on this page\>\#\# ALWAYS PASS \(never BLOCK these\):\- A FIRST, specific query; opening a plausibly\-matching product to inspect it; selecting a CORRECT option;click\[Buy Now\] once every required option is selected on a matching product within price; and click\[Next\>\] while unscanned pages remain\. When unsure, PASS\.\#\# The fix must be a REAL, PRESENT, FORWARD action:\- The fix target of a click MUST be copied verbatim from<current\_page\_buttons\>\. NEVER emit \`Action: click\[search\]\` and NEVERemit \`Action: click\[Back to Search\]\` as a fix \-\- to search again, emit \`Action: search\[\.\.\.\]\` directly \(itworks from any page\)\.\- Prefer, in order: select a required option here \-\> open the matching product here \-\> click\[Next \>\] to scanmore \-\> search\[<more specific query\>\]\. Only re\-search when nothing on this page moves toward the match\.\#\# No privileged framing \- write ONLY what the agent could know from the screen\- NEVER use "gold", "target", "the correct answer", "the system knows"; never call a product id "thetarget"\.\- Justify from the OBSERVABLE instruction, e\.g\. "I searched without the requested color, so I only seelook\-alikes", "this product doesn’t offer the requested size and I haven’t scanned the remaining results","I clicked Buy Now before selecting the requested color"\. Refer to requirements as the INSTRUCTION states\.\#\# First person \+ literal action\- The critique is the AGENT talking about ITSELF \("I \.\.\.", never "the agent"\), ONE sentence\.\- After "Instead: " output ONLY the LITERAL next action: \`Action: search\[\.\.\.\]\`, \`Action: click\[Next \>\]\`,\`Action: click\[xx\-large\]\`, \`Action: click\[B09KLQLLT2\]\`\. NEVER intent/meta phrasing\.\#\# Output formatEmit EXACTLY these tags and nothing else\.For PASS:<decision\>PASS</decision\>For BLOCK:<decision\>BLOCK</decision\><analysis\>One first\-person sentence on what I did wrong\. Instead:Action: <the literal corrected action\></analysis\>Figure 6:WebShop expert annotation prompt: system message\.The expert sees the shopping instruction, the privileged goal specification, the buttons on the agent’s current page, and the episode so far\.WebShop expert annotation prompt — per\-step user message<instruction\>\{instruction\}</instruction\><gold\>\{gold\}</gold\><current\_page\_buttons\>\{buttons\}</current\_page\_buttons\><history\>\{history\}</history\><proposed\_action\>\{action\}</proposed\_action\>Review the proposed\_action\. Emit <decision\>PASS</decision\> to let it through, or <decision\>BLOCK</decision\>plus a first\-person <analysis\> ending in "Instead: Action: \.\.\." \(the literal corrected action\)\.Figure 7:WebShop expert annotation prompt: per\-step user message\.WebShop student system prompt \(ReAct harness\)You are an autonomous shopping agent operating a text\-based web store \(WebShop\)\. Your job: buy the ONEproduct that best matches the given Instruction\.Each turn, first reason about the current observation, then end your reply with EXACTLY ONE action line:Action: search\[<query\>\] \- search the store; works from ANY page\(you do NOT need to click "Back to Search" first\)Action: click\[<button\>\] \- click a button shown in the observationbetween \[button\] and \[button\_\] \(product ids like B08KBVJ4XN, optionslike "white"/"x\-large", "Buy Now", "Description", "Next \>", "Back to Search"\)Guidelines:\- ONE ACTION PER TURN: after your reasoning, output a SINGLE "Action: \.\.\." line and then STOP\. Do NOT emitmultiple "Action:" lines and do NOT write your own "Observation:" lines \- only your FIRST action isexecuted and everything after it is ignored \(so a plan written as several actions loses all but thefirst\)\.\- Copy the button text EXACTLY as shown between \[button\] and \[button\_\]\.\- Normal flow: search\[\.\.\.\] \-\> click a product id \-\> click EACH required option \(color/size/scent/\.\.\.\) \-\>click\[Buy Now\]\.\- Satisfy the WHOLE Instruction: every attribute, every option, and the price limit \(price must be <= thestated dollars\)\. Prefer the cheapest item that satisfies everything\.\- If the system says your action had no effect, you are on the wrong page or used a wrong label: click theproduct id again to return to the item page, or "Back to Search" to search again\.\- Make progress each turn; click Buy Now once your chosen item is configured to match\.Example:Observation:Instruction:i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50\.00dollars\[button\] Search \[button\_\]Action: search\[3 ounce bright citrus deodorant sensitive skin\]Observation:\[button\] Back to Search \[button\_\]Page 1 \(Total results: 50\)\[button\] B078GWRC1J \[button\_\]Bright Citrus Deodorant by Earth Mama, 3\-Ounce$10\.99\[button\] B08KBVJ4XN \[button\_\]Barrel and Oak Aluminum\-Free Deodorant, 2\.7 oz$15\.95Action: click\[B078GWRC1J\]Observation:\[button\] Back to Search \[button\_\]scent \[button\] bright citrus \[button\_\] \[button\] ginger fresh \[button\_\]size \[button\] travel set \(4\-pack\) \[button\_\] \[button\] 3 ounce \(pack of 1\) \[button\_\]Price: $10\.99\[button\] Buy Now \[button\_\]Action: click\[bright citrus\]Observation: You have clicked bright citrus\.Action: click\[3 ounce \(pack of 1\)\]Observation: You have clicked 3 ounce \(pack of 1\)\.Action: click\[Buy Now\]Figure 8:WebShop student system promptused by our ReAct harness\. The CAFT system addendum of Figure[11](https://arxiv.org/html/2610.19367#A2.F11)is appended to it during fine\-tuning\.
### B\.3Prompt\-ablation variants

The ablation of Section[3\.3](https://arxiv.org/html/2610.19367#S3.SS3)varies two axes independently onτ2\\tau^\{2\}AR\.*Privileged information*controls whether the expert receives the gold reference actions, reward criteria and hidden user scenario\. When they are withheld, the opening paragraph of the system prompt is replaced by the one in Figure[9](https://arxiv.org/html/2610.19367#A2.F9)and the correctness and criticality clauses drop their references to the gold reference and reward\.*Critique guidance*controls the elaborateness of the domain note and the criticality and correctness clauses\. The detailed guidance version uses the text of Figure[3](https://arxiv.org/html/2610.19367#A2.F3), while the “minimal” condition drops the domain note and replaces both clauses with the bare questions of Figure[10](https://arxiv.org/html/2610.19367#A2.F10), keeping the output format and the first\-person directive identical\.

No\-privilege opening paragraph — replaces the first two paragraphs of the system promptYou are an expert REVIEWER of a customer\-service AGENT\. You review EVERY step the agent takes \-\- a read\-onlylookup, a database write, or a message to the user \-\- AFTER it has already executed\. You never block, undo,or change anything; you only record a short verdict on the single step taken\.You are NOT given any privileged information: you do NOT see a gold reference, the reward’s evaluationcriteria, the user’s hidden scenario, or the database state\. Judge using ONLY the conversation so far andthe agent’s own DOMAIN POLICY and TOOL SCHEMA \(appended to these instructions\) \-\- exactly the informationthe agent itself has when it acts\.Figure 9:No\-privilege opening paragraph, which replaces the first two paragraphs of the expert system message of Figure[3](https://arxiv.org/html/2610.19367#A2.F3)\.Minimal critique guidance — replaces criticality and correctness clauses 1 and 21\. CRITICALITY \-\- Does this step matter for whether the task ultimately succeeds? Answer YES or NO\.2\. CORRECTNESS \-\- Was the step a mistake? If it was a genuine error, say MISSTEP; otherwise say PASS\. For amisstep: state in ONE sentence what was wrong, then "Instead: " the concrete correct action\. For aCRITICAL PASS: state in ONE sentence why it was the right step\.Figure 10:Minimal critique guidance, which replaces criticality and correctness clauses 1 and 2 of Figure[3](https://arxiv.org/html/2610.19367#A2.F3); the output format and the first\-person directive are unchanged\.All four conditions re\-annotate the*same*student trajectories with the same expert model, so the comparison isolates the annotation prompt\.

### B\.4Interjectionρ\\rhoand ablation variants

The interjectionρ\\rhois a fixed user\-role message inserted after an expert\-flagged action\. The student’s system prompt additionally carries a short addendum disclosing that a coach is reviewing its actions, so that the interjection is in\-distribution during SFT training\. We crucially note that both these interjections are only used at*training*time\. During evaluation, we never use the expert critique to interject in any of its generations\. CAFT and its two ablations differ only in these two strings \(Figures[11](https://arxiv.org/html/2610.19367#A2.F11)–[13](https://arxiv.org/html/2610.19367#A2.F13)\) and in the corresponding targetyty\_\{t\}\.

CAFT \(full\) — system addendum and interjectionρ\\rho\[system addendum\]A privileged coach silently reviews each action you take\. If it judges a step you just took to be a mistake,your NEXT turn will be prefixed with \[COACH\-FLAG\]\. When you see \[COACH\-FLAG\], reconsider: work out what waswrong with the action you just took and what the correct step is\.\[interjection\]\[COACH\-FLAG\] The coach flagged the action you just took as a mistake\. Explain what specifically was wrongwith that action, then take the correct action instead\.Figure 11:CAFT \(full\): student system addendum and interjectionρ\\rho\.CAFT\-Correction — system addendum and interjectionρ\\rho\[system addendum\]A privileged coach silently reviews each action you take\. If it judges a step you just took to be a mistake,your NEXT turn will be prefixed with \[COACH\-FLAG\]\. When you see \[COACH\-FLAG\], simply take the correct actioninstead\.\[interjection\]\[COACH\-FLAG\] The coach flagged the action you just took as a mistake\. Take the correct action instead\.Figure 12:CAFT\-Correction: student system addendum and interjectionρ\\rho\.CAFT\-Critique — system addendum and interjectionρ\\rho\[system addendum\]A privileged coach silently reviews each action you take\. If it judges a step you just took to be a mistake,your NEXT turn will be prefixed with \[COACH\-FLAG\]\. When you see \[COACH\-FLAG\], state in your response whatspecifically was wrong with the action you just took\.\[interjection\]\[COACH\-FLAG\] The coach flagged the action you just took as a mistake\. Explain whatspecifically was wrong with that action\.Figure 13:CAFT\-Critique: student system addendum and interjectionρ\\rho\.

## Appendix CData Collection and Supervised Fine\-Tuning Details

### C\.1Trajectory collection and annotation

For each student and environment we roll out the base policy once over the training task set with1616rollouts per task, recording every agent turn together with its expert annotation\.

All models are fully fine\-tuned in bf16 with FSDP on a single88\-GPU node \(NVIDIA RTX PRO 6000,9696GB\), using AdamW, gradient clipping at1\.01\.0, and right truncation\. For all CAFT, ablation, and action distillation experiments, we use a batch size of 64 and a learning rate of3×10−63\\times 10^\{\-6\}\. We train for 2 epochs onτ2\\tau^\{2\}AR and 1 epoch on WebShop\. For expert distillation and CAFT \+BC mixtures onτ2\\tau^\{2\}AR, we increase the learning rate to1×10−51\\times 10^\{\-5\}for 2 epochs\. Iteration\-2 CAFT retains the same hyperparameters of the first iteration\.

The CAFT \+BC mixtures of Section[3\.4](https://arxiv.org/html/2610.19367#S3.SS4)are expert model compute\-matched\. Each mixture contains a fixed total of16641664trajectory\-equivalents, split between expert trajectories and CAFT examples according to the stated ratio\.

## Appendix DReinforcement Learning Details

We use GRPO as implemented inverl\([Sheng et al\., 2025](https://arxiv.org/html/2610.19367#bib.bib38)\)\. The policy self\-serves through vLLM on the training GPUs, while the user simulator and outcome judge run on an external Gemma 4 31B endpoint\. Each RL run occupies one88\-GPU node\. The reward is the binaryτ2\\tau^\{2\}outcome grader applied at the end of each episode\. Table[5](https://arxiv.org/html/2610.19367#A4.T5)lists the hyperparameter settings\. Only the initialization differs across the three runs of Section[3\.6](https://arxiv.org/html/2610.19367#S3.SS6)\.

Table 5:GRPO hyperparameters for theτ2\\tau^\{2\}AR warm\-start experiments using Nemotron 3 Nano 4B\.
## Appendix EAdditional Results

### E\.1How much should correct turns be supervised?

Section[2](https://arxiv.org/html/2610.19367#S2)retains expert\-approved turns as ordinary behavior\-cloning targets, weighted by a scalarλ\\lambdarelative to the corrective turns \(we useλ=0\.1\\lambda=0\.1throughout, originally chosen from a sweep on a small training subset\)\. In this section we evaluate a sweep ofλ∈\{0\.01,0\.1,0\.5\}\\lambda\\in\\\{0\.01,0\.1,0\.5\\\}on theτ2\\tau^\{2\}AR validation data for Gemma 4 E2B and Nemotron 3 Nano 4B\.

Table[6](https://arxiv.org/html/2610.19367#A5.T6)shows thatλ=0\.1\\lambda=0\.1is best for both students\. Upweighting correct turns toλ=0\.5\\lambda=0\.5costs7\.27\.2% and5\.65\.6% for Gemma 4 E2B and Nemotron 3 Nano 4B, respectively, and nearly removing them \(λ=0\.01\\lambda=0\.01\) costs0\.80\.8% and7\.37\.3%\. We believe that atλ=0\.5\\lambda=0\.5the objective is dominated by imitating the student’s own executed actions, rather than on the critiques of incorect steps\. However, removing the correct turns altogether is even more damaging\. Masking them out entirely costs7\.17\.1% for Nemotron 3 Nano 4B \(53\.2%53\.2\\%versus60\.3%60\.3\\%for a matchedλ=0\.1\\lambda=0\.1run of the same configuration\)\. Without any correct\-turn loss the model sees only mistake\-conditioned targets and degrades on the ordinary, correct turns that make up the large majority of a trajectory\.

Table 6:τ2\\tau^\{2\}AR task success \(%\) as a function of the correct\-turn loss weightλ\\lambda\(CAFT usesλ=0\.1\\lambda=0\.1everywhere else in the paper\)\. All arms share the same student trajectories and expert annotations and differ only inλ\\lambda\.
### E\.2Mixing ratio for CAFT \+BC

Section[3\.4](https://arxiv.org/html/2610.19367#S3.SS4)reports a50/5050/50mixture of expert trajectories and CAFT supervision\. Table[7](https://arxiv.org/html/2610.19367#A5.T7)sweeps the ratio at matched total training compute\. An equal mix is best for both Gemma students, while Nemotron—the only student for which CAFT alone already beats expert distillation—prefers the mixture that contains the*most*expert critique data\. This is consistent with expert trajectories being complementary only when behavior cloning on them is itself effective for the student

Table 7:τ2\\tau^\{2\}AR task success \(%\) as a function of the expert\-trajectory fraction in CAFT \+BC\. The0%0\\%column is pure CAFT and the reference for the sweep\.
### E\.3Do CAFT\-trained agents receive less intervention from the expert model?

Table 8:Expert Interventions\.Task success \(%\) when the expert corrects flagged actions at inference time, alongside the average number of expert corrections per episode \(Corr\./ep\.\)\.To provide further evidence that CAFT effectively teaches the agent to avoid previously identified mistakes, we run an additional experiment where we take the CAFT\-trained model checkpoints and measure the frequency of expert interventions that would take place if the expert was available at inference time\. We hypothesize that if the student policy successfully internalizes the textual critiques provided during training, it should naturally learn to avoid those same errors that the expert model typically flags and thus trigger fewer corrections from the experts than the base untrained model\.

We test this hypothesis by conducting a coach\-in\-the\-loop rollout on the validation split\. During this evaluation, the expert model reviews every action taken by the student and replaces it with a corrected action whenever a mistake is flagged\. We track both the resulting corrected task success rate and the total number of interventions required\.

As shown in Table[8](https://arxiv.org/html/2610.19367#A5.T8), CAFT checkpoints require substantially fewer interventions compared to their corresponding base policies across both environments\. Onτ2\\tau^\{2\}AR, this reduction in necessary corrections corresponds to a higher corrected success rate\. The CAFT \+BC checkpoints require the fewest interventions overall, though their corrected success rates indicate that the few remaining errors they make are more difficult to recover from\. On WebShop, the CAFT policies similarly require far fewer corrections than the base models\. Although similarly the actual success rate with intervention drops – suggesting that while CAFT adapted to prevent the errors the expert critique does catch, the errors not caught by the expert become more fatal\. Ultimately, this drop in required expert interventions from the base model to the CAFT trained version supports the conclusion that CAFT agents successfully internalize the critique feedback\.