AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories

# AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories
Source: [https://arxiv.org/html/2609.05837](https://arxiv.org/html/2609.05837)
Yewen LiLongtao ZhengShengtian YangAffiliation:Nanyang Technological University Kuaishou Technology Southeast UniversityLang FengLei FengAffiliation:Nanyang Technological University Kuaishou Technology Southeast UniversityPeng JiangKun GaiQingpeng CaiBo An

###### Abstract

LLM\-based agents are increasingly deployed in real\-world applications through tool\-use APIs, yet training them for specific environments remains fundamentally difficult: real\-world applications provide no pre\-defined tasks or verifiers, no faithful simulators, and limited budget for large\-scale environment interaction\. In this paper, we proposeAgentBrew, an offline training framework that learns effective tool\-use policies from a single batch of raw interaction trajectories, without task verifiers or iterative on\-policy rollouts\. The agent first explores the target environment to collect a raw trajectory corpus without quality filtering\. To extract training signal from this noisy corpus,*retrospective task inference*reconstructs an aligned instruction for each trajectory based on its actual outcome, and*PMI\-Based credit assignment*decomposes the trajectory’s total information about the inferred instruction into additive per\-action credits via pointwise mutual information \(PMI\)\. These credits weight the policy training objective, amplifying informative actions while suppressing ineffective ones\. On three real\-world MCP applications \(GitHub, Notion, PostgreSQL\), AgentBrew improves Qwen3\-32B by \+8\.7 Acc / \+9\.7 Score on average, surpassing Qwen3\-235B \(\+2\.3 / \+4\.4\) and outperforming rejection sampling \(\+5\.9 / \+10\.3\)\. These results demonstrate that fine\-grained offline learning can recover useful supervision from raw trajectories that filtering\-based approaches would discard\. The code is available at[https://github\.com/alphatogo/AgentBrew](https://github.com/alphatogo/AgentBrew)

## 1Introduction

Tool\-use agents powered by Large Language Models \(LLMs\) are rapidly moving from research prototypes to production deployments\[[44](https://arxiv.org/html/2609.05837#bib.bib44);[21](https://arxiv.org/html/2609.05837#bib.bib21);[29](https://arxiv.org/html/2609.05837#bib.bib29);[2](https://arxiv.org/html/2609.05837#bib.bib2)\]\. Standardized protocols like the Model Context Protocol \(MCP\)\[[2](https://arxiv.org/html/2609.05837#bib.bib2)\]now connect LLM agents to real\-world applications such as Notion\[[22](https://arxiv.org/html/2609.05837#bib.bib22)\]and GitHub\[[10](https://arxiv.org/html/2609.05837#bib.bib10)\], enabling tool\-use across complex, stateful environments\. However, each application presents a distinct tool landscape: Notion exposes dozens of APIs spanning database operations, page management, and content editing, each requiring precise parameter composition and awareness of the application’s live state\. General\-purpose LLMs, even at frontier scale, struggle with such environment\-specific complexity, exhibiting low task completion rates on real\-world tool\-use benchmarks\[[20](https://arxiv.org/html/2609.05837#bib.bib20);[39](https://arxiv.org/html/2609.05837#bib.bib39)\], indicating that reliable deployment demands targeted, environment\-specific training\.

Existing approaches to training tool\-use agents for specific applications follow two main routes\. The first operates on real\-world data\[[42](https://arxiv.org/html/2609.05837#bib.bib42);[48](https://arxiv.org/html/2609.05837#bib.bib48);[31](https://arxiv.org/html/2609.05837#bib.bib31)\]: practitioners design a suite of tasks with ground\-truth verifiers for the target application, deploy an agent to interact with the live environment and collect trajectories, then use the verifiers to filter for successful completions and train via supervised fine\-tuning on the retained high\-quality subset\[[26](https://arxiv.org/html/2609.05837#bib.bib26);[48](https://arxiv.org/html/2609.05837#bib.bib48);[24](https://arxiv.org/html/2609.05837#bib.bib24)\]\. The second pursues on\-policy optimization\[[30](https://arxiv.org/html/2609.05837#bib.bib30);[9](https://arxiv.org/html/2609.05837#bib.bib9);[13](https://arxiv.org/html/2609.05837#bib.bib13)\]: because repeated rollouts against live APIs incur monetary costs, rate limits, and irreversible side effects on production data, these methods instead construct simulated environments that replicate the application’s state and tool behavior, enabling large\-scale iterative training before deploying the resulting policy back to the real application\[[6](https://arxiv.org/html/2609.05837#bib.bib6);[38](https://arxiv.org/html/2609.05837#bib.bib38);[35](https://arxiv.org/html/2609.05837#bib.bib35);[28](https://arxiv.org/html/2609.05837#bib.bib28)\]\. Despite their different trade\-offs, both routes rest on shared prerequisites: pre\-defined task suites, reliable reward signals, and either sufficient real\-world interaction budget or a faithful simulator of the target environment\.

![Refer to caption](https://arxiv.org/html/2609.05837v1/insight.png)Figure 1:Left: trajectory\-level filtering discards entire trajectories that fail the original task, losing potentially useful actions\. Right: AgentBrew infers a better\-aligned task for each trajectory and assigns per\-action credits under the inferred task, enabling the entire corpus to contribute to training\.While these methods have achieved strong results on curated benchmarks, applying them to real\-world applications exposes fundamental difficulties at three levels\. At thetask level, real\-world applications provide neither pre\-defined task nor ground\-truth verifiers\. Tasks must be constructed from scratch with validity dependent on the environment’s live state, and reliable verification remains an open problem\[[12](https://arxiv.org/html/2609.05837#bib.bib12)\]: neither manual annotation\[[7](https://arxiv.org/html/2609.05837#bib.bib7);[28](https://arxiv.org/html/2609.05837#bib.bib28)\]nor automated judging scales to complex, stateful API interactions \(see §[4\.2](https://arxiv.org/html/2609.05837#S4.SS2)\)\. At thetrajectory level, the offline route faces a severe data efficiency problem\. Real\-world tool\-use tasks involve long, complex API compositions with low completion rates, so standard trajectory\-level filtering\[[42](https://arxiv.org/html/2609.05837#bib.bib42);[48](https://arxiv.org/html/2609.05837#bib.bib48)\]discards the majority of an already limited corpus\. Worse, even a ’failed’ trajectory may contain actions that are useful training signal for a different but related task, yet trajectory\-level filtering discards them entirely \(Figure[1](https://arxiv.org/html/2609.05837#S1.F1), left\)\. At thetraining level, the online route introduces a significant sim\-to\-real gap: simulators cannot faithfully reproduce the complex, interconnected state of real\-world applications\. Recent work confirms this empirically: simulator\-trained agents achieve strong results on synthetic benchmarks yet degrade when evaluated on real\-world platforms\[[6](https://arxiv.org/html/2609.05837#bib.bib6)\]\. Together, these challenges rule out standard training pipelines and point to a setting that, despite being the norm for industrial deployment, remains largely unexplored in the research community:

How can we train a tool\-use agent for a specific application, with no pre\-defined tasks, no verifiers, no simulator, and limited environment interaction?

In this paper, we presentAgentBrew, an offline training framework that learns effective tool\-use policies from a single batch of raw interaction trajectories, without task verifiers or iterative rollouts\. Our key insight is that every trajectory, regardless of whether it completes its original task, carries reusable signal: by inferring what each trajectory actually accomplished and scoring which actions genuinely contributed, we extract fine\-grained supervision from the entire raw corpus without discarding any data\.First, the agent explores the target environment to discover its live state and proposes grounded task instructions, then executes these tasks to collect a raw trajectory corpus without any quality filtering \(§[3\.1](https://arxiv.org/html/2609.05837#S3.SS1)\)\.Second, we recover fine\-grained supervision from this noisy corpus through two complementary mechanisms \(§[3\.2](https://arxiv.org/html/2609.05837#S3.SS2)\)\. For each trajectory,*retrospective task inference*reconstructs an instruction that reflects what the trajectory actually achieved rather than what it originally intended, producing aligned instruction\-trajectory pairs\.*PMI\-Based credit assignment*then quantifies how much each action contributes to the inferred task\. We define the total information a trajectory carries about its inferred instruction via pointwise mutual information \(PMI\), and leverage the chain rule of PMI to decompose this trajectory\-level quantity into additive per\-action credits, each measuring the marginal information a single action contributes\.Third, these credits serve as sample weights for policy training, amplifying the gradient from informative actions while suppressing that from ineffective ones \(§[3\.3](https://arxiv.org/html/2609.05837#S3.SS3)\)\. Crucially, once the raw corpus is gathered, all subsequent stages proceed entirely offline with no further environment interaction, making the framework practical for real\-world APIs where iterative exploration is infeasible\.

We evaluate AgentBrew on three real\-world MCP applications: GitHub, Notion, and PostgreSQL\[[20](https://arxiv.org/html/2609.05837#bib.bib20);[39](https://arxiv.org/html/2609.05837#bib.bib39)\]\. Without any external reward signals, human annotations, or iterative environment interaction, AgentBrew improves over the Qwen3\-32B base model by \+8\.7 Acc and \+9\.7 Score on average, surpassing Qwen3\-235B and outperforming rejection sampling by \+5\.9 Acc and \+10\.3 Score\. These results demonstrate that fine\-grained offline learning can extract useful training signal from raw trajectories that filtering\-based approaches would discard entirely\.

## 2Problem Formulation

We consider an LLM agent operating within a real\-world application environmentℰ\\mathcal\{E\}\(e\.g\., GitHub, Notion\) that exposes a set of callable toolsℱ=\{f1,…,fK\}\\mathcal\{F\}=\\\{f\_\{1\},\\ldots,f\_\{K\}\\\}, each defined by a natural language description and a typed parameter specification\. Given a task instructionI∈ℐI\\in\\mathcal\{I\}, the agent interacts withℰ\\mathcal\{E\}over discrete stepst=0,…,Tt=0,\\ldots,T\. At each step, the agent observes the statest=\(I,a0,o0,…,at−1,ot−1\)s\_\{t\}=\(I,a\_\{0\},o\_\{0\},\\ldots,a\_\{t\-1\},o\_\{t\-1\}\)and generates an actionat=\(ut,ct\)a\_\{t\}=\(u\_\{t\},c\_\{t\}\), whereutu\_\{t\}is a chain\-of\-thought trace andctc\_\{t\}is a tool call\. The environment returns an observationot=ℰ⁡\(ct\)o\_\{t\}=\\mathcal\{E\}\(c\_\{t\}\), yielding a complete trajectoryτ=\{\(st,at,ot\)\}t=0T\\tau=\\\{\(s\_\{t\},a\_\{t\},o\_\{t\}\)\\\}\_\{t=0\}^\{T\}\.

Our goal is to improve the agent’s policyπθ\\pi\_\{\\theta\}for tool\-use tasks inℰ\\mathcal\{E\}under three constraints that reflect the practical reality of most real\-world applications \(§[1](https://arxiv.org/html/2609.05837#S1)\): \(1\) no pre\-defined tasks or verifiers are available; \(2\) the agent may collect a single trajectory corpus𝒟raw=\{\(In,τn\)\}n=1N\\mathcal\{D\}\_\{\\text\{raw\}\}=\\\{\(I\_\{n\},\\tau\_\{n\}\)\\\}\_\{n=1\}^\{N\}but further large\-scale interaction is infeasible; and \(3\) trajectories carry no external reward signals\. Each trajectory terminates when the agent issues a completion action or a maximum step limit is reached, but neither condition constitutes ground\-truth verification of success\. The objective is to extract a training signal from this unlabeled𝒟raw\\mathcal\{D\}\_\{\\text\{raw\}\}that improvesπθ\\pi\_\{\\theta\}on the distribution of user tasks inℰ\\mathcal\{E\}\.

## 3Methodology

We proposeAgentBrew, a three\-stage offline framework illustrated in Figure[2](https://arxiv.org/html/2609.05837#S3.F2)\. Given a target environmentℰ\\mathcal\{E\}, the agent first collects a raw trajectory corpus𝒟raw\\mathcal\{D\}\_\{\\text\{raw\}\}through grounded exploration \(§[3\.1](https://arxiv.org/html/2609.05837#S3.SS1)\)\. The raw corpus then undergoes experience distillation \(§[3\.2](https://arxiv.org/html/2609.05837#S3.SS2)\), which produces realigned instruction\-trajectory pairs with per\-action credit scores\. Finally, these credits serve as sample weights for a maximum likelihood training objective \(§[3\.3](https://arxiv.org/html/2609.05837#S3.SS3)\)\. Once the initial corpus is gathered, the entire pipeline requires no further environment interaction\.

![Refer to caption](https://arxiv.org/html/2609.05837v1/overview.png)Figure 2:Overview of AgentBrew\. \(1\) The agent exploresℰ\\mathcal\{E\}to propose grounded tasks and collects raw trajectories\. \(2\) Retrospective task inference recovers aligned instructionsI^n\\hat\{I\}\_\{n\}, and PMI\-based credit assignment scores each action\. \(3\) Normalized credits weight the policy training objective\.### 3\.1Raw Experience Collection

##### Grounded Task Proposal\.

Prior approaches synthesize tasks directly from tool documentation\[[26](https://arxiv.org/html/2609.05837#bib.bib26);[42](https://arxiv.org/html/2609.05837#bib.bib42)\], but frequently produce tasks that reference nonexistent entities or invalid configurations because they lack access to the live state ofℰ\\mathcal\{E\}\. We instead ground task construction in the environment itself\. The agent first generates a set of high\-level task categories from the tool descriptions ofℱ\\mathcal\{F\}\(see Appendix[C](https://arxiv.org/html/2609.05837#A3)\), following the common practice of aligning synthetic training tasks with target capability profiles\[[42](https://arxiv.org/html/2609.05837#bib.bib42);[48](https://arxiv.org/html/2609.05837#bib.bib48);[34](https://arxiv.org/html/2609.05837#bib.bib34)\]\. Each category specifies the capability profile to exercise, not the specific entities or data involved\. Training and evaluation use fully isolated environment instances with no overlap in entities, data entries, or environment states\. For each category, the agent conducts exploratory interactions withℰ\\mathcal\{E\}, invoking tools to retrieve concrete environmental information such as available resources and existing data entries, and then proposes task instructions grounded in the retrieved information\. Generated tasks are validated to ensure all referenced schema entities exist in the target environment\. This yields a task setℐ=\{In\}n=1N\\mathcal\{I\}=\\\{I\_\{n\}\\\}\_\{n=1\}^\{N\}whose instructions reference entities and configurations that verifiably exist inℰ\\mathcal\{E\}\.

##### Task Execution\.

For each proposed taskInI\_\{n\}, the agent interacts withℰ\\mathcal\{E\}over multiple turns following the process described in §[2](https://arxiv.org/html/2609.05837#S2), producing a trajectoryτn=\{\(st,at,ot\)\}t=0Tn\\tau\_\{n\}=\\\{\(s\_\{t\},a\_\{t\},o\_\{t\}\)\\\}\_\{t=0\}^\{T\_\{n\}\}\. The interaction terminates when the agent returns a final answer indicating task completion, or when a maximum turn limitTmaxT\_\{\\max\}is reached\. Trajectories that reachTmaxT\_\{\\max\}are discarded as they typically reflect degenerate behaviors such as repeated failed API calls with no meaningful environmental interaction\. All remaining naturally terminated trajectories are retained without quality filtering, yielding the raw corpus𝒟raw=\{\(In,τn\)\}n=1N\\mathcal\{D\}\_\{\\text\{raw\}\}=\\\{\(I\_\{n\},\\tau\_\{n\}\)\\\}\_\{n=1\}^\{N\}\.

### 3\.2Experience Distillation

The original instructions\{In\}\\\{I\_\{n\}\\\}are necessary to drive trajectory collection, but most trajectories fail to completeInI\_\{n\}\[[20](https://arxiv.org/html/2609.05837#bib.bib20);[39](https://arxiv.org/html/2609.05837#bib.bib39);[6](https://arxiv.org/html/2609.05837#bib.bib6)\], making the raw pairs\(In,τn\)\(I\_\{n\},\\tau\_\{n\}\)unsuitable for direct training: the policy would learn to associate incorrect behaviors with the stated goal\. We address this through two complementary steps\.*Retrospective task inference*\(§[3\.2\.1](https://arxiv.org/html/2609.05837#S3.SS2.SSS1)\) replaces eachInI\_\{n\}with a revised instructionI^n\\hat\{I\}\_\{n\}that reflects what the trajectory actually accomplished, producing aligned pairs for training\.*PMI\-based credit assignment*\(§[3\.2\.2](https://arxiv.org/html/2609.05837#S3.SS2.SSS2)\) then scores each action by how much it contributes toI^n\\hat\{I\}\_\{n\}, so that informative actions receive stronger supervision while irrelevant or erroneous ones are suppressed\.

#### 3\.2\.1Retrospective Task Inference

To construct aligned pairs, we first discard all chain\-of\-thought tracesutu\_\{t\}from each trajectory, retaining only the tool calls and environment responses:

τ¯n=\{\(ct,ot\)\}t=0Tn\\bar\{\\tau\}\_\{n\}=\\\{\(c\_\{t\},o\_\{t\}\)\\\}\_\{t=0\}^\{T\_\{n\}\}\(1\)This removal is necessary becauseutu\_\{t\}is generated conditioned on the original instructionInI\_\{n\}and may contradict the trajectory’s actual outcome: for instance,utu\_\{t\}may claim a sub\-goal is being achieved whileoto\_\{t\}indicates an API error or no state change\. Retaining such traces would bias the inference toward intentions the trajectory did not fulfill\. The resultingτ¯n\\bar\{\\tau\}\_\{n\}is analogous to a user interaction log of application operations and system responses, stripped of subjective rationales\.

Givenτ¯n\\bar\{\\tau\}\_\{n\}, we prompt an LLM to infer a revised instructionI^n\\hat\{I\}\_\{n\}that is consistent with the trajectory’s observable outcomes \(prompt details in Appendix[E](https://arxiv.org/html/2609.05837#A5)\)\. The inference is guided by several key constraints:I^n\\hat\{I\}\_\{n\}must describe the outcome genuinely achieved inℰ\\mathcal\{E\}rather than narrate the action sequence; claims not directly supported by successful operations inτ¯n\\bar\{\\tau\}\_\{n\}are removed; and no entities or results beyond what is evidenced by the trajectory may be introduced\. This produces realigned pairs\(I^n,τ¯n\)\(\\hat\{I\}\_\{n\},\\bar\{\\tau\}\_\{n\}\)for downstream supervision\.

#### 3\.2\.2PMI\-Based Credit Assignment

After retrospective task inference, we obtain aligned pairs\(I^n,τ¯n\)\(\\hat\{I\}\_\{n\},\\bar\{\\tau\}\_\{n\}\)that could in principle be used directly for supervised fine\-tuning\. However, not all actions inτn\\tau\_\{n\}contribute equally toI^n\\hat\{I\}\_\{n\}: some served the original intent ofInI\_\{n\}but are irrelevant toI^n\\hat\{I\}\_\{n\}, and others are low\-quality such as incorrect API calls and redundant queries\. We therefore need a per\-action credit that quantifies each action’s contribution toI^n\\hat\{I\}\_\{n\}\. Our key idea is to use a reference language modelπref\\pi\_\{\\text\{ref\}\}to measure how much information each action\-observation pair provides about the inferred taskI^n\\hat\{I\}\_\{n\}\. This informational measure correlates well with actual contribution becauseI^n\\hat\{I\}\_\{n\}is itself derived from the trajectory’s observable outcomes \(§[3\.2\.1](https://arxiv.org/html/2609.05837#S3.SS2.SSS1)\): actions that produced the described effects leave distinctive traces inoto\_\{t\}that strongly reduce uncertainty about the outcome\-derivedI^n\\hat\{I\}\_\{n\}, receiving high credit, while failed or redundant actions provide little such evidence and are suppressed\. We formalize this using pointwise mutual information \(PMI\)\.

##### Total information of a trajectory\.

To assign per\-action credits, we first define the total information that a trajectory carries about its inferred task, then decompose it across individual actions\. Letπref\\pi\_\{\\text\{ref\}\}denote the reference language model\. The PMI between the reduced trajectoryτ¯n=\{\(ct,ot\)\}t=0Tn\\bar\{\\tau\}\_\{n\}=\\\{\(c\_\{t\},o\_\{t\}\)\\\}\_\{t=0\}^\{T\_\{n\}\}and the inferred instructionI^n\\hat\{I\}\_\{n\}is:

PMI​\(I^n,τ¯n\)=log⁡πref​\(I^n∣τ¯n\)πref​\(I^n\)=\(−log⁡πref​\(I^n\)\)⏟ℒ∅−\(−log⁡πref​\(I^n∣τ¯n\)\)⏟ℒTn=ℒ∅−ℒTn\\text\{PMI\}\(\\hat\{I\}\_\{n\};\\,\\bar\{\\tau\}\_\{n\}\)=\\log\\frac\{\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\\mid\\bar\{\\tau\}\_\{n\}\)\}\{\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\)\}=\\underbrace\{\(\-\\log\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\)\)\}\_\{\\mathcal\{L\}\_\{\\emptyset\}\}\-\\underbrace\{\(\-\\log\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\\mid\\bar\{\\tau\}\_\{n\}\)\)\}\_\{\\mathcal\{L\}\_\{T\_\{n\}\}\}=\\mathcal\{L\}\_\{\\emptyset\}\-\\mathcal\{L\}\_\{T\_\{n\}\}\(2\)whereℒ∅=−log⁡πref​\(I^n\)\\mathcal\{L\}\_\{\\emptyset\}=\-\\log\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\)is the negative log\-likelihood \(NLL\) ofI^n\\hat\{I\}\_\{n\}under the reference model without any context, andℒTn=−log⁡πref​\(I^n∣τ¯n\)\\mathcal\{L\}\_\{T\_\{n\}\}=\-\\log\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\\mid\\bar\{\\tau\}\_\{n\}\)is the NLL after conditioning on the entire trajectory\. We denote this total information asΔ​ℒ=ℒ∅−ℒTn\\Delta\\mathcal\{L\}=\\mathcal\{L\}\_\{\\emptyset\}\-\\mathcal\{L\}\_\{T\_\{n\}\}\. The question is how to decomposeΔ​ℒ\\Delta\\mathcal\{L\}across theTn\+1T\_\{n\}\+1individual actions\.

##### Per\-action credit via the chain rule of PMI\.

The chain rule of PMI decomposes the total information into a sum of conditional PMI terms, each measuring the marginal information contribution of a single action\. Letℒt=−log⁡πref​\(I^n∣c0,o0,…,ct,ot\)\\mathcal\{L\}\_\{t\}=\-\\log\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\\mid c\_\{0\},o\_\{0\},\\ldots,c\_\{t\},o\_\{t\}\)denote the NLL after observing actions up to steptt\. The conditional PMI between\(ct,ot\)\(c\_\{t\},o\_\{t\}\)andI^n\\hat\{I\}\_\{n\}given preceding actions\(c<t,o<t\)\(c\_\{<t\},o\_\{<t\}\)is:

wt=PMI\(I^n;\(ct,ot\)∣c<t,o<t\)=logπref​\(I^n∣c≤t,o≤t\)πref​\(I^n∣c<t,o<t\)=ℒt−1−ℒt,t=1,…,Tnw\_\{t\}=\\text\{PMI\}\(\\hat\{I\}\_\{n\};\\,\(c\_\{t\},o\_\{t\}\)\\mid c\_\{<t\},o\_\{<t\}\)=\\log\\frac\{\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\\mid c\_\{\\leq t\},o\_\{\\leq t\}\)\}\{\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\\mid c\_\{<t\},o\_\{<t\}\)\}=\\mathcal\{L\}\_\{t\-1\}\-\\mathcal\{L\}\_\{t\},\\quad t=1,\\ldots,T\_\{n\}\(3\)withw0=ℒ∅−ℒ0w\_\{0\}=\\mathcal\{L\}\_\{\\emptyset\}\-\\mathcal\{L\}\_\{0\}for the first action\. Eachwtw\_\{t\}quantifies how much additional information\(ct,ot\)\(c\_\{t\},o\_\{t\}\)provides aboutI^n\\hat\{I\}\_\{n\}beyond what is already known from the preceding actions\. An action that is highly informative ofI^n\\hat\{I\}\_\{n\}yields a large positivewtw\_\{t\}, a redundant action yieldswt≈0w\_\{t\}\\approx 0, and an action whose observation misleads the model away fromI^n\\hat\{I\}\_\{n\}yieldswt<0w\_\{t\}<0\.

By the chain rule of PMI, these per\-action credits form an exact additive decomposition of the total information:

∑t=0Tnwt=\(ℒ∅−ℒ0\)\+∑t=1Tn\(ℒt−1−ℒt\)=ℒ∅−ℒTn=Δ​ℒ\\sum\_\{t=0\}^\{T\_\{n\}\}w\_\{t\}=\(\\mathcal\{L\}\_\{\\emptyset\}\-\\mathcal\{L\}\_\{0\}\)\+\\sum\_\{t=1\}^\{T\_\{n\}\}\(\\mathcal\{L\}\_\{t\-1\}\-\\mathcal\{L\}\_\{t\}\)=\\mathcal\{L\}\_\{\\emptyset\}\-\\mathcal\{L\}\_\{T\_\{n\}\}=\\Delta\\mathcal\{L\}\(4\)This guarantees that the per\-action credits neither inflate nor discard any information: each unit of PMI betweenτ¯n\\bar\{\\tau\}\_\{n\}andI^n\\hat\{I\}\_\{n\}is attributed to exactly one action\.

### 3\.3PMI\-weighted Policy Training

The raw credits may be negative and vary in scale across trajectories\. We clamp negative values to zero because any instrumental value of an uncertainty\-increasing action is already captured by the subsequent step that resolves that uncertainty, as guaranteed by the additive decomposition in Eq\. \([4](https://arxiv.org/html/2609.05837#S3.E4)\)\. We then normalize within each trajectory to\[0,2\]\[0,2\]:

w~t\(n\)=2⋅max⁡\(wt\(n\),0\)max0≤j≤Tn⁡max⁡\(wj\(n\),0\)\\tilde\{w\}\_\{t\}^\{\(n\)\}=2\\cdot\\frac\{\\max\(w\_\{t\}^\{\(n\)\},\\,0\)\}\{\\max\_\{0\\leq j\\leq T\_\{n\}\}\\,\\max\(w\_\{j\}^\{\(n\)\},\\,0\)\}\(5\)The policy is then optimized by minimizing the weighted negative log\-likelihood, following the sample\-weighted SFT paradigm used in prior work\[[47](https://arxiv.org/html/2609.05837#bib.bib47);[11](https://arxiv.org/html/2609.05837#bib.bib11)\]:

𝒥\(θ\)=−∑n=1N∑t=0Tnw~t\(n\)⋅logπθ\(at∣I^n,st\)\\mathcal\{J\}\(\\theta\)=\-\\sum\_\{n=1\}^\{N\}\\sum\_\{t=0\}^\{T\_\{n\}\}\\tilde\{w\}\_\{t\}^\{\(n\)\}\\cdot\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid\\hat\{I\}\_\{n\},s\_\{t\}\)\(6\)wherest=\(I^n,a0,o0,…,at−1,ot−1\)s\_\{t\}=\(\\hat\{I\}\_\{n\},\\,a\_\{0\},o\_\{0\},\\ldots,a\_\{t\-1\},o\_\{t\-1\}\)\. Although credits are computed over\(ct,ot\)\(c\_\{t\},o\_\{t\}\), the loss is applied to the full actionat=\(ut,ct\)a\_\{t\}=\(u\_\{t\},c\_\{t\}\)including chain\-of\-thought tokens\. The credit weighting mitigates potential misalignment betweenutu\_\{t\}andI^n\\hat\{I\}\_\{n\}: low\-contribution actions receive near\-zero weight, suppressing gradient from both their tool calls and associated reasoning, while high\-credit actions tend to have reasoning consistent with their executed behavior\. Overall, informative actions receive amplified gradient signal while uninformative ones are effectively masked\. We present pseudo code and additional details in Appendix[A](https://arxiv.org/html/2609.05837#A1)&[D](https://arxiv.org/html/2609.05837#A4)\.

Table 1:Main results across three real\-world applications\. We reportAcc\(binary task completion rate\) andScore\(average per\-task score, 0–1\)\. Best results per column arebolded\.Greenandreddeltas indicate changes relative to Qwen3\-32B\.MethodGitHubNotionPostgresAverageAcc\.ScoreAcc\.ScoreAcc\.ScoreAcc\.ScoreFrontier Proprietary Models![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/gpt.png)GPT\-564\.285\.442\.983\.442\.961\.750\.076\.8![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/claude.png)Claude Sonnet\-4\.557\.176\.939\.384\.638\.153\.344\.871\.6![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/gemini.png)Gemini\-3 Pro60\.781\.253\.686\.257\.175\.157\.180\.8Open\-Source Foundation Models![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/glm.png)GLM\-5\-744B42\.974\.332\.177\.323\.859\.432\.970\.3![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/deepseek.png)DeepSeek\-V3\.2\-685B17\.863\.139\.274\.333\.364\.130\.167\.2![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/qwen.png)Qwen3\-235B17\.963\.010\.765\.719\.034\.315\.954\.3![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/gpt.png)GPT\-OSS\-120B3\.637\.83\.654\.614\.324\.17\.138\.8![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/qwen.png)Qwen3\-32B10\.749\.73\.660\.114\.337\.29\.549\.0Training Methods \(32B\)![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/toucan.png)TOUCAN17\.856\.43\.649\.214\.328\.811\.9\(\+2\.4\)44\.8\(\-4\.2\)![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/qwen.png)Vanilla SFT14\.253\.97\.159\.89\.530\.710\.3\(\+0\.8\)48\.1\(\-0\.9\)![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/Fingure/icons/qwen.png)Rejection Sampling17\.853\.9053\.219\.038\.012\.3\(\+2\.8\)48\.4\(\-0\.6\)![[Uncaptioned image]](https://arxiv.org/html/2609.05837v1/agentbrew.png)AgentBrew21\.463\.514\.270\.123\.841\.818\.2\(\+8\.7\)58\.7\(\+9\.7\)

## 4Experiments

We evaluate AgentBrew across three real\-world applications, examining overall performance \(§[4\.2](https://arxiv.org/html/2609.05837#S4.SS2)\), the effectiveness of PMI\-based credit assignment and retrospective task inference \(§[4\.3](https://arxiv.org/html/2609.05837#S4.SS3)–[4\.4](https://arxiv.org/html/2609.05837#S4.SS4)\), robustness to noisy data \(§[4\.5](https://arxiv.org/html/2609.05837#S4.SS5)\), and transferability across model sizes \(§[4\.6](https://arxiv.org/html/2609.05837#S4.SS6)\)\.

### 4\.1Experimental Settings

##### Setup\.

We evaluate on three real\-world MCP applications\[[20](https://arxiv.org/html/2609.05837#bib.bib20);[39](https://arxiv.org/html/2609.05837#bib.bib39)\]:GitHub\(code collaboration\),Notion\(knowledge management\), andPostgreSQL\(relational database\)\. We compare against frontier proprietary models \(GPT\-5, Claude Sonnet\-4\.5, Gemini\-3 Pro\), open\-source foundation models \(GLM\-5\-744B, DeepSeek\-V3\.2\-685B, Qwen3\-235B, GPT\-OSS\-120B, Qwen3\-32B\), and training\-based methods built on Qwen3\-32B: TOUCAN\[[42](https://arxiv.org/html/2609.05837#bib.bib42)\], Vanilla SFT on our collected trajectories, and Rejection Sampling\[[47](https://arxiv.org/html/2609.05837#bib.bib47)\]which retains only LLM\-judge\-approved trajectories for SFT\. The evaluation tasks and ground\-truth verifiers are provided by the mcp\-universe and mcpmark benchmarks\[[20](https://arxiv.org/html/2609.05837#bib.bib20);[39](https://arxiv.org/html/2609.05837#bib.bib39)\]; that these verifiers are used solely for evaluation; they are not available during any stage of the AgentBrew pipeline\. To prevent data leakage, training and evaluation are isolated at two levels: training tasks are generated from capability descriptions, and the training corpus is collected from environment instances that share no entities, data entries, or states with the benchmark evaluation environments \(details in Appendix[F](https://arxiv.org/html/2609.05837#A6)\)\.

##### Data collection and implementation\.

We use Qwen3\-32B\[[43](https://arxiv.org/html/2609.05837#bib.bib43)\]as the unified model for the entire AgentBrew pipeline, serving as the base policy, the reference modelπref\\pi\_\{\\text\{ref\}\}for PMI credit computation, and the backbone for retrospective task inference\. The raw corpus𝒟raw\\mathcal\{D\}\_\{\\text\{raw\}\}contains approximately 2,500 trajectories for GitHub, 2,500 for Notion, and 3,500 for PostgreSQL\. Since real\-world interactions often exceed 100K tokens, we adopt a memory mechanism that retains the three most recent steps in full and compresses earlier history into LLM\-generated summaries\[[23](https://arxiv.org/html/2609.05837#bib.bib23);[49](https://arxiv.org/html/2609.05837#bib.bib49)\]\. This mechanism is applied uniformly across all methods and models, ensuring fair comparison\. Since all prefixes of a trajectory share a common prompt, PMI credit computation benefits from KV\-cache reuse and can be parallelized across trajectories; computing credits for the entire corpus of approximately 8,500 trajectories completes in roughly 9 hours on 4 H100 GPUs\. Further details on data collection and implementation are provided in Appendix[F](https://arxiv.org/html/2609.05837#A6), and case studies are presented in Appendix[G](https://arxiv.org/html/2609.05837#A7)\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2609.05837#S3.T1)summarizes performance across three real\-world applications\. AgentBrew targets data efficiency rather than model scaling: it extracts stronger signal from limited, unlabeled interactions and is therefore complementary to advances in base model capability\. Built on Qwen3\-32B, AgentBrew achieves an average Acc of 18\.2 and Score of 58\.7, surpassing its own 235B variant \(\+2\.3 Acc, \+4\.4 Score\), demonstrating that targeted offline adaptation on a single environment can be more effective than general\-purpose scaling alone\. Absolute accuracy across all methods remains modest, reflecting the genuine difficulty of these real\-world benchmarks, where even frontier proprietary models such as Gemini\-3 Pro \(57\.1 / 80\.8\) and GPT\-5 \(50\.0 / 76\.8\) have ample room for improvement\.

Among training\-based methods, AgentBrew consistently outperforms all baselines across the three environments\. Compared to Vanilla SFT and Rejection Sampling, AgentBrew achieves improvements of \+7\.9 Acc / \+10\.6 Score and \+5\.9 Acc / \+10\.3 Score, respectively, showing that retrospectively realigning task instructions and extracting fine\-grained per\-action credit from noisy trajectories is more effective than either uniform supervision or trajectory\-level quality filtering\. The advantage is most pronounced on Notion, where Rejection Sampling yields an Acc of 0: the LLM judge fails to reliably assess task completion in complex stateful API interactions, causing the filter to reject nearly all trajectories and leaving insufficient data for effective training\. AgentBrew achieves 14\.2 Acc on the same corpus by bypassing verification entirely and instead recovering informative actions within imperfect trajectories\. AgentBrew also outperforms TOUCAN \(\+6\.3 Acc, \+13\.9 Score\), which is trained on 1\.5M trajectories synthesized across broad MCP environments, further confirming that a small amount of environment\-specific data with fine\-grained credit assignment is more effective than large\-scale cross\-environment trajectory synthesis\.

\(a\)

\(b\)

\(c\)

\(d\)

Figure 3:Effect of credit assignment\.\(a\-b\) Performance comparison of different credit assignment strategies on GitHub and Notion\. \(c\-d\) Distribution of mean credit and total credit for well\-aligned \(red\) and misaligned \(blue\) trajectories on GitHub \(c\) and Notion \(d\)\.
### 4\.3Effect of PMI\-Based Credit Assignment

We examine how PMI\-Based credit assignment \(§[3\.2\.2](https://arxiv.org/html/2609.05837#S3.SS2.SSS2)\) identifies informative actions within imperfect trajectories\. All methods are trained on the same realigned pairs\{\(I^n,τn\)\}\\\{\(\\hat\{I\}\_\{n\},\\tau\_\{n\}\)\\\}from retrospective task inference \(§[3\.2\.1](https://arxiv.org/html/2609.05837#S3.SS2.SSS1)\); only the per\-action weighting strategy differs\. We compare our PMI\-based credit against three alternatives:*Uniform*, which assigns equal weight to all actions;*LLM Score*, which promptsπref\\pi\_\{\\text\{ref\}\}to score each of its own actions; and*InfoPO*\[[14](https://arxiv.org/html/2609.05837#bib.bib14)\], which computes credit based on the information gain advantage over future actions\. As shown in Figure[3](https://arxiv.org/html/2609.05837#S4.F3)\(a\-b\), PMI\-based credit consistently outperforms all alternatives on both GitHub and Notion\. Notably, LLM Score drops to 7\.1 / 54\.7 on Notion, underperforming even Uniform \(10\.7 / 61\.6\), suggesting that LLM self\-evaluation is unreliable for action\-level quality assessment in complex stateful tool\-use settings\.

Beyond downstream performance, PMI\-based credit provides a natural safeguard against imperfect task inference: whenI^n\\hat\{I\}\_\{n\}is poorly aligned with the trajectory, actions provide little information about it, yielding uniformly low credits that are clamped to near\-zero by Eq\. \([5](https://arxiv.org/html/2609.05837#S3.E5)\) and ensuring that the trajectory contributes minimal gradient signal during training\. We verify this by using GPT\-5 to classify each\(I^n,τ¯n\)\(\\hat\{I\}\_\{n\},\\bar\{\\tau\}\_\{n\}\)pair as well\-aligned or misaligned\. As shown in Figure[3](https://arxiv.org/html/2609.05837#S4.F3)\(c\-d\), misaligned trajectories \(blue\) cluster at significantly lower mean and total credit compared to well\-aligned ones \(red\) on both GitHub and Notion, confirming that PMI\-based credit automatically suppresses noisy task inference without requiring any explicit quality filter\.

### 4\.4Effect of Retrospective Task Inference

Table 2:Effect of retrospective task inference on Notion\.Task InstructionAcc\.ScoreOriginalInI\_\{n\}7\.165\.6InferredI^n\\hat\{I\}\_\{n\}14\.270\.1We verify the necessity of retrospective task inference \(§[3\.2\.1](https://arxiv.org/html/2609.05837#S3.SS2.SSS1)\) by comparing two training configurations on Notion: one using the original task instructions\{In\}\\\{I\_\{n\}\\\}and the other using the inferred instructions\{I^n\}\\\{\\hat\{I\}\_\{n\}\\\}, both trained with PMI\-based credit assignment\. As shown in Table[2](https://arxiv.org/html/2609.05837#S4.T2), training withI^n\\hat\{I\}\_\{n\}achieves an Acc of 14\.2 and Score of 70\.1, compared to 7\.1 / 65\.6 with the originalInI\_\{n\}\. The degradation underInI\_\{n\}arises because most trajectories fail to complete their original instruction, causing the policy to associate incomplete or erroneous action sequences with the stated goal \(see Appendix[G](https://arxiv.org/html/2609.05837#A7)for qualitative examples\)\. Although PMI\-based credit assignment can down\-weight misaligned trajectories \(§[4\.3](https://arxiv.org/html/2609.05837#S4.SS3)\), credit is computed as the information gain of each action with respect to whatever instruction is provided; when the instruction itself is misaligned, even useful actions receive distorted credits measured against the wrong reference task\. Retrospective task inference addresses this at the source by replacingInI\_\{n\}withI^n\\hat\{I\}\_\{n\}that reflects the trajectory’s actual outcome, ensuring that credit computation operates on coherent instruction\-trajectory pairs\. The two mechanisms are thus complementary: task inference corrects*what*the trajectory is trained to achieve, while credit assignment determines*how much*each action contributes to that corrected goal\.

### 4\.5Robustness to Noisy Trajectories

![Refer to caption](https://arxiv.org/html/2609.05837v1/robustness_figure.png)Figure 4:Performance on Notion when progressively adding lower\-quality data\.As an offline method, AgentBrew can in principle be applied to trajectories from diverse sources, including agent self\-exploration and human usage logs, where data quality is inherently uncontrolled\. Robustness to such quality variance is therefore a practical requirement\. To evaluate this, we use GPT\-5 to score each raw trajectory pair\(In,τn\)\(I\_\{n\},\\tau\_\{n\}\)based on instruction\-trajectory alignment and rank all trajectories by quality in descending order\. We then train AgentBrew on progressively larger subsets: the top 15%, top 50%, top 75%, and the full corpus, where each expansion introduces trajectories of strictly lower quality than those already included\.

As shown in Figure[4](https://arxiv.org/html/2609.05837#S4.F4), both metrics improve monotonically as lower\-quality data is added, with Acc increasing from 10\.7 to 14\.2 and Score from 61\.7 to 70\.1\. Even adding the lowest\-quality 25% of trajectories yields a further Score gain \(67\.1→\\rightarrow70\.1\)\. This does not imply that lower\-quality trajectories are inherently more valuable; rather, each additional trajectory, regardless of its overall completion quality, still contains individual actions that are informative for the inferred task\. AgentBrew’s credit mechanism extracts these useful portions while suppressing the rest, so the net effect of including more data is consistently positive\. This allows practitioners to include trajectories from diverse and uncontrolled sources without degrading performance\.

### 4\.6Offline Transferability Across Models

To test whether AgentBrew’s distilled signal transfers across model sizes, we apply the data distilled by Qwen3\-32B to train a 14B model via the same weighted training objective\. As shown in Table[3](https://arxiv.org/html/2609.05837#S4.T3), AgentBrew\-14B achieves 20\.2 average Acc / 55\.3 Score, substantially outperforming the Qwen3\-14B base model \(\+11\.5 Acc / \+12\.5 Score\), and on Notion nearly matches the 32B AgentBrew model \(14\.2 / 69\.8 vs 14\.2 / 70\.1\)\. Toucan\-14B and AWM\-14B also leverage larger models in their pipelines for trajectory synthesis and environment construction, yet show limited improvement over the base model, indicating that the transferability stems from the quality of AgentBrew’s distilled signal rather than the capacity gap alone\. This confirms that a single round of offline distillation can be reused to train smaller models, amortizing the cost of real\-world API access\.

Table 3:Offline transferability on 14B models\. AgentBrew distills data collected by Qwen3\-32B into a 14B model via the offline pipeline, without additional environment interaction\.MethodTraining ParadigmGitHubNotionPostgresAvg\.Acc\.ScoreAcc\.ScoreAcc\.ScoreAcc\.ScoreQwen3\-14BNo additional training7\.148\.2048\.819\.031\.38\.742\.8Toucan\-14BSFT on 1\.5M synthetic trajs\.045\.2042\.5011\.40\.033\.3AWM\-14BOnline RL on synthetic envs\.7\.145\.4044\.523\.838\.010\.342\.6AgentBrew\-14BOffline transfer17\.956\.314\.269\.828\.640\.020\.255\.3

## 5Related Work

##### Training Data for Tool\-Use Agents\.

A central challenge in training tool\-use agents is obtaining high\-quality \(task, trajectory\) pairs at scale\. The prevailing forward approach first defines tasks, then collects trajectories by executing them with strong models and retaining only successful completions\[[26](https://arxiv.org/html/2609.05837#bib.bib26);[48](https://arxiv.org/html/2609.05837#bib.bib48);[24](https://arxiv.org/html/2609.05837#bib.bib24);[42](https://arxiv.org/html/2609.05837#bib.bib42);[34](https://arxiv.org/html/2609.05837#bib.bib34);[17](https://arxiv.org/html/2609.05837#bib.bib17)\]\. An alternative reverse approach derives tasks from existing interactions: OS\-Genesis\[[33](https://arxiv.org/html/2609.05837#bib.bib33)\]infers tasks from GUI state transitions, AgentTrek\[[41](https://arxiv.org/html/2609.05837#bib.bib41)\]uses web tutorials to guide trajectory synthesis, and Learn\-by\-interact\[[32](https://arxiv.org/html/2609.05837#bib.bib32)\]synthesizes trajectories from documentation and constructs instructions via backward abstraction of the interaction histories\. Closer to our work, hindsight relabeling methods recover training signal from failed trajectories by replacing the original instruction with one the trajectory actually fulfills\. This idea originates from HER\[[1](https://arxiv.org/html/2609.05837#bib.bib1)\]in classic RL; HSL\[[18](https://arxiv.org/html/2609.05837#bib.bib18)\]adapts it to LLM agents using environment rewards, and AgentHER\[[5](https://arxiv.org/html/2609.05837#bib.bib5)\]introduces failure classification with confidence gating\. Across these lines of work, data quality is ensured through external signals such as execution verification, environment rewards, or LLM\-based filtering\. AgentBrew requires none of these, instead deferring quality control entirely to an information\-theoretic credit assignment mechanism that operates on the trajectories themselves\.

##### Policy Optimization for LLM Agents\.

On\-policy reinforcement learning is widely used to train LLM agents in interactive environments, with methods such as GRPO\[[30](https://arxiv.org/html/2609.05837#bib.bib30)\], DAPO\[[46](https://arxiv.org/html/2609.05837#bib.bib46)\], WebRL\[[25](https://arxiv.org/html/2609.05837#bib.bib25)\], and DigiRL\[[3](https://arxiv.org/html/2609.05837#bib.bib3)\]relying on iterative rollouts and environment rewards\. A central challenge is credit assignment, since trajectory\-level rewards provide sparse supervision and cannot distinguish informative actions from redundant ones\. To obtain finer\-grained signals, IGPO\[[36](https://arxiv.org/html/2609.05837#bib.bib36)\]uses information gain on a ground\-truth answer as turn\-level reward, GiGPO\[[9](https://arxiv.org/html/2609.05837#bib.bib9)\]builds step\-level groups from anchor states, iStar\[[19](https://arxiv.org/html/2609.05837#bib.bib19)\]infers step rewards from trajectory preferences, and AgentPRM\[[40](https://arxiv.org/html/2609.05837#bib.bib40)\]trains a process reward model via temporal\-difference estimation\. These methods remain on\-policy and require repeated interaction with reward feedback\. Offline methods reduce rollout cost: DPO\[[27](https://arxiv.org/html/2609.05837#bib.bib27)\], KTO\[[8](https://arxiv.org/html/2609.05837#bib.bib8)\], and RPO\[[45](https://arxiv.org/html/2609.05837#bib.bib45)\]learn from preference pairs, while Digi\-Q\[[4](https://arxiv.org/html/2609.05837#bib.bib4)\]and OREO\[[37](https://arxiv.org/html/2609.05837#bib.bib37)\]learn value or Q\-functions from offline trajectories\. Yet they still depend on rewards, preferences, or learned value functions\. In contrast, AgentBrew is fully offline and derives per\-action training weights directly from the relationship between trajectories and their inferred tasks, without external reward signals or preference annotations\.

## 6Conclusion, Limitation, and Future Work

We presented AgentBrew, an offline framework that trains tool\-use agents from a single batch of raw interactions without task verifiers or on\-policy rollouts\. By retrospectively inferring aligned instructions and decomposing per\-action credit, AgentBrew extracts training signal from noisy trajectories that filtering\-based methods would discard, enabling a 32B model to surpass Qwen3\-235B across three real\-world applications\. Current limitations include evaluation on only three MCP applications, potential degradation of task inference on extremely long or chaotic trajectories, and the reliance on PMI as an informational proxy for action quality, which, despite structural alignment with actual contribution through the construction ofI^n\\hat\{I\}\_\{n\}\(§[3\.2\.2](https://arxiv.org/html/2609.05837#S3.SS2.SSS2)\), may not capture all causal effects in highly entangled action sequences\.

A promising future direction is applying AgentBrew to real user interaction logs in production applications, which are naturally unlabeled and noisy\. Retrospective task inference could recover user intent directly from operation sequences, and credit assignment could identify key actions within each session, enabling agents to learn from authentic human demonstrations at scale\.

## References

- Andrychowicz et al\. \[2017\]Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba\.Hindsight experience replay\.*Advances in neural information processing systems*, 30, 2017\.
- Anthropic \[2024\]Anthropic\.Introducing the model context protocol\.[https://www\.anthropic\.com/news/model\-context\-protocol](https://www.anthropic.com/news/model-context-protocol), November 2024\.
- Bai et al\. \[2024\]Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar\.Digirl: Training in\-the\-wild device\-control agents with autonomous reinforcement learning\.*Advances in Neural Information Processing Systems*, 37:12461–12495, 2024\.
- Bai et al\. \[2025\]Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar\.Digi\-q: Learning q\-value functions for training device\-control agents\.*arXiv preprint arXiv:2502\.15760*, 2025\.
- Ding \[2026\]Liang Ding\.Agenther: Hindsight experience replay for llm agent trajectory relabeling\.*arXiv preprint arXiv:2603\.21357*, 2026\.
- Dong et al\. \[2026\]Guanting Dong, Junting Lu, Junjie Huang, Wanjun Zhong, Longxiang Liu, Shijue Huang, Zhenyu Li, Yang Zhao, Xiaoshuai Song, Xiaoxi Li, Jiajie Jin, Yutao Zhu, Hanbin Wang, Fangyu Lei, Qinyu Luo, Mingyang Chen, Zehui Chen, Jiazhan Feng, Ji\-Rong Wen, and Zhicheng Dou\.Agent\-world: Scaling real\-world environment synthesis for evolving general agent intelligence\.*arXiv preprint*, 2026\.
- Drouin et al\. \[2024\]Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al\.Workarena: How capable are web agents at solving common knowledge work tasks?*arXiv preprint arXiv:2403\.07718*, 2024\.
- Ethayarajh et al\. \[2024\]Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela\.Kto: Model alignment as prospect theoretic optimization\.*arXiv preprint arXiv:2402\.01306*, 2024\.
- Feng et al\. \[2025\]Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An\.Group\-in\-group policy optimization for llm agent training\.*arXiv preprint arXiv:2505\.10978*, 2025\.
- GitHub \[2025\]GitHub\.Github mcp server\.[https://github\.com/github/github\-mcp\-server](https://github.com/github/github-mcp-server), 2025\.GitHub repository\. Accessed: 2026\-04\-17\.
- Gulcehre et al\. \[2023\]Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al\.Reinforced self\-training \(rest\) for language modeling\.*arXiv preprint arXiv:2308\.08998*, 2023\.
- Guo et al\. \[2024\]Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu\.Stabletoolbench: Towards stable large\-scale benchmarking on tool learning of large language models\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 11143–11156, 2024\.
- He et al\. \[2026\]Shuo He, Lang Feng, Qi Wei, Xin Cheng, Lei Feng, and Bo An\.Hierarchy\-of\-groups policy optimization for long\-horizon agentic tasks\.*arXiv preprint arXiv:2602\.22817*, 2026\.
- Kong et al\. \[2026\]Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, and Bang Liu\.Infopo: Information\-driven policy optimization for user\-centric agents\.*arXiv preprint arXiv:2603\.00656*, 2026\.
- Lei et al\. \[2024\]Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al\.Spider 2\.0: Evaluating language models on real\-world enterprise text\-to\-sql workflows\.*arXiv preprint arXiv:2411\.07763*, 2024\.
- Li et al\. \[2023\]Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al\.Can llm already serve as a database interface? a big bench for large\-scale database grounded text\-to\-sqls\.*Advances in Neural Information Processing Systems*, 36:42330–42357, 2023\.
- Li et al\. \[2025\]Yuetai Li, Huseyin A Inan, Xiang Yue, Wei\-Ning Chen, Lukas Wutschitz, Janardhan Kulkarni, Radha Poovendran, Robert Sim, and Saravan Rajmohan\.Simulating environments with reasoning models for agent training\.*arXiv preprint arXiv:2511\.01824*, 2025\.
- Li et al\. \[2026\]Zichao Li, Gang Wu, Zichao Wang, Ruiyi Zhang, Wanrong Zhu, Ryan A\. Rossi, Vlad I Morariu, and Jihyung Kil\.Spinning straw into gold: Relabeling LLM agent trajectories in hindsight for successful demonstrations\.In*The Fourteenth International Conference on Learning Representations*, 2026\.
- Liu et al\. \[2025\]Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao\.Agentic reinforcement learning with implicit step rewards\.*arXiv preprint arXiv:2509\.19199*, 2025\.
- Luo et al\. \[2025\]Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li\.Mcp\-universe: Benchmarking large language models with real\-world model context protocol servers\.*arXiv preprint arXiv:2508\.14704*, 2025\.
- Lyu et al\. \[2025\]Zhiyi Lyu, Jianguo Huang, Yanchen Deng, Steven Hoi, and Bo An\.Let’s revise step\-by\-step: A unified local search framework for code generation with llms\.*arXiv preprint arXiv:2508\.07434*, 2025\.
- Notion \[2025\]Notion\.Notion: The ai workspace that works for you\.[https://www\.notion\.com/](https://www.notion.com/), 2025\.Accessed: 2026\-04\-17\.
- Packer et al\. \[2023\]Charles Packer, Vivian Fang, Shishir\_G Patil, Kevin Lin, Sarah Wooders, and Joseph\_E Gonzalez\.Memgpt: towards llms as operating systems\.2023\.
- Prabhakar et al\. \[2025\]Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al\.Apigen\-mt: Agentic pipeline for multi\-turn data generation via simulated agent\-human interplay\.*arXiv preprint arXiv:2504\.03601*, 2025\.
- Qi et al\. \[2024\]Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al\.Webrl: Training llm web agents via self\-evolving online curriculum reinforcement learning\.*arXiv preprint arXiv:2411\.02337*, 2024\.
- Qin et al\. \[2023\]Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al\.Toolllm: Facilitating large language models to master 16000\+ real\-world apis\.*arXiv preprint arXiv:2307\.16789*, 2023\.
- Rafailov et al\. \[2023\]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.*Advances in neural information processing systems*, 36:53728–53741, 2023\.
- Rawles et al\. \[2024\]Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell\-Ajala, et al\.Androidworld: A dynamic benchmarking environment for autonomous agents\.*arXiv preprint arXiv:2405\.14573*, 2024\.
- Schick et al\. \[2023\]Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\.Toolformer: Language models can teach themselves to use tools\.*Advances in neural information processing systems*, 36:68539–68551, 2023\.
- Shao et al\. \[2024\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shi et al\. \[2024\]Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng\.Direct multi\-turn preference optimization for language agents\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 2312–2324, 2024\.
- Su et al\. \[2025\]Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Arık\.Learn\-by\-interact: A data\-centric framework for self\-adaptive agents in realistic environments\.*arXiv preprint arXiv:2501\.10893*, 2025\.
- Sun et al\. \[2025\]Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al\.Os\-genesis: Automating gui agent trajectory construction via reverse task synthesis\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 5555–5579, 2025\.
- Tian et al\. \[2026\]Xiaoyu Tian, Haotian Wang, Shuaiting Chen, Hao Zhou, Kaichi Yu, Yudian Zhang, Jade Ouyang, Junxi Yin, Jiong Chen, Baoyan Guo, et al\.Astra: Automated synthesis of agentic trajectories and reinforcement arenas\.*arXiv preprint arXiv:2601\.21558*, 2026\.
- Tu et al\. \[2026\]Dunwei Tu, Hongyan Hao, Hansi Yang, Yihao Chen, Yi\-Kai Zhang, Zhikang Xia, Yu Yang, Yueqing Sun, Xingchen Liu, Furao Shen, et al\.Scaleenv: Scaling environment synthesis from scratch for generalist interactive tool\-use agent training\.*arXiv preprint arXiv:2602\.06820*, 2026\.
- Wang et al\. \[2025a\]Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying\.Information gain\-based policy optimization: A simple and effective approach for multi\-turn llm agents\.*arXiv preprint arXiv:2510\.14967*, 2025a\.
- Wang et al\. \[2025b\]Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu\.Offline reinforcement learning for llm multi\-step reasoning\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 8881–8893, 2025b\.
- Wang et al\. \[2026\]Zhaoyang Wang, Canwen Xu, Boyi Liu, Yite Wang, Siwei Han, Zhewei Yao, Huaxiu Yao, and Yuxiong He\.Agent world model: Infinity synthetic environments for agentic reinforcement learning\.*arXiv preprint arXiv:2602\.10090*, 2026\.
- Wu et al\. \[2025\]Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, Yaoqi Ye, Jiawei Wang, et al\.Mcpmark: A benchmark for stress\-testing realistic and comprehensive mcp use\.*arXiv preprint arXiv:2509\.24002*, 2025\.
- Xi et al\. \[2025\]Zhiheng Xi, Chenyang Liao, Guanyu Li, Yajie Yang, Wenxiang Chen, Zhihao Zhang, Binghai Wang, Senjie Jin, Yuhao Zhou, Jian Guan, et al\.Agentprm: Process reward models for llm agents via step\-wise promise and progress\.*arXiv preprint arXiv:2511\.08325*, 2025\.
- Xu et al\. \[2024\]Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu\.Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials\.*arXiv preprint arXiv:2412\.09605*, 2024\.
- Xu et al\. \[2025\]Zhangchen Xu, Adriana Meza Soria, Shawn Tan, Anurag Roy, Ashish Sunil Agrawal, Radha Poovendran, and Rameswar Panda\.Toucan: Synthesizing 1\.5 m tool\-agentic data from real\-world mcp environments\.*arXiv preprint arXiv:2510\.01179*, 2025\.
- Yang et al\. \[2025\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yao et al\. \[2022\]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao\.React: Synergizing reasoning and acting in language models\.In*The eleventh international conference on learning representations*, 2022\.
- Yin et al\. \[2024\]Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou\.Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts\.*arXiv preprint arXiv:2402\.10958*, 2024\.
- Yu et al\. \[2025\]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al\.Dapo: An open\-source llm reinforcement learning system at scale\.*arXiv preprint arXiv:2503\.14476*, 2025\.
- Yuan et al\. \[2023\]Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou\.Scaling relationship on learning mathematical reasoning with large language models\.*arXiv preprint arXiv:2308\.01825*, 2023\.
- Zhang et al\. \[2025\]Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Quoc Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al\.xlam: A family of large action models to empower ai agent systems\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 11583–11597, 2025\.
- Zhou et al\. \[2023\]Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan\.Recurrentgpt: Interactive generation of \(arbitrarily\) long text\.*arXiv preprint arXiv:2305\.13304*, 2023\.

## Appendix APseudo Code

We provide the complete AgentBrew pipeline in Algorithm[1](https://arxiv.org/html/2609.05837#alg1)\. The procedure consists of three stages: raw experience collection \(§[3\.1](https://arxiv.org/html/2609.05837#S3.SS1)\), experience distillation \(§[3\.2](https://arxiv.org/html/2609.05837#S3.SS2)\), and PMI\-weighted policy training \(§[3\.3](https://arxiv.org/html/2609.05837#S3.SS3)\)\. Stage 1 is the only phase that requires interaction with the live environmentℰ\\mathcal\{E\}; Stages 2 and 3 proceed entirely offline using only the collected corpus and a reference language modelπref\\pi\_\{\\text\{ref\}\}\.

Algorithm 1AgentBrew: Offline Tool\-Use Agent Learning from Raw Trajectories0:Target environment

ℰ\\mathcal\{E\}with tool set

ℱ\\mathcal\{F\}, base policy

πθ\\pi\_\{\\theta\}, reference model

πref\\pi\_\{\\text\{ref\}\}, max turns

TmaxT\_\{\\max\}
0:Trained policy

πθ\\pi\_\{\\theta\}
1:— Stage 1: Raw Experience Collection —

2:Generate high\-level task categories

𝒢\\mathcal\{G\}from tool descriptions of

ℱ\\mathcal\{F\}
3:foreach category

gm∈𝒢g\_\{m\}\\in\\mathcal\{G\}do

4:Explore

ℰ\\mathcal\{E\}via tool calls to retrieve live environmental information

5:Propose grounded task instructions

\{In\}\\\{I\_\{n\}\\\}based on retrieved information

6:foreach task

In∈ℐI\_\{n\}\\in\\mathcal\{I\}do

7:Execute

InI\_\{n\}with

πθ\\pi\_\{\\theta\}in

ℰ\\mathcal\{E\}, producing trajectory

τn=\{\(st,at,ot\)\}t=0Tn\\tau\_\{n\}=\\\{\(s\_\{t\},a\_\{t\},o\_\{t\}\)\\\}\_\{t=0\}^\{T\_\{n\}\}
8:if

Tn=TmaxT\_\{n\}=T\_\{\\max\}\(no natural termination\)then

9:Discard

τn\\tau\_\{n\}
10:Collect raw corpus

𝒟raw=\{\(In,τn\)\}n=1N\\mathcal\{D\}\_\{\\text\{raw\}\}=\\\{\(I\_\{n\},\\tau\_\{n\}\)\\\}\_\{n=1\}^\{N\}⊳\\trianglerightNo quality filtering

11:

12:— Stage 2: Experience Distillation \(Offline\) —

13:foreach

\(In,τn\)∈𝒟raw\(I\_\{n\},\\tau\_\{n\}\)\\in\\mathcal\{D\}\_\{\\text\{raw\}\}do

14:// Retrospective Task Inference

15:Remove chain\-of\-thought traces:

τ¯n=\{\(ct,ot\)\}t=0Tn\\bar\{\\tau\}\_\{n\}=\\\{\(c\_\{t\},o\_\{t\}\)\\\}\_\{t=0\}^\{T\_\{n\}\}
16:Infer revised instruction:

I^n←LLM​\(τ¯n\)\\hat\{I\}\_\{n\}\\leftarrow\\text\{LLM\}\(\\bar\{\\tau\}\_\{n\}\)⊳\\trianglerightGoal\-oriented, evidence\-grounded

17:

18:// PMI\-Based Credit Assignment

19:Compute unconditional NLL:

ℒ∅=−log⁡πref​\(I^n\)\\mathcal\{L\}\_\{\\emptyset\}=\-\\log\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\)
20:for

t=0,1,…,Tnt=0,1,\\ldots,T\_\{n\}do

21:Compute conditional NLL:

ℒt=−log⁡πref​\(I^n∣c0,o0,…,ct,ot\)\\mathcal\{L\}\_\{t\}=\-\\log\\pi\_\{\\text\{ref\}\}\(\\hat\{I\}\_\{n\}\\mid c\_\{0\},o\_\{0\},\\ldots,c\_\{t\},o\_\{t\}\)
22:

w0←ℒ∅−ℒ0w\_\{0\}\\leftarrow\\mathcal\{L\}\_\{\\emptyset\}\-\\mathcal\{L\}\_\{0\}
23:for

t=1,…,Tnt=1,\\ldots,T\_\{n\}do

24:

wt←ℒt−1−ℒtw\_\{t\}\\leftarrow\\mathcal\{L\}\_\{t\-1\}\-\\mathcal\{L\}\_\{t\}⊳\\trianglerightPer\-action PMI credit

25:

26:// Weight Normalization

27:for

t=0,1,…,Tnt=0,1,\\ldots,T\_\{n\}do

28:

w~t←2⋅max⁡\(wt,0\)/max0≤j≤Tn⁡wj\\tilde\{w\}\_\{t\}\\leftarrow 2\\cdot\\max\(w\_\{t\},0\)\\;/\\;\\max\_\{0\\leq j\\leq T\_\{n\}\}w\_\{j\}⊳\\trianglerightNormalize to\[0,2\]\[0,2\]

29:

30:— Stage 3: PMI\-Weighted Policy Training \(Offline\) —

31:Optimize

πθ\\pi\_\{\\theta\}by minimizing:

32:

𝒥\(θ\)=−∑n=1N∑t=0Tnw~t\(n\)⋅logπθ\(at∣I^n,st\)\\mathcal\{J\}\(\\theta\)=\-\\displaystyle\\sum\_\{n=1\}^\{N\}\\sum\_\{t=0\}^\{T\_\{n\}\}\\tilde\{w\}\_\{t\}^\{\(n\)\}\\cdot\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid\\hat\{I\}\_\{n\},s\_\{t\}\)

## Appendix BEnvironment Details

We evaluate AgentBrew on three real\-world MCP applications that span distinct interaction paradigms: code collaboration \(GitHub\), knowledge management \(Notion\), and relational database administration \(PostgreSQL\)\. Table[4](https://arxiv.org/html/2609.05837#A2.T4)lists the complete tool set exposed by each environment\.

##### GitHub\.

The GitHub environment exposes 41 tools covering the full spectrum of platform operations: repository management, file operations, issue tracking, pull request workflows, branch and commit inspection, CI/CD workflow management, code and user search, and security scanning \(code scanning, Dependabot, secret scanning, and security advisories\)\. Tasks in this environment require composing long action sequences across these tool categories\. For example, forking a repository, creating a feature branch, pushing file changes, opening a pull request, and performing a code review, all within a single trajectory\. The live state includes cross\-referenced issues, pull requests, commits, and branches, making precise entity resolution critical\.

##### Notion\.

The Notion environment provides 19 tools organized around four core object types: users, blocks, pages, and databases, along with search and comment capabilities\. Despite the smaller tool count, Notion tasks are challenging because of deeply nested content structures: pages contain blocks that may themselves contain child blocks or inline database references, and databases define typed schemas with relations and rollups linking to other databases\. Operations such as inserting a block at a precise position, restructuring a page layout, or synchronizing rows across related databases require the agent to maintain an accurate mental model of the workspace hierarchy throughout the interaction\.

##### PostgreSQL\.

The PostgreSQL environment exposes 9 tools that combine standard SQL execution with database administration utilities\. Beyondexecute\_sqlfor arbitrary SQL statements, the environment provides schema inspection tools \(list\_schemas,list\_objects,get\_object\_details\), query analysis tools \(explain\_query,get\_top\_queries\), and health diagnostics \(analyze\_workload\_indexes,analyze\_query\_indexes,analyze\_db\_health\)\. Although the tool set is compact, the complexity arises from the SQL reasoning itself: tasks range from multi\-table data migrations and schema design to query optimization and security policy implementation, requiring the agent to generate correct, multi\-step SQL workflows grounded in the actual database schema and data\.

Table 4:Tool sets exposed by each environment\.Env\.\#ToolsToolsGitHub41create\_repository, get\_repository, list\_repositories, fork\_repository, get\_repository\_languages, get\_file\_contents, create\_or\_update\_file, delete\_file, push\_files, create\_issue, get\_issue, list\_issues, update\_issue, add\_issue\_comment, create\_pull\_request, get\_pull\_request, list\_pull\_requests, get\_pull\_request\_files, create\_pull\_request\_review, merge\_pull\_request, create\_branch, list\_branches, list\_commits, list\_workflow\_runs, get\_workflow\_run, list\_workflow\_jobs, cancel\_workflow\_run, search\_repositories, search\_code, search\_issues, search\_users, list\_code\_scanning\_alerts, get\_code\_scanning\_alert, list\_dependabot\_alerts, get\_dependabot\_alert, list\_secret\_scanning\_alerts, get\_secret\_scanning\_alert, list\_global\_security\_advisories, list\_org\_repository\_security\_advisories, list\_repository\_security\_advisories, create\_pull\_request\_with\_copilotNotion19get\-user, get\-users, get\-self, post\-database\-query, post\-search, get\-block\-children, patch\-block\-children, retrieve\-a\-block, update\-a\-block, delete\-a\-block, retrieve\-a\-page, patch\-page, post\-page, create\-a\-database, update\-a\-database, retrieve\-a\-database, retrieve\-a\-page\-property, retrieve\-a\-comment, create\-a\-commentPostgres9list\_schemas, list\_objects, get\_object\_details, execute\_sql, explain\_query, get\_top\_queries, analyze\_workload\_indexes, analyze\_query\_indexes, analyze\_db\_health

## Appendix CTask Proposal

##### Environment preparation\.

Each target environmentℰ\\mathcal\{E\}requires a diverse substrate of real\-world content to ensure that proposed tasks are grounded in meaningful state\. ForGitHub, we curate 200 public repositories spanning a variety of programming languages \(Python, JavaScript, Go, Rust, etc\.\) and project types \(web applications, CLI tools, data pipelines, machine learning libraries\), providing a rich landscape of issues, pull requests, branches, and commit histories for the agent to interact with\. ForNotion, we collect 180 free templates from the official Notion template gallery,111[https://www\.notion\.com/templates](https://www.notion.com/templates)covering domains such as project management, personal planning, knowledge bases, and team collaboration\. These templates provide pre\-populated pages, databases, and relational structures that mirror realistic workspace configurations\. ForPostgreSQL, we assemble 145 relational databases sourced from established open\-source benchmarks including BIRD\[[Li et al\., 2023](https://arxiv.org/html/2609.05837#bib.bib16)\]and Spider\[[Lei et al\., 2024](https://arxiv.org/html/2609.05837#bib.bib15)\], which collectively span domains such as retail, healthcare, education, sports, and enterprise operations\. This diversity ensures that the subsequent task proposal stage can generate instructions grounded in entities and configurations that actually exist within each environment\.

##### High\-level task category generation\.

Given the tool setℱ\\mathcal\{F\}of the target environmentℰ\\mathcal\{E\}, we prompt the agent to generate a set of high\-level task categories𝒢=\{g1,g2,…,gM\}\\mathcal\{G\}=\\\{g\_\{1\},g\_\{2\},\\ldots,g\_\{M\}\\\}that cover representative human intents supported byℰ\\mathcal\{E\}\. The agent receives the names and descriptions of all available tools as input and produces categories that reflect common multi\-step usage patterns, each requiring the composition of multiple tools\. Table[5](https://arxiv.org/html/2609.05837#A3.T5)lists the generated categories for each environment\. GitHub yields 14 categories spanning repository lifecycle management, code collaboration, and project planning; Notion produces 12 categories covering database operations, page editing, and content synthesis; PostgreSQL results in 15 categories ranging from data migration and analytics to security policy design\. The prompt template used for category generation is shown below\.

Prompt: Task Category GenerationYou are an expert task designer for evaluating AI agents that interact with real\-world applications via tool\-use APIs\.You are given the complete tool set of the\{environment\_name\}environment\. Each tool is described by its name, purpose, and parameter specification:\{tool\_descriptions\}Your goal is to generate a diverse set of high\-level task categories that represent realistic human intents on this platform\. Each category should:\(1\) Requirecomposing multiple toolsin a meaningful sequence \(not single\-tool invocations\)\.\(2\) Reflect aconcrete, goal\-oriented workflowthat a real user would perform \(e\.g\., “migrate data between tables with validation”, not “use the search API”\)\.\(3\) Bedistinctfrom other categories in terms of the tool combinations and reasoning patterns involved\.\(4\) Coverdiverse difficulty levels, from straightforward CRUD operations to complex multi\-step workflows involving conditional logic, cross\-entity coordination, or synthesis\.For each category, provide:–Title: A concise name for the category\.–Goal: A one\-sentence description of what a user aims to accomplish\.Generate\{num\_categories\}categories that collectively maximize the coverage of the tool set\.

Table 5:Task categories generated for each environment\.GitHub \(14\)Notion \(12\)PostgreSQL \(15\)1\.Project bootstrap & self\-review1\.Database creation & schema design1\.Bulk migration & transformation2\.Multi\-source code integration2\.Cross\-database relation & sync2\.Hierarchy\-aware CRUD & integrity3\.Ecosystem metrics & reporting3\.Batch query, filter & migration3\.Dashboard reporting & aggregates4\.Decision\-based comparative forking4\.Batch updates & coordinated edits4\.SQL query debugging & fixing5\.Simulated issue automation5\.Aggregate computation & dashboards5\.Query optimization & indexing6\.Fork\-and\-fix contribution6\.Multi\-source query & synthesis6\.Operational workflow management7\.Cross\-repo dependency injection7\.Page layout & column restructuring7\.Descriptive reporting & statistics8\.Epic & sub\-issue planning8\.Block insertion & format matching8\.Multi\-factor performance analytics9\.Release audit & upgrade docs9\.Template filling & nested editing9\.Schema buildout & seed data10\.Repo maintenance & cleanup10\.Content deletion & archival10\.Retention & churn analysis11\.Code archaeology & arch\. docs11\.Style & color batch updates11\.Executive dashboard automation12\.Legacy branch backporting12\.Itinerary & planner composition12\.Structural & bottleneck analysis13\.Research framework scaffolding13\.Consistency enforcement14\.PR queue management & hygiene14\.Security policy & row\-level access15\.Transactional procedures & audit

##### Grounded task instruction generation\.

For each categorygm∈𝒢g\_\{m\}\\in\\mathcal\{G\}, a task generation agent conducts a lightweight exploratory interaction withℰ\\mathcal\{E\}to ground the abstract category in concrete environmental state\. The agent follows a ReAct\-style loop: it first invokes a single read\-only tool call \(e\.g\., listing repository contents, querying database schemas, or browsing page structures\) to discover real entities such as repository names, file paths, issue numbers, page IDs, or table columns\. Based on the retrieved information, the agent then formulates a specific task instructionInI\_\{n\}written as a natural user request that references only verified identifiers\. This grounding step is critical because tasks that reference nonexistent resources \(e\.g\., a deleted branch or a missing database entry\) would produce unexecutable trajectories during the subsequent data collection phase\. To maintain diversity, the agent also receives an example task from a different category as a style reference, ensuring consistent tone and specificity across the generated task set\. The prompt template is shown below\.

Prompt: Grounded Task Instruction GenerationYou are a ReAct \(Reasoning and Acting\) agent specializing in Task Engineering\. You are designing a single, high\-quality task for LLMs that interact with real\-world applications via tool\-use\.Tool Restrictions:You may ONLY call read/search tools to inspect the target environment — do NOT create, modify, or delete anything\. Call exactly 1 tool to ground yourself, then immediately output the final task\.What the generated task CAN include:The task may involve any operation supported by the environment \(reading, creating, updating, deleting resources\)\. Write operations must target user\-owned resources \(newly created or forked\)\.Your Job:\(1\) Read theWorkflow Pattern\(\{task\_category\}\) to understand the type of task to generate\.\(2\) Read theExample Task\(\{example\_task\}\) to learn the desired tone, style, and level of detail\.\(3\) Call 1 read\-only tool to inspect the target environment and discover concrete entities\.\(4\) Write a NEW task that follows the same workflow pattern, mimics the example’s tone, and is grounded in the environment’s real state\.Available tools:\{tool\_descriptions\}Task Output Constraints:\(1\) Output exactly one task\. Do not generate multiple tasks or sub\-task lists\.\(2\) Write it as a natural user request — same tone and style as the example\. No formal specs, no numbered checklists, no headings\.\(3\) Do not mention “tools” or “APIs” in the final task text\.\(4\) Include concrete identifiers confirmed via the tool call\.Output Format:A JSON object with athoughtfield \(verification reasoning\) and either anactionfield \(to call a read\-only tool\) or ananswerfield \(the final natural\-language task instruction\)\.

## Appendix DPMI\-Based Credit Assignment

For each trajectory, we first infer a hindsight instructionI^\\hat\{I\}that describes the outcome actually supported by the observed tool interactions\. We then compute per\-action credit by measuring how much each trajectory prefix reduces the negative log\-likelihood ofI^\\hat\{I\}under a reference modelπref\\pi\_\{\\mathrm\{ref\}\}\. Letτ¯≤t\\bar\{\\tau\}\_\{\\leq t\}denote the serialized tool\-interaction prefix up to steptt, and letI^=\(i^1,…,i^m\)\\hat\{I\}=\(\\hat\{i\}\_\{1\},\\ldots,\\hat\{i\}\_\{m\}\)be the tokenized hindsight instruction\. In implementation, we compute the average token negative log\-likelihood:

Lt=−1m∑k=1mlogπref\(i^k∣i^<k,τ¯≤t\)\.L\_\{t\}=\-\\frac\{1\}\{m\}\\sum\_\{k=1\}^\{m\}\\log\\pi\_\{\\mathrm\{ref\}\}\\left\(\\hat\{i\}\_\{k\}\\mid\\hat\{i\}\_\{<k\},\\bar\{\\tau\}\_\{\\leq t\}\\right\)\.The empty\-prefix loss is denoted byL−1L\_\{\-1\}and is computed in the same way without conditioning on any observed tool interactions:

L−1=−1m∑k=1mlogπref\(i^k∣i^<k\)\.L\_\{\-1\}=\-\\frac\{1\}\{m\}\\sum\_\{k=1\}^\{m\}\\log\\pi\_\{\\mathrm\{ref\}\}\\left\(\\hat\{i\}\_\{k\}\\mid\\hat\{i\}\_\{<k\}\\right\)\.The raw PMI credit for actionttis the reduction in hindsight\-instruction NLL after revealing that action:

wt=Lt−1−Lt\.w\_\{t\}=L\_\{t\-1\}\-L\_\{t\}\.Thus, positive credit indicates that the action makes the inferred instruction easier to predict, while zero or negative credit indicates that the action is redundant or uninformative with respect to the inferred instruction\. Before training, we clamp negative credits and normalize weights within each trajectory:

w~t=2⋅max⁡\(wt,0\)maxj⁡wj\.\\tilde\{w\}\_\{t\}=2\\cdot\\frac\{\\max\(w\_\{t\},0\)\}\{\\max\_\{j\}w\_\{j\}\}\.If all raw credits in a trajectory are non\-positive, we set all normalized weights to zero\.

##### Empirical statistics of normalized weights\.

Across the full training corpus, the mean normalized weightw~t\\tilde\{w\}\_\{t\}is 0\.75, with per\-environment values of 0\.81 \(GitHub\), 0\.71 \(Notion\), and 0\.77 \(PostgreSQL\)\. This value is below the uniform baseline of 1\.0 because negative credits are clamped to zero and many actions within each trajectory receive near\-zero credit, reflecting the intended selective effect of PMI weighting\. The relative ranking of per\-action credits, rather than the absolute scale, is the primary mechanism through which PMI weighting steers the policy toward informative actions\.

## Appendix ERetrospective Task Inference

As described in §[3\.2\.1](https://arxiv.org/html/2609.05837#S3.SS2.SSS1), we infer a revised instructionI^n\\hat\{I\}\_\{n\}from each trajectory’s reduced formτ¯n=\{\(ct,ot\)\}t=0Tn\\bar\{\\tau\}\_\{n\}=\\\{\(c\_\{t\},o\_\{t\}\)\\\}\_\{t=0\}^\{T\_\{n\}\}, which retains only tool calls and environment responses\. We use Qwen3\-32B for this inference, consistent with the rest of the AgentBrew pipeline\. Before prompting, we extract a structured change summary fromτ¯n\\bar\{\\tau\}\_\{n\}by identifying successful state\-changing operations \(e\.g\., pages created, rows inserted, SQL objects defined\) and separating them from read\-only or failed steps\. Both the change summary and the full reduced trajectory are provided as input\.

The prompt follows a consistent template across all three environments, instantiated with environment\-specific terminology and examples\. The general template is shown below\.

Prompt: Retrospective Task InferenceYou revise benchmark\-style\{environment\_name\}tasks from agent tool trajectories\.Your job is to minimally edit the ORIGINAL TASK so it matches what the agent actually completed\.Main goal:•Remove or weaken unsupported subtasks\.•The revised task must be highly consistent with the trajectory\.Use the trajectory as evidence, with priority on: \(1\) successful state\-changing operations; \(2\) query/retrieve steps only when they help identify objects involved in successful writes\.Rules:•Output only the revised task text\.•Preserve the original high\-level task type \(e\.g\., migration stays migration, analysis stays analysis\)\.•If the agent only completed part of a multi\-step task, keep only the completed subset\.•Do not claim outcomes \(e\.g\., performance improvements, verification success\) unless directly supported\.•Do not invent entities, artifacts, or results not evidenced by the trajectory\.•Prefer deleting unsupported details over replacing them with new speculative ones\.•Do not preserve strong claims \(all,each,exact count,verbatim\) unless directly supported by successful operations\.ORIGINAL TASK: \{original\_task\}OBSERVED CHANGES: \{change\_summary\}TRAJECTORY: \{trajectory\_str\}

## Appendix FExperiment Details

### F\.1Context Management

Real\-world tool\-use interactions frequently produce trajectories exceeding 100K tokens, as each tool call may return verbose API responses \(e\.g\., full page content in Notion or lengthy query results in PostgreSQL\)\. To handle this within finite context windows, we adopt a sliding\-window memory mechanism inspired by prior work on memory\-augmented agents\[[Packer et al\., 2023](https://arxiv.org/html/2609.05837#bib.bib23),[Zhou et al\., 2023](https://arxiv.org/html/2609.05837#bib.bib49)\]\.

At each steptt, the agent’s context is composed of two parts: \(1\) a*long\-term memory*consisting of compressed summaries of all steps prior to the recent window, and \(2\) a*short\-term context*containing the full, uncompressed content of the three most recent steps\(st−2,at−2,ot−2\),…,\(st,at,ot\)\(s\_\{t\-2\},a\_\{t\-2\},o\_\{t\-2\}\),\\ldots,\(s\_\{t\},a\_\{t\},o\_\{t\}\)\. This design preserves fine\-grained details for immediate decision\-making while retaining high\-level progress information from earlier steps\.

Concretely, after each steptt, we prompt the LLM to generate a concise summary of the current step conditioned on the task instruction, all previous step summaries, and the full content of steptt\. The summary captures what was attempted, key results or facts discovered, important constraints or decisions, and concrete identifiers \(paths, URLs, IDs\)\. Each summary is kept under 500 words and is appended to the long\-term memory buffer\. When constructing the context for stept\+1t\{\+\}1, all summaries for steps00tottare concatenated as the long\-term memory prefix, followed by the full content of stepst−2t\{\-\}2,t−1t\{\-\}1, andtt\.

This mechanism is applied uniformly across all methods and models in our experiments, ensuring that performance differences reflect the training approach rather than context management advantages\. The prompt template used for step summarization is provided below\.

Prompt: Step\-Level SummarizationYou are a step\-level summarizer for a multi\-step ReAct\-style agent\.Your task:Given the overall question, all past step summaries, and the full content of the current step, extract only the information from this step that will be useful for understanding what has already been done and making better decisions in future steps\.Overall Question:\{question\}Past Step Summaries:\{past\_summaries\}Current Step Index:\{step\_index\}Full Content of the Current Step:\{step\_full\_content\}Produce a concise summary of this current step only, following these rules:•Focus on: what was attempted or executed; key results or facts discovered; important constraints, assumptions, or decisions; identifiers, paths, or URLs needed later\.•Connect to the question and past progress when relevant, but do not re\-summarize previous steps\.•Omit generic reasoning boilerplate, tool framework noise, and large raw outputs \(summarize their essential points instead\)\.•Use fewer than 500 words\.

### F\.2Training Hyperparameters

We train the policyπθ\\pi\_\{\\theta\}using the PMI\-weighted SFT objective \(Eq\.[6](https://arxiv.org/html/2609.05837#S3.E6)\) implemented as a custom weighted cross\-entropy loss\. For each training sample, the per\-action creditw~t\(n\)\\tilde\{w\}\_\{t\}^\{\(n\)\}is attached as a sample\-level weight; during forward passes, the trainer computes per\-sample token\-averaged cross\-entropy and scales it by the corresponding weight before aggregating across the batch\. During evaluation, weights are disabled to ensure comparable eval loss across runs\.

We fine\-tune Qwen3\-32B using DeepSpeed ZeRO\-Stage 3 on 4×\\timesH100 80GB GPUs with the following configuration: learning rate5×10−65\\times 10^\{\-6\}with cosine scheduling and 10% warmup, per\-device batch size 2 with gradient accumulation over 4 steps \(effective batch size 64\), weight decay 0\.1, maximum sequence length 20,000 tokens, BF16 mixed precision, and gradient checkpointing enabled\. Training runs for 1 epoch over the combined corpus of all three environments\. Samples with zero normalized credit \(w~t\(n\)=0\\tilde\{w\}\_\{t\}^\{\(n\)\}=0\) are filtered out before training, as they contribute no gradient signal\. The 14B transfer experiments \(§[4\.6](https://arxiv.org/html/2609.05837#S4.SS6)\) use identical hyperparameters and training data\.

### F\.3Evaluation Protocol

We adopt the evaluation framework from MCP\-Universe\[[Luo et al\., 2025](https://arxiv.org/html/2609.05837#bib.bib20)\]\. Each test task is paired with a programmatic verifier that inspects the post\-execution environment state \(e\.g\., created resources, modified database rows, updated page content\) against a ground\-truth specification\. The verifier produces a per\-task score in\[0,1\]\[0,1\]reflecting partial completion, andAccis the fraction of tasks achieving a score of1\.01\.0\(full completion\)\.Scoreis the average per\-task score across all test tasks\.

## Appendix GCase Studies

We provide qualitative case studies across three real\-world tool\-use environments: GitHub, Notion, and PostgreSQL\. These examples illustrate how AgentBrew distills useful supervision from imperfect raw trajectories\. In each case, retrospective task inference rewrites the original instruction into an aligned task that better matches the trajectory’s observable outcome, while PMI credit assignment identifies which actions are useful for policy learning\.

Figure 5:Case study on the GitHub environment\.![Refer to caption](https://arxiv.org/html/2609.05837v1/notion.png)Figure 6:Case study on the Notion environment\.Figure 7:Case study on the PostgreSQL environment\.