Rethinking Experience Utilization in Self-Evolving Language Model Agents

# Rethinking Experience Utilization in Self-Evolving Language Model Agents
Source: [https://arxiv.org/html/2605.07164](https://arxiv.org/html/2605.07164)
Weixiang Zhao1, Yingshuo Wang1, Yichen Zhang1, Yanyan Zhao1 Yu Zhang1,Yang Wu2,Dandan Tu2,Bing Qin1,Ting Liu1 1Harbin Institute of Technology,2Huawei Technologies Co\., Ltd \{wxzhao,yswang,yczhang,yyzhao\}@ir\.hit\.edu\.cn

###### Abstract

Self\-evolving agents improve by accumulating and reusing experience from past interactions\. Existing work has largely focused on how experience is constructed, represented, and updated, while paying less attention to how experience should be used during runtime decision\-making\. As a result, most agents rely on rigid usage strategies, either injecting experience once at initialization or at every step, without considering whether it is needed for the current decision\. This paper studies experience utilization as a critical design dimension of self\-evolving agents\. We ask whether agents benefit from interweaving experience use with decision\-making, so that experience is invoked only when additional guidance is needed\. To examine this question, we introduce ExpWeaver, a lightweight instantiation that leaves experience construction unchanged and modifies only runtime utilization by exposing experience as an optional resource during reasoning\. Across four representative frameworks, seven LLM backbones, and three types of environments, ExpWeaver consistently achieves the best performance among different utilization strategies\. Reinforcement learning experiments further show that this behavior can be amplified through training\. Usage\-pattern, causal ablation, and entropy\-based analyses reveal that ExpWeaver enables agents to invoke experience selectively, at beneficial decision points, and under higher reasoning uncertainty\. Overall, our findings call for a shift from merely studying*what*experience to store toward understanding*how*and*when*experience should enter decision\-making\.

## 1Introduction

The emergence of self\-evolving agents marks a critical milestone in the development of autonomous systems capable of continuous learning and adaptive behavior\(Zhaoet al\.,[2024b](https://arxiv.org/html/2605.07164#bib.bib2); Douet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib4); Fanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib23)\)\. As we move into the era of experience\(Silver and Sutton,[2025](https://arxiv.org/html/2605.07164#bib.bib3)\), where learning is increasingly driven by agents’ own interaction histories, this paradigm offers substantial potential for building systems that operate robustly in open\-ended environments\(Hu,[2025](https://arxiv.org/html/2605.07164#bib.bib8); Linet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib9)\)\.

At a high level, self\-evolving agents operate by continuously gathering, storing, and reusing experience from interactions with the environment to inform future decisions\(Gaoet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib1); Caiet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib5); Bellet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib6)\)\. This process can be naturally decomposed into two stages: constructing an experience repository and utilizing it during decision\-making\. Existing research has overwhelmingly focused on the former, with extensive efforts devoted to designing diverse representations of experience, such as distilled insights\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib12); Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15)\), reusable skills\(Wanget al\.,[2024](https://arxiv.org/html/2605.07164#bib.bib10); Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\), or executable workflows\(Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14)\)\. In contrast, the question of how experience should be used at runtime remains comparatively underexplored\.

![Refer to caption](https://arxiv.org/html/2605.07164v1/x1.png)Figure 1:Illustration of different experience utilization strategies in self\-evolving agents\. \(a\)Initialization\-only: experience is injected once at the beginning and remains static throughout the decision process\. \(b\)Always\-on: experience is indiscriminately injected at every step, regardless of its necessity\. \(c\)ExpWeave: experience utilization is interwoven with the decision\-making process, allowing the agent to decide during reasoning when to retrieve and use experience\.In practice, as conceptually illustrated in Figure[1](https://arxiv.org/html/2605.07164#S1.F1)\(a\) and \(b\), existing agents rely on rigid experience usage schemes\. Some methods follow an*initialization\-only*strategy, where experience is injected once at the beginning of task execution\(Zhaoet al\.,[2024a](https://arxiv.org/html/2605.07164#bib.bib13)\); others adopt an*always\-on*strategy, where retrieved experience is indiscriminately injected at every decision step\(Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14)\)\. Although these strategies are simple and easy to implement, they implicitly assume that experience is either globally useful throughout a task or uniformly useful at every step\. However, they do not distinguish whether experience is actually needed for the current decision\. This leaves open a fundamental question: what role should experience play within the agent’s runtime decision\-making process?

This gap motivates us to revisit the role of experience in self\-evolving agents\. In human cognition, experience is not used as static background knowledge, but is tightly coupled with decision\-making: individuals selectively apply relevant memories when facing uncertainty, ambiguity, or increased cognitive control demands\(Shiffrin and Schneider,[1977](https://arxiv.org/html/2605.07164#bib.bib25); Botvinicket al\.,[2001](https://arxiv.org/html/2605.07164#bib.bib24)\)\. Cognitive neuroscience further suggests that memory retrieval is actively regulated based on task demands, with the prefrontal cortex gating access to stored knowledge during goal\-directed behavior\(Tomitaet al\.,[1999](https://arxiv.org/html/2605.07164#bib.bib18); Andersonet al\.,[2004](https://arxiv.org/html/2605.07164#bib.bib19); Bauschet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib17)\)\. Inspired by this mechanism, we hypothesize that:*if agents can similarly couple experience utilization with their decision\-making process, enabling experience to be invoked only when needed, they may make more effective use of accumulated experience\.*

To examine this hypothesis, we proposeExpWeaver, a lightweight paradigm that interweaves experience utilization into the agent’s decision\-making process\. ExpWeaver is not designed to change how experience is constructed, represented, or updated\. Instead, it provides a minimal intervention on the usage side: experience is made available as an optional resource during reasoning\. Concretely, ExpWeaver extends the widely adopted ReAct\-style\(Yaoet al\.,[2023](https://arxiv.org/html/2605.07164#bib.bib20)\)decision\-making paradigm in self\-evolving agents, where decisions are made through iterativereasoning→\\rightarrowactionsteps\. As illustrated in Figure[1](https://arxiv.org/html/2605.07164#S1.F1)\(c\), ExpWeaver introduces a unified loop:reasoning→\\rightarrow\(optional experience utilization\)→\\rightarrowaction\. In this loop, the agent may retrieve relevant experience only when it determines during reasoning that additional guidance is needed\.

This design allows us to systematically study whether regulating experience usage, rather than improving experience representation, is a key missing factor in self\-evolving agents\. To this end, we integrate ExpWeaver into four representative frameworks with diverse forms of experience, spanning both single\-agent\(Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15); Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14); Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\)and multi\-agent settings\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib12)\), and compare it against their vanilla experience usage strategies, which follow either initialization\-only or always\-on schemes\. Following the original evaluation settings of these frameworks, we evaluate different experience utilization strategies across seven LLM backbones and three types of environments, including embodied interaction, web navigation, and knowledge\-intensive question answering\. Across these settings, ExpWeaver consistently outperforms the corresponding vanilla frameworks without modifying existing experience construction pipelines\. These results support our central hypothesis that, beyond how experience is constructed and represented, how experience is utilized during decision\-making is also a critical factor for self\-evolving agents \(§[4\.2](https://arxiv.org/html/2605.07164#S4.SS2)\)\.

We further investigate whether interweaving experience utilization with decision\-making is merely an inference\-time prompting effect or a capability that can be strengthened through learning \(§[4\.3](https://arxiv.org/html/2605.07164#S4.SS3)\)\. To this end, we incorporate ExpWeaver into agentic reinforcement learning, where experience serves both as inference\-time context and as a signal for policy optimization\(Wuet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib26); Zhaiet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib27); Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\)\. Experiments on Qwen3\-4B and Qwen3\-14B\(Yanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib28)\), trained with GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.07164#bib.bib29)\), show that ExpWeaver consistently achieves the best performance among different experience utilization strategies\. This suggests that interweaving experience utilization within decision\-making is a learnable capability that can be further amplified through training\.

To further understand what interweaving experience utilization with decision\-making brings to agents, we conduct a series of analyses that reveal ExpWeaver’s behavior and mechanism\. First, ExpWeaver induces a non\-uniform pattern of experience utilization: agents invoke experience more frequently in demanding tasks and at critical stages of a trajectory, rather than using it uniformly across all steps \(§[5\.1](https://arxiv.org/html/2605.07164#S5.SS1)\)\. Second, targeted ablations show that this pattern is not incidental but causally beneficial\. Removing experience at positions selected by ExpWeaver leads to consistent performance degradation, while random experience utilization fails to yield comparable gains, indicating that the effectiveness of experience depends critically on when it is invoked \(§[5\.2](https://arxiv.org/html/2605.07164#S5.SS2)\)\. Finally, we provide mechanistic evidence that ExpWeaver tends to invoke experience at decision points associated with higher uncertainty, as reflected by increased token\-level entropy during the agent’s reasoning process \(§[5\.3](https://arxiv.org/html/2605.07164#S5.SS3)\)\. Together, these findings suggest that coupling experience utilization with decision\-making enables agents to adaptively regulate experience use in response to decision difficulty\.

Overall, our findings suggest that effective self\-evolution depends not only on accumulating better experience, but also on regulating how experience enters the decision\-making process\. This study highlights experience utilization as a critical design dimension for future self\-evolving agents, shifting the focus from merely storing experience to understanding how and when experience should be used\.

## 2Related Works

Existing works on experience\-driven self\-evolving agents can be broadly discussed from two perspectives: experience construction and experience utilization\.

*Experience construction*studies how agents should represent, store, and update past interactions\. A large body of work focuses on constructing external experience repositories from offline datasets or online interactions\. Offline approaches pre\-construct reusable experience from collected trajectories and keep the memory fixed during inference\(Li and Qiu,[2023](https://arxiv.org/html/2605.07164#bib.bib47); Yanget al\.,[2023](https://arxiv.org/html/2605.07164#bib.bib49); Zhonget al\.,[2024](https://arxiv.org/html/2605.07164#bib.bib48); Zhaoet al\.,[2024a](https://arxiv.org/html/2605.07164#bib.bib13); Fuet al\.,[2024](https://arxiv.org/html/2605.07164#bib.bib21); Zhouet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib50); Yanget al\.,[2025b](https://arxiv.org/html/2605.07164#bib.bib52)\)\. Online approaches further allow the experience repository to evolve during deployment, enabling agents to accumulate, retrieve, and refine experience through interaction\(Chenet al\.,[2024](https://arxiv.org/html/2605.07164#bib.bib53); Zhanget al\.,[2025b](https://arxiv.org/html/2605.07164#bib.bib54); Suzgunet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib51); Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15)\)\. Representative systems such as ReasoningBank continuously distill reasoning patterns from recent episodes, while G\-Memory extends online memory accumulation to multi\-agent settings\(Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15); Zhanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib12)\)\. These studies have produced diverse experience forms, including distilled insights, skills, workflows, and raw trajectories\.

In contrast,*experience utilization*concerns how constructed experience should enter the agent’s runtime decision\-making process\. This aspect remains comparatively underexplored\. Recent concurrent studies have begun to explore more adaptive experience usage, such as proactive retrieval in lifelong agents\(Caiet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib55)\)and self\-triggered experience seeking in web agents\(Zhanget al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib56)\)\. However, since these works introduce their own experience representations and construction pipelines, the impact of experience utilization is intertwined with changes in how experience is formed, making it difficult to cleanly isolate the effect of*how*experience is used\.

Our work instead takes a higher\-level view by rethinking the role of experience in self\-evolving agents: experience should not merely serve as static auxiliary context, but as an active component of decision\-making\. To isolate this question, we keep existing experience construction, representation, and update pipelines unchanged, and only modifies runtime utilization\. This allows us to identify experience utilization as a critical design dimension, showing that effective self\-evolution depends not only on what experience is stored, but also on how and when it enters decision\-making\.

## 3Interweaving Experience Utilization within Decision\-Making

In this section, we first formulate experience\-driven self\-evolution as a two\-stage process consisting of experience construction and experience utilization \(§[3\.1](https://arxiv.org/html/2605.07164#S3.SS1)\)\. We then introduce ExpWeaver, a lightweight intervention that interweaves experience utilization with the agent’s reasoning process, enabling us to examine whether experience can be invoked when needed during decision\-making \(§[3\.2](https://arxiv.org/html/2605.07164#S3.SS2)\)\. Finally, we extend this formulation to agentic reinforcement learning to study whether such usage behavior can be further optimized as a learnable capability \(§[3\.3](https://arxiv.org/html/2605.07164#S3.SS3)\)\.

### 3\.1Preliminaries

We study self\-evolving agents, where behavioral adaptation is achieved by accumulating and reusing past interactions without modifying model parameters\(Gaoet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib1)\)\. This process consists of two stages: constructing an experience repository and utilizing it during decision\-making\.

#### Experience Construction\.

After each interaction, the agent produces a trajectoryτ\\tauand receives feedbackrr\. From each\(τ,r\)\(\\tau,r\)pair, an experience unitEEis distilled and stored in an external memory:

M=\{E1,E2,…,En\}\.M=\\\{E\_\{1\},E\_\{2\},\\dots,E\_\{n\}\\\}\.Different frameworks instantiateEEas distilled insights\(Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15)\), reusable skills\(Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\), or executable workflows\(Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14)\), with corresponding memory update rules\. Most works focus on this stage, improving how experience is represented, organized, and maintained\.

#### Experience Utilization\.

Experience utilization determines how the constructed experience memoryMMenters runtime decision\-making, and is largely orthogonal to experience construction\. Given a task inputxx, existing methods retrieve relevant experienceM​\(x\)⊆MM\(x\)\\subseteq Mand typically use it in fixed and rigid ways\. One strategy is*initialization\-only*usage:

y=πθ​\(\[x;M​\(x\)\]\),y=\\pi\_\{\\theta\}\(\[x;M\(x\)\]\),where experience is injected once\. Another is*always\-on*usage in multi\-step decision\-making:

at=πθ​\(\[st;M​\(st\)\]\),a\_\{t\}=\\pi\_\{\\theta\}\(\[s\_\{t\};M\(s\_\{t\}\)\]\),where experience is injected at every steptt\.

Both strategies predefine when experience is used, rather than allowing the agent to decide whether it is needed for the current decision\. This motivates us to study experience utilization as a runtime decision process: how should experience selectively enter decision\-making when useful?

### 3\.2Prompting\-based ExpWeaver

To study this question, we proposeExpWeaver, a lightweight paradigm that interweaves experience utilization with the agent’s decision\-making process\. ExpWeaver leaves experience construction, representation, and updating unchanged, and only modifies runtime utilization by exposing experience as an optional resource during reasoning\.

#### From Fixed Usage to Decision\-Time Utilization\.

We build on the widely adopted ReAct\-style\(Yaoet al\.,[2023](https://arxiv.org/html/2605.07164#bib.bib20)\)paradigm, where agents follow an iterativereasoning→\\rightarrowactionloop\. At steptt, the agent maintains a contextctc\_\{t\}and generates an actionat∼πθ\(⋅∣ct\)a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{t\}\)\.

In existing agents, experience utilization is external to this loop: experience is injected either before execution or at every step\. ExpWeaver instead places experience utilization inside thereasoningprocess, making the decision of whether to use experience conditioned on the agent’s reasoning state\.

#### Interweaving Experience into Reasoning\.

ExpWeaver implements this idea through a minimal prompting\-based intervention\. We augment the system prompt with an instruction that reminds the agent that it may consult past experience during reasoning if additional guidance is needed\. When the agent decides to use experience, it emits a special trigger token,\[Retrieve\], which activates retrieval\. The full prompt template is provided in Appendix[D\.1](https://arxiv.org/html/2605.07164#A4.SS1)\.

Formally, at each steptt, the agent first produces an intermediate reasoning trace:

ht=πθ​\(ct\),h\_\{t\}=\\pi\_\{\\theta\}\(c\_\{t\}\),which reflects its current understanding of the state\. We then check whetherhth\_\{t\}contains the trigger token\. If the trigger appears, the current context is used as a retrieval query:

M​\(ct\)=Retrieve​\(M,ct\),M\(c\_\{t\}\)=\\text\{Retrieve\}\(M,c\_\{t\}\),and the retrieved experience is incorporated for subsequent action generation:

at=πθ​\(\[ct;ht;M​\(ct\)\]\)\.a\_\{t\}=\\pi\_\{\\theta\}\(\[c\_\{t\};h\_\{t\};M\(c\_\{t\}\)\]\)\.If the trigger token does not appear, the agent proceeds without additional experience incorporated\. Overall, this yields the following decision loop:

reasoning→\(optional experience utilization\)→action\.\\texttt\{reasoning\}\\rightarrow\\texttt\{\(optional experience utilization\)\}\\rightarrow\\texttt\{action\}\.

#### Role of ExpWeaver\.

ExpWeaver should be viewed as a minimal instantiation for studying runtime experience utilization, rather than as a modification to experience construction\. It introduces three useful properties for analysis:\(1\) Adaptive usage\.Experience is invoked during reasoning only when the agent determines that additional guidance may be useful\.\(2\) Orthogonality\.ExpWeaver is independent of how experience is represented or updated, making it applicable to diverse self\-evolving frameworks\.\(3\) Observability\.Because retrieval is triggered explicitly within the reasoning process, we can inspect when experience is used and analyze whether such usage is meaningful\.

### 3\.3Agentic Reinforcement Learning\-based ExpWeaver

The prompting\-based setup tests whether existing agents can be guided to regulate experience usage at inference time\. We further ask whether this behavior can be strengthened through learning\. To this end, we integrate ExpWeaver into experience\-driven agentic reinforcement learning\.

#### Policy Optimization with GRPO\.

We train the policy using Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2605.07164#bib.bib29)\)\. For an inputxx, the agent samplesGGtrajectories\{τ\(1\),…,τ\(G\)\}\\\{\\tau^\{\(1\)\},\\dots,\\tau^\{\(G\)\}\\\}\. Each trajectory receives a binary rewardRi=r​\(τ\(i\)\)∈\{0,1\}R\_\{i\}=r\(\\tau^\{\(i\)\}\)\\in\\\{0,1\\\}indicating whether the task is successfully completed\. The policy is updated with a PPO\-style\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.07164#bib.bib30)\)clipped objective:

𝒥​\(θ\)=𝔼​\[1G​∑i=1Gmin⁡\(ri​Ai,clip​\(ri,1−ϵ,1\+ϵ\)​Ai\)−β​DKL​\(πθ∥πref\)\],\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\min\\big\(r\_\{i\}A\_\{i\},\\text\{clip\}\(r\_\{i\},1\-\\epsilon,1\+\\epsilon\)A\_\{i\}\\big\)\-\\beta D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\Bigg\],whereAiA\_\{i\}is computed from intra\-group normalized rewards, and the KL penalty keeps the updated policy close to the reference policy\.

## 4How Should Experience Be Utilized in Self\-Evolving Agents?

### 4\.1Experimental Setup

Experience Utilization Strategies\.We compare four experience utilization strategies\.w/o Experienceremoves the experience repository and performs task execution without using any past experience\.Initialization\-only \(Init\-only\)retrieves relevant experience once at the beginning of task execution and keeps it as static context throughout the trajectory\.Always\-onretrieves and injects experience at every decision step, regardless of whether it is needed for the current state\.ExpWeaverinterweaves experience utilization with the decision\-making process, allowing the agent to invoke experience during reasoning only when additional guidance is needed\.

Agent Framework\.We implement the four experience utilization strategies described above across four representative self\-evolving frameworks\. These frameworks cover diverse forms of experience representation:ReasoningBank\(Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15)\)uses distilled insights,AWM\(Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14)\)represents experience as executable workflows,SkillRL\(Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\)models experience as reusable skills, andG\-Memory\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib12)\)combines raw trajectories with distilled insights\. G\-Memory operates in a*multi\-agent*setting, while the others are*single\-agent*frameworks\.

The vanilla versions of ReasoningBank, SkillRL, and G\-Memory follow an Init\-only usage strategy\. AWM does not include an explicit retrieval mechanism; therefore, its Init\-only and Always\-on variants are equivalent in practice\. These frameworks also cover both*offline*and*online*self\-evolving paradigms: SkillRL relies on a pre\-constructed experience repository, whereas the other frameworks dynamically accumulate experience through interaction with the environment\.

This selection spans a broad spectrum of experience representations, agent settings, and evolution paradigms, enabling a comprehensive evaluation of how different experience utilization strategies affect self\-evolving agents\. Further details are provided in Appendix[B](https://arxiv.org/html/2605.07164#A2)\.

Backbone Model\.We conduct experiments across a diverse set of seven LLMs from multiple families, includingGPT\-5\.2\(Singhet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib31)\),DeepSeek\-V4\-Pro\(DeepSeek\-AI,[2026](https://arxiv.org/html/2605.07164#bib.bib32)\), andKimi\-K2\.5\(Team,[2026](https://arxiv.org/html/2605.07164#bib.bib33)\), as well as open\-weight Qwen series includingQwen3\.5\-397B\-A17B\(Qwen Team,[2026](https://arxiv.org/html/2605.07164#bib.bib34)\)and severalQwen3variants \(32B, 14B and 4B\)\(Yanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib28)\), covering a wide range of model scales and architectures for comprehensive evaluation\. Due to computational constraints, we conduct reinforcement learning experiments only on Qwen3\-14B and Qwen3\-4B\.

Environment & Benchmark\.We evaluate across 8 benchmarks in 3 environments: \(1\) For*embodied interaction*, we adopt the interactive environmentALFWorld\(Shridharet al\.,[2021](https://arxiv.org/html/2605.07164#bib.bib43)\)\. \(2\) For*web navigation*, we evaluate onWebShop\(Yaoet al\.,[2022](https://arxiv.org/html/2605.07164#bib.bib42)\)\. \(3\) For*knowledge\-intensive question answering \(QA\)*, we includeHotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2605.07164#bib.bib35)\),NQ\(Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2605.07164#bib.bib37)\)TriviaQA\(Joshiet al\.,[2017](https://arxiv.org/html/2605.07164#bib.bib38)\),2Wiki\(Hoet al\.,[2020](https://arxiv.org/html/2605.07164#bib.bib39)\),MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2605.07164#bib.bib40)\)andBamboogle\(Presset al\.,[2023](https://arxiv.org/html/2605.07164#bib.bib41)\)\. These benchmarks span diverse environments and task types, providing a comprehensive evaluation of experience\-driven adaptation \(details in Appendix[C](https://arxiv.org/html/2605.07164#A3)\)\.

Implementation Details\.For prompting\-based experiments, we access those models via official APIs\. All agentic reinforcement learning experiments are conducted on a cluster of eight NVIDIA A100 80GB GPUs\. Further implementation details are provided in Appendix[D](https://arxiv.org/html/2605.07164#A4)\.

![Refer to caption](https://arxiv.org/html/2605.07164v1/x2.png)Figure 2:Prompting\-based results under the ReasoningBank framework on ALFWorld and WebShop\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x3.png)Figure 3:Prompting\-based results under the SkillRL framework on ALFWorld and WebShop\.
### 4\.2Results on Prompting\-based ExpWeaver

Figures[2](https://arxiv.org/html/2605.07164#S4.F2)and[3](https://arxiv.org/html/2605.07164#S4.F3)present the prompting\-based results on ALFWorld and WebShop across ReasoningBank and SkillRL frameworks\. Due to space constraints, results on knowledge\-intensive QA tasks and the full results for G\-Memory and AWM are provided in Appendix[E\.1](https://arxiv.org/html/2605.07164#A5.SS1)\. Overall, ExpWeaver consistently achieves the best performance among different experience utilization strategies across models, environments, and experience representations\. This demonstrates the efficacy of interweaving experience utilization into the decision\-making process\. We highlight the following key observations:

Consistent gains over rigid experience usage strategies\.ExpWeaver consistently improves over both*Init\-only*and*Always\-on*baselines across nearly all settings\. This trend holds across different backbones and environments\. The results indicate that neither under\-utilization nor over\-utilization of experience is optimal; instead, integrating experience utilization into the decision\-making process leads to more reasonable behavior and effective performance\.

Robust effectiveness across diverse experience representations\.ExpWeaver achieves consistent improvements across frameworks with fundamentally different experience forms, including distilled insights \(ReasoningBank\), workflows \(AWM\), skills \(SkillRL\), and hybrid memory \(G\-Memory\)\. This demonstrates that the benefits of interweaving experience utilization into the decision\-making process are largely independent of how experience is represented, supporting the claim that experience usage, rather than representation, is the key bottleneck\.

Strong generalization across agent settings and self\-evolving paradigms\.ExpWeaver generalizes across both single\-agent and multi\-agent settings \(i\.e\., G\-Memory\), as well as offline and online self\-evolving paradigms\. This is notable given their distinct experience construction and update mechanisms\. In particular, SkillRL provides a controlled offline setting where ExpWeaver uses the same pre\-constructed experience repository as the vanilla baseline and only changes runtime utilization\. The gains in this setting indicate that the improvement comes from regulating experience usage itself, supporting ExpWeaver as a general mechanism across heterogeneous agent designs\.

![Refer to caption](https://arxiv.org/html/2605.07164v1/x4.png)Figure 4:RL results of Qwen3\-4B on ALFWorld after GRPO training\.
### 4\.3Results on RL\-based ExpWeaver

We further investigate whether interweaving experience utilization with the decision\-making process can be strengthened through agentic reinforcement learning\. Specifically, we adopt the ReasoningBank framework and train Qwen3\-4B on the ALFWorld environment with GRPO under four experience utilization strategies\. Detailed training settings are provided in Appendix[D\.2](https://arxiv.org/html/2605.07164#A4.SS2)\.

Figure[4](https://arxiv.org/html/2605.07164#S4.F4)shows that the way experience is used substantially affects RL performance\. All experience\-based variants outperform w/o Experience, confirming the value of experience under GRPO training\. Yet Always\-on underperforms Init\-only, suggesting that excessive experience exposure can introduce interference\. ExpWeaver achieves the highest success rate, improving over Init\-only by 5\.3 points and Always\-on by 8\.0 points\. This indicates that RL benefits most when experience utilization is interwoven with decision\-making rather than fixed as static context or injected at every step\.

We also observe consistent trends on knowledge\-intensive QA with Qwen3\-14B\. As shown in Appendix[E\.2](https://arxiv.org/html/2605.07164#A5.SS2), ExpWeaver generally achieves stronger learning dynamics and higher final success rates than the vanilla baseline across QA benchmarks such as NQ, TriviaQA, Bamboogle, and 2Wiki\. These results suggest that the learnability of experience utilization is not limited to embodied interaction, but also generalizes to reasoning\-intensive QA scenarios\.

## 5Analysis and Discussions

Table 1:Average number of experience retrievals per sample under ExpWeaver\. We report results on ReasoningBank and G\-Memory\.### 5\.1Usage Pattern Analysis of ExpWeaver

To understand how ExpWeaver utilizes experience in practice, we analyze its usage patterns from two perspectives: \(1\) a*horizontal comparison*across tasks, measured by the average number of experience retrievals per sample; and \(2\) a*temporal analysis*within trajectories, measured by the fraction of trajectories that invoke experience at each step\.

Horizontal comparison across tasks\.Table[1](https://arxiv.org/html/2605.07164#S5.T1)reports the average number of experience retrievals per sample under two representative frameworks, ReasoningBank and G\-Memory\. Additional results on AWM and SkillRL are provided in Appendix[F\.1](https://arxiv.org/html/2605.07164#A6.SS1)\. Overall, ExpWeaver exhibits a clear task\-aware, model\-aware, and framework\-agnostic usage pattern\.

First, experience usage is highly task\-dependent\. Across most frameworks and backbones, ALFWorld induces substantially more retrievals than WebShop and HotpotQA, suggesting that experience is more frequently needed in embodied interaction, where agents must perform long\-horizon planning, track partial observations, and recover from uncertain states\. Second, retrieval frequency tends to decrease as model capability increases\. Stronger backbone models such as GPT\-5\.2 and Qwen3\.5\-397B\-A17B rarely invoke experience, while smaller models such as Qwen3\-32B rely on it more often\. This indicates that ExpWeaver does not mechanically retrieve experience, but treats it as a complementary resource when the model requires additional guidance\. Third, similar patterns appear across different self\-evolving frameworks, despite their distinct experience representations\. This suggests that the observed behavior is not tied to a specific memory format, but is induced by interweaving experience utilization into the decision\-making process\.

![Refer to caption](https://arxiv.org/html/2605.07164v1/x5.png)Figure 5:Temporal pattern of experience utilization under ExpWeaver on ALFWorld with the ReasoningBank framework\.Temporal analysis within decision trajectories\.Figure[5](https://arxiv.org/html/2605.07164#S5.F5)reports the fraction of trajectories that invoke experience at each interaction step in ALFWorld under the ReasoningBank framework\. Experience utilization shows a clearly non\-uniform temporal pattern: retrieval is concentrated at the first step, drops sharply afterward, and occasionally resurges at later stages\. This suggests that agents mainly consult experience when task understanding and high\-level planning are most critical, rather than using it uniformly throughout the trajectory\.

Importantly, later retrievals are not random\. In ALFWorld, they often occur during object\-search phases, where the agent must locate target items under partial observability\. When search attempts fail repeatedly, experience is re\-invoked to provide corrective guidance\. This suggests that experience utilization may be associated with moments of increased decision uncertainty, a hypothesis we further examine through entropy\-based analysis in §[5\.3](https://arxiv.org/html/2605.07164#S5.SS3)\. Additional temporal analyses across more frameworks and tasks are provided in Appendix[F\.2](https://arxiv.org/html/2605.07164#A6.SS2), and specific qualitative examples of ExpWeaver on different tasks are also provided in Appendix[F\.3](https://arxiv.org/html/2605.07164#A6.SS3)\.

Table 2:Causal ablation of ExpWeaver\-invoked experience utilization\. Empty removes retrieved experience at positions where ExpWeaver chooses to invoke it, while Random Utilization invokes experience at randomly selected positions\.
### 5\.2Causal Effect of ExpWeaver\-Invoked Experience Utilization

We further examine whether the observed ExpWeaver’s non\-uniform usage pattern is causally beneficial\. We compare ExpWeaver with two ablations:Empty, which removes retrieved experience at positions invoked by ExpWeaver, andRandom Utilization, which invokes experience at random positions\. For Random Utilization, we first compute the average number of experience invocations and the average trajectory length of ExpWeaver on each benchmark, and then invoke experience at each step with probability given by their ratio\.

As shown in Table[2](https://arxiv.org/html/2605.07164#S5.T2), both ablations consistently underperform ExpWeaver\. We focus on Qwen3\-32B and DeepSeek\-V4\-Pro because these two backbones invoke experience more frequently under ExpWeaver, providing enough usage events for reliable ablation analysis\. For Qwen3\-32B, removing experience leads to clear drops across ReasoningBank and G\-Memory, while random utilization also fails to match ExpWeaver\. DeepSeek\-V4\-Pro exhibits the same trend, with ExpWeaver consistently outperforming both ablation variants across all evaluated settings\.

These results provide causal evidence that interweaving experience utilization with decision\-making is effective because the agent learns*when*experience should enter the reasoning process\.

### 5\.3Mechanistic Analysis: Experience Usage under Uncertainty

![Refer to caption](https://arxiv.org/html/2605.07164v1/x6.png)Figure 6:Mechanistic analysis of experience usage in ExpWeaver\. The blue curve shows the mean token entropy at each reasoning step, while red points denote steps where experience is invoked\.We further analyze what happens inside the agent when ExpWeaver decides to invoke experience\. Our hypothesis is that experience usage is triggered when the agent encounters higher decision uncertainty\. To examine this, we measure the token\-level entropy of the agent’s output distribution during each reasoning step, and compare the entropy at positions with and without experience usage\.

Figure[6](https://arxiv.org/html/2605.07164#S5.F6)presents the entropy analysis results on ALFWorld\. The blue curve denotes the mean token entropy at each step, while red points indicate steps where ExpWeaver invokes experience\. We observe that experience usage tends to occur at entropy peaks\. In contrast, steps without experience usage generally correspond to lower\-entropy regions\. This suggests that ExpWeaver is more likely to retrieve experience when the agent is uncertain about its next decision\.

## 6Conclusion

This work studies how experience should be utilized in self\-evolving agents\. We show that existing rigid usage schemes overlook an important design dimension: when and how experience should enter decision\-making\. To examine this question, we introduce ExpWeaver, a lightweight instantiation that keeps experience construction unchanged while modifying runtime utilization\. Across diverse frameworks, models, and environments, ExpWeaver consistently outperforms rigid usage strategies\. Further analyses show that agents invoke experience selectively, at causally beneficial positions, and under higher reasoning uncertainty\. Overall, our findings suggest that effective self\-evolution depends not only on what experience is stored, but also on how and when it is used\.

## References

- M\. C\. Anderson, K\. N\. Ochsner, B\. Kuhl, J\. Cooper, E\. Robertson, S\. W\. Gabrieli, G\. H\. Glover, and J\. D\. Gabrieli \(2004\)Neural systems underlying the suppression of unwanted memories\.Science303\(5655\),pp\. 232–235\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p4.1)\.
- Distinct neuronal populations in the human brain combine content and context\.Nature,pp\. 1–11\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p4.1)\.
- J\. Bell, L\. Quarantiello, E\. N\. Coleman, L\. Li, M\. Li, M\. Madeddu, E\. Piccoli, and V\. Lomonaco \(2025\)The future of continual learning in the era of foundation models: three key directions\.arXiv preprint arXiv:2506\.03320\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p2.1)\.
- M\. M\. Botvinick, T\. S\. Braver, D\. M\. Barch, C\. S\. Carter, and J\. D\. Cohen \(2001\)Conflict monitoring and cognitive control\.\.Psychological review108\(3\),pp\. 624\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p4.1)\.
- Y\. Cai, Y\. Hao, J\. Zhou, H\. Yan, Z\. Lei, R\. Zhen, Z\. Han, Y\. Yang, J\. Li, Q\. Pan,et al\.\(2025\)Building self\-evolving agents via experience\-driven lifelong learning: a framework and benchmark\.arXiv preprint arXiv:2508\.19005\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p2.1)\.
- Y\. Cai, J\. Zhou, Q\. Chen, and L\. He \(2026\)Ask only when needed: proactive retrieval from memory and skills for experience\-driven lifelong agents\.arXiv preprint arXiv:2604\.20572\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p3.1)\.
- M\. Chen, Y\. Li, Y\. Yang, S\. Yu, B\. Lin, and X\. He \(2024\)Automanual: constructing instruction manuals by llm agents via interactive environmental learning\.Advances in Neural Information Processing Systems37,pp\. 589–631\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p5.1)\.
- S\. Dou, M\. Zhang, C\. Huang, J\. Chen, F\. Chen, S\. Liu, Y\. Liu, C\. Liu, C\. ZHONG, Z\. Zhang,et al\.\(2025\)EvaLearn: quantifying the learning capability and efficiency of llms via sequential problem solving\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p1.1)\.
- J\. Fang, Y\. Peng, X\. Zhang, Y\. Wang, X\. Yi, G\. Zhang, Y\. Xu, B\. Wu, S\. Liu, Z\. Li,et al\.\(2025\)A comprehensive survey of self\-evolving ai agents: a new paradigm bridging foundation models and lifelong agentic systems\.arXiv preprint arXiv:2508\.07407\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p1.1)\.
- L\. Feng, Z\. Xue, T\. Liu, and B\. An \(2025\)Group\-in\-group policy optimization for llm agent training\.arXiv preprint arXiv:2505\.10978\.Cited by:[§D\.2](https://arxiv.org/html/2605.07164#A4.SS2.p1.1)\.
- Y\. Fu, D\. Kim, J\. Kim, S\. Sohn, L\. Logeswaran, K\. Bae, and H\. Lee \(2024\)Autoguide: automated generation and selection of context\-aware guidelines for large language model agents\.Advances in Neural Information Processing Systems37,pp\. 119919–119948\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- H\. Gao, J\. Geng, W\. Hua, M\. Hu, X\. Juan, H\. Liu, S\. Liu, J\. Qiu, X\. Qi, Y\. Wu,et al\.\(2025\)A survey of self\-evolving agents: on path to artificial super intelligence\.arXiv preprint arXiv:2507\.21046\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p2.1),[§3\.1](https://arxiv.org/html/2605.07164#S3.SS1.p1.1)\.
- X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,pp\. 6609–6625\.Cited by:[3rd item](https://arxiv.org/html/2605.07164#A3.I3.i3.p1.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- B\. Hu \(2025\)On improvisation and open\-endedness: insights for experiential ai\.arXiv preprint arXiv:2511\.00529\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p1.1)\.
- B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. O\. Arik, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.InSecond Conference on Language Modeling,Cited by:[§E\.2](https://arxiv.org/html/2605.07164#A5.SS2.p1.1)\.
- M\. Joshi, E\. Choi, D\. S\. Weld, and L\. Zettlemoyer \(2017\)Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1601–1611\.Cited by:[2nd item](https://arxiv.org/html/2605.07164#A3.I3.i2.p1.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.\(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.Cited by:[4th item](https://arxiv.org/html/2605.07164#A3.I3.i4.p1.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- X\. Li and X\. Qiu \(2023\)MoT: memory\-of\-thought enables chatgpt to self\-improve\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 6354–6374\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- M\. Lin, H\. Lu, Z\. Shi, B\. He, R\. Mao, Z\. Zhang, Z\. Wu, X\. Tang, H\. Liu, Z\. Dai,et al\.\(2026\)Position: agentic evolution is the path to evolving llms\.arXiv preprint arXiv:2602\.00359\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p1.1)\.
- S\. Ouyang, J\. Yan, I\. Hsu, Y\. Chen, K\. Jiang, Z\. Wang, R\. Han, L\. T\. Le, S\. Daruki, X\. Tang,et al\.\(2025\)Reasoningbank: scaling agent self\-evolving with reasoning memory\.arXiv preprint arXiv:2509\.25140\.Cited by:[1st item](https://arxiv.org/html/2605.07164#A2.I1.i1.p1.1),[Appendix B](https://arxiv.org/html/2605.07164#A2.p1.1),[1st item](https://arxiv.org/html/2605.07164#A4.I1.i1.p1.1),[§1](https://arxiv.org/html/2605.07164#S1.p2.1),[§1](https://arxiv.org/html/2605.07164#S1.p6.1),[§2](https://arxiv.org/html/2605.07164#S2.p2.1),[§3\.1](https://arxiv.org/html/2605.07164#S3.SS1.SSS0.Px1.p1.5),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p2.1)\.
- W\. Pan, S\. Liu, X\. Zhou, S\. Zhang, W\. Shi, M\. Xu, and X\. Jia \(2026\)M\-star: every task deserves its own memory harness\.arXiv preprint arXiv:2604\.11811\.Cited by:[Appendix A](https://arxiv.org/html/2605.07164#A1.SS0.SSS0.Px1.p3.1)\.
- O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. Lewis \(2023\)Measuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5687–5711\.Cited by:[5th item](https://arxiv.org/html/2605.07164#A3.I3.i5.p1.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p5.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§3\.3](https://arxiv.org/html/2605.07164#S3.SS3.SSS0.Px1.p1.4)\.
- S\. Shao, Q\. Ren, C\. Qian, B\. Wei, D\. Guo, Y\. JingYi, X\. Song, L\. Zhang, W\. Zhang, D\. Liu,et al\.\(2025\)Your agent may misevolve: emergent risks in self\-evolving llm agents\.InSocially Responsible and Trustworthy Foundation Models at NeurIPS 2025,Cited by:[Appendix A](https://arxiv.org/html/2605.07164#A1.SS0.SSS0.Px1.p4.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p7.1),[§3\.3](https://arxiv.org/html/2605.07164#S3.SS3.SSS0.Px1.p1.4)\.
- R\. M\. Shiffrin and W\. Schneider \(1977\)Controlled and automatic human information processing: ii\. perceptual learning, automatic attending and a general theory\.\.Psychological review84\(2\),pp\. 127\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p4.1)\.
- M\. Shridhar, J\. Thomason, D\. Gordon, Y\. Bisk, W\. Han, R\. Mottaghi, L\. Zettlemoyer, and D\. Fox \(2020\)Alfred: a benchmark for interpreting grounded instructions for everyday tasks\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 10740–10749\.Cited by:[1st item](https://arxiv.org/html/2605.07164#A3.I1.i1.p1.1)\.
- M\. Shridhar, X\. Yuan, M\. Cote, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2021\)ALFWorld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations,Cited by:[1st item](https://arxiv.org/html/2605.07164#A3.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- D\. Silver and R\. S\. Sutton \(2025\)Welcome to the era of experience\.Google AI1\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p1.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p5.1)\.
- M\. Suzgun, M\. Yuksekgonul, F\. Bianchi, D\. Jurafsky, and J\. Zou \(2025\)Dynamic cheatsheet: test\-time learning with adaptive memory\.arXiv preprint arXiv:2504\.07952\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- K\. Team \(2026\)Kimi k2\.5: visual agentic intelligence\.External Links:2602\.02276Cited by:[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p5.1)\.
- H\. Tomita, M\. Ohbayashi, K\. Nakahara, I\. Hasegawa, and Y\. Miyashita \(1999\)Top\-down signal from prefrontal cortex in executive control of memory retrieval\.Nature401\(6754\),pp\. 699–703\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p4.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.Cited by:[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p2.1)\.
- Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. Neubig \(2025\)Agent workflow memory\.InInternational Conference on Machine Learning,pp\. 63897–63911\.Cited by:[2nd item](https://arxiv.org/html/2605.07164#A2.I1.i2.p1.1),[Appendix B](https://arxiv.org/html/2605.07164#A2.p1.1),[2nd item](https://arxiv.org/html/2605.07164#A4.I1.i2.p1.1),[§1](https://arxiv.org/html/2605.07164#S1.p2.1),[§1](https://arxiv.org/html/2605.07164#S1.p3.1),[§1](https://arxiv.org/html/2605.07164#S1.p6.1),[§3\.1](https://arxiv.org/html/2605.07164#S3.SS1.SSS0.Px1.p1.5),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p2.1)\.
- R\. Wu, X\. Wang, J\. Mei, P\. Cai, D\. Fu, C\. Yang, L\. Wen, X\. Yang, Y\. Shen, Y\. Wang,et al\.\(2025\)Evolver: self\-evolving llm agents through an experience\-driven lifecycle\.arXiv preprint arXiv:2510\.16079\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p7.1)\.
- P\. Xia, J\. Chen, H\. Wang, J\. Liu, K\. Zeng, Y\. Wang, S\. Han, Y\. Zhou, X\. Zhao, H\. Chen,et al\.\(2026\)SkillRL: evolving agents via recursive skill\-augmented reinforcement learning\.InICLR 2026 Workshop on Memory for LLM\-Based Agentic Systems,Cited by:[3rd item](https://arxiv.org/html/2605.07164#A2.I1.i3.p1.1),[Appendix B](https://arxiv.org/html/2605.07164#A2.p1.1),[4th item](https://arxiv.org/html/2605.07164#A4.I1.i4.p1.3),[§1](https://arxiv.org/html/2605.07164#S1.p2.1),[§1](https://arxiv.org/html/2605.07164#S1.p6.1),[§1](https://arxiv.org/html/2605.07164#S1.p7.1),[§3\.1](https://arxiv.org/html/2605.07164#S3.SS1.SSS0.Px1.p1.5),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p2.1)\.
- Y\. Xiong, S\. Hu, and J\. Clune \(2026\)Learning to continually learn via meta\-learning agentic memory designs\.arXiv preprint arXiv:2602\.07755\.Cited by:[Appendix A](https://arxiv.org/html/2605.07164#A1.SS0.SSS0.Px1.p3.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p7.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p5.1)\.
- W\. Yang, J\. Xiao, H\. Zhang, Q\. Zhang, Y\. Wang, and B\. Xu \(2025b\)Coarse\-to\-fine grounded memory for llm agent planning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 13040–13067\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- Z\. Yang, P\. Li, and Y\. Liu \(2023\)Failures pave the way: enhancing large language models through tuning\-free rule accumulation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 1751–1777\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[1st item](https://arxiv.org/html/2605.07164#A3.I3.i1.p1.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan \(2022\)Webshop: towards scalable real\-world web interaction with grounded language agents\.Advances in Neural Information Processing Systems35,pp\. 20744–20757\.Cited by:[1st item](https://arxiv.org/html/2605.07164#A3.I2.i1.p1.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p6.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p5.3),[§3\.2](https://arxiv.org/html/2605.07164#S3.SS2.SSS0.Px1.p1.4)\.
- Y\. Zhai, S\. Tao, C\. Chen, A\. Zou, Z\. Chen, Q\. Fu, S\. Mai, L\. Yu, J\. Deng, Z\. Cao,et al\.\(2025\)Agentevolver: towards efficient self\-evolving agent system\.arXiv preprint arXiv:2511\.10395\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p7.1)\.
- G\. Zhang, M\. Fu, K\. Wang, G\. Wan, M\. Yu, and S\. YAN \(2025a\)G\-memory: tracing hierarchical memory for multi\-agent systems\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[4th item](https://arxiv.org/html/2605.07164#A2.I1.i4.p1.1),[Appendix B](https://arxiv.org/html/2605.07164#A2.p1.1),[3rd item](https://arxiv.org/html/2605.07164#A4.I1.i3.p1.1),[§1](https://arxiv.org/html/2605.07164#S1.p2.1),[§1](https://arxiv.org/html/2605.07164#S1.p6.1),[§2](https://arxiv.org/html/2605.07164#S2.p2.1),[§4\.1](https://arxiv.org/html/2605.07164#S4.SS1.p2.1)\.
- Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li,et al\.\(2025b\)Agentic context engineering: evolving contexts for self\-improving language models\.arXiv preprint arXiv:2510\.04618\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- W\. Zhang, X\. Zhang, H\. Yu, S\. Nie, B\. Wu, J\. Yue, T\. Liu, and Y\. Li \(2026\)ExpSeek: self\-triggered experience seeking for web agents\.arXiv preprint arXiv:2601\.08605\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p3.1)\.
- A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang \(2024a\)Expel: llm agents are experiential learners\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19632–19642\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p3.1),[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- W\. Zhao, S\. Wang, Y\. Hu, Y\. Zhao, B\. Qin, X\. Zhang, Q\. Yang, D\. Xu, and W\. Che \(2024b\)SAPT: a shared attention framework for parameter\-efficient continual learning of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11641–11661\.Cited by:[§1](https://arxiv.org/html/2605.07164#S1.p1.1)\.
- W\. Zhao, Y\. Wang, Y\. Zhang, Y\. Deng, Y\. Zhao, W\. Che, B\. Qin, and T\. Liu \(2026a\)Large language model agents are not always faithful self\-evolvers\.arXiv preprint arXiv:2601\.22436\.Cited by:[Appendix A](https://arxiv.org/html/2605.07164#A1.SS0.SSS0.Px1.p4.1)\.
- W\. Zhao, Y\. Zhang, Y\. Wang, Y\. Deng, Y\. Zhao, X\. Zhi, Y\. Huang, W\. Che, B\. Qin, T\. Liu,et al\.\(2026b\)On safety risks in experience\-driven self\-evolving agents\.arXiv preprint arXiv:2604\.16968\.Cited by:[Appendix A](https://arxiv.org/html/2605.07164#A1.SS0.SSS0.Px1.p4.1)\.
- W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. Wang \(2024\)Memorybank: enhancing large language models with long\-term memory\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19724–19731\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.
- H\. Zhou, Y\. Chen, S\. Guo, X\. Yan, K\. H\. Lee, Z\. Wang, K\. Y\. Lee, G\. Zhang, K\. Shao, L\. Yang,et al\.\(2025\)Memento: fine\-tuning llm agents without fine\-tuning llms\.arXiv preprint arXiv:2508\.16153\.Cited by:[§2](https://arxiv.org/html/2605.07164#S2.p2.1)\.

## Appendix ALimitations and Impact Statement

#### Limitations\.

This work studies how experience should be utilized in self\-evolving agents and shows that interweaving experience utilization with decision\-making can improve performance across diverse frameworks and environments\. Nevertheless, several limitations remain\.

First, ExpWeaver is implemented through prompting and trigger\-based retrieval, which makes it lightweight and broadly applicable, but also means that its behavior depends on the instruction\-following ability of the underlying LLM\. We observe consistent gains across multiple backbones, but weaker small\-scale models may fail to reliably decide when experience is needed through prompting\.

Second, our analysis focuses on experience utilization under existing experience construction pipelines\. While this design helps isolate the effect of runtime utilization, it does not jointly optimize experience construction and utilization\. A promising direction is the recent trend of fully automated experience memory design, where agents can automatically discover, refine, and reorganize experience representations over time\[Xionget al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib58), Panet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib59)\]\. Combining such automated memory construction with adaptive utilization may provide a more unified path toward self\-evolving agents that can improve not only what they store, but also how and when they use it\.

Finally, our current study primarily evaluates task performance\. Other dimensions, such as safety, robustness, calibration, and long\-term stability under continuous deployment, remain important directions for future investigation\[Shaoet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib60), Zhaoet al\.,[2026b](https://arxiv.org/html/2605.07164#bib.bib61),[a](https://arxiv.org/html/2605.07164#bib.bib16)\]\.

#### Impact Statement\.

This work highlights experience utilization as a critical design dimension for self\-evolving agents\. By showing that agents benefit from selectively invoking experience during decision\-making, our findings may help build more adaptive, efficient, and interpretable agent systems\. In particular, ExpWeaver provides a lightweight way to improve existing self\-evolving frameworks without modifying their experience construction pipelines, which may lower the cost of deploying experience\-driven agents in diverse environments\.

At the same time, more effective experience utilization may also amplify both desirable and undesirable behaviors contained in accumulated experience\. If the experience repository contains biased, unsafe, outdated, or task\-inappropriate content, selectively invoking such experience may still mislead the agent\. Therefore, practical deployment should be accompanied by careful experience filtering, monitoring, and safety evaluation\. We hope this work encourages future research to study not only how agents accumulate experience, but also how experience should be governed, audited, and safely integrated into decision\-making\.

## Appendix BSelf\-Evolving Agents

We present detailed description of the four experience\-driven self\-evolving agents used in our experiments: ReasoningBank\[Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15)\], AWM\[Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14)\], SkillRL\[Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\]and G\-Memory\[Zhanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib12)\]\. Across all frameworks, agents adapt their behavior by accumulating, retrieving, and reusing external experiences stored in explicit memory structures, rather than by updating the parameters of the underlying language model\. In the following, we summarize the core design principles and memory mechanisms of these agents in detail:

- •ReasoningBank\[Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15)\]embodies an instance of an online self\-evolving agent\. It keeps an ever\-expanding memory that records condensed reasoning patterns distilled from the agent’s past interactions, covering both successful cases and failures\. Following each task completion, the agent assesses its own performance and selectively incorporates newly acquired experiences into this memory repository\. During inference, pertinent reasoning strategies are retrieved from the memory and incorporated into the agent’s context to guide future interactions\. The system forms a closed feedback loop where experiences are continuously accumulated, accessed, and reused throughout deployment, enabling the agent’s behavior to progressively adapt over time\.
- •AWM\[Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14)\]embodies another instance of an online self\-evolving agent\. It maintains a growing memory of reusable workflows—common sub‑routines distilled from successful agent trajectories, with example‑specific values abstracted into variables\. After each task, an LLM‑based evaluator judges whether the trajectory succeeded; if so, the agent induces one or more workflows from the trajectory and adds them to the memory\. During inference, relevant workflows are retrieved and incorporated into the agent’s context to guide future actions\. AWM operates both offline \(inducing workflows from annotated training examples\) and online \(learning purely from test queries in a streaming fashion\)\. This closed‑loop process enables the agent to accumulate increasingly complex workflows over time, rapidly improving performance from only a handful of examples and generalising across tasks, websites, and domains\.
- •SkillRL\[Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\]is an online self\-evolving agent that bridges raw experience and policy improvement via automatic skill discovery and recursive evolution\. It maintains a hierarchical skill library—SKILLBANK—comprising general strategies and task‑specific heuristics, both distilled from successful and failed trajectories by a teacher model\. During inference, relevant skills are retrieved to augment the agent’s context, achieving 10–20× token compression over raw trajectories\. A cold‑start supervised fine‑tuning stage teaches the base model how to use skills, followed by reinforcement learning \(GRPO\) that jointly optimises the policy and recursively evolves the skill library: failed validation trajectories trigger generation of new skills, enabling the library and policy to co‑adapt\. This closed‑loop process yields faster convergence, higher asymptotic performance, and strong generalisation across embodied and web tasks\.
- •G\-Memory\[Zhanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib12)\]is an online self\-evolving memory mechanism designed for multi\-agent systems\. It maintains a shared, persistent memory that records past multi\-agent collaboration experiences across tasks, capturing both abstract insights and condensed interaction histories\. When a new task arrives, relevant memory entries are retrieved and selectively injected into the contexts of different agents to support coordination and reasoning\. After task completion, newly generated interactions and distilled insights are incorporated into the memory, updating its contents during deployment\. This continual retrieval\-and\-update process enables agent teams to accumulate and reuse collaborative experience over time, allowing collective behavior to adapt without modifying the underlying language models\.

## Appendix CEnvironment and Benchmark

### C\.1Embodied Interaction

- •ALFWorld\[Shridharet al\.,[2021](https://arxiv.org/html/2605.07164#bib.bib43)\]is an embodied benchmark that aligns abstract, text\-based environments with interactive visual\-based scenes to execute household tasks\. It builds on the ALFRED\[Shridharet al\.,[2020](https://arxiv.org/html/2605.07164#bib.bib44)\]benchmark by providing paired representations of the same underlying tasks, where agents can operate through high\-level textual commands in a simulated environment\. The tasks span multiple categories such as pick\-and\-place, cleaning, heating, and cooling, and require multi\-step interaction with objects and receptacles distributed across diverse room layouts\. The benchmark is constructed to maintain a shared underlying world state across modalities, enabling consistent correspondence between language\-level actions and embodied executions\. We utilized the 134 solvable tasks\.

### C\.2Web Navigation

- •WebShop\[Yaoet al\.,[2022](https://arxiv.org/html/2605.07164#bib.bib42)\]is a large\-scale web interaction benchmark that simulates realistic online shopping scenarios through a self\-contained e\-commerce environment\. It includes over one million real\-world products and 12,087 crowdsourced natural language instructions, each specifying a product requirement to be fulfilled through a sequence of web\-based actions\. Given an instruction, an agent must navigate search results, inspect product pages, select appropriate options, and complete a purchase to satisfy the specified constraints\. We follow the instructions of ExpeL to set the implementation details of the WebShop Environment\.

### C\.3Knowledge\-Intensive Question Answering

- •HotpotQA\[Yanget al\.,[2018](https://arxiv.org/html/2605.07164#bib.bib35)\]is a large\-scale question answering benchmark designed to support multi\-hop reasoning over natural language text\. It contains question–answer pairs constructed from Wikipedia articles, where answering each question requires reasoning across multiple supporting documents\. The questions are diverse in form, including standard factoid queries as well as comparison and yes/no questions, and are not constrained by predefined knowledge base schemas\.
- •TriviaQA\[Joshiet al\.,[2017](https://arxiv.org/html/2605.07164#bib.bib38)\]is a challenging reading comprehension dataset containing over 650K question–answer–evidence triples\. It consists of 95K question\-answer pairs authored by trivia enthusiasts, with evidence documents provided for distant supervision\. The questions have relatively complex, compositional syntax and often require cross‑sentence reasoning to answer\.
- •2WikiMultiHopQA\[Hoet al\.,[2020](https://arxiv.org/html/2605.07164#bib.bib39)\]is constructed using a hybrid of templated generation and logical rules based on Wikidata\. Each question requires multi‑step reasoning across two Wikipedia articles, and the dataset includes explicit evidence information that reveals the full reasoning path from the question to the answer\.
- •Natural Questions\(NQ\)\[Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2605.07164#bib.bib37)\]is derived from real, anonymized Google search queries\. The dataset contains over 300,000 training examples, each annotated by human raters with both long and short answers from the corresponding Wikipedia page, providing a realistic benchmark for open‑domain QA\.
- •Bamboogle\[Presset al\.,[2023](https://arxiv.org/html/2605.07164#bib.bib41)\]is a small yet challenging hand‑crafted multi‑hop QA dataset designed to measure the compositional reasoning gap in language models\. Each question explicitly requires combining information from two separate facts, and it is commonly used as a difficult out‑of‑domain evaluation set\.

For each of the above datasets following the official settings of evaluated self\-evolving frameworks, we randomly selected 100 questions from its development split for our experiments\.

## Appendix DImplementation Details

### D\.1Prompting\-based Setup

ExpWeaver preserves the original design of each self\-evolving agent framework with respect to experience construction, representation, and updating\. Specifically, we leave the mechanisms for extracting, storing, and maintaining experience unchanged, and instead introduce only lightweight prompting\. This is achieved by only augmenting the system prompt with instructions that interweave experience utilization with the reasoning process\. We present the prompt designs for ALFWorld, QA tasks, and WebShop in Figure[7](https://arxiv.org/html/2605.07164#A4.F7), Figure[8](https://arxiv.org/html/2605.07164#A4.F8)and Figure[9](https://arxiv.org/html/2605.07164#A4.F9), respectively\. The detailed hyperparameter settings in our experiments for each self\-evolving framework are described below\. All reported results in our experiments are averaged over 3 independent runs\.

- •ReasoningBankOur experimental setup largely follows the configuration of ReasoningBank\[Ouyanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib15)\]\. For the agent, we set the decoding temperature to 0\.7 and adopt greedy decoding as the decoding strategy\. On the WebArena benchmark, each task is allowed a maximum of 30 interaction steps\. We use*text\-embedding\-ada\-002*as the embedding model to encode queries and memory items, and employ cosine similarity for retrieval\. For each new query, the agent retrieves the top\-3 most relevant memory items, which are then injected into the agent’s prompt\.
- •Agent Workflow Memory \(AWM\)We adopt AWM\[Wanget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib14)\]as a baseline\. Following the original setup, we use temperature 0\.0\. Workflows are induced from successful trajectories, and during online inference the agent uses relevant workflows, which means no additional training data is used; the agent learns entirely from test queries in a streaming fashion\.
- •G\-MemoryWe follow the experimental setup of G\-Memory\[Zhanget al\.,[2025a](https://arxiv.org/html/2605.07164#bib.bib12)\]\. For coarse\-grained retrieval over the query graph, queries are encoded using a MiniLM sentence embedding model and matched with cosine similarity, retrieving the top\-kmost similar historical queries\. For fine\-grained retrieval, we further select the top\-Mrelevant queries using an LLM\-based relevance scorer and sparsify their interaction graphs with an LLM\-based graph compression module\. The values ofkandMare treated as tunable hyperparameters\. Query nodes are labeled with execution status from Failed, Resolved\. All retrieved insights and interaction subgraphs are injected into agent prompts before task execution, and the memory graphs are updated after each task without any gradient\-based training\.
- •SkillRLWe follow SkillRL\[Xiaet al\.,[2026](https://arxiv.org/html/2605.07164#bib.bib11)\]for inference\. The agent uses a hierarchical SkillBank with general and task\-specific skills\. For each query, it retrieves top\-66relevant task\-specific skills \(similarity threshold0\.40\.4\) using an embedding model \(text\-embedding\-ada\-002\) and includes all general skills\. Retrieved skills are added to the prompt, and the agent decodes with temperature0\.70\.7\. The skill library is fixed during testing\.

![Refer to caption](https://arxiv.org/html/2605.07164v1/x7.png)Figure 7:System prompt on ALFWorld\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x8.png)Figure 8:System prompt on QA tasks\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x9.png)Figure 9:System prompt of WebShop\.
### D\.2Reinforcement Learning\-based Setup

We conduct RL training on two types of interactive environments: ALFWorld and a suite of search‑augmented question answering \(QA\) tasks\. For ALFWorld, we use the standard training splits with 3553 training tasks\. All reported results are averaged over 3 independent runs\. For QA training, we follow prior work and merge the training sets of NQ and HotpotQA to form a unified dataset for ExpWeaver\. Out‑of‑domain generalization is assessed on TriviaQA, 2Wiki, and Bamboogle, where we sample 100 test instances per dataset\. Hyperparameters used for ExpWeaver RL training are summarized in Table[3](https://arxiv.org/html/2605.07164#A4.T3)\. Our implementation is built on the verl\-agent framework\[Fenget al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib57)\]\. Training runs for a fixed number of epochs or until validation performance converges\.

Table 3:Hyperparameters for ExpWeaver training\.HyperparameterValueRL TrainingLearning rate1e\-6Batch size8Group sizeGG8Gradient accumulation steps64KL penalty coefficientβ\\beta0\.01Max prompt length \(tokens\)32768Max response length \(tokens\)512Total training epochs150![Refer to caption](https://arxiv.org/html/2605.07164v1/x10.png)Figure 10:Prompting\-based results under the ReasoningBank framework on three QA tasks\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x11.png)Figure 11:Prompting\-based results under the SkillRL framework on three QA tasks\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x12.png)Figure 12:Prompting\-based results under the G\-Memory framework on ALFWorld and WebShop\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x13.png)Figure 13:Prompting\-based results under the G\-Memory framework on three QA tasks\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x14.png)Figure 14:Prompting\-based results under the AWM framework on ALFWorld and WebShop\.![Refer to caption](https://arxiv.org/html/2605.07164v1/x15.png)Figure 15:Prompting\-based results under the AWM framework on three QA tasks\.

## Appendix EAdditional Results with ExpWeaver

### E\.1Additional Prompting\-based Results

#### Results on QA tasks\.

Figures[10](https://arxiv.org/html/2605.07164#A4.F10)and[11](https://arxiv.org/html/2605.07164#A4.F11)present the prompting\-based results on three knowledge\-intensive QA tasks under the ReasoningBank and SkillRL frameworks, respectively\. Overall, ExpWeaver consistently achieves the best or competitive performance across TriviaQA, HotpotQA, and 2Wiki, further confirming the effectiveness of interweaving experience utilization into the decision\-making process\.

Under ReasoningBank, ExpWeaver improves over the vanilla initialization\-only strategy in most settings\. The gains are particularly clear on TriviaQA and HotpotQA, where ExpWeaver achieves the best performance across nearly all backbone models\. On 2Wiki, ExpWeaver also generally improves over rigid usage strategies, though the margin is smaller for stronger models whose baseline performance is already high\. These results suggest that allowing agents to decide when to consult distilled reasoning experiences is beneficial for knowledge\-intensive reasoning\.

Under SkillRL, we observe similar trends despite the different form of experience\. Since SkillRL uses a pre\-constructed skill repository, all variants share the same underlying experience memory and differ only in how skills are utilized at runtime\. ExpWeaver consistently improves over vanilla usage on TriviaQA and HotpotQA, and remains competitive on 2Wiki\. This provides controlled evidence that the gains arise from better runtime experience utilization rather than changes in the experience repository itself\. Together, these QA results complement the main results on ALFWorld and WebShop, showing that ExpWeaver generalizes beyond interactive environments to knowledge\-intensive reasoning tasks\.

#### Results on G\-Memory\.

Figures[12](https://arxiv.org/html/2605.07164#A4.F12)and[13](https://arxiv.org/html/2605.07164#A4.F13)present the prompting\-based results under the G\-Memory framework\. In the G\-Memory framework, we did not design a variant that retrieves experience at every step \(Always\-on\), as doing so would violate its design philosophy and core architecture—namely, building a hierarchical graph memory store with complete task trajectories as the central nodes\. In our experiments, we only adjusted the timing of experience presentation during execution, without changing G\-Memory’s own experience mechanism\. This allows us to independently examine the effect of experience presentation timing while preserving the unique design advantages of the G\-Memory framework\. Unlike the single\-agent frameworks, G\-Memory operates in a multi\-agent setting and represents experience using both raw trajectories and distilled insights\. Across ALFWorld, WebShop, and QA tasks, ExpWeaver generally improves over the vanilla initialization\-only strategy, showing that interweaving experience utilization into decision\-making remains effective in multi\-agent self\-evolving systems\.

On ALFWorld, ExpWeaver consistently improves performance across all models\. The gains are especially clear for Qwen3\.5\-397B\-A17B and DeepSeek\-V4\-Pro, suggesting that even strong models benefit when experience is introduced at appropriate decision points rather than only at the beginning\. On WebShop, ExpWeaver also improves over the vanilla strategy in most cases, indicating that adaptive experience usage helps agents resolve interactive decision uncertainty in web navigation\.

For QA tasks, ExpWeaver achieves consistent or competitive improvements across TriviaQA, HotpotQA, and 2Wiki\. The gains are generally smaller than those on ALFWorld, which is consistent with our usage\-pattern analysis showing that QA tasks require less frequent experience invocation\. Overall, the G\-Memory results demonstrate that ExpWeaver generalizes beyond single\-agent settings and remains effective when experience is shared and utilized in a multi\-agent memory framework\.

#### Results on AWM\.

Figures[14](https://arxiv.org/html/2605.07164#A4.F14)and[15](https://arxiv.org/html/2605.07164#A4.F15)present the prompting\-based results under the AWM framework\. AWM represents experience as executable workflows and does not include an explicit retrieval mechanism during task execution\. Therefore, the initialization\-only and always\-on variants are equivalent in implementation: the same workflow experience is provided as fixed guidance rather than dynamically retrieved at different steps\.

Across ALFWorld, WebShop, and QA tasks, ExpWeaver generally improves over the vanilla AWM setting\. The gains are especially clear on ALFWorld, where ExpWeaver consistently achieves higher success rates across different models, suggesting that even when experience is represented as workflows, allowing the agent to decide when to rely on experience remains beneficial\. On WebShop and QA tasks, ExpWeaver also achieves competitive or improved performance in most settings\.

These results further support our main conclusion: the benefit of ExpWeaver is not tied to retrieval\-based memory alone\. Even in a workflow\-based framework without explicit retrieval, interweaving experience utilization into decision\-making can improve how agents make use of existing experience\.

\(a\)Reinforcement learning results of Qwen3\-14B on knowledge\-intensive QA tasks\.### E\.2Additional RL Results on QA Tasks

We further evaluate RL\-based ExpWeaver on knowledge\-intensive QA tasks using Qwen3\-14B under the ReasoningBank framework\. Specifically, we train the model with GRPO on a merged training set consisting of NQ and HotpotQA\[Jinet al\.,[2025](https://arxiv.org/html/2605.07164#bib.bib46)\]\. During training, we periodically evaluate intermediate checkpoints on five test sets: NQ, TriviaQA, HotpotQA, Bamboogle, and 2Wiki\. Figure[20\(a\)](https://arxiv.org/html/2605.07164#A5.F20.sf1)reports the success rates of ExpWeaver and the vanilla baseline across training steps, where each point corresponds to the performance of a checkpoint on the corresponding test set\.

Overall, ExpWeaver achieves stronger improvements than the vanilla baseline \(Init\-only\) across QA tasks\. On NQ, ExpWeaver shows a clear upward trend and reaches a substantially higher success rate at later training steps, indicating that the model gradually learns to leverage experience more effectively during reasoning\. Similar trends are observed on TriviaQA, Bamboogle, and 2Wiki, where ExpWeaver tends to achieve higher final performance\.

These results suggest that the benefits of ExpWeaver are not limited to embodied interaction\. Even in knowledge\-intensive QA, where reasoning and evidence integration are central, allowing the agent to regulate when experience enters the reasoning process can improve RL optimization\. This further supports our claim that interweaving experience utilization with decision\-making is a learnable capability that generalizes beyond inference\-time prompting and across task domains\.

## Appendix FUsage Pattern Analysis of ExpWeaver

Table 4:Additional usage pattern results on AWM and SkillRL\.### F\.1Additional Horizontal Comparison Results

Table[4](https://arxiv.org/html/2605.07164#A6.T4)reports the average number of experience retrievals per sample on AWM and SkillRL\.

First, ALFWorld again induces substantially more experience retrievals than WebShop and HotpotQA\. For example, under AWM, Qwen3\-32B invokes experience 2\.62 times per sample on ALFWorld, compared with 0\.16 on WebShop and 0\.53 on HotpotQA\. Similarly, under SkillRL, Qwen3\-32B retrieves experience 2\.50 times per sample on ALFWorld, while retrieval frequencies remain below 0\.40 on WebShop and HotpotQA\. This reinforces the observation that embodied interaction, with long\-horizon planning and partial observability, creates stronger demand for experience\.

Second, stronger models tend to invoke experience less frequently\. To be more specific, Qwen3\.5\-397B\-A17B and GPT\-5\.2 exhibit near\-zero retrieval on WebShop and HotpotQA under both AWM and SkillRL, suggesting that stronger backbones can often solve these tasks without additional experience\. In contrast, smaller or less capable models rely on experience more often, especially in ALFWorld\. This further supports the view that ExpWeaver adapts experience usage according to model capability rather than following a fixed retrieval schedule\.

Third, the consistency across AWM and SkillRL indicates that the usage pattern is not specific to a particular form of experience\. AWM represents experience as workflows, while SkillRL represents experience as reusable skills, yet both exhibit similar task\- and model\-dependent retrieval behavior\. This suggests that interweaving experience utilization into decision\-making induces a general regulatory mechanism over experience use, regardless of how experience is constructed or represented\.

Together, these results reveal an important property of ExpWeaver: it does not simply reduce or increase experience usage uniformly\. Instead, it produces a calibrated usage pattern conditioned on task difficulty, model capability, and decision context\. This provides direct evidence that experience utilization should not be treated as an always\-on operation; its value depends critically on when the agent chooses to invoke it\.

\(a\)Temporal pattern of experience utilization under ExpWeaver with ReasoningBank\.
\(b\)Temporal pattern of experience utilization under ExpWeaver with G\-Memory\.
\(c\)Temporal pattern of experience utilization under ExpWeaver with AWM\.\(d\)Temporal pattern of experience utilization under ExpWeaver with SkillRL\.Table 5:The example of ExpWeaver on ALFWorld\.
Table 6:The example of ExpWeaver on WebShop\.
Table 7:The example of ExpWeaver on QA task\.
### F\.2Additional Temporal Analysis

We provide additional temporal analyses of experience utilization across different self\-evolving frameworks and tasks\. Figures[22\(a\)](https://arxiv.org/html/2605.07164#A6.F22.sf1)–[22\(d\)](https://arxiv.org/html/2605.07164#A6.F22.sf4)report the fraction of trajectories that invoke experience at each interaction step under ReasoningBank, G\-Memory, AWM, and SkillRL, respectively\. Overall, these results show that the non\-uniform temporal pattern observed in the main text is consistent across frameworks, tasks, and backbone models\.

Across ALFWorld, experience utilization is highly concentrated at the beginning of trajectories\. This trend appears consistently under G\-Memory, AWM, and SkillRL, where most retrievals occur in the first few interaction steps and then rapidly decrease\. This suggests that agents tend to consult experience when initial task understanding, high\-level planning, and environment exploration are most needed\. Notably, some models continue to invoke experience sporadically in later steps, especially in ALFWorld, which often corresponds to object\-search or recovery phases under partial observability\.

On WebShop, the temporal patterns are more dispersed and generally sparser\. Compared with ALFWorld, agents invoke experience less frequently, and retrieval events are often scattered across later interaction steps\. This reflects the different nature of web navigation, where the agent may not need experience at the beginning but may consult it when encountering uncertain product search, comparison, or navigation decisions\. The scattered peaks indicate that experience is used as situational guidance rather than as a fixed initialization signal\.

On HotpotQA, experience utilization is typically concentrated in early reasoning steps and quickly diminishes afterward\. This is especially clear for models with higher retrieval frequency, where experience is often invoked during initial question decomposition or evidence\-seeking\. Once the agent establishes a reasoning direction, later retrieval becomes less frequent\. This pattern supports the view that experience is mainly used to guide early reasoning or resolve uncertainty, rather than being repeatedly injected throughout the entire reasoning process\.

Importantly, these trends hold across different experience representations, including distilled insights in ReasoningBank, raw trajectories and distilled insights in G\-Memory, workflows in AWM, and reusable skills in SkillRL\. Although the absolute retrieval frequency varies across frameworks and models, the temporal structure remains consistent: ExpWeaver invokes experience selectively, with strong early\-stage usage and occasional later\-stage reactivation when the agent faces uncertainty or difficulty\. These results further support our claim that interweaving experience utilization with decision\-making enables agents to regulate when experience enters the reasoning process, rather than following rigid initialization\-only or always\-on usage strategies\.

### F\.3Qualitative Examples

We provide qualitative examples to illustrate how ExpWeaver invokes experience during decision\-making across different environments\. These cases show that experience retrieval is not triggered uniformly, but is typically invoked when the agent encounters uncertainty, ambiguity, or repeated failure during task execution\.

#### ALFWorld\.

Table[5](https://arxiv.org/html/2605.07164#A6.T5)presents an example from ALFWorld, where the agent needs to heat an apple and place it in the garbage can\. The agent first searches several plausible locations, such as cabinets and the fridge, but fails to find the target object\. After repeated unsuccessful search attempts, ExpWeaver triggers experience retrieval to seek guidance about alternative object locations\. The retrieved experience suggests that the garbage can should be checked, which directly helps the agent locate the apple and continue the task\. This example shows that ExpWeaver invokes experience when the agent is stuck in a partially observable environment and needs corrective guidance\.

#### WebShop\.

Table[6](https://arxiv.org/html/2605.07164#A6.T6)shows a WebShop example where the agent must purchase a black long\-sleeve men’s t\-shirt under a price constraint\. After finding a candidate item, the agent becomes uncertain about whether unspecified options such as size should also be selected before buying\. ExpWeaver retrieves relevant experience indicating that all selectable options should be clicked before purchasing, even when some options are not specified in the instruction\. Guided by this experience, the agent selects the required color, chooses an available size, and successfully completes the purchase\. This case shows that experience is useful for resolving procedural uncertainty in web environments\.

#### Knowledge\-intensive QA\.

Table[7](https://arxiv.org/html/2605.07164#A6.T7)provides an example from a QA task asking for the profession shared by Nicholas Ray and Elia Kazan\. After retrieving information about one entity, the agent recognizes a risk of missing overlapping professions and invokes experience to obtain a comparison checklist\. The retrieved experience reminds the agent to extract profession sets for both entities separately and return their intersection\. Following this, the agent searches for the second entity and correctly computes the shared professions\. This illustrates how ExpWeaver helps regulate reasoning in knowledge\-intensive tasks by retrieving experience when the agent finds a reasoning pitfall\.

Overall, these examples support our temporal and entropy\-based analyses: ExpWeaver tends to invoke experience at moments where the agent faces uncertainty or requires additional guidance\. The retrieved experience then serves as targeted support for the current decision, rather than as static context uniformly injected throughout the trajectory\.