Agent 记忆最危险的时候，往往是它太相信过去。

很多 Memory Agent 找到相似经验后，会直接把它塞进上下文。但任务相似，不代表当前状态相同，旧经验有时反而会把决策带偏。

这篇论文提出 MemHarness，把 Agent 使用记忆的过程改成三步：

先检索过去经验，再结合当前状态进行判断和重构，最后才生成行动。

有用的部分会被保留，不匹配的内容会被修改或直接丢弃。整个能力通过 GRPO 和任务奖励端到端训练，不需要额外标注重构过程。

基于 Qwen2.5-7B，MemHarness 在 ALFWorld 和 WebShop 上分别达到 85.2% 和 75.6% 的成功率，比纯 GRPO 提升 8.8 和 9.5 个百分点。

在未见过的环境中，它也达到 85.9%，而直接回放原始记忆只有 76.3%。这说明记忆用错了，确实可能比没有记忆更糟。

这篇工作把 Agent Memory 从「保存过去」推进到了「根据现在重新理解过去」。

真正可靠的记忆系统，需要知道哪些经验值得保留，哪些需要改写，哪些应该及时忘掉。

arxiv: https://arxiv.org/abs/2607.28272

1 Introduction

Source: https://arxiv.org/html/2607.28272 MemHarness: Memory Is Reconstructed, Not ReplayedRong Wu1,2,Daocheng Fu2,3,Licheng Wen2,4,5,Xuemeng Yang2,Shu Zou6,Jianbiao Mei2,Yuxin Wang7,Hairong Zhang2,Yu Yang1,2,Tao Hu2,Cong Zhang1,Botian Shi2,Pinlong Cai†,2,1Zhejiang University,2Shanghai Artificial Intelligence Laboratory3Fudan University,4Shanghai Innovation Institute5Shanghai Jiao Tong University,6The Australian National University7University of Science and Technology of China†Corresponding AuthorRetrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent’s current situation. This “replay” paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we proposeMemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent’s intrinsic reasoning capabilities.Date:Github Repo:https://github.com/KnowledgeXLab/MemHarness

Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents for sequential decision-makingChenet al.(2023b); Fuet al.(2025); Jinet al.(2025); Meiet al.(2025). Experiential memory further enables these agents to reuse prior successes and avoid repeated mistakesWanget al.(2024); Yanet al.(2025); Xuet al.(2026). However, most memory-augmented agents follow averbatim replayparadigm: retrieved trajectories or principles are treated as static records and directly inserted into the model contextYanget al.(2026); Zhanget al.(2025a). This design conflates retrieval relevance with action-level applicability. A memory may be semantically relevant to the task yet inappropriate for the current interaction state, because it was formed under different environmental conditions. An alternative line of work internalizes experience into model parameters, enabling state-conditioned behavior without explicitly replaying retrieved recordsZhanget al.(2025b). While such parametric memory offers adaptive generation, the underlying experience and its influence on a decision remain implicit and difficult to inspect or revise. Explicit memory banks provide greater traceability, but typically lack this adaptive use of experience. This exposes a desirable middle ground: retaining explicit, attributable memories while dynamically reconstructing them for the current state.

This middle ground echoes a fundamental insight from cognitive science: human remembering is reconstructive rather than reproductiveLoftus and Palmer (1974). Recall does not recover an immutable record of the past; instead, retrieved experience is interpreted and reorganized using present cues and prior knowledge before it informs behavior. As illustrated in Figure1, prior agents largely connect retrieval directly to action generation, whereas human memory introduces an intermediate process that evaluates and adapts recalled experience. This suggests a broader paradigm shift for memory-augmented agents—fromretrieve and replaytoretrieve, evaluate, and reconstruct.

Inspired by this mechanism, we decompose memory-guided decision-making into five stages that parallel the reconstructive process:environment observationcaptures the current situation;experience retrievalrecalls a potentially relevant memory together with its historical context;memory critiqueassesses its applicability and identifies state mismatches;contextual memory reconstructionpreserves transferable knowledge while revising or discarding incompatible content; andaction generationuses the reconstructed guidance to make a context-aligned decision.

Figure 1:Memory utilization paradigms.Top:Prior methods directly replay retrieved memories, risking state misalignment.Middle:Human memory reconstructs past experience according to the current context.Bottom:Inspired by this process, MemHarness reconstructs retrieved memories into state-aligned guidance.Based on this formulation, we introduceMemHarness, designed to actively harness retrieved experiences by explicitly inserting critique and reconstruction between retrieval and action, turning memory from a static prompt fragment into context-sensitive guidance.

Building on this formulation,MemHarnessbridges explicit and parametric memory utilization: it retains retrieved experiences as inspectable records, while parameterizing their state-conditioned critique and reconstruction within a single policy. At each decision step, conditioned on the current observation, agent first critiques the retrieved experience against its original context, then reconstructs it into state-specific guidance, and finally generates the executable action. Crucially, this reconstructive ability requires no additional human annotation: it emerges through end-to-end training with Group Relative Policy Optimization (GRPO). By optimizing for task success, the agent implicitly learns a discriminative reconstruction strategy, preserving transferable memories while rewriting misleading ones to close the gap between historical knowledge and the present state.

We evaluate MemHarness on two challenging agent decision-making benchmarks, ALFWorld and WebShop. It consistently outperforms both pure RL baselines and state-of-the-art static memory-augmented methods. Ablation studies further show that removing the reconstruction stage degrades performance to the level of naive memory replay, confirming that adaptive reconstruction, rather than retrieval alone, is the primary driver of the observed gains.

In summary, our main contributions are as follows:

• •A reconstructive view of agent memory.We identify the applicability gap underlying the prevalentverbatim replayparadigm, and, drawing on the reconstructive nature of human memory, recast memory-guided decision-making as a five-stage process spanning observation, retrieval, critique, reconstruction, and action.
• •A reconstruction-centric framework.We proposeMemHarness, which actively harnesses retrieved experiences by inserting explicit memory reconstruction between retrieval and action, and learns this capability end-to-end via GRPO without external supervision, combining the traceability of explicit memory with the adaptivity of parameterized reconstruction.
• •Comprehensive empirical validation.On interactive decision-making benchmarks, MemHarness delivers robust improvements over static replay, with ablations isolating reconstruction as the key contributing factor.

2Related Works

2.1LLM Agents for Interactive Decision-Making

LLM agents navigate interactive tasks using prompting paradigms like ReActYaoet al.(2023); Weiet al.(2022), often augmented with planning or self-reflectionShinnet al.(2023); Maet al.(2026). However, these training-free methods are bounded by frozen model capabilities. To acquire task-specific skills from interaction, agents are fine-tuned via supervised learningChenet al.(2024b,2023a)or reinforcement learning (RL)Schulmanet al.(2017); Ahmadianet al.(2024); Shaoet al.(2024); Fenget al.(2026). While RL optimizes decision-making directly, standard frameworks lack explicit mechanisms to accumulate and reuse cross-episode experiences, motivating memory-augmented agents.

2.2Memory-Augmented LLM Agents

Memory-augmented agents address this by retrieving past interactions or distilled insightsZhaoet al.(2024); Fanget al.(2026); Chhikaraet al.(2025); Liuet al.(2026). Most rely on frozen policies, while alternative approaches internalize experience directly into model weightsZhanget al.(2025b), sacrificing the traceability of explicit memory. Recent works integrate external memory with RLZhanget al.(2026); Wuet al.(2025), yet they inject retrieved experiencesverbatim. Because interactive tasks exhibit high state variance, replay often introduces misaligned or misleading guidance.

MemHarness bridges these two paradigms. It retains explicit memory banks for traceability but parameterizes their state-conditioned reconstruction. Rather than applying retrieved memory blindly, MemHarness is optimized end-to-end to critique and rewrite past experiences into context-specific guidance before acting.

3Method

Figure 2:Overview of the MemHarness framework. The execution pipeline consists of three stages: (1)Memory Retrieval, where the policy generates a query to fetch relevant past experiences; (2)Contextual Memory Reconstruction, where the policy compares the memory’s source state with the current state to reconstruct adapted guidance (or falls back to self-reasoning if the memory is deemed unhelpful); and (3)Action Generationguided by the reconstructed memory.### 3.1Problem Formulation

Sequential Decision-Making.

We model interactive decision-making as a POMDPℳ=(𝒮,𝒜,𝒪,P,R)\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{O},P,R), where𝒮\mathcal{S},𝒜\mathcal{A}, and𝒪\mathcal{O}are the state, action, and observation spaces, respectively, andPPandRRdenote the transition and reward functions. Given a task description𝒯\mathcal{T}, at stepttthe agent observesoto_{t}and selectsata_{t}using a policyπθ\pi_{\theta}. To account for partial observability, the policy conditions on a recent history windowht=(ot−w+1,at−w+1,…,ot−1,at−1,ot)h_{t}=(o_{t-w+1},a_{t-w+1},\ldots,o_{t-1},a_{t-1},o_{t}). The objective is to maximize the expected episodic return𝔼πθ​[∑t=1TR​(st,at)]\mathbb{E}_{\pi_{\theta}}[\sum_{t=1}^{T}R(s_{t},a_{t})].

Experiential Memory.

The agent maintains a memory bankℬ={mi}i=1N\mathcal{B}=\{m_{i}\}_{i=1}^{N}by periodically distilling collected trajectories into natural-language experiences. Each entrymi=(ei,ois​r​c)m_{i}=(e_{i},o_{i}^{src})contains an abstract strategyeie_{i}and its source observationois​r​co_{i}^{src}. FollowingWuet al.(2025), we perform semantic deduplication, track empirical utility and periodically prune low-utility entries. Given a context-derived queryqtq_{t}, a retrieverℛ\mathcal{R}returns the top-kkentries:

ℰt=ℛ​(qt,ℬ),|ℰt|=k.\mathcal{E}_{t}=\mathcal{R}(q_{t},\mathcal{B}),\qquad|\mathcal{E}_{t}|=k.(1)The source observations enable comparison between historical and current states during reconstruction.

From Replay to Reconstruction.

Verbatim replay directly conditions action generation on retrieved experiences:at∼πθ(⋅∣𝒯,ht,ℰt)a_{t}\sim\pi_{\theta}(\cdot\mid\mathcal{T},h_{t},\mathcal{E}_{t}). MemHarness instead first reconstructs them into state-specific guidance and then acts:

gt\displaystyle g_{t}∼πθ(⋅∣𝒯,ht,ℰt),\displaystyle\sim\pi_{\theta}(\cdot\mid\mathcal{T},h_{t},\mathcal{E}_{t}),(2)g~t\displaystyle\tilde{g}_{t}=f​(gt),\displaystyle=f(g_{t}),at\displaystyle a_{t}∼πθ(⋅∣𝒯,ht,g~t),\displaystyle\sim\pi_{\theta}(\cdot\mid\mathcal{T},h_{t},\tilde{g}_{t}),whereffmaps the raw reconstructiongtg_{t}to final guidanceg~t\tilde{g}_{t}, including rejection of mismatched memories. Both stages share the unified policyπθ\pi_{\theta}.

3.2Overall Framework

We introduceMemHarness, an RL-driven framework that reconstructs retrieved experiences for the current state rather than replaying them verbatim. As shown in Figure2, its inference pipeline comprises three stages: retrieval, contextual memory reconstruction, and action generation. These three stages instantiate the five-stage formulation in Section1: the history windowhth_{t}serves as the environment observation, and memory critique is folded into the reconstruction stage.

1. 1.Retrieval.At steptt, the agent decides whether to query the experiential memory bankℬ\mathcal{B}based onhth_{t}. If triggered, retrieval returns relevant experiencesℰt\mathcal{E}_{t}, which are reconstructed rather than directly added to the action context.
2. 2.Reconstruction.The policyπθ\pi_{\theta}critiquesℰt\mathcal{E}_{t}againsthth_{t}and generates state-specific guidancegtg_{t}, retaining applicable knowledge while discarding or revising misaligned content.
3. 3.Action Generation.The same policyπθ\pi_{\theta}generates an executable actionata_{t}conditioned onhth_{t}andgtg_{t}.

MemHarness thus separates reconstruction from action generation while implementing both with a unified policy. Because supervised reconstruction traces are unavailable, we train the two-step process end-to-end with GRPO using only task rewards. The following sections detail the reconstruction and optimization procedures.

3.3Contextual Memory Reconstruction

MemHarness transforms retrieved experiences into state-specific guidance through an explicit reconstruction. Because experiences abstracted from historical trajectories may be outdated or incompatible with the current state, the policy critiques their applicability before they influence action generation.

Critique and Reconstruction.

The reconstruction context concatenates the task description𝒯\mathcal{T}, current historyhth_{t}, and retrieved memory tuples(et,i,ot,is​r​c)∈ℰt(e_{t,i},o_{t,i}^{src})\in\mathcal{E}_{t}:

xrecon=𝒯⊕ht⊕⋃i=1k(et,i,ot,is​r​c),x_{\mathrm{recon}}=\mathcal{T}\oplus h_{t}\oplus\bigcup_{i=1}^{k}(e_{t,i},o_{t,i}^{src}),(3)whereet,ie_{t,i}is a retrieved experience,ot,is​r​co_{t,i}^{src}is its source observation, and⊕\oplusdenotes text concatenation. The policy then generates:

gt∼πθ(⋅∣xrecon).g_{t}\sim\pi_{\theta}(\cdot\mid x_{\mathrm{recon}}).(4)By comparing each source observation with the current context represented byhth_{t}, the policy identifies state shifts and retains, revises or rejects the retrieved experience to produce context-specific guidance.

If no retrieved experience is applicable, the policy outputs<EMPTY>. The guidance used for action generation is

g~t={pself,if​gt=<EMPTY>,gt,otherwise,\tilde{g}_{t}=\begin{cases}p_{\mathrm{self}},&\text{if }g_{t}=\texttt{<EMPTY>},\\ g_{t},&\text{otherwise},\end{cases}(5)wherepselfp_{\mathrm{self}}instructs the agent to rely on its own reasoning without memory guidance.

Action Generation.

The same policy generates the executable action conditioned on the reconstructed guidance:

at∼πθ(⋅∣𝒯,ht,g~t).a_{t}\sim\pi_{\theta}(\cdot\mid\mathcal{T},h_{t},\tilde{g}_{t}).(6)Thus, action generation uses either adapted memory or independent reasoning, rather than unfiltered historical records. Reconstruction and action generation share parametersθ\thetaand are optimized jointly. Sincegtg_{t}has no ground-truth annotation, we treat it as a latent reasoning process and train the pipeline end-to-end using task-level rewards

3.4Training with GRPO

Since the guidancegtg_{t}is a latent variable without ground-truth supervision, we optimize the retrieve–reconstruct–act pipeline end-to-end using reinforcement learning. The reward combines task outcomes with format-based signals that enforce valid reconstruction and action outputs. We adopt GRPO to estimate advantages from groups of sampled trajectories without training a separate value network.

Reward Design.

The rewardR​(τi)R(\tau_{i})for a trajectory primarily depends on the sparse task outcome (0 for failure, 10 for success), augmented by a minor bonus to enforce structural constraints:R​(τi)=Routcome+0.1×RformatR(\tau_{i})=R_{\text{outcome}}+0.1\times R_{\text{format}}. The format scoreRformat∈[0,1]R_{\text{format}}\in[0,1]equally weights three criteria: (1) exactly one valid<think>and<action>block per step, (2) appropriate memory retrieval frequency with<retrieve_memory>block (1 to 5 times per episode), and (3) strict English output. This formatting bonus explicitly encourages an agentic protocol (i.e., reason first, retrieve memory when necessary, single action per step) without overshadowing the primary task objective.

Policy Update.

For each task, we sample a group ofGGtrajectories{τi}i=1G\{\tau_{i}\}_{i=1}^{G}. GRPO computes the group-normalized advantage:

Ai=R​(τi)−mean​({R​(τk)}k=1G)std​({R​(τk)}k=1G).A_{i}=\frac{R(\tau_{i})-\mathrm{mean}(\{R(\tau_{k})\}_{k=1}^{G})}{\mathrm{std}(\{R(\tau_{k})\}_{k=1}^{G})}.(7)The trajectory-level advantageAiA_{i}is assigned to all tokens inτi\tau_{i}, providing shared credit to thinking process, reconstruction and action generation. For tokenyi,jy_{i,j}, the importance ratio is

ri,j​(θ)=πθ​(yi,j∣yi,<j)πθold​(yi,j∣yi,<j).r_{i,j}(\theta)=\frac{\pi_{\theta}(y_{i,j}\mid y_{i,<j})}{\pi_{\theta_{\mathrm{old}}}(y_{i,j}\mid y_{i,<j})}.(8)The clipped surrogate objective is

ℒi,jCLIP​(θ)=min⁡(ri,j​(θ)​Ai,clip​(ri,j​(θ),1−ϵ,1+ϵ)​Ai),\mathcal{L}^{\mathrm{CLIP}}_{i,j}(\theta)=\min\!\left(r_{i,j}(\theta)A_{i},\,\mathrm{clip}(r_{i,j}(\theta),1-\epsilon,1+\epsilon)A_{i}\right),(9)and the overall objective is

𝒥​(θ)=𝔼​[1∑i|τi|​∑i=1G∑j=1|τi|(ℒi,jCLIP​(θ)−β​𝔻KL​[πθ∥πref])],\mathcal{J}(\theta)=\mathbb{E}\!\left[\frac{1}{\sum_{i}|\tau_{i}|}\sum_{i=1}^{G}\sum_{j=1}^{|\tau_{i}|}\left(\mathcal{L}^{\mathrm{CLIP}}_{i,j}(\theta)-\beta\,\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}\|\pi_{\mathrm{ref}}]\right)\right],(10)whereϵ\epsiloncontrols clipping andβ\betaweights the KL regularization. This objective jointly optimizes memory reconstruction and action generation using the same trajectory-level reward.

4Experiments

In this section, we empirically evaluate our proposed MemHarness framework on long-horizon interactive environments. Our evaluation is designed to answer the following core research questions (RQs):

• •RQ1:Does the MemHarness framework outperform existing agentic and memory-augmented baselines on complex interactive tasks? (Section4.2)
• •RQ2:How do individual components (especially the contextual memory reconstruction module) contribute to the overall performance? (Section4.3)
• •RQ3:Is MemHarness robust against environmental distribution shifts in out-of-distribution (OOD) scenarios? (Section4.4)
• •RQ4:Does the reconstruction mechanism genuinely rely on state comparison to make decisions? (Section4.5)
• •RQ5:How do adaptive memory utilization behaviors evolve during reinforcement learning, and how do they correlate with task success? (Section4.6)

4.1Experimental Setup

Benchmarks and Baselines.

We evaluate MemHarness on two complex, long-horizon interactive benchmarks: ALFWorldShridharet al.(2020)(embodied household tasks) and WebShopYaoet al.(2022)(goal-directed online shopping). Performance is measured by task success rate, together with the average task score on WebShop. We compare MemHarness against closed-source LLMs, including GPT-4oHurstet al.(2024)and Gemini-2.5-ProComaniciet al.(2025), prompt-based or memory-based agents (ReAct, Reflexion, Mem0, ExpeL, MemP, and SimpleMem)Yaoet al.(2023); Shinnet al.(2023); Chhikaraet al.(2025); Zhaoet al.(2024); Fanget al.(2026); Liuet al.(2026), and RL-trained agents (RLOO, GRPO, MemRL, EvolveR, Mem0+GRPO, and SimpleMem+GRPO)Ahmadianet al.(2024); Guoet al.(2025); Zhanget al.(2026); Wuet al.(2025); Liuet al.(2026). We also evaluate ablated variants with verbatim memory replay or the reconstruction harness removed at inference time to isolate the contribution of our design.

Implementation Details.

We adopt Qwen2.5-7B-InstructQwenet al.(2025)as the policy backbone. Our implementation builds on theverl-agentframeworkFenget al.(2026). Following Deepseek-R1 and EvolveRGuoet al.(2025); Wuet al.(2025), we perform a brief cold-start stage before RL training. For each benchmark, the cold-start dataset contains 200 multi-turn interaction trajectories with chain-of-thought reasoning and active retrieval of memories, as well as 200 question–answer examples that summarize trajectories into memories. Whenever memory is retrieved, the agent is required to reconstruct it before taking an action. This stage primarily aligns the model with the required interaction and memory-abstraction formats. The unified policy is subsequently optimized end-to-end via GRPO with sparse task rewards (1010for success and0for failure) and a minor formatting bonus. For fair comparison, all trainable baselines use the same backbone, observation history window (w=3w=3), interaction budget, and optimization configuration. We will show more details in the Appendix.

Memory Configuration.

The experiential memory bank is initialized as empty and progressively populated during RL training with memory principles summarized by the policy from its own interaction trajectories. We use BGE-M3Chenet al.(2024a)as the embedding model and retrieve the top-33most relevant principles.

MethodAlfWorldWebShopPickLookCleanHeatCoolPick2Avg. SRScoreSRClosed-source LLMsGPT-4o75.360.831.256.721.649.849.231.823.7Gemini-2.5-Pro92.863.362.169.026.658.762.142.535.9Qwen2.5-7B-InstructQwen2.533.421.619.36.92.83.214.526.47.8Prompt-based Agentic or Memory-based MethodsReAct48.535.434.313.218.217.627.946.219.5Reflexion62.041.644.930.936.323.839.958.128.8Mem054.055.026.936.420.87.733.523.92.0ExpeL21.067.055.052.071.06.045.330.911.2MemP54.338.548.156.232.016.741.025.36.4SimpleMem64.533.320.012.533.33.827.933.28.6RL-based MethodsRLOO87.678.287.381.371.948.975.980.365.7GRPO90.866.189.374.772.564.776.479.366.1Memory-Augmented RL-based MethodsMemRL62.838.522.212.58.00.024.029.59.2Mem0+GRPO78.154.856.131.065.026.952.058.137.5SimpleMem+GRPO89.536.360.050.064.926.354.567.846.9EvolveR (reproduced)84.861.595.762.561.954.270.184.172.6MemHarness87.078.697.087.571.490.085.287.475.6Table 1:Main results on AlfWorld and WebShop. For AlfWorld, we report the success rate (%) on each of the six task categories, along with the macro-averaged success rate (Avg. SR) across all categories. Best results are highlighted inbold.

4.2Main Results: Overall Effectiveness

Table1summarizes the performance on AlfWorld and WebShop. Overall, MemHarness achieves the best results, reaching success rates of 85.2% on AlfWorld and 75.6% on WebShop.

Comparison with Foundation Models.MemHarness demonstrates massive improvements over its base model, Qwen2.5-7B-Instruct, and substantially outperforms strong closed-source models like Gemini-2.5-Pro (+23.1% on AlfWorld and +39.7% on WebShop) despite its 7B parameter scale.

Advantage over Existing Paradigms.Prompt-based and test-time memory methods (e.g., Reflexion, ExpeL) yield limited success without policy optimization. While pure RL (GRPO) provides a strong baseline, naively combining it with external memory (e.g., Mem0+GRPO, SimpleMem+GRPO) severely degrades performance. MemHarness effectively overcomes this issue via state-conditioned memory reconstruction, safely leveraging historical experience to outperform standard GRPO by absolute margins of +8.8% and +9.5% on AlfWorld and WebShop, respectively.

4.3Ablation Studies: Component Analysis

To evaluate the contribution of each component (RQ2), we conduct ablation studies on ALFWorld and WebShop, with results summarized in Table2. We identify three key findings regarding memory utilization and policy optimization:

Necessity of RL.

The cold-start model, trained solely for format alignment, performs poorly (7.6% on ALFWorld), confirming that RL is necessary for acquiring actual interactive decision-making capabilities.

Raw Memory Introduces Noise.

Injecting unrefined memory (“RL + Raw Memory” or “w/o reconstruction”) hurts ALFWorld performance. This proves verbatim replay introduces state-mismatch noise, which our reconstruction mechanism effectively mitigates.

Intrinsic Policy Improvement.

Disabling test-time memory (“w/o memory”) still substantially outperforms the ‘RL Only‘ baseline (83.0% vs. 76.4% on ALFWorld). Thus, the reconstruction objective during training intrinsically enhances the agent’s base reasoning capabilities, serving as high-quality latent guidance.

Necessity of Policy-Internal Reconstruction.

Replacing MemHarness’s internal reconstruction module with a generic instruction-tuned LLM (Qwen2.5-7B-Instruct) while keeping the actor unchanged degrades performance significantly (e.g., 85.2% to 77.7% on ALFWorld). This demonstrates that zero-shot text rewriting is insufficient; end-to-end RL is crucial for learning task-grounded adaptation that aligns with environment dynamics.

Table 2:Ablation studies on the components of MemHarness. Memory and Reconstruction indicate whether memory retrieval and reconstruction modules are active during inference. Best results are inbold.

4.4Out-of-Distribution Generalization

To address RQ3, we evaluate robustness against environmental shifts on ALFWorld Out-of-Distribution (OOD) settings, where room layouts and object placements are unseen during training. In such scenarios, directly applying historical memories risks severe state mismatch.

As shown in Table3, MemHarness achieves the highest average success rate (85.9%), substantially outperforming the “RL + Raw Memory” baseline (76.3%). Notably, while the intrinsic policy (“w/o memory”) remains competitive (83.0%), injecting unadapted memories (“w/o reconstruction”) degrades performance to 82.4%. This confirms that verbatim memory replay in unseen environments introduces noise. MemHarness overcomes this by dynamically filtering and rewriting mismatched guidance, safely leveraging past experiences to achieve superior OOD generalization.

Table 3:Performance on the ALFWorld Out-of-Distribution (OOD) evaluation. The environment layouts and object configurations are unseen during training.(a) ALFWorld (b) WebShop

Figure 3:Evolution of memory usage behaviors during RL training.The curves show the average number of memory retrievals, accepted reconstructions, and rejected reconstructions per trajectory.(a) ALFWorld (b) WebShop

Figure 4:Training dynamics of success rates conditioned on memory decisions.“SR when Accepted/Rejected” denotes the success rate of trajectories containing at least one accepted or rejected memory reconstruction.

4.5Mechanism Deep Dive: How is Memory Reconstructed?

To answer RQ4, we investigate whether reconstruction genuinely relies on state comparison, examining its behavior from macro-level filtering to micro-level adaptation.

Macro-level Filtering.

We ablate the historical source state (os​r​co^{src}) provided during reconstruction (Table4). Removingos​r​co^{src}entirely leaves the rejection rate unchanged but degrades task success (e.g.,85.2%→80.0%85.2\%\to 80.0\%on ALFWorld), as the agent accepts misaligned guidance. Crucially, replacingos​r​co^{src}with a mismatched state from arandommemory markedly increases rejection rates (8.7%→13.3%8.7\%\to 13.3\%on ALFWorld,56.0%→63.3%56.0\%\to 63.3\%on WebShop). This confirms the policy actively compares historical and current states to filter incompatible memories, rather than reacting to missing fields, thereby maintaining robust downstream performance.

Micro-level Counterfactual Adaptation.

We construct 1,000 offline probes by asking a strong LLM to minimally edit each current state so that its retrieved memory becomes inapplicable, and evaluate only the reconstruction output rather than task execution (AppendixB). Table4reports the resulting within-environment shifts. In ALFWorld, counterfactual edits reduce unchanged outputs and increase adaptation and rejection. In WebShop, they further increase rejection. The different response patterns are consistent with their observation formats: ALFWorld provides concise state descriptions, whereas WebShop exposes substantially longer webpage content. Despite this difference, both show that reconstruction responds to fine-grained state changes.

(a) Source-state ablation

(b) Counterfactual behavior

Table 4:Analysis of memory reconstruction.(a)Ablation on the memory source stateos​r​co^{src}, where RR and SR denote the per-memory average reject rate and task average success rate, respectively.(b)Behavioral shifts under minimal factual edits (s−s^{-}).

4.6Training Dynamics and Behavioral Evolution (RQ5)

To address RQ5, we track internal memory decisions and task success rates during RL training. MemHarness learns task-specific memory strategies rather than fixed rules (Figure3). ALFWorld converges to a sparse policy (retrievals dropping to 2–3 per trajectory), whereas WebShop maintains frequent retrieval with persistent active filtering. These learned decisions align directly with task outcomes (Figure4). Across both environments, trajectories containingacceptedreconstructions consistently achieve higher success rates than those withrejectedones, confirming the agent effectively learns to judge memory applicability.

5Conclusion

We introducedMemHarness, an RL-driven framework that shifts memory-augmented agents fromverbatim replaytostate-conditioned reconstruction. Optimized end-to-end via GRPO, a single unified policy learns to autonomously critique and rewrite historical experiences into state-specific guidance before acting. Experiments on ALFWorld and WebShop show our 7B model significantly outperforms closed-source models and RL baselines, especially in out-of-distribution environments. Analyses reveal this mechanism actively filters mismatched noise and fundamentally improves the agent’s intrinsic reasoning, even when memory is unavailable at inference. Future work will explore scaling to larger models and open-ended environments.

References

• [1](2024-08)Back to basics: revisiting REINFORCE-style optimization for learning from human feedback in LLMs.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),L. Ku, A. Martins, and V. Srikumar (Eds.),Bangkok, Thailand,pp. 12248–12267.External Links:Link,DocumentCited by:§2.1,§4.1.
• [2]B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao(2023)Fireact: toward language agent fine-tuning.arXiv preprint arXiv:2310.05915.Cited by:§2.1.
• [3]J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu(2024)Bge m3-embedding: multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216.Cited by:§A.3,§4.1.
• [4]L. Chen, L. Wang, H. Dong, Y. Du, J. Yan, F. Yang, S. Li, P. Zhao, S. Qin, S. Rajmohan,et al.(2023)Introspective tips: large language model for in-context decision making.arXiv preprint arXiv:2305.11598.Cited by:§1.
• [5]Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao(2024)Agent-flan: designing data and methods of effective agent tuning for large language models.InFindings of the Association for Computational Linguistics: ACL 2024,pp. 9354–9366.Cited by:§2.1.
• [6]P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav(2025)Mem0: building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413.Cited by:§2.2,§4.1.
• [7]G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen,et al.(2025)Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261.Cited by:§4.1.
• [8]R. Fang, Y. Liang, X. Wang, J. Wu, S. Qiao, P. Xie, F. Huang, H. Chen, and N. Zhang(2026)Memp: exploring agent procedural memory.InFindings of the Association for Computational Linguistics: ACL 2026,pp. 17490–17502.Cited by:§2.2,§4.1.
• [9]L. Feng, Z. Xue, T. Liu, and B. An(2026)Group-in-group policy optimization for llm agent training.Advances in Neural Information Processing Systems38,pp. 46375–46408.Cited by:§2.1,§4.1.
• [10]D. Fu, J. Mei, L. Wen, X. Yang, C. Yang, R. Wu, T. Hu, S. Li, Y. Shen, X. Cai,et al.(2025)RE-searcher: robust agentic search with goal-oriented planning and self-reflection.arXiv preprint arXiv:2509.26048.Cited by:§1.
• [11]D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi,et al.(2025)Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948.Cited by:§4.1,§4.1.
• [12]A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford,et al.(2024)Gpt-4o system card.arXiv preprint arXiv:2410.21276.Cited by:§4.1.
• [13]B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han(2025)Search-r1: training llms to reason and leverage search engines with reinforcement learning.External Links:2503.09516,LinkCited by:§1.
• [14]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica(2023)Efficient memory management for large language model serving with pagedattention.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:4th item.
• [15]J. Liu, Y. Su, P. Xia, S. Han, Z. Zheng, C. Xie, M. Ding, and H. Yao(2026)SimpleMem: efficient lifelong memory for llm agents.arXiv preprint arXiv:2601.02553.Cited by:§2.2,§4.1.
• [16]E. F. Loftus and J. C. Palmer(1974)Reconstruction of automobile destruction: an example of the interaction between language and memory.Journal of verbal learning and verbal behavior13(5),pp. 585–589.Cited by:§1.
• [17]E. Ma, L. Zhou, T. Tang, J. Zhang, J. Jiang, Z. Zhang, D. Han, K. Zhan, X. Zhang, X. Lang,et al.(2026)Correctad: a self-correcting agentic system to improve end-to-end planning in autonomous driving.InProceedings of the AAAI Conference on Artificial Intelligence,Vol.40,pp. 7755–7763.Cited by:§2.1.
• [18]J. Mei, T. Hu, D. Fu, L. Wen, X. Yang, R. Wu, P. Cai, X. Cai, X. Gao, Y. Yang,et al.(2025)O2O^{2}-Searcher: a searching-based agent model for open-domain open-ended question answering.arXiv preprint arXiv:2505.16582.Cited by:§1.
• [19]Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu(2025)Qwen2.5 technical report.External Links:2412.15115,LinkCited by:§A.2,§4.1.
• [20]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov(2017)Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347.Cited by:§2.1.
• [21]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu,et al.(2024)Deepseekmath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.Cited by:§A.2,§2.1.
• [22]G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu(2024)HybridFlow: a flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256.Cited by:§A.2.
• [23]N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao(2023)Reflexion: language agents with verbal reinforcement learning.External Links:2303.11366,LinkCited by:§2.1,§4.1.
• [24]M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht(2020)Alfworld: aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768.Cited by:§A.1,§4.1.
• [25]J. Wang, X. Yi, R. Guo, H. Jin, P. Xu, S. Li, X. Wang, X. Guo, C. Li, X. Xu,et al.(2021)Milvus: a purpose-built vector data management system.InProceedings of the 2021 International Conference on Management of Data,pp. 2614–2627.Cited by:§A.3.
• [26]L. Wang, X. Zhang, H. Su, and J. Zhu(2024)A comprehensive survey of continual learning: theory, method and application.External Links:2302.00487,LinkCited by:§1.
• [27]J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou,et al.(2022)Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35,pp. 24824–24837.Cited by:§2.1.
• [28]R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y. Shen, Y. Wang,et al.(2025)Evolver: self-evolving llm agents through an experience-driven lifecycle.arXiv preprint arXiv:2510.16079.Cited by:§A.3,§2.2,§3.1,§4.1,§4.1.
• [29]Z. Xi, Y. Ding, W. Chen, B. Hong, H. Guo, J. Wang, D. Yang, C. Liao, X. Guo, W. He, S. Gao, L. Chen, R. Zheng, Y. Zou, T. Gui, Q. Zhang, X. Qiu, X. Huang, Z. Wu, and Y. Jiang(2024)AgentGym: evolving large language model-based agents across diverse environments.External Links:2406.04151Cited by:§A.4.
• [30]W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang(2026)A-mem: agentic memory for llm agents.Advances in Neural Information Processing Systems38,pp. 17577–17604.Cited by:§1.
• [31]S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, H. Schütze, V. Tresp, and Y. Ma(2025)Memory-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828.Cited by:§1.
• [32]C. Yang, X. Yang, L. Wen, D. Fu, J. Mei, R. Wu, P. Cai, Y. Shen, N. Deng, J. Xu,et al.(2026)Towards self-evolving agents: enabling autonomy through interactive experience refinement.InFindings of the Association for Computational Linguistics: ACL 2026,pp. 30424–30451.Cited by:§1.
• [33]S. Yao, H. Chen, J. Yang, and K. Narasimhan(2022)WebShop: towards scalable real-world web interaction with grounded language agents.InAdvances in Neural Information Processing Systems,S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.),Vol.35,pp. 20744–20757.External Links:LinkCited by:§A.1,§4.1.
• [34]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao(2023)ReAct: synergizing reasoning and acting in language models.External Links:2210.03629,LinkCited by:§2.1,§4.1.
• [35]G. Zhang, M. Fu, G. Wan, M. Yu, K. Wang, and S. Yan(2025)G-memory: tracing hierarchical memory for multi-agent systems.External Links:2506.07398,LinkCited by:§1.
• [36]G. Zhang, M. Fu, and S. Yan(2025)Memgen: weaving generative latent memory for self-evolving agents.arXiv preprint arXiv:2509.24704.Cited by:§1,§2.2.
• [37]S. Zhang, J. Wang, R. Zhou, J. Liao, Y. Feng, Z. Li, Y. Zheng, W. Zhang, Y. Wen, Z. Li,et al.(2026)Memrl: self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192.Cited by:§2.2,§4.1.
• [38]A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang(2024)ExpeL: llm agents are experiential learners.External Links:2308.10144,LinkCited by:§2.2,§4.1.

Appendix

Contents

Appendix AImplementation Details

A.1Environment and Evaluation Setup

We evaluate our method on two challenging interactive benchmarks: ALFWorld and WebShop.ALFWorldShridharet al.(2020)is an embodied text-based environment containing six sub-task categories (e.g., Pick, Look, Clean). The maximum interaction horizon is set to 50 steps, with a maximum prompt length of 2048 tokens and a generation limit of 512 tokens.WebShopYaoet al.(2022)is an e-commerce website simulation containing over 1.1 million products. The maximum interaction horizon is restricted to 15 steps, with a maximum prompt length of 4096 tokens and a generation limit of 512 tokens.

Reward Function Design:Across both environments, we employ a straightforward outcome-driven reward function combined with a format penalty. The agent receives a sparse outcome reward of+10+10upon successful task completion and0for failure. To encourage adherence to the prescribed interaction protocol, we add a small formatting bonus, weighted by0.10.1, to the sparse outcome reward. The format score equally weights three criteria: (1) each step contains exactly one valid block followed by one valid<action>…</action>block, (2) memory is retrieved through valid<retrieve_memory>…</retrieve_memory>blocks between one and five times per episode, (3) thinking contents are including by one valid<think>…</think>block and (4) all generated content is strictly in English.

A.2Training Hyperparameters

Our reinforcement learning optimization is based on Group Relative Policy Optimization (GRPO)Shaoet al.(2024)and implemented using the veRLShenget al.(2024)framework. We initialize the policy with instruction-tuned base models Qwen2.5-7BQwenet al.(2025)and freeze the reference policyπr​e​f\pi_{ref}during training to compute the KL divergence penalty.

The core hyperparameter settings are consistent across both ALFWorld and WebShop:

• •Rollout & Sampling:During the environment interaction phase, we sample a group size ofG=8G=8trajectories for each prompt. The rollout generation temperature is set to1.01.0, and the validation temperatue is set to0.40.4. We sample 16 different groups per rollout, utilizing a total of 128 parallel environments.
• •Optimization:The policy network is updated using the Adam optimizer with a learning rate of1×10−61\times 10^{-6}.
• •GRPO Objectives:The PPO-style clipping parameter is set toϵ=0.2\epsilon=0.2. To prevent the policy from degrading and deviating too far from the base model, the KL divergence penalty coefficient is set toβ=0.01\beta=0.01.
• •Batching:The training utilizes a mini-batch size of 256 for Alfworld and 64 for Webshop with 1 tensor parallel size. We use vLLMKwonet al.(2023)for high-throughput rollout generation with a GPU memory utilization ratio of 0.75.

A.3Memory Bank and Retrieval Setup

Following prior work, we implement the memory storeℬ\mathcal{B}as a vector database using MilvusWanget al.(2021)for efficient similarity search. Each memory entry consists of a distilled experienceeie_{i}paired with its source interaction stateois​r​co_{i}^{src}. We use BGE-M3Chenet al.(2024a)as the embedding model to encode both the retrieval queryqtq_{t}and the stored entries, and retrieve the top-kk(k=3k=3) most relevant experiences at each step based on cosine similarity.

Memory Initialization.Inspired by prior workWuet al.(2025), the memory bank is not manually curated but is instead populated online through the policy’s own interactions. During training, the policy modelπθ\pi_{\theta}distills successful and failed trajectories into concise natural-language principles paired with its source interaction stateois​r​co_{i}^{src}, which are then inserted intoℬ\mathcal{B}. To ensure the efficiency of the pipeline, we retain only 50% of the trajectories generated during the GRPO process for distillation. Specifically, we prioritize a balanced retention of successful and failed trajectories; when such a balance cannot be achieved, the remaining quota is filled by randomly sampling from the leftover trajectories.

Deduplication and Pruning.To prevent the memory bank from growing unboundedly and accumulating redundant entries (e.g., near-identical principles arising from GRPO group sampling), we apply curation at three stages.(1) Write-time deduplication:before inserting a new memory entry, we perform an embedding-based nearest-neighbor probe; if its maximum cosine similarity to any existing entry exceeds the thresholdθs​i​m\theta_{sim}(default0.850.85), the insertion is skipped. In practice, we enable it for WebShop and disable it for ALFWorld.(2) Retrieval-time deduplication:at inference, we first retrieve a larger candidate pool from bank, then apply greedy embedding-based deduplication before truncating to the top-kkresults, ensuring the returned memories are not semantically redundant.(3) Capacity and quality maintenance:we adopt an EvolveR-style Laplace-smoothed utility scores​(p)=(cs​u​c​c​(p)+1)/(cu​s​e​(p)+2)s(p)=(c_{succ}(p)+1)/(c_{use}(p)+2), and periodically prune low-utility entries whose score falls below the pruning thresholdθp​r​u​n​e\theta_{prune}(default0.30.3) once they have been retrieved at leastmin_uses_before_prunetimes (default 3).

A.4Cold-Start Dataset Construction

We construct a small cold-start dataset to familiarize the policy with the interaction protocol, particularly how to initiate memory retrieval and summarize trajectories into reusable memories. For each benchmark, we collect seed interaction trajectories from the AgentGym datasetXiet al.(2024).

The cold-start dataset contains two subsets, each comprising200examples per benchmark. First, we constructmemory-augmented interaction trajectories. For each seed trajectory, we randomly sample an intermediate decision points and insert a memory-retrieval turn. GPT-5.1 generates a context-appropriate retrieval query from the task description and interaction prefix at the selected point. The retrieved memory is produced by asking GPT-5.1 to distill the corresponding source trajectory into concise, reusable guidance. The resulting trajectory therefore demonstrates the complete interaction protocol, including issuing a retrieval request, receiving an experience from the memory bank, and continuing environment interaction in the required output format.

Second, we construct 200trajectory-to-memory summarization pairs. Each pair takes a completed AgentGym trajectory as input and uses its GPT-5.1 generated summary as the target, following the structured memory schema described in AppendixC. These examples teach the policy to convert interaction histories into compact memories containing a situational precondition and a reusable principle.

We combine the two subsets and perform supervised fine-tuning for 2 epochs with a learning rate of1×10−51\times 10^{-5}and an effective batch size of 2. GPT-5.1 is used only for offline cold-start data construction and is not involved in RL training or evaluation. The purpose of this stage is protocol and format alignment rather than task-skill acquisition; consistent with this role, the cold-start model alone achieves limited task performance, while the actual reconstruction and decision-making capabilities are subsequently learned through task-level reinforcement learning.

Appendix BCounterfactual Probe Construction

To evaluate whether MemHarness genuinely relies on fine-grained state comparison rather than superficial pattern matching, we constructed a counterfactual probe dataset comprising 2,000 examples (1,000 for ALFWorld and 1,000 for WebShop). For each instance, we sample a real interaction state (s+s^{+}) and a genuinely retrieved memory (mm) that is highly applicable. We then use a strong LLM to minimally edit the state to produce a counterfactual state (s−s^{-}), renderingmmeither inapplicable or misleading.

B.1Generation Prompt

We use GPT-5.1 to generate the counterfactual states. The prompt (Appendix C) enforces that the edits remain minimal and structurally identical to the original observations, preventing the agent from relying on formatting artifacts to reject the memory.

B.2Examples of State Edits

Below we provide concrete examples of the factual edits for both benchmarks. In these examples, the MemHarness agent successfully accepts the memory under the original state (s+s^{+}) but adaptively rewrites or rejects it when presented with the minimal edit (s−s^{-}).

ALFWorld Example: Premise Invalidation.

In ALFWorld, memories often provide search heuristics based on past failures. The factual edit invalidates the heuristic’s premise by revealing the target object.

• •Task:heat some mug and put it in coffeemachine.
• •Retrieved Memory (mm):If you cannot find a mug in nearby cabinets or drawers after opening several of them, shift your search to visible surfaces like countertops or the sinkbasin.
• •Original State (s+s^{+}):On the drawer 5, you see nothing.
• •Edited State (s−s^{-}):On the drawer 5, you see a mug.
• •Agent Behavior Shift:Ins+s^{+}, the agent accepts the memory and redirects its search to open surfaces. Ins−s^{-}, the premise (failing to find a mug) is broken; the agent rejects the memory (<EMPTY>) and directly picks up the mug.

WebShop Example: Constraint Violation.

In WebShop, memories often recommend finalizing a purchase when the product matches all instructions. The edit subtly alters a product attribute, violating the user’s initial constraint.

• •Task:Find me machine wash men’s t-shirts … and price lower than 50.00 dollars.
• •Retrieved Memory (mm):After selecting all required attributes … and confirming the price constraint is satisfied, finish the task by clicking the “Buy Now” button.
• •Original State (s+s^{+}):… “Price: $19.99” [SEP] “Rating: N.A.” … “Buy Now”
• •Edited State (s−s^{-}):… “Price: $59.99” [SEP] “Rating: N.A.” … “Buy Now”
• •Agent Behavior Shift:Ins+s^{+}, the agent follows the memory to click “Buy Now”. Ins−s^{-}, noticing the price now exceeds the $50 limit, the agent rejects the memory to prevent a wrong purchase and navigates back to search.

Appendix CPrompt Templates

This section presents the prompt templates used for memory retrieval, state-conditioned memory reconstruction, agent–environment interaction, trajectory summarization, and fallback reasoning when no adapted memory. Text enclosed in braces denotes a runtime placeholder.

Environment feedback Prompt for Agent

The agent system prompts specify the environment context, interaction history, admissible actions, and the required reasoning–action output format. The templates for ALFWorld and WebShop are presented in Figures5and6, respectively. The memory retrieval instruction guides the agent to issue a retrieval query when additional experience is needed, while preventing retrieval and environment actions from appearing in the same response.

Alfworld Environment feedback Prompt for AgentYou are an expert agent operating in the ALFRED Embodied Environment. Your task is to:{task_description}Prior to this step, you have already taken{step_count}step(s). Below are the most recent{history_length}observations and the corresponding actions you took:{action_history}You are now at step{current_step}, and your current observation is:{current_observation}Your admissible actions in the current situation are:{admissible_actions}.Now it is your turn to take an action. First, reason step by step about the current situation. The reasoning process must be enclosed within<think>and</think>tags. After finishing the reasoning, choose one admissible action for the current step and present it within<action>and</action>tags.You have access to a memory bank of past experiences from similar tasks. When you are uncertain about how to proceed, prefer retrieving relevant memory for guidance before taking an environment action. To retrieve memory, output exactly one search query wrapped by<retrieve_memory>and</retrieve_memory>tags. Do not output both a memory retrieval query and an action in the same response. After the memory is returned, take an environment action in the next response.Figure 5:Agent system prompts used in ALFWorld.Webshop Environment feedback Prompt for AgentYou are an expert autonomous agent operating in the WebShop e-commerce environment.Your task is to:{task_description}.Prior to this step, you have already taken{step_count}step(s). Below are the most recent{history_length}observations and the corresponding actions you took:{action_history}You are now at step{current_step}, and your current observation is:{current_observation}.Your admissible actions in the current situation are:[{available_actions}].Now it is your turn to take one action for the current step. First, reason step by step about the current situation and determine which admissible action best advances the shopping goal. The reasoning process must be enclosed within<think>and</think>tags. After finishing the reasoning, choose one admissible action and present it within<action>and</action>tags.You have access to a memory bank of past experiences from similar tasks. When you are uncertain about how to proceed, prefer retrieving relevant memory for guidance before taking an environment action. To retrieve memory, output exactly one search query wrapped by<retrieve_memory>and</retrieve_memory>tags. Do not output both a memory retrieval query and an action in the same response. After the memory is returned, take an environment action in the next response.Figure 6:Agent system prompts used in WebShop, with and without recent interaction history.

Self-Reasoning Fallback Prompt

When no retrieved principle is applicable, the fallback prompt instructs the agent to rely on the current observation and its own reasoning instead. The corresponding prompt is shown in Figure7.

Self-Reasoning Fallback InstructionNo validated memory principle applies; rely on observation and reasoning.Figure 7:Prompt used to self-reasoning fallback

Contextual Memory Reconstruction Prompt

The reconstruction prompt adapts a retrieved historical principle to the current task and state, returning<EMPTY>when the principle is not applicable. The full template is presented in Figure8.

State-Conditioned Memory Reconstruction PromptSystem PromptYou adapt a retrieved memory principle into concise, reusable guidance for the current situation and initial task, or output exactly<EMPTY>if the principle does not apply.Do not write chain-of-thought, first-person reasoning, or a step-by-step action plan.User Message TemplateInitial task:{task}Current situation (state):{s_curr}Retrieved historical state:{s_old}Historical principle (memory):{p_old}Output Rules:•Output exactly one short adapted principle, or exactly<EMPTY>.•Write in an imperative or neutral guidance style. •Do not use first person (e.g., “I,” “my,” or “we”) or phrases such as “therefore,” “I need to,” or “since”.•Do not explain the reasoning or describe multiple next steps.Figure 8:Prompt used to reconstruct a retrieved memory principle according to the current environment state.

Trajectory Summarization Prompts

After an episode is completed, the trajectory summarization prompt extracts concise and reusable memories grounded in the interaction trajectory. Its required JSON output schema and extraction constraints are provided in Figure9.

Trajectory Summarization PromptSystem PromptYou are a JSON-only memory extractor.Read a completed agent trajectory and write reusable advice for future similar states.Critical Rules:•Return exactly one JSON object and nothing else.•Do not continue the trajectory.•Do not write thoughts, actions, markdown, XML tags, or explanations.•Do not copy raw<think>,<action>, or<retrieve_memory>text.•Each memory must be grounded in the trajectory and useful later.User Message TemplateBenchmark: “{task_name}”.Task: extract at most{num_memories}reusable memories from the completed trajectory below.Return only this JSON shape:{ ‘‘memories’’: [ { ‘‘situation’’: ‘‘one short sentence ...’’, ‘‘memory’’: ‘‘one short sentence ...’’ } ]}Field Rules:•situation: generalized state or precondition, not a full observation dump.•memory: reusable advice, not a recap and not a next action command.•If no useful memory exists, return{‘‘memories’’: []}.Trajectory:{trajectory_text}Figure 9:Prompt used for offline trajectory summarization. The model distills raw interaction experiences into a structured JSON format containing situational preconditions and reusable guidance.

Counterfactual Probe Construction Prompts

This section details the prompts used to construct the counterfactual probes for the analysis in Section4.5. The complete template is shown in Figure10and Figure11.

ALFWorld Counterfactual Probe Generation Prompt[System Instructions]You create ONE counterfactual mismatched observation for a memory probe.You are given a matched current observation (s_plus) where a retrieved memory applies. Produces_minus:s_pluswith ONE minimal factual change so the memory NO LONGER applies.Rules:- Keep the same observation style ass_plus(AlfWorld), including entity names, ids, and length.- Change only one applicability-related fact (see memory preconditions / principle).- Do NOT produce unnatural or gibberish text.- Output valid JSON only (no markdown fences):{‘‘s_minus’’: ‘‘...’’, ‘‘flip’’: ‘‘one-line description of the single fact you changed’’}[User Input]Initial task (fixed):{task}Retrieved historical state s_old (fixed; NOT the current observation):{s_old}Memory principle p_old:{p_old}Matched current observation s_plus (fixed; do NOT rewrite):{s_plus}Memory metadata (for applicability):{metadata_json}Return JSON with exactly: s_minus, flip.Figure 10:Prompt used to generate counterfactual mismatched observations for ALFWorld.WebShop Counterfactual Probe Generation Prompt[System Instructions]You create ONE counterfactual mismatched WebShop observation for a memory probe.You are given a matched current observation (s_plus) where a retrieved memory applies. Produces_minus:s_pluswith ONE minimal factual change so the memory NO LONGER applies.WebShop format rules:-s_minusMUST look like a WebShop formatted observation: ’segment’ [SEP] “segment” [SEP] …- Every segment MUST be wrapped in single quotes.- Preserve [SEP] separators exactly as ins_plus.- Change only ONE segment/value (or remove/add one segment) so the memory no longer applies.- Do NOT output plain prose (e.g. “No Search”, “nothing available”) without quoted [SEP] segments.- Keep overall length and structure similar tos_plus.Output valid JSON only (no markdown fences):{‘‘s_minus’’: ‘‘...’’, ‘‘flip’’: ‘‘one-line description of the single fact you changed’’}[User Input]Initial task (fixed):{task}Retrieved historical state s_old (fixed; NOT the current observation):{s_old}Memory principle p_old:{p_old}Matched current observation s_plus (fixed; do NOT rewrite):{s_plus}Memory metadata (for applicability):{metadata_json}Return JSON with exactly: s_minus, flip.Figure 11:Prompt used to generate counterfactual mismatched observations for WebShop.

Appendix DCase Studies

To intuitively illustrate how MemHarness bridges the gap between historical knowledge and current interaction states, we provide two real reconstruction cases from our evaluation. These examples demonstrate why verbatim replay of past experiences often fails and how our policy actively rewrites memory to provide state-grounded guidance.

D.1ALFWorld: Abstracting and Transferring Skills

In ALFWorld, verbatim replay of the retrieved memory would introduce hallucinated objects (“credit card” and “coffee table”) into the context, likely causing the agent to output an invalid action. As shown in Figure12, MemHarness instead extracts the underlying procedural skill (placing a held item at the final destination after a prerequisite step) and explicitly grounds it with the current entities (“cup” and “sidetable”).

Case Study: ALFWorld Skill TransferIntial Task:cool some cup and put it in sidetableCurrent State (sts_{t}):You cool the cup 2 using the fridge 1.Historical State (ss​r​cs_{src}):Holding a creditcard in inventory; at the coffeetable where two credit cards need to be placed on it; the task is to put two credit cards on the coffeetable.Retrieved Memory (mm):Once you have acquired and prepared the target object (e.g., a credit card), proceed directly to the final receptacle (e.g., the coffee table) and place it there to complete the goal.MemHarness Reconstruction (gtg_{t}):If you are holding a target object (here, a cup) and you have already used the fridge to cool it, the next step is to place it at the destination location (here, the sidetable). The fridge is not required in this step; it was only needed to cool the cup. The sidetable is the goal location where the cup should be placed.Figure 12:ALFWorld case demonstrating how MemHarness prevents hallucination by rewriting a mismatched historical memory into applicable, grounded guidance.

D.2WebShop: Grounding Heuristics into Actionable Targets

In WebShop, the retrieved memory is often an abstract heuristic (e.g., advising the agent to cross-check specific constraints before clicking). A naive replay would simply append this abstract rule to the prompt, offering no concrete operational help. As shown in Figure13, MemHarness executes this heuristic during the reconstruction phase: it scans the current noisy observation, aligns it with the user’s constraints, and directly outputs the exact target ID to act upon.

Case Study: WebShop Heuristic GroundingIntial Task:Find me day comfort, anti slip, non slip women’s oxfords with high heel, closed toe, ankle strap, memory foam, rubber sole with color: black, and size: 6Current State (sts_{t}):‘‘Back to Search’’ [SEP] ‘‘Page 1 (Total results: 50)’’ [SEP] ‘‘Next >’’ [SEP] ‘‘B06W51MMKY’’ [SEP] ‘‘Alegria Women’s Kourtney’’ [SEP] ‘‘$62.99 to $129.99’’ [SEP] ... [Other 9 products omitted for brevity] ...Historical State (ss​r​cs_{src}):On a WebShop search results page with many clothing items listed only by ID, color, size, and price ranges, with no visible product details yet.Retrieved Memory (mm):When facing a search results page with mixed items and highly detailed constraints (e.g., fit, fabric, color, size), carefully cross-reference the visible titles and attributes against all instructions before clicking a specific product ID.MemHarness Reconstruction (gtg_{t}):B06W51MMKY: Alegria Women’s Kourtney - seems to match the criteria of day comfort, anti slip, non slip women’s oxfords with high heel, closed toe, ankle strap, memory foam, rubber sole with color: black, and size: 6.Figure 13:WebShop case demonstrating how MemHarness translates an abstract heuristic rule into a concrete, actionable target based on the current observation.