The most dangerous moment for an agent’s memory is often when it trusts the past too much. Many memory agents, after finding similar experiences, directly stuff them into the context. But similar tasks do not mean the current state is the same; old experience can sometimes lead decisions astray. This paper proposes MemHarness, which changes the process of an agent using memory into three steps: first retrieve past experience, then judge and reconstruct it in light of the current state, and finally generate actions. Useful parts are retained, while mismatched content is revised or directly discarded. The entire capability is trained end-to-end with GRPO and task rewards, without requiring additional annotation of the reconstruction process. Based on Qwen2.5-7B, MemHarness achieves success rates of 85.2% and 75.6% on ALFWorld and WebShop, respectively, improving over pure GRPO by 8.8 and 9.5 percentage points. In unseen environments, it reaches 85.9%, whereas directly replaying raw memory only achieves 76.3%. This shows that misused memory can indeed be worse than no memory at all. This work advances agent memory from “preserving the past” to “reinterpreting the past based on the present.” A truly reliable memory system needs to know which experiences are worth keeping, which need rewriting, and which should be forgotten in time.

arxiv: https://arxiv.org/abs/2607.28272

1 Introduction

Source: https://arxiv.org/html/2607.28272

MemHarness: Memory Is Reconstructed, Not Replayed

Rong Wu1,2,Daocheng Fu2,3,Licheng Wen2,4,5,Xuemeng Yang2,Shu Zou6,Jianbiao Mei2,Yuxin Wang7,Hairong Zhang2,Yu Yang1,2,Tao Hu2,Cong Zhang1,Botian Shi2,Pinlong Cai†,2,1

1Zhejiang University,2Shanghai Artificial Intelligence Laboratory
3Fudan University,4Shanghai Innovation Institute
5Shanghai Jiao Tong University,6The Australian National University
7University of Science and Technology of China

†Corresponding Author

Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent’s current situation. This “replay” paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we proposeMemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent’s intrinsic reasoning capabilities.

Date:Github Repo:https://github.com/KnowledgeXLab/MemHarness

Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents for sequential decision-makingChenet al.(2023b (https://arxiv.org/html/2607.28272#bib.bib2)); Fuet al.(2025 (https://arxiv.org/html/2607.28272#bib.bib26)); Jinet al.(2025 (https://arxiv.org/html/2607.28272#bib.bib30)); Meiet al.(2025 (https://arxiv.org/html/2607.28272#bib.bib25)). Experiential memory further enables these agents to reuse prior successes and avoid repeated mistakesWanget al.(2024 (https://arxiv.org/html/2607.28272#bib.bib33)); Yanet al.(2025 (https://arxiv.org/html/2607.28272#bib.bib29)); Xuet al.(2026 (https://arxiv.org/html/2607.28272#bib.bib32)). However, most memory-augmented agents follow averbatim replayparadigm: retrieved trajectories or principles are treated as static records and directly inserted into the model contextYanget al.(2026 (https://arxiv.org/html/2607.28272#bib.bib27)); Zhanget al.(2025a (https://arxiv.org/html/2607.28272#bib.bib31)). This design conflates retrieval relevance with action-level applicability. A memory may be semantically relevant to the task yet inappropriate for the current interaction state, because it was formed under different environmental conditions. An alternative line of work internalizes experience into model parameters, enabling state-conditioned behavior without explicitly replaying retrieved recordsZhanget al.(2025b (https://arxiv.org/html/2607.28272#bib.bib39)). While such parametric memory offers adaptive generation, the underlying experience and its influence on a decision remain implicit and difficult to inspect or revise. Explicit memory banks provide greater traceability, but typically lack this adaptive use of experience. This exposes a desirable middle ground: retaining explicit, attributable memories while dynamically reconstructing them for the current state. This middle ground echoes a fundamental insight from cognitive science: human remembering is reconstructive rather than reproductiveLoftus and Palmer (1974 (https://arxiv.org/html/2607.28272#bib.bib28)). Recall does not recover an immutable record of the past; instead, retrieved experience is interpreted and reorganized using present cues and prior knowledge before it informs behavior. As illustrated in Figure1 (https://arxiv.org/html/2607.28272#S1.F1), prior agents largely connect retrieval directly to action generation, whereas human memory introduces an intermediate process that evaluates and adapts recalled experience. This suggests a broader paradigm shift for memory-augmented agents—fromretrieve and replaytoretrieve, evaluate, and reconstruct. Inspired by this mechanism, we decompose memory-guided decision-making into five stages that parallel the reconstructive process:environment observationcaptures the current situation;experience retrievalrecalls a potentially relevant memory together with its historical context;memory critiqueassesses its applicability and identifies state mismatches;contextual memory reconstructionpreserves transferable knowledge while revising or discarding incompatible content; andaction generationuses the reconstructed guidance to make a context-aligned decision.

Refer to caption

Figure 1:Memory utilization paradigms.Top:Prior methods directly replay retrieved memories, risking state misalignment.Middle:Human memory reconstructs past experience according to the current context.Bottom:Inspired by this process, MemHarness reconstructs retrieved memories into state-aligned guidance.

Based on this formulation, we introduceMemHarness, designed to actively harness retrieved experiences by explicitly inserting critique and reconstruction between retrieval and action, turning memory from a static prompt fragment into context-sensitive guidance. Building on this formulation,MemHarnessbridges explicit and parametric memory utilization: it retains retrieved experiences as inspectable records, while parameterizing their state-conditioned critique and reconstruction within a single policy. At each decision step, conditioned on the current observation, agent first critiques the retrieved experience against its original context, then reconstructs it into state-specific guidance, and finally generates the executable action. Crucially, this reconstructive ability requires no additional human annotation: it emerges through end-to-end training with Group Relative Policy Optimization (GRPO). By optimizing for task success, the agent implicitly learns a discriminative reconstruction strategy, preserving transferable memories while rewriting misleading ones to close the gap between historical knowledge and the present state. We evaluate MemHarness on two challenging agent decision-making benchmarks, ALFWorld and WebShop. It consistently outperforms both pure RL baselines and state-of-the-art static memory-augmented methods. Ablation studies further show that removing the reconstruction stage degrades performance to the level of naive memory replay, confirming that adaptive reconstruction, rather than retrieval alone, is the primary driver of the observed gains. In summary, our main contributions are as follows:

• •A reconstructive view of agent memory.We identify the applicability gap underlying the prevalentverbatim replayparadigm, and, drawing on the reconstructive nature of human memory, recast memory-guided decision-making as a five-stage process spanning observation, retrieval, critique, reconstruction, and action.
• •A reconstruction-centric framework.We proposeMemHarness, which actively harnesses retrieved experiences by inserting explicit memory reconstruction between retrieval and action, and learns this capability end-to-end via GRPO without external supervision, combining the traceability of explicit memory with the adaptivity of parameterized reconstruction.
• •Comprehensive empirical validation.On interactive decision-making benchmarks, MemHarness delivers robust improvements over static replay, with ablations isolating reconstruction as the key contributing factor.

2Related Works

2.1LLM Agents for Interactive Decision-Making

LLM agents navigate interactive tasks using prompting paradigms like ReActYaoet al.(2023 (https://arxiv.org/html/2607.28272#bib.bib3)); Weiet al.(2022 (https://arxiv.org/html/2607.28272#bib.bib35)), often augmented with planning or self-reflectionShinnet al.(2023 (https://arxiv.org/html/2607.28272#bib.bib4)); Maet al.(2026 (https://arxiv.org/html/2607.28272#bib.bib20)). However, these training-free methods are bounded by frozen model capabilities. To acquire task-specific skills from interaction, agents are fine-tuned via supervised learningChenet al.(2024b (https://arxiv.org/html/2607.28272#bib.bib21),2023a (https://arxiv.org/html/2607.28272#bib.bib22))or reinforcement learning (RL)Schulmanet al.(2017 (https://arxiv.org/html/2607.28272#bib.bib23)); Ahmadianet al.(2024 (https://arxiv.org/html/2607.28272#bib.bib16)); Shaoet al.(2024 (https://arxiv.org/html/2607.28272#bib.bib17)); Fenget al.(2026 (https://arxiv.org/html/2607.28272#bib.bib7)). While RL optimizes decision-making directly, standard frameworks lack explicit mechanisms to accumulate and reuse cross-episode experiences, motivating memory-augmented agents.

2.2Memory-Augmented LLM Agents

Memory-augmented agents address this by retrieving past interactions or distilled insightsZhaoet al.(2024 (https://arxiv.org/html/2607.28272#bib.bib5)); Fanget al.(2026 (https://arxiv.org/html/2607.28272#bib.bib15)); Chhikaraet al.(2025 (https://arxiv.org/html/2607.28272#bib.bib14)); Liuet al.(2026 (https://arxiv.org/html/2607.28272#bib.bib19)). Most rely on frozen policies, while alternative approaches internalize experience directly into model weightsZhanget al.(2025b (https://arxiv.org/html/2607.28272#bib.bib39)), sacrificing the traceability of explicit memory. Recent works integrate external memory with RLZhanget al.(2026 (https://arxiv.org/html/2607.28272#bib.bib18)); Wuet al.(2025 (https://arxiv.org/html/2607.28272#bib.bib1)), yet they inject retrieved experiencesverbatim. Because interactive tasks exhibit high state variance, replay often introduces misaligned or misleading guidance. MemHarness bridges these two paradigms. It retains explicit memory banks for traceability but parameterizes their state-conditioned reconstruction. Rather than applying retrieved memory blindly, MemHarness is optimized end-to-end to critique and rewrite past experiences into context-specific guidance before acting.

3Method

Refer to caption

Figure 2:Overview of the MemHarness framework. The execution pipeline consists of three stages: (1)Memory Retrieval, where the policy generates a query to fetch relevant past experiences; (2)Contextual Memory Reconstruction, where the policy compares the memory’s source state with the current state to reconstruct adapted guidance (or falls back to self-reasoning if the memory is deemed unhelpful); and (3)Action Generationguided by the reconstructed memory.

3.1Problem Formulation

Sequential Decision-Making.

We model interactive decision-making as a POMDPM=(S,A,O,P,R)\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{O},P,R), whereS\mathcal{S},A\mathcal{A}, andO\mathcal{O}are the state, action, and observation spaces, respectively, andPPandRRdenote the transition and reward functions. Given a task descriptionT\mathcal{T}, at stepttthe agent observesoto_{t}and selectsata_{t}using a policyπθ\pi_{\theta}. To account for partial observability, the policy conditions on a recent history windowht=(ot−w+1,at−w+1,…,ot−1,at−1,ot)h_{t}=(o_{t-w+1},a_{t-w+1},\ldots,o_{t-1},a_{t-1},o_{t}). The objective is to maximize the expected episodic returnEπθ[∑t=1TR(st,at)]\mathbb{E}_{\pi_{\theta}}[\sum_{t=1}^{T}R(s_{t},a_{t})].

Experiential Memory.

The agent maintains a memory bankB={mi}i=1N\mathcal{B}={m_{i}}_{i=1}^{N}by periodically distilling collected trajectories into natural-language experiences. Each entrymi=(ei,oisrc)m_{i}=(e_{i},o_{i}^{src})contains an abstract strategyeie_{i}and its source observationoisrco_{i}^{src}. FollowingWuet al.(2025 (https://arxiv.org/html/2607.28272#bib.bib1)), we perform semantic deduplication, track empirical utility and periodically prune low-utility entries. Given a context-derived queryqtq_{t}, a retrieverR\mathcal{R}returns the top-kkentries: Et=R(qt,B),|Et|=k.\mathcal{E}_{t}=\mathcal{R}(q_{t},\mathcal{B}),\qquad|\mathcal{E}_{t}|=k.(1)The source observations enable comparison between historical and current states during reconstruction.

From Replay to Reconstruction.

Verbatim replay directly conditions action generation on retrieved experiences:at∼πθ(⋅∣T,ht,Et)a_{t}\sim\pi_{\theta}(\cdot\mid\mathcal{T},h_{t},\mathcal{E}_{t}). MemHarness instead first reconstructs them into state-specific guidance and then acts: gt\displaystyle g_{t}∼πθ(⋅∣T,ht,Et),\displaystyle\sim\pi_{\theta}(\cdot\mid\mathcal{T},h_{t},\mathcal{E}_{t}),(2) g~t\displaystyle\tilde{g}_{t}=f(gt),\displaystyle=f(g_{t}),at\displaystyle a_{t}∼πθ(⋅∣T,ht,g~t),\displaystyle\sim\pi_{\theta}(\cdot\mid\mathcal{T},h_{t},\tilde{g}_{t}),whereffmaps the raw reconstructiongtg_{t}to final guidanceg~t\tilde{g}_{t}, including rejection of mismatched memories. Both stages share the unified policyπθ\pi_{\theta}.

3.2Overall Framework

We introduceMemHarness, an RL-driven framework that reconstructs retrieved experiences for the current state rather than replaying them verbatim. As shown in Figure2 (https://arxiv.org/html/2607.28272#S3.F2), its inference pipeline comprises three stages: retrieval, contextual memory reconstruction, and action generation. These three stages instantiate the five-stage formulation in Section1 (https://arxiv.org/html/2607.28272#S1): the history windowhth_{t}serves as the environment observation, and memory critique is folded into the reconstruction stage.

1. 1.Retrieval.At steptt, the agent decides whether to query the experiential memory bankB\mathcal{B}based onhth_{t}. If triggered, retrieval returns relevant experiencesEt\mathcal{E}_{t}, which are reconstructed rather than directly added to the action context.
2. 2.Reconstruction.The policyπθ\pi_{\theta}critiquesEt\mathcal{E}_{t}againsthth_{t}and generates state-specific guidancegtg_{t}, retaining applicable knowledge while discarding or revising misaligned content.
3. 3.Action Generation.The same policyπθ\pi_{\theta}generates an executable actionata_{t}conditioned onhth_{t}andgtg_{t}. MemHarness thus separates reconstruction from action generation while impl