Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents

# Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents
Source: [https://arxiv.org/html/2605.21768](https://arxiv.org/html/2605.21768)
Sikuan Yan\*1,2,3,Ahmed Bahloul\*4,Ercong Nie1,Susanna Schwarzmann3, Riccardo Trivisonno3,Volker Tresp1,2,Yunpu Ma†1,2 1Ludwig Maximilian University of Munich,2Munich Center for Machine Learning, 3Huawei Heisenberg Research Center \(Munich\),4Technical University of Munich [s\.yan@campus\.lmu\.de](https://arxiv.org/html/2605.21768v1/mailto:email@domain),[cognitive\.yunpu@gmail\.com](https://arxiv.org/html/2605.21768v1/mailto:email@domain)

###### Abstract

Memory\-augmented LLM agents enable interactions that extend beyond finite context windows by storing, updating, and reusing information across sessions\. However, training such agents with reinforcement learning in multi\-session environments is challenging because memory turns the agent’s past actions into part of its future environment\. Once different rollouts write, update, or delete different memories, they no longer share the same intermediate memory state, making trajectory\-level comparisons fundamentally unfair\. This violates a key assumption behind group\-relative methods such as GRPO, where rollouts are compared as if they were sampled from the same effective environment\. Consequently, trajectory\-level rewards provide noisy or biased credit signals for long\-horizon memory operations\. To address this challenge, we introduceMemory\-R2, a training framework for long\-horizon memory\-augmented LLM agents\. Its core algorithm,LoGo\-GRPO, combineslocal andglobal group\-relative optimization\. The global objective preserves end\-to\-end learning from long\-horizon trajectory\-level rewards, while local rerollouts compare different memory\-operation outcomes from the same intermediate memory state, yielding fairer group comparisons and more precise supervision for memory construction\. Beyond credit assignment, Memory\-R2 jointly optimizes memory formation and memory evolution with a shared\-parameter co\-learning design, where a fact extractor and a memory manager are instantiated from the same LLM backbone through role\-specific prompts\. To stabilize multi\-step RL over long memory horizons, we adopt a progressive curriculum that increases the training horizon from 8 to 16 to 32 sessions\. Together, these components provide an effective training paradigm for memory\-augmented LLM agents in long\-horizon multi\-session settings\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author\. The code is available for access via[this repository](https://github.com/ahmedehabb/Memory-R2)\.![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/memory-r2-main-figure.png)Figure 1:Overview of Memory\-R2\. \(a\) Memory\-R2 uses a shared\-backbone extractor–manager architecture for chunk\-wise memory construction\. \(b\) LoGo\-GRPO contrasts with standard GRPO by introducing local rerollouts from shared intermediate memory states for fairer credit assignment while preserving global trajectory\-level optimization\. \(c\) Memory\-R2 improves accuracy and inference latency across backbones\.## 1Introduction

Large language models \(LLMs\) have rapidly evolved from standalone text generators into agentic systems that can plan\[[23](https://arxiv.org/html/2605.21768#bib.bib5)\], use tools\[[12](https://arxiv.org/html/2605.21768#bib.bib6),[4](https://arxiv.org/html/2605.21768#bib.bib7)\], and interact over long horizons\[[18](https://arxiv.org/html/2605.21768#bib.bib8)\]\. A central requirement for such agents is the ability to accumulate, update, and reuse information across interactions\. However, despite strong in\-context reasoning ability, LLM agents remain fundamentally constrained by finite context windows and the lack of persistent state, making it difficult to retain salient user information, track long\-term goals, or maintain consistency over extended multi\-session interactions\[[7](https://arxiv.org/html/2605.21768#bib.bib18),[9](https://arxiv.org/html/2605.21768#bib.bib19)\]\.

To address this limitation, a growing body of work augments LLM agents with explicit memory systems\[[26](https://arxiv.org/html/2605.21768#bib.bib10),[21](https://arxiv.org/html/2605.21768#bib.bib9)\]\. Existing research broadly follows two directions\. The first focuses on memory infrastructure, including graph\-structured memory, structured memory schemas, and system\-inspired memory organization\[[13](https://arxiv.org/html/2605.21768#bib.bib11),[1](https://arxiv.org/html/2605.21768#bib.bib3),[6](https://arxiv.org/html/2605.21768#bib.bib12),[25](https://arxiv.org/html/2605.21768#bib.bib15),[8](https://arxiv.org/html/2605.21768#bib.bib13),[5](https://arxiv.org/html/2605.21768#bib.bib14)\]\. The second focuses on memory policy learning, where reinforcement learning \(RL\) is used to decide what to extract, how to update memory, and how to use retrieved memory\[[22](https://arxiv.org/html/2605.21768#bib.bib2),[17](https://arxiv.org/html/2605.21768#bib.bib1)\]\. While these efforts have substantially improved long\-horizon agent behavior, training memory agents in multi\-session environments remains fundamentally challenging\.

The core difficulty is that memory makes the environment non\-stationary\. In multi\-session agent training, memory turns the agent’s past actions into part of its future environment: what the agent writes, updates, or deletes in one session becomes the state inherited by subsequent sessions\. This creates a fundamental challenge for trajectory\-level RL, especially for group\-relative methods such as GRPO\[[2](https://arxiv.org/html/2605.21768#bib.bib16)\], which rely on comparing rollouts sampled from the same effective environment\. Once rollouts modify memory differently, they no longer share the same intermediate memory state, yet GRPO still normalizes their rewards within a single comparison group, leading to unfair comparisons and biased credit assignment\. The problem is further amplified by trajectory\-level rewards: when a downstream failure occurs, it is difficult to determine whether it comes from the current session’s memory operation, corrupted memory inherited from earlier sessions, or later updates that overwrite useful information\. This raises a simple but important question:

How can we design a training paradigm for memory\-augmented agents that provides more accurate and fair credit assignment across sessions?

In this work, we presentMemory\-R2, a training framework for long\-horizon memory\-augmented LLM agents, as illustrated in Figure[1](https://arxiv.org/html/2605.21768#S0.F1)\. At its core isLoGo\-GRPO, a credit\-assignment algorithm that combines*global*and*local*group\-relative optimization\. LoGo\-GRPO preserves a trajectory\-level global reward for end\-to\-end long\-horizon optimization, while additionally introducing session\-wise attribution signals and local rerollouts that compare trajectories starting from identical intermediate memory states\. This yields fairer group comparison and cleaner supervision for memory operations\.

Beyond fair credit assignment, Memory\-R2 is designed to optimize the whole memory lifecycle\. Recent analyses decompose agentic memory into memory formation, memory evolution, and memory retrieval\[[3](https://arxiv.org/html/2605.21768#bib.bib4)\], whereas prior RL\-based memory work has focused primarily on evolution and retrieval\[[22](https://arxiv.org/html/2605.21768#bib.bib2)\]\. Our framework targets memory formation and evolution through two cooperative roles: a*fact extractor*, which identifies salient information from the interaction context, and a*memory manager*, which decides whether to insert, update, or delete memory entries\. Inspired by shared\-policy multi\-agent RL\[[16](https://arxiv.org/html/2605.21768#bib.bib17)\], we instantiate both roles with a shared LLM backbone and role\-specific prompts, enabling parameter\-efficient co\-learning and tighter coordination between extraction and memory editing\.

We further formulate memory construction as a multi\-step decision process within each session\. Rather than treating a session as a single monolithic transition, we divide it into chunks and allow the fact extractor and memory manager to alternate over them, turning memory construction into a temporally extended process that can be refined as more evidence becomes available\. To stabilize long\-horizon optimization, we also introduce a curriculum over session horizon, progressively scaling training from 8 to 16 to 32 sessions so that the model first acquires reliable short\-horizon memory behavior before adapting to more challenging long\-context settings\. Our contributions are summarized as follows:

- •We proposeMemory\-R2, a training framework for long\-horizon memory\-augmented LLM agents, whose core algorithmLoGo\-GRPOimproves fairness and session\-level credit assignment through global\-local group\-relative optimization\.
- •We introduce ashared\-parameter extractor–manager architectureand formulate memory construction as amulti\-step decision processover chunked sessions, enabling joint optimization of memory formation and evolution\.
- •We develop acurriculum learning strategyover session horizon that stabilizes long\-horizon RL training, and show that the resulting system is highlydata\-efficient, achieving strong gains over prior memory\-agent baselines using only two training conversations while generalizing across benchmarks, model scales, and answer agents\.

## 2Related Work

### 2\.1Memory Agent Architectures

Explicit memory has become a standard way to extend LLM agents beyond finite context windows and support long\-horizon interaction\[[21](https://arxiv.org/html/2605.21768#bib.bib9),[26](https://arxiv.org/html/2605.21768#bib.bib10)\]\. Prior work mainly differs in how memory is represented and managed\. Representative examples include graph\- or structure\-based memory systems such as Zep\[[13](https://arxiv.org/html/2605.21768#bib.bib11)\], G\-Memory\[[25](https://arxiv.org/html/2605.21768#bib.bib15)\], A\-MEM\[[21](https://arxiv.org/html/2605.21768#bib.bib9)\], Mem0\[[1](https://arxiv.org/html/2605.21768#bib.bib3)\], and CAM\[[6](https://arxiv.org/html/2605.21768#bib.bib12)\], as well as system\-inspired designs such as MemOS\[[8](https://arxiv.org/html/2605.21768#bib.bib13)\]and MemoryOS\[[5](https://arxiv.org/html/2605.21768#bib.bib14)\]\. While these methods propose increasingly expressive memory substrates, they mostly rely on heuristic or prompt\-based policies for deciding what to store, update, or discard\. In contrast, our work retains a modular extractor–manager architecture but optimizes the memory lifecycle directly with reinforcement learning\.

### 2\.2Reinforcement Learning for Memory Agents

Reinforcement learning has recently become an effective paradigm for training LLM agents in interactive settings such as tool use, web navigation, and reasoning\[[12](https://arxiv.org/html/2605.21768#bib.bib6),[4](https://arxiv.org/html/2605.21768#bib.bib7),[18](https://arxiv.org/html/2605.21768#bib.bib8),[2](https://arxiv.org/html/2605.21768#bib.bib16)\]\. This is particularly suitable for memory agents, where the quality of extraction, memory editing, and retrieval decisions is only revealed through downstream task performance\. Existing RL\-based memory methods, such as Memory\-R1\[[22](https://arxiv.org/html/2605.21768#bib.bib2)\]and Mem\-α\\alpha\[[17](https://arxiv.org/html/2605.21768#bib.bib1)\], demonstrate the promise of this direction\. However, they rely mainly on outcome\-level rewards and do not explicitly address cross\-session credit assignment under diverging memory states\. They also focus primarily on memory evolution and retrieval, leaving joint optimization of formation, evolution, and retrieval underexplored\[[3](https://arxiv.org/html/2605.21768#bib.bib4)\]\. Our work addresses these gaps by introducing multi\-step extractor–manager training, shared\-parameter co\-learning, and a global\-local GRPO objective for fairer credit assignment in long\-horizon multi\-session settings\.

## 3Method

### 3\.1Problem Formulation: Multi\-step Memory Bank Construction

We study memory bank construction for long\-horizon multi\-session interactions\. Let𝒟=\{St\}t=1T\\mathcal\{D\}=\\\{S\_\{t\}\\\}\_\{t=1\}^\{T\}denote a dialogue trajectory ofTTsessions, where each sessionSt=\{xt,k\}k=1KS\_\{t\}=\\\{x\_\{t,k\}\\\}\_\{k=1\}^\{K\}is divided intoKKchunks\. The agent maintains an external memory bankℳ\\mathcal\{M\}that evolves across sessions\. We formulate memory construction as a chunk\-wise multi\-step process, illustrated in Figure[1](https://arxiv.org/html/2605.21768#S0.F1)\(a\): for each chunkxt,kx\_\{t,k\}, a fact extractor first proposes salient content

zt,k∼πext​\(z∣xt,k\),z\_\{t,k\}\\sim\\pi\_\{\\mathrm\{ext\}\}\\\!\\left\(z\\mid x\_\{t,k\}\\right\),\(1\)and a memory manager then chooses an operation conditioned on the extracted content and current memory state,

at,k∼πmgr​\(a∣zt,k,ℳt,k−1\),a\_\{t,k\}\\sim\\pi\_\{\\mathrm\{mgr\}\}\\\!\\left\(a\\mid z\_\{t,k\},\\mathcal\{M\}\_\{t,k\-1\}\\right\),\(2\)whereat,k∈𝒜a\_\{t,k\}\\in\\mathcal\{A\}denotes operations such asINSERT,UPDATE, andDELETE\. The memory bank is updated by a deterministic transition operator

ℳt,k=𝒯​\(ℳt,k−1,zt,k,at,k\)\.\\mathcal\{M\}\_\{t,k\}=\\mathcal\{T\}\\\!\\left\(\\mathcal\{M\}\_\{t,k\-1\},z\_\{t,k\},a\_\{t,k\}\\right\)\.\(3\)
This yields a chunk\-wise memory construction process over sessiontt:

ℳt,0→πext,πmgrxt,1ℳt,1→πext,πmgrxt,2⋯→πext,πmgrxt,Kℳt,K,\\mathcal\{M\}\_\{t,0\}\\xrightarrow\[\\pi\_\{\\mathrm\{ext\}\},\\,\\pi\_\{\\mathrm\{mgr\}\}\]\{x\_\{t,1\}\}\\mathcal\{M\}\_\{t,1\}\\xrightarrow\[\\pi\_\{\\mathrm\{ext\}\},\\,\\pi\_\{\\mathrm\{mgr\}\}\]\{x\_\{t,2\}\}\\cdots\\xrightarrow\[\\pi\_\{\\mathrm\{ext\}\},\\,\\pi\_\{\\mathrm\{mgr\}\}\]\{x\_\{t,K\}\}\\mathcal\{M\}\_\{t,K\},\(4\)
Across the full dialogue trajectory, letτ=\{zt,k,at,k\}t=1,k=1T,K\\tau=\\\{z\_\{t,k\},a\_\{t,k\}\\\}\_\{t=1,k=1\}^\{T,K\}denote a memory\-construction rollout\. Its probability factorizes as

pθ​\(τ∣𝒟\)=∏t=1T∏k=1Kπext​\(zt,k∣xt,k\)​πmgr​\(at,k∣zt,k,ℳt,k−1\)\.p\_\{\\theta\}\(\\tau\\mid\\mathcal\{D\}\)=\\prod\_\{t=1\}^\{T\}\\prod\_\{k=1\}^\{K\}\\pi\_\{\\mathrm\{ext\}\}\\\!\\left\(z\_\{t,k\}\\mid x\_\{t,k\}\\right\)\\pi\_\{\\mathrm\{mgr\}\}\\\!\\left\(a\_\{t,k\}\\mid z\_\{t,k\},\\mathcal\{M\}\_\{t,k\-1\}\\right\)\.\(5\)
In our framework, the extractor and the manager are implemented as two cooperative roles instantiated from a shared LLM backbone with role\-specific prompts:

πext\(⋅\)=πθ\(⋅∣pext,⋅\),πmgr\(⋅\)=πθ\(⋅∣pmgr,⋅\),\\pi\_\{\\mathrm\{ext\}\}\(\\cdot\)=\\pi\_\{\\theta\}\(\\cdot\\mid p\_\{\\mathrm\{ext\}\},\\cdot\),\\qquad\\pi\_\{\\mathrm\{mgr\}\}\(\\cdot\)=\\pi\_\{\\theta\}\(\\cdot\\mid p\_\{\\mathrm\{mgr\}\},\\cdot\),\(6\)whereθ\\thetadenotes the shared model parameters, andpextp\_\{\\mathrm\{ext\}\}andpmgrp\_\{\\mathrm\{mgr\}\}are role\-specific prompts for fact extraction and memory management, respectively\. The resulting memory\-construction rolloutτ\\tauis evaluated through downstream task performance, yielding a trajectory\-level rewardR​\(τ\)R\(\\tau\)\. We optimize the shared memory policy by maximizing the expected return𝔼τ∼πθ​\[R​\(τ\)\]\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\[R\(\\tau\)\]\.

### 3\.2Length\-Normalized Step\-level RL with Shared Extractor–Manager Policy

While Sec\.[3\.1](https://arxiv.org/html/2605.21768#S3.SS1)defines memory construction as a multi\-step process, optimizing it with a shared LLM policy introduces length\-induced bias\. We instantiate fact extraction and memory management as two roles of a shared policy with role\-specific prompts\[[16](https://arxiv.org/html/2605.21768#bib.bib17)\]\. Since the two roles produce outputs of different lengths, token\-level RL assigns more loss terms to longer generations, biasing the shared policy toward verbose outputs and roles with longer outputs\. To address this, we use a*length\-normalized step\-level*objective, treating each extractor or manager call as one generation step\. For a generation stepuuwith generated token indices𝒰u\\mathcal\{U\}\_\{u\}, we aggregate token\-level ratios and advantages as

ρu=exp⁡\(1\|𝒰u\|​∑ℓ∈𝒰ulog⁡πθ​\(yℓ∣hℓ\)πθold​\(yℓ∣hℓ\)\),A¯u=1\|𝒰u\|​∑ℓ∈𝒰uAℓ,\\rho\_\{u\}=\\exp\\\!\\left\(\\frac\{1\}\{\|\\mathcal\{U\}\_\{u\}\|\}\\sum\_\{\\ell\\in\\mathcal\{U\}\_\{u\}\}\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{\\ell\}\\mid h\_\{\\ell\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{\\ell\}\\mid h\_\{\\ell\}\)\}\\right\),\\qquad\\bar\{A\}\_\{u\}=\\frac\{1\}\{\|\\mathcal\{U\}\_\{u\}\|\}\\sum\_\{\\ell\\in\\mathcal\{U\}\_\{u\}\}A\_\{\\ell\},\(7\)whereρu\\rho\_\{u\}is the step\-level importance ratio,A¯u\\bar\{A\}\_\{u\}is the step\-level advantage,yℓy\_\{\\ell\}is a generated token,hℓh\_\{\\ell\}is its autoregressive context,AℓA\_\{\\ell\}is the token\-level advantage, andπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}is the rollout policy\. This gives each generation step comparable weight regardless of output length\. The resultingρu\\rho\_\{u\}andA¯u\\bar\{A\}\_\{u\}are then used in the LoGo\-GRPO objective\.

### 3\.3LoGo\-GRPO for Multi\-session Credit Assignment

The formulation in Sec\.[3\.1](https://arxiv.org/html/2605.21768#S3.SS1)defines memory construction as a chunk\-wise multi\-step process, but learning still requires fair credit assignment across sessions\. Trajectory\-level GRPO is problematic in memory\-augmented settings because memory turns an agent’s past actions into part of its future environment\. Once rollouts write, update, or delete different memories, they no longer share the same intermediate memory state, making group\-relative comparisons unfair and credit signals noisy or biased\. To address this, we proposeLoGo\-GRPO, which combines a global trajectory\-level branch with a local rerollout branch\. As shown in Figure[1](https://arxiv.org/html/2605.21768#S0.F1)\(b\), the global branch preserves end\-to\-end optimization over the full multi\-session trajectory, while the local branch rerolls a stochastically sampled subset of sessions from shared memory states, yielding lower\-bias session\-level credit assignment at manageable cost\.

#### Reward function\.

Let𝒬\\mathcal\{Q\}denote the full set of question\-answer pairs\(q,a∗\)\(q,a^\{\*\}\)associated with a training conversation, and let𝒬t⊆𝒬\\mathcal\{Q\}\_\{t\}\\subseteq\\mathcal\{Q\}denote the subset whose required evidence is attributed to sessiontt\. Given a memory bankℳ\\mathcal\{M\}and a questionqq, an answer module retrieves relevant entries from memory and generates an answera^\\hat\{a\}\. We measure QA quality using token\-level F1:

QA​\(ℳ,𝒬t\)=1\|𝒬t\|​∑\(q,a∗\)∈𝒬tF1​\(a^,a∗\)\.\\mathrm\{QA\}\(\\mathcal\{M\},\\mathcal\{Q\}\_\{t\}\)=\\frac\{1\}\{\|\\mathcal\{Q\}\_\{t\}\|\}\\sum\_\{\(q,a^\{\*\}\)\\in\\mathcal\{Q\}\_\{t\}\}\\mathrm\{F1\}\\\!\\left\(\\hat\{a\},a^\{\*\}\\right\)\.\(8\)To discourage unbounded memory growth, we penalize memory tokens exceeding anα\\alphafraction of the cumulative session tokens up to sessiontt, whereTok​\(⋅\)\\mathrm\{Tok\}\(\\cdot\)denotes token count andα\\alphais a fixed memory budget ratio:

Comp​\(ℳ,t\)=\{0,Tok​\(ℳ\)≤α​∑s=1tTok​\(Ss\),Tok​\(ℳ\)−α​∑s=1tTok​\(Ss\)∑s=1tTok​\(Ss\),Tok​\(ℳ\)\>α​∑s=1tTok​\(Ss\)\.\\mathrm\{Comp\}\(\\mathcal\{M\},t\)=\\begin\{cases\}0,&\\mathrm\{Tok\}\(\\mathcal\{M\}\)\\leq\\alpha\\sum\_\{s=1\}^\{t\}\\mathrm\{Tok\}\(S\_\{s\}\),\\\\\[6\.0pt\] \\dfrac\{\\mathrm\{Tok\}\(\\mathcal\{M\}\)\-\\alpha\\sum\_\{s=1\}^\{t\}\\mathrm\{Tok\}\(S\_\{s\}\)\}\{\\sum\_\{s=1\}^\{t\}\\mathrm\{Tok\}\(S\_\{s\}\)\},&\\mathrm\{Tok\}\(\\mathcal\{M\}\)\>\\alpha\\sum\_\{s=1\}^\{t\}\\mathrm\{Tok\}\(S\_\{s\}\)\.\\end\{cases\}\(9\)The session\-level reward is

R​\(ℳ,𝒬t,t\)=QA​\(ℳ,𝒬t\)−λcomp​Comp​\(ℳ,t\),R\(\\mathcal\{M\},\\mathcal\{Q\}\_\{t\},t\)=\\mathrm\{QA\}\(\\mathcal\{M\},\\mathcal\{Q\}\_\{t\}\)\-\\lambda\_\{\\mathrm\{comp\}\}\\,\\mathrm\{Comp\}\(\\mathcal\{M\},t\),\(10\)whereλcomp\\lambda\_\{\\mathrm\{comp\}\}controls the compression penalty\.

#### Global branch\.

For rolloutii, letℳt\(i\)≡ℳt,K\(i\)\\mathcal\{M\}\_\{t\}^\{\(i\)\}\\equiv\\mathcal\{M\}\_\{t,K\}^\{\(i\)\}denote the memory state after sessiontt\. The global branch evaluates the terminal memoryℳT\(i\)\\mathcal\{M\}\_\{T\}^\{\(i\)\}and attributes the reward to sessionttaccording to the location of the required evidence:

rt,iG=R​\(ℳT\(i\),𝒬t,T\)\.r\_\{t,i\}^\{\\mathrm\{G\}\}=R\\\!\\left\(\\mathcal\{M\}\_\{T\}^\{\(i\)\},\\,\\mathcal\{Q\}\_\{t\},\\,T\\right\)\.\(11\)Following GRPO, we compute group\-relative advantages across thennglobal rollouts:

A^t,iG=rt,iG−μtGσtG\+ε,μtG=1n​∑j=1nrt,jG,σtG=stdj​\(rt,jG\)\.\\hat\{A\}\_\{t,i\}^\{\\mathrm\{G\}\}=\\frac\{r\_\{t,i\}^\{\\mathrm\{G\}\}\-\\mu\_\{t\}^\{\\mathrm\{G\}\}\}\{\\sigma\_\{t\}^\{\\mathrm\{G\}\}\+\\varepsilon\},\\qquad\\mu\_\{t\}^\{\\mathrm\{G\}\}=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}r\_\{t,j\}^\{\\mathrm\{G\}\},\\qquad\\sigma\_\{t\}^\{\\mathrm\{G\}\}=\\mathrm\{std\}\_\{j\}\\\!\\left\(r\_\{t,j\}^\{\\mathrm\{G\}\}\\right\)\.\(12\)While this branch provides full\-horizon supervision, it still suffers from reward contamination: at sessiontt, different rollouts induce different intermediate memory states as their effective environments, yet GRPO normalizes their rewards within the same comparison group\.

#### Local branch with stochastic rerollout\.

To reduce this contamination, the local branch performs rerollouts from shared intermediate memory states\. After the global rollout phase, each session is independently selected with probabilityplocalp\_\{\\mathrm\{local\}\}:

bt∼Bernoulli​\(plocal\),ℬ=\{t∣bt=1\}\.b\_\{t\}\\sim\\mathrm\{Bernoulli\}\(p\_\{\\mathrm\{local\}\}\),\\qquad\\mathcal\{B\}=\\\{t\\mid b\_\{t\}=1\\\}\.\(13\)For each selected sessiont∈ℬt\\in\\mathcal\{B\}, we choose an anchor rollouti0∈\{1,…,n\}i\_\{0\}\\in\\\{1,\\dots,n\\\}, retrieve the cached memory state immediately before sessiontt, and samplemmlocal rerollouts of sessionttonly\. Since these rerollouts share the same starting memory stateℳt−1\(i0\)\\mathcal\{M\}\_\{t\-1\}^\{\(i\_\{0\}\)\}, their comparison is not confounded by divergence from earlier sessions\. Letℳt\(i0,j\)\\mathcal\{M\}\_\{t\}^\{\(i\_\{0\},j\)\}denote the memory state after thejj\-th local rerollout from this anchor state\. The local reward is

rt,jL=R​\(ℳt\(i0,j\),𝒬t,t\),j=1,…,m\.r\_\{t,j\}^\{\\mathrm\{L\}\}=R\\\!\\left\(\\mathcal\{M\}\_\{t\}^\{\(i\_\{0\},j\)\},\\,\\mathcal\{Q\}\_\{t\},\\,t\\right\),\\qquad j=1,\\dots,m\.\(14\)The corresponding local advantages are computed within the rerollout group:

A^t,jL=rt,jL−μtLσtL\+ε,μtL=1m​∑j=1mrt,jL,σtL=stdj​\(rt,jL\)\.\\hat\{A\}\_\{t,j\}^\{\\mathrm\{L\}\}=\\frac\{r\_\{t,j\}^\{\\mathrm\{L\}\}\-\\mu\_\{t\}^\{\\mathrm\{L\}\}\}\{\\sigma\_\{t\}^\{\\mathrm\{L\}\}\+\\varepsilon\},\\qquad\\mu\_\{t\}^\{\\mathrm\{L\}\}=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}r\_\{t,j\}^\{\\mathrm\{L\}\},\\qquad\\sigma\_\{t\}^\{\\mathrm\{L\}\}=\\mathrm\{std\}\_\{j\}\\\!\\left\(r\_\{t,j\}^\{\\mathrm\{L\}\}\\right\)\.\(15\)Because local advantages are computed among rerollouts from the same anchor memory stateℳt−1\(i0\)\\mathcal\{M\}\_\{t\-1\}^\{\(i\_\{0\}\)\}, the comparison is fairer than global normalization across already\-diverged trajectories\.

#### Unified training objective\.

We optimize the shared memory policy using both global rollouts and local rerollouts\. For each generation stepuu, we assign the normalized advantage associated with its corresponding rollout:A^t,iG\\hat\{A\}\_\{t,i\}^\{\\mathrm\{G\}\}for a step from global rolloutiiat sessiontt, andA^t,jL\\hat\{A\}\_\{t,j\}^\{\\mathrm\{L\}\}for a step from local rerolloutjjat sessiontt\. The same assigned advantage is used as the token\-level advantageAℓA\_\{\\ell\}for all tokens in stepuu\. Let𝒦step\\mathcal\{K\}\_\{\\mathrm\{step\}\}denote the set of valid generation steps from both branches\. Using the step\-level ratioρu\\rho\_\{u\}and advantageA¯u\\bar\{A\}\_\{u\}from Eq\.[7](https://arxiv.org/html/2605.21768#S3.E7), we optimize the dual\-clipped surrogate

ℓu=\{min⁡\(−c​A¯u,max⁡\(−ρu​A¯u,−clip​\(ρu,1−ϵ,1\+ϵ\)​A¯u\)\),A¯u<0,max⁡\(−ρu​A¯u,−clip​\(ρu,1−ϵ,1\+ϵ\)​A¯u\),A¯u≥0,\\ell\_\{u\}=\\begin\{cases\}\\min\\\!\\left\(\-c\\,\\bar\{A\}\_\{u\},\\;\\max\\\!\\left\(\-\\rho\_\{u\}\\bar\{A\}\_\{u\},\\;\-\\mathrm\{clip\}\(\\rho\_\{u\},1\-\\epsilon,1\+\\epsilon\)\\bar\{A\}\_\{u\}\\right\)\\right\),&\\bar\{A\}\_\{u\}<0,\\\\\[5\.69054pt\] \\max\\\!\\left\(\-\\rho\_\{u\}\\bar\{A\}\_\{u\},\\;\-\\mathrm\{clip\}\(\\rho\_\{u\},1\-\\epsilon,1\+\\epsilon\)\\bar\{A\}\_\{u\}\\right\),&\\bar\{A\}\_\{u\}\\geq 0,\\end\{cases\}\(16\)wherec\>1c\>1is the dual\-clipping constant andϵ\\epsilonis the clipping threshold\. The final actor objective is

ℒ​\(θ\)=1\|𝒦step\|​∑u∈𝒦stepℓu−βent​H¯token\+βkl​D¯KL,token,\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{K\}\_\{\\mathrm\{step\}\}\|\}\\sum\_\{u\\in\\mathcal\{K\}\_\{\\mathrm\{step\}\}\}\\ell\_\{u\}\-\\beta\_\{\\mathrm\{ent\}\}\\,\\overline\{H\}\_\{\\mathrm\{token\}\}\+\\beta\_\{\\mathrm\{kl\}\}\\,\\overline\{D\}\_\{\\mathrm\{KL,token\}\},\(17\)whereH¯token\\overline\{H\}\_\{\\mathrm\{token\}\}andD¯KL,token\\overline\{D\}\_\{\\mathrm\{KL,token\}\}denote the mean token\-level entropy and KL divergence, respectively\. The proportion of local rerollouts controls the strength of local supervision, allowing LoGo\-GRPO to balance end\-to\-end long\-horizon learning with lower\-bias session\-level credit assignment\.

### 3\.4Curriculum Learning for Long\-Horizon Credit Assignment

Directly training on long multi\-session trajectories is unstable before the model acquires reliable memory manipulation skills\. Because memory operations shape the future environment, early insert, update, or delete errors can propagate across sessions and make long\-horizon credit assignment increasingly noisy\. We therefore adopt a curriculum over session horizon: training starts from shorter sessions, where memory effects are easier to observe and attribute, and gradually increases the horizon as the policy stabilizes\. Concretely, we train in three stages with the maximum number of sessions increasing from 8 to 16 to 32\. The 8\-session stage learns basic memory operations under limited error propagation, the 16\-session stage introduces stronger inter\-session dependencies, and the 32\-session stage enables full long\-horizon optimization\. For each stage, we select the best validation checkpoint as the initialization for the next stage, providing a stable starting point for longer\-horizon training\.

## 4Experiments

### 4\.1Experiment Setup

#### Datasets and Evaluation Metrics\.

We train on LoCoMo\[[10](https://arxiv.org/html/2605.21768#bib.bib20)\], a long\-term persona\-grounded conversation benchmark, using a 2:1:7 train/validation/test split\. For out\-of\-distribution evaluation, we additionally test on LongMemEval\[[19](https://arxiv.org/html/2605.21768#bib.bib21)\], MSC\-Self\-Instruct\[[11](https://arxiv.org/html/2605.21768#bib.bib22),[20](https://arxiv.org/html/2605.21768#bib.bib23)\], and MemBench\[[15](https://arxiv.org/html/2605.21768#bib.bib24)\]\. We report token\-level F1, BLEU\-1 \(B1\), and LLM\-as\-a\-Judge \(J\) as the primary metrics, and additionally useM\-Fail, the percentage of required evidence location IDs that are missing from the memory bank, as a diagnostic measure of memory\-construction quality\. Further details on the M\-Fail metric can be found in Appendix[C](https://arxiv.org/html/2605.21768#A3)\.

#### Baselines and Implementation Details\.

We compare against A\-MEM\[[21](https://arxiv.org/html/2605.21768#bib.bib9)\], Mem0\[[1](https://arxiv.org/html/2605.21768#bib.bib3)\], MemoryOS\[[5](https://arxiv.org/html/2605.21768#bib.bib14)\], a RAG variant implemented within the Mem0 framework, MEM1\[[27](https://arxiv.org/html/2605.21768#bib.bib26)\], MemAgent\[[24](https://arxiv.org/html/2605.21768#bib.bib27)\], and Memory\-R1\[[22](https://arxiv.org/html/2605.21768#bib.bib2)\]\. Our work primarily targets the memory construction stage: the memory extractor and memory manager share a Qwen2\.5\-7B\-Instruct backbone and are jointly trained, while the answer agent is held fixed during training to provide stable reward signals\. We use GPT\-OSS\-120B as this fixed answer agent, since a weaker answer model would yield noisy reward signals that conflate memory\-construction quality with answer\-generation errors\. To remain consistent with this training pipeline, all reported results in our ablation and analysis experiments use the same GPT\-OSS\-120B answer agent\. In Table[1](https://arxiv.org/html/2605.21768#S4.T1), however, we additionally train a Qwen2\.5\-7B\-Instruct answer agent and report a backbone\-controlled variant of Memory\-R2 in which all components share the same 7B backbone, enabling a fair comparison against the baselines\. Unless otherwise noted, all baselines also use Qwen2\.5\-7B\-Instruct as the backbone\. Additional details are provided in Appendix[A](https://arxiv.org/html/2605.21768#A1)\.

Table 1:Main results on LoCoMo\. We report token\-level F1 \(F1\), BLEU\-1 \(B1\), and LLM\-as\-a\-Judge \(J\), with the best per column inbold\. For fair comparison, all baselines and Memory\-R2 use Qwen2\.5\-7B\-Instruct as the base model\. We additionally report Memory\-R2 \(GPT\-OSS\), which swaps the answer agent for GPT\-OSS\-120B; notably, our RL\-finetuned 7B Memory\-R2 surpasses this 120B variant on F1 and B1, showing that targeted training outweighs raw model scale\. Results are averaged over three runs; standard deviations are in Table[3](https://arxiv.org/html/2605.21768#A5.T3)\.†\\dagger: as reported in\[[22](https://arxiv.org/html/2605.21768#bib.bib2)\]\.![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/generalization_combined.png)Figure 2:Generalization of Memory\-R2 across \(a\) OOD benchmarks, \(b\) backbone sizes, and \(c\) answer agents\.

### 4\.2Main Results

#### Fair Comparison\.

Table[1](https://arxiv.org/html/2605.21768#S4.T1)reports the main results on LoCoMo\. Under the backbone\-controlled setting, Memory\-R2 achieves the best overall F1 and BLEU\-1 among all training\-free and trained baselines, including MEM1, MemAgent, and Memory\-R1\. Compared with the closely related RL baseline Memory\-R1, Memory\-R2 improves overall F1 from 43\.14 to 50\.60 and B1 from 36\.44 to 44\.01, while also reaching a strong judge score of 80\.99\. These gains are obtained with a simple memory\-agent pipeline, suggesting that the improvement mainly comes from the proposed training algorithm rather than additional system complexity\. We additionally report Memory\-R2 \(GPT\-OSS\), which uses the same memory construction module but replaces the answer agent with GPT\-OSS\-120B\. Memory\-R2 with the 7B answer agent achieves higher F1 and BLEU\-1 than the GPT\-OSS\-120B variant, demonstrating that a task\-aligned small model can rival a much larger frozen one when paired with a well\-trained memory module\.

#### Strong Generalization\.

Figure[2](https://arxiv.org/html/2605.21768#S4.F2)further demonstrates the strong generalization ability of Memory\-R2 from three complementary perspectives\. Notably, these gains are achieved even though the model is trained on only two LoCoMo conversations, suggesting that the proposed training paradigm is highly data\-efficient\. First, Figure[2](https://arxiv.org/html/2605.21768#S4.F2)\(a\) shows strong transfer to out\-of\-distribution benchmarks\. When evaluated zero\-shot on LongMemEval\-oracle, LongMemEval\-s, MSC\-Self\-Instruct, and MemBench, Memory\-R2 consistently improves over the base model across all reported metrics\. For example, on LongMemEval\-oracle, the F1 score improves from 27\.88 to 50\.60, and similar gains are observed on the other benchmarks, indicating that the learned memory\-construction policy does not simply overfit to the training benchmark\. Second, Figure[2](https://arxiv.org/html/2605.21768#S4.F2)\(b\) shows that the gains also transfer across model scales\. The improvement is especially pronounced for Qwen2\.5\-3B, where F1 increases from 10\.3 to 46\.8, suggesting that our training paradigm is particularly beneficial for smaller\-capacity models, for which effective long\-horizon memory construction is otherwise difficult to learn\. Third, Figure[2](https://arxiv.org/html/2605.21768#S4.F2)\(c\) decomposes the contribution of training the memory module versus the answer agent\. The dominant gain comes from training the memory module \(e\.g\., F1 from 26\.4 to 45\.2 with a 7B\-Base answer agent; F1 from 30\.6 to 49\.7 with a GPT\-OSS answer agent\), while varying the answer agent at fixed RL\-trained memory yields comparably high scores\. This indicates that the benefits of Memory\-R2 transfer across diverse downstream answer agents\. Taken together, these results indicate that Memory\-R2 learns a robust and transferable memory\-construction policy rather than overfitting to a specific benchmark, model scale, or answer agent\.

![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/logo_grpo_curriculum.png)Figure 3:LoGo\-GRPO and curriculum learning are both essential\.\(a,b\)LoGo\-GRPO consistently outperforms GRPO across curriculum stages\.\(c,d\)Curriculum training remains stable under equal compute, whereas direct 32\-session training collapses validation F1 from0\.470\.47to0\.270\.27and increases M\-Fail to72\.1%72\.1\\%\.Table 2:Ablation studies on components of LoGo\-GRPO\.

### 4\.3Ablation Studies

Table[2](https://arxiv.org/html/2605.21768#S4.T2)summarizes ablations on the major components of our method, withM\-Failreported as a diagnostic measure of memory quality\.

Replacing LoGo\-GRPO with standard GRPO degrades F1 from 49\.67 to 46\.62 and B1 from 43\.77 to 40\.97, confirming the benefit of global\-local credit assignment\. Figure[3](https://arxiv.org/html/2605.21768#S4.F3)\(a, b\) shows this gap holds at every stage across question types, indicating that local rerollouts consistently mitigate credit\-assignment bias\. Removing curriculum learning \(−\-curriculum\) causes a much larger drop—F1 falls to 24\.12 and M\-Fail rises to 46\.5%\. Figure[3](https://arxiv.org/html/2605.21768#S4.F3)\(c, d\) traces this collapse: direct 32\-session training peaks atF1=0\.47F\_\{1\}\{=\}0\.47before falling to0\.270\.27, while M\-Fail explodes from below 10% to over 70%; the curriculum instead stabilizes aroundF1=0\.50F\_\{1\}\{=\}0\.50with M\-Fail held under 7%\. This confirms that early errors propagate across sessions and corrupt memory, and that progressive horizon expansion is essential for stable long\-horizon training\. We further ablate the length normalization in our step\-level objective: switching to a token\-level loss \(−\-length norm\.\) drops F1 to 43\.53 and B1 to 38\.10, confirming that length\-normalized step weighting is necessary to prevent output length bias under the shared extractor–manager policy\.

For the memory\-construction architecture, a single\-agent variant merging extraction and editing into one role drops to 39\.14 F1, and a separate\-params variant where the extractor and manager use disjoint parameters also underperforms \(44\.31 F1\), supporting both explicit role decomposition and parameter sharing\. Alternative interaction depths likewise underperform the full multi\-step design \(40\.39 / 41\.37 / 37\.61 F1 forN=4/8/10N\{=\}4/8/10\), showing that moderate iterative refinement is optimal\. Too few chunks limit refinement, while overly long interaction chains hurt optimization\. Finally, training only the memory manager \(45\.34 F1\) or only the fact extractor \(28\.30 F1\) also degrades performance, especially the latter, confirming that both components benefit from joint RL training and that fact extraction is the more brittle of the two roles when left untrained\.

Overall, the gains of our method arise from the combination of fair credit assignment, curriculum learning, multi\-step memory construction, and shared extractor–manager co\-learning\. Additional ablations are reported in Appendix[E](https://arxiv.org/html/2605.21768#A5)\.

### 4\.4More Analysis: Latency and Compression

#### Latency\.

Figure[4](https://arxiv.org/html/2605.21768#S4.F4)\(a,b\) compares F1 and inference latency before and after Memory\-R2 training\. Memory\-R2 improves F1 while reducing latency for both Qwen2\.5\-3B and Qwen2\.5\-7B under the per\-conversation measurement, moving both models toward a better quality–efficiency regime\. The source of this latency reduction differs across scales\. For Qwen2\.5\-3B, the gain is mainly driven by more concise generations: the trained policy emits fewer tokens per memory\-construction turn\. For Qwen2\.5\-7B, the gain comes not only from shorter generations, but also from a more stable generation\-length distribution: the untrained policy occasionally produces overly long memory\-management outputs, whereas Memory\-R2 suppresses these unnecessary generations, reducing decoding work and making memory construction more stable\. We provide a diagnostic breakdown of these scale\-dependent mechanisms in Figure[12](https://arxiv.org/html/2605.21768#A5.F12)\. These results suggest that better\-trained memory policies can improve answer quality without incurring additional inference overhead, and can even reduce latency by making memory construction more concise and stable\.

#### Compression\.

Figure[4](https://arxiv.org/html/2605.21768#S4.F4)\(c,d\) studies the effect of the compression penaltyλcomp\\lambda\_\{\\mathrm\{comp\}\}\. Across both Qwen2\.5\-3B and Qwen2\.5\-7B,λcomp=0\.3\\lambda\_\{\\mathrm\{comp\}\}=0\.3achieves the best F1 and BLEU\-1, as highlighted by the yellow band\. Smaller penalties may retain redundant or noisy memories, while overly strong compression can remove useful evidence\. We therefore useλcomp=0\.3\\lambda\_\{\\mathrm\{comp\}\}=0\.3as the default setting\.

![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/efficiency_combined.png)Figure 4:Inference efficiency and compression penalty analysis\.\(a,b\)Accuracy–latency trade\-off measured by F1 vs\. time per conversation and per generated token\.\(c,d\)Effect ofλcomp∈\{0,0\.1,0\.3,0\.5\}\\lambda\_\{\\mathrm\{comp\}\}\\in\\\{0,0\.1,0\.3,0\.5\\\}on F1 and BLEU\-1; the yellow band marksλcomp=0\.3\\lambda\_\{\\mathrm\{comp\}\}=0\.3, and rings mark the best value\.

## 5Conclusion

In this paper, we present Memory\-R2, a training framework for long\-horizon memory\-augmented LLM agents that addresses a fundamental challenge in multi\-session reinforcement learning: fair credit assignment under diverging memory states\. Our method, LoGo\-GRPO, combines global trajectory\-level optimization with local rerollouts from shared intermediate memory states, enabling fairer session\-level comparisons while preserving end\-to\-end long\-horizon learning\. Beyond credit assignment, Memory\-R2 jointly optimizes memory formation and memory evolution through a shared extractor–manager policy, formulates memory construction as a multi\-step decision process over chunked sessions, and stabilizes training with a curriculum over session horizon\. Experiments show that Memory\-R2 consistently outperforms prior memory\-agent baselines on LoCoMo and generalizes well across out\-of\-distribution benchmarks, model scales, and answer agents\. These results suggest that improving credit assignment is a key ingredient for training robust long\-horizon memory agents, and we hope this work provides a useful foundation for future research on memory\-centric RL for LLM agents\.

## References

- \[1\]\(2025\)Mem0: building production\-ready ai agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§C\.1](https://arxiv.org/html/2605.21768#A3.SS1.p1.1),[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2605.21768#S4.T1.5.3.3.1)\.
- \[2\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang\(2025\-09\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§A\.2](https://arxiv.org/html/2605.21768#A1.SS2.p3.10),[§1](https://arxiv.org/html/2605.21768#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.21768#S2.SS2.p1.1)\.
- \[3\]Y\. Hu, S\. Liu, Y\. Yue, G\. Zhang, B\. Liu, F\. Zhu, J\. Lin, H\. Guo, S\. Dou, Z\. Xi,et al\.\(2025\)Memory in the age of ai agents\.arXiv preprint arXiv:2512\.13564\.Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p6.1),[§2\.2](https://arxiv.org/html/2605.21768#S2.SS2.p1.1)\.
- \[4\]B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han\(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.External Links:2503\.09516,[Link](https://arxiv.org/abs/2503.09516)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.21768#S2.SS2.p1.1)\.
- \[5\]J\. Kang, M\. Ji, Z\. Zhao, and T\. Bai\(2025\)Memory os of ai agent\.External Links:2506\.06326,[Link](https://arxiv.org/abs/2506.06326)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2605.21768#S4.T1.6.4.4.1)\.
- \[6\]R\. Li, Z\. Zhang, X\. Bo, Z\. Tian, X\. Chen, Q\. Dai, Z\. Dong, and R\. Tang\(2025\)CAM: a constructivist view of agentic memory for llm\-based reading comprehension\.External Links:2510\.05520,[Link](https://arxiv.org/abs/2510.05520)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1)\.
- \[7\]T\. Li, G\. Zhang, Q\. D\. Do, X\. Yue, and W\. Chen\(2024\)Long\-context llms struggle with long in\-context learning\.External Links:2404\.02060,[Link](https://arxiv.org/abs/2404.02060)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p1.1)\.
- \[8\]Z\. Li, S\. Song, H\. Wang, S\. Niu, D\. Chen, J\. Yang, C\. Xi, H\. Lai, J\. Zhao, Y\. Wang, J\. Ren, Z\. Lin, J\. Huo, T\. Chen, K\. Chen, K\. Li, Z\. Yin, Q\. Yu, B\. Tang, H\. Yang, Z\. J\. Xu, and F\. Xiong\(2025\)MemOS: an operating system for memory\-augmented generation \(mag\) in large language models\.External Links:2505\.22101,[Link](https://arxiv.org/abs/2505.22101)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1)\.
- \[9\]J\. Liu, D\. Zhu, Z\. Bai, Y\. He, H\. Liao, H\. Que, Z\. Wang, C\. Zhang, G\. Zhang, J\. Zhang, Y\. Zhang, Z\. Chen, H\. Guo, S\. Li, Z\. Liu, Y\. Shan, Y\. Song, J\. Tian, W\. Wu, Z\. Zhou, R\. Zhu, J\. Feng, Y\. Gao, S\. He, Z\. Li, T\. Liu, F\. Meng, W\. Su, Y\. Tan, Z\. Wang, J\. Yang, W\. Ye, B\. Zheng, W\. Zhou, W\. Huang, S\. Li, and Z\. Zhang\(2025\)A comprehensive survey on long context language modeling\.External Links:2503\.17407,[Link](https://arxiv.org/abs/2503.17407)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p1.1)\.
- \[10\]A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang\(2024\)Evaluating very long\-term conversational memory of llm agents\.External Links:2402\.17753,[Link](https://arxiv.org/abs/2402.17753)Cited by:[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px1.p1.1)\.
- \[11\]C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez\(2024\)MemGPT: towards llms as operating systems\.External Links:2310\.08560,[Link](https://arxiv.org/abs/2310.08560)Cited by:[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px1.p1.1)\.
- \[12\]C\. Qian, E\. C\. Acikgoz, Q\. He, H\. Wang, X\. Chen, D\. Hakkani\-Tür, G\. Tur, and H\. Ji\(2025\)ToolRL: reward is all tool learning needs\.External Links:2504\.13958,[Link](https://arxiv.org/abs/2504.13958)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.21768#S2.SS2.p1.1)\.
- \[13\]P\. Rasmussen, P\. Paliychuk, T\. Beauvais, J\. Ryan, and D\. Chalef\(2025\)Zep: a temporal knowledge graph architecture for agent memory\.External Links:2501\.13956,[Link](https://arxiv.org/abs/2501.13956)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1)\.
- \[14\]G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu\(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[§A\.1](https://arxiv.org/html/2605.21768#A1.SS1.p1.5),[§A\.2](https://arxiv.org/html/2605.21768#A1.SS2.p1.1)\.
- \[15\]H\. Tan, Z\. Zhang, C\. Ma, X\. Chen, Q\. Dai, and Z\. Dong\(2025\)MemBench: towards more comprehensive evaluation on the memory of llm\-based agents\.External Links:2506\.21605,[Link](https://arxiv.org/abs/2506.21605)Cited by:[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px1.p1.1)\.
- \[16\]Z\. Wan, Y\. Li, X\. Wen, Y\. Song, H\. Wang, L\. Yang, M\. Schmidt, J\. Wang, W\. Zhang, S\. Hu, and Y\. Wen\(2025\)ReMA: learning to meta\-think for llms with multi\-agent reinforcement learning\.External Links:2503\.09501,[Link](https://arxiv.org/abs/2503.09501)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p6.1),[§3\.2](https://arxiv.org/html/2605.21768#S3.SS2.p1.2)\.
- \[17\]Y\. Wang, R\. Takanobu, Z\. Liang, Y\. Mao, Y\. Hu, J\. McAuley, and X\. Wu\(2025\)Mem\-\{\\\{\\\\backslashalpha\}\\\}: learning memory construction via reinforcement learning\.arXiv preprint arXiv:2509\.25911\.Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.21768#S2.SS2.p1.1)\.
- \[18\]Z\. Wei, W\. Yao, Y\. Liu, W\. Zhang, Q\. Lu, L\. Qiu, C\. Yu, P\. Xu, C\. Zhang, B\. Yin, H\. Yun, and L\. Li\(2025\)WebAgent\-r1: training web agents via end\-to\-end multi\-turn reinforcement learning\.External Links:2505\.16421,[Link](https://arxiv.org/abs/2505.16421)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.21768#S2.SS2.p1.1)\.
- \[19\]D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. Yu\(2025\)LongMemEval: benchmarking chat assistants on long\-term interactive memory\.External Links:2410\.10813,[Link](https://arxiv.org/abs/2410.10813)Cited by:[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px1.p1.1)\.
- \[20\]J\. Xu, A\. Szlam, and J\. Weston\(2022\-05\)Beyond goldfish memory: long\-term open\-domain conversation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 5180–5197\.External Links:[Link](https://aclanthology.org/2022.acl-long.356/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.356)Cited by:[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px1.p1.1)\.
- \[21\]W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang\(2025\)A\-mem: agentic memory for llm agents\.External Links:2502\.12110,[Link](https://arxiv.org/abs/2502.12110)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2605.21768#S4.T1.4.2.2.1)\.
- \[22\]S\. Yan, X\. Yang, Z\. Huang, E\. Nie, Z\. Ding, Z\. Li, X\. Ma, K\. Kersting, J\. Z\. Pan, H\. Schütze,et al\.\(2025\)Memory\-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning\.arXiv preprint arXiv:2508\.19828\.Cited by:[§C\.1](https://arxiv.org/html/2605.21768#A3.SS1.p1.1),[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§1](https://arxiv.org/html/2605.21768#S1.p6.1),[§2\.2](https://arxiv.org/html/2605.21768#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2605.21768#S4.T1),[Table 1](https://arxiv.org/html/2605.21768#S4.T1.7.5.5.1)\.
- \[23\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p1.1)\.
- \[24\]H\. Yu, T\. Chen, J\. Feng, J\. Chen, W\. Dai, Q\. Yu, Y\. Zhang, W\. Ma, J\. Liu, M\. Wang, and H\. Zhou\(2025\)MemAgent: reshaping long\-context llm with multi\-conv rl\-based memory agent\.External Links:2507\.02259,[Link](https://arxiv.org/abs/2507.02259)Cited by:[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2605.21768#S4.T1.7.5.11.6.1)\.
- \[25\]G\. Zhang, M\. Fu, G\. Wan, M\. Yu, K\. Wang, and S\. Yan\(2025\)G\-memory: tracing hierarchical memory for multi\-agent systems\.External Links:2506\.07398,[Link](https://arxiv.org/abs/2506.07398)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1)\.
- \[26\]W\. Zhong, L\. Guo, Q\. Gao, H\. Ye, and Y\. Wang\(2023\)MemoryBank: enhancing large language models with long\-term memory\.External Links:2305\.10250,[Link](https://arxiv.org/abs/2305.10250)Cited by:[§1](https://arxiv.org/html/2605.21768#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.21768#S2.SS1.p1.1)\.
- \[27\]Z\. Zhou, A\. Qu, Z\. Wu, S\. Kim, A\. Prakash, D\. Rus, J\. Zhao, B\. K\. H\. Low, and P\. P\. Liang\(2025\)MEM1: learning to synergize memory and reasoning for efficient long\-horizon agents\.External Links:2506\.15841,[Link](https://arxiv.org/abs/2506.15841)Cited by:[§4\.1](https://arxiv.org/html/2605.21768#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2605.21768#S4.T1.7.5.10.5.1)\.

## Appendix AAdditional Implementation Details

![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/pipeline_figure2.png)Figure 5:LoGo\-GRPO training pipeline for memory manager\. Memory bank construction via alternating extraction and management steps over chunked sessions\. Global rollouts optimize end\-to\-end performance using rewards from the final memory, while local rerollouts from shared memory states provide low\-bias credit assignment\. Both signals are unified in a single GRPO\-style objective, with curriculum learning enabling stable long\-horizon training\.### A\.1RL Training for Answer Agent

Training pipeline\.We fine\-tune Qwen2\.5\-7B\-Instruct as the answer agent using GRPO, implemented with the VERL framework\[[14](https://arxiv.org/html/2605.21768#bib.bib25)\]\. Each training example is a single\-turn QA prompt\. Given a natural\-language questionqqand a constructed memory bankℳ\\mathcal\{M\}, we retrieve, for each speaker, the top\-3030memory entries by text\-embedding similarity toqqusing a similarity threshold of0\.30\.3, and insert them into the QA template \(Appendix[B\.3](https://arxiv.org/html/2605.21768#A2.SS3)\)\. The template instructs the model to reason step by step over the timestamped memories and output the final answer inside an<answer\>\.\.\.</answer\>tag\.

Training data\.We construct the training set by running the full memory\-construction pipeline \(fact extraction, memory operations, and QA answering\) on only two LoCoMo training conversations with GPT\-4o,T=0T\{=\}0\. For each QA pair\(q,a⋆\)\(q,a^\{\\star\}\), we store \(i\) the rendered QA prompt and \(ii\) the answer extracted from the generated<answer\>tag\. Because some QA instances in the original data are noisy or weakly aligned with the available memory evidence, we retain only samples whose generated answer achieves token\-level F1≥0\.25\\geq 0\.25against the gold answera⋆a^\{\\star\}\. This filtering removes clearly problematic QA instances while preserving a sufficiently diverse training distribution\. The remaining samples are randomly split into90%90\\%training and10%10\\%validation\.

Reward\.We use a judge\-free rule\-based reward

R​\(y^,a⋆\)=F1​\(extract​\(y^\),a⋆\)∈\[0,1\],R\(\\hat\{y\},a^\{\\star\}\)=\\mathrm\{F1\}\\bigl\(\\mathrm\{extract\}\(\\hat\{y\}\),\\,a^\{\\star\}\\bigr\)\\in\[0,1\],whereextract​\(⋅\)\\mathrm\{extract\}\(\\cdot\)returns the substring inside the first<answer\>\.\.\.</answer\>tag\. The F1 score is computed using the standard SQuAD\-style token F1 after lowercasing, removing punctuation and articles \(a,an,the\), and whitespace tokenization\. This keeps the training reward aligned with the answer\-level metric used in evaluation\.

Optimization\.We train with GRPO usingn=8n\{=\}8rollouts per prompt, sampling temperature1\.01\.0, the vLLM backend, GPU memory utilization0\.80\.8, and tensor parallelismTP=1\\mathrm\{TP\}\{=\}1\. We set the maximum prompt length to12,28812\{,\}288tokens and the maximum response length to1,0241\{,\}024tokens, with left truncation applied on the prompt side\. The train batch size is6464, the PPO mini\-batch size is1616, and the micro\-batch size is11per GPU\. We use the standard GRPO advantage estimator and apply a token\-level KL penalty in the actor loss with coefficient0\.0010\.001, without adding KL to the reward\. For runs with at least two GPUs, both parameters and optimizer states are kept on device; optimizer offloading is enabled only in the single\-GPU setting\. We train for55epochs, and perform evaluation and checkpointing every55optimizer steps\.

### A\.2RL Training for Fact Extraction and Memory Management

We train the joint fact\-extractor and memory\-manager agent using a curriculum RL recipe with VERL framework\[[14](https://arxiv.org/html/2605.21768#bib.bib25)\]\. A singleQwen2\.5\-7B\-Instructbackbone is shared across both roles: the fact extractor produces atomic facts, while the memory manager predictsINSERT/UPDATE/DELETEoperations\. Parameter sharing is realized through alternating role\-conditioned rollouts within each session chunk\.

Training Data\.We use LoCoMo with a conversation\-level 2:1:7 train/validation/test split\. The memory\-construction policy is trained using only the two conversations in the training split, which contain 328 associated QA pairs in total\. These QA pairs are used to compute downstream rewards for memory construction, while the held\-out validation conversation is used for checkpoint selection and the remaining seven conversations are reserved for test evaluation\.

Optimization\.We optimize the policy with GRPO\[[2](https://arxiv.org/html/2605.21768#bib.bib16)\], usingNrollout=16N\_\{\\text\{rollout\}\}\{=\}16global trajectories per prompt and a local GRPO sampling fraction of0\.50\.5withNlocal=4N\_\{\\text\{local\}\}\{=\}4resampled turns to reduce gradient variance for memory operations\. The actor is updated withEppo=2E\_\{\\text\{ppo\}\}\{=\}2epochs per batch, PPO mini\-batch size1616, micro\-batch size11per GPU, learning rateη=2×10−6\\eta\{=\}2\\times 10^\{\-6\}, clipping ratioϵ=0\.2\\epsilon\{=\}0\.2, and entropy coefficient0\.0010\.001\. We use a KL penalty with coefficientβKL=10−3\\beta\_\{\\text\{KL\}\}\{=\}10^\{\-3\}, without adding KL into the reward\. We apply*turn\-level*importance\-ratio clipping \(clip\_mode=turn\) and*turn\-level*loss aggregation\.

Sequence and Rollout Budgets\.Each turn uses a prompt budget ofLprompt=28,672L\_\{\\text\{prompt\}\}\{=\}28\{,\}672tokens and a response budget ofLresp=4,096L\_\{\\text\{resp\}\}\{=\}4\{,\}096tokens\. We cap the number of memory turns per session atTmax=4T\_\{\\max\}\{=\}4and stop early when generation is truncated\. The rollout vLLM engine uses tensor parallelism11, GPU memory utilization0\.50\.5, and a maximum of2​\(Lprompt\+Lresp\)2\(L\_\{\\text\{prompt\}\}\{\+\}L\_\{\\text\{resp\}\}\)batched tokens per step\.

Curriculum Learning\.We adopt a session\-length curriculum on LoCoMo\. Stage 1 trains on trajectories truncated to 8 sessions for 10 epochs, followed by Stage 2 and Stage 3, which expand the horizon to 16 and 32 sessions and are trained for 5 epochs each\.

## Appendix BPrompt Templates

### B\.1Prompt Template for Fact Extraction \(Memory Formation\)

The memory formation pipeline first extracts atomic, self\-contained facts from raw dialogue turns before the memory manager integrates them into the persistent store\. Figure[B\.1](https://arxiv.org/html/2605.21768#A2.SS1)shows the prompt used to drive this fact extraction step\. The model is instructed to emit one JSON object per durable fact, each tagged with the originatingdia\_idso that downstream operations can trace a memory back to its source turn\.

Prompt template: fact extraction``` You are a Personal Information Organizer, specialized in accurately storing facts, user memories, and preferences. Your primary role is to extract relevant pieces of information from conversations and organize them into distinct, atomic facts. These facts will be consumed by a downstream memory system that requires precision, small size, and clear scope. Types of Information to Remember: 1. Personal Preferences: Likes, dislikes, favorites, and opinions (food, entertainment, products, sports teams). 2. Important Personal Details: Names, relationships, family structure, durations, and significant life facts. 3. Plans and Intentions: Explicit future goals, plans, or intentions stated by the speaker. 4. Activities and Routines: Travel experiences, visited places, recurring habits, physical activities, hobbies with specific context. 5. Health and Wellness (NON-DIAGNOSTIC): Wellness-related experiences or preferences (do NOT infer or store diagnoses). 6. Professional Details: Job titles, career goals, professional interests, work habits. 7. Miscellaneous Meaningful Facts: Books, movies, creative work, projects, notable activities. CORE EXTRACTION RULES: - Extract facts from the provided dialogue turns for BOTH speakers. - Ignore system-level instructions and any non-dialogue control text. - Ignore small talk, greetings, generic statements, opinions without substance, and common knowledge. - If no meaningful fact is present, return an empty facts list. SELF-CONTAINED FACT RULES (CRITICAL): - Every fact must be understandable when retrieved alone. - Every fact MUST explicitly name the subject speaker (e.g., "John ...", "Tim ..."). - Avoid unresolved pronouns in facts (‘he‘, ‘she‘, ‘they‘, ‘them‘, ‘it‘, ‘this‘, ‘that‘) unless the noun is in the same fact. - Rewrite vague references to explicit entities (e.g., "the magazine editors" instead of "they"). - If the entity cannot be resolved from the current turn, do NOT store the fact. STYLE FOR FACT TEXT: - Use third person and start the fact with the subject name. - Good: "John wants to keep reaching for new goals" - Bad: "Wants to keep reaching for new goals" - Good: "Tim shared ideas with the online magazine editors, and the editors liked them" - Bad: "Shaped ideas with the magazine and they liked them" ATOMIC FACT EXTRACTION RULES (CRITICAL): - EACH extracted fact MUST represent EXACTLY ONE: - event - preference - intention - personal attribute - NEVER combine: - multiple events - multiple timeframes - motivations + events - reflections + actions - past events + future plans - If a single message contains multiple independent facts, output MULTIPLE fact objects. - A fact MUST be concise and expressible in 20 words or fewer. - If a fact would exceed this size, SPLIT it into multiple smaller, independent facts. - Do not store bare dialogue acts such as "asks", "says hello", "thanks", unless they contain a durable personal fact. INTENT VS EVENT RULE: - Past events (what happened) and intentions or goals (what the speaker wants or plans) MUST ALWAYS be extracted as SEPARATE facts. TEMPORAL INFORMATION RULE: - If a fact includes temporal information (dates, durations, relative times), always include it explicitly. - Prefer absolute time when resolvable from context (e.g., "in 2018" not "five years ago" if the session year is known). - Do NOT drop temporal details — they are often critical for answering questions correctly. Important Notes on dia_id: - ‘dia_id‘ uniquely identifies the dialogue turn. - EACH fact must include the ‘dia_id‘ of its source message. - Do NOT attach multiple dia_ids to a single fact. Here are some few-shot examples: Input: [{"speaker": "John", "text": "Hi, how are you?", "dia_id": "D1:1"}] Output: {"facts": []} Input: [{"speaker": "John", "text": "There are branches in trees.", "dia_id": "D2:3"}] Output: {"facts": []} Input: [ {"speaker": "Maria", "text": "What’s your favorite sport?", "dia_id": "D3:1"}, {"speaker": "John", "text": "I love playing basketball with friends.", "dia_id": "D3:2"} ] Output: {"facts": [{"speaker": "John", "dia_id": "D3:2", "fact": "John loves playing basketball with friends"}]} Input: [ {"speaker": "Maria", "text": "What did you do yesterday?", "dia_id": "D3:5"}, {"speaker": "John", "text": "Yesterday, I had a meeting at 3pm. We discussed a new project.", "dia_id": "D3:6"} ] Output: {"facts": [{"speaker": "John", "dia_id": "D3:6", "fact": "John had a meeting at 3pm yesterday about a new project"}]} Input: [{"speaker": "John", "text": "I am a software engineer.", "dia_id": "D4:2"}] Output: {"facts": [{"speaker": "John", "dia_id": "D4:2", "fact": "John is a software engineer"}]} Input: [{"speaker": "John", "text": "My favorite movies are Inception and Interstellar.", "dia_id": "D4:3"}] Output: {"facts": [{"speaker": "John", "dia_id": "D4:3", "fact": "John’s favorite movies are Inception and Interstellar"}]} Input: [{"speaker": "John", "text": "I attended an LGBTQ workshop last Friday and it inspired me to pursue counseling.", "dia_id": "D5:1"}] Output: {"facts": [ {"speaker": "John", "dia_id": "D5:1", "fact": "John attended an LGBTQ workshop last Friday"}, {"speaker": "John", "dia_id": "D5:1", "fact": "John feels inspired to pursue a counseling career"} ]} Input: [ {"speaker": "Maria", "text": "That sounds wonderful!", "dia_id": "D6:3"}, {"speaker": "John", "text": "Thanks! I really appreciate your support.", "dia_id": "D6:4"} ] Output: {"facts": []} Input: [{"speaker": "John", "text": "So what do you think about that?", "dia_id": "D7:2"}] Output: {"facts": []} Return the facts in JSON format exactly as shown above. Remember the following: - If multiple statements describe the SAME EVENT at the SAME TIME and PLACE within the SAME dialogue turn, they MAY be merged into a SINGLE fact. - If they differ by time, place, motivation, outcome, or reflection -> extract SEPARATE facts. - Do NOT output standalone facts that depend on another fact for context unless merged into a complete event. - Do not return anything from the custom few-shot example prompts provided above. - If no relevant facts are found, return {"facts": []}. - The response MUST be valid JSON with a single top-level key: "facts". ```

Figure 6:Prompt template for atomic fact extraction\. Each extracted fact is a self\-contained, third\-person statement tagged with the originatingdia\_id, and is then passed to the memory manager \(Figure[B\.2](https://arxiv.org/html/2605.21768#A2.SS2)\) for integration into the persistent memory store\.

### B\.2Prompt Template for the Memory Manager \(Memory Evolution\)

The memory manager is the second stage of our pipeline\. It takes the atomic facts produced by the fact\-retrieval prompt \(Section[B\.1](https://arxiv.org/html/2605.21768#A2.SS1)\) together with the current memory store, and decides for each new fact whether to insert, update, delete, or take no operation on the store\. Compared with a naive memory writer, our prompt enforces three properties that proved important in practice: \(i\)*atomicity*, so that each memory entry encodes exactly one fact; \(ii\)*monotonicity*, so that prior factual claims are never silently dropped during anUpdate; and \(iii\)*noise tolerance*over the embedding\-based candidates, which can be topically unrelated\. Figure[B\.2](https://arxiv.org/html/2605.21768#A2.SS2)shows the full prompt\.

Prompt template: memory bank management``` You are a smart memory manager which controls the memory of a system. You can perform four operations: (1) insert into the memory, (2) update the memory, (3) delete from the memory, and (4) no change. Your primary goal is to preserve accurate factual evidence over time. Memory updates must be SAFE, NON-DESTRUCTIVE, and FACT-PRESERVING. CONTENT LENGTH RULE (enforced before all other rules): - Every ‘content‘ field you write (INSERT or UPDATE) MUST be at most 20 words. - ONE fact per memory item -- never combine multiple independent facts into one entry. - If you cannot express a fact in 20 words, write the most essential part only. INPUT FORMAT: The input contains two sections: - "memories": a flat list of existing memory entries (each appears exactly once, identified by memory_id). - "facts": new facts to process, each with a "related_memory_ids" list -- IDs pointing into "memories" as candidates for UPDATE or DELETE for that specific fact. To find candidates for a fact: look up its "related_memory_ids" in the "memories" list by memory_id. WARNING: "related_memory_ids" are retrieved by embedding similarity and MAY CONTAIN NOISE -- some IDs may point to memories that are topically unrelated to the fact. Do NOT blindly UPDATE or DELETE just because an ID appears in "related_memory_ids". Always verify the memory content actually refers to the same entity and topic before acting on it. If no entry is genuinely relevant, treat "related_memory_ids" as empty. For each new fact, decide whether to: - INSERT: The fact is new and not captured by any entry in its "related_memory_ids" (or list is empty or has no genuinely relevant entry). - UPDATE: The fact refers to the SAME entity or event as a "related_memory_ids" entry and enriches, refines, or corrects it WITHOUT removing prior factual information. - DELETE: The fact explicitly proves a "related_memory_ids" entry is false or invalid (not merely outdated). - NO OPERATION: The fact is already captured by a "related_memory_ids" entry, redundant, irrelevant, or insignificant. There are specific guidelines to select which operation to perform: 1. INSERT: If the fact contains new information not captured in its ‘related_memory_ids‘, then you have to add it. - Assign ‘speaker‘ as who the fact is ABOUT. - Assign ‘content‘ as a concise summary in third person. - Keep tense faithful to the source fact (past events may stay past tense). - Do NOT assign ‘memory_id‘ for INSERT operations; the system will auto-generate it. - Always include the ‘dia_id‘ with each inserted fact to ensure the memory is accurately linked to the correct dialogue. - ‘content‘ must be SELF-CONTAINED and include the subject name explicitly (e.g., "John ...", "Tim ..."). - ‘content‘ must avoid vague pronouns unless the referenced noun appears in the same sentence. - If a fact cannot be made self-contained without guessing, skip it (NO OPERATION). Example: - Input: { "memories": [{"memory_id": "a32b32c1", "speaker": "John", "content": "John works as a software engineer", "session_time": "6:59 pm on 26 August, 2023", "dia_ids": ["D1:4"]}], "facts": [ {"speaker": "John", "dia_id": "D3:6", "fact": "John had a meeting at 3pm", "related_memory_ids": ["a32b32c1"]}, {"speaker": "John", "dia_id": "D3:6", "fact": "John discussed a new project", "related_memory_ids": ["a32b32c1"]} ] } - Operations: {"operations": [ {"operation": "INSERT", "speaker": "John", "content": "John had a meeting at 3pm about a new project", "dia_id": "D3:6"} ]} ATOMICITY RULE: - Each memory item MUST represent a single fact or event. - Do NOT merge multiple independent facts into one memory item. - If a new fact represents a genuinely different event, topic, or attribute -> INSERT instead of UPDATE. - Exception: if the new fact is a progression or status change of the SAME entity’s story (e.g., "exploring a job" -> "accepted the job"), UPDATE the existing entry rather than inserting a duplicate. 2. UPDATE: Use UPDATE only when the new fact clearly refers to the SAME entity or event as an entry in its ‘related_memory_ids‘ and ADDS detail, refinement, or correction WITHOUT removing prior facts. - NEVER remove existing factual information during an UPDATE. - If the new fact is more specific, merge it with the existing content. - If both convey the same meaning, keep the more informative version. - If the new fact introduces a completely unrelated event, goal, or topic -> INSERT instead. - If the new fact is a later development or confirmation of the SAME entity’s ongoing story (e.g., plan -> outcome, exploring -> confirmed) -> UPDATE even if the time is different. - Please keep in mind while updating you have to use the same ID. - Always include the ‘dia_id‘ with each updated fact to ensure the memory is accurately linked to the correct dialogue. - Please note to return the IDs in the output from the input IDs only and do not generate any new ID. Example (refinement -- same entity, added detail): - Input: { "memories": [{"memory_id": "a0299e69", "speaker": "Emily", "content": "Likes to play cricket", "session_time": "2:04 pm on 3 September, 2021", "dia_ids": ["D5:2"]}], "facts": [{"speaker": "Emily", "fact": "Emily loves to play cricket with friends", "dia_id": "D5:4", "related_memory_ids": ["a0299e69"]}] } - Operations: {"operations": [{"operation": "UPDATE", "memory_id": "a0299e69", "content": "Emily loves to play cricket with friends", "dia_id": "D5:4"}]} Example (cross-session fact evolution -- status changed from exploring to confirmed): - Input: { "memories": [{"memory_id": "f3a91b44", "speaker": "Sarah", "content": "Sarah is exploring a job opportunity at a tech company in Seattle", "session_time": "3:00 pm on 10 March, 2022", "dia_ids": ["D2:5"]}], "facts": [{"speaker": "Sarah", "dia_id": "D3:8", "fact": "Sarah accepted a senior software engineer role at TechCorp in Seattle", "related_memory_ids": ["f3a91b44"]}] } - Operations: {"operations": [{"operation": "UPDATE", "memory_id": "f3a91b44", "content": "Sarah accepted a senior software engineer role at TechCorp in Seattle", "dia_id": "D3:8"}]} Explanation: Same entity (Sarah’s job in Seattle), status evolved from "exploring" to "accepted" -- UPDATE the same memory entry. Do NOT insert a duplicate about the Seattle job. INVALID UPDATE EXAMPLE (DO NOT DO THIS): - Memory content: "Sarah traveled to Paris and Rome on her European trip" - Wrong UPDATE content: "Sarah traveled to Paris" -- removes Rome, destroys stored fact - Correct action: NO OPERATION (no new information) or INSERT a separate fact about Rome if it was new. 3. DELETE: Use DELETE only when a new fact explicitly contradicts and invalidates an entry in its ‘related_memory_ids‘. - Do NOT delete memories just because they are old or less relevant. - Please note to return the IDs in the output from the input IDs only and do not generate any new ID. Example: - Input: { "memories": [{"memory_id": "6v0k193d", "speaker": "Samy", "content": "I went to Paris last summer", "session_time": "8:04 am on 3 February, 2009", "dia_ids": ["D6:5"]}], "facts": [{"speaker": "Samy", "fact": "Samy never went to Paris", "dia_id": "D7:1", "related_memory_ids": ["6v0k193d"]}] } - Operations: {"operations": [{"operation": "DELETE", "memory_id": "6v0k193d"}]} 4. NO OPERATION: If the new fact is already captured by an entry in its ‘related_memory_ids‘ -- even if worded differently -- do NOT insert a new entry. Before deciding INSERT, look up the fact’s ‘related_memory_ids‘ in "memories" and check for semantic overlap: same person, same topic, same meaning. If a semantically equivalent memory already exists -> NO OPERATION (not INSERT). If ‘related_memory_ids‘ is empty -> INSERT is safe. Example (exact match): - Input: { "memories": [{"memory_id": "9b3c82e0", "speaker": "Sofia", "content": "Sofia loves cheese pizza", "session_time": "11:10 am on 18 March, 2020", "dia_ids": ["D8:3"]}], "facts": [{"speaker": "Sofia", "fact": "Sofia loves cheese pizza", "dia_id": "D8:10", "related_memory_ids": ["9b3c82e0"]}] } - Operations: {"operations": []} Example (semantic match -- paraphrase is NOT a new fact): - Input: { "memories": [{"memory_id": "c4184b6a", "speaker": "Alex", "content": "Alex is training for a marathon with a local running club", "session_time": "9:00 am on 5 January, 2022", "dia_ids": ["D1:3"]}], "facts": [{"speaker": "Alex", "dia_id": "D1:9", "fact": "Alex is preparing for a marathon competition with teammates", "related_memory_ids": ["c4184b6a"]}] } - Operations: {"operations": []} Explanation: The new fact describes the same activity already in "memories". It is a paraphrase, not new information -> NO OPERATION. Example (same memory shared by two facts): - Input: { "memories": [{"memory_id": "a32b32c1", "speaker": "John", "content": "John works as a software engineer", "session_time": "6:59 pm on 26 August, 2023", "dia_ids": ["D1:4"]}], "facts": [ {"speaker": "John", "dia_id": "D3:6", "fact": "John changed careers to become a teacher", "related_memory_ids": ["a32b32c1"]}, {"speaker": "John", "dia_id": "D3:7", "fact": "John no longer works in tech", "related_memory_ids": ["a32b32c1"]} ] } - Operations: {"operations": [{"operation": "UPDATE", "memory_id": "a32b32c1", "content": "John became a teacher, left software engineering", "dia_id": "D3:6"}]} Explanation: Both facts point to the same memory. Produce ONE UPDATE -- do not UPDATE the same memory_id twice. DECISION ORDER (follow this sequence for EVERY new fact): 1. Does the new fact explicitly contradict a memory entry in its ‘related_memory_ids‘? -> DELETE the contradicted entry. 2. Does a semantically equivalent entry already exist in ‘related_memory_ids‘ (same person, same topic, same meaning)? -> NO OPERATION. Stop. 3. Does an entry in ‘related_memory_ids‘ exist and the new fact refines, progresses, or confirms the same entity’s story? -> UPDATE. Stop. 4. No matching entry found -> INSERT. Follow the instruction mentioned below: - Memory is MONOTONIC: factual information must never be lost unless explicitly contradicted. - UPDATE operations MUST preserve all previously stored factual claims. An UPDATE must preserve all existing factual claims, but may rephrase them concisely within size limits. - Do not return anything from the custom few shot prompts provided above. - You should return the operations in only JSON format as shown above. - Do not store small talk, greetings, generic questions. Only store information that conveys meaningful or significant facts. - If there is an insert, must include speaker field. must not include memory_id, session_time fields because the system auto-generates it. - If there is a deletion or update, must use exact memory_id from the "memories" list (looked up via the fact’s ‘related_memory_ids‘). Do not invent or guess memory IDs. - If two facts share a related_memory_ids entry, produce at most ONE operation on that memory_id -- do not UPDATE or DELETE the same memory_id twice. - Before outputting operations, run a strict self-check: 1) Every ‘content‘ is understandable alone. 2) Every ‘content‘ explicitly names the subject speaker. 3) No unresolved vague pronouns remain. 4) No entry is only a conversational act without durable fact value. Do not return anything except the JSON format. ```

Figure 7:Prompt template for the memory manager\. The model receives the current memory store and a batch of atomic facts \(output of the fact\-retrieval stage, Appendix[B\.1](https://arxiv.org/html/2605.21768#A2.SS1)\) and emits a JSON list ofINSERT/UPDATE/DELETEedits\. A fixed decision order, an atomicity constraint, and explicit non\-destructive update semantics together prevent the common failure modes of LLM\-based memory writers, namely fact loss, duplicated entries, and noisy retrieval\-driven overwrites\.

### B\.3Prompt Template for Answer Agent \(Memory Usage\)

The answer agent is the final stage of our pipeline\. It takes a user question together with the memory entries written by the memory manager \(Section[B\.2](https://arxiv.org/html/2605.21768#A2.SS2)\) and produces a concise, evidence\-grounded answer\. Two design choices are worth noting\. First, each memory entry carries a timestamp, and questions in our benchmark frequently involve relative time expressions \(*“last year”*,*“two months ago”*\); the prompt therefore instructs the model to resolve such expressions to absolute dates using the timestamp of the supporting memory, rather than the question’s own utterance time\. Second, the two speakers’ memories are presented in separate blocks labeled by speaker name, which prevents the model from confusing third\-party names mentioned within a memory with the speaker who owns that memory\. The model is required to terminate its response with an<answer\>\.\.\.</answer\>span, which is then extracted and scored against the gold answer using SQuAD\-style token\-level F1\. Figure[B\.3](https://arxiv.org/html/2605.21768#A2.SS3)shows the full prompt\.

Prompt template: memory\-grounded question answering``` You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories. # CONTEXT: You have access to memories from two speakers in a conversation. These memories contain timestamped information that may be relevant to answering the question. # INSTRUCTIONS: 1. Carefully analyze all provided memories from both speakers 2. Pay special attention to the timestamps to determine the answer 3. If the question asks about a specific event or fact, look for direct evidence in the memories 4. If the memories contain contradictory information, prioritize the most recent memory 5. If there is a question about time references (like "last year", "two months ago", etc.), calculate the actual date based on the memory timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021. 6. Always convert relative time references to specific dates, months, or years. For example, convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory timestamp. Ignore the reference while answering the question. 7. Focus only on the content of the memories from both speakers. Do not confuse character names mentioned in memories with the actual users who created those memories. 8. If memories are insufficient and the question is about a general world fact, you may use reliable general world knowledge. 9. Keep the final answer concise, typically no more than 10-12 words; do not omit essential entities or dates. # APPROACH (Think step by step): 1. First, examine all memories that contain information related to the question 2. Examine the timestamps and content of these memories carefully 3. Look for explicit mentions of dates, times, locations, or events that answer the question 4. If the answer requires calculation (e.g., converting relative time references), show your work 5. Formulate a precise, concise answer based on the evidence in the memories, using general world knowledge only if memories are insufficient 6. Double-check that your answer directly addresses the question asked 7. Ensure your final answer is specific and avoids vague time references 8. Output the final answer only in this format, with no extra text: <answer>YOUR_FINAL_ANSWER</answer> Memories for user speaker_1: speaker_1_memories Memories for user speaker_2: speaker_2_memories Question: question Answer step by step, and output the final answer in this format, with no extra text: <answer>YOUR_FINAL_ANSWER</answer> ```

Figure 8:Prompt template used for memory\-based question answering\. Double\-braced tokens denote runtime placeholders\. Model outputs are parsed from the<answer\>\.\.\.</answer\>span and scored with SQuAD\-style token F1\.

## Appendix CEvaluation Metrics

### C\.1LLM\-as\-a\-Judge

In addition to F1, B1, we report an LLM\-as\-a\-Judge \(J\) score that captures semantic equivalence between the generated answer and the gold answer, mitigating the well\-known brittleness of token\-level metrics on free\-form generations\. We follow the judging protocol established by prior work on memory\-augmented dialogue agents\[[1](https://arxiv.org/html/2605.21768#bib.bib3),[22](https://arxiv.org/html/2605.21768#bib.bib2)\], and usegpt\-4o\-minias the judge model for all reported J scores\. The judge receives the question, the gold answer, and the generated answer, and is asked to return a binary Correct/Wrong label, with explicit instructions to be lenient toward formatting differences \(e\.g\.,*“May 7”*versus*“7 May”*\) and toward the generated answer being more verbose than the gold\. The model is required to emit its decision as a JSON object with a singlelabelfield, which we parse for downstream aggregation\. We chosegpt\-4o\-minias a deliberate cost\-quality trade\-off: it is strong enough to reliably handle the lenient string\-matching judgments required here, while being cheap enough to run across the full evaluation set without distorting our compute budget\. Figure[C\.1](https://arxiv.org/html/2605.21768#A3.SS1)shows the full prompt\.

Prompt template: LLM\-as\-a\-Judge``` Your task is to label an answer to a question as ’CORRECT’ or ’WRONG’. You will be given: (1) a question, (2) a gold (ground truth) answer, (3) a generated answer. The gold answer is usually concise; the generated answer may be longer. Be generous: if the generated answer touches on the same topic/date as the gold, count CORRECT. Different formats for the same date (e.g. "May 7" vs "7 May") are CORRECT. Question: {question} Gold answer: {gold_answer} Generated answer: {generated_answer} First give a one-sentence reasoning, then finish with CORRECT or WRONG. Do NOT include both. Return a JSON object with key "label" whose value is exactly "CORRECT" or "WRONG". ```

Figure 9:Prompt template for the LLM\-as\-a\-Judge evaluator, instantiated withgpt\-4o\-mini\. Single\-braced tokens \(\{question\},\{gold\_answer\},\{generated\_answer\}\) are runtime placeholders\.

### C\.2Memory\-Failure Rate \(M\-Fail\)

To diagnose memory\-bank construction quality beyond answer\-level metrics, we defineM\-Failas the fraction of gold evidence that is missing from the memory bank\. For each questionqq, letℰq⊆𝒟\\mathcal\{E\}\_\{q\}\\subseteq\\mathcal\{D\}be the set of gold evidence dialogue\-turn IDs required to answerqq, and letℳ⊆𝒟\\mathcal\{M\}\\subseteq\\mathcal\{D\}denote the set of dialogue\-turn IDs currently stored in the memory bank\. We compute

M​\-​Fail=∑q\|ℰq∖ℳ\|∑q\|ℰq\|\.\\mathrm\{M\\text\{\-\}Fail\}=\\frac\{\\sum\_\{q\}\\left\|\\mathcal\{E\}\_\{q\}\\setminus\\mathcal\{M\}\\right\|\}\{\\sum\_\{q\}\|\\mathcal\{E\}\_\{q\}\|\}\.\(18\)Lower is better;M​\-​Fail=0\\mathrm\{M\\text\{\-\}Fail\}=0means all required evidence is present in memory\. This metric isolates*memory\-construction*errors \(evidence never stored\) from downstream retrieval or answer\-generation effects\.

## Appendix DLimitations and Future Work

Our study focuses on long\-horizon text\-only multi\-session dialogue, where memory is constructed from conversational turns and evaluated through downstream QA\. Extending the same training paradigm to multimodal settings, such as image\-grounded dialogue, video memory, or embodied interaction, remains unexplored\. While the proposed credit\-assignment framework is in principle agnostic to the modality of memory content, its practical effectiveness in richer environments still requires further investigation\.

This work has both potential positive and negative societal impacts\. On the positive side, Memory\-R2 improves the training of long\-horizon memory\-augmented LLM agents, which may help make current memory systems more reliable, consistent, and practically useful\. More effective memory construction can benefit applications such as personalized assistants, long\-term educational support, and tools that require maintaining user context across multiple interactions\. More broadly, improving memory quality may reduce failure modes caused by forgetting or inconsistent recall, which are common limitations of current LLM\-based agents\.

At the same time, stronger persistent memory systems also raise important risks\. In real\-world deployments, memory mechanisms may store sensitive personal information over extended interactions, creating privacy and security concerns if such information is retained unnecessarily, retrieved inappropriately, or exposed to unauthorized parties\. As memory becomes more accurate and durable, these risks may become more consequential\. In addition, errors in stored memory may persist across sessions and affect later interactions\. We therefore view privacy\-preserving memory design, secure storage and access control, and safer memory\-update mechanisms as important directions for future work\.

## Appendix EAdditional Ablations and Generalization Results

This section provides additional evidence for the effectiveness and stability of Memory\-R2\.

Figure[10](https://arxiv.org/html/2605.21768#A5.F10)breaks down LoGo\-GRPO versus standard GRPO across question types and curriculum stages, showing consistent gains from local rerollouts\.

Figure[11](https://arxiv.org/html/2605.21768#A5.F11)compares the proposed 8→\\rightarrow16→\\rightarrow32\-session curriculum with direct 32\-session training under equal compute, demonstrating that curriculum learning stabilizes validation performance, memory growth, memory quality, and reward signals\.

Figure[12](https://arxiv.org/html/2605.21768#A5.F12)further diagnoses the latency improvements of Memory\-R2, showing that the 3B model reduces latency mainly through more concise generations, while the 7B model reduces latency by suppressing long output\-length tails and improving batched decoding efficiency\.

Table[3](https://arxiv.org/html/2605.21768#A5.T3)reports mean and standard deviation over three independent runs, confirming that the improvements of Memory\-R2 are robust across random seeds\.

![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/logo_grpo_curriculum_1x4.png)Figure 10:LoGo\-GRPO consistently outperforms GRPO across all question types and curriculum stages\.Judge accuracy \(J\) at curriculum stages 8→\\rightarrow16→\\rightarrow32 sessions, broken down by question type:\(a\)Single\-hop,\(b\)Multi\-hop,\(c\)Temporal,\(d\)Open\-domain\. LoGo\-GRPO \(blue\) dominates GRPO \(gray\) at every stage and on every category, with the shaded band visualizing the gap, indicating that local rerollouts constantly mitigate credit\-assignment bias\. Accuracy continues to improve with the session range on long\-horizon question types \(Multi\-hop, Temporal\), while remaining stable on the others\.![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/direct_vs_curriculum.png)Figure 11:Curriculum learning is essential for stable long\-horizon training\.Training dynamics of curriculum 8→\\rightarrow16→\\rightarrow32 sessions \(blue\) vs\. direct 32\-session training \(orange\) under equal compute\. The x\-axis is the cumulative epochs within the curriculum; direct\-32sess is linearly stretched onto the same axis for fair comparison\.\(a\)ValidationF1F\_\{1\}on LoCoMo: the curriculum stabilizes around0\.500\.50, while direct\-32sess collapses from a peak of0\.470\.47down to0\.270\.27\.\(b\)Memory bank size grows steadily under the curriculum \(194→416→483194\\,\\rightarrow\\,416\\,\\rightarrow\\,483items at stage ends\), reflecting healthy accumulation; the direct run instead inflates and then truncates erratically\.\(c\)Memory failure rate \(M\-Fail\) stays below7%7\\%throughout the curriculum, but explodes to over70%70\\%for direct\-32sess once training enters the long\-horizon regime\.\(d\)Per\-sessionF1F\_\{1\}reward remains high across the curriculum \(0\.61/0\.58/0\.450\.61/0\.58/0\.45at stage ends\), while the direct run degrades to0\.230\.23\. Together, these dynamics show that early errors in the long\-horizon setting propagate across sessions and corrupt the memory bank, and that the curriculum allows the policy to acquire reliable short\-horizon memory behaviors before tackling longer trajectories\.![Refer to caption](https://arxiv.org/html/2605.21768v1/figure/acc_vs_latency_mechanism.png)Figure 12:Latency\-mechanism diagnostics\. \(a\) Memory\-R2 improves F1 while reducing per\-conversation latency\. \(b\) Memory\-R2 shortens the output\-length tail, indicating fewer overly long generations\. \(c\) Memory\-R2 reduces total completion tokens, especially for Qwen2\.5\-3B, indicating a more concise memory\-construction policy\. Together, these diagnostics suggest that the latency gains come from more controlled and less redundant generation\.Table 3:Main results per question category, averaged over 3 independent runs with different random seeds\. We report mean±\\pmstandard deviation\. The J score is computed by an LLM\-as\-a\-Judge withgpt\-4o\-minias the judge model \(Section[C\.1](https://arxiv.org/html/2605.21768#A3.SS1)\)\.