Multi-Head Recurrent Memory Agents

# Multi-Head Recurrent Memory Agents
Source: [https://arxiv.org/html/2607.01523](https://arxiv.org/html/2607.01523)
\\undefine@key

newfloatplacement\\undefine@keynewfloatname\\undefine@keynewfloatfileext\\undefine@keynewfloatwithin

Jiatong Li Samuel Yeh Sharon Li Department of Computer Science, University of Wisconsin\-Madison \{jli2947, samuelyeh, sharonli\}@cs\.wisc\.edu

###### Abstract

Recurrent memory agents extend LLMs to arbitrarily long contexts by iteratively consolidating input into a fixed\-size memory window\. Despite their scalability, these agents exhibit a well\-documented reliability problem: end\-to\-end performance degrades systematically as context length grows\. We diagnose this failure by decomposing performance into two factors—memory*capture*and memory*retention*—and quantitatively confirm that retention is the dominant bottleneck\. Retention collapses because existing designs maintain memory as a monolithic text block, forcing every update to risk overwriting previously retained content\. Motivated by this diagnosis, we proposeMulti\-Head Recurrent Memory \(MHM\), a general, training\-free framework that partitions memory into independent heads governed by a stage\-wise select\-then\-update strategy\. At each step, exactly one head is selected for update while the remaining heads are structurally shielded from overwriting, shifting the burden of retention from model behavior to architectural design\. As a lightweight instantiation, we introduce Least\-Recently\-Updated MHM \(MHM\-LRU\), which guarantees uniform head utilization with zero additional token overhead\. Extensive experiments on long\-context benchmarks show that MHM\-LRU substantially improves both retention and end\-to\-end accuracy across the 100K–1M token range, where baselines degrade sharply\. On RULER\-HQA at 896K tokens, MHM\-LRU improves the memory retention rate from less than 30% to73\.96%\. These gains generalize across model families, scales, and task types, positioning architectural optimization as a practical and cost\-efficient path toward reliable long\-context recurrent memory\. Our code is available[here](https://github.com/deeplearning-wisc/multi-head-memory)\.

### 1Introduction

The ability to reason over long contexts is a fundamental requirement for LLM agents operating in the real world\[zhang2024chain,jiang2024longllmlingua,li2023long,li2024loogle,xiao2024infllm\]\. Tasks such as document analysis\[wang2024leave,ma2024mmlongbench,li2025long,wu2025resum\], multi\-step research\[huang2025deep,zhang2025web,chen2026iterresearch\], and extended dialogue\[zhang2024chain,tan2025prospect\]demand that agents track and integrate information across inputs that far exceed the context window\. Recurrent memory\[zhou2023recurrengpt,yuan2025memsearcher,zhou2026mem1\]addresses this challenge by processing input sequentially in chunks and iteratively compressing a long context into a fixed\-size memory window\. At each step, the agent reads a new context chunk and updates its memory to integrate new information with what has been retained so far\. This paradigm is appealing because it places no hard limit on context length and requires no modification to the underlying LLM\. In practice, however, recurrent memory agents exhibit a well\-documented reliability problem: performance degrades systematically as context length grows\[yu2026memagent,du2025context\]\.

To motivate our work, we first diagnose how existing recurrent memory fails under long contexts by decomposing end\-to\-end performance into two factors: whether the memory ever*captures*the relevant information during memory update, and whether it*retains*that information through all subsequent memory updates\. Measuring both memory capture rate and memory retention rate on a representative method MemAgent\[yu2026memagent\], we find that capture remains stable across all context lengths, while memory retention decreases sharply, falling below 30% at 896K tokens\. This diagnosis pinpoints memory retention failure as the dominant bottleneck\. The root cause is architectural: existing recurrent memory designs maintain memory as a monolithic text block, so every update risks erasing key information\. Information written at stepttmust survive\(T−t\)\(T\-t\)subsequent overwrites to reach the final state, an increasingly unlikely outcome asTTgrows\.

![Refer to caption](https://arxiv.org/html/2607.01523v1/x1.png)Figure 1:Left: An overview of the multi\-head recurrent memory \(MHM\)\. Right: An overview of the memory head utilization pattern across time\. In every time step, only one head is written\. As time goes by, MHM achieves asymptotic uniform memory utilization\.Motivated by the diagnosis, we proposeMulti\-Head Recurrent Memory \(MHM\), a general, training\-free memory framework designed to target memory retention failure\. Rather than maintaining a single monolithic memory block, MHM partitions memory into multiple independent heads and governs updates through a stage\-wise select\-then\-update strategy\. At each step, one of the heads is selected for update while the remaining heads are held fixed\. This shifts the burden of retention from model behavior to the architectural design: rather than relying on the model to avoid overwriting important content, MHM structurally shields the information retained in unselected heads from erasure at each step\. The selection criterion is left general, accommodating any rule\-based or model\-based strategy, and existing single\-headed methods are subsumed as the special case\.

As a lightweight instantiation, we showcase MHM\-LRU, which selects the least\-recently\-updated head at each step\. To diversify memory head use to persist memory retention, this rule provably guarantees uniform head utilization across heads and incurs zero additional token overhead relative to a single\-headed baseline\. Extensive experiments on realistic long\-context benchmarks demonstrate that MHM\-LRU substantially improves memory retention, outperforming baselines across all evaluated context lengths\. Notably, on RULER\-HQA at 896K tokens, MHM\-LRU improves memory retention rate from less than 30% to73\.96%\. The retention improvements directly translate into end\-to\-end performance gains, where MHM\-LRU’s accuracy remains strong and stable across the 100K\-1M context range while baselines degrade sharply\. These benefits generalize across model families, scales, and task types, positioning architectural optimization as a practical and cost\-efficient path toward reliable long\-context recurrent memory\. Overall, the contributions of this paper are summarized as follows:

1. 1\.We identify memory retention failure as the dominant bottleneck of recurrent memory agents under extremely long contexts, supported by a quantitative capture\-retention decomposition analysis\.
2. 2\.We propose Multi\-Head Recurrent Memory \(MHM\), a general and training\-free memory architecture\-level framework that structurally limits overwrite pressure through independent memory heads and stage\-wise selective updates\.
3. 3\.We extensively validate MHM on realistic long\-context benchmarks, demonstrating substantial retention and accuracy improvements across context lengths, model families, and task types\.

### 2Preliminaries and Related Work

##### Long\-Context LLMs\.

Scaling LLMs to long contexts has been pursued along two primary directions at the model level: extending the context window via positional encoding modifications\[kazemnejad2023positional,he2024positional,wu2024positional\]and reducing attention complexity via sparse or linear attention mechanisms\[lai2025sparseattn,yang2025sparseattn,roy2021sparseattention\]\. These advances have enabled million\-token\-level context windows in both open\-source models such as Qwen2\.5\-1M\[yang2025qwen251m\], and proprietary systems such as Claude\-Sonnet\-4\.6\[anthropic2026claude\]and Gemini\-3\.1\-Pro\[google2026gemini\]\. Despite their effectiveness, native long\-context LLMs remain bounded by fixed context limits or quadratic memory costs, motivating recurrent memory as a complementary paradigm that sidesteps both constraints\.

##### Recurrent Memory Agents\.

The idea of recurrent memory was foreshadowed in earlier architecture\-level approaches\[bulatov2022recurrent,gu2023mamba,peng2023rwkv,behrouz2024titans\]\. However, constrained by the representational capacity of the underlying architecture, the generality of these methods are limited in a broader range of applications\. It was not until the emergence of token\-level recurrent memory\[yu2026memagent,zhou2023recurrengpt\]that this paradigm was concretely established\.

Formally, let\(C,q\)\(C,q\)denote a context\-query pair where an LLM agent must produce a correct responseyy\. SinceCCmay far exceed the model’s context window, the agent partitionsCCinto an ordered chunk sequenceC=\(c1,c2,…,cT\)C=\(c\_\{1\},c\_\{2\},\\ldots,c\_\{T\}\)and maintains a bounded memory statemtm\_\{t\}through sequential updates\. The memory agentfθf\_\{\\theta\}then processes these chunks sequentially, updating a bounded memory statemtm\_\{t\}at each step:

mt←fθ​\(mt−1,ct,q\),t=1,2,…,T\.m\_\{t\}\\leftarrow f\_\{\\theta\}\(m\_\{t\-1\},\\,c\_\{t\},\\,q\),\\quad t=1,2,\\ldots,T\.\(1\)After all chunks have been processed, the LLM agent generates its final response conditioned solely on the accumulated memory:

y←LLM​\(mT,q\)\.y\\leftarrow\\text\{LLM\}\(m\_\{T\},\\,q\)\.\(2\)
BecausemTm\_\{T\}is the sole information carrier from the firstTTchunks to the final response, the fidelity of the update in Eq\. \([1](https://arxiv.org/html/2607.01523#S2.E1)\) under growingTTis the central reliability challenge of this paradigm\. This paradigm has been concretely established by MemAgent\[yu2026memagent\]for long\-context query answering\. Subsequent work has extended it in several directions: GRU\-Mem\[sheng2026grumem\]introduces text\-controlled gating for selective updates, while ReMemR1\[shi2026remem\]augments recurrent memory with a revisitable retrieval component, yet the core recurrent working memory remains a monolithic block\.

##### Single\-Headed Memory as a Structural Bottleneck\.

Across existing recurrent memory designs\[yu2026memagent,shi2026remem,sheng2026grumem\], the core recurrent memory statemtm\_\{t\}is maintained as a single monolithic text block—a design we term*single\-head*memory\. At every update step, the memory agent must overwrite this block with content derived fromctc\_\{t\}, creating an inherent tension between retaining previously captured information and incorporating new context\. This tension compounds with context length: information captured at stepttmust survive\(T−t\)\(T\-t\)subsequent overwrites intact to reachmTm\_\{T\}\. Under sufficiently long contexts, compression into a single representational slot makes information loss structurally inevitable\. In the next section, we rigorously characterize and measure this failure, identifying it as the binding constraint on long\-context performance\.

### 3Characterizing Recurrent Memory Failure

To motivate our memory architectural design, we first analyze*how*recurrent memory fails under long contexts\. Incentivized by the working memory\[baddeley2012working\]and memory forgetting detection\[spear1971forgetting,eldridge2005dissociation\]in neuroscience, we decompose end\-to\-end performance into two memory factors:captureandretention\. For a correct response, the memory must \(1\)*capture*the relevant information at some point during iterative updates, and \(2\)*retain*it through all subsequent updates until the final statemTm\_\{T\}\. Failure at either stage produces a wrong answer, but the two modes need not be equally prevalent in practice\.

###### Definition 1\(Memory Capture Rate \(MCR\)\)\.

Let𝒟=\{\(Ci,qi,yi∗\)\}i=1N\\mathcal\{D\}=\\\{\(C\_\{i\},q\_\{i\},y^\{\*\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}be an evaluation dataset andℳ=\{mi\}i=1N\\mathcal\{M\}=\\\{m\_\{i\}\\\}\_\{i=1\}^\{N\}the corresponding memory trajectories\. Themi=\{mi,1,…,mi,T\}m\_\{i\}=\\\{m\_\{i,1\},\\ldots,m\_\{i,T\}\\\}denotes the memory state trajectory, whereTTis the total number of trajectory steps for the respective instance\. We assess answer membership𝟙​\(yi∗∈mi,t\)\\mathbbm\{1\}\(y^\{\*\}\_\{i\}\\in m\_\{i,t\}\)of the ground\-truth answer against the memory content at steptt\. MCR is defined as the fraction of queries for which the relevant information was present in memory at least once during processing:

MCR​\(ℳ,𝒟\)=1N​∑i=1N𝟙​\(yi∗∈mi,t,∃t∈\[T\]\)\.\\mathrm\{MCR\}\(\\mathcal\{M\},\\mathcal\{D\}\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\!\\left\(y^\{\*\}\_\{i\}\\in m\_\{i,t\},\\;\\exists\\,t\\in\[T\]\\right\)\.\(3\)

###### Definition 2\(Memory Retention Rate \(MRR\)\)\.

Among queries where memory capture occurred, the fraction for which the captured information survived through to the final memory statemTm\_\{T\}:

MRR​\(ℳ,𝒟\)=∑i=1N𝟙​\(yi∗∈mi,t,∃t∈\[T−1\]\)⋅𝟙​\(yi∗∈mi,T\)∑i=1N𝟙​\(yi∗∈mi,t,∃t∈\[T\]\)\.\\mathrm\{MRR\}\(\\mathcal\{M\},\\mathcal\{D\}\)=\\frac\{\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\(y^\{\*\}\_\{i\}\\in m\_\{i,t\},\\exists\\,t\\in\[T\-1\]\)\\cdot\\mathbbm\{1\}\(y^\{\*\}\_\{i\}\\in m\_\{i,T\}\)\}\{\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\(y^\{\*\}\_\{i\}\\in m\_\{i,t\},\\;\\exists\\,t\\in\[T\]\)\}\.\(4\)A high MRR indicates that the memory agent is capable of retaining previously captured information when processing irrelevant context\.

![Refer to caption](https://arxiv.org/html/2607.01523v1/x2.png)Figure 2:\(a\)MCR and MRR of MemAgent\[yu2026memagent\]\(Qwen2\.5\-14B\-Instruct\) on RULER\-HQA across context lengths from 7K to 896K tokens\. While MCR remains stable throughout, MRR degrades sharply with context length, falling below 30% at 896K tokens\.\(b\)MRR and Acc correlation analysis of MemAgent and ReMem\[shi2026remem\]on RULER\-HQA across context lengths from 7K to 896K tokens\. The ‘LR’ denotes linear regression\. End\-to\-end accuracy correlates strongly with MRR \(ρ=0\.98\\rho=0\.98\), confirming that retention failure is the dominant factor\.##### Retention Failure Dominates\.

We apply this analysis to MemAgent, a representative recurrent memory baseline built on Qwen2\.5\-14B\-Instruct\[yang2024qwen25\], on the RULER\-HQA\[hsieh2024ruler,yang2018hotpotqa\]benchmark\. As shown in Figure[2](https://arxiv.org/html/2607.01523#S3.F2)\(left\), MCR remains stable across all evaluated context lengths, indicating that the agent consistently captures relevant information regardless of input length\. MRR, by contrast, degrades sharply, falling below 30% at 896K tokens\. Since MCR is roughly stable, the observed accuracy degradation is almost entirely attributable to retention failure\. Appending results from ReMem\[shi2026remem\], Figure[2](https://arxiv.org/html/2607.01523#S3.F2)\(right\) further confirms that end\-to\-end accuracy correlates strongly with MRR, with correlation coefficientρ=0\.98\\rho=0\.98\. This isolates memory retention as the main structural bottleneck, and directly motivates the design question we address in Section[4](https://arxiv.org/html/2607.01523#S4): how to make the memory structure itself resistant to overwrite pressure\.

### 4Multi\-Head Recurrent Memory

Motivated by our analysis, we proposeMulti\-Head Recurrent Memory\(MHM\), a general framework designed to alleviate retention failure\. The core insight is that retention failure is not a failure of the model but the memory structure: when a single block holds all context, every update is a potential erasure\. MHM addresses this at the source by splitting memory intoHHmutually independent memory heads and governing updates through a selective overwrite strategy\. More specifically, MHM represents the memory state as a set ofHHindependent text blocks,mt=\{mt1,…,mtH\}m\_\{t\}=\\\{m^\{1\}\_\{t\},\\ldots,m^\{H\}\_\{t\}\\\}, where each headmthm^\{h\}\_\{t\}is delimited by<memory\_\{h\}\> … </memory\_\{h\}\>tags\. Each update affects only one memory head, while the remaining\(H−1\)\(H\-1\)heads are structurally shielded from erasure by construction\. Our design addresses two challenges: which head to select for update, and how to perform the update given the selected head\. We describe the two stages below\.

##### Selection Stage\.

The selection stage addresses the memory retention failure throughdiversifiedhead selection—by spreading updates across different heads over time, the agent avoids repeatedly overwriting the same head and thereby reduces the risk of erasing pivotal information\. Formally, given the current memory statemt−1=\{mt−11,…,mt−1H\}m\_\{t\-1\}=\\\{m\_\{t\-1\}^\{1\},\\ldots,m\_\{t\-1\}^\{H\}\\\}, the selection stage identifies the headh⋆∈\[H\]h^\{\\star\}\\in\[H\]most worthy of being updated:

h⋆=arg⁡maxh∈\[H\]⁡rselect​\(h∣mt−1,q,ct\)\.h^\{\\star\}=\\arg\\max\_\{h\\in\[H\]\}\\;r\_\{\\mathrm\{select\}\}\(h\\mid m\_\{t\-1\},q,c\_\{t\}\)\.\(5\)whererselect​\(h∣mt−1,q,ct\)r\_\{\\text\{select\}\}\(h\\mid m\_\{t\-1\},q,c\_\{t\}\)is a criterion function that assigns a priority score to each head candidate\. The criterionrselectr\_\{\\mathrm\{select\}\}can encode a wide range of selection strategies depending on the available information and computational budget\. A rule\-based criterion may select based on head age or a fixed schedule\. A model\-based criterion may assess the relevance of each head given the current chunk and query\. Regardless of the specific instantiation, a well\-designed criterion should satisfy a key property:*diversity*, ensuring that updates are spread across heads over time so that no single head bears disproportionate overwrite pressure\. We will provide an ablation on different selection strategies in Section[5\.3](https://arxiv.org/html/2607.01523#S5.SS3)\.

##### Update Stage\.

The update stage overwrites the selected headh⋆h^\{\\star\}with content derived from the current context chunk\. Formally, given the current memory statemt−1m\_\{t\-1\}, the selected head indexh⋆h^\{\\star\}, the queryqq, and the current chunkctc\_\{t\}, the memory agent generates new head content and updates the memory state as follows:

mth⋆=fθ​\(mt−1,h⋆,q,ct\),mth=mt−1h​∀h≠h⋆\.m^\{h^\{\\star\}\}\_\{t\}=f\_\{\\theta\}\(m\_\{t\-1\},\\,h^\{\\star\},\\,q,\\,c\_\{t\}\),\\qquad m^\{h\}\_\{t\}=m^\{h\}\_\{t\-1\}\\;\\;\\forall\\,h\\neq h^\{\\star\}\.\(6\)By receiving the full memory statemt−1m\_\{t\-1\}as context, the update allows the memory agent to reason jointly about what has already been retained across all heads and what new information fromctc\_\{t\}is worth writing\. This encourages both capturing newly relevant content and avoiding redundancy with information already preserved in unselected heads\. This stage\-wise design subsumes existing single\-headed methods as the special caseH=1H=1, making MHM a flexible and general framework\.

##### A Lightweight Instantiation\.

As a concrete instantiation of MHM, inspired by the least\-recently\-used cache replacement algorithm\[lee1999existence,jelenkovic2004least,megiddo2004outperforming\], we adopt a least\-recently\-updated \(LRU\) selection rule: at each step, the head that has gone the longest without an update is selected\. This instantiation has two advantages\. First, the selection strategy ensures diversified head selection\. When all heads are initialized identically, this provides a structural guarantee that no head is updated more than once before all others are updated at least once\. This ensures uniform utilization across heads and maximizes memory redundancy\. Second, this method is computationally lightweight since head selection requires only a simple age comparison rather than an additional LLM inference pass, incurring*no extra token overhead*relative to a single\-headed baseline\. We refer to this instantiation asMHM\-LRU\. As our experiments demonstrate, the structural guarantee of uniform head utilization can substantially improve retention and end\-to\-end performance in long context scenarios, suggesting that the benefits of MHM do not depend on sophisticated selection strategies\. Full prompt templates and implementation details are provided in Appendix[A\.2](https://arxiv.org/html/2607.01523#A1.SS2)\.

### 5Experiments

#### 5\.1Experimental Setup

##### Baselines\.

We compare MHM\-LRU against two latest recurrent memory baselines: \(1\) MemAgent\[yu2026memagent\], which maintains a single memory head and applies a full overwrite at every update step, and \(2\) ReMem\[shi2026remem\], which maintains two heterogeneous heads \(an update memory and a retrieval memory\) where the retrieval memory is refreshed via a rule\-based callback query\. GRU\-Mem\[sheng2026grumem\]is excluded as its implementation is not publicly available\. All methods use Qwen2\.5\-14B\-Instruct as the backbone of the recurrent memory structure\. We additionally evaluate on Qwen2\.5\-32B\-Instruct and gpt\-oss\-120b in the ablation study\. To ensure fair comparison,*the total memory capacity is kept the same across all methods*: 4,096 tokens for MemAgent \(one head\), 2,048 per head for ReMem \(two heads\), and 1,024 per head for MHM\-LRU \(4 heads\)\. Full decoding and evaluation settings are provided in Appendix[B](https://arxiv.org/html/2607.01523#A2)\.

##### Datasets\.

We evaluate on two benchmarks: \(1\)RULER\-HQA\[yu2026memagent\], a multi\-hop QA dataset spanning context lengths from 7K to 892K tokens, following the evaluation protocol of\[yu2026memagent\], and \(2\)BABILong\[kuratov2024babilong\], a long\-context reasoning benchmark covering ten structurally distinct task types \(Q1–Q10\) across context lengths from 1K to 10M tokens\. For BABILong, we evaluate on context lengths from 8K to 1M tokens\. For each length, we sample 128 entries via systematic sampling to ensure uniform coverage of all task types\. All recurrent memory methods are run three times independently; we report the mean and standard deviation throughout\.

#### 5\.2Main Results

##### MHM Substantially Improves Memory Retention

Figure[3](https://arxiv.org/html/2607.01523#S5.F3)shows MRR across context lengths for all methods\. We observe that MHM\-LRU consistently and substantially outperforms both baselines, with the gap widening at longer contexts\. For example, on RULER\-HQA at 896K tokens, MHM\-LRU achieves an MRR of73\.96% compared to less than 30% for MemAgent and ReMem\. On BABILong at 1M tokens, MHM\-LRU reaches 68\.68% versus 42\.96% for MemAgent\. These results demonstrate the effectiveness of MHM\-LRU in alleviating the memory retention issue of recurrent memories as context length increases\. Moreover, at shorter context lengths \(under 50K tokens\), MHM\-LRU achieves near\-perfect retention \(MRR\>\>99%\) on both datasets, confirming that the multi\-head design incurs no penalty in the common short\-to\-mid\-length regime\. Overall, the experiments demonstrate the effectiveness of MHM in addressing the retention bottleneck of current memory\.

![Refer to caption](https://arxiv.org/html/2607.01523v1/x3.png)Figure 3:MRR results across different context lengths\. Thexx\-axis represents context length\. Each data point is an average across 3 random runs with standard deviation as the error bar\.Table 1:End\-to\-end answer accuracy↑\\uparrow\(%\) results on RULER\-HQA\.Table 2:End\-to\-end answer accuracy↑\\uparrow\(%\) results on BABILong\.
##### Retention Gains Translate Directly to End\-to\-End Performance\.

Table[1](https://arxiv.org/html/2607.01523#S5.T1)and Table[2](https://arxiv.org/html/2607.01523#S5.T2)report end\-to\-end accuracy across all context lengths\. We observe that MHM\-LRU consistently outperforms both recurrent memory baselines and the native long\-context LLM across nearly all settings\. The advantage is most pronounced at extreme context lengths: on RULER\-HQA at 896K tokens, MHM\-LRU achieves 49\.74% accuracy versus 21\.62% for MemAgent and 0\.00% for the LLM baseline; on BABILong at 1M tokens, 41\.41% versus 25\.26% for MemAgent\. Crucially, MHM\-LRU’s accuracy remains stable across the 100K–1M range \(48\.70% – 50\.26% on RULER\-HQA; 41\.41% – 48\.70% on BABILong\), whereas MemAgent and ReMem degrade sharply\. Taken together with the MRR results in Figure[3](https://arxiv.org/html/2607.01523#S5.F3), these findings confirm that the retention improvements from MHM directly translate into downstream performance gains\.

#### 5\.3Ablation Studies on Key Design

We ablate two design dimensions of MHM: the number of heads and the memory selection strategy\. All ablations are conducted on RULER\-HQA using Qwen2\.5\-14B\-Instruct, with per\-head capacity fixed at 1,024 tokens\.

![Refer to caption](https://arxiv.org/html/2607.01523v1/x4.png)Figure 4:Effect of number of memory heads across context lengths\.##### More Memory Heads Improve Retention\.

We evaluate the impact of the number of headsHHon the memory retention rate \(MRR\)\. We varyHHfrom 1 to 16, under each context length\. As shown in Figure[4](https://arxiv.org/html/2607.01523#S5.F4), MRR shows approximately logistic\-like improvement as the number of heads grows\. For example, under context length 896K, our approach improves MRR from 26\.53% with one head \(equivalent to MemAgent\) to 69\.89% with two heads, 74\.67% with four heads, and 87\.12% with eight heads, before plateauing at sixteen heads\. This trend holds consistently across all evaluated context lengths\. This ablation study reveals that increasing the number of heads can significantly improve memory retention, confirming the structural advantage of our design\.

Table 3:Ablation study of the multi\-headed memory selection strategy\.
##### Stage\-wise Selection Reduces Retention Failure\.

A critical design in MHM is the stage\-wise head selection and memory update\. We thus compare MHM\-LRU against a variant MHM\-Concur to isolate the effect of the stage\-wise design\. In MHM\-Concur, head selection and memory update are merged into a single step: the model is asked to simultaneously decide which head to overwrite and produce the updated content\. As shown in Table[3](https://arxiv.org/html/2607.01523#S5.T3), while MHM\-Concur achieves a notably higher MCR than the stage\-wise approach, it suffers from substantially lower MRR \(54\.12% vs\. 73\.96% at context length 896K\), resulting in lower end\-to\-end accuracy across all context lengths\. The concurrent design introduces instruction complexity that degrades the model’s ability to retain captured content: when the model must simultaneously reason about head routing and content generation, it is more likely to inadvertently erase previously retained information\. In contrast, the stage\-wise design eliminates this conflict by dedicating separate prompts to each operation, allowing the update stage to focus entirely on capture and retention without the competing objective of head routing\.

##### Rule\-based vs\. Relevance\-based Selection\.

We compare MHM\-LRU \(rule\-based\) against another variant, MHM\-Relevance, to isolate the effect of the selection criterion, holding the stage\-wise design fixed\. In MHM\-Relevance, the model is explicitly prompted to select the head most worthy of being updated, while diversifying the head usage \(details in Appendix[A\.4](https://arxiv.org/html/2607.01523#A1.SS4)\)\. The intuition is that the model can make relevance\-aware routing decisions\. Table[3](https://arxiv.org/html/2607.01523#S5.T3)shows that, despite this appeal, MHM\-Relevance usually incurs sub\-optimal MCR and MRR, leading to lower end\-to\-end accuracy \(45\.31% vs\. 49\.74% at context length 896K\)\. The model\-driven routing has two compounding issues\. First, model\-based relevance assessment at each step is inherently noisy: the model cannot reliably predict which information will be needed at query time, leading to routing decisions that concentrate updates on heads perceived as salient and starve others of updates\. Second, this concentration undermines coverage diversity, reducing the breadth of information captured across heads\. MHM\-LRU sidesteps the issues by guaranteeing uniform head utilization without any model involvement, requires no additional inference for selection, and achieves overall better performance\.

#### 5\.4Generality Analysis

Table 4:MRR \(%\) of different model families and scales\. Best metric value given the model and context length isbolded\.##### Generality Across Model Family and Scale\.

We evaluate MHM\-LRU with two additional base models: Qwen2\.5\-32B\-Instruct and gpt\-oss\-120b\. We focus on the 112K–896K settings, where retention failure is most severe\. Across both models, MHM\-LRU maintains significantly higher and more stable MRR than MemAgent, as shown in Table[4](https://arxiv.org/html/2607.01523#S5.T4)\. On gpt\-oss\-120b, MemAgent’s MRR falls below 40%, while MHM\-LRU maintains MRR to 68\.85%\. On Qwen2\.5\-32B\-Instruct, MHM\-LRU’s MRR consistently achieves 90% on 112K–448K and maintains to 81\.91% on 896K\.

This consistent pattern across model families and scales confirms that the retention benefit of MHM does not depend on any model\-specific property, but follows from the memory architectural advantage\.

\\caption@setoptions

floatrow\\caption@setoptionsfigurerow\\caption@setpositionb

\\caption@setoptionsfigure\\caption@setpositionbFigure 6:Accuracy by task type \(QA1 – QA10\) and context length \(128K – 1M\) on BABILong\.![Refer to caption](https://arxiv.org/html/2607.01523v1/x5.png)\\caption@setoptionsfigure\\caption@setpositionbFigure 8:VRAM cost and accuracy change on RULER\-HQA\.![Refer to caption](https://arxiv.org/html/2607.01523v1/x6.png)

##### Generality Across Task Types\.

To evaluate the generality of MHM\-LRU across diverse task types, we conduct experiments on BABILong, which encompasses ten structurally distinct reasoning tasks \(QA1–QA10\)\. As shown in Figure[8](https://arxiv.org/html/2607.01523#S5.F8), MHM\-LRU outperforms MemAgent across all ten task types at long contexts \(1M\), with particularly pronounced gains on tasks requiring relational reasoning and yes\-no judgment \(QA4–QA6\), where MemAgent’s accuracy degrades sharply with context length\. This pattern aligns with our retention analysis: tasks involving multi\-step relational chains are most susceptible to memory retention failure, making them the primary beneficiaries of MHM’s structural redundancy\. These results confirm that MHM\-LRU’s advantage is not task\-specific but reflects a systematic improvement that generalizes across task types\.

Table 5:Time and output token costs of recurrent memory agents on RULER\-HQA\. For each method, the test is run on an NVIDIA A100 80GB, with four parallel threads\.
##### Computational Efficiency\.

Figure[8](https://arxiv.org/html/2607.01523#S5.F8)plots accuracy degradation \(7K to 896K\) against total VRAM cost for all methods\. We additionally include MemAgent\+RL \(RL\-MemAgent\[yu2026memagent\]\), a reinforcement learning post\-trained variant of MemAgent that requires significantly more VRAM due to the overhead of RL training\. MHM\-LRU lies on the Pareto frontier between these two extremes: it recovers a substantial fraction of the accuracy gain achieved by MemAgent\+RL at no additional memory overhead beyond vanilla MemAgent\. This is notable because MHM\-LRU requires neither model weight access nor task\-specific training, and the gain comes entirely from the architectural design with multiple heads\. This positions architectural optimization as a practical and resource\-efficient path toward reliable long\-context recurrent memory, particularly in settings where model retraining or large VRAM budgets are unavailable\.

We further evaluate the time and output token costs of different recurrent memory methods under long contexts, as shown in Table[5](https://arxiv.org/html/2607.01523#S5.T5)\. MHM\-LRU is consistently faster than MemAgent at short\-to\-mid context lengths \(e\.g\., 25\.9s vs\. 31\.5s at 112K\), with negligible overhead at extreme lengths\. Similarly, MHM\-LRU consumes substantially fewer output tokens at shorter contexts \(142\.8 vs\. 330\.6 at 7K\), converging to parity with MemAgent only at 896K\. Both effects reflect a spontaneous amortization property of MHM: by writing to a single head per step, the LLM naturally generates fewer tokens per update, amortizing the cost of memory maintainance\. These results confirm that MHM\-LRU achieves its retention and accuracy gains at no meaningful increase in computational cost\.

![Refer to caption](https://arxiv.org/html/2607.01523v1/x7.png)Figure 9:A case study of memory trajectory on BABILong\-512K\. The task type is QA4: 2\-arg relations\.

#### 5\.5Memory Dynamics Case Study

To concretely illustrate how MHM\-LRU alleviates retention failure, Figure[9](https://arxiv.org/html/2607.01523#S5.F9)presents a representative memory trajectory on a BABILong\-512K sample from the 2\-arg relations task\. The query asks “What is the bathroom south of?” \(ground truth: “office”\), with relevant chunks falling within steps \[70, 96\)\. MemAgent captures the answer from step 74 onward but overwrites it during the final updates \(steps 91–95\), yielding the incorrect response “There is not enough information to determine what is south of the bathroom\.”, which is a representative instance of retention failure\. MHM\-LRU, in contrast, captures the answer at step 74 and then spontaneously propagates it to the remaining heads in subsequent updates, maintaining multiple redundant copies that persist through step 95 and yielding the correct answer “Therefore, the answer is the office\.” This emergent backup behavior concretely demonstrates how partitioning memory into independent heads converts the stochastic overwrite risk of a monolithic block into an architectural retention guarantee\.

### 6Conclusion and Limitation

We presented Multi\-Head Recurrent Memory \(MHM\), a new framework for improving the reliability of recurrent memory agents under long contexts\. Through a diagnostic analysis of memory dynamics, we identified retention failure as the dominant bottleneck of long\-context performance\. Motivated by this diagnosis, MHM is designed to address the memory retention failure through multiple memory heads and the stage\-wise memory update strategy\. MHM structurally limits overwrite pressure on retained content without requiring any model retraining\. MHM\-LRU, a lightweight instantiation, substantially improves memory retention and end\-to\-end accuracy on challenging long\-context query benchmarks\. We hope these findings encourage future work on architectural approaches to long\-context reliability, and on combining MHM with training\-based methods for further gains\.

##### Limitation\.

In our experiments, the number of headsHHis fixed\. While our ablation shows that MRR improves monotonically withHH, the optimal number of heads is likely dependent on context length and task structure\. An adaptive head allocation strategy that adjustsHHbased on input characteristics is a natural direction for future work\.

### Acknowledgment

We sincerely thank Changdae Oh, Linyue Cai and Hyeong Kyu Choi for proofreading the manuscript, and Jimmy Di for providing additional computational resources\. The work is supported in part by the AFOSR Young Investigator Program under award number FA9550\-23\-1\-0184, National Science Foundation under awards IIS\-2237037 and IIS\-2331669, Office of Naval Research, Schmidt Sciences Foundation, Open Philanthropy \(now Coefficient Giving\), and Alfred P\. Sloan Fellowship\.

### References

## Appendix

### Appendix AMulti\-Head Recurrent Memory Paradigm

#### A\.1Overview of MHM Pipeline

The pseudo\-code of the MHM pipeline is shown in Algorithm[1](https://arxiv.org/html/2607.01523#algorithm1)\.

Input:Context

CC, query

qq, backbone LLM, number of heads

HH, chunk size

LL
Output:Response

yy
Initialize memory

m0←\{m01,…,m0H\|m0h=∅,h∈\[H\]\}m\_\{0\}\\leftarrow\\\{m\_\{0\}^\{1\},\\ldots,m\_\{0\}^\{H\}\|m\_\{0\}^\{h\}=\\emptyset,h\\in\[H\]\\\};

Equally split

CCinto context chunks

Csplit←\[c1,c2,…,cT\],T=⌈\|C\|L⌉C\_\{\\text\{split\}\}\\leftarrow\[c\_\{1\},c\_\{2\},\\ldots,c\_\{T\}\],T=\\left\\lceil\\frac\{\|C\|\}\{L\}\\right\\rceil\.

for*t←1t\\leftarrow 1toTT*do

Head selection:

h⋆←arg⁡maxh′∈\[H\]⁡rselect​\(h′∣mt−1,q,ct\)h^\{\\star\}\\leftarrow\\arg\\max\_\{h^\{\\prime\}\\in\[H\]\}\\,r\_\{\\text\{select\}\}\(h^\{\\prime\}\\mid m\_\{t\-1\},q,c\_\{t\}\);

Head update:

mth⋆←LLM​\(mt−1,h⋆,q,ct\)m\_\{t\}^\{h^\{\\star\}\}\\leftarrow\\text\{LLM\}\(m\_\{t\-1\},\\,h^\{\\star\},\\,q,\\,c\_\{t\}\);

Memory Overwrite:

mt=\(mt−1∖\{mt−1h⋆\}\)∪\{mth⋆\}m\_\{t\}=\\left\(m\_\{t\-1\}\\setminus\\\{m\_\{t\-1\}^\{h^\{\\star\}\}\\\}\\right\)\\cup\\\{m\_\{t\}^\{h^\{\\star\}\}\\\}\.

Generate response

y←πθ​\(mT,q\)y\\leftarrow\\pi\_\{\\theta\}\(m\_\{T\},\\,q\);

return

yy;

Algorithm 1Multi\-Headed Memory For Long\-Context Query
#### A\.2MHM\-LRU: MHM Instantiation with Least Recently Updated Strategy

The prompt template of MHM\-LRU is shown in Figure[10](https://arxiv.org/html/2607.01523#A1.F10)\.

Prompt Template: Stage\-wise Memory Head Update and Final AnswerPrompt Template for Memory Update:You are presented with a problem, a section of an article that may contain the answer to the problem, and previous memory\. Please read the provided section carefully and update the memory\*\[memory ID\]\*with the new information that helps to answer the problem\. Be sure to retain all relevant details from the previous memory while adding any new, useful information\. Directly output the updated content without including the memory id\. Be sure to note the source context location of the updated memory\.<problem\>
\[Queryqq\]
</problem\><memory\_1\>
\[Memory Headmt−11m\_\{t\-1\}^\{1\}\]
</memory\_1\><memory\_2\>
\[Memory Headmt−12m\_\{t\-1\}^\{2\}\]
</memory\_2\><memory\_3\>
\[Memory Headmt−13m\_\{t\-1\}^\{3\}\]
</memory\_3\><memory\_4\>
\[Memory Headmt−14m\_\{t\-1\}^\{4\}\]
</memory\_4\>\}\\left\.\\rule\{0\.0pt\}\{80\.00012pt\}\\right\\\}\[Memory Head Setmt−1=\{mt−11,mt−12,mt−13,mt−14\}m\_\{t\-1\}=\\\{m\_\{t\-1\}^\{1\},m\_\{t\-1\}^\{2\},m\_\{t\-1\}^\{3\},m\_\{t\-1\}^\{4\}\\\}\]<section\>
\[Context Chunkctc\_\{t\}\]
</section\>Updated\[memory ID\]:
Prompt Template for Final Answer:You are presented with a problem and a previous memory containing multiple heads\. Please answer the problem based on the previous memory and format your response as follows "Therefore, the answer is \(insert answer here\)"\.<problem\>
\[Queryqq\]
</problem\>\[Memory Head SetmT=\{mT1,mT2,mT3,mT4\}m\_\{T\}=\\\{m\_\{T\}^\{1\},m\_\{T\}^\{2\},m\_\{T\}^\{3\},m\_\{T\}^\{4\}\\\}\]Your answer:Figure 10:LLM prompt template used in MHM\-LRU and MHM\-Relevance\. Italicized tokens in brackets denote runtime input parameters\. It should be noticed thatthe targeted memory head is selected and input as a parameter by the previous selection stage\.
#### A\.3MHM\-Concur: MHM Instantiation with Concurrent Selection And Update

MHM\-Concur is a vanilla instantiation of MHM\. The motivation of MHM\-Concur is that the memory agent should be capable of selecting and updating the memory head at the same time\. As shown in Figure[11](https://arxiv.org/html/2607.01523#A1.F11), in the memory update prompt template, the LLM is required to output the updated memory in a specific format that consists of the selected head ID and updated content\. The final answer prompt template is the same as in Figure[10](https://arxiv.org/html/2607.01523#A1.F10)\.

Prompt Template: Concurrent Memory Head UpdateYou are presented with a problem, a section of an article that may contain the answer to the problem, and previous memory\. Please read the provided section carefully and update the memory with the new information that helps to answer the problem\. Specifically, the memory consists of four memory heads:memory\_1, memory\_2, memory\_3, memory\_4\. Please select one of the memory heads and update the memory\. Be sure to diversify the use of different memory heads withinmemory\_1tomemory\_4and avoid repeatedly selecting the same memory head for updates\. Be sure to retain all relevant details from the previous memory while adding any new, useful information\. Be sure to format your update as follows: "<memory\_id\> \(your updated memory content\) </memory\_id\>"\.<problem\>
\[Queryqq\]
</problem\>\[Memory Head Setmt−1=\{mt−11,mt−12,mt−13,mt−14\}m\_\{t\-1\}=\\\{m\_\{t\-1\}^\{1\},m\_\{t\-1\}^\{2\},m\_\{t\-1\}^\{3\},m\_\{t\-1\}^\{4\}\\\}\]<section\>
\[Context Chunkctc\_\{t\}\]
</section\>Updated memory:Figure 11:LLM prompt template for memory update in MHM\-Concur\. Italicized tokens in brackets denote runtime input parameters\. Different from the stage\-wise paradigm, MHM\-Concur requires the LLM toselect and update memory heads at the same time\.
#### A\.4MHM\-Relevance: MHM Instantiation with Model\-Based Selection Strategy

MHM\-Relevance is characterized by the stage\-wise, model\-driven, semantic\-aware head selection — another important potential instantiation of MHM\. Specifically, the selection criteria is explicitly regulated as selecting the head that is the most semantically similar to the section, while prioritizing selecting empty memory heads to avoid single\-head memory pattern, as shown in Figure[12](https://arxiv.org/html/2607.01523#A1.F12)\. The consequential memory update and final answer prompt templates are the same as MHM\-LRU, as shown in Figure[10](https://arxiv.org/html/2607.01523#A1.F10)\.

Prompt Template: Model\-Based Relevance\-Driven SelectionYou are presented with a problem, a section of an article that may contain the answer to the problem, and previous memory\. Please read the provided section carefully and select a memory head inmemory\_1, memory\_2, memory\_3, memory\_4that is \*the most semantically similar to the section\*\. If there is an empty memory head, you should prioritize selecting it\. Be sure to diversify the use of different memory heads withinmemory\_1tomemory\_4and avoid repeatedly selecting the same memory head for updates\.<problem\>
\[Queryqq\]
</problem\><memory\_1\>
\[Memory Headmt−11m\_\{t\-1\}^\{1\}\]
</memory\_1\><memory\_2\>
\[Memory Headmt−12m\_\{t\-1\}^\{2\}\]
</memory\_2\><memory\_3\>
\[Memory Headmt−13m\_\{t\-1\}^\{3\}\]
</memory\_3\><memory\_4\>
\[Memory Headmt−14m\_\{t\-1\}^\{4\}\]
</memory\_4\><section\>
\[Context Chunkctc\_\{t\}\]
</section\>Selected memory:Figure 12:LLM prompt template for memory selection in MHM\-Relevance\. Italicized tokens in brackets denote runtime input parameters\.

### Appendix BDetailed Experiment Setting

#### B\.1Memory Setting

##### Memory Setting\.

The context chunk size is set to 5,000 tokens per chunk for long context splitting, consistent with the memory setting in\[yu2026memagent\]\. The total memory capacity is set to 4,096 tokens for all recurrent memory agents, which requires setting the token generation limit for each method according to its number of heads\.

- •MemAgent\(one single memory tag\): Themax\_new\_tokensis set to 4,096\.
- •ReMem\(one current memory and one recalled memory tags\): Themax\_new\_tokensis set to 2,048, ensuring that the total capacity of the two memory heads equals 4,096\.
- •MHM\-LRU: The number of heads is set to four, and themax\_new\_tokensis set to 1,024, ensuring that the total capacity of the four memory heads equals 4,096\. At the beginning, when multiple memory heads have the same highest age, MHM\-LRU chooses the lowest available head index by default\.
- •Ablation on \# of heads for MHM\-LRU: Themax\_new\_tokensis fixed to 1,024 regardless of the number of heads, ensuring that the agent’s performance is not affected by the change of token generation limit\.

#### B\.2Runtime Setting

##### Decoding Setting\.

In the LLM decoding setting, thetemperatureis set as 0\.7\. Thetop\_pis set as 0\.95\. The timeout limit is set as 60 seconds per 1,024 output tokens\. As a result, the timeout limit is 240 seconds for MemAgent, 120 seconds for ReMem, and 60 seconds for MHM\-LRU\. Each LLM calling has three tries in case of encountering an exception \(e\.g\., a timeout exception\)\. All experiments are run on a Linux server with 8×\\timesA100 GPUs\. LLMs are deployed with vLLM 0\.15\.0\.

##### Evaluation Setting\.

In evaluating the end\-to\-end performance \(accuracy\) of recurrent memory agents, we adopt substring exact matching\. In the model’s final response, we instruct the model to output the answer in the format “Therefore, the answer is \(insert answer here\)”\. The evaluation pipeline matches the output sentence starting with “Therefore, the answer is”, with the rest part is distinguished as the agent’s prediction\. The pipeline then executes substring exact matching between normalized ground truth and the agent’s prediction to decide the accuracy score of the agent’s final response\.

### Appendix CMemory Head Diversity Analysis

#### C\.1Experiment Setup

An inherent research question for the multi\-head recurrent memory agent is:Are memory heads differentiated or not?To answer this question, we propose a semantic distance\-based analysis of memory head diversity, demonstrating whether and to what extent memory heads semantically differ from one another\.

##### Evaluation Metric\.

We measure the diversity of memory heads using the average cosine distance for memory states at each time step\. Specifically, given a memory trajectory of a test item, denotemt=\{mt1,…,mtH\}m\_\{t\}=\\\{m\_\{t\}^\{1\},\\ldots,m\_\{t\}^\{H\}\\\}as the memory state of MHM at time steptt\. We first use a sentence transformer \(all\-mpnet\-base\-v2in the experiment\) to obtain the semantic vector of each memory head, i\.e\.,

vth=SentenceTransformer​\(mth\),h∈\[H\]\.v\_\{t\}^\{h\}=\\text\{SentenceTransformer\}\(m\_\{t\}^\{h\}\),h\\in\[H\]\.\(7\)
Next, we calculate the average memory vectorv¯t=1H​∑h=1Hvth\\overline\{v\}\_\{t\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}v\_\{t\}^\{h\}\. We define the cosine distance between each memory head and the averaged memory by:

dcos​\(vth,v¯t\)=1−cos⁡\(vth,v¯t\),d\_\{\\cos\}\(v\_\{t\}^\{h\},\\overline\{v\}\_\{t\}\)=1\-\\cos\(v\_\{t\}^\{h\},\\overline\{v\}\_\{t\}\),\(8\)
Finally, theaveraged cosine distanced¯t\\overline\{d\}\_\{t\}is defined by:

d¯t=1H​∑h=1Hdcos​\(vth,v¯t\)\\overline\{d\}\_\{t\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}d\_\{\\cos\}\(v\_\{t\}^\{h\},\\overline\{v\}\_\{t\}\)\(9\)
Thed¯t\\overline\{d\}\_\{t\}is non\-negative and measures the diversity of memory heads at time steptt\. Whend¯t=0\\overline\{d\}\_\{t\}=0, all memory heads are semantically identical\. The higher thed¯t\\overline\{d\}\_\{t\}, the more diverse the memory heads are\. For each memory trajectory, we visualized¯t\\overline\{d\}\_\{t\}with respect to time stepttto see how memory diversity evolves as time goes by\.

#### C\.2Experiment Results

We evaluate memory diversity on three representative tasks in BABILong: the QA1 \(1\-supporting fact\), QA4 \(2\-arg relations\) and QA7 \(Counting\), choosing five trajectories with continuous item IDs for visualization for each task\. Results are shown in Figure[13](https://arxiv.org/html/2607.01523#A3.F13)\.

##### MHM Enables Diverse Memory Allocation\.

Figure[13](https://arxiv.org/html/2607.01523#A3.F13)shows that MHM\-LRU consistently produces non\-trivial semantic diversity across memory heads throughout processing\. Across all three tasks,d¯t\\overline\{d\}\_\{t\}is positive at most time steps, confirming that heads spontaneously specialize without any explicit diversity objective\. Diversity is highest in the initial steps when each head is first written, then gradually decreases and stabilizes at a lower plateau as MHM\-LRU propagates task\-relevant information redundantly across heads—the same emergent backup behavior documented in Section[5\.5](https://arxiv.org/html/2607.01523#S5.SS5)\. This trajectory reflects a natural balance between early\-stage coverage and late\-stage retention robustness\.

##### Diversity Varies In Different Tasks\.

The magnitude and duration of head diversity are modulated by task complexity\. For QA1 \(1\-supporting fact\),d¯t\\overline\{d\}\_\{t\}collapses rapidly once the single target fact is captured, with heads converging to near\-identical content by step 20–40\. QA4 \(2\-arg relations\) sustains higher diversity for longer\. QA7 \(Counting\) exhibits the most persistent and fluctuating diversity, consistent with the incremental nature of counting where each new chunk may modify the running tally\. This pattern aligns with the retention gains in Section[5\.4](https://arxiv.org/html/2607.01523#S5.SS4): tasks requiring multi\-step information accumulation benefit most from heads that remain semantically differentiated over longer processing horizons\.

![Refer to caption](https://arxiv.org/html/2607.01523v1/x8.png)\(a\)QA1: 1\-supporting fact\.
![Refer to caption](https://arxiv.org/html/2607.01523v1/x9.png)\(b\)QA4: 2\-arg relations\.
![Refer to caption](https://arxiv.org/html/2607.01523v1/x10.png)\(c\)QA7: Counting\.

Figure 13:Average memory head distance \(d¯t\\overline\{d\}\_\{t\}\) with respect to time steps\. The higher thed¯t\\overline\{d\}\_\{t\}, the more semantically diverse the memory heads in the time step\.

### Appendix DBroader Impact

In this section, we discuss the broader impact of MHM not only on recurrent memory for long\-context query, but more general research field of agentic memory in the future\.

##### Towards Long\-term and Heterogeneous Memory\.

Despite the success of MHM in single\-turn long\-context query, the current research focus of recurrent memory is still limited by query\-specific, short\-term memory\. An inevitable following research question is: what a long\-term memory should be, and how to maintain it\. Long\-term memory is valuable in the whole life cycle of an agent, containing a variety of heterogeneous memory information, and is query\-agnostic\. The multi\-head memory has the potential to achieve this: by allowing each memory head to specialize in storing a distinct type of information, such as episodic experience, declarative knowledge, procedural skills, or relational context, MHM provides a natural architectural substrate for heterogeneous long\-term memory\. Unlike the monolithic single\-head design, which is forced to compress all memory types into a single undifferentiated representation, the multi\-head structure admits a principled partition of memory responsibilities across heads\. This partitioning can be further guided by semantic\-aware or type\-aware selection strategies, enabling the agent to organize, retrieve, and update memory content in a structured and persistent manner across its entire operational lifetime\.

##### Towards Self\-Evolving Agent\.

We believe that a must\-go path for memory agent is to achieve a realistic self\-evolving agent—the agent can learn experience from past failure or success, and accumulate knowledge from observation and simulation\. The multi\-head memory essentially provides a technical path towards the self\-evolving agent: by endowing agents with the ability to spontaneously and freely manage and utilize multiple memory heads, the agent can achieve self\-evolvement of its experience through self\-evolvement of memory internalization\. In the long run, we envision that MHM\-style architectures, when coupled with continual learning and reflection mechanisms, could enable agents to autonomously consolidate short\-term observations into long\-term behavioral priors, progressively refine their internal representations in response to environmental feedback, and ultimately exhibit sustained improvement over time without human intervention, which is a foundational step toward truly autonomous and adaptive AI systems\.