Stacking fancier “memory” architectures on LLM agents does not necessarily yield better results. A new paper empirically tested 12 memory systems and found no universal winner.

It decomposes agent memory like a database — breaking it into four modules: representation and storage, extraction, retrieval and routing, and maintenance. They benchmarked 12 systems, including Mem0, Letta, Zep, Cognee, MemOS, MemTree, A-MEM, LightMem, plus two baselines (Long Context and Embedding RAG), across 5 workload types and 11 datasets.

Several counterintuitive findings:

1. On DB-Bench, a database operation task, bare Long Context (48.20 EM) and the simplest MemoChat outperform all fancy memory systems. Strong memory depends on how well it aligns with the dominant bottleneck, not how fancy the representation is.

2. The key to retrieval is how evidence is organized for later reconstruction, not just ranking the most relevant item first. When evidence is far away, graph/hierarchical structures crush flat ones: A-MEM achieves 85.9 Recall@10, while Embedding RAG’s Answer-F1 drops sharply from 37.1 to 7.4.

3. Cost is determined by “maintenance scope,” not the structure itself. Local maintenance far outperforms global reorganization. On LongBench, LightMem stays at 17.3 seconds, while Mem0/MemoChat/MemoryOS/A-MEM, which do global coordination, spike to 374~552 seconds — a 20-30x latency difference.

4. Don’t compress prematurely: preserving the original text beats summarization; compressing drops Substring-EM from 26.0 to 10.7. Extraction should preserve context (MemOS Fast 25.5 EM vs Fine 2.5). Maintenance should be conservative; aggressive delayed updating actually lowers scores.

There is also a common flaw called “hallucinations of the past”: after facts are updated, the system still returns old values. That’s why the paper title asks: are we really ready for agent-native memory systems?

Are We Ready For An Agent-Native Memory System?

Source: https://arxiv.org/html/2606.24775

Abstract.

Memory for large language model (LLM) agents has rapidly evolved from simple retrieval-augmented mechanisms into a data management system that supports persistent information storage, retrieval, update, consolidation, and dynamic lifecycle governance throughout agent execution. Despite this evolution, existing evaluations still benchmark agent memory mainly through end-to-end task success metrics (e.g., F1, BLEU), while treating the underlying system as a monolithic black box. As a result, critical system-level concerns, including operational costs, architectural trade-offs across memory modules, and robustness under dynamic knowledge updates, remain insufficiently explored.

In this paper, we present a systematic experimental study of agent memory from a data management perspective. We propose an analytical framework that decomposes agent memory into four core modules: memory representation and storage, extraction, retrieval and routing, and maintenance. Under this framework, we evaluate 12 representative memory systems and two reference baselines across five benchmark workloads spanning 11 datasets. Our extensive end-to-end evaluation shows that no single architecture dominates across all scenarios; instead, effectiveness depends heavily on how well the memory structure aligns with the workload bottleneck. Furthermore, through fine-grained ablation studies, we quantify their individual effects on representation fidelity, retrieval precision, update correctness, and long-horizon stability. Finally, we reveal cost-performance trade-offs under realistic workloads, showing localized maintenance is more cost-efficient than global reorganization. Based on these findings, we identify promising directions towards building truly agent-native memory systems. The code is publicly available athttps://github.com/OpenDataBox/MemoryData.

††copyright:none## 1.Introduction

The rapid evolution of Large Language Model (LLM) agents has sparked a large body of exciting research and industrial efforts in building agent memory, i.e., the data management system of the LLM agent that supports long-horizon stateful execution and personalized interaction(Luoet al.,2026 (https://arxiv.org/html/2606.24775#bib.bib225); Khanet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib226); Liuet al.,2026b (https://arxiv.org/html/2606.24775#bib.bib227); Singhet al.,2024 (https://arxiv.org/html/2606.24775#bib.bib253); OpenAI,2026 (https://arxiv.org/html/2606.24775#bib.bib243); Microsoft,2025 (https://arxiv.org/html/2606.24775#bib.bib246); Google,2025 (https://arxiv.org/html/2606.24775#bib.bib247)).

As shown in Figure1 (https://arxiv.org/html/2606.24775#S1.F1), existing agent memory systems span a diverse set of architectural designs.(1) Stream-and-Reflection Memory System(e.g., MemoryBank(Zhonget al.,2024 (https://arxiv.org/html/2606.24775#bib.bib255))) maintains experiences as timestamped memory streams and periodically summarizes them into higher-level reflections that are written back into the stream;(2) Hierarchical Tiered Memory System(e.g., MemGPT(Packeret al.,2023 (https://arxiv.org/html/2606.24775#bib.bib230))) organizes memory into multiple levels with different capacities and access properties, separating core memory from archival storage with explicit movement (e.g., eviction and promotion) across tiers;(3) Knowledge Graph Memory System(e.g.,Mem0g\text{Mem0}^{g}(Chhikaraet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib231)), Zep(Rasmussenet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib233))) represents entities, relations, and their temporal evolution in structured forms (e.g., temporal knowledge graphs), often incorporating entity disambiguation and conflict resolution;(4) Composite Hybrid Memory System(e.g., A-MEM(Xu and others,2025 (https://arxiv.org/html/2606.24775#bib.bib232))) routes schema-aware memory objects across multiple storage substrates, explicitly separating runtime state (e.g., KV caches) from long-term storage (e.g., vector, graph, keyword indexes), managed by dedicated maintenance modules. However, this rapid proliferation has also led to a highly fragmented landscape that lacks systematic evaluation from a data management perspective, raising a natural question:Are we ready for an agent-native memory system?

Refer to captionFigure 1.Typical Execution Workflows of Agent Memory.In this paper, we revisit this question for agent memory. In particular, we focus onsystem-level memoryover textual, structured, and even parametric representations(Chhikaraet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib231); Packeret al.,2023 (https://arxiv.org/html/2606.24775#bib.bib230); Rasmussenet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib233); Xu and others,2025 (https://arxiv.org/html/2606.24775#bib.bib232)), a fundamental infrastructure component for modern autonomous agents. We focus on memory-centric systems, rather than task-specific agent frameworks where memory is an auxiliary module(1 (https://arxiv.org/html/2606.24775#bib.bib269);W. Zhou, X. Zhou, Q. He, G. Li, B. He, Q. Xu, and F. Wu (2026) (https://arxiv.org/html/2606.24775#bib.bib222)). It is the persistent data management system that maintains information beyond a single inference step (e.g., historical interactions, environmental observations, and intermediate tool executions) decoupled from the LLMs’ parametric weights and volatile context windows. Agent frameworks rely on these external memory systems (e.g., Mem0(Chhikaraet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib231)), Letta(Packeret al.,2023 (https://arxiv.org/html/2606.24775#bib.bib230)), Zep(Rasmussenet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib233)), and A-MEM(Xu and others,2025 (https://arxiv.org/html/2606.24775#bib.bib232))) to actively write, update, index, and route relevant context back into the reasoning loop. The capability of a long-horizon agent largely depends on the reliability and efficiency of this memory layer. An agent adopting a poorly designed memory architecture can suffer from factual contradictions, catastrophic forgetting, or unacceptable latencies during continuous execution(Du,2026 (https://arxiv.org/html/2606.24775#bib.bib258); Zhenget al.,2025b (https://arxiv.org/html/2606.24775#bib.bib257)).

Recent benchmarks(Maharanaet al.,2024 (https://arxiv.org/html/2606.24775#bib.bib240); Wuet al.,2024 (https://arxiv.org/html/2606.24775#bib.bib241); MemoryAgentBench Team,2026 (https://arxiv.org/html/2606.24775#bib.bib242); Tanet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib256))have evaluated agent memory and shown that external memory can improve agent performance on tasks requiring factual recall and long-context understanding. However, these evaluations are largely rooted in natural language processing and have multiple limitations when treating agent memory as a data management system (see Section2 (https://arxiv.org/html/2606.24775#S2)). First, they fail to evaluate many representative memory architectures (e.g., systems such as MemoChat, MemTree, and LightMem have not been included in prior evaluations) under unified workloads, making principled cross-system comparisons difficult. Efforts from the database community(Wuet al.,2026 (https://arxiv.org/html/2606.24775#bib.bib259))limit their scope to a few chatbot-centric datasets (e.g., LoCoMo and LongMemEval only), neglecting complex agentic execution scenarios. Second, existing benchmarks predominantly rely on single-sided, end-to-end task success metrics (e.g., F1 and BLEU scores) rather than a comprehensive evaluation suite. They fail to explicitly isolate and measure multi-dimensional performance indicators such as evidence-level retrieval fidelity, dynamic update robustness under conflicting knowledge, and long-horizon stability. Third, they rarely measure key operational costs from a systems perspective, such as index construction time and query latency, which are critical for production deployments. Last, they treat memory systems as monolithic black boxes rather than decomposing them into fundamental data management modules for isolated, fine-grained analysis.

We overcome these limitations and conduct comprehensive experiments and analyses from a data management perspective. The contributions are as follows.

(1) Technology Decomposition and Taxonomy (Section3 (https://arxiv.org/html/2606.24775#S3)).We decompose existing agent memory systems into four core components: (i) memory representation and storage, (ii) memory extraction, (iii) memory retrieval and routing, and (iv) memory maintenance. For each component, we further establish a structured taxonomy111https://github.com/OpenDataBox/awesome-agent-memoryby categorizing existing approaches according to their underlying design principles, enabling principled comparisons.

(2) Overall End-to-End Performance Evaluation (Section4 (https://arxiv.org/html/2606.24775#S4)).We conduct end-to-end evaluations under a unified and fair testbed (e.g., unified time-overhead traces)222https://github.com/OpenDataBox/MemoryDataacross five distinct benchmark workloads encompassing 11 datasets. Our study includes 12 representative memory systems, each embodying different combinations of representation, storage, routing, and maintenance strategies. We evaluate their performance from five perspectives: task effectiveness (RQ1), retrieval fidelity (RQ2), dynamic update robustness (RQ3), long-horizon stability (RQ4), and operational cost (RQ5).

(3) Fine-Grained Technical Component Evaluation (Section5 (https://arxiv.org/html/2606.24775#S5)).Leveraging our four-module framework, we conduct controlled and fine-grained experiments on representative strategies within each technique component. By systematically generating controlled variants that modify one module at a time, we quantify their performance trade-offs and assess their individual impacts on representation fidelity, routing precision, and update correctness.

(4) Insightful Findings.Based on our experimental results and in-depth analysis, we distill a set of insightful findings regarding the cost–performance trade-offs of agent memory systems:

❶Are Memory Systems Effective Across Different Agent Request Workloads?No single memory architecture dominates all scenarios. Composite hybrid systems lead on conversational QA, while graph-based methods excel in single-hop factual recall but struggle with temporal reasoning. Moreover, effective memory systems remain robust acrossLLMbackbone variants because they externalize evidence localization before answer generation.

❷How Accurately Do Memory Systems Retrieve Stored Evidence?Explicit query planning and balanced hybrid search maximize contextual relevance. However, retrieval accuracy degrades significantly as the temporal distance between the evidence and the query increases, exposing limitations of similarity-based retrieval.

❸Are Memory Systems Robust Under Dynamic Updates?Graph-based methods handle knowledge updates most reliably, whereas popular fact-extraction plugins and append-only stores struggle with targeted overwrites. Systems lacking lifecycle management return stale facts, leading to “hallucinations of the past”.

❹Do Memory Systems Remain Stable Over Long Horizons?Many append-only memory stores suffer from catastrophic degradation as evidence becomes more distant. For time-dependent queries, raw long-context retrieval still outperforms most memory-backed approaches, indicating that standard semantic consolidation often destroys crucial chronological cues.

❺What Are the Operational Costs of Agent Memory?Highly structured systems incur orders-of-magnitude higher index construction time and query latency than lightweight stores, yet do not consistently deliver proportional accuracy gains.

❻When Do Individual Memory Components Go Wrong?Each layer of abstraction (e.g., compression, summarization, and fact extraction) progressively discards information. Furthermore, fine-grainedLLM-based extraction can yield modest precision gains but substantially degrade multi-hop reasoning. Finally, conservative memory consolidation serves as the best default maintenance strategy, whereas delayed flushing creates a deceptive trade-off between surface-level coverage and actual answerability.

2.Preliminaries

To support the discussion in the rest of this paper, we first clarify the scope ofagent memoryfrom a data management perspective. Although recent studies have examined memory from viewpoints such as cognitive taxonomy, agent architecture, and graph-based organization(Zhanget al.,2025 (https://arxiv.org/html/2606.24775#bib.bib260); Du,2026 (https://arxiv.org/html/2606.24775#bib.bib258); Huet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib238); Wuet al.,2026 (https://arxiv.org/html/2606.24775#bib.bib259); Tang and others,2026 (https://arxiv.org/html/2606.24775#bib.bib261); Yanget al.,2026 (https://arxiv.org/html/2606.24775#bib.bib239)), the underlying concept is still often treated primarily as an algorithmic component of the LLM or agent pipeline(Zhanget al.,2025 (https://arxiv.org/html/2606.24775#bib.bib260); Du,2026 (https://arxiv.org/html/2606.24775#bib.bib258); Huet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib238)). In contrast, we study agent memory as a standalone data management object and system infrastructure, with explicit attention to how it is represented, stored, retrieved, updated, and maintained under real agent workloads. Under this view, we introduce a set of definitions below.

Table 1.Taxonomy and Characteristics of Agent Memory Systems.\rowcolor[HTML]1F1F1F\cellcolor[HTML]1F1F1F\cellcolor[HTML]1F1F1F\cellcolor[HTML]1F1F1FMemory Representation & Storage\cellcolor[HTML]1F1F1F\cellcolor[HTML]1F1F1F\cellcolor[HTML]1F1F1F\rowcolor[HTML]1F1F1F\cellcolor[HTML]1F1F1FCategory\cellcolor[HTML]1F1F1FMethod\cellcolor[HTML]1F1F1FRepresentation\cellcolor[HTML]1F1F1FStorage\cellcolor[HTML]1F1F1FMemory Extraction\cellcolor[HTML]1F1F1FMemory Retrieval &Query Routing\cellcolor[HTML]1F1F1FMemory MaintenanceMemoChat(Luet al.,2023 (https://arxiv.org/html/2606.24775#bib.bib4))❶ Token-Level Sequence(Structured JSON Memos)❶ Transient In-Context Registers❸ Schema-Constrained Extraction(LLM Topic Segmentation)❹ Autonomous Agentic Routing(LLM Topic Selection)❸ LLM-Driven Semantic Consolidation(Turn-Triggered)Mem0(Chhikaraet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib231))❶ Token-Level Sequence(Discrete Facts)❷ Specialized Single-Engine(Vector DB)❷ Schema-Free Extraction❷ Semantic-Based Retrieval❸ LLM-Driven Semantic Consolidation(Tool-Calling)MEM1(Zhouet al.,2025 (https://arxiv.org/html/2606.24775#bib.bib266))❶ Token-Level Sequence❶ Transient In-Context Registers❶