MemDefrag: Latent Memory Defragmentation for Large Language Models
Summary
Proposes MemDefrag, a training-free framework that uses a middle-layer tracing signal to defragment latent memory in LLMs, achieving significantly better knowledge retention than existing methods like MemoryLLM and M+.
View Cached Full Text
Cached at: 07/08/26, 04:42 AM
# MemDefrag: Latent Memory Defragmentation for Large Language Models Source: [https://arxiv.org/abs/2607.05969](https://arxiv.org/abs/2607.05969) [View PDF](https://arxiv.org/pdf/2607.05969) > Abstract:Latent memory, which stores past knowledge fragments as per\-layer hidden states, has emerged as a promising paradigm \(e\.g\., MemoryLLM and M\+\) for long\-term memory in large language models \(LLMs\)\. However, the paradigm suffers from significant performance degradation during memory updates, due to positional encoding misalignment and the absence of any tracing mechanism to distinguish target memory fragments from irrelevant ones\. To discover such a tracing mechanism, we probe the layer\-wise attention density over stored memory fragments, and find that a small set of middle transformer layers consistently concentrates the highest density on the target fragment \- exposing an inherent tracing signal\. In light of this, we propose MemDefrag, a training\-free and model\-agnostic framework that \(1\) uses a middle\-layer tracing signal to conduct memory defragmentation \(rank, reorder, and filter memories\), and \(2\) applies an informativeness\-guided proportional forgetting mechanism once capacity is exceeded\. Experiments show that MemDefrag substantially outperforms MemoryLLM and M\+ on knowledge retention \(e\.g\., 43\.0% vs\. 17\.4%/17\.6% after 50 memory updates\) and long\-context benchmarks, and generalizes well across various LLMs and latent\-memory variants\. ## Submission history From: Ruiyi Yan \[[view email](https://arxiv.org/show-email/1b90dbb8/2607.05969)\] **\[v1\]**Tue, 7 Jul 2026 08:04:34 UTC \(763 KB\)
Similar Articles
Δ-Mem: Efficient Online Memory for Large Language Models
Proposes delta-Mem, a lightweight online memory mechanism that uses a compact state matrix updated by delta-rule learning to improve long-context performance of frozen LLMs without full fine-tuning or context extension.
δ-mem: Efficient Online Memory for Large Language Models
The paper introduces δ-mem, a lightweight memory mechanism that enhances large language models by augmenting a frozen attention backbone with a compact associative memory state. It demonstrates improved performance on memory-heavy benchmarks with minimal computational overhead.
HasMem: Hard-Origin Adaptively Softened Memory for Long-Term LLM Agents
The paper proposes HasMem, an adaptive memory compression method for long-term LLM agents that adjusts memory widths while maintaining frozen LLM parameters, demonstrating improved performance on reconstruction and QA benchmarks.
@omarsar0: // Latent Semantic Memory for LLMs // This paper presents some really interesting findings about agent memory. They fin…
The paper introduces Memorilla, a memory system that compresses an entire document collection into 16 query-dependent memory tokens for frozen LLMs, outperforming full-context inference on FactKG (84.33% vs 80.43%) while using 108x fewer tokens and beating baselines like Mem0 and MemoRAG across five of eight benchmarks.
Reducing LLM Latency
Techniques and methods for reducing latency in large language models, improving inference speed.