MemDefrag: Latent Memory Defragmentation for Large Language Models

arXiv cs.CL Papers
llm memory-management latent-memory defragmentation training-free model-agnostic

Summary

Proposes MemDefrag, a training-free framework that uses a middle-layer tracing signal to defragment latent memory in LLMs, achieving significantly better knowledge retention than existing methods like MemoryLLM and M+.

arXiv:2607.05969v1 Announce Type: new Abstract: Latent memory, which stores past knowledge fragments as per-layer hidden states, has emerged as a promising paradigm (e.g., MemoryLLM and M+) for long-term memory in large language models (LLMs). However, the paradigm suffers from significant performance degradation during memory updates, due to positional encoding misalignment and the absence of any tracing mechanism to distinguish target memory fragments from irrelevant ones. To discover such a tracing mechanism, we probe the layer-wise attention density over stored memory fragments, and find that a small set of middle transformer layers consistently concentrates the highest density on the target fragment - exposing an inherent tracing signal. In light of this, we propose MemDefrag, a training-free and model-agnostic framework that (1) uses a middle-layer tracing signal to conduct memory defragmentation (rank, reorder, and filter memories), and (2) applies an informativeness-guided proportional forgetting mechanism once capacity is exceeded. Experiments show that MemDefrag substantially outperforms MemoryLLM and M+ on knowledge retention (e.g., 43.0% vs. 17.4%/17.6% after 50 memory updates) and long-context benchmarks, and generalizes well across various LLMs and latent-memory variants.
Original Article
View Cached Full Text

Cached at: 07/08/26, 04:42 AM 

# MemDefrag: Latent Memory Defragmentation for Large Language Models
Source: [https://arxiv.org/abs/2607.05969](https://arxiv.org/abs/2607.05969)
[View PDF](https://arxiv.org/pdf/2607.05969)

> Abstract:Latent memory, which stores past knowledge fragments as per\-layer hidden states, has emerged as a promising paradigm \(e\.g\., MemoryLLM and M\+\) for long\-term memory in large language models \(LLMs\)\. However, the paradigm suffers from significant performance degradation during memory updates, due to positional encoding misalignment and the absence of any tracing mechanism to distinguish target memory fragments from irrelevant ones\. To discover such a tracing mechanism, we probe the layer\-wise attention density over stored memory fragments, and find that a small set of middle transformer layers consistently concentrates the highest density on the target fragment \- exposing an inherent tracing signal\. In light of this, we propose MemDefrag, a training\-free and model\-agnostic framework that \(1\) uses a middle\-layer tracing signal to conduct memory defragmentation \(rank, reorder, and filter memories\), and \(2\) applies an informativeness\-guided proportional forgetting mechanism once capacity is exceeded\. Experiments show that MemDefrag substantially outperforms MemoryLLM and M\+ on knowledge retention \(e\.g\., 43\.0% vs\. 17\.4%/17\.6% after 50 memory updates\) and long\-context benchmarks, and generalizes well across various LLMs and latent\-memory variants\.

## Submission history

From: Ruiyi Yan \[[view email](https://arxiv.org/show-email/1b90dbb8/2607.05969)\] **\[v1\]**Tue, 7 Jul 2026 08:04:34 UTC \(763 KB\)

Similar Articles

Δ-Mem: Efficient Online Memory for Large Language Models

Hacker News Top

Proposes delta-Mem, a lightweight online memory mechanism that uses a compact state matrix updated by delta-rule learning to improve long-context performance of frozen LLMs without full fine-tuning or context extension.

δ-mem: Efficient Online Memory for Large Language Models

Hugging Face Daily Papers

The paper introduces δ-mem, a lightweight memory mechanism that enhances large language models by augmenting a frozen attention backbone with a compact associative memory state. It demonstrates improved performance on memory-heavy benchmarks with minimal computational overhead.

Reducing LLM Latency

Reddit r/AI_Agents

Techniques and methods for reducing latency in large language models, improving inference speed.

StageMem: Lifecycle-Managed Memory for Language Models

arXiv cs.CL

StageMem proposes a lifecycle-managed memory framework for language models that organizes memory into transient, working, and durable stages with explicit confidence and strength metrics, treating memory as a stateful process rather than a static store to better manage retention and forgetting under bounded capacity.

SimpleMem: Efficient Lifelong Memory for LLM Agents

Papers with Code Trending

Introduces SimpleMem, an efficient memory framework for LLM agents that uses semantic lossless compression to improve accuracy and reduce token consumption, achieving 26.4% F1 improvement and up to 30x reduction in inference-time token usage.