Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
Summary
Introduces Belief Entropy and Metacognitive Memory Policy Optimization (MMPO) to improve memory quality in long-horizon LLM agents, outperforming existing methods and maintaining performance over long contexts.
View Cached Full Text
Cached at: 06/05/26, 06:07 AM
Paper page - Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
Source: https://huggingface.co/papers/2605.30159 Published on May 28
·
Submitted byhttps://huggingface.co/ziyan2003
liuon Jun 5
Abstract
Memory-augmented language models struggle with long-horizon tasks due to information loss in recursive summaries, but a new method using belief entropy and metacognitive policy optimization improves performance by focusing on memory quality rather than just outcome success.
Memory-augmented LLM agentstackle complexlong-horizon tasksby recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-basedreinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbatesbelief deviation, obscuring the agent’s estimate of thelatent task stateand ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduceBelief Entropy, a self-supervised proxy that probes how uncertain the model remains about thelatent task stategiven its current memory. Based on this proxy, we proposeMetacognitive Memory Policy Optimization(MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce highepistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverselong-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.30159
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.30159 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.30159 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.30159 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Belief Memory: Agent Memory Under Partial Observability
This paper introduces BeliefMem, a novel memory paradigm for LLM agents that stores multiple candidate conclusions with probabilities to handle partial observability and reduce self-reinforcing errors. Empirical evaluations show it outperforms deterministic baselines on LoCoMo and ALFWorld benchmarks.
Learning What to Remember: Observability-Safe Memory Retention via Constrained Optimization for Long-Horizon Language Agents
This paper formulates memory retention for long-horizon language agents as a constrained stochastic optimization problem, introducing OSL-MR, a framework that enforces observability-safe learning with a Mixed-Score heuristic. Experiments show consistent improvements over existing heuristic baselines under tight memory budgets.
MemGym: a Long-Horizon Memory Environment for LLM Agents
MemGym is a benchmark for evaluating memory formation in LLM agents over long-horizon tasks, unifying existing agent gyms and synthetic pipelines with memory-isolated scores. It spans tool-use dialogue, multi-turn search, coding, and computer use, and includes a lightweight reward model (MemRM) for efficient evaluation.
Retrieval-Driven Memory Reconsolidation for Long-Term LLM Agents
The paper introduces REALM, a framework for long-term memory in LLM agents that uses retrieval-driven reconsolidation to autonomously organize memories into a cognitive graph, achieving improved performance on long-term memory benchmarks.
Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents
Memory-R2 introduces LoGo-GRPO, a training framework that combines local and global group-relative optimization to provide fairer credit assignment for long-horizon memory-augmented LLM agents, improving accuracy and inference latency across backbones.