Agent memory layers don't need an LLM deciding what to remember

Reddit r/LocalLLaMA Tools
agent-memory llm-extraction retrieval embedding sqlite open-source

Summary

The author argues that agent memory layers should skip LLM-based extraction for deciding what to remember, instead using simple storage, embeddings, and retrieval, exemplified by their open-source memU tool.

Most agent memory setups run a model call on the way in. Something reads the turn, decides whether it's worth keeping, rewrites it into a "memory", tags it with a type and an importance score. That's a second inference pass on every interaction and I think it's the wrong place to spend one. Cost isn't the issue. The judgment being unauditable is. When the agent fails to recall something you can't tell whether retrieval missed it or the extractor decided six days ago it wasn't worth keeping. Two different bugs, same symptom, and separating them means reading logs that mostly don't exist. Take that stage out and you have storage, an embedding, retrieval. Which is what a memory layer is. I run memU partly for this reason. Core is around 500 lines doing those three things, exposed as commit, list, retrieve. What it keeps is readable Markdown, embedded and indexed into a local sqlite db. Apache-2.0. You lose the distillation, obviously. Raw turns are noisier than summarised ones so retrieval has to be better to compensate. I'll take that trade because a noisy store you can read beats a tidy one you can't inspect, but I'd take the other side seriously from anyone who has actually measured it. Being straight about the cost since someone will find it anyway: self-hosting does not get you off the hook for an embedding provider, you still need a key for that. It's single machine too, syncing across boxes is their hosted mode. And recall quality is entirely downstream of whichever embedder you point it at. With a small one the recalls go vague on anything abstract. Facts and procedures come back fine. A preference I mentioned once in passing, much less reliably. Anyone still running the extraction step in their harness and finding it earns its keep? Would like to know what it's buying that retrieval isn't.
Original Article

Similar Articles

Human-Inspired Memory Architecture for LLM Agents

arXiv cs.AI

Microsoft researchers propose a biologically-inspired memory architecture for LLM agents that incorporates mechanisms like sleep-phase consolidation and interference-based forgetting to manage persistent memory efficiently.

Agent memory is not just RAG over user facts

Reddit r/AI_Agents

The article argues that simple RAG-based agent memory systems fail in production due to issues like stale preferences, missed keywords, and prompt injection, and advocates for a layered memory architecture with active selection, deterministic fallback, governance, and testing.

@chenchengpro: The more fancy "memory" architectures you stack on an LLM Agent, the better the results? Not necessarily. A new paper tested 12 memory systems and found no universal winner. It decomposes Agent memory like a database — representation & storage, extraction, retrieval & routing, and maintenance — and tested Mem0, Letta, Zep, C…

X AI KOLs Timeline

A paper systematically evaluates 12 LLM Agent memory systems, breaks them into four modules, finds no single architecture dominates all scenarios, and reveals cost-performance trade-offs and common issues (e.g., 'past hallucinations').

From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms

Hugging Face Daily Papers

This survey paper proposes an evolutionary framework for LLM agent memory mechanisms, categorizing their development into three stages: storage, reflection, and experience. It analyzes core drivers such as long-range consistency and continual learning to provide design principles for next-generation agents.