Every agent memory system is benchmarked on recall. Almost none check whether the memory is still true
Summary
Argues that agent memory systems are evaluated on recall but not on whether recalled facts remain true; introduces a 'correctness under staleness' benchmark showing vector stores return stale facts 68% of the time.
Similar Articles
does anyone else's agent confidently "remember" stuff that's already changed?
A user describes an issue where AI agents confidently retrieve outdated facts from memory layers without flagging changes, and asks the community for solutions to invalidate old memories or track fact freshness.
STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
This paper identifies a critical failure mode in LLM agents where they fail to update personalized memories when new evidence conflicts with prior beliefs. It introduces the STALE benchmark and a three-dimensional probing framework, revealing that even the best models achieve only 55.2% accuracy, and proposes CUPMem as a prototype for robust memory revision.
Agents don't need memory, they need documentation
An argument that current AI agent memory plugins all boil down to flawed RAG-based recall, and that agents would be better served by explicit, document-based knowledge rather than vector-store snippet retrieval.
What Should an Agent Remember? Disentangling Retention from Retrieval in Bounded-Memory Evaluation
This paper builds a streaming-recall benchmark that separates an agent's query-independent retention decision from its query-aware selection decision, showing that much of a prior evaluation's reported memory advantage was actually an access-regime artifact and that recall under bounded memory is upper-bounded by retention rate.
How do you find out afterwards whether your agent's memory was right?
The author discusses the challenge of verifying the correctness of AI agents' long-term memory and asks for community experiences on tracking and correcting memory errors.