Every agent memory system is benchmarked on recall. Almost none check whether the memory is still true

Reddit r/AI_Agents Tools

Summary

Argues that agent memory systems are evaluated on recall but not on whether recalled facts remain true; introduces a 'correctness under staleness' benchmark showing vector stores return stale facts 68% of the time.

Recall is the metric every agent memory system reports: hit rate, token savings, context recovered. None of it tells you the thing that actually breaks agents in production: whether the fact it just recalled is still true. A memory that returns a fact that was true once and is wrong now is worse than no memory, because the agent acts on it with full confidence. Say you stored "prod database host is db-prod-03" six months ago. Infra moved. The store still recalls db-prod-03, the agent runs a migration against a box that no longer exists, and nothing in the recall step knew the fact had expired. So we built a small benchmark for one thing, correctness under staleness: whether memory returns the current truth after the ground truth changes. Method, to reproduce: Freeze a task set of facts (service hosts, versions, regions, owners). Seed each into memory as a statement. Mutate the ground truth over time, feeding each change as a new observation. Query for the current value, and score it against the current source. 50 facts, 40 of them changed at least once. On those 40: memory pattern returns current truth returns a stale value append-only vector store (top-1 by similarity) 32% 68% plain key-value store, overwrite per entity 100% 0% The vector store retrieved the right entity every time. It just had no notion that a newer fact replaced the old one, so similarity ranking handed back the outdated version two times out of three. Recency-weighting the retrieval cut the stale hits but started returning the newest fact about the wrong service, so it traded one error for another. Scope that recency to the entity and you have rebuilt the overwrite store. The differentiator was never the store. It was whether a changed fact invalidates the old one. Correctness under staleness catches that, and almost nobody measures it. How are you catching stale-but-confident memory before the agent acts on it?
Original Article

Similar Articles

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

Hugging Face Daily Papers

This paper identifies a critical failure mode in LLM agents where they fail to update personalized memories when new evidence conflicts with prior beliefs. It introduces the STALE benchmark and a three-dimensional probing framework, revealing that even the best models achieve only 55.2% accuracy, and proposes CUPMem as a prototype for robust memory revision.

Agents don't need memory, they need documentation

Hacker News Top

An argument that current AI agent memory plugins all boil down to flawed RAG-based recall, and that agents would be better served by explicit, document-based knowledge rather than vector-store snippet retrieval.