What would a memory benchmark have to do before you'd trust the number?

Reddit r/AI_Agents News
memory-benchmarks ai-evaluation llm benchmark-quality locomo longmemeval

Summary

The author critiques the reliability of memory benchmarks for LLMs, citing issues like incorrect golden answers in LoCoMo and inconsistent scoring metrics yielding large score gaps. They question whether practitioners trust these numbers and discuss recent improvements like LongMemEval's knowledge update questions.

Working on a memory eval and I've hit a wall on whether it's even worth building. Spent the last couple weeks going through the actual benchmark repos instead of the writeups, and two things stuck. There's a public audit of LoCoMo that found 99 of the 1540 questions have wrong golden answers, with code and the full error list included so you can check it yourself. That puts the real ceiling around 93.5 and a few published scores sit above it. The other one bothers me more: I found a repo where the same set of predictions gets scored three ways in the same committed file, token overlap F1 gives 51.4 and an LLM judge on the identical answers gives 75.8. Nobody did anything shady, that's just what happens when there's no agreed metric, but a 24 point gap from the scoring method alone is wider than most of the gaps between systems that people argue about. So does anyone here actually use these numbers when picking a memory layer, or is it all throw my own data at two options and see which one annoys me less. And if you do look at them, what's the bar. Fixed judge model, raw per question output published so you can recount it yourself, something I'm not thinking of. Half expecting the answer to be that none of it matters and everyone picks on docs and pricing. One more thing since it changed my mind halfway through. A lot of the complaints I see are aimed at stuff that already got fixed, "memory benchmarks don't test knowledge updates" comes up constantly but LongMemEval has 78 of exactly those questions and there's a whole separate benchmark for fabrication now. So some of what gets repeated is really about the 2024 versions, and probably some of my own assumptions are too.
Original Article

Similar Articles

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

Hugging Face Daily Papers

This paper introduces MemTrapBench, a benchmark for evaluating cognitive traps in LLM memory use, and proposes AdaptiveMem, an inference-time method to mitigate these traps while preserving performance on standard benchmarks.

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

arXiv cs.AI

MemTrace is a benchmark that evaluates LLM agent memory at the knowledge point level, probing how facts behave under varying memory age, question type, and evidence conditions. It reveals that pooled accuracy hides distinct failure modes, and that the main bottleneck is evidence use rather than retrieval.

MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents

arXiv cs.CL

MemEvoBench introduces the first benchmark for evaluating memory safety in LLM agents, measuring behavioral degradation from adversarial memory injection, noisy outputs, and biased feedback across QA and workflow tasks. The work reveals that memory evolution significantly contributes to safety failures and that static defenses are insufficient.