Same system, very different scores depending on the judge. How do you read memory benchmarks?

Reddit r/AI_Agents News
memory-benchmarks llm-evaluation benchmark-reliability llm-judge agent-memory reproducibility

Summary

A discussion on how unreliable LLM memory benchmarks can be, noting that scores swing wildly depending on the evaluator model (e.g., 51.4 under token-overlap F1 vs. 75.8 under an LLM judge), the reader model, and run counts — while many results are vendor-reported with unclear noise levels.

I've been reading memory benchmark results for a while and I can't tell how much to trust any of them. The same system seems to score very differently depending on things that usually aren't printed next to the number, like which model reads the retrieved memories, which model judges the answer, and how many times it was run. I read a post here recently claiming the same set of answers scored 51.4 under token overlap F1 and 75.8 under an LLM judge. On top of that, a lot of the numbers I see are reported by the vendor itself, and it's rarely clear how many runs they come from, so I don't even know what the noise is. Am I overthinking this, or is it a real problem? How do you decide whether a benchmark number means anything, and has a published score ever actually changed which memory tool you picked?
Original Article

Similar Articles

What would a memory benchmark have to do before you'd trust the number?

Reddit r/AI_Agents

The author critiques the reliability of memory benchmarks for LLMs, citing issues like incorrect golden answers in LoCoMo and inconsistent scoring metrics yielding large score gaps. They question whether practitioners trust these numbers and discuss recent improvements like LongMemEval's knowledge update questions.

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

arXiv cs.AI

This paper investigates the measurement limitations of LLM-judged benchmarks, showing that generalizability is constrained by judge variability and that protocol design significantly affects performance ceilings, with implications for AI benchmarking methodologies.

anyone else notice memory API benchmark numbers are all over the place?

Reddit r/AI_Agents

The post questions the reliability of benchmark scores for memory APIs like Mem0 and Zep, noting significant discrepancies between self-reported and third-party numbers on the LoCoMo benchmark, suggesting these metrics may be more marketing than accurate comparisons.