Same retrieval, different answers: 32 of 500 LongMemEval results flipped

Reddit r/AI_Agents Tools
benchmark retrieval memory-layer long-memeval engrava variance ai-memory

Summary

This article highlights that benchmark scores for Engrava, a memory layer tool, can vary due to reader or judge stage variance even when retrieval results are unchanged, using LongMemEval results to illustrate the issue.

Disclosure: I build Engrava, so these are my own project's results. If you are picking a memory layer based on someone's LongMemEval number, here is a failure mode: a gap of a few questions can happen after retrieval, even when retrieval itself is unchanged for those questions. I hit that on my own releases. Engrava 0.6.0 scored 81.6% micro on the full 500-question LongMemEval-S set in August 2026. Version 0.5.0 scored 82.4% in July. So the newer release looked worse by four correct answers, and I wanted to know whether it had actually got worse at finding things. Both runs stored the retrieved IDs, so I diffed the artifacts instead of running the benchmark again. On 457 of the 500 questions, retrieval returned the same IDs in the same order. On the other 43, retrieval differed, but not one graded outcome changed. Meanwhile, 32 outcomes did change: 18 down and 14 up, which nets exactly to the four-answer gap. Every one of those 32 questions had the same retrieved IDs in the same order in both runs. So the score movement happened after retrieval, in the reader or judge stage. Those models can vary even at temperature zero. The runs do sit on different harness commits, though, so this does not measure reader and judge variance by itself. Two things made this diff useful. First, there is no generative LLM deciding what to store or reranking results on read. There is still an embedding call during ingest. Second, the runs record the retrieved IDs for every question, in order. That is the part I wish more published comparisons included. What I can't tell you is how much the score normally moves on its own. Each configuration was run once, so I have no measured variance and I'm not going to invent a confidence interval. It is also one retrieval benchmark on one dataset, and it says nothing about your workload. Both rows are still published, including the older and higher one. The artifacts are in the repo if you want to repeat the diff. The README has the full reproduction steps. The newer row pins commit a45dde9 and engrava==0.6.0. A full run needs the cleaned LongMemEval-S split and an OpenAI API key. There is also a free offline smoke run if you only want to inspect the wiring. I'll put the two repo links in a comment rather than in the post. If you compare memory layers, do you diff the retrieved context, or just the final score?
Original Article

Similar Articles

What would a memory benchmark have to do before you'd trust the number?

Reddit r/AI_Agents

The author critiques the reliability of memory benchmarks for LLMs, citing issues like incorrect golden answers in LoCoMo and inconsistent scoring metrics yielding large score gaps. They question whether practitioners trust these numbers and discuss recent improvements like LongMemEval's knowledge update questions.

#1 on memory benchmark LongMemEval with Gemini Flash, not Pro [R]

Reddit r/MachineLearning

A novel memory retrieval system inspired by episodic memory theory achieves state-of-the-art 96.4% top-50 accuracy on the LongMemEval benchmark using Gemini Flash, outperforming larger Pro-based baselines by isolating retrieval quality from model capability.

Memory retrieval is broken under the hood.

Reddit r/AI_Agents

PrecisionMemBench is an open-source benchmark that tests retrieval precision as a strict unit test, revealing that popular memory frameworks like Mem0, Zep, and Hindsight have very low precision (0.05-0.09) and rely on LLMs to compensate. The article argues for zero-tolerance hard fail on precision for production memory infrastructure.

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

arXiv cs.AI

MemTrace is a benchmark that evaluates LLM agent memory at the knowledge point level, probing how facts behave under varying memory age, question type, and evidence conditions. It reveals that pooled accuracy hides distinct failure modes, and that the main bottleneck is evidence use rather than retrieval.