Same retrieval, different answers: 32 of 500 LongMemEval results flipped

Disclosure: I build Engrava, so these are my own project's results. If you are picking a memory layer based on someone's LongMemEval number, here is a failure mode: a gap of a few questions can happen after retrieval, even when retrieval itself is unchanged for those questions. I hit that on my own releases. Engrava 0.6.0 scored 81.6% micro on the full 500-question LongMemEval-S set in August 2026. Version 0.5.0 scored 82.4% in July. So the newer release looked worse by four correct answers, and I wanted to know whether it had actually got worse at finding things. Both runs stored the retrieved IDs, so I diffed the artifacts instead of running the benchmark again. On 457 of the 500 questions, retrieval returned the same IDs in the same order. On the other 43, retrieval differed, but not one graded outcome changed. Meanwhile, 32 outcomes did change: 18 down and 14 up, which nets exactly to the four-answer gap. Every one of those 32 questions had the same retrieved IDs in the same order in both runs. So the score movement happened after retrieval, in the reader or judge stage. Those models can vary even at temperature zero. The runs do sit on different harness commits, though, so this does not measure reader and judge variance by itself. Two things made this diff useful. First, there is no generative LLM deciding what to store or reranking results on read. There is still an embedding call during ingest. Second, the runs record the retrieved IDs for every question, in order. That is the part I wish more published comparisons included. What I can't tell you is how much the score normally moves on its own. Each configuration was run once, so I have no measured variance and I'm not going to invent a confidence interval. It is also one retrieval benchmark on one dataset, and it says nothing about your workload. Both rows are still published, including the older and higher one. The artifacts are in the repo if you want to repeat the diff. The README has the full reproduction steps. The newer row pins commit a45dde9 and engrava==0.6.0. A full run needs the cleaned LongMemEval-S split and an OpenAI API key. There is also a free offline smoke run if you only want to inspect the wiring. I'll put the two repo links in a comment rather than in the post. If you compare memory layers, do you diff the retrieved context, or just the final score?