Tag
A discussion on how unreliable LLM memory benchmarks can be, noting that scores swing wildly depending on the evaluator model (e.g., 51.4 under token-overlap F1 vs. 75.8 under an LLM judge), the reader model, and run counts — while many results are vendor-reported with unclear noise levels.
The author critiques the reliability of memory benchmarks for LLMs, citing issues like incorrect golden answers in LoCoMo and inconsistent scoring metrics yielding large score gaps. They question whether practitioners trust these numbers and discuss recent improvements like LongMemEval's knowledge update questions.