Deterministic on every query, 0 LLM calls to retrieve, and it still hits 1.00 recall: we put our memory tool head-to-head with 4 others
Summary
This article presents a head-to-head comparison of a new deterministic memory tool that achieves perfect recall with zero LLM calls during retrieval against four other tools.
Similar Articles
Memory retrieval is broken under the hood.
PrecisionMemBench is an open-source benchmark that tests retrieval precision as a strict unit test, revealing that popular memory frameworks like Mem0, Zep, and Hindsight have very low precision (0.05-0.09) and rely on LLMs to compensate. The article argues for zero-tolerance hard fail on precision for production memory infrastructure.
#1 on memory benchmark LongMemEval with Gemini Flash, not Pro [R]
A novel memory retrieval system inspired by episodic memory theory achieves state-of-the-art 96.4% top-50 accuracy on the LongMemEval benchmark using Gemini Flash, outperforming larger Pro-based baselines by isolating retrieval quality from model capability.
Same system, very different scores depending on the judge. How do you read memory benchmarks?
A discussion on how unreliable LLM memory benchmarks can be, noting that scores swing wildly depending on the evaluator model (e.g., 51.4 under token-overlap F1 vs. 75.8 under an LLM judge), the reader model, and run counts — while many results are vendor-reported with unclear noise levels.
Memoria 1.0.0 — a local, model-agnostic memory system for LLMs
The author released Memoria 1.0.0, a local-first, model-agnostic memory system for LLM applications that runs without a GPU or cloud API key. Benchmarked on an Intel Celeron N4020 (3.7 GiB RAM), it achieved 89.8% Recall@1 and 99.6% Recall@50 on LongMemEval-S using parallel FAISS, BM25, graph, phrase, attribute, and temporal retrieval with multi-signal ranking, and is installable via PyPI.
Same retrieval, different answers: 32 of 500 LongMemEval results flipped
This article highlights that benchmark scores for Engrava, a memory layer tool, can vary due to reader or judge stage variance even when retrieval results are unchanged, using LongMemEval results to illustrate the issue.