Tag
The author critiques the reliability of memory benchmarks for LLMs, citing issues like incorrect golden answers in LoCoMo and inconsistent scoring metrics yielding large score gaps. They question whether practitioners trust these numbers and discuss recent improvements like LongMemEval's knowledge update questions.
This paper introduces Regimes, an auditable, held-out-gated improvement loop built on the ActiveGraph runtime for self-improving agents. It demonstrates modest improvements on the LongMemEval dataset by autonomously discovering prompt repairs that pass static checks, sandbox execution, and held-out validation.
Regimes is an auditable, held-out-gated improvement loop built on the ActiveGraph event-sourced runtime. It diagnoses failures in AI agents, proposes repairs, and promotes them only after passing multiple gates, improving accuracy on LongMemEval by up to +0.10.
In our second longmemeval experiment, we introduce semantic ingestion into recall leveraging the ActiveGraph runtime, improving retrieval from 60.6% to 83.4%/84.8% for flat/agentic retrieval with LLM ingestion.
A novel memory retrieval system inspired by episodic memory theory achieves state-of-the-art 96.4% top-50 accuracy on the LongMemEval benchmark using Gemini Flash, outperforming larger Pro-based baselines by isolating retrieval quality from model capability.