locomo

#locomo

What would a memory benchmark have to do before you'd trust the number?

Reddit r/AI_Agents ↗ · 2026-08-12

The author critiques the reliability of memory benchmarks for LLMs, citing issues like incorrect golden answers in LoCoMo and inconsistent scoring metrics yielding large score gaps. They question whether practitioners trust these numbers and discuss recent improvements like LongMemEval's knowledge update questions.

0 favorites 0 likes

#locomo

@elliotchen100: This Chinese article is the clearest I've seen recently on Agent memory. Full disclosure: the EverOS mentioned is built by us @EverMind. Three additions: 1. That 93.05% LoCoMo accuracy isn't just a paper claim—it's a script you can run from the open-source repo, reproducible by anyone. 2. Skill self-evolution: The first trajectory feed only produces cases; you need to run several similar tasks before distilling a skill. Many people integrate and see no skill and think it's broken. 3. Easter egg: A cool new product is launching at the end of the month. If it's not cool, I'll pay up.

X AI KOLs Timeline ↗ · 2026-05-13

This article discusses the implementation of AI Agent memory, introduces the reproducible 93.05% LoCoMo accuracy of the EverOS system and the Skill self-evolution mechanism, and teases a new cool product launching at the end of the month.

0 favorites 0 likes

locomo

What would a memory benchmark have to do before you'd trust the number?

Submit Feedback