I ran 8 AI agent memory systems through 2176 tasks and a plain markdown wiki beat every product.

Reddit r/AI_Agents News

Summary

A developer ran 2,176 tasks across 8 AI agent memory systems and found that a plain markdown wiki curated by the agent outperformed every commercial product, with Mitosis Cortex as the best hosted option.

A few weeks ago I asked here whether anyone had success with second brains. About 75k people read that thread and most of the comments were frustration. When I published my web search benchmark last month I promised memory tools were next. Here it is: the Agentic Memory Index. The setup: the same agent setup worked with each of 8 memory systems. Each system got 272 scored tasks: 200 questions about facts stored across simulated multi-week working relationships, plus 72 questions about facts that were never stored, to catch invented memories. There was also a separate scale test with a 5,000-page store. The judge was calibrated against two independent human labelers before the run. What I found: The winner is not a product. A plain markdown wiki that the agent curates itself, following Karpathy's llm-wiki gist, scored 98.5. Every product came in below it. Mitosis Cortex was the top hosted product at 96.9. gbrain, a free open-source local tool, scored 92.9, ahead of every hosted API except Mitosis Cortex. Mem0 (92.3) was the cheapest per 1,000 successful answers at $341. Zep's biggest problem was freshness: a just-stored fact took 162.7 seconds at the median before it became answerable. It passed 8 of 24 update questions. Supermemory was near perfect for recently stored memories (59/60 recall) but passed only 11 of 72 long-horizon questions. If I were choosing today: for a hosted memory API I would start with Mitosis Cortex, it ranked first of the five hosted tools. If I wanted free and local, the boring answer held up: markdown files curated by the agent or gbrain. If cost per answer is the constraint, Mem0 was the cheapest per successful answer in the whole set. The full rankings, confidence intervals, failure breakdowns, cost and speed data and the methodology are in the first comment. Next up: a free tool that shows you what tools your agent should use and how much smarter your agent would be with them.
Original Article

Similar Articles

@tricalt: https://x.com/tricalt/status/2057173322924806651

X AI KOLs Timeline

A founder discusses the scaling challenges of using markdown files for AI agent memory in production, highlighting common pitfalls with permissions, multi-agent interaction, and temporal queries, and suggests that teams often end up patching around these issues without realizing they are rebuilding a more complex system.