@yoheinakajima: ran my first benchmark this weekend (longmemeval) mostly to test activegraph, learned a lot! - this is a stepping stone…
Summary
Yohei Nakajima ran the LongMemEval benchmark on ActiveGraph, achieving 85.6% QA accuracy and 86.2% turn answer-in-context, demonstrating the effectiveness of event-based agent systems for long-term memory.
View Cached Full Text
Cached at: 05/26/26, 07:05 AM
ran my first benchmark this weekend (longmemeval) mostly to test activegraph, learned a lot!
- this is a stepping stone to show the event based agent system works. the AI convinced me not to start with graph extraction of facts/entities
- learned running benchmarks takes a long time
- understand more what a good benchmark vs bad benchmark is (I think this is pretty thorough)
- fully reproducible open source repo and tests
- seems like ActiveGraph is well suited for this, performed solid, which is a good start
no extraction here, just deterministic ingestion as a first test
for now just research, but i’m also testing cofounder from @intelligenceco which is handling the website/newsletter/blog which has been a fun way to add some polish
Awesome
that’s so cool, please do! i can really only work on this on weekends so would love to see other ppl push it faster
Similar Articles
@yoheinakajima: ActiveGraph: 1 month in: Paper #1: The Log is the Agent 3 LongMemEval Experiments Paper #2: Regimes, self-improvement l…
ActiveGraph announces two new papers on agent memory (LongMemEval) and self-improvement regimes, along with reference agents, pack templates, and upcoming meetups in Seattle and San Francisco.
Benchmarking agent memory retrieval on LongMemEval‑S — 98% Recall@5, 100% recall by R@23, local embeddings only (all-MiniLM-L6-v2), no LLM, no API key
The author shares benchmark results for memweave, a Python library for agent memory, achieving 98% Recall@5 on LongMemEval-S using only local embeddings without LLM calls. The post details the methodology and compares performance against mempalace, highlighting stable retrieval across different question types.
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
This paper introduces LongMemEval-V2, a benchmark for evaluating long-term memory systems in web agents, along with two memory methods: AgentRunbook-R and AgentRunbook-C.
@yoheinakajima: i know it's backwards order, but experiment #2:
In our second longmemeval experiment, we introduce semantic ingestion into recall leveraging the ActiveGraph runtime, improving retrieval from 60.6% to 83.4%/84.8% for flat/agentic retrieval with LLM ingestion.
@yoheinakajima: longmemeval experiment arch: 1) deterministic ingestion/extraction (85.6% accuracy, 86.2% retrieval) 2) semantic ingest…
The tweet shares results from LongMemEval experiments comparing deterministic and semantic ingestion/extraction methods, achieving up to 87.6% accuracy and 90.0% retrieval with a hybrid approach.