I measured memory vs "just send the whole history" over 90 simulated days: 23-62x fewer context tokens, same or better recall on personal facts, and one place where memory clearly loses (numbers + method)

Reddit r/AI_Agents Papers

Summary

This study compares memory systems to full conversation history in AI agents over simulated days, showing 23-62x fewer context tokens with similar or better recall on personal facts, but memory loses on numerical data and specific details like identifiers.

Everyone building an agent hits the same fork: keep a memory layer, or send the full conversation history every request. I run a memory API, so discount accordingly; the point of this post is the measurement, and the losses are in it. Setup. Three synthetic dialogue corpora with ground truth (a companion app, a customer-support desk, a coding assistant), each at 7, 30 and 90 simulated days. Facts are planted on known days, distractors (someone else's birthday, a colleague's allergy, a staging port) planted later. After the last day, 6-8 questions per corpus that need the old facts. Two strategies get the same question and the same answer model (gpt-4o-mini): full history: the whole dialogue as context memory: extract facts per day as they arrive, then retrieve into a 600-token budget Tokens counted exactly (tiktoken o200k), not estimated. One run per cell for the cost table; the memory side was repeated 3x on companion earlier and the run-to-run spread is 0.12-0.17 on recall, so read small recall differences as noise. Context tokens per request, full history -> memory: corpus 30 days 90 days companion 2,837 -> 169 (17x) 7,532 -> 122 (62x) support 2,622 -> 278 (9x) 7,562 -> 331 (23x) coding 2,226 -> 201 (11x) 5,914 -> 186 (32x) Memory stays flat (120-330 tokens) whatever the history length. At $0.15/M input tokens, 1,000 users x 20 requests/day: $630/month for full history at 90 days vs $19 with memory. Context input only; extraction and search costs are on top and not in that number. Recall of the old facts (full history / memory): corpus 30 days 90 days companion 1.00 / 1.00 0.88 / 1.00 support 0.62 / 0.25 0.38 / 0.75 coding 1.00 / 0.88 1.00 / 0.88 Three things I did not expect: Full history is not a free lunch. On support at 90 days the answer model finds the month-old fact in 7.5k tokens of chatter only 38% of the time. Memory got 75% there. "Just send everything" degrades with length too, you just don't see it. Memory loses badly on support at 30 days (0.25). I looked at what was stored: the extractor turned "loyalty number LR-11560" into "has a loyalty number", never extracted the pickup office, and filed the customer's card under the assistant. Identifiers, options, named things: exactly what end-user memory for a product is supposed to hold. That is the next thing to fix and I'll post the number when it's fixed, not before. Measuring found two bugs in my own filter before I could publish. A "don't store session chatter" gate I shipped this week was dropping "noted that the service is pinned to Python 3.12" as reported speech. Coding recall went 0.75 -> 0.88 after the fix. If you write a salience filter, test it on facts phrased as speech. Reproduce: the bench, the corpus generator (fixed seed), a full-history baseline, a reproduction of the OpenAI Agents sandbox memory, and the cost report with the price as a stated input are all in the repo; link in the comments per sub rules. Every experiment has its criterion written down before it ran and its result recorded, including the rejected ones. Question for people who ship agents: what do you send today, and at what history length did it start hurting?
Original Article

Similar Articles