The article discusses key challenges in memory retrieval for voice agents, emphasizing the need for measuring P99 latency per turn, using prefetching, and budgeting memory tokens to reduce latency and improve user experience.
I'm seeing every memory tool market themselves using retrieval delay but it doesn't mean much on its own. I found this out after putting memory into a voice agent. The limits I faced A voice turn has a hard time limit, from end of user speech to first audio out and if you take too long the person hears it as the line dropping and they start talking again, which triggers barge in and messes up the next turn too. A cascaded stack splits that time across end of speech detection, TTFT and speech synthesis plus transport and once those three are done, whatever's left is small and memory has to fit inside that. Problem 1: P50 is not the right metric A call is a sequence of turns not one turn so across a few calls your P99 will absolutely show up. The user feels your worst turn and it sounds like the agent going quiet mid sentence. I only caught the tail latency after measuring per turn instead of in aggregate, mostly graph traversals on entities with hundreds of links. A customer with 200 interactions retrieves nothing like one with 5 and you can't see any of that in a P50 on a clean test store. If you're evaluating any memory layer, ask for P99 under concurrency on real data. Problem 2: Retrieval blocking the response The basic flow is (end of speech - transcript - retrieve - build prompt - call model), which puts retrieval right in the silence the user is already in and every millisecond adds directly to what they feel. The solution is prefetching, streaming transcription sends partial results while the person is still talking so I retrieve against the partial and by end of speech the memory is already there and once you prefetch, retrieval speed only matters on misses. Problem 3: Fast retrieval that returns too much is worse Injected tokens push TTFT up roughly linearly at these scales, so a 15ms retrieval that dumps a few thousand tokens into the prompt adds more delay than the retrieval saved so you get a good benchmark but lose the turn. I started thinking in token budget instead of top k. Rank across stores and fill to budget plus cut whatever isn't worth the tokens and remove markdown from your system prompt for voice, it messes with synthesis and on top you're paying prefill for useless characters. Problem 4: The user changes their mind mid turn For instance "book me tuesday, actually no, thursday and make it afternoon." This is where low retrieval latency matters as recovery cost when the guess was wrong. I treat every prefetch as speculative and check it against the final transcript before anything hits the model. My advice for anyone building in this Measure P99 per turn, under concurrency and on a store with realistic data density. Prefetch on partial transcripts. Budget memory in tokens rather than in results and count the TTFT cost of whatever you inject. Instrument the whole chain before you optimise any single link. Happy to answer anything, shoot your questions :)
A voice agent team found that despite lower end-to-end latency (280ms vs competitor's 450ms), users perceived it as slower due to poor barge-in interrupt rate (380ms vs 60ms). They identified three fixes—memory pinning, VAD threshold tuning, and smaller TTS chunks—that improved barge-in rate from 41% to 89% at 100ms, making users feel it's faster.
After 6 months running a voice AI agent for service businesses, the author reveals that real-world latency is bimodal (median ~800ms, p95 ~2.4s) and this p95 determines user perception. Issues like VAD misfires, function call degradation with long prompts, and TTS quality matter more than LLM choice, with multilingual support adding significant costs.
Memora is a scalable memory system for AI agents that decouples storage from retrieval, enabling long-horizon tasks with up to 98% fewer context tokens while setting new state-of-the-art on benchmarks. The paper is published at ICML 2026.
Voice agents often face latency issues not mainly from the model but from components like endpointing; optimizing VAD and using proper benchmarking can reduce turn latency effectively.
AgentMemBench is a systematic benchmark that evaluates five long-term memory management strategies for conversational AI agents across three datasets, finding that external key-value store retrieval dominates on quality but incurs a larger memory footprint.