prefetching

The article discusses key challenges in memory retrieval for voice agents, emphasizing the need for measuring P99 latency per turn, using prefetching, and budgeting memory tokens to reduce latency and improve user experience.

OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.

The article explains how the author achieved p99 zero-millisecond perceived latency for autocomplete on 240 million domain names by prefetching suggestions on keyDown and caching, with a fast API built on Tranco and CZDS data.