prefetching

PreSTO 提出一种预测性子树预取方法加速 PowerMH 功率采样器，通过批量预取未来的 MH 提案并保持转移核不变，在七个数据集和五个 LLM 上实现 1.71×–2.33× 的中位加速。

The paper introduces KVFetch, a training-free framework that adds a temporal recall channel to score-based KV cache compressors to fix "sequential forgetting" (verbatim copying breaking under compression), recovering copying performance from 0.8 to 78.4 and boosting the RULER-16K average by +8.4 at iso-budget.

The article discusses key challenges in memory retrieval for voice agents, emphasizing the need for measuring P99 latency per turn, using prefetching, and budgeting memory tokens to reduce latency and improve user experience.

OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.

The article explains how the author achieved p99 zero-millisecond perceived latency for autocomplete on 240 million domain names by prefetching suggestions on keyDown and caching, with a fast API built on Tranco and CZDS data.