Predictive Speculative KV Replication for Bursty LLM Inference
Summary
A paper introducing predictive speculative KV replication to handle bursty LLM inference workloads, with code available on GitHub.
Similar Articles
OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
This paper introduces SparseSpec-L, a training-free self-speculative decoding framework for long-context LLM inference that uses a dynamically sparsified and recallable KV cache along with an entropy-based controller for speculation length, achieving up to 2.79x speedup over autoregressive decoding.
proveKV – Honest 36× lossless (vs f32, 18x vs fp16) KV‑cache compression for LLMs (zero PPL regression)
An open-source repo, proveKV, demonstrates a reproducible KV-cache compression technique achieving 36x lossless (vs f32) and 68x lossy memory reduction on SmolLM2-1.7B with zero PPL regression, including Rust examples and an audit pipeline.
DKV: Open-source KV-cache compression framework for local LLM inference (CLI + technical report)
DKV is an open-source framework for compressing KV-cache during local LLM inference, providing a CLI and a technical report.
RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference
Introduces RKSC, a training-free inference framework for multi-branch LLM reasoning that reduces KV cache redundancy via similarity-based sharing and early exit, achieving up to 3x speedup with minimal error.