Tag
This paper introduces QV-PIC, a query-aware dual-resolution position-independent caching framework for efficient RAG serving, which improves F1 by 21.6 points over vanilla rendered-image PIC while reducing time-to-first-token by 83.8% relative to full prefill.
This paper proposes Kara, a sliding-window KV cache compression method for efficient serving of reasoning LLMs, addressing limitations in existing compression techniques by using bidirectional attention and a Token2Chunk module. The method is integrated into the KvLLM inference framework built upon vLLM, improving output throughput while maintaining performance.
TurboServe is a serving system designed for streaming video generation workloads, using an online scheduling approach that coordinates session placement and GPU provisioning to reduce latency and cost.
This paper proposes Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV cache transmission with compact semantic codes, achieving up to 2.65x TTFT speedup while keeping generation quality within 5% F1 of the oracle.