Tag
TOPAS is a workflow-aware prefix-state scheduling method for multi-agent LLM serving that optimizes cache usage and request scheduling to reduce job completion time.
CacheScout is an agent-aware KV-cache runtime layer for multi-agent LLM serving that learns agent execution semantics online to guide cache eviction and prefetching, improving cache hit rate and reducing latency.
A developer shares how they cut DeepSeek API costs by roughly 90% through prefix caching optimizations, reducing browser agent task costs below $0.005 and enabling a free, ads-supported browser agent.
CacheBlend, a EuroSys 2025 Best Paper, solves the problem that 90% of KV cache is never reused due to rigid prefix-matching in prompt caching. By selectively recomputing only boundary tokens between documents, it achieves 2-4x faster multi-document processing without quality loss, implemented in the open-source LMCache layer.
The author shares their work over 6-8 months in ML systems and AI infrastructure, including a lightweight Python LLM inference engine (tachyon) that achieves 600+ tokens/s on consumer hardware with continuous batching and prefix caching, alongside blog posts on CUDA/CUTE DSL and collective communication, and contributions to SGLang and vLLM.
Trellis introduces RadixAttention to optimize LLM inference prefill phase by caching prefix tokens using a radix tree, reducing redundant computation in chat and agentic sessions.
vLLM integrates Mooncake Store for distributed KV cache reuse, enabling cross-node prefix caching to efficiently serve agentic workloads with high token reuse.
ObjectCache proposes using S3-compatible object storage for LLM KV cache reuse to reduce cost and increase capacity, with a co-designed storage protocol and transfer schedule that minimizes latency overhead. Experiments show it adds only 5.6% latency over local DRAM for 64K contexts.
Reasonix is a terminal-based AI coding agent optimized for DeepSeek models, achieving 99.82% cache hit rate and reducing token costs from ~$61 to ~$12 per workload through stable prefix caching.
A new semantic-adaptive eviction policy for LLM prefix caches that learns token reuse patterns across different token types, achieving 1.4x-2.7x TTFT improvement over existing policies.
Reasonix is a terminal AI coding agent designed specifically for DeepSeek API prefix caching mechanism, achieving ultra-low token costs in long sessions through a cache-first architecture. In testing, 435 million input tokens cost only about $12, with a cache hit rate of 99.82%.
This paper introduces sparse prefix caching for hybrid and recurrent LLMs, which stores recurrent states at a limited set of checkpoint positions to avoid dense caching while minimizing recomputation. The method outperforms standard heuristics on real-world data, especially when requests share substantial but non-identical prefixes.