Tag
A paper proposing 'cache engineering', letting LLM agents directly manipulate their KV cache via drop and repos operations without invalidating the trajectory, co-designed with the AKV serving engine, achieving 27.7% cost reduction, 1.2x throughput, and 7.1% further savings via recursive self-improvement of cache policies.
SparseEngine is a ground-up sparse-first inference engine whose shared lifecycle contract lets each sparse attention method control its KV representation and computation, delivering over 10x higher throughput with KV eviction and over 2x end-to-end speedup on agent benchmarks compared to vLLM.
LLMSys-PaperList is a curated reading list on GitHub that organizes LLM systems research papers and resources into practical categories such as training systems, serving systems, and multi-modal coverage, helping AI/ML engineers and researchers stay updated.