Tag
Minima-KV presents a retention-preserving KV cache compression method using mixed-format paged attention to reduce memory footprint in long-context LLM serving, with evaluated performance on benchmarks.
The article explains how prompt caching works in LLM inference using paged attention and prefix caching, with practical tips for developers to optimize performance.
PagedAttention applies virtual memory concepts to the KV cache in language models, reducing memory fragmentation and enabling inference engines to serve 2-4 times more users per GPU.
A deep dive into vLLM's architecture and components for high-throughput LLM inference, covering scheduling, paged attention, continuous batching, advanced features, scaling, serving, and benchmarking.
A developer shares their day 6 of inference engineering, writing a CUDA kernel for 1D convolution, explaining PagedAttention's memory efficiency, and providing an overview of GPU memory hierarchy (global, register, local, constant, shared).
Recommends an introduction to LLM serving, inference basics, and VLLM, covering paged attention and continuous batching.
Tiny-vLLM is a high-performance LLM inference engine implemented in C++ and CUDA, offering features like continuous batching and PagedAttention, and serves as an educational resource.
This paper introduces PagedAttention, an algorithm inspired by virtual memory paging, and vLLM, a serving system that significantly improves LLM throughput by reducing memory fragmentation in key-value caches.