@Alacritic_Super: If you want to master LLM inference, start with these three papers. They introduced many of the ideas powering today's …

X AI KOLs Timeline Papers
llm-inference kv-cache attention-mechanism batch-serving throughput pagedattention sglang

Summary

This thread recommends three key papers for mastering LLM inference: PagedAttention, Sarathi-Serve, and SGLang, which introduce efficient memory management, chunked prefills, and structured generation techniques used in modern inference engines like vLLM and TensorRT-LLM.

If you want to master LLM inference, start with these three papers. They introduced many of the ideas powering today's fastest inference engines. 1. PagedAttention (vLLM) arXiv: 2309.06180 https://arxiv.org/abs/2309.06180 Why it matters • Efficient KV cache memory management • Virtual-memory-inspired paging • Near-zero memory fragmentation • 2–4× higher serving throughput • Foundation of modern vLLM serving 2. Sarathi-Serve arXiv: 2403.02310 https://arxiv.org/abs/2403.02310 Why it matters • Chunked prefills • Stall-free scheduling • Better throughput–latency tradeoff • Higher GPU utilization • Smarter batching for production inference 3. SGLang: Efficient Execution of Structured Language Model Programs NeurIPS 2024 https://arxiv.org/abs/2312.07104 Why it matters • Structured generation • RadixAttention • Prefix/KV cache reuse • Continuous batching • Fast structured decoding • High-performance LLM & VLM serving These three papers explain many of the core ideas behind vLLM, SGLang, TensorRT-LLM, and modern production LLM serving.
Original Article
View Cached Full Text

Cached at: 07/09/26, 03:46 PM

If you want to master LLM inference, start with these three papers. They introduced many of the ideas powering today’s fastest inference engines.

  1. PagedAttention (vLLM) arXiv: 2309.06180 https://arxiv.org/abs/2309.06180

Why it matters • Efficient KV cache memory management • Virtual-memory-inspired paging • Near-zero memory fragmentation • 2–4× higher serving throughput • Foundation of modern vLLM serving

  1. Sarathi-Serve arXiv: 2403.02310 https://arxiv.org/abs/2403.02310

Why it matters • Chunked prefills • Stall-free scheduling • Better throughput–latency tradeoff • Higher GPU utilization • Smarter batching for production inference

  1. SGLang: Efficient Execution of Structured Language Model Programs NeurIPS 2024 https://arxiv.org/abs/2312.07104

Why it matters • Structured generation • RadixAttention • Prefix/KV cache reuse • Continuous batching • Fast structured decoding • High-performance LLM & VLM serving

These three papers explain many of the core ideas behind vLLM, SGLang, TensorRT-LLM, and modern production LLM serving.

Efficient Memory Management for Large Language Model Serving with PagedAttention

Source: https://arxiv.org/abs/2309.06180 View PDF

Abstract:High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4\\times with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM’s source code is publicly available atthis https URL

Submission history

From: Woosuk Kwon [view email] **[v1]**Tue, 12 Sep 2023 12:50:04 UTC (831 KB)

Similar Articles

Inference Engines for LLMs & Local AI Hardware (2026 Edition)

X AI KOLs

This article provides a comprehensive guide to LLM inference engines for local AI hardware in 2026, explaining how to choose based on hardware strategy, workload, and serving model, and covering engines like llama.cpp, MLX, ExLlamaV2/3, vLLM, SGLang, TensorRT-LLM, and NVIDIA Dynamo.