Efficient Memory Management for Large Language Model Serving with PagedAttention

Papers with Code Trending 09/12/23, 12:50 PM Papers

llm-serving memory-management paged-attention vllm kv-cache inference-optimization

Summary

This paper introduces PagedAttention, an algorithm inspired by virtual memory paging, and vLLM, a serving system that significantly improves LLM throughput by reducing memory fragmentation in key-value caches.

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4times with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm

Original Article

View Cached Full Text

Cached at: 05/08/26, 08:38 AM

Paper page - Efficient Memory Management for Large Language Model Serving with PagedAttention

Source: https://huggingface.co/papers/2309.06180

Abstract

PagedAttention algorithm and vLLM system enhance the throughput of large language models by efficiently managing memory and reducing waste in the key-value cache.

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because thekey-value cache(KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we proposePagedAttention, an attention algorithm inspired by the classicalvirtual memoryandpaging techniquesin operating systems. On top of it, we buildvLLM, an LLM serving system that achieves (1) near-zero waste inKV cachememory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show thatvLLMimproves the throughput of popular LLMs by 2-4times with the same level of latency compared to the state-of-the-art systems, such asFasterTransformerandOrca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms.vLLM’s source code is publicly available at https://github.com/vllm-project/vllm