A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
Summary
This paper analyzes a one-year production trace from Chutes to study LLM serving workloads, revealing temporal evolution and user-model interactions to improve serving system benchmarking.
View Cached Full Text
Cached at: 08/17/26, 09:37 AM
# A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing Source: [https://arxiv.org/abs/2608.13573](https://arxiv.org/abs/2608.13573) [View PDF](https://arxiv.org/pdf/2608.13573) > Abstract:Large Language Model \(LLM\) serving has become a critical cloud workload, and realistic traces are essential for motivating and benchmarking serving systems\. However, existing LLM serving workload studies remain limited in scale and scope\. They often observe short time periods and provide limited visibility into how users interact with models in production\. As a result, they do not fully capture how LLM serving workloads evolve over time or how user\-model interactions shape production traffic\. In this work, we further the understanding of real\-world LLM serving workloads through both a global characterization and a longitudinal study of a one\-year production trace from Chutes\. Unlike prior studies, our trace captures full production behavior across many models and users, including both popular and long\-tail models\. We analyze the workload from aggregate, temporal, model\-level, and user\-level perspectives, revealing workload evolution and user\-model structure that are typically hidden behind aggregate views\. To support future research, we will release the full one\-year trace with the paper, enabling downstream studies of production behavior without relying on sampled or synthetically generated workloads\. ## Submission history From: William Nixon \[[view email](https://arxiv.org/show-email/aeda041d/2608.13573)\] **\[v1\]**Fri, 3 Jul 2026 17:45:43 UTC \(3,765 KB\)
Similar Articles
Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)
A deep dive into vLLM's architecture and components for high-throughput LLM inference, covering scheduling, paged attention, continuous batching, advanced features, scaling, serving, and benchmarking.
@Alacritic_Super: If you are building production LLM applications, learn LLM Caching. Caching can reduce latency, GPU utilization, and AP…
This article emphasizes the importance of LLM caching in production systems to reduce latency, GPU utilization, and costs, and introduces LMCache, an open-source KV cache management layer for scalable LLM inference.
We stopped optimizing our LLM stack manually — it optimizes itself now
The article describes a company's transition to a self-optimizing LLM stack that uses production traces to automatically route requests and fine-tune models, resulting in significant cost reductions and performance improvements.
@Mayhem4Markets: https://x.com/Mayhem4Markets/status/2069090022117019928
A detailed technical comparison of two dominant LLM serving frameworks, SGLang and vLLM, covering architectural differences in KV cache management (RadixAttention vs PagedAttention), throughput, latency, and deployment considerations for self-hosted environments.
Your LLM inference benchmark is lying to you
The article explains why synthetic benchmarks for LLM inference can be misleading, as production traffic is bursty and variable, and recommends testing with real workloads to choose the right inference framework.