continuous-batching

An educational walkthrough of the LLM serving scheduling loop, explaining how inference requests move through waiting queues, batch construction, prefill, decode, and continuous batching to keep GPUs saturated as requests finish at different times.

This article explains static, dynamic, and continuous batching strategies for serving large language models on GPUs, detailing their trade-offs and how they optimize throughput and reduce idle time.

An educational overview of key concepts in LLM inference engineering, covering techniques like KV cache, PagedAttention, Flash Attention, and continuous batching to optimize inference performance.

A tutorial on using vLLM to efficiently serve LLM inference for AI agents, covering concepts like continuous batching and PagedAttention to address performance bottlenecks.

A deep dive into vLLM's architecture and components for high-throughput LLM inference, covering scheduling, paged attention, continuous batching, advanced features, scaling, serving, and benchmarking.

BlockServe introduces block-grained continuous batching to address convergence heterogeneity in diffusion LLMs, enabling 1.9–10.6x throughput improvement over Fast-dLLM while maintaining generation quality.

The author shares their work over 6-8 months in ML systems and AI infrastructure, including a lightweight Python LLM inference engine (tachyon) that achieves 600+ tokens/s on consumer hardware with continuous batching and prefix caching, alongside blog posts on CUDA/CUTE DSL and collective communication, and contributions to SGLang and vLLM.

A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.

A blog post explaining continuous batching, a technique for improving LLM serving throughput by dynamically adding new requests to a batch as old ones finish, keeping the GPU busy and reducing idle time.

Recommends an introduction to LLM serving, inference basics, and VLLM, covering paged attention and continuous batching.

Continuous batching has been added to TRL for GRPO, improving speed and VRAM usage without needing vLLM. The tweet explains how it works and when to use it.

zml/llmd now runs fully on Apple's Metal API, serving 8 simultaneous requests at full bf16 precision, with continuous batching and other modern features.

Based on the SGLang Omni team's internal decision-making article, this post introduces the operating principles of LLM inference systems in an accessible way, starting from basic concepts such as autoregressive decoding, KV cache, and continuous batching.

dlmserve is the first open-source serving engine for diffusion language models, providing an OpenAI-compatible API, continuous batching, and 2.5x throughput over Hugging Face, all within 12GB VRAM.

oMLX 0.3.9rc1, an LLM inference server optimized for Apple Silicon Macs, adds low-memory stability, chunked prefill, multi-tasking admin chat, and more.

This article explains how to implement asynchronous continuous batching for LLM inference, overlapping CPU batch preparation with GPU computation to maximize utilization and reduce idle time.