Tag
A technical explainer on how vLLM's chunked prefill prevents long prompts from stalling concurrent decoding requests, covering latency tradeoffs, mixed batching, and the `--max-num-batched-tokens` knob in vLLM V1.
The paper introduces Decode-Latency Feedback Prefill (DLFP), a model-free proportional controller implemented in vLLM that adapts chunked prefill sizing based on observed decode latency, reducing P99 inter-token latency by ~28% on Qwen3-0.6B on an A100, while also documenting a clear generalization failure on larger models and multi-GPU setups.
This note explains the implementation of chunked prefill and mixed batching in nano-vllm-v1, and benchmarks performance against the previous version.
A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.
CompactAttention introduces Block-Union KV Selection to accelerate chunked prefill for long-context LLMs, achieving up to 2.72x attention speedup on LLaMA-3.1-8B at 128K context while maintaining accuracy close to dense attention.