@_avichawla: The LLM scheduling loop, clearly explained: (bookmark this) Whenever a user sends a prompt to an LLM, the resulting inf…

The LLM scheduling loop, clearly explained: (bookmark this) Whenever a user sends a prompt to an LLM, the resulting inference request does not go straight from the API endpoint to the GPU. Instead, it first enters the serving engine, where a scheduler decides when and how its tokens will be processed. Here is the complete lifecycle: 1) The request enters the waiting queue Each request arrives with a prompt, sampling settings, and a maximum output length. Prompt lengths vary, and the scheduler cannot know the final generation length in advance. It therefore manages requests one iteration at a time. 2) The scheduler builds the next batch Before every model step, the scheduler checks two main constraints: - How many tokens can be processed in this iteration - How much KV-cache space remains It then applies a scheduling policy, often prioritizing running decode requests before admitting new work. 3) New requests enter prefill During prefill, the model processes the prompt tokens and creates the K and V tensors needed by attention. This stage can process many prompt tokens in parallel. Long prompts may be split into chunks so they do not block active generations for too long. 4) Running requests enter decode Once prefill finishes, the request moves to decode. The model now generates the next token using the KV cache created for all previous tokens. Standard autoregressive decoding usually adds one new token per active request during a model step. 5) The GPU executes the selected work Modern serving engines can place prefill and decode work in the same GPU batch. For example, request A may be processing its prompt while requests B and C generate their next tokens. The batch is therefore not a fixed group that runs until every request finishes. Its composition can change after every model step. 6) The engine updates request state After execution, generated tokens are sampled and appended to their requests. A finished request releases its KV-cache blocks and leaves the batch. An unfinished request keeps its state and becomes eligible for the next scheduling iteration. The scheduler can then use the freed token budget and cache space to admit another waiting request. In the visual below, C finishes after iteration 1, so D enters during iteration 2. By iteration 3, the active batch contains only A and D. This iteration-level replacement is continuous batching. It keeps the GPU working while requests with different prompt and output lengths progress independently. If you want to dive deeper, I wrote a detailed article covering the full LLM inference pipeline, including tokenization, prefill, decode, KV caching, and the latency metrics each stage controls. Read it below.