@junupark_: new note: inside nano-vllm-v1 this can be seen as the second part of my previous note on nano-vllm. in this note, i go …
Summary
This note explains the implementation of chunked prefill and mixed batching in nano-vllm-v1, and benchmarks performance against the previous version.
View Cached Full Text
Cached at: 07/28/26, 10:26 AM
new note: inside nano-vllm-v1
this can be seen as the second part of my previous note on nano-vllm. in this note, i go over the two main ideas - chunked prefill and mixed batching (prefill and decode can be scheduled in the same step) - and how they are implemented at the code level. also benchmarked nano-vllm against nano-vllm-v1 to see how scheduling behavior affects the end-to-end performance.
link to note: https://github.com/junuxyz/mlsys-notes/blob/main/notes/vllm/inside-nano-vllm-v1.md… link to benchmark repo: https://github.com/junuxyz/nano-vllm-bench…
junuxyz/mlsys-notes
Source: https://github.com/junuxyz/mlsys-notes
mlsys-notes
Learning notes and experiments for understanding modern Machine Learning System.
Currently focusing on LLM serving system and inference optimization.
Notes
- Introduction to LLM Inference Part 1
- ORCA paper review
- PagedAttention paper review
- Softmax: From Naive to Blocked Softmax
Inference Engine
Distributed
Hardware
- NVIDIA GPU Architecture: From GPC to SM
- GPU Memory Hierarchy in CUDA
- CUDA Intro
- [KR] NPU Memory Architecture and Data Reuse
Diffusion
Labs
- microengine: a minimal serving engine
- tinyorca: a minimal implementation of ORCA
- tiny-speculators: a from-scratch implementation of speculative decoding model training.
Similar Articles
@junupark_: new article: inside nano-vLLM. while v0 architecture is a bit outdated at this point, still thought this repo was optim…
A new article by @junupark_ provides an end-to-end code walkthrough of nano-vLLM, an LLM inference engine, with plans to cover speculative decoding techniques like Eagle3 and dFlash.
@akshay_pachaar: How vLLM stops one long prompt from stalling others: (chunked prefill, clearly explained) Every LLM request has two pha…
A technical explainer on how vLLM's chunked prefill prevents long prompts from stalling concurrent decoding requests, covering latency tradeoffs, mixed batching, and the `--max-num-batched-tokens` knob in vLLM V1.
@no_stp_on_snek: while everyone is talking about @SpaceXAI , @AnthropicAI , and @OpenAI updates (but where @GoogleAI?)... went and teste…
A detailed comparison of Unsloth's NVFP4 quantized model inference performance between vLLM and llama.cpp, highlighting prefill speed advantages for vLLM but decode and caching advantages for llama.cpp in single-stream agent workloads.
Question: Why is prefill unbelievably faster in vLLM than other inference engines?
The user shares benchmark results showing vLLM's significantly faster prefill performance compared to llama.cpp and other engines, and questions the technical reasons behind this speed difference.
@athleticKoder: A 1600-word note on how llm inference work: Covering: 1. Attention - the only place tokens interact 2. KV caching - why…
A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.