@junupark_: new note: inside nano-vllm-v1 this can be seen as the second part of my previous note on nano-vllm. in this note, i go …
Summary
This note explains the implementation of chunked prefill and mixed batching in nano-vllm-v1, and benchmarks performance against the previous version.
View Cached Full Text
Cached at: 07/28/26, 10:26 AM
new note: inside nano-vllm-v1
this can be seen as the second part of my previous note on nano-vllm. in this note, i go over the two main ideas - chunked prefill and mixed batching (prefill and decode can be scheduled in the same step) - and how they are implemented at the code level. also benchmarked nano-vllm against nano-vllm-v1 to see how scheduling behavior affects the end-to-end performance.
link to note: https://github.com/junuxyz/mlsys-notes/blob/main/notes/vllm/inside-nano-vllm-v1.md… link to benchmark repo: https://github.com/junuxyz/nano-vllm-bench…
junuxyz/mlsys-notes
Source: https://github.com/junuxyz/mlsys-notes
mlsys-notes
Learning notes and experiments for understanding modern Machine Learning System.
Currently focusing on LLM serving system and inference optimization.
Notes
- Introduction to LLM Inference Part 1
- ORCA paper review
- PagedAttention paper review
- Softmax: From Naive to Blocked Softmax
Inference Engine
Distributed
Hardware
- NVIDIA GPU Architecture: From GPC to SM
- GPU Memory Hierarchy in CUDA
- CUDA Intro
- [KR] NPU Memory Architecture and Data Reuse
Diffusion
Labs
- microengine: a minimal serving engine
- tinyorca: a minimal implementation of ORCA
- tiny-speculators: a from-scratch implementation of speculative decoding model training.
Similar Articles
@junupark_: new article: inside nano-vLLM. while v0 architecture is a bit outdated at this point, still thought this repo was optim…
A new article by @junupark_ provides an end-to-end code walkthrough of nano-vLLM, an LLM inference engine, with plans to cover speculative decoding techniques like Eagle3 and dFlash.
@no_stp_on_snek: while everyone is talking about @SpaceXAI , @AnthropicAI , and @OpenAI updates (but where @GoogleAI?)... went and teste…
A detailed comparison of Unsloth's NVFP4 quantized model inference performance between vLLM and llama.cpp, highlighting prefill speed advantages for vLLM but decode and caching advantages for llama.cpp in single-stream agent workloads.
vllm-project/vllm v0.21.0rc1
vLLM v0.21.0rc1 is a pre-release update for the high-performance LLM inference and serving library, featuring optimizations for throughput, quantization, and hardware support.
@athleticKoder: A 1600-word note on how llm inference work: Covering: 1. Attention - the only place tokens interact 2. KV caching - why…
A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.
@_avichawla: Prefill & decode in LLM inference. Have you ever noticed that the first token from an LLM always takes a moment to appe…
Explains the two phases of LLM inference - prefill and decode - detailing how GPU bottlenecks shift from compute-bound during prefill to memory-bound during decode, and the importance of KV caching.