@junupark_: new article: inside nano-vLLM. while v0 architecture is a bit outdated at this point, still thought this repo was optim…
Summary
A new article by @junupark_ provides an end-to-end code walkthrough of nano-vLLM, an LLM inference engine, with plans to cover speculative decoding techniques like Eagle3 and dFlash.
View Cached Full Text
Cached at: 07/06/26, 04:04 AM
new article: inside nano-vLLM.
while v0 architecture is a bit outdated at this point, still thought this repo was optimal for an end-to-end code walkthrough and to understand the mental model of llm inference engine. think i can polish it better later but feel free to read and give feedback!
next i’m planning to dive more into speculative decoding - mainly eagle3 and dflash. open to other suggestions as well.
Similar Articles
@junupark_: new note: inside nano-vllm-v1 this can be seen as the second part of my previous note on nano-vllm. in this note, i go …
This note explains the implementation of chunked prefill and mixed batching in nano-vllm-v1, and benchmarks performance against the previous version.
@vllm_project: vLLM v0.21.0 is out! 367 commits from 202 contributors (49 new). Highlights: KV Offload + HMA, spec decode with thinkin…
vLLM v0.21.0 has been released with KV Offload + HMA, speculative decoding with thinking budget for reasoning models, TOKENSPEED_MLA on Blackwell for DSR1/Kimi K2.5, Mooncake distributed KV, DeepSeek V4 pipeline parallelism, and a C++20 + Transformers v5 baseline.
@no_stp_on_snek: while everyone is talking about @SpaceXAI , @AnthropicAI , and @OpenAI updates (but where @GoogleAI?)... went and teste…
A detailed comparison of Unsloth's NVFP4 quantized model inference performance between vLLM and llama.cpp, highlighting prefill speed advantages for vLLM but decode and caching advantages for llama.cpp in single-stream agent workloads.
@_avichawla: Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effe…
Researchers introduced DFlash, a technique using block diffusion models for speculative decoding that accelerates LLM inference by up to 8.5x without accuracy loss. It is already integrated with major frameworks like vLLM and SGLang.
@ngxson: My talk about llama.cpp speculative decoding (MTP, dflash, dspark) at dotAI replay available soon!
A talk about llama.cpp speculative decoding methods (MTP, dflash, dspark) at the dotAI conference will have its replay available soon.