Tag
This pull request enables CUDA graph for MTP draft in llama.cpp to improve performance of LLM inference on GPUs.
DominoTree introduces a training-free best-first draft tree for speculative decoding that uses conditional (non-factorized) correction from Domino to achieve up to 6.6x speedup over autoregressive decoding and the highest mean accept length across evaluated methods on Qwen3 models.
A researcher debuted Shard, achieving 30 tok/s inference on a 744B parameter model distributed across 6 consumer GPUs over the open internet, a 15-20x improvement over previous methods.
A tweet urging AI researchers to learn inference-acceleration basics and spotlighting CUDA Graph as the key to vLLM’s GPU utilization.