Tag
A 10-week, 30-minutes-a-day roadmap for engineers to learn LLM inference serving, covering vLLM, SGLang, load testing, quantization, and optimization techniques to build a reproducible benchmark.
This paper introduces Decoupled Contrastive Decoding (DCD), which uses an expert-aligned lightweight proposer for speculative decoding while keeping the contrastive signal only in verification, achieving speedups over vanilla contrastive decoding without degrading output distribution.
This paper introduces the trie automaton, a specialized constrained decoding mechanism for finite sets that precomputes token masks via Aho-Corasick matching, achieving up to 29x end-to-end throughput improvements over XGrammar in vLLM batch serving while guaranteeing 100% output validity.
This paper argues that the optimal tokenizer vocabulary size is not fixed but depends on the deployment regime, such as batch size and inference volume. Through roofline analysis and experiments on A10G and A100 GPUs, it shows the lifecycle-optimal vocabulary can shift by up to 16x between on-device and datacenter serving, with minimal quality impact.
LinearKV is a training-free framework that enables position-independent caching for hybrid LLMs by using a single cached state to initialize linear layers, outperforming exact prefix-state composition and remaining compatible with existing PIC methods.
SPECTRA is a training-free codec that re-encodes LLM KV caches via spectral transform to concentrate bit budgets on important channels, achieving near-lossless 4x compression and usable compression up to 12x, surpassing the 2-bit quantization cliff.
CommitKV proposes a lifecycle-aware KV cache compression method for multi-turn ReAct agents, distinguishing dormant tokens from completed ones via commit transitions to reduce memory use and speed up inference.
A detailed benchmark comparing 16 quantizations of Qwen3.6 27B across GGUF, NVFP4, AWQ, AutoRound, and FP8 formats, measuring KL divergence from the unquantized reference. Weight-only GGUF quants generally offer the best quality-size tradeoffs, while vLLM quants vary substantially.
A PSA about Meta's Muse Glimmer: setting reasoning strength via system prompt is partially overridden by the chat template's high default, and the absent default is high, not off. Using chat_template_kwargs.reasoning_strength provides more control.
A discussion on why speculative decoding matured in 2026 for LLM inference, citing Uber's early use, Apple and DeepMind papers, and Tri Dao's research, with observations on adoption in frameworks and local deployments.
AoH is a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections, enabling sparse attention without runtime attention scores. At 50% sparsity it retains 96.5% of full-attention performance while reducing prefill/decode latency and KV-cache memory.
CubicQuant proposes a parametric non-uniform scalar quantization format for LLM weights, using a monotonic cubic curve to adapt reconstruction levels at 1-8 bit widths while retaining dense integer code streams for GPU efficiency. Experiments show RMSE reductions over uniform and floating-point baselines, with preliminary H200 kernel measurements.
This paper introduces HYMELL, a hybrid analytical-machine-learning framework for estimating LLM inference latency and energy across prefill and decode phases, validated on NVIDIA H100 with under 5% error for LLaMA 3 8B.
A user seeks advice on building a budget home AI server with 32-48GB VRAM, debating between AMD RX 9060 XT and Nvidia RTX 5060 Ti GPUs, and whether to use AM5 or used EPYC platforms for local LLM inference and large MoE model offloading.
OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.
This preprint introduces DBLast, a dependent block drafter for stochastic speculative decoding, using a low-rank latent mixture over token positions and an acceptance-oriented training objective to improve accepted draft length in higher-entropy decoding regimes. Experiments with Qwen3-4B and Qwen3-8B show consistent improvements over independent block sampling.
This paper introduces QEvict, a KV-cache management scheme for LLMs that uses recoverable quantized eviction to handle attention drift during long-context decoding, improving memory efficiency while preserving important historical context.
vLLM, open-sourced by UC Berkeley's Sky Computing Lab, is a high-performance LLM inference and serving library. Through PagedAttention and continuous batching, it dramatically improves throughput and reduces VRAM waste, while being compatible with the OpenAI API and various hardware.
DeepSeek announced a significant API price hike, and analysis suggests the move goes beyond GPU cost pass-through to reflect broader market shifts toward value-based pricing and open-source ecosystem pressures.
A deep dive into vLLM's architecture and components for high-throughput LLM inference, covering scheduling, paged attention, continuous batching, advanced features, scaling, serving, and benchmarking.