Tag
A tweet thread shares a curated list of resources for learning LLM inference, covering tokenization, GPU hardware, optimization techniques like quantization and speculative decoding, and inference engines such as vLLM and SGLang, followed by an Nvidia architect's talk summary.
A paper introducing predictive speculative KV replication to handle bursty LLM inference workloads, with code available on GitHub.
This paper proposes a KV cache eviction strategy that scores tokens by counter-causal surprise, removing past tokens that are well-predicted from future context. The method is training-free, in-distribution, and achieves competitive performance with a fast single-layer approximation.
Prox is a training-free framework for sparse SwiGLU FFNs in LLMs, using approximate intermediate-channel salience to build channel masks without dense computation. It outperforms training-free baselines across ten LLMs, achieving up to 1.99x end-to-end decoding speedup at 70% sparsity.
This paper introduces SparseSpec-L, a training-free self-speculative decoding framework for long-context LLM inference that uses a dynamically sparsified and recallable KV cache along with an entropy-based controller for speculation length, achieving up to 2.79x speedup over autoregressive decoding.
This paper proposes functional reconstruction for converting MHA/GQA checkpoints into MLA draft models for speculative decoding, directly optimizing attention modules to preserve token acceptance. It reports consistent improvements across 192 configurations involving Llama/Qwen models and multiple conversion methods.
Introduces Tritium, an open-source Rust/CUDA engine for ternary (1.58-bit) quantization, serving, and training of LLMs on consumer GPUs. It claims faster inference than llama.cpp for ternary models and introduces a new quantization method called SALT.
This paper analyzes lossy verification schemes in speculative decoding, categorizing them into truncation-based and collaborative verification, and identifies pitfalls and principles for maintaining generation quality.
DigitalOcean introduces prompt caching for LLM inference, automatically caching repeated context like system prompts to reduce input token costs by up to 80% without code changes, with a detailed tutorial on break-even math.
Tokenless is a YC-backed startup that offers a drop-in replacement for API calls to LLMs, automatically routing requests to the most cost-effective model without sacrificing quality, potentially cutting inference costs in half.
The author shares a runbook for deploying in-house LLM inference on Kubernetes, based on their experience building the infrastructure at their organization.
This note explains the implementation of chunked prefill and mixed batching in nano-vllm-v1, and benchmarks performance against the previous version.
LMCache is an open-source KV cache management layer that separates cache I/O from compute, plugging into vLLM, SGLang, and TensorRT-LLM to achieve up to 14x faster time-to-first-token and 4x faster decoding by parallelizing cache lookups and sharing GPU memory.
Qwen3.6-27B model shows improved speculative decoding performance when using heavier quantization levels, enhancing inference efficiency.
RIS-Kernel introduces a model-agnostic sparse attention architecture (RIS) that reduces self-attention complexity from O(N^2) to O(N log N) for long-context LLM inference, enabling operation on commodity CPU hardware without GPU acceleration.
Minimax M3 support with MSA has been merged into llama.cpp, enabling inference for the Minimax M3 model using the MSA architecture.
BeeLlama.cpp v0.4.1 introduces KVarN, KV precision tail, and improved KV cache support, with benchmarks showing that using tail 1024 allows kvarn5 and q6_0 to match q8_0 precision while using less VRAM.
DKV is an open-source framework for compressing KV-cache during local LLM inference, providing a CLI and a technical report.
Hetzner has launched an experimental LLM inference API service, offering an OpenAI-compatible endpoint with the Qwen3.6-35B-A3B-FP8 model. The service is free during the experiment period, has no SLA, and is intended to gather user feedback.
Pulsar Attention replaces the static anchor in Star Attention with content-aware summaries and attention sinks, reducing FLOPs by 3.3x while outperforming dense attention on long-context benchmarks.