Tag
AcceptMoE is a verifier-side expert selection method for speculative decoding of MoE LLMs, which reduces expert-weight traffic and improves throughput by up to 2.06x under expert offloading with minimal accuracy loss.
ATFlash introduces a per-RoPE-wavelength distance window that prunes query-key inner-product terms proportional to each frequency pair's wavelength, cutting 37-48% of attention compute with minimal quality loss and up to 1.31x speedups on long-context LLM inference.
AnchorKV is a new KV cache compression scheme that shrinks the cache by 20x without discarding any tokens, using anchor-residual representations to preserve 99% of full-cache accuracy at the 70B scale. The paper is a preprint under review.
Proposes OptR, an output-aware rotation method for INT2 KV-cache quantization that minimizes post-output attention error, improving QuaRot and OSCAR across models and benchmarks.
TaskPress introduces a query-agnostic KV cache compression framework that uses a task guide as a meta-query and quantization scale factors to prune irrelevant tokens, enabling reusable caches across diverse queries with negligible overhead.
A developer updates MLX-Serve, a fast local inference server for Apple Silicon, to support recent models like LiquidAI 2.6B, MiniMax H3 video generation, and DeepSeek V4 Flash, with AntLing 3.0-flash coming soon.
SK Hynix introduces CMM-Ax, an ASIC-based CXL-PNM solution developed with Marvell Technology, designed to overcome memory bottlenecks in long-context LLM inference, achieving up to 5.5× higher throughput than GPU-only systems.
This paper introduces S4R, a low-rank KV cache compression method that combines selective token sampling, subspace construction, and sparse reconstruction to achieve up to 5× compression with near full-cache accuracy on LongBench and RULER benchmarks.
Demonstrates running an 80B Qwen model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone, showcasing extreme memory optimization for local LLM inference.
A tweet thread shares a curated list of resources for learning LLM inference, covering tokenization, GPU hardware, optimization techniques like quantization and speculative decoding, and inference engines such as vLLM and SGLang, followed by an Nvidia architect's talk summary.
A paper introducing predictive speculative KV replication to handle bursty LLM inference workloads, with code available on GitHub.
This paper proposes a KV cache eviction strategy that scores tokens by counter-causal surprise, removing past tokens that are well-predicted from future context. The method is training-free, in-distribution, and achieves competitive performance with a fast single-layer approximation.
Prox is a training-free framework for sparse SwiGLU FFNs in LLMs, using approximate intermediate-channel salience to build channel masks without dense computation. It outperforms training-free baselines across ten LLMs, achieving up to 1.99x end-to-end decoding speedup at 70% sparsity.
This paper introduces SparseSpec-L, a training-free self-speculative decoding framework for long-context LLM inference that uses a dynamically sparsified and recallable KV cache along with an entropy-based controller for speculation length, achieving up to 2.79x speedup over autoregressive decoding.
This paper proposes functional reconstruction for converting MHA/GQA checkpoints into MLA draft models for speculative decoding, directly optimizing attention modules to preserve token acceptance. It reports consistent improvements across 192 configurations involving Llama/Qwen models and multiple conversion methods.
Introduces Tritium, an open-source Rust/CUDA engine for ternary (1.58-bit) quantization, serving, and training of LLMs on consumer GPUs. It claims faster inference than llama.cpp for ternary models and introduces a new quantization method called SALT.
This paper analyzes lossy verification schemes in speculative decoding, categorizing them into truncation-based and collaborative verification, and identifies pitfalls and principles for maintaining generation quality.
DigitalOcean introduces prompt caching for LLM inference, automatically caching repeated context like system prompts to reduce input token costs by up to 80% without code changes, with a detailed tutorial on break-even math.
Tokenless is a YC-backed startup that offers a drop-in replacement for API calls to LLMs, automatically routing requests to the most cost-effective model without sacrificing quality, potentially cutting inference costs in half.
The author shares a runbook for deploying in-house LLM inference on Kubernetes, based on their experience building the infrastructure at their organization.