Tag
SPECTRA is a training-free codec that re-encodes LLM KV caches via spectral transform to concentrate bit budgets on important channels, achieving near-lossless 4x compression and usable compression up to 12x, surpassing the 2-bit quantization cliff.
CommitKV proposes a lifecycle-aware KV cache compression method for multi-turn ReAct agents, distinguishing dormant tokens from completed ones via commit transitions to reduce memory use and speed up inference.
A detailed benchmark comparing 16 quantizations of Qwen3.6 27B across GGUF, NVFP4, AWQ, AutoRound, and FP8 formats, measuring KL divergence from the unquantized reference. Weight-only GGUF quants generally offer the best quality-size tradeoffs, while vLLM quants vary substantially.
A PSA about Meta's Muse Glimmer: setting reasoning strength via system prompt is partially overridden by the chat template's high default, and the absent default is high, not off. Using chat_template_kwargs.reasoning_strength provides more control.
A discussion on why speculative decoding matured in 2026 for LLM inference, citing Uber's early use, Apple and DeepMind papers, and Tri Dao's research, with observations on adoption in frameworks and local deployments.
AoH is a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections, enabling sparse attention without runtime attention scores. At 50% sparsity it retains 96.5% of full-attention performance while reducing prefill/decode latency and KV-cache memory.
CubicQuant proposes a parametric non-uniform scalar quantization format for LLM weights, using a monotonic cubic curve to adapt reconstruction levels at 1-8 bit widths while retaining dense integer code streams for GPU efficiency. Experiments show RMSE reductions over uniform and floating-point baselines, with preliminary H200 kernel measurements.
This paper introduces HYMELL, a hybrid analytical-machine-learning framework for estimating LLM inference latency and energy across prefill and decode phases, validated on NVIDIA H100 with under 5% error for LLaMA 3 8B.
A user seeks advice on building a budget home AI server with 32-48GB VRAM, debating between AMD RX 9060 XT and Nvidia RTX 5060 Ti GPUs, and whether to use AM5 or used EPYC platforms for local LLM inference and large MoE model offloading.
OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.
This preprint introduces DBLast, a dependent block drafter for stochastic speculative decoding, using a low-rank latent mixture over token positions and an acceptance-oriented training objective to improve accepted draft length in higher-entropy decoding regimes. Experiments with Qwen3-4B and Qwen3-8B show consistent improvements over independent block sampling.
This paper introduces QEvict, a KV-cache management scheme for LLMs that uses recoverable quantized eviction to handle attention drift during long-context decoding, improving memory efficiency while preserving important historical context.
vLLM, open-sourced by UC Berkeley's Sky Computing Lab, is a high-performance LLM inference and serving library. Through PagedAttention and continuous batching, it dramatically improves throughput and reduces VRAM waste, while being compatible with the OpenAI API and various hardware.
DeepSeek announced a significant API price hike, and analysis suggests the move goes beyond GPU cost pass-through to reflect broader market shifts toward value-based pricing and open-source ecosystem pressures.
A deep dive into vLLM's architecture and components for high-throughput LLM inference, covering scheduling, paged attention, continuous batching, advanced features, scaling, serving, and benchmarking.
Introduces BinaryPC, a training-free hashing-based sparse attention method for long-context LLMs that uses binary principal components to construct hash codes, preserving accuracy while improving decoding throughput by 3.56x over FlashAttention.
This paper proposes NOVA-KV, a transform-coding approach to KV cache quantization that uses attention-preserving transforms to allocate bits where queries actually attend, improving long-context retrieval accuracy at low bit rates compared to prior methods.
This paper presents the first architectural characterization of agentic AI workflows, revealing fragmented, heterogeneous execution patterns that mismatch conventional server designs, and introduces a prototype server called Agora to improve CPU/GPU utilization and throughput.
bootai is an open-source UEFI application that boots directly into an AI chat/code REPL, running Qwen2.5 and SmolLM model inference on bare metal without an OS, with hand-written network drivers and a TCP/IP stack.
Discusses the bottlenecks of using Transformer for large-scale inference and reviews the evolution of inference optimization technologies for large models from 2023 to 2026, including KV cache quantization, speculative decoding, architectural innovation, and hardware-software co-design.