Tag
SPECTRA is a training-free codec that re-encodes LLM KV caches via spectral transform to concentrate bit budgets on important channels, achieving near-lossless 4x compression and usable compression up to 12x, surpassing the 2-bit quantization cliff.
CommitKV proposes a lifecycle-aware KV cache compression method for multi-turn ReAct agents, distinguishing dormant tokens from completed ones via commit transitions to reduce memory use and speed up inference.
A blog post explaining that cache read costs dominate LLM inference spending for agentic workloads, with cumulative costs growing quadratically as context is re-read each turn, and advice to reduce tool call count to cut costs.
A user reports successfully running a 1M-token context on a single RTX 3090 using a Qwen-based 35B A3B model (17GB VRAM) with KVarN 4-bit KV-cache quantization in a BeeLlama.cpp fork, extracting 7 needles from different parts of the text.
CoinRAG is a new method for long-context RAG that reuses fine-grained contextualized information nugget KV caches instead of full chunks, improving efficiency and answering quality. It achieves a new Pareto frontier with 5.3% relative F1 improvement on LongBench multi-hop QA tasks.
AoH is a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections, enabling sparse attention without runtime attention scores. At 50% sparsity it retains 96.5% of full-attention performance while reducing prefill/decode latency and KV-cache memory.
Discovered a KV cache invalidation bug in DeepSeek V4 Flash with vLLM local deployment: after switching sessions, prefix cache hit rate drops to 0, causing re-prefill, with latency exploding from ~1 second to ~100 seconds. Root cause suspected to be vLLM calling _maybe_evict_cached_block() when reusing free queue blocks, which removes the cached index.
A Microsoft paper analyzing 13.5M GitHub Copilot sessions shows that coding-agent workloads are dominated by autonomous LLM call chains, with KV-cache and container idle time strongly dependent on turn/session structure, arguing for workflow-level scheduling instead of request-level policies.
The author shares insights from training a small model with DeepSeek's latent attention, observing layer-dependent latent usage and a test-time trick that reduces KV cache 4x without loss change.
OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.
A detailed report on quantizing LiquidAI's LFM2.5-2.6B model with various GGUF and KV cache quantizations, showing it fits on an 8GB Raspberry Pi with minimal degradation, but warning against Q4_K_M.
This paper introduces QEvict, a KV-cache management scheme for LLMs that uses recoverable quantized eviction to handle attention drift during long-context decoding, improving memory efficiency while preserving important historical context.
A developer shows how to run a 28M parameter LLM locally on an Amazon Echo Dot 2 using llama.cpp, achieving around 4 tokens/s generation and ~2.3s latency with prompt caching, enabling simple offline voice commands.
Technical post sharing performance stats for running Qwen 27B on 2x RTX 5070 Ti GPUs with vLLM cu129-nightly, achieving up to 94-87 tps decode and 170k GPU KV cache.
Introduces BinaryPC, a training-free hashing-based sparse attention method for long-context LLMs that uses binary principal components to construct hash codes, preserving accuracy while improving decoding throughput by 3.56x over FlashAttention.
This paper proposes NOVA-KV, a transform-coding approach to KV cache quantization that uses attention-preserving transforms to allocate bits where queries actually attend, improving long-context retrieval accuracy at low bit rates compared to prior methods.
Discusses the bottlenecks of using Transformer for large-scale inference and reviews the evolution of inference optimization technologies for large models from 2023 to 2026, including KV cache quantization, speculative decoding, architectural innovation, and hardware-software co-design.
AnchorKV is a new KV cache compression scheme that shrinks the cache by 20x without discarding any tokens, using anchor-residual representations to preserve 99% of full-cache accuracy at the 70B scale. The paper is a preprint under review.
Proposes OptR, an output-aware rotation method for INT2 KV-cache quantization that minimizes post-output attention error, improving QuaRot and OSCAR across models and benchmarks.
TaskPress introduces a query-agnostic KV cache compression framework that uses a task guide as a meta-query and quantization scale factors to prune irrelevant tokens, enabling reusable caches across diverse queries with negligible overhead.