Tag
CommitKV proposes a lifecycle-aware KV cache compression method for multi-turn ReAct agents, distinguishing dormant tokens from completed ones via commit transitions to reduce memory use and speed up inference.
The author adds Maple-Preview support to Mference, a tool that streams MoE experts from disk to run large models on low-RAM devices, achieving 40 tps with 500MB RAM on an Air M4.
Highlights a paper introducing ZeRO (Zero Redundancy Optimizer), a memory optimization system for efficiently training extremely large models on limited GPU memory.
Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.
AnchorKV is a new KV cache compression scheme that shrinks the cache by 20x without discarding any tokens, using anchor-residual representations to preserve 99% of full-cache accuracy at the 70B scale. The paper is a preprint under review.
Discusses running Gemma 4 on a device with only 500MB of memory, likely through quantization or other optimization techniques.
Technical write-up on running DeepSeek-V4-Flash-0731 with full 1M context on a single RTX 5090 + 256GB DDR5 desktop using vLLM with CPU/RAM offloading, achieving ~800 tps prefill and ~15 tps decode.
Demonstrates running an 80B Qwen model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone, showcasing extreme memory optimization for local LLM inference.
PyTorch 2.13 release brings FlexAttention to Apple Silicon with up to 12x speedups and 4x peak memory reduction for large-vocabulary models via fused LinearCrossEntropyLoss, alongside updates to distributed training, compilation, and on-device inference.
Testing a 1.56TB Mixture-of-Experts model on a 6GB RTX 4050 laptop, requiring patched memory streaming with NVMe to achieve 0.106 tokens/s decode speed.
Valkey uses different encodings like listpack and hashtable for its data types to balance memory and performance; understanding these internal structures can lead to significant memory savings.
AirLLM is an open-source tool that enables running 405-billion-parameter models on a single 8GB gaming GPU by streaming layers, priced at free and Apache 2.0 licensed.
Google open-sourced a vector index that compresses 31GB of AI memory to 4GB, fitting 10 million documents and searching faster than FAISS, with zero training or GPU required.
A technique to remove 423 GB from GLM-5.2 (753B weights) without quantization or retraining, achieving bit-exact compression by keeping weights compressed in VRAM.
MosaicKV introduces dynamic two-dimensional KV cache compression for long-context LLM serving, achieving up to 16x attention speedup and 3x memory reduction with minimal accuracy loss.
Proposes Block-GTQ, a RoPE-aware bit allocation method for key-value cache quantization that improves long-context performance and memory efficiency by allocating more bits to high-energy RoPE blocks.
A visual deep dive into FlashAttention, explaining memory optimization and operator fusion for efficient attention computation in language model training.
Reverse engineering the Qualcomm NPU compiler reveals undocumented VTCM memory management, MILP-based placement, automatic precision alteration, and a hidden analytical simulator (Hextimate) for edge deployment optimization.
LMCache is a KV cache management layer that accelerates large model inference and reduces VRAM consumption by caching and reusing KV cache. It has received 9.2K stars and joined the PyTorch Foundation, and is integrated by NVIDIA Dynamo.
NVFP4 KV cache quantization on sm120 significantly improves memory efficiency for large language models, enabling 32GB VRAM systems to achieve ~60 tok/sec inference at 196k context size with Qwen3.6-27B.