Tag
A discussion of NVIDIA's TriAttention research on KV cache compression in vLLM, arguing that real gains come from freeing GPU memory blocks rather than evicting tokens, with reported 2.5× faster decoding and 10.7× less KV cache memory.
vLLM's nightly build now supports PLE disk offload, helping offload KV-cache or memory pressure to disk for LLM inference serving. The update is available on vLLM's nightly channel.
Part 6 of an open-source Inference Engineering repo is released, covering KV cache memory allocation, growth, release, and capacity-aware scheduling using real model inference and DynamicCache.
PyTorch Conference North America in San Jose (Oct 20-21) will feature a poster by Bongwoo Bak of Moreh on KV cache block placement, covering vLLM offloading strategies between GPU, CPU, and shared storage. The post also promotes conference registration tiers and pricing.
DisaggKD 提出将教师与学生模型分离到不同节点上进行知识蒸馏，并设计一种融合损失在行窗口与词汇表分片上等价于全词汇表损失，从而突破单 GPU 内存限制，在 Qwen3.5 等大模型对上实现与同位部署相当的损失且训练更快。
This repository helps DGX Spark users utilize a spare GPU to free up memory for better context or quantization quality by offloading the draft model via remote inference with vllm modifications.
SlideDP presents an asynchronous data-parallel runtime for host-resident layer streaming that fine-tunes LLMs beyond GPU memory across multiple GPUs, achieving 1.46–2.64x throughput over SlideFormer, MegaTrain, and ZeRO-Offload and even exceeding FSDP2 peak throughput while fine-tuning Qwen2.5-72B on four RTX 4090s.
KVCMAS introduces an efficient KV cache correction framework for multi-agent systems that reduces computation and memory overhead while achieving faster inference and maintaining accuracy.
Nokia Applied Research has open-sourced WiSP, a vLLM plug-in for running MoE models on GPUs with insufficient VRAM by intelligently paging experts and reallocating memory with the KV cache, achieving up to 2.0× higher decode throughput.
This paper introduces methods to bound four key memory peaks in large Mixture-of-Experts training, enabling training at 1M context length with fixed GPU memory and up to 10.4× throughput improvement over baselines.
This article explains how GPU memory is utilized during large language model (LLM) inference, breaking it down into four key components: model weights, KV cache, activations/workspace, and runtime overhead. It highlights the importance of quantization in optimizing memory usage for better performance.
This paper introduces techniques to manage memory peaks in training large Mixture-of-Experts models with long context lengths, including Pipelined LLEP, Ring-DTP, SCO, and OffloadStreamAdamW, which enable fixed GPU working sets and improve throughput up to 10.4x.
By implementing a Rust-based daemon to dynamically sleep inactive AI models, GPU memory usage in local agents can be reduced from 122 GB to 43 GB with sub-200ms wakeup latency, making it viable to run multiple models on a single workstation.
System76 launches the Thelio Mira AI Linux Workstation, featuring up to 192 GB of GPU memory for AI and high-performance computing applications.
At PyTorch Conference North America, Ricardo Noriega de Soto and Alexander Brooks will demonstrate how extending vLLM's prefix caching mechanism to multistage pipelines boosts inference speeds while reducing GPU memory overhead, providing practical strategies for optimizing complex AI workloads.
This article details the process of GPU memory writes, tracing the STG.E instruction through various hardware components like the load/store unit, coalescer, and L1 cache on an RTX 4090.
NVIDIA's open kernel driver for DGX Spark received two fixes that return freed GPU memory to the OS when a process exits and enable huge pages for GPU page faults on system memory, boosting first-touch bandwidth from 0.4 to 19.6 GiB/s.
TOPAS is a workflow-aware prefix-state scheduling method for multi-agent LLM serving that optimizes cache usage and request scheduling to reduce job completion time.
Benchmarking shows that 4-bit quantized Qwen3.8 27B retains performance on benchmarks like Terminal-Bench 2.1 while fitting on 24GB GPUs, but 1-bit quantization causes severe degradation.
PagedAttention applies virtual memory concepts to the KV cache in language models, reducing memory fragmentation and enabling inference engines to serve 2-4 times more users per GPU.