Tag
torch-preflight is a new linter for PyTorch code that catches common GPU-wasting bugs without executing code or requiring a GPU, plus estimates VRAM usage to check if a training run fits before paying for an instance.
A developer shares their day 6 of inference engineering, writing a CUDA kernel for 1D convolution, explaining PagedAttention's memory efficiency, and providing an overview of GPU memory hierarchy (global, register, local, constant, shared).
Samsung published a paper on offloading KV-cache over a CXL memory pool, demonstrating that even with early-gen CXL hardware, GPUs can be fed as effectively as with real DRAM, making the approach easily replicable.
Unsloth releases a new version of Qwen3.6 27B NVFP4 quantization scheme, introducing FP8_E4M3 intermediate precision layer and refined weight protection, achieving 2.5x speed improvement on 24GB VRAM, while improving accuracy and tool-calling capabilities.
Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers that uses hierarchical two-level routing to enable exact attention over a small routed working set, allowing models like Qwen3-30B to run at 64K context on a single RTX 5090 with minimal quality loss.
A thread sharing practical tips for freeing up GPU memory in llama.cpp, such as offloading mmproj to CPU and adjusting KV cache types, while discussing parameters like --cache-type-k/v and --spec-draft-n-max.
A detailed blog post explaining how vLLM works, including PagedAttention, KV cache management, and continuous batching for efficient LLM serving.
A tool that estimates which LLMs fit on a user's GPU memory, ranking models by performance while considering memory constraints and quantization levels.
InfiniteKV is an open-source KV cache technique that compresses old tokens into 104-byte searchable records stored in RAM or on disk, enabling models to handle million-token contexts beyond their trained window without discarding data. Verified working with Mistral-7B and SmolLM2.
Proposes Lookahead Sparse Attention with a Neural Memory Indexer on DeepSeek-V4, reducing GPU memory usage to ~13.5% of full-context baseline while maintaining or slightly improving accuracy.
A practical guide explaining how to calculate VRAM requirements for LLMs based on parameter count and quantization level, plus additional overhead from KV cache, activations, and batching.
This paper presents an empirical study on scheduling multiple LLMs on shared heterogeneous hardware, focusing on performance implications of CPU-GPU offloading and preemption. It finds that offloading causes non-linear decode degradation, especially for smaller models, and preemption overhead is dominated by model state reload, providing design guidance for future multi-model schedulers.
Discusses the trade-off between dense and Mixture-of-Experts (MoE) models for local AI, noting that high-RAM users have limited MoE options beyond Qwen 3.5 122B, and questioning if large GPU is the only viable path.
IceCache introduces Dynamic Continuous Indexing to keep GPU memory usage constant during long LLM generations with minimal accuracy loss.