Tag
A tweet explains the optimization principle behind Flash attention, focusing on minimizing data movement and maximizing on-chip memory reuse, in the context of a language modeling course lecture.
A tweet asking why FlashAttention achieves both speed and GPU memory efficiency in AI computations.
This pull request introduces Flash Attention tuning optimizations for CUDA/HIP in the llama.cpp project, targeting gfx1201 hardware to enhance inference performance.
An educational overview of key concepts in LLM inference engineering, covering techniques like KV cache, PagedAttention, Flash Attention, and continuous batching to optimize inference performance.
A discussion on why speculative decoding matured in 2026 for LLM inference, citing Uber's early use, Apple and DeepMind papers, and Tri Dao's research, with observations on adoption in frameworks and local deployments.
Astral is open sourcing its build pipelines for pre-built wheels of GPU-enabled Python packages like FlashAttention and DeepSpeed, making them available to all via standard Python indexes.
Modified SGLang to support Qwen and Laguna models on V100 GPUs using custom FlashAttention and Marlin kernels, achieving decent throughput on 4xV100 hardware.
Google updated Gemma 4's chat templates with major fixes to tool calling, reduced laziness, enabled Flash Attention 4 on Hopper GPUs, and released an interactive vision guide. The updates are available on Hugging Face.
An educational thread explaining that the main bottleneck in LLM inference is data movement, not computation, and highlighting techniques like quantization, KV cache optimization, and FlashAttention to reduce memory traffic.
Introduces Flash-MSA, the first performant open-source training kernels for MiniMax Sparse Attention on Hopper and Blackwell GPUs, enabling efficient million-token training.
A tweet recommending studying FlashAttention for LLM inference, highlighting its importance in optimizing GPU memory traffic and speeding up attention mechanisms, with links to the GitHub repository and papers for FlashAttention, FlashAttention-2, and FlashAttention-3.
AI model Claude was used to write a FlashAttention forward kernel using the pyptx DSL, achieving near-parity performance with hand-tuned FlashAttention-4 on NVIDIA B200 hardware.
Claude Fable used the pyptx DSL to write a FlashAttention forward kernel for NVIDIA B200 that achieves near-parity performance with the hand-tuned CUTLASS kernel, demonstrating the potential for AI agents in compiler and DSL design.
Hybrid Sliding Window Attention (Hybrid SWA) is a mixed attention mechanism in long-context language models that balances computational efficiency with full long-range dependencies. By alternating between local SWA layers and global attention layers, it significantly compresses KV cache while maintaining inference capability. This article details its design principles, application in models such as Gemma and Qwen, and best practices in open-source projects like vLLM and HuggingFace.
Proposes mechanism-driven monitors for preemptive detection of LLM training instability by deriving internal signals from low-precision flash attention and MoE routers, enabling detection thousands of steps before loss divergence.
A new book from CMU's Machine Learning Systems course teaches modern GPU programming for ML systems, covering Blackwell architecture, GEMM, and FlashAttention using the TIRx Python DSL.
NVIDIA Research publishes a technical blog post examining KV cache compression techniques and their infrastructure problems, including how FlashAttention and paged attention create practical obstacles for production deployment of long-context LLMs, with a proposed geometric solution using RoPE.
Discussion about rewriting parallelism to improve kernel performance using CuTe DSL and tile programming models for the FA4 (FlashAttention 4) kernel.
Explains that inference kernels differ from training, with Flash Attention 4 focusing on changing parallelism across KV and supporting small irregular loads.
Parallax is a new parametrized form of Local Linear Attention that eliminates numerical solvers and matches FlashAttention 2/3 in decoding. Its effectiveness depends on the optimizer, working with Muon but not AdamW, highlighting the role of optimizer geometry.