Tag
A free "GPU Kernel Engineering" course series of 44 lessons has been released, starting from CPU parallel computing and CUDA basics, and progressively diving into memory optimization, GEMM, Tensor Cores, Flash Attention, and all the way to cutting-edge hardware like H100/Blackwell and AI-written kernels.
rdna35-perf branch of strix-llama.cpp 为 Strix Halo (Radeon 8060S) 带来新的 RDNA3.5 flash attention 与 GEMM 内核优化，prompt processing 比 Vulkan 快 21–54%，在 128k 长上下文下比原版 ROCm 最快达 2.7 倍，且输出质量与原版 ROCm 基本一致。
PyTorch/Meta blog reveals Jagged Flash Attention built with TLX (Triton Low-level Extensions) on NVIDIA Blackwell B200, using ~3.2K lines of Triton code—3x less than FlashAttention-4—while outperforming FA4 by ~13% forward and ~50% backward on jagged attention shapes used in Meta's Generative Ads Model.
A tweet explains the optimization principle behind Flash attention, focusing on minimizing data movement and maximizing on-chip memory reuse, in the context of a language modeling course lecture.
A tweet asking why FlashAttention achieves both speed and GPU memory efficiency in AI computations.
This pull request introduces Flash Attention tuning optimizations for CUDA/HIP in the llama.cpp project, targeting gfx1201 hardware to enhance inference performance.
An educational overview of key concepts in LLM inference engineering, covering techniques like KV cache, PagedAttention, Flash Attention, and continuous batching to optimize inference performance.
A discussion on why speculative decoding matured in 2026 for LLM inference, citing Uber's early use, Apple and DeepMind papers, and Tri Dao's research, with observations on adoption in frameworks and local deployments.
Astral is open sourcing its build pipelines for pre-built wheels of GPU-enabled Python packages like FlashAttention and DeepSpeed, making them available to all via standard Python indexes.
Modified SGLang to support Qwen and Laguna models on V100 GPUs using custom FlashAttention and Marlin kernels, achieving decent throughput on 4xV100 hardware.
Google updated Gemma 4's chat templates with major fixes to tool calling, reduced laziness, enabled Flash Attention 4 on Hopper GPUs, and released an interactive vision guide. The updates are available on Hugging Face.
An educational thread explaining that the main bottleneck in LLM inference is data movement, not computation, and highlighting techniques like quantization, KV cache optimization, and FlashAttention to reduce memory traffic.
Introduces Flash-MSA, the first performant open-source training kernels for MiniMax Sparse Attention on Hopper and Blackwell GPUs, enabling efficient million-token training.
A tweet recommending studying FlashAttention for LLM inference, highlighting its importance in optimizing GPU memory traffic and speeding up attention mechanisms, with links to the GitHub repository and papers for FlashAttention, FlashAttention-2, and FlashAttention-3.
AI model Claude was used to write a FlashAttention forward kernel using the pyptx DSL, achieving near-parity performance with hand-tuned FlashAttention-4 on NVIDIA B200 hardware.
Claude Fable used the pyptx DSL to write a FlashAttention forward kernel for NVIDIA B200 that achieves near-parity performance with the hand-tuned CUTLASS kernel, demonstrating the potential for AI agents in compiler and DSL design.
Hybrid Sliding Window Attention (Hybrid SWA) is a mixed attention mechanism in long-context language models that balances computational efficiency with full long-range dependencies. By alternating between local SWA layers and global attention layers, it significantly compresses KV cache while maintaining inference capability. This article details its design principles, application in models such as Gemma and Qwen, and best practices in open-source projects like vLLM and HuggingFace.
Proposes mechanism-driven monitors for preemptive detection of LLM training instability by deriving internal signals from low-precision flash attention and MoE routers, enabling detection thousands of steps before loss divergence.
A new book from CMU's Machine Learning Systems course teaches modern GPU programming for ML systems, covering Blackwell architecture, GEMM, and FlashAttention using the TIRx Python DSL.
NVIDIA Research publishes a technical blog post examining KV cache compression techniques and their infrastructure problems, including how FlashAttention and paged attention create practical obstacles for production deployment of long-context LLMs, with a proposed geometric solution using RoPE.