@PyTorch: Normalization layers often introduce memory-bound bottlenecks in large language models and recommendation systems due t…
Summary
Meta introduces techniques like Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention to fuse normalization operations with GEMM and Attention kernels, hiding up to 90% of normalization latency on NVIDIA B200 hardware and achieving up to 35% latency reduction in attention blocks.
View Cached Full Text
Cached at: 07/10/26, 08:14 PM
Normalization layers often introduce memory-bound bottlenecks in large language models and recommendation systems due to unique hardware tiling requirements. To address this, Meta has been working on techniques to fuse normalization operations directly with GEMM and Attention kernels. By overlapping CUDA Core normalization math with Tensor Core execution pipelines, notable speedups can be unlocked.
Multiple novel ideas and techniques have been introduced to tackle the key blocker to norm-related fusions: the tiling difference. These include Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention. Testing on real-world RecSys traffic and NVIDIA B200 hardware shows these techniques can hide up to 90% of normalization latency behind GEMM kernels. This yields up to a 35% latency reduction for full Attention blocks with pre-norm, post-norm, and residual connections.
This milestone shows what is possible when you combine algorithmic insights with low-level hardware optimization. We are moving closer to a world of free normalization, where developers get full modeling power without performance compromises.
Read the complete blog, link in comments
Similar Articles
@PyTorch: https://bit.ly/4yawNqB..*
This blog post from PyTorch presents novel kernel fusion techniques for normalization ops like LayerNorm and RMSNorm, achieving significant speedups by reducing memory-IO overhead. Techniques include Lazy Pre-Norm and Multi-CTA Norm Fusion, hiding up to 90% of normalization latency when fused with GEMMs, and the FlashNormAttention algorithm achieving up to 35% kernel speedup.
@PyTorch: PyTorch member Meta just open-sourced a GPU kernel that makes attention 2.3x faster on NVIDIA Blackwell. TLX Block Atte…
Meta open-sources TLX Block Attention, a warp-specialized Triton kernel that achieves 2.3x speedup for block-diagonal self-attention on NVIDIA Blackwell GPUs, with up to 3.5x speedup when fused with rotary embeddings.
@PyTorch: PyTorch-native NeMo AutoModel handles transformer pretraining in @nvidia's end-to-end workflow for building a transacti…
NVIDIA's blog post describes an end-to-end workflow using PyTorch-native NeMo AutoModel for pretraining a transaction foundation model. The workflow uses GPU-accelerated data processing and tokenization, decoder-only model pretraining, and embedding extraction to improve fraud classification performance by over 40% on the IBM TabFormer dataset.
Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models
Independent study shows 227M-parameter hypernetwork adds zero gain over well-crafted few-shot prompts for tool-use in 3B Llama, achieving 79.7% of GPT-5 performance at 10× lower latency.
The memory wall gets expensive: KV cache is why you should stop worshiping softmax attention
The article discusses how rising DDR5 memory prices signal a broader memory bottleneck in AI, particularly the KV cache in softmax attention for LLMs, and highlights post-transformer architectures like linear attention and state space models that aim to reduce memory usage.