@PyTorch: Normalization layers often introduce memory-bound bottlenecks in large language models and recommendation systems due t…

X AI KOLs Following Papers
normalization fusion gemm attention meta hardware-optimization latency-reduction

Summary

Meta introduces techniques like Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention to fuse normalization operations with GEMM and Attention kernels, hiding up to 90% of normalization latency on NVIDIA B200 hardware and achieving up to 35% latency reduction in attention blocks.

Normalization layers often introduce memory-bound bottlenecks in large language models and recommendation systems due to unique hardware tiling requirements. To address this, Meta has been working on techniques to fuse normalization operations directly with GEMM and Attention kernels. By overlapping CUDA Core normalization math with Tensor Core execution pipelines, notable speedups can be unlocked. Multiple novel ideas and techniques have been introduced to tackle the key blocker to norm-related fusions: the tiling difference. These include Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention. Testing on real-world RecSys traffic and NVIDIA B200 hardware shows these techniques can hide up to 90% of normalization latency behind GEMM kernels. This yields up to a 35% latency reduction for full Attention blocks with pre-norm, post-norm, and residual connections. This milestone shows what is possible when you combine algorithmic insights with low-level hardware optimization. We are moving closer to a world of free normalization, where developers get full modeling power without performance compromises. Read the complete blog, link in comments
Original Article
View Cached Full Text

Cached at: 07/10/26, 08:14 PM

Normalization layers often introduce memory-bound bottlenecks in large language models and recommendation systems due to unique hardware tiling requirements. To address this, Meta has been working on techniques to fuse normalization operations directly with GEMM and Attention kernels. By overlapping CUDA Core normalization math with Tensor Core execution pipelines, notable speedups can be unlocked.

Multiple novel ideas and techniques have been introduced to tackle the key blocker to norm-related fusions: the tiling difference. These include Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention. Testing on real-world RecSys traffic and NVIDIA B200 hardware shows these techniques can hide up to 90% of normalization latency behind GEMM kernels. This yields up to a 35% latency reduction for full Attention blocks with pre-norm, post-norm, and residual connections.

This milestone shows what is possible when you combine algorithmic insights with low-level hardware optimization. We are moving closer to a world of free normalization, where developers get full modeling power without performance compromises.

Read the complete blog, link in comments

Similar Articles

@PyTorch: https://bit.ly/4yawNqB..*

X AI KOLs Timeline

This blog post from PyTorch presents novel kernel fusion techniques for normalization ops like LayerNorm and RMSNorm, achieving significant speedups by reducing memory-IO overhead. Techniques include Lazy Pre-Norm and Multi-CTA Norm Fusion, hiding up to 90% of normalization latency when fused with GEMMs, and the FlashNormAttention algorithm achieving up to 35% kernel speedup.