reading about tile-level activation overlap today

https://t.co/TQmsC1xJgI https://t.co/KBaJuO9gfU

---

Tile-Level Activation Overlap for Efficient LLM Inference

Source: https://arxiv.org/html/2607.02521

Abstract

SwiGLU is the dominant MLP activation in modern large language models, yet its intermediate tensor materialization costs 9–37% of MLP execution time. We present two complementary CUTLASS-based SM90 kernels that fuse SwiGLU into GeMM at the tile level. Kernel-1 overlaps Swish computation on the Gate accumulator with Up-tile loading using the Pingpong warp-specialized schedule; Kernel-2 interleaves SwiGLU with tile stores via a custom Epilogue Visitor Tree. Evaluated on Qwen-2.5 models (0.5B–72B) on NVIDIA H100, our kernels achieve up to 2.47×\timesspeedup over PyTorch, shifting workloads from memory-bound to compute-bound and reaching 79.5% peak BF16 utilization. We demonstrate thattorch.compilecannot replicate this fusion (3–7×\timesslower than our kernels), validating the need for hand-crafted tile-level design. Our fused kernels are also numerically superior, achieving zero mismatches compared to 4.5–11% for cuBLAS.

1Introduction

SwiGLU6has become the dominant activation function in modern large language models. Qwen-2.510, LLaMA9, Mistral, and Gemma all employ the gated MLP structure:Gate=A×W1\text{Gate}=A\times W_{1},Up=A×W2\text{Up}=A\times W_{2},Y=SiLU​(Gate)⊙UpY=\text{SiLU}(\text{Gate})\odot\text{Up}. This pattern requires two independent matrix multiplications followed by an element-wise gated activation, materializing two full intermediate tensors (GateandUp) in high-bandwidth memory (HBM) between the GeMM and activation stages.

As tensor core compute density increases through quantization (FP8, INT4) and architectural improvements, the relative cost of memory-bound operations grows. We profile the SwiGLU MLP on NVIDIA H100 and find that the activation computation and its associated intermediate tensor materialization consume9–37% of total MLP execution timedepending on model size (Figure1). For edge-deployment models (Qwen-2.5 0.5B), SwiGLU accounts for over 30% of MLP time — a substantial overhead that will only worsen as GeMM arithmetic becomes cheaper relative to memory traffic.

Figure 1:SwiGLU activation as a fraction of total MLP execution time across Qwen-2.5 model sizes. Smaller models spend up to 37% of MLP time on SwiGLU and intermediate materialization, motivating tile-level fusion.Existing compiler infrastructure cannot address this bottleneck. PyTorch’storch.compilewith maximum optimization is unable to fuse across two separate GeMMs with different weight matrices — a fundamental limitation of graph-level fusion passes. Our experiments show thattorch.compileachieves only 34–94% of eager PyTorch performance for this pattern, and explicit fusion hints provide no meaningful improvement (<<4% change). This validates the need for hand-crafted, hardware-aware kernel design.

Inspired by FlashAttention’s1success in eliminating intermediate materialization for attention, we apply IO-aware kernel design to the MLP block. However, the MLP fusion challenge is fundamentally different: it involvestwo independent GeMMswith separate weight matrices that must be coordinated, rather than a single attention computation.

We present two complementary CUTLASS-based SM90 kernels7that fuse SwiGLU into GeMM at the tile level:

1. 1.First fine-grained GeMM-SwiGLU fusionat the tile level using warp-specialized scheduling — Kernel-1 overlaps Swish computation with Up MMA during the Pingpong schedule’s consumer phase, creating[M,N][M,N]threadblocks optimized for large batch sizes.
2. 2.Complementary dual-kernel approach— Kernel-2 uses a custom Epilogue Visitor Tree (PairMulStore) to interleave SwiGLU with tile stores, creating[M,2​N][M,2N]threadblocks that provide2×2\timesbetter occupancy for small batch sizes.
3. 3.Systematic experimental evaluationacross 4 model sizes×\times5 batch sizes showing up to 2.47×\timesspeedup over PyTorch, with roofline analysis explaining how fusion shifts workloads from memory-bound to compute-bound (reaching 79.5% of peak BF16 utilization).
4. 4.Demonstration that compiler infrastructure cannot replicate this—torch.compilewith all fusion hints is 3–7×\timesslower than our kernels, and our fused kernels are also numerically superior (0 mismatches vs. 4.5–11% for cuBLAS).

2Related Work

2.1SwiGLU and Gated Activations

Shazeer6introduced gated linear unit variants (GEGLU, SwiGLU, ReGLU) as replacements for the standard FFN activation, demonstrating consistent quality improvements. SwiGLU, which computesSiLU​(x​W1)⊙(x​W2)\text{SiLU}(xW_{1})\odot(xW_{2})wherex​W1xW_{1}is the gate projection andx​W2xW_{2}is the up projection, has since become the default activation in LLaMA9, Qwen-2.510, Mistral, and Gemma. The two-input gated structure that makes SwiGLU effective — requiring both a Gate and Up projection through separate weight matrices — is precisely what creates the fusion challenge our work addresses. Prior optimizations have treated SwiGLU as a lightweight element-wise kernel launched after the GeMMs; we show that for small-to-medium models, this element-wise operation and its intermediate materialization constitute up to 37% of MLP time.

2.2IO-Aware Kernel Design

FlashAttention1;2established the paradigm of IO-aware GPU kernel design for transformers, reducing attention’s HBM accesses fromO⁡(N2)O(N^{2})toO⁡(N2/M)O(N^{2}/M)by tiling computation to fit in SRAM and never materializing the full attention matrix. Our work extends this IO-awareness principle from the attention block to the MLP block. However, the fusion challenge differs fundamentally: FlashAttention fuses operations within a single computation flow (Q​KT→softmax→VQK^{T}\rightarrow\text{softmax}\rightarrow V), whereas our kernels must coordinate outputs fromtwo independent GeMMswith separate weight matrices before applying the gated activation. This requires novel scheduling strategies (Pingpong overlap, dual threadblock grids) not needed in the attention case.

2.3GPU Kernel Optimization Frameworks

CUTLASS 3.x7provides the substrate for our implementation, offering warp-specialized scheduling (Pingpong, Cooperative), Epilogue Visitor Trees (EVTs) for composable post-GeMM operations, and TMA for hardware-accelerated async data movement on SM90. Our work demonstrates two novel uses of CUTLASS: (1) inserting activation computation between two GeMM phases within the Pingpong consumer loop (Kernel-1), and (2) extending EVTs with a custom PairMulStore node that fuses a gated activation during the store phase (Kernel-2).

Triton8offers a higher-level alternative for kernel development, operating on tile-level abstractions with compiler-managed scheduling. While Triton enables rapid prototyping of fused kernels, it provides insufficient control over warp-level scheduling to implement our fine-grained MMA-activation overlap. A Triton-based SwiGLU fusion would be limited to basic epilogue fusion without the tile-level temporal overlap that drives our largest speedups.

2.4LLM Serving and Inference Systems

Production LLM serving systems employ varying levels of kernel fusion. TensorRT-LLM5uses graph-level pattern matching with pre-compiled fused kernels for common operations, but its MLP fusion operates at a coarser granularity than our tile-level approach. vLLM3and SGLang use PyTorch’s default execution path (cuBLAS for GeMM, separate SwiGLU kernel), representing exactly the baseline our kernels improve upon. Megatron-LM4implements fused bias+GeLU but not the full SwiGLU+GeMM fusion, as its focus is on distributed training rather than single-GPU inference optimization.

Our fused kernels are designed as drop-in replacements: they accept the same input tensors (AA,W1W_{1},W2W_{2}) and produce the same output (YY) as the unfused baseline, enabling integration into any of these serving frameworks without architectural changes.

3Method

We present two complementary CUTLASS-based SM90 kernels that fuse SwiGLU activation into the GeMM computation at the tile level. Both kernels eliminate intermediate tensor materialization to HBM, but employ different strategies for overlapping activation computation with matrix arithmetic.

3.1Background: SwiGLU MLP Structure

The MLP block in modern LLMs using SwiGLU6computes:

Gate=A×W1∈ℝM×N\displaystyle=A\times W_{1}\in\mathbb{R}^{M\times N}(1)Up=A×W2∈ℝM×N\displaystyle=A\times W_{2}\in\mathbb{R}^{M\times N}(2)Y\displaystyle Y=SiLU​(Gate)⊙Up\displaystyle=\text{SiLU}(\text{Gate})\odot\text{Up}(3)whereA∈ℝM×KA\in\mathbb{R}^{M\times K}is the input activation,W1,W2∈ℝK×NW_{1},W_{2}\in\mathbb{R}^{K\times N}are separate weight matrices,SiLU​(x)=x⋅σ​(x)\text{SiLU}(x)=x\cdot\sigma(x)is the Swish activation, and⊙\odotdenotes element-wise multiplication.

Memory Traffic Problem.

In a standard (unfused) implementation, the computation requires 8 HBM operations for the intermediate tensors:

1. 1.Write Gate to HBM (M×N×2M\times N\times 2bytes, BF16)
2. 2.Write Up to HBM (M×N×2M\times N\times 2bytes)
3. 3.Read Gate from HBM for SwiGLU
4. 4.Read Up from HBM for SwiGLU
5. 5.WriteYYto HBM (M×N×2M\times N\times 2bytes)

plus the reads ofAA,W1W_{1}, andW2W_{2}. The four intermediate operations (items 1–4) transfer4×M×N×24\times M\times N\times 2bytes that could be eliminated if SwiGLU were computed while tiles remain in registers or shared memory.

UnfusedAAGeMM1Gate (HBM)GeMM2Up (HBM)SwiGLUYY(HBM)wrwrrdrdwrFusedAAFusedGeMM+SwiGLUYY(HBM)wr4 HBM ops eliminated8 HBM ops (4 intermediate)4 HBM ops onlyFigure 2:Memory access pattern comparison.Left:Unfused SwiGLU requires writing/reading intermediate Gate and Up tensors through HBM (8 total HBM operations).Right:Fused kernel computes SwiGLU in registers, eliminating 4 intermediate HBM operations.

Quantifying the Bottleneck.

Our profiling (Section5) shows that SwiGLU and its associated intermediate memory traffic consume 9–37% of total MLP execution time on H100, with larger fractions for smaller models: 30% for Qwen-2.5 0.5B versus 9% for Qwen-2.5 72B. As tensor core arithmetic intensity increases with quantization (FP8, INT4), this memory-bound activation overhead will become an even larger relative bottleneck.

Arithmetic Intensity Analysis.

For the unfused baseline, the arithmetic intensity is:

AIunfused=2⋅2​M​K​N+2​M​N2​(M​K+2​K​N+4​M​N+M​N)⋅2\text{AI}_{\text{unfused}}=\frac{2\cdot 2MKN+2MN}{2(MK+2KN+4MN+MN)\cdot 2}(4)where the numerator counts FLOPs (two GeMMs of2​M​K​N2MKNeach plus2​M​N2MNfor SwiGLU) and the denominator counts bytes transferred (inputAA, two weights, four intermediate transfers, and outputYY, all in BF16). By fusing, we eliminate4​M​N4MNelements of intermediate traffic:

AIfused=4​M​K​N+2​M​N2​(M​K+2​K​N+M​N)⋅2\text{AI}_{\text{fused}}=\frac{4MKN+2MN}{2(MK+2KN+MN)\cdot 2}(5)This increases arithmetic intensity by 6–247% depending on the ratioM/KM/K(Table3in Section5.3).

3.2Kernel-1: Sync SwiGLU via Pingpong Schedule Overlap

Pingpong Warp-Specialized Schedule.

CUTLASS 3.x’s SM90 Pingpong schedule7partitions warps within a threadblock intoproducersandconsumers. Producers issue TMA (Tensor Memory Accelerator) loads to fill shared memory buffers asynchronously, while consumers execute MMA (Matrix Multiply-Accumulate) instructions on previously loaded tiles. The schedule alternates (“ping-pongs”) between two shared memory buffers, overlapping loads of the next tile with computation on the current tile.

Key Insight: Swish During Consumer Off-Phase.

Within the Pingpong schedule, after the consumer warp completes MMA on a tile and before the next tile’s data arrives, there exists a brief window where consumer warps are idle (waiting on the producer’s TMA load). We exploit this window to compute the Swish activationSiLU​(Gatetile)=Gatetile⋅σ⁡(Gatetile)\text{SiLU}(\text{Gate}_{\text{tile}})=\text{Gate}_{\text{tile}}\cdot\sigma(\text{Gate}_{\text{tile}})on the accumulated Gate tile stored in registers.

TimeProducerTMA GateTMA UpTMA GateTMA UpConsumerMMA GateSwishMMA UpStoreYYOverlapFigure 3:Kernel-1 Pingpong schedule timeline. Swish computation (green) on the Gate accumulator overlaps with the producer’s TMA loads for Up tiles, hiding activation latency within the load-compute pipeline.

Threadblock Design.

Kernel-1 creates an[M,N][M,N]threadblock grid. Each threadblock sequentially computes:

1. 1.Gate tile: MMA ofAtile×W1,tileA_{\text{tile}}\times W_{1,\text{tile}}, accumulating across theKKdimension
2. 2.Swish computation:SiLU​(Gatetile)\text{SiLU}(\text{Gate}_{\text{tile}})computed in registers during the synchronization barrier between Gate and Up phases
3. 3.Up tile: MMA ofAtile×W2,tileA_{\text{tile}}\times W_{2,\text{tile}}, accumulated similarly
4. 4.Epilogue: Element-wise multiplicationSiLU​(Gatetile)⊙Uptile\text{SiLU}(\text{Gate}_{\text{tile}})\odot\text{Up}_{\text{tile}}followed by store to HBM

The critical advantage is that Swish computation (step 2) overlaps temporally with the producer warp’s TMA loads for the Up weight tiles. Since Swish involves only element-wise operations on registers (sigmoid approximation and multiply), it completes within the load latency without extending the critical path.

Epilogue Efficiency.

Because both Gate (post-Swish) and Up accumulator results reside in registers within the same threadblock, the final element-wise multiply and store to HBM are maximally efficient — a single fused store operation writes the final outputYYwith no additional global memory reads. This makes Kernel-1 particularly effective at large batch sizes (M≥2048M\geq 2048) where the[M,N][M,N]grid provides sufficient threadblocks to saturate the GPU’s Streaming Multiprocessors (SMs).

Implementation.

We implement Kernel-1 as a customGemmSwiGLUCollective Builder that extends CUTLASS’sCollectiveMainloopwith a modified consumer loop. The builder inserts Swish computation between the two GeMM phases, coordinating viacute::cp_async_fencebarriers to ensure the Gate accumulator is complete before applying Swish, and that Swish completes before Up MMA stores overwrite shared memory buffers.

3.3Kernel-2: Interleave SwiGLU via Custom Epilogue Visitor Tree

Epilogue Visitor Trees (EVTs).

CUTLASS 3.x provides Epilogue Visitor Trees7— a composable framework for expressing post-GeMM operations as a directed acyclic graph (DAG) of compute and store nodes. Each EVT node operates on register-resident tile fragments during the epilogue phase, enabling fusion of arbitrary element-wise operations with the GeMM store without additional kernel launches.

Key Insight: PairMulStore EVT Node.

We introduce a custom EVT node,PairMulStore, that operates on the[M,2​N][M,2N]accumulator within a single threadblock. For each output position(m,n)(m,n), the node reads the adjacent column pair from the accumulator:

PairMulStore​(D⁡[m,2​n],D⁡[m,2​n+1])=SiLU​(D⁡[m,2​n])⊙D⁡[m,2​n+1]\texttt{PairMulStore}(D[m,2n],D[m,2n{+}1])=\text{SiLU}(D[m,2n])\odot D[m,2n{+}1](6)where even columns correspond to Gate values and odd columns to Up values. This integrates SwiGLU computation into the store phase itself, overlapping the arithmetic ofSiLUand element-wise multiply with the TMA stores of the previous tile’s output.

Threadblock Design.

Unlike Kernel-1’s sequential Gate→\rightarrowUp approach, Kernel-2 concatenatesW1W_{1}andW2W_{2}into a single fused weight matrixWfused∈ℝK×2​NW_{\text{fused}}\in\mathbb{R}^{K\times 2N}with columns interleaved: even columns holdW1W_{1}(Gate) and odd columns holdW2W_{2}(Up). The kernel launches a standard GeMM over[M,2​N,K][M,2N,K], creating an[M,2​N][M,2N]threadblock grid. Each threadblock computes a tile of the fullA×WfusedA\times W_{\text{fused}}product, producing accumulator values where adjacent column pairs(2​n,2​n+1)(2n,2n{+}1)correspond to matched Gate and Up elements.

Intra-Threadblock Fusion via PairMulStore.

The key design is thatno cross-threadblock communication is needed. The PairMulStore EVT node operates entirely within each threadblock’s local accumulator: it reads adjacent column pairs(D⁡[m,2​n],D⁡[m,2​n+1])(D[m,2n],D[m,2n{+}1])from registers, appliesout​[m,n]=SiLU​(D⁡[m,2​n])⊙D⁡[m,2​n+1]\text{out}[m,n]=\text{SiLU}(D[m,2n])\odot D[m,2n{+}1], and stores theM×NM\times Nresult via TMA. Since both Gate and Up values for any output element reside in thesame threadblock’sregisters (by construction of the interleaved weight layout), the fusion is purely local — eliminating any need for global memory exchange between threadblocks.

Kernel-1:[M,N][M,N]Each block: Gate+Up+SwiGLUMMNNKernel-2:[M,2​N][M,2N]Even cols: GateOdd cols: UpMM2​N2NEVT pairs(2​n,2​n+1)(2n,2n{+}1)within tileFigure 4:Threadblock grid comparison.Left:Kernel-1 uses[M,N][M,N]blocks, each sequentially computing Gate and Up.Right:Kernel-2 uses[M,2​N][M,2N]blocks over interleaved weights with even columns for Gate and odd columns for Up; the PairMulStore EVT fuses adjacent columns within each block’s accumulator.

Occupancy Advantage.

The2×2\timesincrease in threadblock count directly improves SM occupancy at small batch sizes. For Qwen-2.5 0.5B atM=256M=256with tile size 128, Kernel-1 creates only2×38=762\times 38=76threadblocks, while Kernel-2 creates2×76=1522\times 76=152threadblocks. On H100 with 132 SMs, this difference is significant: Kernel-1 achieves<1<1wave of execution while Kernel-2 sustains>1>1wave, keeping more SMs active.

Compute-Store Overlap.

The interleaved design provides a second performance advantage: SwiGLU computation for the current output tile overlaps with TMA stores of the previous tile. Since TMA stores are asynchronous, the SM’s compute units remain active during store completion, effectively hiding the store latency behind useful arithmetic. This explains why Kernel-2 achieves fusion efficiency>100%>100\%on several configurations — the compute overlap provides benefit beyond pure memory traffic elimination.

Implementation.

Kernel-2 is implemented by extending CUTLASS’sCollectiveEpiloguewith thePairMulStoreEVT node. The weight matricesW1W_{1}andW2W_{2}are pre-interleaved intoWfused∈ℝK×2​NW_{\text{fused}}\in\mathbb{R}^{K\times 2N}(a one-time preprocessing step). The EVT node registers a callback that receives the[M,2​N][M,2N]accumulator tile, extracts adjacent column pairs, appliesSiLU(implemented asx⋅(1+exp⁡(−x))−1x\cdot(1+\exp(-x))^{-1}with fast BF16 approximation), performs element-wise multiply, and feeds theM×NM\times Nresult to the TMA store descriptor. The kernel is compatible with both Pingpong and Cooperative CUTLASS schedules, though we use Pingpong for consistency with Kernel-1.

3.4Kernel Selection Strategy

The two kernels are complementary:

• •Kernel-2 (default): Preferred for most configurations. Wins 13/20 benchmarked settings due to better small-batch occupancy and compute-store overlap. Recommended forM≤2048M\leq 2048or models withN≤14000N\leq 14000.
• •Kernel-1: Preferred for very large models (72B) at high batch sizes (M>512M>512) where the[M,N][M,N]grid already saturates SMs and the sequential Gate→\rightarrowSwish→\rightarrowUp design provides more efficient epilogue stores.

The crossover point decreases with model size: no crossover for 0.5B (Kernel-2 always wins),M≈2500M\approx 2500for 1.5B, andM≈425M\approx 425for 72B. Near crossover points, performance differences are<1%<1\%, making the selection non-critical in practice.

4Experimental Setup

We evaluate our fused GeMM-SwiGLU kernels on a single NVIDIA H100 80GB HBM3 GPU with peak BF16 throughput of 989.4 TFLOPS and 3.35 TB/s HBM bandwidth. All experiments use CUDA 13.0, CUTLASS 3.x (SM90 target), and PyTorch 2.11.0+cu130.

4.1Model Configurations

We benchmark using the hidden dimension (KK) and Feed-Forward Network (FFN) intermediate size (NN) from four Qwen-2.5 models10, spanning edge-deployment to datacenter scale. No model weights are downloaded; we construct random BF16 input tensorsA∈ℝM×KA\in\mathbb{R}^{M\times K}and weight matricesW1,W2∈ℝK×NW_{1},W_{2}\in\mathbb{R}^{K\times N}matching each model’s dimensions. Table1summarizes the configurations.

Table 1:Qwen-2.5 model configurations used for benchmarking.

4.2Batch Configurations

We vary the batch dimensionM∈{256,512,1024,2048,4096}M\in\{256,512,1024,2048,4096\}, representing the product of batch sizeBBand sequence lengthS=256S=256withB∈{1,2,4,8,16}B\in\{1,2,4,8,16\}. This yields 20 total configurations (4 models×\times5 batch sizes), covering both latency-sensitive single-request serving (M=256M=256) and throughput-oriented batched inference (M=4096M=4096).

4.3Baselines

We compare against two PyTorch baselines:

1. 1.PyTorch Eager (cuBLAS): The standard implementation using twotorch.mmcalls followed by element-wise SwiGLU:Y=SiLU​(A​W1)⊙(A​W2)Y=\text{SiLU}(AW_{1})\odot(AW_{2}), whereW1W_{1}is the gate projection andW2W_{2}is the up projection. We enableallow_bf16_reduced_precision_reductionfor best cuBLAS performance.
2. 2.torch.compile (max-autotune): PyTorch’s compiler with maximum optimization, enabling CUDA graphs, Triton autotuning, and kernel selection between cuBLAS and Triton-generated kernels.

We deliberately use PyTorch as the baseline rather than standalone CUTLASS GeMM, as PyTorch represents the default execution path in production serving frameworks such as vLLM3.

4.4Measurement Methodology

All latency measurements use CUDA events for GPU-side timing with 50 warmup iterations followed by 200 measured iterations per configuration. We report the median latency to minimize sensitivity to outliers. Measurements achieve a coefficient of variation (CV) below 5% across all configurations, with most below 2%, confirming measurement stability. No tensor parallelism is used, isolating single-GPU kernel-level performance. All data uses BF16 precision throughout.

5Results

We evaluate both fused kernels across all 20 configurations (4 models×\times5 batch sizes) and compare against PyTorch eager andtorch.compilebaselines.

5.1Main Speedup Results

Table2presents the complete speedup results for both kernels relative to the PyTorch eager baseline. Both kernels achieve speedups across 19 of 20 configurations, with the sole exception being 72B atM=4096M=4096where both kernels are at parity (0.99×\times).

Table 2:Speedup over PyTorch eager baseline (cuBLAS + separate SwiGLU). Bold indicates the faster kernel per configuration. Both kernels achieve≥1.0×\geq 1.0\timesin 19/20 configs.00footnotetext:All entries are medians over 200 iterations after 50 warmup. Coefficient of variation: baseline CV<<5%, Kernel-1 CV<<1%, Kernel-2 CV<<0.5%. Speedup uncertainty is±\pm0.03×\timesin the worst case.#### Key Observations.

Kernel-2 (Interleave) achieves the highest speedup of 2.47×\timeson Qwen-2.5 0.5B atM=256M=256and wins 13/20 configurations overall. Kernel-1 (Sync) wins the remaining 7/20, predominantly for larger models at high batch sizes. The speedup magnitude decreases with model size: peak 2.47×\times(0.5B), 1.77×\times(1.5B), 1.32×\times(14B), and 1.25×\times(72B). This trend is explained by the diminishing fraction of time spent on SwiGLU and intermediate materialization as the GeMM computation (which scales withK×NK\times N) dominates.

Figure5visualizes the speedup landscape as heatmaps, clearly showing that both kernels provide the largest benefits in the top-left region (small model, small batch) where memory traffic for intermediates is proportionally largest.

Figure 5:Speedup heatmaps for Kernel-1 (left) and Kernel-2 (right) relative to PyTorch eager baseline. Darker colors indicate higher speedup. Kernel-2 achieves uniformly higher speedups for small models, while Kernel-1 is competitive only for large models at high batch sizes.

5.2Crossover Analysis and Kernel Selection

To provide practical deployment guidance, we characterize the batch size at which Kernel-1 overtakes Kernel-2 for each model.

• •Qwen-2.5 0.5B: No crossover — Kernel-2 is faster at all batch sizes (M=256M=256–40964096).
• •Qwen-2.5 1.5B: Crossover atM≈2500M\approx 2500. Below this, Kernel-2 leads by 1.5–6.4μ\mus; above, both kernels achieve parity (<<1% difference).
• •Qwen-2.5 14B: Complex oscillating pattern due to tile quantization effects. No single clean crossover, but Kernel-2 generally leads forM∈[320,1280]M\in[320,1280].
• •Qwen-2.5 72B: Clear crossover atM≈425M\approx 425. Kernel-1 leads forM>512M>512.

The crossover point decreases with model size because largerNNprovides sufficient threadblocks for Kernel-1’s[M,N][M,N]grid even at smallMM, negating Kernel-2’s occupancy advantage. Near crossover points, performance differences are below 1%, making kernel selection non-critical.

Figure 6:Kernel-1 to Kernel-2 latency ratio across batch sizes. Values>>1.0 indicate Kernel-2 is faster. The crossover point (ratio = 1.0) shifts left with increasing model size.

5.3Roofline Analysis

We place all configurations on the H100 roofline to explain the mechanism behind the observed speedups.

Arithmetic Intensity Shift.

Fusion increases arithmetic intensity by eliminating4×M×N×24\times M\times N\times 2bytes of intermediate traffic. Table3quantifies this shift across all model sizes. The increase ranges from 6.1% (72B,M=256M=256) to 246.7% (0.5B,M=4096M=4096), with larger relative gains for small models where intermediate tensor size (M×NM\times N) is large relative to weight matrices (K×NK\times N).

Table 3:Arithmetic intensity shift from fusion (FLOP/Byte). Fusion eliminates intermediate tensor traffic, increasing AI by 6–247%.

Regime Transition.

The baseline operates in the memory-bound regime for 7/20 configurations (small models at small batch sizes). After fusion, only 4/20 configurations remain memory-bound — fusion shifts 3 configurations across the ridge point into the compute-bound regime. This regime transition is the fundamental mechanism enabling the largest speedups (2.0–2.5×\times): the baseline is bottlenecked by HBM bandwidth while the fused kernel is bottlenecked by tensor core throughput.

Compute Utilization.

Table4shows achieved compute utilization as a percentage of peak BF16 TFLOPS (989.4 TFLOPS on H100).

Table 4:Compute utilization (% of peak BF16 TFLOPS). Fusion doubles utilization for small models, reaching 79.5% peak.Figure 7:H100 roofline plot showing baseline (circles) and fused kernel (triangles) operating points. Fusion shifts configurations rightward (higher arithmetic intensity) and upward (higher achieved TFLOPS). The largest speedups occur where baseline is memory-bound but fused kernels operate compute-bound.

5.4Comparison with torch.compile

We evaluate whether PyTorch’s compiler infrastructure can automatically achieve similar fusion.

Default max-autotune.

Surprisingly,torch.compilewithmode=“max-autotune“isslowerthan eager PyTorch for the SwiGLU pattern, achieving only 0.35–0.94×\timesof eager performance (Table5). The slowdown is largest at small batch sizes (0.35×\timesfor 72B atM=256M=256) and diminishes toward parity at large batch sizes. The compiler’s Triton-generated kernels and CUDA graph overhead cannot compensate for the inability to fuse across two separate weight matrices.

Table 5:torch.compile (max-autotune) performance relative to eager PyTorch. Values<<1.0 indicate slowdown. The compiler cannot fuse separate GeMMs with different weight matrices.

Fusion Hints.

We additionally testedtorch.compilewith explicit fusion hints:fullgraph=True, wrapping the operation in a customtorch.autograd.Function, and enablingcoordinate_descent_tuning. None of these variations provides meaningful improvement over default max-autotune — all remain within±\pm4% of the baseline compile performance. The fundamental limitation is architectural: the compiler’s fusion passes operate on single-kernel graphs and cannot merge two GeMMs withdifferentweight matrices into one kernel.

Gap vs. Fused Kernels.

At small batch sizes where our kernels achieve 2.0–2.5×\timesspeedup over eager PyTorch, the gap versustorch.compileis even larger: our fused kernels are 3–7×\timesfaster than the besttorch.compilevariant. This definitively validates that hand-crafted tile-level fusion is necessary for this workload — compiler infrastructure cannot replicate it.

6Discussion

6.1Ablation: Decomposing the Speedup

To understand the source of performance gains, we decompose each kernel’s speedup into two components: (a) time saved from eliminating intermediate memory traffic, and (b) additional benefit or cost from compute overlap and fusion overhead.

We estimate the theoretical memory savings asΔ​tmemory=4​M​N⋅2/BWeff\Delta t_{\text{memory}}=4MN\cdot 2/\text{BW}_{\text{eff}}, whereBWeff\text{BW}_{\text{eff}}is the effective HBM bandwidth measured from standalone SwiGLU profiling (0.48–1.62 TB/s depending on configuration). We definefusion efficiencyas the ratio of actual speedup to theoretical memory savings:η=Δ​tactual/Δ​tmemory×100%\eta=\Delta t_{\text{actual}}/\Delta t_{\text{memory}}\times 100\%.

Table 6:Fusion efficiency (%) by model. Values near 100% indicate memory savings fully explains the speedup;>>100% indicates additional compute overlap benefit;<<100% indicates compute overhead partially offsetting memory savings.Table6reveals three key insights:

Memory savings is the dominant driver.

For 0.5B, fusion efficiency is near 100%, meaning the entire speedup is explained by eliminating intermediate tensor reads and writes. The small model’s GeMM is memory-bound, so removing4​M​N4MNbytes of traffic directly translates to proportional time savings.

Kernel-2 achieves super-linear efficiency.

On several configurations (0.5B atM=512M=512–40964096, 1.5B atM=256M=256), Kernel-2 achieves>>100% efficiency. This “extra” benefit comes from the interleaved store design: SwiGLU computation overlaps with TMA stores, providing genuine compute-store overlap beyond pure memory elimination.

Compute overhead grows with model size.

For 14B and 72B models, fusion efficiency drops to 69–77%. The fused MMA is slightly less efficient than cuBLAS’s highly optimized decomposition for very large GeMMs, introducing compute overhead that partially offsets memory savings. This fully explains the speedup degradation from 2.5×\times(0.5B) to∼\sim1.0×\times(72B at largeMM).

Figure 8:Ablation decomposing fusion benefit into memory savings (blue) and compute overlap/overhead (orange). Memory traffic elimination dominates for small models; compute overhead increasingly offsets gains for larger models.

6.2Why Speedup Decreases with Model Scale

The monotonic decrease in speedup from 2.47×\times(0.5B) to∼\sim1.0×\times(72B) is explained by three compounding factors:

1. 1.SwiGLU fraction decreases: From 30–37% of MLP time for 0.5B down to 8–9% for 72B. There is simply less to gain from fusing a smaller fraction of the computation.
2. 2.Baseline arithmetic intensity increases: Large models haveK×NK\times Nweights that dominate total memory traffic. The intermediate tensors (M×NM\times N) become a smaller fraction of total bytes moved, reducing the relative impact of eliminating them.
3. 3.Fused MMA inefficiency: For very large tiles (72B:K=8192K=8192,N=29568N=29568), cuBLAS’s multi-level tiling and split-K decomposition achieves near-peak utilization. Our fused kernel’s constraint of processing both Up and Gate within one threadblock limits tiling flexibility, introducing 1–3% compute overhead that negates the small memory savings.

The practical implication is that our kernels provide the largest benefit for edge and mobile deployment models (0.5B–14B) — precisely the models where inference latency is most critical and compute budgets are constrained.

6.3Numerical Accuracy

An unexpected finding is that our fused kernels aremorenumerically accurate than the PyTorch baseline.

Table 7:Numerical comparison against FP32 reference. Mismatch rate counts elements with>>1% relative error. Fused kernels achieve zero mismatches while PyTorch cuBLAS has 4.5–11% error rate.Both fused kernels achievezero mismatches(no elements exceeding 1% relative error) across all 20 configurations when compared to an FP32 reference. Remarkably, Kernel-2 isbit-exact— producing identical results to the FP32 reference. In contrast, PyTorch’s cuBLAS backend shows 4.5–11% of output elements exceeding the 1% error threshold.

This superiority stems from accumulation order. cuBLAS uses aggressive tile decomposition with BF16 partial accumulation across tiles, introducing rounding errors that compound across theKKdimension. Our fused kernels maintain strict FP32 accumulation within each tile’s reduction, applying BF16 conversion only at the final store. The SwiGLU computation itself uses FP32 arithmetic on register-resident values, avoiding any intermediate precision loss.

We note that all BF16 accumulation orders are mathematically valid — cuBLAS’s results are not “incorrect” but rather reflect a different (hardware-optimized) reduction order. Our kernels’ strict per-tile FP32 accumulation happens to match the sequential reference more closely, which is desirable for debugging and reproducibility but does not imply cuBLAS produces inferior model outputs in practice.

6.4Limitations

We acknowledge several limitations of this work:

• •No end-to-end benchmarks: We measure MLP kernel latency in isolation. End-to-end LLM inference involves attention, layer norms, and communication that may shift the bottleneck.
• •SM90 only: Our kernels target H100 (SM90) and have not been tested on A100 (SM80) or consumer GPUs. The Pingpong schedule and TMA are SM90-specific features.
• •No TensorRT-LLM comparison: We compare against PyTorch but not against TensorRT-LLM’s proprietary fused MLP kernels, which may employ similar (unpublished) techniques.
• •Diminishing returns at scale: For 72B models atM≥2048M\geq 2048, our kernels provide≤\leq1.3% improvement, limiting applicability to large-scale datacenter workloads.
• •BF16 only: We have not evaluated FP8 or INT8 variants, though we expect even larger relative gains as quantized GeMM becomes cheaper.

7Conclusion

We presented two complementary CUTLASS-based SM90 kernels that fuse SwiGLU activation into GeMM at the tile level, eliminating intermediate tensor materialization that accounts for 9–37% of MLP execution time in modern LLMs.

Kernel-1 (Sync SwiGLU) exploits the Pingpong warp-specialized schedule to overlap Swish computation on the Gate accumulator with Up-tile loading, creating an[M,N][M,N]threadblock grid optimized for large batch sizes. Kernel-2 (Interleave SwiGLU) introduces a custom PairMulStore Epilogue Visitor Tree node that interleaves SwiGLU with tile stores, creating an[M,2​N][M,2N]grid with2×2\timesbetter occupancy for small batches. Together, they achieve up to 2.47×\timesspeedup over PyTorch on NVIDIA H100, with Kernel-2 winning 13/20 configurations and serving as the recommended default.

Our ablation reveals that memory traffic elimination is the primary speedup mechanism, with Kernel-2 providing additional compute-store overlap benefit (>>100% fusion efficiency on several configurations). The roofline analysis shows that fusion shifts workloads from memory-bound to compute-bound, achieving 79.5% of peak BF16 utilization. We definitively demonstrate thattorch.compilecannot replicate this fusion (3–7×\timesslower than our kernels), validating the necessity of hand-crafted tile-level design. As a bonus, our fused kernels are numerically superior to cuBLAS, achieving zero mismatches versus 4.5–11% for the baseline.

Future Work.

Several directions extend this work: (1) end-to-end integration into vLLM and SGLang for full inference serving benchmarks; (2) extension to FP8/INT8 quantized GeMMs where the activation bottleneck will be proportionally larger; (3) adaptive runtime kernel selection based on problem dimensions; (4) generalization to other gated activations (GeGLU, ReGLU); (5) multi-GPU tensor parallelism scenarios where communication-computation overlap interacts with our fusion; and (6) ports to A100 (SM80) and consumer GPUs.

References

• Daoet al.(2022)T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. RéFlashAttention: fast and memory-efficient exact attention with IO-awareness.InAdvances in Neural Information Processing Systems (NeurIPS),Cited by:§1,§2.2.
• Dao (2024)T. DaoFlashAttention-2: faster attention with better parallelism and work partitioning.InInternational Conference on Learning Representations (ICLR),Cited by:§2.2.
• Kwonet al.(2023)W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. StoicaEfficient memory management for large language model serving with PagedAttention.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP),Cited by:§2.4,§4.3.
• Narayananet al.(2021)D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kasber, and B. CatanzaroEfficient large-scale language model training on GPU clusters using Megatron-LM.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC),Cited by:§2.4.
• NVIDIA (2023)NVIDIATensorRT-LLM.Note:https://github.com/NVIDIA/TensorRT-LLMCited by:§2.4.
• Shazeer (2020)N. ShazeerGLU variants improve transformer.arXiv preprint arXiv:2002.05202.Cited by:§1,§2.1,§3.1.
• Thakkaret al.(2023)V. Thakkar, P. Ramani, C. Cecka, A. Shivam, H. Lu, E. Yan, V. Korthikanti,et al.CUTLASS 3.x: composable CUDA template abstractions for high-performance GeMM.Note:NVIDIA, GitHub:https://github.com/NVIDIA/cutlassCited by:§1,§2.3,§3.2,§3.3.
• Tilletet al.(2019)P. Tillet, H.T. Kung, and D. CoxTriton: an intermediate language and compiler for tiled neural network computations.InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages,Cited by:§2.3.
• Touvronet al.(2023)H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar,et al.LLaMA: open and efficient foundation language models.arXiv preprint arXiv:2302.13971.Cited by:§1,§2.1.
• Yanget al.(2024)A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou,et al.Qwen2 technical report.arXiv preprint arXiv:2407.10671.Cited by:§1,§2.1,§4.1.