In this paper is proposed a hardware-software co-design framework for N:M sparse vision Transformer inference, enabling more accurate and reliable acceleration under the same target speedup.

https://t.co/uga3Ba6C1L https://t.co/7rZBHxWQMr

Realizable N:M Sparse Transformer Inference via Search–Kernel Co-Design

Source: https://arxiv.org/html/2607.12505 11institutetext:University of Science and Technology of China, Hefei, China22institutetext:Suzhou Institute of Advanced Research, University of Science and Technology of China, Suzhou, China33institutetext:National Key Laboratory of Modeling and Simulation for Complex Systems, China 33email:[email protected],33email:[email protected],33email:[email protected] Lou✉https://orcid.org/0000-0002-2240-6672Zhiguang Wanghttps://orcid.org/0009-0002-5829-3339Zhiwei Kehttps://orcid.org/0009-0003-7636-2446 Fengrui Zuohttps://orcid.org/0009-0003-3468-9280Chao Wanghttps://orcid.org/0000-0002-9403-5575Xuehai Zhouhttps://orcid.org/0000-0002-8360-3143

Abstract

Vision Transformers (ViTs) achieve strong accuracy but incur high inference latency. Semi-structured N:M sparsity can reduce arithmetic cost, yet its theoretical savings often fail to translate into proportional end-to-end speedups on modern GPUs. This mismatch arises because deployment latency depends not only on arithmetic reduction but also on execution regularity and hardware scheduling under sparsity. Achieving practical acceleration, therefore, requires coordinated design across sparse execution and sparsity configuration. To this end, we propose a hardware-software co-design framework for N:M sparse ViT inference. On the hardware side, we design MD-SpMM, an N:M sparse CUDA kernel that reorganizes sparse GEMM into micro-dense, Tensor-Core-aligned dataflow and uses inference-aware adaptive parallelism to sustain utilization.On the software side, we perform layer-wise sparsity search under explicit end-to-end latency budgets using a three-stage heuristic search with constraint relaxation to avoid premature convergence and enable deployment-aware sparsity allocation. Experiments on multiple ViT/Swin models and GPU platforms show that the framework achieves over 2.2× latency speedup while maintaining comparable accuracy and delivering superior accuracy under the same latency constraint. The source code is publicly available athttps://github.com/liuganhuo/realizable-nm-sparse-transformer.

1Introduction

Vision Transformers (ViTs) leverage self-attention to capture long-range dependencies and achieve strong performance in tasks such as image classification and semantic segmentation[20]. However, this benefit comes with high inference cost: ViTs rely heavily on matrix multiplications in projection and feed-forward layers, imposing substantial compute and memory demands that make deployment on real hardware increasingly challenging[24,16].

Model compression, especially pruning, has been widely studied to mitigate this burden by removing redundant parameters while preserving accuracy[2]. Among pruning schemes, semi-structured N:M sparsity has emerged as a practical compromise between accuracy preservation and hardware regularity, and has been adopted across CNNs, ViTs, and related accelerators[25,10,12,9].

Despite this progress, the theoretical savings of N:M sparsity often fail to translate into proportional end-to-end latency gains in real inference, especially under INT8 deployment, as evidenced by the experimental comparisonsin Section 6. In this regime, latency depends not only on arithmetic reduction but also on execution regularity, memory behavior, and scheduling effects under sparsity. This reveals a persistent gap between theoretical efficiency and practical performance. The gap is often misinterpreted as sparsity being “ineffective”, whereas it is more accurately attributed to the difficulty of realizing sparsity efficiently on current deployment stacks.

We therefore focus onrealizable acceleration: whether N:M sparsity can deliver a target end-to-end latency reduction on a real deployment stack, rather than merely improving proxy metrics such as FLOPs, global sparsity, or isolated operator speedups[4,23,18]. From a system perspective, realizable acceleration requires satisfying two tightly coupled conditions. First, sparse computation must beexecution-realizable: sparse operators should deliver stable and efficient execution on real hardware instead of exhibiting irregular behavior that prevents consistent latency reduction[10]. Second, sparsity assignment must belatency-grounded: when N:M sparsity is applied in a layer-wise manner, per-layer ratios should be selected under explicit deployment-level latency constraints rather than abstract sparsity or FLOPs-based objectives that poorly reflect end-to-end cost[8]. Violating either condition leads to unrealizable speedups in practice, even when theoretical computation reduction is significant. Yet existing approaches often optimize only one side[7,9,10,13], motivating an end-to-end co-design framework that couples execution-realizable sparse kernels with latency-grounded layer-wise configuration and lightweight candidate evaluation.

To this end, we propose a hardware-software co-design framework for realizable N:M sparse inference acceleration. On the hardware side, we design MD-SpMM, an N:M sparse CUDA kernel that restructures sparse computation into Tensor-Core-aligned execution through micro-dense restructuring and adaptive parallelism.On the software side, we introduce a latency-grounded layer-wise search pipeline that combines a profiling-based latency predictor and a feasible-region-aware evolutionary solver to evaluate candidates efficiently.By integrating execution-realizable kernels with latency-grounded sparsity search, the framework consistently translates sparsity-induced computation reduction into end-to-end latency improvement on real deployment stacks.

In summary, this work makes the following contributions:

• •We present asearch–kernel co-design frameworkforrealizableN:M sparse Transformer acceleration, built on execution realizability and configuration realizability under deployment latency constraints, and enabling more accurate and reliable acceleration under the same target speedup.
• •We designMD-SpMM, an execution-realizable Tensor-Core-native N:M sparse kernel that converts sparse computation into regular MMA-centric execution with scalable inference parallelism. Across diverse matrix multiplication shapes and sparsity levels, it delivers 2.0×\timesaverage speedup over cuBLAS dense GEMM and 1.6×\times–4.0×\timesspeedup overnmSPARSE[10].
• •We propose a latency-grounded layer-wise N:M search pipeline that integrates a calibrated LUT-based latency predictor with a feasible-region-aware evolutionary solver to efficiently discover high-quality sparse configurations. With our hardware–software co-design, the resulting solutions achieve over 2.2× latency reduction while delivering higher accuracy, yielding up to +1.4% Top-1 accuracy on ImageNet-1K and consistently outperforming prior methods across ViT and Swin inference settings.

2Background and Motivation

2.1Vision Transformers and Layer-wise N:M Configuration

Vision Transformers (ViTs) process an image as a sequence of patch embeddings and forward it through a stack of Transformer blocks. In practice, inference is dominated by the Q/K/V projections, the attention output projection, and the two FFN linear layers. For common vision workloads, these matrix-heavy operators account for most of the end-to-end inference time, making them the primary targets for sparsification[24].

N:M sparsity constrains each group ofMMweights to retainNNnonzeros. In ViTs, it is often applied uniformly across all Transformer blocks, as illustrated in Fig.2. However, despite their similar block structure, layers at different depths contribute unequally to model accuracy and inference latency due to their depth-dependent roles[8,18,5]. This leads to heterogeneous sparsity sensitivity across blocks, making a single global N:M ratio suboptimal. Consequently, ViT sparsification is more naturally formulated as a layer-wise N:M configuration problem.

Figure 1:N:M sparsity and uniform vs. layer-wise configurations in ViTs. Figure 2:Roofline analysis of N:M sparse INT8 inference on GPUs.

2.2The Lack of Efficient Sparse Execution Mechanisms

Although N:M sparsity reduces multiply-accumulate operations, it does not directly yield efficient execution on modern Tensor Core GPUs. This mismatch is especially pronounced for INT8 inference, where dense GEMM already achieves high efficiency through regular dataflow, predictable control flow, and high MMA utilization[15,14,1]. In this regime, sparse speedup depends not only on arithmetic reduction but also on whether execution remains compatible with the hardware execution model. Existing sparse kernels often rely on metadata handling, runtime decoding, and irregular memory accesses, which disrupt Tensor Core dataflow and reduce utilization[6,10,22].

As illustrated by the Roofline analysis in Fig.2, existing approaches exhibit a clear execution trade-off. nmSPARSE[10]follows sparse decoding and irregular access patterns, leading to low arithmetic intensity and limited utilization, whereas cuBLAS sustains high efficiency with dense GEMM but cannot exploit N:M structure. This gap highlights the need for sparse execution mechanisms that preserve Tensor-Core-friendly regularity while exposing the arithmetic benefit of N:M sparsity.

2.3The Need for Latency-Grounded Layer-wise Configuration

Introducing latency constraints into sparse configuration search is necessary but challenging, since meeting a target latency requires different sparsity levels across models, layers, and deployment environments. Traditional methods that optimize proxy metrics such as sparsity ratio or FLOPs often fail to capture real deployment behavior. Even with similar overall sparsity, different layer-wise assignments can lead to substantially different end-to-end latencies. Moreover, enforcing latency constraints fragments the feasible space by discarding invalid configurations, weakening search-space connectivity and making exploration prone to narrow feasible regions.

To address this issue, sparse configuration optimization must incorporate deployment-level latency semantics. In particular, latency models built on the target hardware and deployment stack are needed to guide the search toward configurations that both satisfy latency constraints and enable realizable end-to-end acceleration. This mitigates the mismatch between abstract optimization objectives and real hardware behavior[3,4,19].

3Problem Formulation and Method Overview

Given a Transformer model and a target end-to-end latency budget, the problem is to realize N:M sparse inference that preserves model quality while delivering actual speedup on the target hardware.

As shown in Fig.3, we address this problem through two coordinated components. At the execution level, we design MD-SpMM, a sparse kernel that realizes N:M sparsity through Tensor-Core-aligned execution. At the configuration level, we build a latency-grounded layer-wise search pipeline, in which a calibrated LUT-based latency predictor enforces deployment feasibility and a hardware-aware search strategy explores the fragmented feasible region under explicit latency constraints.

Together, these components instantiate the two requirements of realizable acceleration introduced in Section 1. MD-SpMM addresses execution realizability, while the latency-grounded search addresses configuration realizability by identifying layer-wise N:M assignments aligned with end-to-end deployment behavior. In this sense, realizable acceleration is achieved only when sparse execution and sparsity configuration are jointly optimized.

Figure 3:Overview of the search-kernel co-design framework for N:M sparse Transformer acceleration.

4MD-SpMM: Dataflow-Centric N:M Sparse Kernels

4.1Design Methodology

From the perspective of execution realizability, INT8 inference with N:M sparsity depends not only on reduced arithmetic but also on whether sparse execution matches the hardware execution model. We identify two first-order requirements: (i)dataflow regularityto sustain Tensor Core MMA utilization, and (ii)inference-scale parallelismto maintain high SM occupancy across varying operator shapes. Violating either requirement by introducing index-driven control, irregular memory access, or insufficient thread-block parallelism can erase theoretical sparsity gains in end-to-end latency. As shown in Fig. 4, MD-SpMM uses three mechanisms: (1) Weight Packing for decode-friendly sparse storage, (2) Micro-Dense Execution for MMA-centric regular tiles, and (3) Adaptive Parallelism to improve occupancy while bounding reduction overhead.

4.2Micro-Dense: From N:M Sparsity to Tensor Core Dataflow

To make N:M sparsity compatible with Tensor Core execution, we introduceMicro-Dense, which reformulates N:M SpMM as an MMA-centric regular dataflow. Under N:M sparsity, each window of sizeMMcontainsNNnonzeros. Rather than exposing them to the compute stage as irregular sparse operands, Micro-Dense materializes only theseNNnonzeros into fixed MMA-compatible dense tiles aligned with Tensor Core fragment shapes, so Tensor Cores still operate on regular dense fragments and runtime computation remains entirely MMA-based.

This design shifts sparsity irregularity out of the MMA compute path and into the loading-and-decoding stage. To keep the resulting overhead low, we adopt a decode-friendly packed representation (B_pack) that jointly encodes nonzero values and their positions within each N:M window. This removes separate index loads and enables lightweight decoding, which is overlapped with shared-memory placement and MMA issuance.

Therefore, Micro-Dense confines sparsity handling to packed loading and lightweight decoding, allowing runtime computation to operate directly on MMA-aligned tiles. Overall, it establishes a clear execution mapping: N:M sparsity→\rightarrowMicro-Dense tiles→\rightarrowTensor Core dataflow.

Figure 4:Overview of the MD-SpMM kernel design.

4.3Adaptive Parallelism: Split-K for Inference-Scale SpMM

Micro-Dense improves per-block Tensor Core efficiency, but inference can still suffer fromunder-occupancywhen operator shapes expose limited parallel work. End-to-end performance therefore depends not only on efficient single-block execution, but also on scalable and predictable parallelism across workloads.

We address this with split-K parallelism, which partitions the reduction dimension so multiple thread blocks compute partial sums in parallel, expanding the parallel space from the output dimensions(M,N)(M,N)to(M,N,splitK)(M,N,\textit{splitK}). Split-K boosts occupancy but introduces reduction and extra write-back overhead, so its benefit must be controlled.

We therefore adopt a rule-driven splitK selection strategy. First,parallelism priority: splitK is increased until the number of blocks is sufficient to fully schedule SMs. Second,controlled reduction: splitK is restricted to a small discrete candidate set so that K-splits remain aligned with Tensor Core granularity and reduction overhead stays predictable. WhenKKis small or reduction dominates, the strategy reverts to a smaller splitK.

In this approach, all splits share the same compute template and partial sums are merged only in the final stage, avoiding additional control complexity. Combined with Micro-Dense, this adaptive parallelism enables consistent speedup across deployment scenarios.

5Latency-Grounded Layer-wise Sparsity Search

While MD-SpMM makes N:M sparsity execution-realizable, determining an effective layer-wise sparsity configuration remains challenging, since accuracy alone is insufficient and the end-to-end latency impact of sparsity is strongly deployment-dependent. We therefore develop a latency-grounded search strategy that explicitly incorporates latency constraints to guide the search toward higher-quality and deployment-feasible configurations.

5.1Lookup Table-based Latency Modeling

Accuracy ranking alone is insufficient, since the search must also satisfy an end-to-end latency budget. Because measuring full-model latency for every candidate is too costly, we build a lightweight predictor for feasibility checks[3,5,11,21].

Figure 5:Predicted vs. measured end-to-end latency on platforms and architectures.Under a fixed hardware–software stack, we profile MD-SpMM operators and build an operator-level sparsity–latency lookup table (LUT). For each operator shape and candidate N:M ratio, we benchmark sparse matrix multiplication latency (average repeated runs). Empirically, the latency is largely determined by operator shape and sparsity level, yielding a LUT indexed by(shape,sparsity)(\textit{shape},\textit{sparsity}). Since ViT-family models reuse only a small set of operator shapes, the LUT can be rebuilt offline within seconds when the deployment environment changes.

To predict end-to-end latency, we fit a first-order linear model on top of the LUT. Lettℓ​(sℓ)t_{\ell}(s_{\ell})denote the LUT latency of layerℓ\ellunder sparsity settingsℓs_{\ell}, then

Tpred=∑ℓ=1Lγℓ​tℓ​(sℓ)+b,T_{\text{pred}}=\sum_{\ell=1}^{L}\gamma_{\ell}\,t_{\ell}(s_{\ell})+b,(1)whereγℓ≥0\gamma_{\ell}\geq 0andb≥0b\geq 0are calibrated by non-negative least squares (NNLS) using a small number of full-model measurements. This calibration absorbs graph-level scheduling, runtime overhead, and compiler optimizations beyond single-operator profiling. After calibration, the predictor enables constant-time feasibility checks during search with within-5% error (Fig.5), grounding layer-wise sparsity decisions in deployment latency rather than abstract sparsity metrics.

5.2Hardware-Aware Layer-wise Sparsity Search

Input:ModelMM; latency budget𝒯target\mathcal{T}_{\text{target}}; number of generations𝒢\mathcal{G}; number of offspringλ\lambda;//define: parentxx; childyy.

1Initialization:

c​a​n​d​i​d​a​t​e=𝚄𝚗𝚒𝚂𝚙𝚊𝚛𝙶𝚎𝚗​(𝒯t​a​r​g​e​t,M);candidate=\mathtt{UniSparGen}(\mathcal{T}_{target},M); 2

x1=𝚁𝚊𝚗𝚍​_​𝚂𝚙𝚊𝚛​(c​a​n​d​i​d​a​t​e,t​o​w​a​r​d=𝒯t​a​r​g​e​t);x_{1}=\mathtt{Rand\_Spar}(candidate,\,toward=\mathcal{T}_{target}); 3

4Search: for*ggin[1,𝒢][1,\mathcal{G}]*do

5

o​f​f​s​p​r​i​n​g​s=[];offsprings=[]; 63Stage-OffspringGen: for*iiin[1,λ][1,\;\lambda]*do

7

yi=xg;y_{i}=x_{g}; 8for*llin𝚁𝚊𝚗𝚍𝚘𝚖​_​𝙺​({l|yi​[l]<8})\mathtt{Random\_K}(\,\{\,l\,|\,y_{i}[l]<8\,\})*do

9

yi​[l]=N+1;y_{i}[l]=N+1;//Densify thell-th layer ofyiy_{i}(e.g., 4:8 to 5:8), keeping other layers unchanged.

10while*L​a​t​e​n​c​y​(yi)>𝒯t​a​r​g​e​tLatency(y_{i})>\mathcal{T}_{target}*do

11

l=𝚁𝚊𝚗𝚍𝚘𝚖​_​𝟷​({l|yi​[l]>1});l=\mathtt{Random\_1}(\,\{\,l\,|\,y_{i}[l]>1\,\}\,); 12

yi​[l]=N−1;y_{i}[l]=N-1;//Sparsify thell-th layer ofyiy_{i}

13

14

D:={l∣(yi​[l]<8)∧Δ​𝙵𝚒𝚝𝚗𝚎𝚜𝚜l≥0};D:=\{\,l\mid(y_{i}[l]<8)\wedge\ \Delta\mathtt{Fitness}_{l}\geq 0\};//layers whose densification yields a fitness gain.

15whileD≠∅D\neq\emptysetdo

16

yi​[𝚁𝚊𝚗𝚍𝚘𝚖​_​𝟷​(D)]=N+1;y_{i}[\mathtt{Random\_1}(D)]\!=\!N\!+\!1;//Random densify one layer

17

D:={l∣(yi​[l]<8)∧Δ​𝙵𝚒𝚝𝚗𝚎𝚜𝚜l≥0};D:=\{\,l\mid(y_{i}[l]<8)\wedge\ \Delta\mathtt{Fitness}_{l}\geq 0\};//Update

18

19

o​f​f​s​p​r​i​n​g​s.a​p​p​e​n​d​(yi);offsprings.append(y_{i}); 20

21Selection:

xg+1=𝙼𝙰𝚇​_​𝙵𝚒𝚝𝚗𝚎𝚜𝚜​(o​f​f​s​p​r​i​n​g​s);x_{g+1}=\mathtt{MAX\_Fitness}(offsprings); 22

return

x𝒢+1;x_{\mathcal{G}+1};(final sparse configuration)

Algorithm 1HW-aware heuristic search algorithmGiven the latency predictor, we search for a layer-wise N:M configuration that maximizes model quality under a target latency budget. LetMMdenote the model, and lets=[s1,…,sL]s=[s_{1},\dots,s_{L}]denote the layer-wise N:M configuration, where eachsℓs_{\ell}is selected from a discrete set of hardware-supported N:M ratios. We solve

maxs⁡A​c​c​(ℳ​(s))s.t.𝒯pred​(ℳ​(s))≤𝒯target,\max_{s}\;Acc\big(\mathcal{M}(s)\big)\quad\text{s.t.}\quad\mathcal{T}_{\text{pred}}\big(\mathcal{M}(s)\big)\leq\mathcal{T}_{\text{target}},(2)whereA​c​c​(⋅)Acc(\cdot)denotes the Top-1 accuracy on ImageNet-1K and𝒯target\mathcal{T}_{\text{target}}is the target end-to-end latency budget.

This problem is challenging because, under layer-wise N:M sparsity, the mapping from sparsity configuration to deployment latency is highly nonlinear. The latency constraint therefore partitions the search space into multiple disconnected feasible regions. As a result, prior local-perturbation-based methods[17]tend to get trapped on small “feasibility islands”, causing the search to stagnate.

To address this, we adopt a feasible-region-aware(1+λ)(1+\lambda)evolutionary search that follows anovershoot–repair–refineprocedure, enabling effective transitions across fragmented feasible regions in the sparsity–latency landscape. The overall pipeline is summarized in Algorithm1. The search is instantiated within the standard one-shot sparsity search framework[8], where the supernet training stage remains unchanged and each candidate configuration is scored by its inherited accuracy under the trained supernet.

During initialization, we first construct several latency-feasible configurations with approximately uniform sparsity distributions and select the one with the highest𝚏𝚒𝚝𝚗𝚎𝚜𝚜\mathtt{fitness}score (Eq.3) as the initial parent. We then repeatedly invoke3Stage-OffspringGento produceλ\lambdaoffspring via the following three stages:

𝙵𝚒𝚝𝚗𝚎𝚜𝚜​(𝒮)={𝙰𝚌𝚌Inherit​(ℳ​(s)),𝒯pred​(ℳ​(s))≤Ttarget,−∞,otherwise.\mathtt{Fitness}(\mathcal{S})=\begin{cases}\mathtt{Acc}_{\text{Inherit}}\big(\mathcal{M}(s)\big),&\mathcal{T}_{\text{pred}}\big(\mathcal{M}(s)\big)\leq T_{\text{target}},\\ -\infty,&\text{otherwise}.\end{cases}(3) 1)Overshoot(Lines 7–8) adopts a “relax-first” strategy: a small random subset of layers is moved to the next denser sparsity level, deliberately violating the latency budget. This avoids the vanishing effect (noise-level score variations) of conventional 1:1 “sparsify-dens” perturbations near the feasibility boundary.

2)Repair(Lines 9–11) then restores feasibility by increasing sparsity on randomly sampled layers until the latency constraint is met, avoiding the directional bias of greedy, latency-guided repair and encouraging diverse search trajectories.

3)Refine(Lines 12–15) then explores the feasible boundary by gradually densifying the model while retaining only latency-valid updates. Candidate configurations with non-decreasing𝚏𝚒𝚝𝚗𝚎𝚜𝚜\mathtt{fitness}are accepted, allowing the search to advance along the feasible boundary iteratively.

Throughout the search, the latency predictor provides constant-time feasibility checks, and the final configuration is selected through offspring competition under the latency constraint. Overall, this design enables more effective exploration of the fragmented feasible space and improves the quality of deployment-feasible layer-wise N:M configurations.

6Experiments

6.1MD-SpMM Kernel Evaluation

BenchmarkWe evaluate MD-SpMM using a benchmark across matrix shapes and sparsity levels. The matrices are defined by the dimensionsMM,NN, andKKfor matrix multiplication, whereMMis the number of rows, andNNandKKare the number of columns in the input and output matrices, respectively. We consider sparsity levels of 50%, 75%, and 87.5%, withM∈{64,128,256,512}M\in\{64,128,256,512\}andN,K∈{128,256,512,1024}N,K\in\{128,256,512,1024\}. Experiments are performed on A100, RTX 4060, and RTX 4090 GPUs using Ubuntu 22.04 and CUDA 12.2, with C++/CUDA kernel implementation compiled usingnvcc. We report latency excluding one-time offline weight packing, sinceB_packconstruction is performed once before deployment. All runtime costs during inference are included, including decoding overhead, data movement, and MMA computation. Thus, both kernel latency and end-to-end latency correspond to pure inference-time execution.

BaselinesWe compare MD-SpMM against state-of-the-art dense and sparse baselines: cuBLAS, NVIDIA’s highly optimized dense GEMM library representing the vendor-optimized upper bound, and nmSPARSE[10], a representative library for N:M sparse matrix multiplication.

Table 1:Adaptive parallel strategy across platforms and sparsity levels.PlatformA100RTX 4090RTX 4060Sparsity50%75%87.5%50%75%87.5%50%75%87.5%Top-1 Hit Rate0.8130.8750.8270.7990.7110.6900.7200.7630.714Avg. Norm. Perf.0.9900.9900.9900.9830.9650.9580.9720.9680.960Worst Norm. Perf.0.8180.7740.8030.7860.7460.7390.7730.6940.696

Figure 6:Speedup over cuBLAS across matrix shapes under different sparsity levels.ResultsWe first evaluate the adaptive parallel strategy against fixed Split-K settings. For each shape–sparsity pair, we test multiple Split-K candidates and use the lowest latency as the oracle optimum. Table1summarizes the Top-1 hit rate, average normalized performance, and worst-case normalized performance. The adaptive strategy selects the oracle-optimal configuration in most cases and remains close to optimal even in non-hit cases, while also maintaining strong worst-case robustness. Compared with a fixed parallel setting, it improves overall kernel performance by more than 50%.

Fig.6reports the kernel speedup of MD-SpMM and nmSPARSE over dense cuBLAS across matrix shapes under different N:M sparsity levels. The vertical axis shows speedup normalized to dense cuBLAS, while the horizontal axis indexes different matrix shapes; speedups above 4 are truncated for clarity.

MD-SpMM consistently outperforms nmSPARSE and delivers stable acceleration across platforms. On A100, it achieves around 2.0×\timesaverage speedup over cuBLAS and 1.6×\times–4.0×\timesspeedup over nmSPARSE[10]. Similar trends are observed on RTX 4060 and RTX 4090, with absolute gains varying due to hardware characteristics such as compute capability and memory bandwidth.

These results show that practical N:M sparse acceleration requires not only arithmetic reduction, but also regularized Tensor-Core-friendly execution together with shape-aware parallel adaptation. Overall, MD-SpMM substantially improves the execution realizability of N:M sparsity under deployment-relevant conditions.

6.2Feasible-Region-Aware Search Validation

Figure 7:Comparison of search trajectories between Evo-Press and our method across models and speedup settings.Benchmark:We evaluate the proposed search strategy on DeiT-Small, DeiT-Base, and Swin-Small under 1.5×\times, 2.0×\times, and 2.5×\timestarget speedups on A100 , RTX 4060 and RTX 4090 GPUs. To improve efficiency, search is conducted on ImageNet-300, a subset of ImageNet-1K with 300 classes, 500 training images, and 50 validation images per class. We find that it preserves the search trends and final configurations observed on full ImageNet-1K while substantially reducing search cost.

Baseline:We instantiate our method within the standard one-shot layer-wise sparsity search framework, keeping supernet training unchanged and replacing only the post-training search stage with our latency-grounded search algorithm. We compare against Evo-Press[17], a representative evolutionary method for layer-wise sparsity search, under the same one-shot framework and latency prediction setup. Under this setting, the dominant cost still comes from supernet training; the full pipeline with 200 search generations takes about 10 hours on three A100 GPUs.

Results:Fig.7shows the search trajectories of our method and Evo-Press across different models and target speedups. Our method exhibits more stable convergence and consistently reaches better final solutions, while Evo-Press more often stagnates near inferior local regions. In many cases, our method overtakes Evo-Press as early as the 20th–30th generation, and the gap further widens under higher speedup targets, where the feasible region becomes more constrained and fragmented. Moreover, the search typically converges within about 100 generations, indicating good search efficiency in practice. These results demonstrate that explicitly incorporating latency constraints into the search process improves both search quality and robustness under practical deployment budgets.

6.3Overall Evaluation

BenchmarkTo evaluate the end-to-end effectiveness of the proposed search-kernel co-design framework, we conduct INT8 sparse inference experiments on DeiT-Small and DeiT-Base using an NVIDIA A100 GPU. All methods are tested under the same inference stack and deployment setting.

BaselinesWe compare different combinations of sparsity configuration strategies and execution backends. On the search side, we consider uniform N:M sparsity, EvoPress[17], and our latency-grounded layer-wise search. On the execution side, we compare nmSPARSE[10]and MD-SpMM as sparse execution backends, while cuBLAS serves as the dense reference. This allows us to isolate the effect of search, kernel, and their combination under the same deployment constraint.

Figure 8:Accuracy-speedup trade-offs across search and kernel combinations.Results:Fig.8summarizes the resulting accuracy–speedup trade-offs, where the shaded feasible region denotes configurations satisfying the target speedup. Improving sparse execution alone is insufficient if sparsity allocation remains weak or latency-unaware. When deploying N:M sparse models with nmSPARSE, irregular execution overhead often offsets arithmetic savings, so more aggressive sparsity is required to meet the target speedup, which in turn causes noticeable accuracy degradation. Conversely, improving sparsity allocation alone is also insufficient without a realizable sparse backend.Uniform N:M sparsity and proxy-driven approaches, such as EvoPress, optimize abstract objectives that are misaligned with deployment latency, often over-pruning latency-critical layers or producing configurations that fail to satisfy the latency constraint.Together, these results show that practical sparse acceleration requires both realizable sparse execution and latency-aligned layer-wise sparsity search.

By contrast, the proposed framework achieves the best overall result by coupling latency-grounded search with MD-SpMM. The former allocates layer-wise sparsity according to deployment behavior, while the latter ensures that these allocations translate into realizable end-to-end speedup on hardware. As a result, compared with conventional strategy combinations, our method consistently delivers better accuracy–speedup trade-offs across all evaluated settings, with an average 2.2×\timesspeedup gain and 1.4% accuracy improvement.

Overall, these results highlight a key principle for practical N:M acceleration: neither sparsity search nor sparse execution alone is sufficient. Favorable accuracy–speedup trade-offs require jointly optimizing layer-wise sparsity configuration and hardware-realizable sparse execution, so that algorithmic sparsity decisions remain aligned with deployment performance.

7Conclusion

This paper argues that practical N:M sparse Transformer acceleration is a search-kernel co-design problem. MD-SpMM makes sparse execution more realizable on Tensor-Core GPUs, while the proposed latency-grounded layer-wise search pipeline allocates sparsity by deployment behavior rather than abstract sparsity objectives. Together, they turn N:M sparsity into a more stable and deployment-effective acceleration mechanism under explicit latency targets.

{credits}

7.0.1Acknowledgements

This work was supported in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant Nos. XDB0660101, XDB0660000, and XDB0660100, in part by the National Natural Science Foundation of China under Grant No. 62502489, in part by Jiangsu Provincial Natural Science Foundation under Grants BK20241818 and BK20250479.

7.0.2\discintname

The authors have no competing interests to declare that are relevant to the content of this article.

References

• [1]Z. Chenet al.(2021)Efficient tensor core-based gpu kernels for structured sparsity under reduced precision.InSC,pp. 1–14.Cited by:§2.2.
• [2]L. Denget al.(2020)Model compression and hardware acceleration for neural networks: a comprehensive survey.Proc. IEEE108(4),pp. 485–532.Cited by:§1.
• [3]C. Fenget al.(2024)LitePred: transferable and scalable latency prediction for hardware-aware neural architecture search.InProc. NSDI,pp. 1463–1477.Cited by:§2.3,§5.1.
• [4]E. Frantaret al.(2022)SPDY: accurate pruning with speedup guarantees.InProc. ICML,pp. 6726–6743.Cited by:§1,§2.3.
• [5]W. Fuet al.(2025)UniCoS: a unified neural and accelerator co-search framework for cnns and vits.InDAC,pp. 1–6.Cited by:§2.1,§5.1.
• [6]T. Galeet al.(2020)Sparse gpu kernels for deep learning.InProc. SC,pp. 1–14.Cited by:§2.2.
• [7]Y. Heet al.(2018)AMC: automl for model compression and acceleration on mobile devices.InECCV,pp. 784–800.Cited by:§1.
• [8]N. Huanget al.(2024)ELSA: exploiting layer-wise n: m sparsity for vision transformer acceleration.InProc. CVPR,pp. 8006–8015.Cited by:§1,§2.1,§5.2.
• [9]X. Kanget al.(2025)SUArch: accelerating layer-wise n: m sparse pattern with a unified architecture for deep-learning edge device.InProc. ASP-DAC,pp. 700–705.Cited by:§1,§1.
• [10]B. Linet al.(2023)Efficient gpu kernels for n: m-sparse weights in deep learning.InProc. MLSys,pp. 513–525.Cited by:2nd item,§1,§1,§2.2,§2.2,§6.1,§6.1,§6.3.
• [11]W. Liuet al.(2024)Arch2End: two-stage unified system-level modeling for heterogeneous intelligent devices.IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst.43(11),pp. 4154–4165.Cited by:§5.1.
• [12]Z. Liuet al.(2022)S2TA: exploiting structured sparsity for energy-efficient mobile cnn acceleration.InProc. HPCA,pp. 573–586.Cited by:§1.
• [13]W. Louet al.(2024)Unleashing network/accelerator co-exploration potential on fpgas: a deeper joint search.IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst.43(10),pp. 3041–3054.Cited by:§1.
• [14]A. Mishraet al.(2021)Accelerating sparse deep neural networks.arXiv.Cited by:§2.2.
• [15]NVIDIA(2020)NVIDIA A100 Tensor Core GPU Architecture.Note:WhitepaperCited by:§2.2.
• [16]Y. Qinet al.(2024)Enhancing long sequence input processing in fpga-based transformer accelerators through attention fusion.InProc. GVLSI,pp. 599–603.Cited by:§1.
• [17]O. Sieberlinget al.(2024)EvoPress: towards optimal dynamic model compression via evolutionary search.InICML,Cited by:§5.2,§6.2,§6.3.
• [18]W. Sunet al.(2021)DominoSearch: find layer-wise fine-grained n: m sparse schemes from dense neural networks.Adv. Neural. Inf. Process. Syst.34,pp. 20721–20732.Cited by:§1,§2.1.
• [19]C. Tanget al.(2026)CloserToMe: a unified framework for accurate and transferable latency prediction across heterogeneous devices.InProc. AAAI,pp. 25805–25813.Cited by:§2.3.
• [20]H. Touvronet al.(2021)Training data-efficient image transformers & distillation through attention.InProc. ICML,pp. 10347–10357.Cited by:§1.
• [21]Z. Wanget al.(2025)UniCoX: a unified cost model for tensorized program tuning across ubiquitous accelerators.IEEE Trans. Comput.75(1),pp. 305–319.Cited by:§5.1.
• [22]Z. Wanget al.(2025)UniSparTa: a unified sparse tensor program tuning framework.IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst..Cited by:§2.2.
• [23]T. Yanget al.(2018)NetAdapt: platform-aware neural network adaptation for mobile applications.InECCV,pp. 285–300.Cited by:§1.
• [24]H. Youet al.(2023)ViTCoD: vision transformer acceleration via dedicated algorithm and accelerator co-design.InProc. HPCA,pp. 273–286.Cited by:§1,§2.1.
• [25]A. Zhouet al.(2021)Learning n: m fine-grained structured sparse neural networks from scratch.InProc. ICLR,Cited by:§1.