CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

# Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits
Source: [https://arxiv.org/html/2607.20518](https://arxiv.org/html/2607.20518)
Xue\-Jian Gao\*, Deng Pan\*, Yueming Su\*, Jiasheng Li, Bin Du, Fengming Zhu, Chengdi Ma, Junyi Fan, Qichen Liao, Chengqiu Hu, Xinxian Chen, Lingchao Zheng, Jun Li, Jiwei Yang, Yuwei Fan\#Huawei

###### Abstract

AI agents are now capable of writing, compiling, and iteratively optimizing low\-level operator kernels on different hardware platforms\. Existing benchmarks, however, focus almost exclusively on CUDA and Triton, leaving hardware ecosystems with less\-exposed programming models without a common evaluation baseline\. We presentCANN Bench, an open benchmark for AI\-generated operator code on Huawei’s Ascend NPU\. The current release covers53 operatorsand1060 test casesorganized into four difficulty tiers—from simple elementwise primitives to MoE dispatch and FlashAttention kernels—spanning FP16, BF16, FP32, and INT8 precision formats\. Evaluation adopts athree\-dimensional weighted composite scorethat treats compilation, functional correctness, and performance as independent axes, providing a principled reward signal for kernel\-generation agents\. Performance is graded against an out\-of\-the\-box PyTorch\-on\-Ascend baseline and an analytical per\-case Hardware\-Anchored Performance \(HAP\) limit on real NPU hardware, ensuring scores reflect genuine optimization headroom rather than measurement artifacts\. The evaluation harness is designed to resist reward hacking from the ground up\. CANN Bench is versioned within the official CANN repository and is designed for long\-term community co\-construction, providing the Ascend ecosystem with a quantitative, reproducible, and sustainably maintained yardstick for AI operator\-authoring capability\.

\*\*footnotetext:These authors contributed equally: Xue\-Jian Gao, Deng Pan, Yueming Su\#\#footnotetext:To whom the correspondence should be addressed:fanyuwei2@huawei\.com*K*eywordsBenchmark⋅\\cdotAscend NPU⋅\\cdotCANN⋅\\cdotKernel Generation⋅\\cdotLLM Agents

## 1Introduction

### 1\.1Why an Ascend\-Oriented Benchmark Now

Code\-generation agents now write, compile, and profile operator kernels end to end\[[6](https://arxiv.org/html/2607.20518#bib.bib11),[23](https://arxiv.org/html/2607.20518#bib.bib12),[12](https://arxiv.org/html/2607.20518#bib.bib13),[1](https://arxiv.org/html/2607.20518#bib.bib9),[27](https://arxiv.org/html/2607.20518#bib.bib10)\], and the benchmarks they target supply the execution\-feedback signal that drives each next round of optimization\[[3](https://arxiv.org/html/2607.20518#bib.bib8),[18](https://arxiv.org/html/2607.20518#bib.bib14),[8](https://arxiv.org/html/2607.20518#bib.bib7),[15](https://arxiv.org/html/2607.20518#bib.bib27)\]\. On Huawei’s Ascend NPU, kernel generation runs through CANN—Huawei’s heterogeneous computing architecture that bridges upper\-layer AI frameworks and the underlying Ascend processors\. Writing a kernel under CANN requires explicit control of the memory hierarchy, tile shapes, and Vector–Cube pipeline scheduling, a programming surface analogous to CUDA on the NVIDIA GPU architecture\. Recent benchmarks have begun to include Ascend targets\[[28](https://arxiv.org/html/2607.20518#bib.bib6),[4](https://arxiv.org/html/2607.20518#bib.bib18),[29](https://arxiv.org/html/2607.20518#bib.bib17),[30](https://arxiv.org/html/2607.20518#bib.bib20),[31](https://arxiv.org/html/2607.20518#bib.bib21),[14](https://arxiv.org/html/2607.20518#bib.bib22)\], but they either treat Ascend as one platform among many or are coupled to a specific agent\-training recipe\. The Ascend ecosystem therefore still lacks a shared evaluation scale for AI\-generated CANN kernels—one independent of any single training recipe and maintained as a cross\-project reference\.

The catalog of operators that must be supported continues to expand\. Each new model family introduces new attention variants and MoE routing patterns that have to be re\-tiled and re\-scheduled for Ascend under CANN\. Precision compounds the growth: modern pipelines mix FP16 and BF16 with low\-precision formats such as FP8 and MXFP4\[[22](https://arxiv.org/html/2607.20518#bib.bib15),[26](https://arxiv.org/html/2607.20518#bib.bib16)\], each with distinct granularity and performance trade\-offs\. Workload churn and precision growth together push the operator catalog beyond what hand\-authored kernels can track\. An Ascend\-oriented benchmark therefore needs to evolve with the hardware and software stack rather than remain fixed at release\. We define six design criteria for such a benchmark below\.

### 1\.2Six Criteria for an Ascend\-Oriented Benchmark

We organize the six criteria into two layers\. The first four concern the task set; the remaining two concern benchmark delivery and maintenance\.

C1\. Difficulty span\.The task set must span the full difficulty range, from element\-wise and reduction primitives, through compute\-bound cores such as convolution and matrix\-multiplication, up to fusion\-level operators such as MoE dispatch and Flash Attention\. Without this range, scores cluster at the trivial or fusion\-only extremes and fail to separate capability tiers\.

C2\. Workload provenance\.Tasks come from real Ascend production workloads—LLM training, quantized inference, and vision/multimodality pipelines—rather than from synthetic cases whose failure modes do not transfer\.

C3\. Precision and quantization coverage\.FP16 and BF16 are the baseline\. Quantized integer paths \(INT8 with per\-tensor and per\-channel granularities\) are equally important in production under CANN, and lower\-precision floating\-point formats such as FP8, HiF8, MXFP8, FP4, and MXFP4\[[22](https://arxiv.org/html/2607.20518#bib.bib15),[26](https://arxiv.org/html/2607.20518#bib.bib16)\]are on the near\-term roadmap \(§[1\.6](https://arxiv.org/html/2607.20518#S1.SS6)\)\.

C4\. Evaluation on three axes\.Compilation, functional correctness, and performance on Ascend NPUs all contribute to the final weighted score; failing any one of them strictly limits the credit a submission can receive on the remaining axes\.

C5\. Self\-contained reproducibility kit\.Each operator ships with a specification, a golden implementation, public test cases, and a performance baseline, so that third parties can reproduce the evaluation\.

C6\. Anti\-reward\-hacking and sustained evolution\.The harness must detect reward\-hacking attempts such as bypassing accuracy assertions, memorizing outputs, or hard\-coding shapes\. The task set and evaluation protocol must also evolve with Ascend hardware, CANN releases, and model families rather than remain fixed at release\.

### 1\.3Existing Landscape Against These Criteria

Measured against the criteria in §[1\.2](https://arxiv.org/html/2607.20518#S1.SS2), existing benchmarks cover only subsets of the design space\. We group the closest prior work by target hardware and summarize their positions in Table[1](https://arxiv.org/html/2607.20518#S1.T1)\.

The CUDA and Triton ecosystem hosts the largest body of kernel\-generation benchmarks\. KernelBench\[[21](https://arxiv.org/html/2607.20518#bib.bib2)\]decomposes evaluation into compilation, correctness, and speedup across 250 PyTorch\-to\-CUDA tasks\. TritonBench\[[16](https://arxiv.org/html/2607.20518#bib.bib3)\]applies a similar three\-axis setup to 184 real\-world Triton operators, where DSL\-concept hallucination is a dominant failure mode\. SOL\-ExecBench\[[17](https://arxiv.org/html/2607.20518#bib.bib4)\]anchors performance to hardware Speed\-of\-Light bounds on NVIDIA Blackwell and adds a sandboxed harness for timing\. All three target NVIDIA platforms\. Their task sets do not transfer directly to Ascend: CANN exposes different tiling semantics and a different multi\-level buffer structure, so a mechanical port fails C2 \(workload provenance\)\.

The same non\-transfer appears on other NPU stacks\. NPUEval\[[13](https://arxiv.org/html/2607.20518#bib.bib19)\]evaluates frontier LLMs on 102 AMD\-NPU kernel tasks and reports an average vectorization score of around 10%\. KernelCraft\[[20](https://arxiv.org/html/2607.20518#bib.bib5)\]studies close\-to\-metal kernel generation across the PLENA, AMD NPU, and Coral NPU ISAs on 33 tasks\. Neither targets CANN nor Ascend\.

Two benchmarks target the Ascend ecosystem more directly, but neither fully satisfies our six criteria\. MultiKernelBench\[[28](https://arxiv.org/html/2607.20518#bib.bib6)\]covers 285 tasks across 14 kernel categories and three hardware platforms—NVIDIA GPUs, Huawei Ascend NPUs, and Google TPUs—using author\-provided references and a category\-aware one\-shot prompting scheme\. Breadth costs per\-platform depth: Ascend tasks are one slice of the total, so per\-operator specification depth is limited \(partial C5\) and Ascend precision coverage is narrow \(partial C3\)\. NPUKernelBench, released alongside the AscendKernelGen fine\-tuning recipe\[[4](https://arxiv.org/html/2607.20518#bib.bib18)\], covers Ascend tasks at three difficulty levels; AscendKernelGen reports a Level\-2 compile rate rising from 0% to 95\.5% Pass@10 with chain\-of\-thought data plus reinforcement learning from on\-device execution feedback\. Two limitations weaken its role as a shared reference\. First, on the static\-shape track that constitutes a substantial portion of NPUKernelBench, all test cases for a given task correspond to a single fixed input configuration\[[4](https://arxiv.org/html/2607.20518#bib.bib18)\], so a kernel specialized to one canonical shape already passes per\-task evaluation\. Second, the benchmark is released and evolved alongside the AscendKernelGen recipe rather than on an independent maintenance track, so its task set tracks the cadence of one training pipeline rather than the broader Ascend ecosystem \(partial C6\)\.

Several kernel\-generation methods now target CANN directly\. AscendCraft\[[29](https://arxiv.org/html/2607.20518#bib.bib17)\]applies DSL\-guided transcompilation through a constraint\-directed intermediate representation\. AscendOptimizer\[[30](https://arxiv.org/html/2607.20518#bib.bib20)\]runs an episodic evolutionary agent with optimization\-rewind over 127 operators from thecann\-opsrepository and reports a 1\.19×\\timesgeometric\-mean speedup\. EvoKernel\[[31](https://arxiv.org/html/2607.20518#bib.bib21)\]addresses cold\-start and continual refinement through value\-driven memory; on a self\-constructed NPU variant of KernelBench it lifts frontier\-model correctness from 11% to 83%\. KernelCAT\[[14](https://arxiv.org/html/2607.20518#bib.bib22)\]reports deployment\-side gains on FlashAttention and DeepSeek\-OCR\-2 on Ascend 910B2\. Each of these efforts uses either a cross\-platform benchmark, a custom task set, or an author\-chosen operator collection, so they still lack a common evaluation scale\.

C6 \(anti\-reward\-hacking and sustained evolution\) has a concrete precedent\. KernelBench has documented silent\-dispatch exploits in practice: submissions that call high\-level operators such ascuBLASin place of the generated CUDA kernel, or no\-op kernels whose output memory is reused from the reference tensor\[[25](https://arxiv.org/html/2607.20518#bib.bib32),[7](https://arxiv.org/html/2607.20518#bib.bib23),[24](https://arxiv.org/html/2607.20518#bib.bib24)\]\. Table[1](https://arxiv.org/html/2607.20518#S1.T1)positions the above benchmarks along five axes: golden\-reference provenance, scoring format, generalization discipline, anti\-reward\-hacking stance, and maintenance mode\. Among publicly described Ascend NPU benchmarks, we are not aware of another that combines co\-location in the official CANN repository, maintenance alongside vendor operator contracts, and a dedicated leaderboard protocol; CANN Bench is designed to cover all three\.

Table 1:Ascend ecosystem kernel\-generation benchmarks: positioning of CANN Bench against relevant benchmarks targeting Ascend NPUs\. Benchmarks targeting other ISAs \(KernelBench, TritonBench, SOL\-ExecBench, KernelCraft, NPUEval\) are omitted from direct comparison\. Columns report provenance of the golden reference, performance\-score composition, test\-case generalization discipline, anti\-reward\-hacking stance, and maintenance mode; “—” denotes an attribute that is not documented for a given benchmark\.
### 1\.4CANN Bench: Dataset, Scoring, and Online Leaderboard

CANN Bench is an operator\-generation benchmark maintained in the official CANN repository\. Its specifications, golden implementations, test cases, and performance baselines sit alongside Huawei’s vendor\-maintained operator contracts \(C2, C5\)\. The initial release targets Ascend 910B2 with a kernel\-DSL\-agnostic specification format designed to extend to Triton, PyPTO, TileLang, and other CANN\-compatible paradigms in future releases\.

#### Dataset\.

CANN Bench currently contains 53 operators and 1060 test cases, distributed 8/16/21/8 across four difficulty levels \(L1–L4\) that track Ascend hardware\-unit usage, from Vector\-only kernels at L1 to fused Matrix–Vector pipelines at L4\. §[2](https://arxiv.org/html/2607.20518#S2)lists representative operators, the full per\-level breakdown, and the four per\-operator reproducibility artifacts \(desc\.md,proto\.yaml,cases\.csv,golden\.py\)\.

#### Public and hidden case split\.

Each operator’s cases split into a 20\-case public set for local iteration and an 80\-case hidden set that the harness retains for leaderboard evaluation\. The leaderboard aggregates both splits, so the hidden set guards against fit to disclosed inputs alone\.

#### Three\-dimensional weighted scoring\.

Compilation acts at the operator level, while functional correctness and performance are scored case by case; the three signals are combined into a per\-operator weighted composite \(§[3\.3](https://arxiv.org/html/2607.20518#S3.SS3.SSS0.Px2)\),

EachOperatorScore=\[wc⋅δpass\+∑i∈casesδaccuracy,i​\(wf\+wp⋅scorei\)num​\_​of​\_​cases\]⋅100,\\mathrm\{EachOperatorScore\}=\[w\_\{c\}\\cdot\\delta\_\{\\mathrm\{pass\}\}\+\\sum\_\{i\\in\\mathrm\{cases\}\}\\frac\{\\delta\_\{\\mathrm\{accuracy\},i\}\(w\_\{f\}\+w\_\{p\}\\cdot\\mathrm\{score\}\_\{i\}\)\}\{\\mathrm\{num\\\_of\\\_cases\}\}\]\\cdot 100,whereδpass∈\{0,1\}\\delta\_\{\\mathrm\{pass\}\}\\in\\\{0,1\\\}is the operator\-level compilation flag \(a kernel that fails to compile zeros out every term\),δaccuracy,i∈\{0,1\}\\delta\_\{\\mathrm\{accuracy\},i\}\\in\\\{0,1\\\}is the per\-case functional indicator,scorei\\mathrm\{score\}\_\{i\}is the per\-case performance sub\-score, andwc,wf,wpw\_\{c\},w\_\{f\},w\_\{p\}are linear weights \(defaultwc=0\.2w\_\{c\}=0\.2,wf=0\.3w\_\{f\}=0\.3,wp=0\.5w\_\{p\}=0\.5\)\. The performance sub\-score is the HAP score: it grades the candidate kernel’s measured runtimeTcand,iT\_\{\\text\{cand\},i\}between a PyTorch\-on\-Ascend baselineTbaseline,iT\_\{\\text\{baseline\},i\}and an analytically derived per\-case Hardware\-Anchored Performance \(HAP\) limitTHW,iT\_\{\\text\{HW\},i\}\(§[3\.2](https://arxiv.org/html/2607.20518#S3.SS2), §[3\.3](https://arxiv.org/html/2607.20518#S3.SS3)\)\.

#### Precision scope\.

The current release covers FP16, BF16, FP32, and quantized integer paths \(notably INT8 for weight and activation quantization\)\. FP8, HiF8, MXFP8, FP4, and MXFP4 remain planned extensions discussed in §[1\.6](https://arxiv.org/html/2607.20518#S1.SS6)\.

#### Online leaderboard\.

CANN Bench is delivered with an online leaderboard111The portal is being prepared and will be released in a future update\.\.

### 1\.5Engineering Substrate and Anti\-Reward\-Hacking

Reliable scoring depends on the harness\. CANN Bench runs compilation, accuracy verification, and performance sampling in a single pipeline\. To control measurement noise on the NPU path, it issues a heavy MatMul\+\+ReduceMax warmup that drives the NPU to its sustained peak DVFS state, evicts the L2 cache between timed runs to neutralize cross\-step cache reuse, and collects kernel\-level timings through thetorch\_npu\.profilersurface, which exposes the same per\-kernel metrics as the Ascendmsproftoolchain\[[9](https://arxiv.org/html/2607.20518#bib.bib26)\]\. To isolate cross\-case state, it executes each operator in a fresh subprocess\. Performance baselines are versioned alongside each task incases\.yamlrather than frozen at release: the values shipped today reflect the CANN release used to record them, and a refresh utility that regeneratesbaseline\_perf\_usas hardware and CANN versions evolve is slated for an upcoming release, so speedups can be re\-anchored to the current attainable reference rather than a stale snapshot\.

Our reward\-hacking defense follows a layered model rather than one monolithic check\. The submission contract rules out high\-level compute delegation, same\-op routing to preinstalled CANN kernels, CPU fallback, output caching, fake or lazy tensors, and timing\-API tampering\. At runtime, the evaluator enforces this contract around framework calls, operator dispatch, device residency, and measurement integrity, while the scorer grants no performance credit when a correctness\-passing run cannot be attributed to candidate NPU\-kernel execution\. Numerical validation remains dtype\-aware, and leaderboard scores aggregate over both public and hidden splits, so a submission cannot be tuned to disclosed inputs alone\.

Two surfaces remain deliberately conservative\. Multi\-stream submissions are restricted rather than fully ranked because stable all\-stream visibility acrosstorch\_npuversions is still fragile, and opaque pre\-compiled binary loading remains a manual\-review item\. CANN Bench also supports evaluator\-machine hardening against same\-op library routing on dedicated evaluation hosts\. The mitigation principles and remaining limitations are detailed in §[3\.4](https://arxiv.org/html/2607.20518#S3.SS4)\.

### 1\.6Status, Roadmap, and Open\-Source Plan

As of this release, CANN Bench provides specifications, golden implementations, public test cases, and performance baselines for 53 operators across L1–L4\. The harness and baseline\-collection pipeline have completed end\-to\-end runs on NPU hardware\. The public leaderboard is being prepared and will be released in a future update\.

Three extensions are planned\. The leaderboard portal will index and date every submission so that results stay directly citable\. Kernel\-DSL coverage will extend the initial Ascend\-optimized support to Triton, PyPTO, TileLang, and other CANN\-compatible paradigms under the same specification and scoring protocol\. Precision coverage will expand the current FP16 / BF16 / FP32 / INT8 baseline to FP8, MXFP8, FP4, MXFP4, HiF8, and related low\-precision formats as operators and baselines are validated\.

The operator set, golden implementations, harness, and documentation are open\-source\. The project accepts contributions of new operators, test cases, baseline updates, and sub\-leaderboards, so the benchmark tracks Ascend hardware and workload evolution over time\.

#### Availability and license\.

CANN Bench is hosted in the official CANN repository at[https://gitcode\.com/cann/cann\-bench](https://gitcode.com/cann/cann-bench)and released under the*CANN Open Software License Agreement Version 2\.0*\[[11](https://arxiv.org/html/2607.20518#bib.bib25)\]\. The harness targets Ascend 910B2 and is validated against CANN≥9\.0\.0\\geq 9\.0\.0, the version we recommend for reproducing the reported baselines; older toolkit releases are not guaranteed to match the publishedbaseline\_perf\_usvalues, which are versioned alongside the harness in each operator’scases\.yamland will be refreshable via a baseline\-update utility scheduled for an upcoming release\.

## 2Benchmark and Dataset

### 2\.1Design Principles

CANN Benchis organized around three principles for Ascend C kernel evaluation\.

1. 1\.Vendor\-authoritative provenance\.Every workload in CANN Bench draws on official artifacts from the CANN repository, including standardized operator definitions, reference implementations, and standard test cases\. CANN Bench is distinct from the vendor\-maintained operator baseopbase222[https://gitcode\.com/cann/opbase](https://gitcode.com/cann/opbase)\.:opbaseis the production operator collection that ships with CANN, while CANN Bench is an evaluation benchmark for AI\-generated kernels whose task set partially overlaps withopbasebut is not identical\. Even on shared tasks, CANN Bench specifications are not transcribed verbatim: we relax a subset of the original functional requirements—for example, narrowing the supported dtype and shape envelope or trimming peripheral code paths—so the evaluation focuses on the core kernel logic rather than the surrounding production machinery\. Beyond these canonical operators, the benchmark also includes emerging custom operators absent from the official codebase, such asEngramfrom DeepSeek\[[5](https://arxiv.org/html/2607.20518#bib.bib1)\]\. Combining canonical and newly proposed operators broadens the scope on which generation and optimization can be evaluated\.
2. 2\.Four levels with increasing difficulty\.Operators are grouped into four hierarchical levels \(L1–L4\), from element\-wise kernels at L1 through progressively composite, contraction\-heavy, and fused production kernels at L4\. The levels also map to a hardware\-utilization progression: L1 and L2 use only vector operations, L3 introduces basic combinations of vector and cube operations, and L4 requires tightly coordinated vector–cube pipelines within a single fused kernel\. Table[2](https://arxiv.org/html/2607.20518#S2.T2)summarizes the four levels\. Our levels reflect how each operator maps onto Ascend’s execution model, not shallow features such as code length or operator arity that prior kernel\-generation benchmarks have used as difficulty proxies\. The level at which an agent first stalls thus localizes its capability gap: a system that clears L1 and L2 cleanly but degrades sharply at L3 reveals a specific weakness on cube\-side reasoning rather than producing only a lower aggregate score\. The resulting per\-level profile delivers the capability\-tier separation that C1 calls for, not a single\-axis difficulty score\.
3. 3\.Case diversity rather than single\-shape scoring\.Each operator carries a family of concrete cases that vary tensor shape, dtype, and attribute settings\. The design avoids rewarding kernels that overfit a single canonical configuration and instead tests whether generated code generalizes across the combinations that matter in real Ascend C development\. Across the current split, the cases cover float16 / bfloat16 / float32 plus integer paths \(int8, uint8, int32, int64\) and mixed\-dtype configurations\.

### 2\.2Benchmark Structure and Dataset Overview

Table[2](https://arxiv.org/html/2607.20518#S2.T2)summarizes the current CANN Bench release:53 operators, each with about 20 test cases\. The benchmark is versioned alongside the CANN repository and grows as additional operators are promoted from the experimental track into the curated set\. Figure[1](https://arxiv.org/html/2607.20518#S2.F1)complements Table[2](https://arxiv.org/html/2607.20518#S2.T2)with four views of the 1060 test cases\.

Table 2:A summary of CANN Bench\. The four levels track a progression in implementation difficulty, from local pointwise kernels to production\-grade fused pipelines\.#### Level distribution \(Figure[1\(a\)](https://arxiv.org/html/2607.20518#S2.F1.sf1)\)\.

Operators are distributed across four hierarchical levels as mentioned above\. The third level contains the most operators \(21\), spanning contraction, sorting, layout conversion, and fused quantization kernels under a heterogeneous mix of vector and cube workloads\. The second level follows \(16\), highlighting the widespread use of multi\-input normalization, indexing, and fused utility kernels in practical Ascend workloads\. The first and fourth levels are tied for the fewest \(8 each\): L1 concentrates on single\-primitive vector kernels, while L4 gathers the most engineering\-heavy fused pipelines under the current classification\.

#### Operation category distribution \(Figure[1\(b\)](https://arxiv.org/html/2607.20518#S2.F1.sf2)\)\.

The category labels span a wide range of operator types\.*FusedComposite*is the largest category, followed by*Contraction*,*Elementwise*, and*SortSelect*\. Composition differs across levels: L1 centers on elementwise kernels, L2 on fused composite and normalization operators, L3 on contraction and reordering\-heavy kernels, and L4 consists entirely of fused composite operators\. The distribution shows that CANN Bench extends beyond pure GEMM evaluation to indexing, reduction, layout manipulation, and control\-intensive operators—common failure points for AI\-assisted Ascend C code generation\.

#### Data\-type distribution \(Figure[1\(c\)](https://arxiv.org/html/2607.20518#S2.F1.sf3)\)\.

The figure reports the primary compute dtype of each operator\. L1 and L2 are dominated by float16, bfloat16 and float32 paths, which matches the typical workload of activation, normalization and utility kernels\. L3 and L4 contain a larger share of mixed\-dtype and integer cases, as several operators in those tiers fuse floating\-point activations with quantized weights, integer metadata, and multi\-input specifications\.

![Refer to caption](https://arxiv.org/html/2607.20518v1/x1.png)\(a\)Operator count per difficulty level\.
![Refer to caption](https://arxiv.org/html/2607.20518v1/x2.png)\(b\)Operation category distribution by level\.
![Refer to caption](https://arxiv.org/html/2607.20518v1/x3.png)\(c\)Primary compute dtype distribution per level\.
![Refer to caption](https://arxiv.org/html/2607.20518v1/x4.png)\(d\)Total number of elements of the test cases per level\.

Figure 1:Dataset overview of our CANN Bench\. \([1\(a\)](https://arxiv.org/html/2607.20518#S2.F1.sf1)\) Operator count per difficulty level\. \([1\(b\)](https://arxiv.org/html/2607.20518#S2.F1.sf2)\) Operation category distribution, stacked by level contribution\. \([1\(c\)](https://arxiv.org/html/2607.20518#S2.F1.sf3)\) Distribution of the primary compute data type per level\. \([1\(d\)](https://arxiv.org/html/2607.20518#S2.F1.sf4)\) Distribution of the provided baseline execution times \(log\-scale\)\.

### 2\.3Test\-Case Design Methodology

Each operator in CANN Bench carries a structured case set rather than a hand\-picked collection of inputs\. The construction follows four explicit coverage axes and ends with per\-operator uniqueness constraints, so each case contributes evidence the others do not\.

#### Tensor\-size coverage\(Figure[1\(d\)](https://arxiv.org/html/2607.20518#S2.F1.sf4)\)\.

Each case is sized by its total*input element count*—the sum of∏idi\\prod\_\{i\}d\_\{i\}over all of its input tensors—which spans roughly10610^\{6\}to3×1083\{\\times\}10^\{8\}\. The benchmark covers this range across three regimes: small cases \(a few million elements\) sit inside the on\-chip buffers and surface tiling and launch\-overhead issues, large cases \(around10810^\{8\}and above\) force off\-chip memory traffic and expose bandwidth\-bound behavior, and a medium band \(tens of millions\) covers the bulk of production shapes\. A kernel that overfits to a single regime cannot collect the per\-case credit at the other two\.

#### Axis alignment\.

We label a shape*aligned*when the byte length of its innermost dimension is a multiple of512 B, and*non\-aligned*otherwise\. The threshold is grounded in the Ascend MTE \(Memory Transfer Engine\) DMA path: a 512 B\-aligned innermost stride lets MTE issue full\-burst transfers between global memory and the on\-chip Unified Buffer at peak bandwidth, whereas a non\-aligned trailing dimension forces MTE to break the transfer into a head/tail pair of slower unaligned bursts plus masked tail handling, materially degrading effective bandwidth even when functional correctness is preserved\. Concretely, a 512 B boundary corresponds to256256elements for FP16/BF16,128128elements for FP32, and512512elements for INT8; common ML\-friendly shapes such as10241024,20482048, or40964096in the innermost axis fall on the boundary, while a prime or odd innermost dimension generally does not\. Within each operator’s case set we require*at least 50% non\-aligned cases*, where the ratio is computed over the operator’s 20 public cases and a case counts as non\-aligned whenever the byte length of any input tensor’s innermost dimension is not a multiple of 512 B\. Inside the non\-aligned subset we further bias toward primes \(e\.g\.10091009,10211021,15371537,40974097,1,000,0031\{,\}000\{,\}003\) so that residual sizes do not collapse onto convenient divisors that an aggressive auto\-tiler might still find friendly\. This is the operational definition behind the alignment\-stress requirement referenced in §[1\.5](https://arxiv.org/html/2607.20518#S1.SS5)\.

#### Dimensionality\.

The case set spans 1D through 5D inputs rather than freezing each operator at a single rank\. Mixing ranks stresses tiling along axes other than the canonical innermost one and surfaces broadcast and reshape paths that a 2D\-only or 3D\-only test would not\.

#### Value\-range sweep\.

Inputs are drawn from a fixed taxonomy \(Table[3](https://arxiv.org/html/2607.20518#S2.T3)\) that covers symmetric and asymmetric ranges of three magnitudes, dtype boundaries \(e\.g\.±65504\\pm 65504for fp16,±88\\pm 88for the fp32 exponent\), the special values±∞\\pm\\inftyandNaN\\mathrm\{NaN\}, and the all\-zero degenerate case\. For ranges containing±∞\\pm\\infty, the data generator injects the corresponding extremes into the head and tail of the flattened tensor while filling the interior with regular random values, so that overflow handling, denormal flushing, and special\-value propagation are observable side\-by\-side with normal arithmetic\.

Table 3:Value\-range categories that populate test\-case inputs\. Each operator’s 20\-case public split is required to draw from at least the symmetric\-small, type\-boundary, and special\-value groups, so that no operator is evaluated only on benign data\.
#### Uniqueness constraints\.

Within each operator’s 20\-case public split we require three properties to be unique across cases: total input element count, the JSON\-serialized attribute combination, and the value range\. The constraint blocks a common failure mode in hand\-curated benchmarks where two cases differ only cosmetically and effectively count once\. A self\-check loop with at least three rounds of validation runs in the contributing pipeline before a new case set lands in the repository\.

#### Public and hidden splits\.

The 20 public cases per operator ship in the open\-source repository and support local iteration, inspection, and reproducibility\. An additional 80 hidden cases per operator are*not*open\-sourced: they reside inside the leaderboard’s server\-side evaluation backend, where submissions are scored after upload\. Hidden cases follow the same coverage spec and uniqueness constraints as the public split, so a kernel tuned only against the public set has no informational shortcut to the hidden one\. Keeping the hidden split off\-repo is deliberate—releasing it would collapse the split into a single 100\-case public set and remove the protection against fitting to disclosed inputs that is its entire purpose\. The submission backend is being prepared together with the online leaderboard and will be released in a future update\.

### 2\.4Problem Specification Format

Each task ships with four artifacts; SOL\-ExecBench\[[17](https://arxiv.org/html/2607.20518#bib.bib4)\]adopts a similar but smaller set\.

1. 1\.Operator specification \(desc\.md\)\.A structured description covering the mathematical definition, interface contract \(input/output shapes, dtypes, and constraints\), and numerical accuracy tolerance\.
2. 2\.Operator prototype \(proto\.yaml\)\.A machine\-readable YAML file capturing the operator name, category, schema, attributes, shape support, and supported data types for inputs and outputs\.
3. 3\.Test cases \(cases\.csv\)\.A CSV table whose rows are concrete test cases, each specifying input shapes, data types, hyper\-parameters, and the baseline NPU execution time in microseconds\.
4. 4\.Golden implementation \(golden\.py\)\.A Python reference implementation, typically in PyTorch, that produces the expected output for each test case\.

Together, the four difficulty levels, the case\-design methodology, and the four\-artifact specification format fix the dataset surface that any submission is evaluated against: which operators are in scope, which configurations are tested per operator, and what each task hands the candidate kernel as input\. The next section turns from the dataset to the scoring side: how the harness converts a submission’s behavior on these artifacts into compilation, functional, and performance signals, and how those signals are combined into a per\-operator and benchmark\-wide composite score\.

## 3Evaluation

CANN Bench scores each test case on three axes: compilation, functional correctness, and performance\. The first two are binary gates; performance is continuous, anchored to a hardware\-derived bound rather than a moving framework baseline\. Linear weights combine the three into a per\-operator composite\.

For each test case, three reference times are recorded:TbaselineT\_\{\\text\{baseline\}\}, the measured runtime of the operator’s CANN\-internal implementation on the target Ascend NPU;THWT\_\{\\text\{HW\}\}, the analytical Hardware\-Anchored Performance \(HAP\) limit serving as a per\-case lower bound; andTcandT\_\{\\text\{cand\}\}, the measured runtime of the candidate kernel\. The first two are computed once per case at benchmark\-construction time;TcandT\_\{\\text\{cand\}\}is collected at evaluation alongside the compilation and accuracy flags\. Section[3\.1](https://arxiv.org/html/2607.20518#S3.SS1)specifies the functional\-correctness standard\. Section[3\.2](https://arxiv.org/html/2607.20518#S3.SS2)motivates the HAP design and definesTHWT\_\{\\text\{HW\}\}\. Section[3\.3](https://arxiv.org/html/2607.20518#S3.SS3)gives the per\-case performance score and assembles the three axes into per\-operator and suite scores\. Section[3\.4](https://arxiv.org/html/2607.20518#S3.SS4)closes with reward\-hacking surfaces and the mitigation principles the evaluator deploys\.

### 3\.1Functional\-Correctness Standard

CANN Bench adopts a general and systematic precision standard as its functional\-correctness gate\. The standard summarizes per\-tensor agreement with the reference through two statistics on the elementwise relative error—its mean and its max\. If that first screen fails, the checker attributes the failing positions to the normal, small\-value, or cancellation regions; only failures confined to the two numerically unstable regions are eligible for an ErrorCount\-ratio fallback\. The standard also admits bounded per\-operator relaxation when the underlying compute is unavoidably lossy\. The four engineering safeguards collected at the end of this subsection layer around the gate so that a kernel cannot pass it by exploiting the harness rather than by computing correctly\.

#### Mean and max relative error\.

For each elementiiat which both the kernel outputy^i\\hat\{y\}\_\{i\}and the referenceyiy\_\{i\}are finite, define the elementwise relative error \(RE\) as

ei=\|y^i−yi\|\|yi\|\+10−7,e\_\{i\}\\;=\\;\\frac\{\|\\hat\{y\}\_\{i\}\-y\_\{i\}\|\}\{\|y\_\{i\}\|\+10^\{\-7\}\},\(1\)where the10−710^\{\-7\}guard prevents a degenerate division whenyiy\_\{i\}is exactly zero\. For dtypeddwith default thresholdτd\\tau\_\{d\}, the tensor clears the first accuracy screen when both the mean and the max of\{ei\}\\\{e\_\{i\}\\\}sit below their respective bounds:

meani⁡ei<τdandmaxi⁡ei<10​τd\.\\operatorname\{mean\}\_\{i\}e\_\{i\}\\;<\\;\\tau\_\{d\}\\quad\\text\{and\}\\quad\\max\_\{i\}e\_\{i\}\\;<\\;10\\,\\tau\_\{d\}\.\(2\)We refer to these two statistics as the mean RE and the max RE\. Positions containing±∞\\pm\\inftyorNaN\\mathrm\{NaN\}on either side are excluded from\{ei\}\\\{e\_\{i\}\\\}and handled separately by the special\-value rule below\. When the first screen fails, any max\-RE failure outside the small\-value and cancellation regions is a normal\-value failure; failures inside those two regions use the fallback rule below\. For multi\-output operators every output tensor must clear the gate independently\.

#### Per\-dtype thresholds\.

Table[4](https://arxiv.org/html/2607.20518#S3.T4)lists the normal\-rangeτd\\tau\_\{d\}entries for the floating\-point dtypes covered by the precision standard and mirrored in the evaluator\. The values are powers of two anchored to each format’s mantissa width, so the bound tightens monotonically as the dtype gains precision; the max\-RE bound is taken to be exactly10​τd10\\,\\tau\_\{d\}rather than the nearest power of two, matching the evaluator’s implementation\. Integer dtypes are evaluated by exact equality by default, while an operator may declare a bounded integer tolerance for quantized outputs\. FP16, BF16, and FP32 are the active floating\-point leaderboard gates in the current release; HiF32 and FP8 E4M3/E5M2 are implementation\-backed threshold entries included for consistency with the region\-specific tables in Appendix[A](https://arxiv.org/html/2607.20518#A1)\. MXFP8, FP4, and MXFP4 remain roadmap formats \(§[1\.6](https://arxiv.org/html/2607.20518#S1.SS6)\)\.

Table 4:Per\-dtype thresholds for the mean/max\-RE gate\. Eachτd\\tau\_\{d\}is a power of two anchored to the format’s mantissa width; the max\-RE bound is permitted to be ten times looser than the mean\-RE bound so that an isolated outlier on a small subset of positions does not by itself fail an otherwise\-correct tensor\.
#### Three\-region verdict\.

The mean/max\-RE rule is the headline gate, but treats every elementwise position identically—which is the wrong default in two regions where relative error stops being a reliable signal on its own\. The evaluator therefore inspects the failing positions after the first screen and applies a region\-specific fallback rather than rejecting the tensor outright\.

*Specific consideration on the small\-value region\.*When the truncated referenceytruncy\_\{\\mathrm\{trunc\}\}sits at or below the dtype’s representable noise floor, the denominator\|yi\|\+10−7\|y\_\{i\}\|\+10^\{\-7\}ineie\_\{i\}amplifies a single\-ULP quantization difference into a spuriously large relative error even when the kernel is computing correctly\. This regime is routine in the tails of normalization layers, in softmax outputs that approach zero, and in gradient updates near a local minimum; flagging these positions as functional failures would penalize correct kernels for the dtype’s own granularity rather than for any real error\.

*Specific consideration on the cancellation region\.*When two intermediate quantities of comparable magnitude subtract to produce a near\-zero output, catastrophic cancellation \(Kahan\) wipes out the leading significant bits of the result\. A correct kernel can then emit\|y^i\|≈0\|\\hat\{y\}\_\{i\}\|\\approx 0whose elementwise relative error against the FP64 Golden is arbitrarily large—not because the arithmetic is wrong but because the result’s dynamic range no longer matches the reference at the device dtype\. Attention score differences, mean\-zero shifts in layer normalization, and residual deltas in optimizer state all routinely visit this regime\.

For both regions the evaluator switches from relative error to an absolute\-error count and compares the NPU count against a same\-precision CPU run of the Golden:

ErrorCountNPUmax⁡\(ErrorCountCPU,1\)≤2,\\frac\{\\mathrm\{ErrorCount\}\_\{\\mathrm\{NPU\}\}\}\{\\max\\bigl\(\\mathrm\{ErrorCount\}\_\{\\mathrm\{CPU\}\},\\,1\\bigr\)\}\\;\\leq\\;2,\(3\)applied independently to the small\-value and cancellation populations with the region\-defining and counting thresholds tabulated in Appendix[A](https://arxiv.org/html/2607.20518#A1)\. The CPU same\-precision reference is what makes this comparison meaningful: errors attributable to the dtype itself appear on both sides of the ratio and cancel out, so only kernel\-induced contributions survive\. Once the first screen has failed, the tensor can still pass only if no failing position lies in the normal region and both special\-region ratios clear their respective bounds\.

#### Per\-operator relaxation\.

The thresholds in Table[4](https://arxiv.org/html/2607.20518#S3.T4)are defaults rather than invariants\. The actual numerical loss of a correct implementation depends on the operator’s compute graph: chained transcendental evaluations, repeated reductions, quantization, and stateful updates accumulate error in ways a single dtype\-level bound cannot capture uniformly\. The benchmark therefore allows an operator’sproto\.yamlto override the effective threshold for any subset of its supported dtypes, decided case by case during operator authorship\. The published standard recommends capping floating\-point relaxation at10×10\\timesthe default so that relaxations remain bounded; submissions are scored against the effective threshold declared by the operator, and that threshold is carried in the evaluation result so that a relaxed pass remains distinguishable from a strict one\.

The four engineering safeguards below operate orthogonally to the threshold rule and apply to every dtype\.

#### High\-precision reference\.

By default, the CANN Bench Golden path runs on CPU after upcasting floating\-point inputs to FP64, which keeps the reference well above the device’s native precision and prevents overflow or underflow during the reference computation itself from corrupting the truth\. The evaluator also supports native\-precision CPU and NPU Golden modes for specialized benchmark integrations, but the default CANN configuration uses the FP64 CPU strategy\. Beforeeie\_\{i\}is computed, the Golden output is cast to the kernel output’s dtype, so that the thresholdτd\\tau\_\{d\}is interpreted at the device dtype rather than at the reference dtype\. Integer and boolean inputs retain their original dtype throughout, which keeps the path compatible with operators that reject a Double substitute \(e\.g\.Gcdon integer inputs,CrossEntropyLossonint64targets\)\.

#### Special\-value handling\.

NaN\\mathrm\{NaN\}positions must match between the kernel output and the Golden\. Matching±∞\\pm\\inftyentries with the same sign are excluded from the mean\-RE and max\-RE statistics in Eq\. \([2](https://arxiv.org/html/2607.20518#S3.E2)\), while an infinity sign mismatch is a hard failure\. One\-sided±∞\\pm\\inftyentries are treated as saturation candidates: the infinite side is replaced by the maximum finite value of the output dtype and then subjected to the ordinary error gate, so saturation is not a blanket pass\.

#### Opt\-in two\-trial fresh\-input verification\.

The evaluator ships an opt\-in second\-pass mode for the accuracy gate\. When the accuracy\-retry mode is enabled and a case clears the gate on the first trial, the evaluator re\-samples a fresh input batch through the same case generator, perturbs every floating\-point tensor by0\.010\.01to defeat seed reuse, and re\-runs Golden and candidate execution; only cases that clear both trials count toward functional correctness\. A submission that caches the first\-trial output by data pointer or short\-circuits its kernel after the first execution fails the second trial with stale or garbage results\. The default leaderboard pipeline currently invokes the single\-trial path; the second pass is retained as an explicit hardening option\.

#### Strict tensor type\-identity\.

The evaluator validates each returned tensor through a strict type\-identity check against the concretetorch\.Tensorclass rather than a permissiveisinstancetest, which rejectsFakeTensorand other lazy\-evaluation wrappers that can satisfy anisinstancecheck without performing any real computation\.

#### Roadmap toward a more scientific standard\.

The mean/max\-RE rule with three\-region verdicts is the current functional\-correctness gate; two follow\-on directions are planned\. First, the benchmark will migrate from the ecosystem\-operator standard now in use to Huawei’s commercial operator precision standard as the latter stabilizes\. Second, the per\-dtype tier will extend to cover the emerging low\-precision formats on the precision roadmap in §[1\.6](https://arxiv.org/html/2607.20518#S1.SS6)\. Appendix[A](https://arxiv.org/html/2607.20518#A1)records the HiF32/FP8 starting thresholds already mirrored in the evaluator; MXFP8, FP4, and MXFP4 will require additional validation before they become leaderboard gates\. These refinements are co\-developed with the CANN operator\-base maintainers\.

### 3\.2Performance anchor: the Hardware\-Anchored Performance \(HAP\) limit

The most natural design for a kernel\-generation benchmark is to score a submission by its speedup over a software reference\. For a benchmark that aims to remain stable across years, this design has a fundamental defect: the reference drifts with framework and driver versions\. A kernel that scored2×2\{\\times\}over last quarter’s CANN reference may score only1\.4×1\.4\{\\times\}against a faster successor without any change to the kernel itself\. Longitudinal comparison loses meaning, and a published number becomes a relative quantity that ages out within months\.

Prior GPU benchmarks have also argued for hardware\-derived anchors when software\-reference speedup is too volatile\[[17](https://arxiv.org/html/2607.20518#bib.bib4)\]\. CANN Bench instantiates this idea through an Ascend\-specific analytical anchor that we refer to as the*Hardware\-Anchored Performance*\(HAP\) limit, denotedTHWT\_\{\\text\{HW\}\}\. This HAP limit is the upper end of the score, whileTbaselineT\_\{\\text\{baseline\}\}remains the lower anchor so that progress is still framed against the engineering reference users actually have access to\. The framework\-dependent component appears only at the lower end of the score; the upper end is anchored to the chip itself\.

To make “how fast can a kernel run” precise on Ascend 910B2, two facts must be acknowledged: 910B2 is a heterogeneous architecture in which six execution units operate concurrently in time, and achievable performance depends on a non\-trivial space of implementation choices\.

#### Six\-unit decomposition\.

Let

K=\{cube,vector,mte1,mte2,mte3,fixp\}K\\;=\\;\\\{\\,\\text\{cube\},\\ \\text\{vector\},\\ \\text\{mte1\},\\ \\text\{mte2\},\\ \\text\{mte3\},\\ \\text\{fixp\}\\,\\\}denote the set of execution units that operate concurrently within a single AIC group\. Cube performs matrix contractions in the L0A×\\timesL0B→\\toL0C accumulator; Vector handles elementwise, reduction, and complex\-activation work in UB; MTE2 and MTE3 are the GM↔\\leftrightarrowUB data\-movement channels; MTE1 issues L1→\\toL0A/L0B prefetch for Cube; FixP is the L0C→\\toGM write\-back path that fuses simple epilogues such as bias, scale, quant, clip, and ReLU\. The 24 AIC groups on a 910B2 die replicate this layout in parallel\.

For a fixed kernel, lettk​\(δ\)t\_\{k\}\(\\delta\)denote the work time on unitkkunder implementation choiceδ\\delta\(work divided by peak rate\), whereδ\\deltaranges over active\-core count, tile shape, schedule, buffer allocation, and pipeline depth\. Because the six units overlap in time, the device\-side dataflow time is bounded below by their maximum; the hardware limit is the minimization of this maximum across the implementation space:

THW=minδ⁡maxk∈K⁡tk​\(δ\)\.T\_\{\\text\{HW\}\}\\;=\\;\\min\_\{\\delta\}\\;\\max\_\{k\\in K\}\\;t\_\{k\}\(\\delta\)\.\(4\)THWT\_\{\\text\{HW\}\}is the runtime a kernel would achieve under three idealizations: every unit operating at peak rate, perfect overlap across all six units, and zero host\-side launch overhead\. It depends only on the operator’s intrinsic FLOP and byte counts and on the published 910B2 peak parameters; it remains constant across software\-stack upgrades and CANN releases\. No real implementation can violate this bound\.

#### Derivation procedure\.

THWT\_\{\\text\{HW\}\}is computed once per case through an analytical procedure grounded in 910B2 architectural documentation and published peak parameters\. The procedure takes as input the operator’s mathematical definition, its interface contract, and the case’s shape and dtype\. For each unitk∈Kk\\in Kit estimatestkt\_\{k\}from the operator’s FLOP and byte counts and that unit’s peak rate, takes the per\-unit maximum, and minimizes over a structured implementation\-choice spaceδ\\deltathat respects on\-chip buffer constraints \(L1 capacity, L0A/L0B capacity, L0C accumulator size\)\. The procedure outputsTHWT\_\{\\text\{HW\}\}together with the dominant\-bottleneck label\. The value ofTHWT\_\{\\text\{HW\}\}is determined entirely by the hardware specification: should the target hardware platform or its six\-unit characterization be revised,THWT\_\{\\text\{HW\}\}is regenerated accordingly\.

#### Per\-case anchor pair\.

Each case is characterized by the analytical anchor pair\(Tbaseline,THW\)\(T\_\{\\text\{baseline\}\},\\,T\_\{\\text\{HW\}\}\), which combines at evaluation time with the candidate kernel’s measured runtimeTcandT\_\{\\text\{cand\}\}\. By the physical meaning of the hardware limit,THW≤TbaselineT\_\{\\text\{HW\}\}\\leq T\_\{\\text\{baseline\}\}should hold on every case\. Any observation that violates this inequality triggers an evaluation review as an audit signal on either the analytical lower\-bound derivation or the timing measurement\.

#### Bottleneck Attribution

Each test case is annotated with the dominant hardware bottleneck that limitsTHWT\_\{\\text\{HW\}\}, classified into three categories:*Vector*\(Vector Core arithmetic\),*Cube*\(Cube Core matrix computation\), and*MTE*\(Memory Transfer Engine for data movement, rolling up MTE1/MTE2/MTE3 and the FixP writeback path\)\. As shown in Figure[2\(a\)](https://arxiv.org/html/2607.20518#S3.F2.sf1), MTE dominates at 63% of the 1060 cases, with Vector at 26% and Cube at 11%\. This distribution reflects the full Ascend pipeline workload, not just the compute engines\.

![Refer to caption](https://arxiv.org/html/2607.20518v1/figs/hw_bottleneck_pie.png)\(a\)Dominant hardware bottleneck across the 1060 cases: MTE 63%, Vector 26%, Cube 11%\.
![Refer to caption](https://arxiv.org/html/2607.20518v1/figs/perf_score.png)\(b\)Per\-case performance score versus candidate runtimeTcandT\_\{\\text\{cand\}\}: anchored at11whenTcand=THWT\_\{\\text\{cand\}\}=T\_\{\\text\{HW\}\}and at0\.50\.5whenTcand=TbaselineT\_\{\\text\{cand\}\}=T\_\{\\text\{baseline\}\}, decaying toward0for slower kernels\.

### 3\.3Per\-case performance score and composite scoring

#### Per\-case performance score

LetTcandT\_\{\\text\{cand\}\}denote the candidate kernel’s device\-side measured runtime\. The per\-case HAP score is defined as

Scorei=Tbaseline,i−THW,i\(Tcand,i−THW,i\)\+\(Tbaseline,i−THW,i\)\.\\mathrm\{Score\}\_\{i\}\\;=\\;\\frac\{T\_\{\\text\{baseline\},i\}\-T\_\{\\text\{HW\},i\}\}\{\(T\_\{\\text\{cand\},i\}\-T\_\{\\text\{HW\},i\}\)\+\(T\_\{\\text\{baseline\},i\}\-T\_\{\\text\{HW\},i\}\)\}\.\(5\)The score has three anchor properties:

- •Tcand=Tbaseline⇒Score=0\.5T\_\{\\text\{cand\}\}=T\_\{\\text\{baseline\}\}\\;\\Rightarrow\\;\\mathrm\{Score\}=0\.5\(matching the reference implementation is the midpoint\);
- •Tcand=THW⇒Score=1T\_\{\\text\{cand\}\}=T\_\{\\text\{HW\}\}\\;\\Rightarrow\\;\\mathrm\{Score\}=1\(matching the hardware limit is a perfect score\);
- •Tcand→∞⇒Score→0T\_\{\\text\{cand\}\}\\to\\infty\\;\\Rightarrow\\;\\mathrm\{Score\}\\to 0\(slower kernels decay smoothly toward zero\)\.

Figure[2\(b\)](https://arxiv.org/html/2607.20518#S3.F2.sf2)plotsScorei\\mathrm\{Score\}\_\{i\}as a function ofTcandT\_\{\\text\{cand\}\}for fixed\(Tbaseline,THW\)\(T\_\{\\text\{baseline\}\},T\_\{\\text\{HW\}\}\), making the three anchor properties visible at a glance\. The midpoint design separates three regimes on a single bounded scale: worse than the reference implementation \(Score<0\.5\\mathrm\{Score\}<0\.5\); better than the reference implementation but sub\-limit \(0\.5<Score<10\.5<\\mathrm\{Score\}<1\); and at or beyond the hardware limit \(Score≥1\\mathrm\{Score\}\\geq 1\)\. Submissions that exceed the limit are not capped at11; the formula remains well\-defined and grows past11, but the case is flagged for review, sinceTcand<THWT\_\{\\text\{cand\}\}<T\_\{\\text\{HW\}\}implies one of three things: an under\-tightTHWT\_\{\\text\{HW\}\}derivation, a measurement\-side exploit \(§[3\.4](https://arxiv.org/html/2607.20518#S3.SS4)\), or a genuine modelling gap in the analytical procedure\. Each merits case\-by\-case investigation rather than silent absorption into the score\.

Compared with a raw baseline ratioTbaseline/TcandT\_\{\\text\{baseline\}\}/T\_\{\\text\{cand\}\}, equation \([5](https://arxiv.org/html/2607.20518#S3.E5)\) is bounded, comparable across operators, and remains well\-defined whenTbaselineT\_\{\\text\{baseline\}\}is already close to the hardware limit\. The formula awards similar credit for similar fractional progress within the reference\-to\-limit window, regardless of how generous the reference happens to be on the case at hand\.

#### Composite Score

Equation \([5](https://arxiv.org/html/2607.20518#S3.E5)\) grades a single test case along the performance axis\. A submission, however, is ranked at the operator level, and a leaderboard entry covers all evaluated operators\. We therefore lift the per\-case score to a per\-operator composite that also folds in compilation and functional correctness\.

Three signals enter the composite\. At the operator level, a single compilation flagδpass∈\{0,1\}\\delta\_\{\\mathrm\{pass\}\}\\in\\\{0,1\\\}records whether the submitted source compiles; under the official “one submission per operator” protocol this flag applies to the entire operator rather than to individual cases\. At the case level, each case carries a functional indicatorδaccuracy,i∈\{0,1\}\\delta\_\{\\mathrm\{accuracy\},i\}\\in\\\{0,1\\\}\(does the kernel match the golden reference under the per\-dtype precision standard\) and a per\-case performance scorescorei\\mathrm\{score\}\_\{i\}from Equation \([5](https://arxiv.org/html/2607.20518#S3.E5)\)\. The three axes are combined under linear weightswc=0\.2w\_\{c\}=0\.2,wf=0\.3w\_\{f\}=0\.3,wp=0\.5w\_\{p\}=0\.5for compilation, functional correctness, and performance, withδaccuracy,i≡0\\delta\_\{\\mathrm\{accuracy\},i\}\\equiv 0wheneverδpass=0\\delta\_\{\\mathrm\{pass\}\}=0\(a kernel that fails to compile cannot pass any case\)\. The per\-operator composite averages the weighted contributions across the operator’s cases and is rescaled to a0–100100range:

EachOperatorScore=\[wc⋅δpass\+∑i∈casesδaccuracy,i​\(wf\+wp⋅scorei\)len​\(cases\)\]⋅100,\\mathrm\{EachOperatorScore\}=\\left\[w\_\{c\}\\cdot\\delta\_\{\\mathrm\{pass\}\}\+\\sum\_\{i\\in\\mathrm\{cases\}\}\\frac\{\\delta\_\{\\mathrm\{accuracy\},i\}\\,\(w\_\{f\}\+w\_\{p\}\\cdot\\mathrm\{score\}\_\{i\}\)\}\{\\mathrm\{len\}\(\\mathrm\{cases\}\)\}\\right\]\\cdot 100,\(6\)where the operator\-level flagδpass\\delta\_\{\\mathrm\{pass\}\}is replicated across every case in the sum, so the compilation term contributes a constantwc⋅δpassw\_\{c\}\\cdot\\delta\_\{\\mathrm\{pass\}\}to the average regardless of how many cases the operator carries\. The structure is therefore additive rather than multiplicative: compilation contributes a single operator\-widewcw\_\{c\}floor on success, and each functionally passing case independently contributes a fixedwfw\_\{f\}floor plus a performance termwp⋅scoreiw\_\{p\}\\cdot\\mathrm\{score\}\_\{i\}that scales with the HAP\-anchored fractional score of Equation \([5](https://arxiv.org/html/2607.20518#S3.E5)\), before the whole quantity is normalised bylen​\(cases\)\\mathrm\{len\}\(\\mathrm\{cases\}\)and lifted onto a0–100100scale\. Per\-level and benchmark\-level totals are simple sums over the operators they cover:

ScoreLevel\-​N=∑op∈Level\-​NEachOperatorScoreop,Scorebenchmark=∑N=14ScoreLevel\-​N\.\\mathrm\{Score\}\_\{\\text\{Level\-\}N\}=\\sum\_\{\\text\{op\}\\in\\text\{Level\-\}N\}\\mathrm\{EachOperatorScore\}\_\{\\text\{op\}\},\\qquad\\mathrm\{Score\}\_\{\\text\{benchmark\}\}=\\sum\_\{N=1\}^\{4\}\\mathrm\{Score\}\_\{\\text\{Level\-\}N\}\.\(7\)
This formulation keeps the three axes co\-active\. A submission that fails to compile contributes nothing on any axis, since both the compilation term and every case\-level term collapse to zero\. A submission that compiles but fails the functional gate on a given case forfeits both the0\.30\.3floor and the0\.5⋅scorei0\.5\\cdot\\mathrm\{score\}\_\{i\}term on that case while preserving the0\.20\.2compilation contribution and the contributions from any cases that do pass\. A functionally correct but slow submission keeps the0\.30\.3floor on every passing case and only loses \(part of\) the0\.5⋅scorei0\.5\\cdot\\mathrm\{score\}\_\{i\}component\. The weights\(wc,wf,wp\)=\(0\.2,0\.3,0\.5\)\(w\_\{c\},w\_\{f\},w\_\{p\}\)=\(0\.2,0\.3,0\.5\)are normalised to sum to11, so a fully compiled, functionally correct, hardware\-limit\-matching operator scores exactly100100\. The per\-case ratiowp/wf=5/3w\_\{p\}/w\_\{f\}=5/3keeps two extreme strategies on comparable footing: a fully correct submission at the baseline anchor \(scorei=0\.5\\mathrm\{score\}\_\{i\}=0\.5on every case\) scores7575on the operator, while a partial\-but\-fully\-optimised submission only overtakes that by clearing more than11/1611/16of its cases \(roughly1414of the current2020\)\.

### 3\.4Reward Hacking and Mitigation

The composite score in Eq\. \([6](https://arxiv.org/html/2607.20518#S3.E6)\) is only meaningful if the measured outputs and runtimes come from the submitted NPU kernel\. Kernel\-generation benchmarks are vulnerable to evaluator loopholes: an optimizer can increase its score by routing computation to existing library kernels, hiding work outside the profiled window, caching outputs, or tampering with timing APIs rather than improving the generated kernel itself\[[15](https://arxiv.org/html/2607.20518#bib.bib27),[17](https://arxiv.org/html/2607.20518#bib.bib4)\]\. CANN Bench therefore treats anti\-reward\-hacking as part of the benchmark specification, not merely as post\-hoc leaderboard review\.

The Ascend evaluation stack exposes timing, synchronization, profiling, and custom\-kernel entry points through thetorch\.npu/torch\_npuruntime and compiled Ascend C extensions\[[10](https://arxiv.org/html/2607.20518#bib.bib29),[19](https://arxiv.org/html/2607.20518#bib.bib30),[2](https://arxiv.org/html/2607.20518#bib.bib31)\]\. CANN Bench defends this surface through four principles: the submitted kernel must be the implementation being measured; candidate execution must stay on the target device; outputs must be materialized and validated case by case; and the measurement path must remain under evaluator control\. Table[5](https://arxiv.org/html/2607.20518#S3.T5)summarizes the deployed defense principles without exposing evaluator\-internal trigger details\.

Table 5:Reward\-hacking surfaces covered by deployed CANN Bench defenses\. The table describes mitigation principles rather than evaluator trigger details\.The same\-op routing surface deserves separate treatment because it crosses the boundary between the submitted package and the evaluator’s CANN installation\. On dedicated evaluator machines, CANN Bench can harden the environment so that leaderboard runs do not silently dispatch target semantics to preinstalled same\-op binaries\. This is an evaluator\-machine policy, not a restriction on legitimate Ascend C intrinsics or other low\-level building blocks compiled into the candidate kernel\.

The remaining limitations are handled conservatively\. Multi\-stream submissions are restricted rather than fully ranked until stream visibility is stable across the supported runtime stack, and opaque precompiled binary loading remains subject to policy and manual review\. Input refresh and address\-diversity controls are useful hardening knobs when enabled by the evaluation profile, but the core security claim is simpler: performance credit is awarded only when the evaluator can validate the output and attribute the measured runtime to candidate NPU\-kernel execution\.

## 4Conclusion

We introducedCANN Bench, an Ascend\-oriented operator\-generation benchmark designed to give AI\-generated kernels under Huawei’s CANN stack a shared, vendor\-aligned evaluation scale\. The initial release ships 53 operators and 1060 test cases distributed across four difficulty levels \(L1–L4\), each operator accompanied by a self\-contained reproducibility kit—desc\.md,proto\.yaml,cases\.csv, andgolden\.py—and scored on the three axes that matter for kernel work on Ascend 910B2: compilation, functional correctness, and performance\. Performance is graded between a PyTorch\-on\-Ascend baseline and an analytically derived per\-case HAP limit, so absolute scores carry a hardware\-grounded meaning rather than only a relative ranking\. The harness, baseline\-collection pipeline, and operator suite are co\-located in the official CANN repository and released under the CANN Open Software License v2\.0, with an online leaderboard prepared as the durable maintenance surface\.

Relative to existing kernel\-generation benchmarks, CANN Bench occupies a position that none of the closest prior efforts simultaneously cover\. Cross\-platform suites such as KernelBench, TritonBench, and SOL\-ExecBench target the NVIDIA GPU architecture and do not transfer mechanically to Ascend’s tiling semantics and multi\-level buffer structure; MultiKernelBench spans three hardware backends but treats Ascend as one slice with limited per\-operator depth; and NPUKernelBench is co\-evolved with a specific training recipe and includes a static\-shape track where a single canonical input configuration governs an entire task\. CANN Bench differs in three concrete ways: golden references are vendor\-authoritative and maintained alongside official CANN operator contracts; every operator carries a 20\-case public split for local iteration plus an 80\-case hidden split for leaderboard evaluation, which prevents fitting to disclosed inputs; and the harness combines specification\-level invalid\-behavior rules, runtime enforcement, measurement\-integrity checks, and NPU\-execution attribution so that performance credit remains tied to the submitted kernel\.

We are explicit about the current limits of this release\. Precision coverage is restricted to FP16, BF16, FP32, and INT8; FP8, HiF8, MXFP8, FP4, and MXFP4 are deliberate near\-term extensions rather than shipped capabilities\. Several reward\-hacking surfaces remain intentionally conservative: multi\-stream execution is restricted rather than fully ranked, because stream\-level visibility acrosstorch\_npuversions is still fragile; evaluator\-machine hardening is a controlled leaderboard policy rather than an automatic property of every local install; and opaque pre\-compiled binary loading is presently addressed by policy and manual review rather than automated provenance checks\. Input refresh and address\-diversity controls should be treated as run\-profile\-dependent hardening knobs unless enabled by the official leaderboard configuration\. The benchmark also begins on a single hardware target \(Ascend 910B2\) and a single kernel programming surface, so coverage of newer Ascend silicon and additional DSLs is a near\-term task rather than an established result\.

Looking forward, CANN Bench is intended to evolve with the Ascend ecosystem rather than freeze at release\. Three extensions are scheduled: the online leaderboard portal will index and date every submission so that results stay directly citable; kernel\-DSL coverage will extend beyond the initial Ascend\-optimized path to Triton, PyPTO, TileLang, and other CANN\-compatible paradigms under the same specification and scoring protocol; and precision coverage will expand to FP8, MXFP8, FP4, MXFP4, HiF8, and related low\-precision formats as operators and baselines are validated\. Versioned performance baselines, refreshable through a baseline\-update utility planned for an upcoming release, ensure that reported speedups continue to track the current attainable reference rather than a stale snapshot as CANN and Ascend hardware progress\. The operator set, golden implementations, harness, and documentation are open\-source, and the project actively accepts contributions of new operators, test cases, baseline updates, and sub\-leaderboards\. By coupling vendor\-authoritative golden references, the HAP anchor, a hidden\-split evaluation protocol, and a deployed anti\-reward\-hacking harness inside the official CANN repository, CANN Bench aims to serve as the shared, sustained evaluation scale that AI\-generated kernels on Ascend have so far lacked, and to grow alongside the workloads, precisions, and silicon it is built to measure\.

## Appendix ARegion\-specific precision thresholds

This appendix collects the per\-dtype thresholds used by the small\-value and cancellation gates referenced in §[3\.1](https://arxiv.org/html/2607.20518#S3.SS1)\(Eq\.[3](https://arxiv.org/html/2607.20518#S3.E3)\)\. The values are taken from the ecosystem\-operator precision standard and from the evaluator implementation; they are organized by region so that the main text can refer to a single ratio test rather than reproduce three threshold tables inline\.

#### Small\-value region\.

A position is classified as small\-value when\|ytrunc\|<τdsv\|y\_\{\\mathrm\{trunc\}\}\|<\\tau^\{\\mathrm\{sv\}\}\_\{d\}, whereytruncy\_\{\\mathrm\{trunc\}\}is the FP64 Golden cast down to the device dtype and back to FP64 \(i\.e\. the best value representable at the device precision\)\. Within the region, a position is counted as an error when\|y^−ytrunc\|\|\\hat\{y\}\-y\_\{\\mathrm\{trunc\}\}\|exceeds the per\-dtype absolute\-error toleranceεdsv\\varepsilon^\{\\mathrm\{sv\}\}\_\{d\}\. The NPU and CPU runs are both scored against the sameytruncy\_\{\\mathrm\{trunc\}\}, so that the ratio in Eq\. \([3](https://arxiv.org/html/2607.20518#S3.E3)\) reflects only the kernel’s contribution to the small\-value error count\.

Table 6:Small\-value region thresholds\. A position is in the small\-value region when\|ytrunc\|<τdsv\|y\_\{\\mathrm\{trunc\}\}\|<\\tau^\{\\mathrm\{sv\}\}\_\{d\}; within the region it contributes toErrorCount\\mathrm\{ErrorCount\}when\|y^−ytrunc\|\>εdsv\|\\hat\{y\}\-y\_\{\\mathrm\{trunc\}\}\|\>\\varepsilon^\{\\mathrm\{sv\}\}\_\{d\}\.
#### Cancellation region\.

A position is classified as cancellation\-affected when the kernel output is approximately zero,\|y^\|<τdcz\|\\hat\{y\}\|<\\tau^\{\\mathrm\{cz\}\}\_\{d\}, while the Golden lies above the dtype’s representable noise floor but inside the cancellation boundary,τdsv≤\|y\|<τdcb\\tau^\{\\mathrm\{sv\}\}\_\{d\}\\leq\|y\|<\\tau^\{\\mathrm\{cb\}\}\_\{d\}\. The lower bound excludes positions already covered by the small\-value rule\. The cancellation count in Eq\. \([3](https://arxiv.org/html/2607.20518#S3.E3)\) is the number of positions within the region whose relative error exceeds the max\-RE bound10​τd10\\,\\tau\_\{d\}from Table[4](https://arxiv.org/html/2607.20518#S3.T4); the NPU and CPU runs are evaluated against the same Golden so that the ratio isolates kernel\-induced cancellation from dtype\-floor cancellation\.

Table 7:Cancellation region thresholds\. A position is cancellation\-affected when\|y^\|<τdcz\|\\hat\{y\}\|<\\tau^\{\\mathrm\{cz\}\}\_\{d\}andτdsv≤\|y\|<τdcb\\tau^\{\\mathrm\{sv\}\}\_\{d\}\\leq\|y\|<\\tau^\{\\mathrm\{cb\}\}\_\{d\}; the per\-dtype values are anchored to the format’s mantissa width so that the region captures losses of roughly one decade of significance around the dtype’s precision floor\.

## References

- \[1\]\(2025\)GPU Kernel Scientist: an LLM\-driven framework for iterative kernel optimization\.arXiv preprint arXiv:2506\.20807\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[2\]AscendAscend extension for pytorch\.Note:GitHub repositoryAccessed 2026\-04\-23External Links:[Link](https://github.com/Ascend/pytorch)Cited by:[§3\.4](https://arxiv.org/html/2607.20518#S3.SS4.p2.1)\.
- \[3\]C\. Baronio, P\. Marsella, B\. Pan, S\. Guo, and S\. Alberti\(2025\)Kevin: multi\-turn RL for generating CUDA kernels\.arXiv preprint arXiv:2507\.11948\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[4\]X\. Cao, J\. Zhai, P\. Li, Z\. Hu, C\. Yan, B\. Mu,et al\.\(2026\)AscendKernelGen: a systematic study of LLM\-based kernel generation for neural processing units\.External Links:2601\.07160,[Link](https://arxiv.org/abs/2601.07160)Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1),[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p4.1),[Table 1](https://arxiv.org/html/2607.20518#S1.T1.1.1.2.1.1)\.
- \[5\]X\. Cheng, W\. Zeng, D\. Dai, Q\. Chen, B\. Wang, Z\. Xie,et al\.\(2026\)Conditional memory via scalable lookup: a new axis of sparsity for large language models\.External Links:2601\.07372,[Link](https://arxiv.org/abs/2601.07372)Cited by:[item 1](https://arxiv.org/html/2607.20518#S2.I1.i1.p1.1)\.
- \[6\]W\. Dai, H\. Wu, Q\. Yu, H\. Gao, J\. Li, C\. Jiang,et al\.\(2026\)CUDA Agent: large\-scale agentic RL for high\-performance CUDA kernel generation\.arXiv preprint arXiv:2602\.24286\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[7\]DeepReinforce Team\(2025\)Hacks and defenses in automatic GPU kernel generation\.Note:[project page](https://deep-reinforce.com/defense_kernel_hack.html)Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p6.1)\.
- \[8\]P\. Guo, C\. Zhu, S\. Chen, F\. Liu, X\. Lin, Z\. Lu,et al\.\(2025\)EvoEngineer: mastering automated CUDA kernel code evolution with large language models\.arXiv preprint arXiv:2509\.18570\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[9\]Huawei Ascend\(2023\)msprof profiling tool: Ascend CANN auxiliary development toolkit documentation\.Note:[Huawei Ascend documentation page](https://www.hiascend.com/document/detail/zh/canncommercial/601/devtools/auxiliarydevtool/atlasprofiling_16_0041.html)CANN Commercial 6\.0\.1Cited by:[§1\.5](https://arxiv.org/html/2607.20518#S1.SS5.p1.1)\.
- \[10\]HuaweiNPU and cuda function alignment\.Note:Huawei Technical Support DocumentationAccessed 2026\-04\-23External Links:[Link](https://support.huawei.com/enterprise/en/doc/EDOC1100191777/dc6e533e/npu-and-cuda-function-alignment)Cited by:[§3\.4](https://arxiv.org/html/2607.20518#S3.SS4.p2.1)\.
- \[11\]Huawei\(2026\)CANN open software license agreement version 2\.0\.Note:[CANN repository LICENSE](https://gitcode.com/cann/cann-bench/blob/master/LICENSE)Cited by:[§1\.6](https://arxiv.org/html/2607.20518#S1.SS6.SSS0.Px1.p1.1)\.
- \[12\]J\. Jaber and O\. Jaber\(2026\)AutoKernel: autonomous GPU kernel optimization via iterative agent\-driven search\.arXiv preprint arXiv:2603\.21331\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[13\]S\. Kalade and G\. Schelle\(2025\)NPUEval: optimizing NPU kernels with LLMs and open source compilers\.External Links:2507\.14403,[Link](https://arxiv.org/abs/2507.14403)Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p3.1)\.
- \[14\]KernelCAT Team\(2025\)KernelCAT: an expert\-level agent for compute acceleration on Ascend NPU\.Note:[Zhihu article](https://zhuanlan.zhihu.com/p/2000688517639070651)Industrial technical blog postCited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1),[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p5.1)\.
- \[15\]R\. T\. Lange, Q\. Sun, A\. Prasad, M\. Faldor, Y\. Tang, and D\. Ha\(2025\)Towards robust agentic cuda kernel benchmarking, verification, and optimization\.External Links:2509\.14279,[Document](https://dx.doi.org/10.48550/arXiv.2509.14279),[Link](https://arxiv.org/abs/2509.14279)Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1),[§3\.4](https://arxiv.org/html/2607.20518#S3.SS4.p1.1)\.
- \[16\]J\. Li, S\. Wang, Z\. Zhang,et al\.\(2025\)TritonBench: benchmarking large language model capabilities for generating Triton operators\.InFindings of ACL,Note:arXiv:2502\.14752Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p2.1)\.
- \[17\]E\. Lin, S\. Modi, S\. K\. S\. Hari, Q\. Huang,et al\.\(2026\)SOL\-ExecBench: speed\-of\-light benchmarking for real\-world GPU kernels against hardware limits\.arXiv preprint arXiv:2603\.19173\.Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p2.1),[§2\.4](https://arxiv.org/html/2607.20518#S2.SS4.p1.1),[§3\.2](https://arxiv.org/html/2607.20518#S3.SS2.p2.2),[§3\.4](https://arxiv.org/html/2607.20518#S3.SS4.p1.1)\.
- \[18\]W\. Liu, J\. Xu, Y\. Li, L\. Zheng, T\. Li, Q\. Liu,et al\.\(2026\)Dr\. Kernel: reinforcement learning done right for Triton kernel generations\.arXiv preprint arXiv:2602\.05885\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[19\]ModelScope SWIFT ContributorsAscend npu performance data collection\.Note:Megatron\-SWIFT DocumentationAccessed 2026\-04\-23External Links:[Link](https://swift.readthedocs.io/en/v3.12/Megatron-SWIFT/Ascend.html)Cited by:[§3\.4](https://arxiv.org/html/2607.20518#S3.SS4.p2.1)\.
- \[20\]J\. Nie, H\. Wu, Y\. Lai, Z\. Cao, C\. Zhang, B\. Lou,et al\.\(2026\)KernelCraft: benchmarking for agentic close\-to\-metal kernel generation on emerging hardware\.External Links:2603\.08721,[Link](https://arxiv.org/abs/2603.08721)Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p3.1)\.
- \[21\]A\. Ouyang, S\. Guo, S\. Arora, A\. L\. Zhang, W\. Hu, C\. Ré,et al\.\(2025\)KernelBench: can LLMs write efficient GPU kernels?\.arXiv preprint arXiv:2502\.10517\.Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p2.1)\.
- \[22\]B\. D\. Rouhani, R\. Zhao, A\. More, M\. Hall, A\. Khodamoradi, S\. Deng,et al\.\(2023\)Microscaling data formats for deep learning\.arXiv preprint arXiv:2310\.10537\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p2.1),[§1\.2](https://arxiv.org/html/2607.20518#S1.SS2.p4.1)\.
- \[23\]T\. Saba, A\. Ouyang, X\. Si, and F\. Long\(2026\)CuTeGen: an LLM\-based agentic framework for generation and optimization of high\-performance GPU kernels using CuTe\.arXiv preprint arXiv:2604\.01489\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[24\]Scaling Intelligence Lab\(2025\)Fall 2025 KernelBench maintenance and improvement plan\.Note:GitHub Issue \#74,[issue page](https://github.com/ScalingIntelligence/KernelBench/issues/74)Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p6.1)\.
- \[25\]Scaling Intelligence Lab\(2025\)KernelBench v0\.1\.Note:Stanford Scaling Intelligence Lab blog post,[blog page](https://scalingintelligence.stanford.edu/blogs/kernelbenchv01/)Cited by:[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p6.1)\.
- \[26\]A\. Tseng, T\. Yu, and Y\. Park\(2025\)Training LLMs with MXFP4\.arXiv preprint arXiv:2502\.20586\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p2.1),[§1\.2](https://arxiv.org/html/2607.20518#S1.SS2.p4.1)\.
- \[27\]J\. Wang, V\. Joshi, S\. Majumder, X\. Chao, B\. Ding, Z\. Liu,et al\.\(2025\)GEAK: introducing triton kernel AI agent & evaluation benchmarks\.arXiv preprint arXiv:2507\.23194\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1)\.
- \[28\]Z\. Wen, Y\. Zhang, Z\. Li, Z\. Liu, L\. Xie, and T\. Zhang\(2025\)MultiKernelBench: a multi\-platform benchmark for kernel generation\.arXiv preprint arXiv:2507\.17773\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1),[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p4.1),[Table 1](https://arxiv.org/html/2607.20518#S1.T1.1.3.1.1.1.1)\.
- \[29\]Z\. Wen, S\. Shao, Z\. Li, Y\. Ge, T\. Xu, Y\. Lin,et al\.\(2026\)AscendCraft: automatic Ascend NPU kernel generation via DSL\-guided transcompilation\.arXiv preprint arXiv:2601\.22760\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1),[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p5.1)\.
- \[30\]J\. Wu, Z\. Huang, W\. Li, C\. Shen, J\. Sheng, and X\. Wang\(2026\)AscendOptimizer: episodic agent for Ascend NPU operator optimization\.arXiv preprint arXiv:2603\.23566\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1),[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p5.1),[Table 1](https://arxiv.org/html/2607.20518#S1.T1.1.4.2.1.1.1)\.
- \[31\]Y\. Zheng, Z\. Li, S\. Zhang, H\. Wang, J\. Sheng, J\. Wang,et al\.\(2026\)Towards cold\-start drafting and continual refining: a value\-driven memory approach with application to NPU kernel synthesis\.arXiv preprint arXiv:2603\.10846\.Cited by:[§1\.1](https://arxiv.org/html/2607.20518#S1.SS1.p1.1),[§1\.3](https://arxiv.org/html/2607.20518#S1.SS3.p5.1)\.