KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
Summary
KernelBench-X is a new benchmark for evaluating LLM-generated GPU kernels, revealing that task structure impacts correctness more than method design and that correctness does not guarantee hardware efficiency.
View Cached Full Text
Cached at: 05/08/26, 10:53 AM
Paper page - KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
Source: https://huggingface.co/papers/2605.04956
Abstract
KernelBench-X benchmark reveals that task structure significantly impacts LLM-generated Triton kernel correctness more than method design, while iterative refinement improves correctness at the expense of performance, and correctness does not guarantee efficiency.
LLM-basedTriton kernel generationhas attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capability break down, and why? We presentKernelBench-X, a benchmark designed to answer this question through category-aware evaluation ofcorrectnessandhardware efficiencyacross 176 tasks in 15 categories. Our systematic comparison of five representative methods yields three main findings. First, task structure determinescorrectnessmore than method design. Category explains nearly three times more variance in semanticcorrectnessthan method (9.4% vs 3.3% explained deviance), and 72% of Fusion tasks fail across all five methods while Math tasks are solved consistently. Second,iterative refinementimprovescorrectness, but not performance. Across GEAK iterations,compile raterises from 52.3% to 68.8% while averagespeedupdeclines from 1.58times to 1.44times; newly rescued kernels consistently underperform persistently correct ones (1.16times vs 1.58timesspeedupin round~0to1). Third,correctnessdoes not imply efficiency. 46.6% of correct kernels are slower than the PyTorch eager baseline, and cross-hardwarespeedupvariance reaches 21.4times. Besides,quantizationremains completely unsolved (0/30 successes) despite non-trivial compilation rates, revealing systematic misunderstanding of numerical computation contracts rather than surface-level syntax errors. These findings suggest that future progress depends on handling global coordination, explicitly modelingnumerical precision, and incorporatinghardware efficiencyinto generation. The code is available at https://github.com/BonnieW05/KernelBenchX
View arXiv pageView PDFProject pageGitHub14Add to collection
Get this paper in your agent:
hf papers read 2605\.04956
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.04956 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.04956 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.04956 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX
PTXBench is introduced as a benchmark to evaluate and adapt large language models for optimizing GPU kernels using architecture-specific PTX, showing uneven performance and fine-tuning insights.
DataKernelBench: Can LLMs Optimize Database Queries on GPUs?
This paper introduces DataKernelBench, a benchmark for evaluating LLMs on optimizing GPU kernels for database queries, achieving speedups over baseline methods like torch.compile.
KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?
Paper introduces KernelBench-Verified, an extended evaluation framework for LLM-generated CUDA kernels that incorporates TF32-enabled baselines and hidden test suites. It finds that frontier models like GPT-5.5 often engage in reward hacking and do not consistently outperform PyTorch under realistic conditions, with the best model achieving only 0.88x geometric mean speedup.
ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation
ParBench is a benchmark framework for evaluating LLM-based parallel API translation across CUDA, OpenMP, OpenCL, and OpenMP target offload, focusing on executable, reproducible conditions and robustness testing.
Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles
This paper introduces mutation analysis as a metric to evaluate benchmark oracles for GPU kernels, revealing limitations in current testing methods and proposing optimizations for better fault detection in LLM-generated code.