LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization

# LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization
Source: [https://arxiv.org/html/2608.21836](https://arxiv.org/html/2608.21836)
Pengfei YangAffiliation:National Key Laboratory of Advanced Communication Networks, Shijiazhuang, Hebei, P\. R\. ChinaAffiliation:Corresponding authors:[pfyang@xidian\.edu\.cn](mailto:[email protected]),[weixinran@zgci\.ac\.cn](mailto:[email protected])Yanxin ChenFusong JuXinran WeiAffiliation:Xidian University, Zhongguancun Academy, Zhongguancun Institute of Artificial IntelligenceAffiliation:Corresponding authors:[pfyang@xidian\.edu\.cn](mailto:[email protected]),[weixinran@zgci\.ac\.cn](mailto:[email protected])

###### Abstract

Large language models have become increasingly capable agents for low\-level code and kernel optimization, but isolated kernel benchmarks provide only a proxy for the deployment behavior that matters in language\-model inference\. We identify a benchmark\-to\-deployment gap: candidate kernels that appear correct and fast in standalone harnesses can exhibit different performance, safety, or phase behavior after integration into a real inference workload\. We introduce LLM4LLM, a deployment\-aware closed\-loop optimization framework that starts from a target inference script, extracts phase\-aware optimization tasks, searches with an experience\-guided episodic agent, and accepts patches through in\-model validation\. Across ten language\-model inference workloads on A100 and H100 GPUs, LLM4LLM improves end\-to\-end latency for every evaluated model, achieving 3\.91×\\times/6\.98×\\timesgeometric\-mean speedups on A100/H100; as supporting kernel\-level evidence, it also attains up to 2\.745×\\timesGeoMean speedup on KernelBench Level 2\.

## 1Introduction

Recent progress in LLM\-driven code generation has made automated kernel and program optimization a credible direction for performance engineering\. Code\-specialized models support synthesis, infilling, and editing\([6](https://arxiv.org/html/2608.21836#bib.bib1);[39](https://arxiv.org/html/2608.21836#bib.bib35)\)\. Competitive\-programming and execution\-feedback systems extend this ability to search\-guided generation\([28](https://arxiv.org/html/2608.21836#bib.bib37);[26](https://arxiv.org/html/2608.21836#bib.bib38)\)\. Iterative agents improve programs through feedback and debugging traces\([30](https://arxiv.org/html/2608.21836#bib.bib4);[9](https://arxiv.org/html/2608.21836#bib.bib39)\)\. Recent kernel benchmarks further show that LLMs can generate low\-level GPU programs\([31](https://arxiv.org/html/2608.21836#bib.bib3)\)\. Most existing evaluations, however, are centered on isolated kernels or extracted program harnesses: a candidate is compiled, executed on synthetic inputs, checked against a reference implementation, and ranked by standalone latency\. For language\-model inference, this measures an intermediate signal\. The deployment objective is the behavior of the patched model under the target workload, including phase semantics, cache state, dispatch overhead, memory residency, and end\-to\-end latency\.

We study the mismatch between benchmark\-level optimization and model\-level deployment\. Autoregressive inference exposes this mismatch clearly\. Prefill and decode have different shapes, cache states, memory\-access patterns, and latency sensitivity; a candidate optimized for one phase can transfer unevenly to the other\. Integration can also change the execution environment through shape guards, dispatch logic, allocator state, and interactions with existing high\-performance kernels\. We refer to this prediction mismatch as the*benchmark\-to\-deployment gap*\. The gap appears as speedup reversal, deployment\-time runtime failure, and phase\-specific behavior that is absent from isolated qualification\.

LLM4LLM111Code is available at[https://github\.com/hzeng2000/LLM4LLM](https://github.com/hzeng2000/LLM4LLM)\.addresses this gap by making deployment context part of the optimization loop\. Starting from a user\-provided inference script, the system profiles real workloads and extracts optimization tasks at deployable module boundaries\. It constructs phase\-aware tasks for prefill and decode, searches for candidates through experience\-guided episodic optimization, and accepts patches only after in\-model validation\. The episodic agent compresses useful experience from earlier attempts into concise constraints, allowing search to reuse successful patterns while discarding stale context\. The resulting loop ties profiling, generation, verification, acceptance, and deployment into a single optimization process\.

Our contributions are:

- •We characterize the benchmark\-to\-deployment gap for LLM\-based kernel optimization through a taxonomy of failure modes, showing how isolated qualification can diverge from deployment\-time acceptance\.
- •We introduce LLM4LLM, a closed\-loop agentic optimization framework that combines real\-model profiling, phase\-aware task formulation, experience\-guided episodic search, and in\-model acceptance\.
- •We evaluate LLM4LLM on real inference workloads and KernelBench Level 2, showing end\-to\-end gains across transformer, state\-space, and recurrent language\-model families with 3\.91×\\times/6\.98×\\timesgeometric\-mean speedups on A100/H100, and GeoMean kernel speedups of 2\.745×\\times/2\.628×\\timeson KernelBench Level 2\.

Figure 1:Divergent isolated and deployment outcomes for an optimizedResNetConvLayer\. The left panel contrasts standalone cold\-cache benchmarking with in\-model warm\-context execution for the same candidate kernel\. The right panel reports speedup from standalone kernel testing, the same kernel measured inside the model context, and the resulting full\-model end\-to\-end execution\.
## 2The Benchmark\-to\-Deployment Gap

### 2\.1Motivating Evidence

We begin with a representative case that isolates the effect of execution context, shown in Figure[1](https://arxiv.org/html/2608.21836#S1.F1)\. A convolution layer extracted from a standard vision backbone is optimized by an LLM\-based agent and evaluated in two settings\. In the isolated setting, the candidate is compiled as a standalone module, invoked on freshly allocated random inputs, and timed independently\. In the deployment setting, the same candidate is patched into the full model and evaluated in the target inference workload\. The isolated benchmark reports a substantial speedup, while the integrated model becomes slower end to end\.

The mechanism is memory residency\. In the standalone benchmark, the operator pays the cost of loading cold inputs from memory\. Inside the model, the same operator consumes activations produced immediately by the preceding layer, so the baseline benefits from warm\-cache execution and producer\-consumer locality\. A memory\-access pattern that improves cold standalone timing can lose its advantage once the surrounding model supplies different cache state and scheduling context\. This case shows how proxy measurements can misrank candidates before deployment\.

The same gap also appears in forms that are specific to deployability and autoregressive inference\. First, a candidate may pass isolated correctness while carrying a latent memory\-safety error\. Standalone tensors are often allocated with adjacent mapped memory, so an out\-of\-bounds read can return a value without faulting\. The allocator state of a full model can place the same access in a faulting region, producing a deployment\-time CUDA error\. Second, phase specialization can make a kernel\-level gain local to only one part of generation\. For example, a candidate that optimizes attention during prefill may be applicable whenq\_len \> 1, while decode attention runs withq\_len = 1and a populated KV cache\. The prefill fast path can therefore fall back, or provide no benefit, in the decode phase\.

Figure 2:Deployment\-specific manifestations of the benchmark\-to\-deployment gap\. The left panel shows how a latent out\-of\-bounds access can pass isolated qualification under a fresh allocator state yet fail after integration into the full model runtime\. The right panel shows that an attention candidate specialized for prefill can be inapplicable to decode attention, so a kernel\-level gain does not necessarily transfer across generation phases\.These cases establish three deployment concerns: performance transfer, runtime validity, and phase\-correct behavior\. They also motivate a distinction between isolated qualification and deployment\-time acceptance, as illustrated in Figure[2](https://arxiv.org/html/2608.21836#S2.F2)\.

### 2\.2Formulation and Implications

We distinguish four evaluation stages\.*Isolated qualification*checks numerical correctness and latency in a standalone harness\.*Search\-time validation*strengthens the proxy by adding real inputs, runtime\-safety gates, or in\-context timing\.*Deployment\-time acceptance*evaluates a candidate after it has been inserted into the target model instance\.*Final end\-to\-end evaluation*measures the patched model under the full workload\.

The benchmark\-to\-deployment gap is the mismatch between isolated qualification and deployment\-time acceptance\. Execution context, phase behavior, and integration constraints all contribute to this mismatch\. Isolated benchmarks remain valuable because they allow rapid candidate generation and screening, while deployment\-time acceptance determines whether a candidate improves the target model\. This leads to a methodological requirement: candidate generation, validation, and patch acceptance form a closed loop around the real workload\.

## 3Method

Figure[3](https://arxiv.org/html/2608.21836#S3.F3)summarizes the LLM4LLM optimization loop\. Starting from a target inference script, the system profiles the real workload, extracts phase\-aware optimization tasks, searches with an experience\-guided agent, and accepts patches through model\-integrated validation\. We denote the target model instance byMM, the deployment workload induced by the inference script by𝒲\\mathcal\{W\}, and the set of autoregressive phases by𝒫\\mathcal\{P\}\. HereMMis a concrete model instance, while𝒲\\mathcal\{W\}and𝒫\\mathcal\{P\}are structured collections\. For a module instancemmand phasep∈𝒫p\\in\\mathcal\{P\}, profiling records the module timet⁡\(m,p\)t\(m,p\)and the total phase latencyT⁡\(p\)T\(p\)\. LLM4LLM selects deployable optimization units using the workload\-weighted hotspot score

s⁡\(m\)=∑p∈𝒫ωp​t⁡\(m,p\)T⁡\(p\),s\(m\)=\\sum\_\{p\\in\\mathcal\{P\}\}\\omega\_\{p\}\\frac\{t\(m,p\)\}\{T\(p\)\},\(1\)whereωp\\omega\_\{p\}is determined by the measured phase frequency or by the evaluation workload\. The score is used for task extraction, and final acceptance is made inside the target model\.

We next detail the three stages of this loop: phase\-aware task extraction, experience\-guided search, and deployment\-time acceptance\.

### 3\.1Hotspot Discovery and Phase\-Aware Extraction

LLM4LLM starts from the inference script supplied by the user\. The script defines the workload, input regime, runtime path, and performance objective\. The system profiles this execution hierarchically and selects semantic module instances whose replacement can affect end\-to\-end latency\. A module is considered deployable when it has a stable call boundary, reproducible input and output tensors, and a fallback implementation for unoptimized regimes\.

For each selected module, LLM4LLM serializes an optimization task containing representative inputs, output references, shape information, module\-family metadata, hardware scope, and phase tags\. For autoregressive inference, prefill and decode are represented as separate tasks when their execution semantics diverge\. When both phases are optimized, extraction also records a dispatch template that later reassembles phase\-specialized candidates into a single patched module\. The task preserves the deployment facts that affect validity, including tensor layout, cache state, phase predicate, and observed shape regimes\.

Figure 3:Overview of LLM4LLM\. The framework starts from a target inference script, identifies deployable hotspots through real\-model profiling, constructs phase\-aware tasks, searches for candidate kernels with experience\-guided agentic optimization, and accepts patches only after in\-model correctness and latency validation\. Validation feedback is returned to the search loop, while accepted patches produce the optimized deployment\.
### 3\.2Experience\-Guided Episodic Optimization

The optimizer searches through repeated generate\-verify\-decide episodes\. Within an episode, the agent proposes candidate kernels, compiles them, runs correctness checks, measures latency, and inspects failures\. Each episode therefore produces a concrete validation trace: candidate code, compiler diagnostics, numerical errors, runtime failures, and latency measurements\. LLM4LLM uses this trace to decide whether the current search context is still productive or has converged to local repair of a narrow failure mode\. For a candidateccon taskτ\\tau, search\-time validation assigns the utility

u⁡\(c,τ\)=tref​\(τ\)tc​\(τ\)​𝟏​\{err⁡\(c,τ\)≤ϵ,safe⁡\(c,τ\)\},u\(c,\\tau\)=\\frac\{t\_\{\\mathrm\{ref\}\}\(\\tau\)\}\{t\_\{c\}\(\\tau\)\}\\mathbf\{1\}\\\{\\mathrm\{err\}\(c,\\tau\)\\leq\\epsilon,\\ \\mathrm\{safe\}\(c,\\tau\)\\\},\(2\)wheretreft\_\{\\mathrm\{ref\}\}is the reference task latency,tct\_\{c\}is the candidate latency,𝟏​\{⋅\}\\mathbf\{1\}\\\{\\cdot\\\}is the indicator function,err\\mathrm\{err\}is the numerical error against recorded outputs, andsafe\\mathrm\{safe\}denotes compilation and runtime\-safety checks\. This utility ranks promising candidates within the extracted task; deployment\-time acceptance remains the final decision\.

At episode boundaries, the system distills the validation trace into compact experience\. The distilled record contains durable constraints and search evidence, including valid tiling choices, boundary\-mask requirements, phase predicates, numerical constraints, failure signatures, and the best observed performance regime\. The next episode is initialized with the original task specification and this experience record; the full turn\-by\-turn transcript is archived\. The summary for episoderris a bounded recordErE\_\{r\}that stores accepted constraints, rejected failure modes, and the best candidate family observed so far\. Restarting from the task plusErE\_\{r\}preserves useful evidence while reducing the influence of long local repair trajectories\.

Figure 4:Experience\-guided episodic optimization\. Candidate generation and validation form a bounded episode\. At the episode boundary, LLM4LLM summarizes the useful validation evidence into compact experience, archives the transient repair history, and restarts search from the task specification plus the distilled record\.Figure[4](https://arxiv.org/html/2608.21836#S3.F4)illustrates this mechanism on a representative optimization episode\. This design treats validation feedback as reusable optimization evidence\. Episode\-level memory captures task\-specific lessons, while family\-scoped memory captures patterns that transfer across related modules, shape buckets, and hardware targets\. The search process remains grounded in observed verification constraints while allowing later episodes to explore implementation structures beyond earlier local edits\.

### 3\.3Deployment\-Time Acceptance and Patching

Search produces qualified candidates; deployment\-time acceptance decides which candidates enter the model\. LLM4LLM ranks candidates by module compatibility, shape coverage, phase compatibility, and predicted impact\. Each candidate is inserted into the target model instance and evaluated for in\-context correctness and latency\. Accepted candidates become patches; rejected candidates provide feedback for subsequent search\. Letℓ⁡\(c\)=L⁡\(M⁡\[c\],𝒲\)/L⁡\(M,𝒲\)\\ell\(c\)=L\(M\[c\],\\mathcal\{W\}\)/L\(M,\\mathcal\{W\}\)be normalized deployment latency, and let𝒞dep=\{c∈𝒞:okdep​\(c\)\}\\mathcal\{C\}\_\{\\mathrm\{dep\}\}=\\\{c\\in\\mathcal\{C\}:\\mathrm\{ok\}\_\{\\mathrm\{dep\}\}\(c\)\\\}be the candidates passing in\-model correctness, runtime compatibility, and shape\-guard coverage\. Deployment\-time acceptance selects

c⋆=arg⁡minc∈𝒞dep⁡ℓ⁡\(c\),ℓ⁡\(c⋆\)≤1−δ\.c^\{\\star\}=\\arg\\min\_\{c\\in\\mathcal\{C\}\_\{\\mathrm\{dep\}\}\}\\ell\(c\),\\qquad\\ell\(c^\{\\star\}\)\\leq 1\-\\delta\.\(3\)whereδ\\deltais the minimum deployment improvement required to accept a patch\. Equation[3](https://arxiv.org/html/2608.21836#S3.E3)ties candidate generation to the evaluation objective\.

For phase\-specialized replacements, the patched module uses the dispatch template generated during extraction\. Runtime guards preserve shape compatibility, and fallback paths preserve execution for unoptimized regimes\. The final patch therefore reflects both the agent’s generated implementation and the deployment constraints of the target workload\.

Figure 5:Scope\-matched comparison on transformer\-family attention workloads\. Each panel reports latency for eager execution, LLM4LLM, and expert deployment kernels across representative workload settings on A100 and H100\. Lower is better\.

## 4Evaluation

### 4\.1Experimental Setup

We evaluate LLM4LLM on language\-model inference workloads spanning transformer\-family models, Mamba\-family state\-space models, and RecurrentGemma\. Experiments are conducted on A100 and H100 GPUs\. The primary metric is end\-to\-end latency of the target inference workload, reported together with speedup over eager execution\. LLM4LLM starts optimization from the eager PyTorch execution path\. For scope\-matched comparisons, we evaluate attention\-only and mixer\-only settings against strong deployment baselines, including FlashAttention for attention and Mamba fast paths for state\-space mixers\. Here, Mamba fast paths refer to hand\-optimized kernels used by the Mamba and Mamba\-2 implementations:mamba\_ssmfor selective\-scan and state\-space mixer execution, andcausal\-conv1dfor causal depthwise convolution\([19](https://arxiv.org/html/2608.21836#bib.bib10);[12](https://arxiv.org/html/2608.21836#bib.bib44);[14](https://arxiv.org/html/2608.21836#bib.bib45)\)\. All reported patches pass model\-integrated correctness checks before latency measurement\. The evaluation follows the same acceptance path used by the method: candidates are validated on extracted tasks, inserted into the model instance, and measured through the target inference script\. We report latency because the objective includes launch overhead, guards, cache behavior, and interactions with surrounding model code\.

### 4\.2End\-to\-End Results on Language\-Model Families

Table 1:End\-to\-end latency \(ms\) across language\-model families on A100 and H100\. Lower latency and higher speedup are better\.Table[1](https://arxiv.org/html/2608.21836#S4.T1)shows that the closed\-loop optimization produces end\-to\-end gains across all evaluated families\. These numbers are measured after patch insertion, so they reflect the realized effect of candidate kernels together with dispatch overhead, shape guards, cache state, and surrounding model code\. The transformer\-family results show steady improvements on both GPUs\. For these models, profiling usually selects attention\-dominated regions and adjacent tensor operations whose cost remains visible after existing fused attention paths are enabled\. The gains therefore reflect deployment\-level replacement boundaries and model\-integrated acceptance\.

The state\-space and recurrent families show a different pattern\. Mamba\-family workloads concentrate latency in mixer modules whose computation is regular enough for generated Triton kernels to cover a large share of the inference path\. RecurrentGemma exposes another profile: the dominant recurring module isRecurrentGemmaRglru, whose cost is tied to gated recurrent updates, state movement, and phase\-dependent memory behavior\. This observation motivates the first profiling stage of LLM4LLM\. The dominant deployable kernel is model\-family dependent: transformer traces emphasize attention regions, Mamba traces emphasize state\-space mixers, and RecurrentGemma traces emphasize the RGLRU recurrent core\. A fixed attention\-first policy allocates budget poorly for these families; profiling\-first extraction directs search to the modules that dominate the actual workload\. The A100–H100 differences further show that the accepted patch depends on both model structure and hardware execution context\.

### 4\.3Comparisons with Strong Deployment Baselines

Figure 6:Scope\-matched comparison on state\-space mixer workloads\. Each panel reports latency for eager execution, LLM4LLM, and Mamba\-family fast paths based onmamba\_ssmandcausal\-conv1dacross representative workload settings on A100 and H100\. Lower is better\.Figures[5](https://arxiv.org/html/2608.21836#S3.F5)and[6](https://arxiv.org/html/2608.21836#S4.F6)compare LLM4LLM against strong deployment baselines under matched optimization scope\. The attention comparison evaluates the setting where expert kernels are especially mature\. LLM4LLM reaches competitive latency in several prompt/decode regimes, while the remaining gaps identify workload shapes where specialized attention implementations retain an advantage\. This result is useful for deployment because it separates two questions: whether an LLM\-generated candidate can execute correctly inside the model, and whether the measured attention region is the best use of the search budget for that model\.

The mixer comparison gives complementary evidence on a family with a different dominant operation\. When profiling selects state\-space mixer boundaries, generated replacements can absorb surrounding reshapes, projections, and elementwise updates that are outside the scope of a single vendor or library kernel\. This wider deployable boundary explains why LLM4LLM can approach or improve over the hand\-optimizedmamba\_ssmandcausal\-conv1dpaths in several regimes\. Together with the RecurrentGemma results in Table[1](https://arxiv.org/html/2608.21836#S4.T1), the scope\-matched figures support a profiling\-driven view of optimization: attention, state\-space mixers, and recurrent RGLRU kernels each become the right target only when they dominate the measured workload\. The deployment baselines therefore serve as strong references for their own scopes, while the closed\-loop acceptance step determines which generated patch improves the full inference path\.

Table 2:KernelBench Level\-2 comparison on A100 and H100 GPUs\. Pass, Fast1, and Fast2 are reported as percentages\. Mean and GeoMean are speedups over PyTorch eager unless otherwise specified\.
### 4\.4KernelBench Comparison

KernelBench provides supporting evidence for kernel\-level capability and agent search behavior while the preceding experiments measure deployment transfer\.

We first compare LLM4LLM with recent LLM\-based kernel optimization systems on KernelBench Level 2\. The comparison includes CUDA\-L1 and KernelSkill\([27](https://arxiv.org/html/2608.21836#bib.bib12);[36](https://arxiv.org/html/2608.21836#bib.bib13)\)\. It also covers STARK and QiMeng\-Kernel\([15](https://arxiv.org/html/2608.21836#bib.bib14);[45](https://arxiv.org/html/2608.21836#bib.bib15)\), as well as KernelBlaster and AI CUDA Engineer\([16](https://arxiv.org/html/2608.21836#bib.bib16);[25](https://arxiv.org/html/2608.21836#bib.bib17)\)\. Unless otherwise noted, rows are evaluated on the full KernelBench Level\-2 split\. Because existing papers report results with different aggregation conventions, Table[2](https://arxiv.org/html/2608.21836#S4.T2)keeps arithmetic and geometric means as separate columns while standardizing the most common correctness and fast\-ppmetrics\. The comparison separates correctness, broad speedup, and heavy\-tailed arithmetic gains, which are often conflated in aggregate kernel benchmark reports\. LLM4LLM delegates exploit detection in isolated KernelBench runs to the benchmark harness: unconstrained LLMs can produce benchmark\-exploiting candidates that hard\-code fixed\-test outputs or bypass the intended computation\. This is one reason our primary evidence comes from deployment evaluations, where candidates are patched into real model code and accepted through model\-integrated correctness and end\-to\-end latency\.

### 4\.5Ablation Study

We further isolate the contribution of sampling, iterative refinement, and LLM4LLM’s restart strategy across GPT\-5\.4, Claude Sonnet 4\.6, and GLM\-5\. Each row reports one complete evaluation over the 100 KernelBench Level\-2 tasks\. For each task, we apply the candidate budget shown in the method name, retain the fastest correct candidate after KernelBench warmup and repeated timing, and aggregate the resulting task\-level speedups using Mean, GeoMean, P50, and P75\. Table[3](https://arxiv.org/html/2608.21836#S4.T3)reports the same method grid against PyTorch eager andtorch\.compile\. The first three rows in each model block follow KernelBench\-style sampling and execution\-feedback iteration\([31](https://arxiv.org/html/2608.21836#bib.bib3)\), separating candidate diversity from refinement\. Sample\-10 improves substantially over Sample\-1, indicating that independent diversity is a strong baseline for finding compilable and occasionally fast kernels\. Iter\-10 reaches a 100% pass rate for all three models, while its GeoMean and Fast2 metrics trail the stronger sampling runs in several settings; the search often spends many turns repairing one trajectory after the first viable implementation\.

The LLM4LLM variants add deployment\-aware task construction, validation feedback, and episodic restart with compact experience\. With restart, the 15\-trial setting gives the best GeoMean for all three models against eager execution \(2\.153, 2\.546, and 2\.745\) and also improves thetorch\.compilecomparison\. The percentile columns contextualize the heavy\-tailed arithmetic means: GLM\-5 reaches P50/P75 speedups of 1\.680/5\.539 against eager and 1\.434/5\.281 againsttorch\.compile, while Claude Sonnet 4\.6 reaches 1\.683/5\.017 against eager\. Comparing “w/o restart” with full LLM4LLM across all three models shows that restart contributes beyond a larger optimization budget, supporting the episodic design in Section[3\.2](https://arxiv.org/html/2608.21836#S3.SS2)\. Increasing the budget from 10 to 15 trials yields consistent gains in GeoMean and Fast2, so the 15\-trial configuration is used as the strongest KernelBench setting\.

Table 3:KernelBench Level\-2 ablation against PyTorch eager andtorch\.compile\. Pass, Fast1, and Fast2 are reported as percentages; Mean, GeoMean, P50, and P75 are speedups\.

## 5Related Work

### 5\.1Tensor Program and Kernel Optimization

Compiler and scheduling systems generate efficient tensor programs across operator and graph levels\. Halide separates algorithms from schedules\([32](https://arxiv.org/html/2608.21836#bib.bib18)\), while Tensor Comprehensions and TensorIR expose schedule\-oriented tensor abstractions\([38](https://arxiv.org/html/2608.21836#bib.bib28);[17](https://arxiv.org/html/2608.21836#bib.bib29)\)\. TVM, AutoTVM, and Ansor combine tensor\-program generation with learned search and task scheduling\([7](https://arxiv.org/html/2608.21836#bib.bib19);[8](https://arxiv.org/html/2608.21836#bib.bib20);[43](https://arxiv.org/html/2608.21836#bib.bib21)\); Triton and OpenTuner cover handwritten GPU programs and extensible autotuning\([37](https://arxiv.org/html/2608.21836#bib.bib22);[3](https://arxiv.org/html/2608.21836#bib.bib30)\)\. Graph\- and model\-level systems optimize substitutions, fusion, and dynamic execution across larger computation regions\([21](https://arxiv.org/html/2608.21836#bib.bib31);[22](https://arxiv.org/html/2608.21836#bib.bib32);[29](https://arxiv.org/html/2608.21836#bib.bib33);[4](https://arxiv.org/html/2608.21836#bib.bib34)\)\. LLM4LLM accepts generated kernels through the patched model under a deployment workload\.

### 5\.2LLM Agents for Code and Kernel Generation

Large language models have been applied to code synthesis, repair, feedback\-driven improvement, and repository\-level editing\. Codex, CodeT5, InCoder, and Code Llama establish generation, identifier\-aware pretraining, and infilling foundations\([6](https://arxiv.org/html/2608.21836#bib.bib1);[39](https://arxiv.org/html/2608.21836#bib.bib35);[18](https://arxiv.org/html/2608.21836#bib.bib36);[33](https://arxiv.org/html/2608.21836#bib.bib2)\)\. Search and feedback improve programs in competitive programming, repository repair, and self\-refinement settings\([28](https://arxiv.org/html/2608.21836#bib.bib37);[26](https://arxiv.org/html/2608.21836#bib.bib38);[23](https://arxiv.org/html/2608.21836#bib.bib40);[30](https://arxiv.org/html/2608.21836#bib.bib4)\)\. Reflexion, ReAct, self\-debugging, and LLM compiler models connect reasoning traces with execution feedback\([35](https://arxiv.org/html/2608.21836#bib.bib5);[40](https://arxiv.org/html/2608.21836#bib.bib23);[9](https://arxiv.org/html/2608.21836#bib.bib39);[10](https://arxiv.org/html/2608.21836#bib.bib41)\)\. KernelBench focuses this capability on GPU kernels\([31](https://arxiv.org/html/2608.21836#bib.bib3)\); recent systems add reinforcement learning and multi\-agent planning\([27](https://arxiv.org/html/2608.21836#bib.bib12);[36](https://arxiv.org/html/2608.21836#bib.bib13);[15](https://arxiv.org/html/2608.21836#bib.bib14);[45](https://arxiv.org/html/2608.21836#bib.bib15)\)\. Memory and verification further address cross\-task reuse and benchmark reliability\([16](https://arxiv.org/html/2608.21836#bib.bib16);[25](https://arxiv.org/html/2608.21836#bib.bib17)\)\. LLM4LLM places the search loop inside a deployment\-aware pipeline conditioned on the target inference script\.

### 5\.3Language Model Inference and Serving Optimization

Language\-model serving systems optimize memory, batching, scheduling, and execution phases\. ORCA and vLLM target iteration scheduling and KV\-cache management\([42](https://arxiv.org/html/2608.21836#bib.bib24);[24](https://arxiv.org/html/2608.21836#bib.bib6)\), while Sarathi and SGLang study chunked execution and structured generation runtimes\([1](https://arxiv.org/html/2608.21836#bib.bib7);[44](https://arxiv.org/html/2608.21836#bib.bib25)\)\. FlexGen, DeepSpeed Inference, and DeepSpeed\-FastGen address memory\- and throughput\-oriented generation at larger scales\([34](https://arxiv.org/html/2608.21836#bib.bib42);[2](https://arxiv.org/html/2608.21836#bib.bib43);[20](https://arxiv.org/html/2608.21836#bib.bib26)\)\. At the kernel level, FlashAttention/FlashAttention\-2 and FlashInfer provide optimized attention paths for inference workloads\([11](https://arxiv.org/html/2608.21836#bib.bib8);[13](https://arxiv.org/html/2608.21836#bib.bib9);[41](https://arxiv.org/html/2608.21836#bib.bib27)\)\. State\-space and hybrid recurrent models add fast paths through Mamba, Mamba\-2, causal depthwise convolution, and RecurrentGemma\-style recurrent blocks\([19](https://arxiv.org/html/2608.21836#bib.bib10);[12](https://arxiv.org/html/2608.21836#bib.bib44);[14](https://arxiv.org/html/2608.21836#bib.bib45);[5](https://arxiv.org/html/2608.21836#bib.bib11)\)\. LLM4LLM uses this deployment context as the environment in which generated candidates are validated and accepted\.

## 6Conclusion

LLM4LLM reframes LLM\-based kernel optimization as a deployment\-aware closed\-loop problem\. The framework connects real\-model profiling, phase\-aware task construction, experience\-guided episodic search, and deployment\-time acceptance\. By making the target workload part of candidate generation and acceptance, LLM4LLM aligns kernel search with the execution context that determines inference performance\. Across diverse language\-model families and two GPU platforms, this loop converts generated candidates into end\-to\-end inference gains while clarifying the relationship between isolated benchmark performance and deployment behavior\. The results suggest a practical path for deployable LLM\-generated kernels: profile the real model, search under phase\-aware constraints, and accept candidates through target\-runtime validation\.

## Limitations

LLM4LLM currently targets single\-GPU inference workloads and assumes access to a representative inference script\. The system studies deployment\-aware optimization at the module and model\-instance level; tensor parallelism, pipeline parallelism, continuous batching, and multi\-tenant serving introduce additional acceptance criteria, including communication cost, scheduler interaction, and batch\-level interference\. The method specializes patches to observed shape and phase regimes, so deployment settings with substantially different prompts, decode lengths, batching behavior, or model configurations require re\-profiling and renewed acceptance checks\. Optimization cost is also a practical consideration: search is most attractive when the resulting patch is reused across many inference calls or related model instances\. The quality of generated candidates depends on the coding ability of the underlying model, the search budget, and the availability of relevant implementation patterns\. Future work can extend deployment\-time acceptance to distributed serving systems and richer workload mixtures\.

## Acknowledgments

We thank the anonymous reviewers and the meta\-reviewer for their constructive feedback\. This work was supported by the Zhongguancun Academy \(Grant No\. C20250501\)\. This work was also supported in part by the Shaanxi Key Technology R&D Program under Grant 2024GX\-ZDCYL\-02\-15, in part by the Natural Science Funds for Distinguished Young Scholar of Shaanxi under Grant 2025JC\-JCQN\-079\. This work was also supported by the National Key Laboratory of Advanced Communication Networks \(Grant No\. FFX26641X006\)\.

## References

- Agrawalet al\.\(2023\)A\. Agrawal, A\. Panwar, J\. Mohan, N\. Kwatra, B\. S\. Gulavani, and R\. RamjeeSarathi: efficient LLM inference by piggybacking decodes with chunked prefills\.arXiv preprint arXiv:2308\.16369\.Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Aminabadiet al\.\(2022\)R\. Y\. Aminabadi, S\. Rajbhandari, A\. A\. Awan, C\. Li, D\. Li, E\. Zheng, O\. Ruwase, S\. Smith, M\. Zhang, J\. Rasley, and Y\. HeDeepSpeed\-Inference: enabling efficient inference of transformer models at unprecedented scale\.InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis,Vol\.,pp\. 1–15\.External Links:[Document](https://dx.doi.org/10.1109/SC41404.2022.00051)Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Anselet al\.\(2014\)J\. Ansel, S\. Kamil, K\. Veeramachaneni, J\. Ragan\-Kelley, J\. Bosboom, U\. O’Reilly, and S\. AmarasingheOpenTuner: an extensible framework for program autotuning\.InProceedings of the 23rd international conference on Parallel architectures and compilation,pp\. 303–316\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Anselet al\.\(2024\)J\. Ansel, E\. Yang, H\. He, N\. Gimelshein, A\. Jain, M\. Voznesensky, B\. Bao, P\. Bell, D\. Berard, E\. Burovski, G\. Chauhan, A\. Chourdia, W\. Constable, A\. Desmaison, Z\. DeVito, E\. Ellison, W\. Feng, J\. Gong, M\. Gschwind, B\. Hirsh, S\. Huang, K\. Kalambarkar, L\. Kirsch, M\. Lazos, M\. Lezcano, Y\. Liang, J\. Liang, Y\. Lu, C\. K\. Luk, B\. Maher, Y\. Pan, C\. Puhrsch, M\. Reso, M\. Saroufim, M\. Y\. Siraichi, H\. Suk, S\. Zhang, M\. Suo, P\. Tillet, X\. Zhao, E\. Wang, K\. Zhou, R\. Zou, X\. Wang, A\. Mathews, W\. Wen, G\. Chanan, P\. Wu, and S\. ChintalaPyTorch 2: faster machine learning through dynamic python bytecode transformation and graph compilation\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2,ASPLOS ’24,New York, NY, USA,pp\. 929–947\.External Links:ISBN 9798400703850,[Link](https://doi.org/10.1145/3620665.3640366),[Document](https://dx.doi.org/10.1145/3620665.3640366)Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Botevet al\.\(2024\)A\. Botev, S\. De, S\. L\. Smith, A\. Fernando, G\. Muraru, R\. Haroun, L\. Berrada, R\. Pascanu, P\. G\. Sessa, R\. Dadashi, L\. Hussenot, J\. Ferret, S\. Girgin, O\. Bachem, A\. Andreev, K\. Kenealy, T\. Mesnard, C\. Hardin, S\. Bhupatiraju, S\. Pathak, L\. Sifre, M\. Rivière, M\. S\. Kale, J\. Love, P\. Tafti, A\. Joulin, N\. Fiedel, E\. Senter, Y\. Chen, S\. Srinivasan, G\. Desjardins, D\. Budden, A\. Doucet, S\. Vikram, A\. Paszke, T\. Gale, S\. Borgeaud, C\. Chen, A\. Brock, A\. Paterson, J\. Brennan, M\. Risdal, R\. Gundluru, N\. Devanathan, P\. Mooney, N\. Chauhan, P\. Culliton, L\. G\. Martins, E\. Bandy, D\. Huntsperger, G\. Cameron, A\. Zucker, T\. Warkentin, L\. Peran, M\. Giang, Z\. Ghahramani, C\. Farabet, K\. Kavukcuoglu, D\. Hassabis, R\. Hadsell, Y\. W\. Teh, and N\. de FrietasRecurrentGemma: moving past transformers for efficient open language models\.External Links:2404\.07839,[Link](https://arxiv.org/abs/2404.07839)Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§1](https://arxiv.org/html/2608.21836#S1.p1.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Chenet al\.\(2018a\)T\. Chen, T\. Moreau, Z\. Jiang, L\. Zheng, E\. Yan, H\. Shen, M\. Cowan, L\. Wang, Y\. Hu, L\. Ceze, C\. Guestrin, and A\. KrishnamurthyTVM: an automated End\-to\-End optimizing compiler for deep learning\.In13th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 18\),Carlsbad, CA,pp\. 578–594\.External Links:ISBN 978\-1\-939133\-08\-3,[Link](https://www.usenix.org/conference/osdi18/presentation/chen)Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Chenet al\.\(2018b\)T\. Chen, L\. Zheng, E\. Yan, Z\. Jiang, T\. Moreau, L\. Ceze, C\. Guestrin, and A\. KrishnamurthyLearning to optimize tensor programs\.Advances in Neural Information Processing Systems31\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Chenet al\.\(2023\)X\. Chen, M\. Lin, N\. Schärli, and D\. ZhouTeaching large language models to self\-debug\.arXiv preprint arXiv:2304\.05128\.Cited by:[§1](https://arxiv.org/html/2608.21836#S1.p1.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Cumminset al\.\(2024\)C\. Cummins, V\. Seeker, D\. Grubisic, B\. Roziere, J\. Gehring, G\. Synnaeve, and H\. LeatherMeta large language model compiler: foundation models of compiler optimization\.arXiv preprint arXiv:2407\.02524\.Cited by:[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. RéFlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.Advances in neural information processing systems35,pp\. 16344–16359\.Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Dao and Gu \(2024\)T\. Dao and A\. GuTransformers are SSMs: generalized models and efficient algorithms through structured state space duality\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§4\.1](https://arxiv.org/html/2608.21836#S4.SS1.p1.1),[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Dao \(2023\)T\. DaoFlashAttention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Dao \(2024\)T\. DaoCausal\-conv1d: causal depthwise conv1d in CUDA with a PyTorch interface\.Note:[https://github\.com/Dao\-AILab/causal\-conv1d](https://github.com/Dao-AILab/causal-conv1d)Accessed: 2026\-05\-04Cited by:[§4\.1](https://arxiv.org/html/2608.21836#S4.SS1.p1.1),[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Donget al\.\(2025\)J\. Dong, Y\. Yang, T\. Liu, Y\. Wang, F\. Qi, V\. Tarokh, K\. Rangadurai, and S\. YangSTARK: strategic team of agents for refining kernels\.arXiv preprint arXiv:2510\.16996\.Cited by:[§4\.4](https://arxiv.org/html/2608.21836#S4.SS4.p2.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Donget al\.\(2026\)K\. S\. Dong, S\. Modi, D\. Nikiforov, S\. Damani, E\. Lin, S\. K\. S\. Hari, and C\. KozyrakisKernelBlaster: continual cross\-task CUDA optimization via memory\-augmented in\-context reinforcement learning\.arXiv preprint arXiv:2602\.14293\.Cited by:[§4\.4](https://arxiv.org/html/2608.21836#S4.SS4.p2.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Fenget al\.\(2023\)S\. Feng, B\. Hou, H\. Jin, W\. Lin, J\. Shao, R\. Lai, Z\. Ye, L\. Zheng, C\. H\. Yu, Y\. Yu, and T\. ChenTensorIR: an abstraction for automatic tensorized program optimization\.InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2,ASPLOS 2023,New York, NY, USA,pp\. 804–817\.External Links:ISBN 9781450399166,[Link](https://doi.org/10.1145/3575693.3576933),[Document](https://dx.doi.org/10.1145/3575693.3576933)Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Friedet al\.\(2022\)D\. Fried, A\. Aghajanyan, J\. Lin, S\. Wang, E\. Wallace, F\. Shi, R\. Zhong, W\. Yih, L\. Zettlemoyer, and M\. LewisInCoder: a generative model for code infilling and synthesis\.arXiv preprint arXiv:2204\.05999\.Cited by:[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Gu and Dao \(2023\)A\. Gu and T\. DaoMamba: linear\-time sequence modeling with selective state spaces\.arXiv preprint arXiv:2312\.00752\.Cited by:[§4\.1](https://arxiv.org/html/2608.21836#S4.SS1.p1.1),[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Holmeset al\.\(2024\)C\. Holmes, M\. Tanaka, M\. Wyatt, A\. A\. Awan, J\. Rasley, S\. Rajbhandari, R\. Y\. Aminabadi, H\. Qin, A\. Bakhtiari, L\. Kurilenko, and Y\. HeDeepSpeed\-FastGen: high\-throughput text generation for LLMs via MII and DeepSpeed\-Inference\.External Links:2401\.08671,[Link](https://arxiv.org/abs/2401.08671)Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Jiaet al\.\(2019a\)Z\. Jia, O\. Padon, J\. Thomas, T\. Warszawski, M\. Zaharia, and A\. AikenTASO: optimizing deep learning computation with automatic generation of graph substitutions\.InProceedings of the 27th ACM Symposium on Operating Systems Principles,pp\. 47–62\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Jiaet al\.\(2019b\)Z\. Jia, J\. Thomas, T\. Warszawski, M\. Gao, M\. Zaharia, and A\. AikenOptimizing dnn computation with relaxed graph substitutions\.Proceedings of Machine Learning and Systems1,pp\. 27–39\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Jimenezet al\.\(2023\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.arXiv preprint arXiv:2310\.06770\.Cited by:[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Langeet al\.\(2025\)R\. T\. Lange, Q\. Sun, A\. Prasad, M\. Faldor, Y\. Tang, and D\. HaTowards robust agentic CUDA kernel benchmarking, verification, and optimization\.arXiv preprint arXiv:2509\.14279\.Cited by:[§4\.4](https://arxiv.org/html/2608.21836#S4.SS4.p2.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Leet al\.\(2022\)H\. Le, Y\. Wang, A\. D\. Gotmare, S\. Savarese, and S\. C\. H\. HoiCodeRL: mastering code generation through pretrained models and deep reinforcement learning\.Advances in Neural Information Processing Systems35,pp\. 21314–21328\.Cited by:[§1](https://arxiv.org/html/2608.21836#S1.p1.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Liet al\.\(2025\)X\. Li, X\. Sun, A\. Wang, J\. Li, and C\. ShumCUDA\-L1: improving CUDA optimization via contrastive reinforcement learning\.arXiv preprint arXiv:2507\.14111\.Cited by:[§4\.4](https://arxiv.org/html/2608.21836#S4.SS4.p2.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Liet al\.\(2022\)Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. D\. Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. S\. Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. VinyalsCompetition\-level code generation with AlphaCode\.Science378\(6624\),pp\. 1092–1097\.External Links:[Document](https://dx.doi.org/10.1126/science.abq1158),[Link](https://www.science.org/doi/abs/10.1126/science.abq1158),https://www\.science\.org/doi/pdf/10\.1126/science\.abq1158Cited by:[§1](https://arxiv.org/html/2608.21836#S1.p1.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Maet al\.\(2020\)L\. Ma, Z\. Xie, Z\. Yang, J\. Xue, Y\. Miao, W\. Cui, W\. Hu, F\. Yang, L\. Zhang, and L\. ZhouRammer: enabling holistic deep learning compiler optimizations with\{\\\{rtasks\}\\\}\.In14th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 20\),pp\. 881–897\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. ClarkSelf\-Refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 46534–46594\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/91edff07232fb1b55a505a9e9f6c0ff3-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.21836#S1.p1.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Ouyanget al\.\(2025\)A\. Ouyang, S\. Guo, S\. Arora, A\. L\. Zhang, W\. Hu, C\. Ré, and A\. MirhoseiniKernelBench: can LLMs write efficient GPU kernels?\.arXiv preprint arXiv:2502\.10517\.Cited by:[§1](https://arxiv.org/html/2608.21836#S1.p1.1),[§4\.5](https://arxiv.org/html/2608.21836#S4.SS5.p1.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Ragan\-Kelleyet al\.\(2013\)J\. Ragan\-Kelley, C\. Barnes, A\. Adams, S\. Paris, F\. Durand, and S\. AmarasingheHalide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines\.Acm Sigplan Notices48\(6\),pp\. 519–530\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Rozièreet al\.\(2024\)B\. Rozière, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez, J\. Rapin, A\. Kozhevnikov, I\. Evtimov, J\. Bitton, M\. Bhatt, C\. C\. Ferrer, A\. Grattafiori, W\. Xiong, A\. Défossez, J\. Copet, F\. Azhar, H\. Touvron, L\. Martin, N\. Usunier, T\. Scialom, and G\. SynnaeveCode Llama: open foundation models for code\.External Links:2308\.12950,[Link](https://arxiv.org/abs/2308.12950)Cited by:[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Shenget al\.\(2023\)Y\. Sheng, L\. Zheng, B\. Yuan, Z\. Li, M\. Ryabinin, B\. Chen, P\. Liang, C\. Ré, I\. Stoica, and C\. ZhangFlexGen: high\-throughput generative inference of large language models with a single GPU\.InInternational Conference on Machine Learning,pp\. 31094–31116\.Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Sunet al\.\(2026\)Q\. Sun, J\. Han, T\. Li, Z\. Tang, S\. Chen, F\. Yang, A\. Liu, X\. Liu, and Y\. LiuKernelSkill: a multi\-agent framework for GPU kernel optimization\.arXiv preprint arXiv:2603\.10085\.Cited by:[§4\.4](https://arxiv.org/html/2608.21836#S4.SS4.p2.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Tilletet al\.\(2019\)P\. Tillet, H\. Kung, and D\. CoxTriton: an intermediate language and compiler for tiled neural network computations\.InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages,pp\. 10–19\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Vasilacheet al\.\(2018\)N\. Vasilache, O\. Zinenko, T\. Theodoridis, P\. Goyal, Z\. DeVito, W\. S\. Moses, S\. Verdoolaege, A\. Adams, and A\. CohenTensor comprehensions: framework\-agnostic high\-performance machine learning abstractions\.arXiv preprint arXiv:1802\.04730\.Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Wanget al\.\(2021\)Y\. Wang, W\. Wang, S\. Joty, and S\. C\. HoiCodeT5: identifier\-aware unified pre\-trained encoder\-decoder models for code understanding and generation\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 8696–8708\.Cited by:[§1](https://arxiv.org/html/2608.21836#S1.p1.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.
- Yeet al\.\(2025\)Z\. Ye, L\. Chen, R\. Lai, W\. Lin, Y\. Zhang, S\. Wang, T\. Chen, B\. Kasikci, V\. Grover, A\. Krishnamurthy, and L\. CezeFlashInfer: efficient and customizable attention engine for LLM inference serving\.InProceedings of Machine Learning and Systems,M\. Zaharia, G\. Joshi, and Y\. Lin \(Eds\.\),Vol\.7,pp\.\.External Links:[Link](https://proceedings.mlsys.org/paper_files/paper/2025/file/dbf02b21d77409a2db30e56866a8ab3a-Paper-Conference.pdf)Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Yuet al\.\(2022\)G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. ChunOrca: a distributed serving system for Transformer\-Based generative models\.In16th USENIX symposium on operating systems design and implementation \(OSDI 22\),pp\. 521–538\.Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Zhenget al\.\(2020\)L\. Zheng, C\. Jia, M\. Sun, Z\. Wu, C\. H\. Yu, A\. Haj\-Ali, Y\. Wang, J\. Yang, D\. Zhuo, K\. Sen, J\. E\. Gonzalez, and I\. StoicaAnsor: generating High\-Performance tensor programs for deep learning\.In14th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 20\),pp\. 863–879\.External Links:ISBN 978\-1\-939133\-19\-9,[Link](https://www.usenix.org/conference/osdi20/presentation/zheng)Cited by:[§5\.1](https://arxiv.org/html/2608.21836#S5.SS1.p1.1)\.
- Zhenget al\.\(2024\)L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. ShengSGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 62557–62583\.External Links:[Document](https://dx.doi.org/10.52202/079017-2000),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf)Cited by:[§5\.3](https://arxiv.org/html/2608.21836#S5.SS3.p1.1)\.
- Zhuet al\.\(2025\)X\. Zhu, S\. Peng, J\. Guo, Y\. Chen, Q\. Guo, Y\. Wen, H\. Qin, R\. Chen, Q\. Zhou, K\. Gao, Y\. Wu, C\. Zhao, and L\. LiQiMeng\-kernel: macro\-thinking micro\-coding paradigm for llm\-based high\-performance gpu kernel generation\.External Links:2511\.20100,[Link](https://arxiv.org/abs/2511.20100)Cited by:[§4\.4](https://arxiv.org/html/2608.21836#S4.SS4.p2.1),[§5\.2](https://arxiv.org/html/2608.21836#S5.SS2.p1.1)\.

## Appendix AAppendix

This appendix expands on mechanisms that are only summarized in the main paper\. The examples come from saved run artifacts and use the same acceptance principle as the main experiments: a candidate is timed only after it passes the corresponding correctness check\. Section[A\.1](https://arxiv.org/html/2608.21836#A1.SS1)analyzes restart as search\-state control; Section[A\.2](https://arxiv.org/html/2608.21836#A1.SS2)isolates search diversity from linear repair; Section[A\.3](https://arxiv.org/html/2608.21836#A1.SS3)discusses API compatibility for fast\-moving kernel DSLs; and Sections[A\.4](https://arxiv.org/html/2608.21836#A1.SS4)–[A\.5](https://arxiv.org/html/2608.21836#A1.SS5)summarize model\-backbone behavior and accepted deployment kernels\.

### A\.1Search\-State Control by Restart

Table[A\.1](https://arxiv.org/html/2608.21836#A1.T1)shows a restart case from KernelBench Level 2, problem 97,Matmul\_BatchNorm\_BiasAdd\_Divide\_Swish\. Both variants use GPT\-5\.4, Triton fp32, an A100, and a 15\-candidate budget\. The no\-restart run first found a correct epilogue\-only candidate, then repeated the same output\-mismatch signature for three turns, and eventually returned to a family of correct epilogue\-only implementations around 8 ms\. The restart\-enabled run produced fullLinear \+ BatchNorm \+ bias/divide/Swishfusion inside the GEMM epilogue, reducing the best latency from 8\.01 ms to 1\.42 ms\.

Table A\.1:Restart case on KernelBench Level 2 problem 97, measured on A100\. The baseline eager runtime is 8\.23 ms for both rows\.
This case illustrates the role of restart as search\-state control\. The useful feedback is compact: the operation sequence, the inference\-mode BatchNorm contract, and the need to preserve the bias and activation semantics\. The long repair history contains many local choices tied to one partially fused layout\. Restart carries forward the compact constraints while allowing the next episode to reselect the optimization scope, which can move the search from epilogue fusion to GEMM\-level fusion\. The generated code exposes the difference directly:

> ``` # no restart: best candidate x = self.matmul(x) x = self.bn(x) x = fused_bias_div_swish( x, self.bias, self.divide_value) # restart: best candidate return fused_linear_bn_bias_div_swish( x, self.matmul.weight, self.matmul.bias, self.bn.running_mean, self.bn.running_var, self.bn.weight, self.bn.bias, self.bias, self.bn.eps, self.divide_value) ```

The corresponding kernel bodies show the same distinction:

> ``` # no restart: epilogue-only kernel y = (x + b) * inv_divide_value y = y * tl.sigmoid(y) # restart: GEMM-level fused kernel acc += tl.dot(x, w) acc = acc + lin_b[None, :] acc = (acc - mean[None, :]) * inv_std[None, :] acc = acc * gamma[None, :] + beta[None, :] acc = (acc + extra_b[None, :]) / divide_value acc = acc * tl.sigmoid(acc) ```

In the no\-restart trace, iteration 1 is correct at 8\.37 ms, iterations 2–4 repeat the same output\-mismatch signature, and iterations 5–15 return to correct epilogue\-only variants around 8 ms\. With restart enabled, correct full\-fusion candidates appear at iteration 2 \(1\.45 ms\) and iteration 3 \(1\.42 ms\), while later candidates explore both fast and slow alternatives\.

### A\.2Sampling and Iteration Are Complementary

Table[A\.2](https://arxiv.org/html/2608.21836#A1.T2)gives two complementary contrasts from GPT\-5\.4 KernelBench Level 2 runs\. Problem 76,Gemm\_Add\_ReLU, shows the value of search diversity: sampling found a full\-fusion implementation, while the recorded iterative run stopped at a correct candidate that left GEMM in PyTorch and fused only the bias\-ReLU epilogue\. Problem 1,Conv2D\_ReLU\_BiasAdd, shows the reverse pattern: the best one\-shot sample was correct but slower than eager execution, while iterative repair fixed an initial output mismatch and produced a faster NCHW\-specialized epilogue kernel\. The point is not that either sampling or iteration dominates; the two mechanisms expose different useful candidates\.

Table A\.2:Complementary sampling and iteration examples from GPT\-5\.4 KernelBench Level 2 runs on A100\. Speedup is against the eager baseline for each problem\.
For problem 76, the core generated code shows the optimization\-scope distinction\. The sampled candidate fuses the matrix multiplication and epilogue into one Triton implementation:

> ``` # Sample-10 best candidate return fused_linear_bias_relu(x, weight, bias) # kernel core acc += tl.dot(a, b) acc += bias[None, :] acc = tl.maximum(acc, 0.0) ```

The iterative candidate is correct, but its generated structure preserves the PyTorch GEMM and only moves the epilogue to Triton:

> ``` # Iter-10 best candidate y = self.gemm(x) y = triton_bias_relu(y, self.bias) return y # kernel core y = tl.maximum(x + b[None, :], 0.0) ```

For problem 1, the useful signal is different\. The iterative run first produced an output mismatch; after one repair turn, it kept the same conservative module boundary as the sampled code but generated a cleaner NCHW\-specialized epilogue kernel:

> ``` # Sample-10 best candidate x = self.conv(x) x = triton_relu_bias(x, self.bias) return x # kernel core hw = H * W c = (offs // hw) % C b = tl.load(bias_ptr + c, mask=mask, other=0.0) y = tl.maximum(x, 0.0) + b ```

> ``` # Iter-10 best candidate, after repair x = self.conv(x) x = triton_relu_bias_nchw(x, self.bias) return x # kernel core c = (offs // HW) % C b = tl.load(bias_ptr + c, mask=mask, other=0.0) y = tl.maximum(x, 0.0) + b ```

Together, these examples explain why LLM4LLM combines sampling, feedback, and restart instead of treating iterative repair as a purely monotonic process\. Sampling exposes alternative decompositions of the same PyTorch graph, while linear repair is effective for turning a nearby candidate into a valid and better\-specialized one\. Restart combines these roles: each new episode can choose a fresh optimization scope, but it still receives compact correctness and implementation constraints learned from previous attempts\.

### A\.3API Compatibility and User\-Defined Fixes

This section separates interface compatibility from the optimization\-scope issue in Section[A\.2](https://arxiv.org/html/2608.21836#A1.SS2)\. Kernel DSLs such as Triton and TileLang evolve quickly, and local installations can differ from the APIs seen during model training\. In KernelBench Level 2 problem 86,Matmul\_Divide\_GELU, an iterative candidate failed because the generated GELU approximation called an unavailable Triton math entry point:

> ``` # failed iterative candidate inner = c0 * (x + c1 * x * x * x) y = 0.5 * x * (1.0 + tl.math.tanh(inner)) ```

After repair, the run obtained a correct epilogue\-only kernel, while an independent sampled candidate reached a fusedLinear \+ divide \+ GELUimplementation:

Table A\.3:API\-affected example on KernelBench Level 2 problem 86, measured on A100\.
The generated code differs at the kernel boundary:

> ``` # Sample-10 best candidate return triton_linear_div_gelu( x, self.linear.weight, self.linear.bias, self.divisor) # Iter-10 repaired candidate x = self.linear(x) x = triton_div_gelu(x, self.divisor) return x ```

LLM4LLM therefore includes an API\-correction layer that users can extend for their local backend versions\. The correction file is backend specific and currently covers Triton, TileLang, and CUDA extension patterns, for example:

> ``` tl.math.tanh -> libdevice.tanh tl.math.max -> tl.maximum tl.math.min -> tl.minimum T.Ranged -> tilelang.language.Range data<T>() -> data_ptr<T>() ```

These fixes keep version\-dependent interface repair separate from performance\-relevant kernel design decisions such as fusion scope, tiling, masking, accumulator precision, cache updates, and launch structure\.

### A\.4Behavior Across LLM Backbones

The ablation table shows that LLM backbones differ in more than final pass rate\. GPT\-5\.4 follows repair feedback reliably, but the examples above show that it can become conservative once a correct partial\-fusion implementation is available\. Restart and experience summaries are useful in this setting because they preserve stable constraints without preserving every local edit in the failed trajectory\.

Claude Sonnet 4\.6 tends to produce well\-guarded code with explicit fallback paths\. This behavior helps coverage and makes deployment\-time acceptance easier to apply, while the search loop still has to test whether guarded candidates enter the optimized path under real prefill/decode conditions\. GLM\-5 produces more aggressive candidates and higher upper\-tail speedups in the KernelBench ablation\. The same search loop accommodates these behaviors by using identical correctness, timing, and deployment\-acceptance rules for all backbones\.

### A\.5Deployment Kernel Families

Table[A\.4](https://arxiv.org/html/2608.21836#A1.T4)summarizes representative accepted kernels from extracted language\-model tasks\. The numbers are task\-level validation speedups on extracted modules; the end\-to\-end effect after patching is reported in Table[1](https://arxiv.org/html/2608.21836#S4.T1)and Figures[5](https://arxiv.org/html/2608.21836#S3.F5)–[6](https://arxiv.org/html/2608.21836#S4.F6)\.

Table A\.4:Representative accepted Triton kernels on extracted deployment tasks, measured on A100\. Speedup is measured on the extracted validation task for the module, before full\-model aggregation\.Transformer attention\.The accepted attention kernels are phase specialized\. For prefill, the kernel maps one Triton program to a query\-head and query\-token tile, computes the KV head from the grouped\-query structure, embeds causal masking in the score update, and maintains online\-softmax statistics in fp32\. This structure reduces memory traffic by keeping the attention matrix implicit and by sharing KV loads according to the model’s head grouping\. For decode, the kernel targets the single\-token regime and processes one KV head per program while computing the associated query\-head group together\. The decode path also keeps mask handling and output layout aligned with the patched module, so the surrounding projection and cache update see the same tensor contract as eager execution\.

Mamba\-family state\-space mixers\.The Mamba\-family cases show the value of optimizing a deployable module boundary that spans multiple primitive operations\. For Mamba2 prefill, the accepted path preserves the mixer sequence of input projection, depthwise convolution, split hidden/state parameters, recurrent scan, gated RMSNorm, and output projection\. Inside the recurrent kernel, compactBBandCCstate vectors are loaded once per time step and broadcast logically across heads, while the state tile remains in registers across the scan\. For decode, the accepted kernels specialize toseq\_len = 1with an existing cache, update convolution and SSM state explicitly, and minimize launch overhead for the recurring single\-token path\. These cases explain why profiling selects Mamba mixers as high\-impact targets in Table[1](https://arxiv.org/html/2608.21836#S4.T1)\.

RecurrentGemma RGLRU\.RecurrentGemma exposes a gated recurrent core whose latency comes from both elementwise gate transformations and recurrent state movement\. The accepted RGLRU implementation fuses sigmoid, softplus\-derived recurrent gating, reset handling, normalization, and the recurrent scan into Triton kernels that preserve the state update semantics\. The key advantage is that the recurrent state is carried through the sequence inside the kernel and the final state is written once, reducing Python\-level dispatch and intermediate tensor traffic\. This case is distinct from attention and Mamba: the dominant optimization target is the recurrent update itself, which supports the profiling\-first design used by LLM4LLM\.

Across these families, the accepted kernels are useful because the generated code respects the module boundary that the deployment patch will actually replace\. For attention, this means that the code must match the phase\-local cache contract: a prefill kernel can assume a query block and construct causal tiles, while a decode kernel must read a populated KV cache and update only the single\-token output path\. For Mamba and recurrent blocks, the same principle appears as state ownership rather than KV ownership\. The kernel must update convolution, SSM, or recurrent state exactly once and return tensors with the same layout expected by the surrounding model\. These examples are therefore not just faster isolated kernels\. They are accepted because the optimized path can be inserted into the profiled model instance without changing the caller\-visible tensor, cache, or phase semantics\.

### A\.6Summary

The concrete run traces support the design choices in the main paper\. Restart is useful when a short set of constraints should survive but a long repair trajectory should not dominate the next implementation\. Sampling provides optimization\-scope diversity that a single linear repair path may not expose\. API fixes separate version\-dependent DSL compatibility from performance\-relevant kernel design\. The deployment cases show that the accepted kernels are not one generic template: attention, Mamba mixers, and recurrent blocks require different phase contracts and cache semantics\.