$\Phi$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

# Can Large Language Models Engineer the Infrastructure That Powers Them?
Source: [https://arxiv.org/html/2609.10226](https://arxiv.org/html/2609.10226)
\\xpatchcmd

\\aaai@affiliations

\\xpatchcmd

\\aaai@affiliations

Shumin WangYuting HuangFanqi WanYinmin ZhangQi HanYiming XuFeiyuan ZhangXiaomeng ChuGuoliang YouWuyang ZhangDaxin JiangYanyong Zhang

###### Abstract

Large language models \(LLMs\) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them\. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre\-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open\-ended, long\-horizon LLM infrastructure engineering\. To address this gap, we presentΦ\\Phi\-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack\. Derived from optimization problems studied in frontier research and grounded in real\-world code repositories,Φ\\Phi\-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel\-level function completion to long\-horizon implementation and end\-to\-end system optimization\. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure\.

1University of Science and Technology of China

2StepFun

3Peking University

4The Hong Kong University of Science and Technology

5Yale University

6University of Pennsylvania

attr/Border \[0 0 0\] user/Subtype /Link /A << /S /URI /URI \(https://faibench\.org\) \>\>Leaderboardattr/Border \[0 0 0\] user/Subtype /Link /A << /S /URI /URI \(https://github\.com/one2piece2hello/faibench\_LLM\_Infra\_Bench\) \>\>GitHubattr/Border \[0 0 0\] user/Subtype /Link /A << /S /URI /URI \(https://huggingface\.co/datasets/faibench\-Frontier\-Infra\-Bench/faibench\_Frontier\_Infra\_Bench\) \>\>Hugging Face

‡‡footnotetext:Work done as an intern at StepFun\.\*\*footnotetext:Equal contribution\.††footnotetext:Correspondence to: Daxin Jiang<djiang@stepfun\.com\>and Yanyong Zhang<yanyongz@ustc\.edu\.cn\>\.## Introduction

With recent advances in large language models \(LLMs\) in reasoning\([Guo et al\. 2025](https://arxiv.org/html/2609.10226#bib.bib20)\)and code generation\([Z\.AI 2026](https://arxiv.org/html/2609.10226#bib.bib16)\), leveraging LLMs to support the development of next\-generation AI models has emerged as a promising research direction\([Ma et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib22);[Ishibashi et al\. 2025](https://arxiv.org/html/2609.10226#bib.bib21)\)\. A particularly important challenge in this context is engineering and optimizing the software infrastructure underlying LLM training and inference, hereafter referred to as*LLM infrastructure*\. Prior work\([Kwon et al\. 2023](https://arxiv.org/html/2609.10226#bib.bib10);[Narayanan et al\. 2021](https://arxiv.org/html/2609.10226#bib.bib12)\)has demonstrated that infrastructure\-level optimizations can substantially improve GPU utilization and reduce computational costs\. These developments naturally raise an intriguing question:Can large language models engineer and optimize the infrastructure that powers them?

Although several benchmarks have evaluated LLMs on GPU kernel implementation\([Ouyang et al\. 2025](https://arxiv.org/html/2609.10226#bib.bib1);[Li et al\. 2025](https://arxiv.org/html/2609.10226#bib.bib2)\)and optimization\([Nangia et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib3);[Li et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib4)\), their evaluation settings are typically restricted to individual functions or small collections of isolated components\. Consequently, they do not fully capture the complexity of engineering and optimizing real\-world LLM infrastructure\. In practice, such tasks are inherently long\-horizon and open\-ended: developers must understand and navigate an existing infrastructure stack, identify bottlenecks and optimization opportunities, and iteratively implement, profile, debug, and refine their solutions\. This end\-to\-end workflow extends far beyond code completion or isolated, single\-commit modifications\. Furthermore, existing benchmarks cover only a narrow subset of the topics involved in LLM infrastructure engineering, limiting their ability to comprehensively assess whether an LLM can develop and optimize modern AI infrastructure\. These limitations motivate the need for a comprehensive, long\-horizon, and open\-ended benchmark that more faithfully evaluates whether LLMs can engineer the infrastructure that powers them\.

![Refer to caption](https://arxiv.org/html/2609.10226v1/Performance.png)Figure 1:Performance of frontier models onΦ\\Phi\-Bench\.Left: Overall scores achieved by different models\.Right: Model performance across different infrastructure topics\.To bridge this gap, we introduceΦ\\Phi\-Bench, the Frontier AI Infrastructure Benchmark, as a systematic evaluation of frontier LLMs on real\-world workloads derived from top\-tier systems papers and public LLM infrastructure repositories\. Three principles guide its design: long\-horizon, open\-ended problem solving, comprehensive coverage of LLM infrastructure, and scalable task synthesis\. Unlike benchmarks centered only on completing isolated functions,Φ\\Phi\-Bench allows agents to navigate existing codebases and iteratively implement, profile, debug, and optimize their solutions\.

To achieve broad coverage, we construct a bottom\-up taxonomy of modern LLM infrastructure from 2,260 papers and 1,852 artifacts collected from public repositories, and use it to guide task synthesis\. To scale benchmark construction beyond the limited supply of manually curated engineering artifacts, we develop an agent\-loop\-based pipeline that automatically mines high\-value challenges from repositories and iteratively generates comprehensive test cases\. Together, these designs enable a broad, realistic, and scalable evaluation of LLM agents on infrastructure engineering\.

The resulting benchmark comprises 85 challenging tasks in three task formats with progressively increasing scope and open\-endedness: Kernel Function Completion \(KFC\), Long\-Horizon Implementation \(LHI\), and End\-to\-End Optimization \(E2EO\)\. Together, these formats form a graduated evaluation spanning local kernel implementation, repository\-scale infrastructure development, and end\-to\-end system optimization\. This design enablesΦ\\Phi\-Bench to distinguish an agent’s ability to implement efficient computational primitives, conduct long\-horizon codebase engineering, and perform open\-ended, hypothesis\-driven infrastructure optimization\.

UsingΦ\\Phi\-Bench, we conduct a systematic evaluation of frontier LLMs\. The best\-performing model, Claude Opus 5, achieves a score of36\.53%36\.53\\%, leaving substantial room for improvement\. Further experiments show how the number of refinement iterations and the reasoning budget affect model performance\. Detailed analyses of model solution trajectories further reveal their distinct strengths and weaknesses, providing insights into future model improvements, and highlight the key challenges that must be addressed before LLMs can reliably contribute to engineering and optimizing the infrastructure that powers them\.

In summary, we make the following contributions:

- •We introduceΦ\\Phi\-Bench, the Frontier AI Infrastructure Benchmark, comprising 85 challenging tasks grounded in real\-world efforts to engineer and optimize the infrastructure for LLM training and inference\.
- •We develop a systematic, taxonomy\-guided benchmark construction methodology grounded in research papers and repository artifacts and an agent\-loop\-based synthesis pipeline that automatically mines candidate engineering problems from repositories and constructs comprehensive test cases through iterative test generation\.
- •We systematically evaluate frontier LLMs onΦ\\Phi\-Bench and analyze their solution trajectories, revealing substantial room for improvement and providing insights into future model improvements\.

## Related Works

### LLM Infrastructure Optimization

LLM infrastructure optimization aims to improve the performance, efficiency, and scalability of LLM training and inference across multiple layers\. At the computation layer, CUTLASS\([NVIDIA 2017](https://arxiv.org/html/2609.10226#bib.bib8)\)provides reusable building blocks for high\-performance GPU kernels, while Triton\([Tillet et al\. 2019](https://arxiv.org/html/2609.10226#bib.bib6)\)offers a programming language and compiler for developing optimized GPU programs\. Complementing these low\-level abstractions, systems such as FlashAttention\([Dao et al\. 2022](https://arxiv.org/html/2609.10226#bib.bib29)\)and FlashInfer\([Ye et al\. 2025](https://arxiv.org/html/2609.10226#bib.bib30)\)further accelerate LLM training and inference by IO\-aware attention algorithms, optimized inference kernels, and hardware\-efficient execution\.

Beyond individual kernels and runtimes, training frameworks such as Megatron\-LM\([Shoeybi et al\. 2019](https://arxiv.org/html/2609.10226#bib.bib5)\)and DeepSpeed\([Rasley et al\. 2020](https://arxiv.org/html/2609.10226#bib.bib7)\)support parallel execution, memory partitioning, and communication optimization, whereas inference systems such as Orca\([Yu et al\. 2022](https://arxiv.org/html/2609.10226#bib.bib9)\), vLLM\([Kwon et al\. 2023](https://arxiv.org/html/2609.10226#bib.bib10)\), and SGLang\([Zheng et al\. 2024](https://arxiv.org/html/2609.10226#bib.bib11)\)provide KV\-cache management, continuous batching, request scheduling, and distributed serving\.

### Benchmarks for LLM Infrastructure Engineering

Recent benchmarks have evaluated the ability of LLMs to implement and optimize components of LLM infrastructure\([Lin et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib27);[Wang et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib28)\)\. KernelBench\([Ouyang et al\. 2025](https://arxiv.org/html/2609.10226#bib.bib1)\), TritonBench\([Li et al\. 2025](https://arxiv.org/html/2609.10226#bib.bib2)\), and FlashInfer\-Bench\([Xing et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib26)\)primarily focus on individual GPU operators or fused operator compositions, with predefined interfaces, input\-output specifications, and optimization objectives\. Consequently, they do not require LLMs to navigate complete infrastructure repositories, identify performance bottlenecks, or coordinate modifications across multiple layers of the software stack\. ISO\-Bench\([Nangia et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib3)\)and CUDAHercules\([Li et al\. 2026](https://arxiv.org/html/2609.10226#bib.bib4)\)extend this evaluation to repository\-level GPU optimization, but their tasks typically specify the target component and performance bottleneck and therefore provide limited evidence of whether an LLM can resolve open\-ended LLM infrastructure engineering challenges\.

![Refer to caption](https://arxiv.org/html/2609.10226v1/framework.png)Figure 2:The task synthesis pipeline ofΦ\\Phi\-Bench\.

## Benchmark Design and Construction

In this section, we describe the task formats, the construction process, and the evaluation metrics illustrated in Figure[2](https://arxiv.org/html/2609.10226#Sx2.F2)\.

### Task Formats

EachΦ\\Phi\-Bench task comprises a natural\-language specification, a complete LLM infrastructure repository, executable workloads or test cases, and an evaluation harness\. All materials, including the test cases, are visible to the agent except the evaluation harness\. The specification defines either the required functionality or the performance objective\.

Φ\\Phi\-Bench includes three task formats with increasing scope and open\-endedness\. KFC specifies the target function and its interface; LHI specifies a feature but leaves its implementation path open; and E2EO specifies only a system\-level objective and constraints\. Their editable and submission scopes expand accordingly, as summarized in Table[1](https://arxiv.org/html/2609.10226#Sx3.T1)\.

Table 1:Comparison of the three task types\.TaskTypeImplementationObjectiveEditableRegionSubmissionLimitKFCSpecifiedSingle fileSingleLHISpecifiedMultiple filesMultipleE2EOFreeWhole repositoryMultiple#### Kernel Function Completion \(KFC\)\.

A KFC task isolates a performance\-critical kernel or operator whose interface and input\-output semantics are explicitly specified\. The agent completes or optimizes its implementation within a single file without changing the external interface\. Submissions are first tested for functional correctness and numerical accuracy, after which correct solutions are benchmarked for efficiency\. KFC therefore evaluates the implementation of correct and efficient computational primitives\.

#### Long\-Horizon Implementation \(LHI\)\.

An LHI task provides an issue\-style feature request and a coarse\-grained editable scope, such as a repository submodule, while leaving the relevant files, dependencies, and implementation strategy unspecified\. Solving it requires navigating the codebase, understanding interactions among modules, modifying multiple files, and iteratively building, testing, and debugging the implementation\. LHI thus evaluates nontrivial repository\-level development rather than isolated function completion\.

#### End\-to\-End Optimization \(E2EO\)\.

An E2EO task provides a representative workload, a system\-level optimization objective, and a set of constraints, without prescribing which components or strategies to use\. The agent must profile the system, identify bottlenecks, formulate an optimization plan, and implement potentially repository\-wide changes\. E2EO therefore evaluates bottleneck discovery, cross\-layer reasoning, and hypothesis\-driven infrastructure optimization\.

Figure 3:The high\- and middle\-level topics of our coverage taxonomy\. The lines in the outer ring represent the topics covered by different benchmarks\.

### Task Synthesis

Φ\\Phi\-Bench tasks are derived from real\-world research and engineering artifacts related to LLM infrastructure rather than from manually authored synthetic programming prompts\. We collect papers published at top\-tier systems conferences together with artifacts from public GitHub repositories implementing representative LLM infrastructure\. We then construct a coverage taxonomy of LLM infrastructure engineering and synthesize tasks under its guidance\.

#### Source collection and filtering\.

For the academic source pool, we collect papers published at top\-tier systems conferences during the four\-year period from 2023 to 2026\. We then use an LLM\-based filtering pipeline to retain papers that are directly related to LLM infrastructure, primarily study the implementation, acceleration, or resource optimization of LLM training or inference, and provide publicly available implementations\. For the engineering source pool, we inspect issues and pull requests from public repositories of widely used LLM infrastructure\. We similarly use LLMs to filter for nontrivial and technically challenging artifacts, prioritizing those that introduce substantial optimizations, add important infrastructure capabilities, address hardware\- or workload\-specific limitations, or document difficult engineering problems\. This process yields 2,260 papers and 1,852 engineering artifacts\.

#### Coverage taxonomy construction\.

We organize the selected sources into a hierarchical, three\-level coverage taxonomy\. For each source, we prompt a large language model to assign labels at the three predefined levels\. The top level captures broad areas of LLM infrastructure engineering and optimization\. The middle level identifies more specific research problems, techniques, infrastructure components, and workload characteristics, while the bottom level consists of fine\-grained tags describing the concrete problems addressed by the sources\.

From the 4,112 sources described above, we extract 410 fine\-grained tags capturing the concrete problems, techniques, and infrastructure characteristics represented in the source pool\. We then cluster these tags into 62 middle\-level topics, which are further organized into nine broad top\-level categories\. We use the middle\-level topics as the primary index for task synthesis, while the provenance and characteristics of each source determine whether it is used to construct a KFC, LHI, or E2EO task\. As illustrated in figure[3](https://arxiv.org/html/2609.10226#Sx3.F3), the taxonomy helpsΦ\\Phi\-Bench to achieve much more comprehensive coverage of LLM infrastructure engineering topics than any other related benchmarks

Guided by the coverage taxonomy, we construct tasks through three complementary approaches: PR\- and Issue\-Grounded Synthesis, Agent\-Assisted Synthesis, and Expert\-Curated Synthesis\. We describe each synthesis approach in detail below\.

#### PR\- and Issue\-Grounded Synthesis\.

We identify important pull requests and issues from public LLM infrastructure repositories and reconstruct them as benchmark tasks\. For each selected artifact, we use the repository state before the corresponding change as the task input and treat the implementation after the change as the reference solution\. Whenever available, the associated unit tests are adapted to construct the evaluation harness\. Small, well\-scoped changes confined to a single file are typically converted into KFC tasks, whereas larger changes that span multiple files or infrastructure components are used to construct LHI or E2EO tasks\. This procedure preserves both the real\-world provenance of each task and the original engineering objective of the underlying repository change\.

#### Agent\-Assisted Synthesis\.

We employ agents to scan code repositories using a predefined synthesis pipeline and identify high\-value implementation sites that can be converted into benchmark tasks\. Depending on the scope and complexity of a selected site, we remove either a localized code segment or an entire implementation module, thereby creating tasks with different levels of implementation horizon and infrastructure context\. To automate test generation, we further construct an agent loop that analyzes the execution flow of the target code\. During execution, the agent tracks the branches exercised by the current test suite and iteratively generates additional test cases when it identifies previously uncovered execution paths\. The resulting tests are used to evaluate whether a submitted implementation correctly handles the behaviors represented by the target code\.

#### Expert\-Curated Synthesis\.

For problems that cannot be reliably identified or reconstructed from repository changes alone, human experts curate challenging and high\-value tasks from influential research papers and conference challenges\. After identifying a target problem, the annotators construct an appropriate initial repository state by removing the relevant implementation, reverting the repository to a simpler baseline, or, when no mature implementation is available, formulating the task directly from the problem specification\. They then manually design test cases and evaluation workloads that precisely exercise the targeted capability\. This approach is primarily used for problems that require substantial infrastructure\-level reasoning, involve open\-ended solution strategies, or represent emerging challenges for which repository history does not provide a complete reference trajectory\.

#### Quality Control\.

We impose additional requirements during benchmark construction to ensure that the resulting evaluation is well posed and discriminative\. For a task evaluated with the performance metric, the reference solution must achieve a stable performance of at least1\.151\.15in the official execution environment, whereas the starter or no\-op solution must not outperform the baseline\. For a task evaluated with the implementation metric, the reference solution must pass all test cases in the official environment, while the starter solution must fail at least one test\.

Each task evaluated with the implementation metric contains at least five test cases covering multiple behavioral categories, including normal inputs, boundary conditions, error paths, and regression scenarios\. A task whose reference solution fails these requirements is considered invalid and is revised or excluded rather than assigned a score\.

### Task Distribution

The final version ofΦ\\Phi\-Bench contains 85 tasks, comprising 55 KFC tasks, 20 LHI tasks, and 10 E2EO tasks\. Figure[4](https://arxiv.org/html/2609.10226#Sx3.F4)summarizes their distribution across task formats and infrastructure topics\.Φ\\Phi\-Bench covers all nine major infrastructure topic categories and, where permitted by repository characteristics, includes multiple task formats within each category\.

### Evaluation Metrics

Φ\\Phi\-Bench assigns each task attempt a reward in\[0,1\]\[0,1\], with higher values indicating better performance\. Across the three task formats defined above, we use two scoring metrics according to the evaluation objective of each task: a continuous performance metric for tasks with an explicit efficiency objective and a binary implementation metric for tasks evaluated primarily by functional completion\. Both metrics are correctness\-gated: buildability, functional correctness, edit constraints, and anti\-cheating checks are evaluated before any reward is awarded\.

![Refer to caption](https://arxiv.org/html/2609.10226v1/temp.png)Figure 4:Task distribution inΦ\\Phi\-Bench\.#### Performance Metric\.

For tasks with an explicit performance objective, a candidate solution must first pass a set of hard validity checks, including static checks, editable region checks, and functionality checks\. For a valid candidate, we measure its performance relative to the unmodified baseline using an AB\-BA paired measurement protocol with at least five measurement pairs\. Lettibaset\_\{i\}^\{\\mathrm\{base\}\}andticandt\_\{i\}^\{\\mathrm\{cand\}\}denote the performance score of the baseline and candidate implementations in theii\-th pair, respectively\. The candidate performance is defined as

s=mediani⁡\(tibaseticand\)\.s=\\operatorname\{median\}\_\{i\}\\left\(\\frac\{t\_\{i\}^\{\\mathrm\{base\}\}\}\{t\_\{i\}^\{\\mathrm\{cand\}\}\}\\right\)\.
We obtain the reference performancesrefs\_\{\\mathrm\{ref\}\}by repeatedly evaluating the reference solution in the same execution environment and taking the median performance over at least five runs\. A candidate receives a reward of zero if the reference solution does not yield a valid performance, i\.e\.,sref≤1s\_\{\\mathrm\{ref\}\}\\leq 1\. Otherwise, the reward is computed as

Rperf=\{0,s≤sref,min⁡\(1,ln⁡\(s/sref\)ln⁡sref\),s\>sref\.R\_\{\\mathrm\{perf\}\}=\\begin\{cases\}0,&s\\leq s\_\{\\mathrm\{ref\}\},\\\\ \\displaystyle\\min\\\!\\left\(1,\\;\\frac\{\\ln\(s/s\_\{\\mathrm\{ref\}\}\)\}\{\\ln s\_\{\\mathrm\{ref\}\}\}\\right\),&s\>s\_\{\\mathrm\{ref\}\}\.\\end\{cases\}
This logarithmic normalization assigns a reward of zero when the candidate matches or underperforms the reference solution\. For candidates that outperform the reference, the reward increases logarithmically and reaches its maximum value of1\.01\.0whens≥sref2s\\geq s\_\{\\mathrm\{ref\}\}^\{2\}\.

#### Implementation Metric\.

For tasks whose primary objective is functional implementation, we use a binary metric:

Rimpl=\{1,if all conditions and test cases pass,0,otherwise\.R\_\{\\mathrm\{impl\}\}=\\begin\{cases\}1,&\\text\{if all conditions and test cases pass\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}
A candidate receives full credit only if it builds and imports successfully, passes every test case, respects all task\-specific forbidden\-edit constraints\.

#### Cheating Detection\.

To protect the integrity ofΦ\\Phi\-Bench, we employ two complementary proctoring mechanisms: rule\-based monitoring and agent\-based inspection\. The rule\-based monitor scans the complete solution trajectory and final code changes for predefined prohibited behaviors, such as searching GitHub for the original implementation, retrieving the corresponding patch or commit diff and recovering upstream code from published Python packages\. A dedicated proctor agent additionally inspects trajectories and submissions for detecting more complex hacking behaviors, such as hard\-coding expected outputs, skipping or bypassing correctness checks, and introducing branches that artificially inflate the measured performance\. If either proctoring mechanism detects a prohibited behavior, the corresponding task attempt receives a reward of zero\.

Table 2:Category\-wise performance onΦ\\Phi\-Bench\. Scores are reported as percentages, with the best result in each category highlighted in bold\. Models are ordered by their overall scores\.ModelTrainingInference& ServingCompressionKernelI/OHardware& EdgeDataInfrastructureSystemOptimizationSystemAssuranceFullClaude Opus 556\.4026\.2015\.7030\.0057\.103\.9049\.4034\.2032\.2036\.53Kimi K335\.5030\.5013\.2022\.2035\.703\.6027\.4041\.1039\.1028\.12Qwen3\.8 Max43\.9029\.908\.3023\.0018\.300\.1026\.5026\.8046\.3027\.73GPT 5\.6 Sol39\.9021\.803\.0017\.7038\.500\.0018\.4022\.8037\.8024\.51GLM 5\.228\.7019\.209\.0018\.6019\.101\.4027\.4022\.1057\.4021\.92Claude Sonnet 534\.007\.206\.6014\.1020\.600\.0023\.501\.5033\.7017\.58Qwen3\.7 Max27\.4012\.705\.9013\.2017\.405\.400\.006\.8032\.2016\.07DeepSeek V4Pro27\.803\.900\.5013\.103\.700\.900\.007\.0031\.7013\.31

## Evaluation

### Experimental Setup

#### Models & Scaffolds\.

We evaluate a range of state\-of\-the\-art proprietary and open\-weight models onΦ\\Phi\-Bench\. Proprietary models include claude\-opus\-5\([Anthropic 2026b](https://arxiv.org/html/2609.10226#bib.bib13)\), claude\-sonnet\-5\([Anthropic 2026c](https://arxiv.org/html/2609.10226#bib.bib23)\), gpt\-5\.6\-sol\([OpenAI 2026](https://arxiv.org/html/2609.10226#bib.bib14)\), qwen3\.8\-max\([Qwen Team 2026b](https://arxiv.org/html/2609.10226#bib.bib25)\), and qwen3\.7\-max\([Qwen Team 2026a](https://arxiv.org/html/2609.10226#bib.bib24)\)\. On the open\-weight side, we evaluate kimi\-k3\([Moonshot AI 2026](https://arxiv.org/html/2609.10226#bib.bib15)\), glm\-5\.2\([Z\.AI 2026](https://arxiv.org/html/2609.10226#bib.bib16)\)and deepseek\-v4\-pro\([DeepSeek\-AI 2026](https://arxiv.org/html/2609.10226#bib.bib17)\)\. To evaluate each model under its strongest available inference configuration, we enable the highest reasoning setting exposed by the corresponding model and allow the maximum supported context length\. We evaluate gpt\-5\.6\-sol with its default scaffold, Codex\([OpenAI 2025](https://arxiv.org/html/2609.10226#bib.bib18)\), and all other models with Claude Code\([Anthropic 2026a](https://arxiv.org/html/2609.10226#bib.bib19)\)scaffold\.

#### Implementation Details\.

We allocate 8 CPU cores, 32 GiB of memory, and a single NVIDIA H20 GPU to every task, while disabling all other GPU types to ensure a consistent evaluation environment\. We allow each agent to submit a single submission for each KFC task and at most 16 candidate solutions for each LHI and E2EO task\. We take the best submissions to compute the final score\. Unless otherwise specified, the models were evaluated with reasoning effort uniformly set to max\.

![Refer to caption](https://arxiv.org/html/2609.10226v1/ablation2.png)Figure 5:Performance over reasoning efforts on LHI tasks\.Figure 6:Performance curve of the best historical submission up to each round; cross marks indicate that the model has not yet produced a valid submission by the current round\.

### Main Results

Table[2](https://arxiv.org/html/2609.10226#Sx3.T2)presents the performance of leading frontier models across the nine major categories of our benchmark\. Claude Opus 5 achieves the highest overall score of36\.53%36\.53\\%, followed by Kimi K3 at28\.12%28\.12\\%and Qwen3\.8 Max at27\.73%27\.73\\%\. Despite having access to the complete repository and test cases, even the strongest model achieves only slightly more than one\-third of the maximum score, highlighting substantial headroom in real\-world LLM infrastructure engineering\.

Performance also varies considerably across infrastructure domains\. Claude Opus 5 leads in five of the nine categories, demonstrating relatively broad competence, whereas other models exhibit more specialized strengths: Kimi K3 performs best on Inference & Serving and System Optimization, Qwen3\.7 Max on Hardware & Edge, and GLM 5\.2 on System Assurance\. No model achieves uniformly strong performance across all categories\. The Hardware & Edge category appears to be particularly challenging for the models: even the best\-performing model achieved only 5\.4%\. This suggests that current frontier models still lack sufficient understanding of the hardware aspects of AI infrastructure and do not yet demonstrate strong engineering capabilities\.

### Ablation Studies

In this section, we investigate how model performance on LLM systems engineering tasks changes under different computational and interaction budgets\. Specifically, we study two questions: \(1\) whether models can progressively improve their solutions through iterative implementation, evaluation, and refinement; and \(2\) whether increasing the test\-time reasoning budget leads to better task performance\.

#### Performance over iterations\.

To evaluate the models’ ability to iteratively refine their solutions, we select a challenging E2EO task from LLM training infrastructure\. Given a fully editable nanoGPT training system, wall\-clock budget, and parameter\-count floor, the objective is to minimize validation bits per byte\. Standard optimizations for this setting include efficient token dispatch \(sorting, grouped batched GEMMs, per\-expert capacity buffers\), load balancing to prevent router collapse and token dropping, and routing configuration, and capacity factor — trading off quality, throughput, and drop rate\.

The results in figure[6](https://arxiv.org/html/2609.10226#Sx4.F6)show that most models exhibit certain ability to iteratively refine their solutions\. Claude Opus 5 achieves a very low BPB score on its very first submission, and is able to continuously improve this metric over subsequent iterations\. This result is highly impressive and clearly demonstrates its long\-horizon reasoning and task execution capabilities\. Although Qwen3\.8\-Max and Kimi K3 performed poorly in their early submissions, they are able to rapidly refine their approaches through iteration and ultimately achieved relatively low BPB scores\. In contrast, DeepSeek V4Pro, Qwen3\.7\-Max, GLM5\.2, and GPT5\.6 Sol exhibite prolonged plateau periods during the iterative process and ultimately failed to optimize BPB to a low level\.

#### Performance over reasoning budget\.

We evaluate three models including Claude Opus 5, Kimi K3, and GPT\-5\.6 Sol on 20 LHI problems across different reasoning effort levels, and plotted their scores as a function of effort level\. The results in Figure[5](https://arxiv.org/html/2609.10226#Sx4.F5)show that although all three models achieve their best performance at the max level, performance does not increase consistently with reasoning budget\. In particular, GPT\-5\.6 sol exhibits notable instability at intermediate effort levels\.

In addition, the performance gap between Claude Opus 5 and GPT\-5\.6 Sol across different budget settings is not particularly significant, whereas Kimi K3 loses about 45% of its score in the low setting compared with the max setting\. This indicates that Kimi K3’s strong performance depends heavily on substantial test\-time reasoning compute\.

## Analysis

Beyond the main evaluation, we collect detailed solution trajectories from different models and analyze them to identify their predominant error modes and detect potential cheating or hacking behaviors\. This trajectory\-level analysis provides more fine\-grained insights into current model limitations and informs future improvements to their capabilities\.

### Error Modes Analysis

We collect error messages that appeared during each model’s problem\-solving trajectories and categorize them into four major classes: Python runtime error, CUDA execution error, Triton / MLIR / CUDA compile error, and tensor shape mismatch\. Figure[7](https://arxiv.org/html/2609.10226#Sx5.F7)presents the number of errors and the distribution of error types across different models\.

Figure 7:Error mode breakdown in model trajectories\.Most high\-scoring models—Claude Opus 5, Kimi K3, and Qwen3\.8 Max—produce more errors\. This pattern suggests that they complete more challenging tasks through additional rounds of trial, diagnosis, and correction\. In contrast, the weaker\-performing models, including DeepSeek V4 Pro, Qwen3\.7\-Max, and Claude Sonnet 5, produced fewer total errors, suggesting that they tended to give up after a limited number of attempts or resort to simpler solutions\. This contrast indicates that the ability to persist through trial and error and to iteratively correct mistakes is one of the core capabilities required for LLMs to handle challenging AI infrastructure engineering tasks\.

In terms of error\-type distribution, Python runtime errors account for more than half of all errors for most models\. By contrast, Claude Opus 5 has a significantly lower proportion of this type of error than the other models, with a larger share of its errors concentrated in CUDA execution errors\. This suggests that Claude Opus 5 is markedly better at generating Python code that works correctly within the repository on the first attempt, allowing it to focus more of its attention on the more critical low\-level implementation and optimization issues\. This is likely one of the reasons why it achieves a leading advantage in AI infrastructure engineering\.

### Capacity over open\-endedness\.

Table[3](https://arxiv.org/html/2609.10226#Sx5.T3)compares model performance across three task categories with progressively increasing levels of open\-endedness\. Claude Opus 5 ranks first in every task format, achieving37\.16%37\.16\\%on KFC,21\.60%21\.60\\%on LHI, and62\.94%62\.94\\%on E2EO\. Kimi K3 ranks second overall and is particularly competitive on E2EO, while Qwen3\.8 Max ranks third overall with consistently strong performance across all three task formats\. Scores on LHI are consistently lower than those on KFC, indicating that long\-horizon repository\-level implementation remains challenging across all evaluated models\.

Table 3:Performance across the three task formats inΦ\\Phi\-Bench\. Scores are reported as percentages, with the best result in each column highlighted in bold\. Models are ordered by their overall scores\.ModelKFCLHIE2EOFullClaude Opus 537\.1621\.6062\.9436\.53Kimi K326\.0919\.5556\.4128\.12Qwen3\.8 Max28\.7916\.6144\.1027\.73GPT 5\.6 Sol25\.4613\.9840\.3324\.51GLM 5\.224\.3513\.6525\.1121\.92Claude Sonnet 518\.1413\.4622\.7417\.58Qwen3\.7 Max16\.4712\.7920\.4016\.07DeepSeek V4Pro16\.0511\.451\.9713\.31
### Hacking Prevention & Detection

In our experiments, we employed two approaches to minimize hacking behavior: \(i\) Soft network disconnection — we hijacked pip and URLs related to the test repositories, returning warning messages that block requesting such resources\. \(ii\) Prompting — we prompt the models with hacking behavior definition and explicitly prohibit such actions\.

Throughout all evaluation, the rule\-based cheating detector identified only three instances in which DeepSeek V4 Pro triggered requests to retrieve code from the PyTorch website\. The Proctor Agent confirmed that no actual hacking behavior occurred\. This demonstrates that our two anti\-hacking mechanisms effectively prevented cheating or hacking behavior during the evaluation process\.

## Case Study

Using the problem shown in Figure[6](https://arxiv.org/html/2609.10226#Sx4.F6), we examine the optimization trajectories of three representative models: Claude Opus 5, the best\-performing model on this task, and the substantially weaker Qwen3\.7 Max and DeepSeek V4Pro\. Figure[8](https://arxiv.org/html/2609.10226#Sx6.F8)shows their per\-round submission results\. We ask a central question:*what behaviors enable strong models to effectively conduct iterative optimization of LLM infrastructure?*We view these trajectories through two coupled capabilities: selecting promising optimization directions and executing local changes reliably\. The three observations below show how strong models connect these capabilities by establishing experimental foundations, controlling variables and noise, and cautiously attributing experimental outcomes\.

### Experimental Foundations

Observations\.Claude Opus 5 establishes lightweight local validation experiments to rapidly screen candidate hypotheses before formal submission\. These experiments reproduce only the relevant part of the workload and serve as inexpensive local checks\. For example, Opus compares a short local run against the development measurement and observes only a small BPB difference, confirming that the local experiment is sufficiently informative for preliminary screening\. It then uses such checks to reject unpromising changes before committing to full evaluation\.

These local checks form part of a coherent search strategy: Opus explores candidate directions broadly before focusing on the most promising ones\.

Qwen3\.7 Max and DeepSeek V4Pro, by contrast, largely follow an implementation–submission–observation loop without preliminary local validation\. Formal submissions therefore serve simultaneously as debugging, hypothesis testing, and evaluation, increasing the cost of unsuccessful hypotheses\. For example, DeepSeek appliestorch\.compileto expert modules without first validating checkpoint compatibility, eventually wasting a submission because the resulting checkpoint cannot be loaded\.

Takeaway\.Good models establish low\-cost mechanisms for hypothesis screening and validation, reducing the cost of each optimization iteration\.

Figure 8:Performance of current submission at each round\. Crosses denote submissions that failed the correctness check\.
### Variable and Noise Control

Observations\.DeepSeek frequently modifies multiple variables simultaneously, making experimental outcomes difficult to attribute\. Qwen3\.7 Max exhibits the opposite failure mode: although it performs more disciplined single\-variable tuning, many of its experiments produce differences comparable to measurement noise\. Variable isolation alone is therefore insufficient when the observed effect is not reliably distinguishable\.

Opus 5 explicitly considers both factors\. It estimates measurement variation, accounts for differences in completed optimization steps, and focuses on experiments whose effects are sufficiently large to interpret\. One representative transition changes the learning\-rate schedule and worsens BPB from 1\.3646 to 1\.3932\. Because the two runs use the same seed, complete the same number of steps, and have nearly identical runtime, Opus treats the degradation as reliable evidence rather than noise\. This leads to the hypothesis that excessive time at very low learning rates is harmful, motivating the warmup–stable–decay schedule, which improves BPB to 1\.3248\. Rather than chasing each immediate improvement, Opus uses these results to update a consistent optimization plan\.

Takeaway\.Good models conduct meaningful experiments and maximize the information gained from controlling variables and noise in each iteration\.

### Cautious Attribution

Observations\.Opus 5 also differs in how cautiously it interprets experimental outcomes\. Rather than directly attributing a performance change to its latest modification, it actively checks for alternative explanations and confounding factors\.

For example, one candidate modification initially appears to cause a substantial throughput regression\. Opus identifies that the code change invalidated the Inductor compilation cache, causing the run to incur additional cold\-compilation overhead\. It therefore repeats the comparison under matched cache conditions instead of rejecting the modification\. The corrected experiment shows essentially unchanged throughput while retaining a BPB improvement\. Similar cache\-related artifacts are detected multiple times in its trajectory, with Opus repeatedly delaying attribution until an appropriate control is available\. This protects its overall optimization plan from being redirected by a local measurement artifact\. In contrast, Qwen3\.7 Max and DeepSeek V4Pro more often interpret observed changes directly in terms of the latest intervention\.

Takeaway\.Good models rigorously test alternative explanations before attributing experimental outcomes, preventing measurement artifacts from becoming incorrect optimization conclusions\.

Overall, these trajectories suggest that long\-horizon infrastructure optimization requires more than reliable local execution\. Claude Opus 5 maintains a coherent optimization strategy across rounds: it explores broadly, uses controlled comparisons to identify promising directions, and updates its plan based on reliable evidence\. The weaker models more often make local, greedy decisions, making it difficult for useful knowledge to accumulate across iterations\.

## Conclusion

We introducedΦ\\Phi\-Bench, a benchmark of 85 tasks spanning nine LLM infrastructure domains and three increasingly open\-ended task formats\. Its taxonomy\-guided, agent\-assisted construction pipeline grounds task synthesis in systems research and public repositories while enabling coverage and scalability\. Evaluating eight frontier models reveals substantial but uneven capabilities: Claude Opus 5 leads with36\.53%36\.53\\%, yet no model performs consistently well\. On a challenging E2EO task, most models improve through iterative refinement; across 20 LHI tasks, greater reasoning effort does not yield consistent gains, although models perform best at the maximum effort level\. Trajectory analysis further reveals notable differences in problem\-solving capability and efficiency\. Overall,Φ\\Phi\-Bench provides a realistic testbed for tracking progress toward more reliable and broadly capable LLM infrastructure agents\.

## References

- Anthropic \(2026a\)AnthropicClaude code: overview\.Note:https://docs\.anthropic\.com/en/docs/claude\-code/overviewOfficial documentationCited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Anthropic \(2026b\)AnthropicIntroducing claude opus 5\.Note:https://www\.anthropic\.com/news/claude\-opus\-5Official model announcementCited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Anthropic \(2026c\)AnthropicIntroducing claude sonnet 5\.Note:https://www\.anthropic\.com/news/claude\-sonnet\-5Accessed: 2026\-07\-28Cited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. RéFlashAttention: fast and memory\-efficient exact attention with io\-awareness\.InAdvances in Neural Information Processing Systems,Cited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, and e\. al\. ZhangDeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\)\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p1.1)\.
- Ishibashiet al\.\(2025\)Y\. Ishibashi, T\. Yano, and M\. OyamadaCan large language models invent algorithms to improve themselves?: algorithm discovery for recursive self\-improvement through reinforcement learning\.External Links:2410\.15639,[Link](https://arxiv.org/abs/2410.15639)Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p1.1),[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p2.1)\.
- Liet al\.\(2025\)J\. Li, S\. Li, Z\. Gao, Q\. Shi, Y\. Li, Z\. Wang, J\. Huang, W\. WangHaojie, J\. Wang, X\. Han,et al\.Tritonbench: benchmarking large language model capabilities for generating triton operators\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 23053–23066\.Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p2.1),[Benchmarks for LLM Infrastructure Engineering](https://arxiv.org/html/2609.10226#Sx2.SSx2.p1.1)\.
- Liet al\.\(2026\)S\. Li, Z\. Zhang, G\. Sun, Y\. Luo, W\. Chen, Y\. Wang, M\. Hong, and C\. DingCUDAHercules: benchmarking hardware\-aware expert\-level cuda optimization for llms\.arXiv preprint arXiv:2605\.08467\.Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p2.1),[Benchmarks for LLM Infrastructure Engineering](https://arxiv.org/html/2609.10226#Sx2.SSx2.p1.1)\.
- Linet al\.\(2026\)E\. Lin, S\. Modi, S\. K\. S\. Hari, Q\. Huang, Z\. Ye, N\. Qin, F\. Zhou, Y\. Zhang, J\. Wang, S\. Damani,et al\.SOL\-execbench: speed\-of\-light benchmarking for real\-world gpu kernels against hardware limits\.arXiv preprint arXiv:2603\.19173\.Cited by:[Benchmarks for LLM Infrastructure Engineering](https://arxiv.org/html/2609.10226#Sx2.SSx2.p1.1)\.
- Maet al\.\(2026\)Z\. Ma, G\. Wang, X\. Xie, Y\. Chen, H\. Du, B\. Li, Y\. Sun, W\. Liu, K\. Chen, and Y\. LiTREX: automating llm fine\-tuning via agent\-driven tree\-based exploration\.External Links:2604\.14116,[Link](https://arxiv.org/abs/2604.14116)Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p1.1)\.
- Moonshot AI \(2026\)Moonshot AIKimi K3: open frontier intelligence\.Note:https://www\.kimi\.com/blog/kimi\-k3Official technical blogCited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Nangiaet al\.\(2026\)A\. Nangia, S\. Mishra, A\. Gokrani, and P\. ChopraISO\-bench: can coding agents optimize real\-world inference workloads?\.arXiv preprint arXiv:2602\.19594\.Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p2.1),[Benchmarks for LLM Infrastructure Engineering](https://arxiv.org/html/2609.10226#Sx2.SSx2.p1.1)\.
- Narayananet al\.\(2021\)D\. Narayanan, M\. Shoeybi, J\. Casper, P\. LeGresley, M\. Patwary, V\. A\. Korthikanti, D\. Vainbrand, P\. Kashinkunti, J\. Bernauer, B\. Catanzaro, A\. Phanishayee, and M\. ZahariaEfficient large\-scale language model training on GPU clusters using Megatron\-LM\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,SC ’21\.External Links:[Document](https://dx.doi.org/10.1145/3458817.3476209)Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p1.1)\.
- NVIDIA \(2017\)NVIDIACUTLASS: cuda templates for linear algebra subroutines\.Note:https://github\.com/NVIDIA/cutlassCited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p1.1)\.
- OpenAI \(2025\)OpenAIIntroducing codex\.Note:https://openai\.com/index/introducing\-codex/Official product announcementCited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- OpenAI \(2026\)OpenAIGPT\-5\.6: frontier intelligence that scales with your ambition\.Note:https://openai\.com/index/gpt\-5\-6/Official model announcementCited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Ouyanget al\.\(2025\)A\. Ouyang, S\. Guo, S\. Arora, A\. L\. Zhang, W\. Hu, C\. Ré, and A\. MirhoseiniKernelbench: can llms write efficient gpu kernels?\.arXiv preprint arXiv:2502\.10517\.Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p2.1),[Benchmarks for LLM Infrastructure Engineering](https://arxiv.org/html/2609.10226#Sx2.SSx2.p1.1)\.
- Qwen Team \(2026a\)Qwen TeamQwen3\.7: the agent frontier\.Note:https://qwen\.ai/blog?id=qwen3\.7Accessed: 2026\-07\-28Cited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Qwen Team \(2026b\)Qwen TeamQwen3\.8\-max: a new bar for coding and cowork\.Note:https://qwen\.ai/blog?id=qwen3\.8Accessed: 2026\-08\-09Cited by:[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Rasleyet al\.\(2020\)J\. Rasley, S\. Rajbhandari, O\. Ruwase, and Y\. HeDeepspeed: system optimizations enable training deep learning models with over 100 billion parameters\.InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,pp\. 3505–3506\.Cited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p2.1)\.
- Shoeybiet al\.\(2019\)M\. Shoeybi, M\. Patwary, R\. Puri, P\. LeGresley, J\. Casper, and B\. CatanzaroMegatron\-lm: training multi\-billion parameter language models using model parallelism\.arXiv preprint arXiv:1909\.08053\.Cited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p2.1)\.
- Tilletet al\.\(2019\)P\. Tillet, H\. Kung, and D\. CoxTriton: an intermediate language and compiler for tiled neural network computations\.InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages,pp\. 10–19\.Cited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p1.1)\.
- Wanget al\.\(2026\)H\. Wang, J\. Zhang, K\. Jiang, H\. Wang, J\. Chen, and J\. ZhuKernelbenchx: a comprehensive benchmark for evaluating llm\-generated gpu kernels\.arXiv preprint arXiv:2605\.04956\.Cited by:[Benchmarks for LLM Infrastructure Engineering](https://arxiv.org/html/2609.10226#Sx2.SSx2.p1.1)\.
- Xinget al\.\(2026\)S\. Xing, Y\. Zhai, A\. Jiang, Y\. Dong, Y\. Wu, Z\. Ye, C\. Ruan, Y\. Huang, Y\. Zhang, L\. Yin,et al\.FlashInfer\-bench: building the virtuous cycle for ai\-driven llm systems\.arXiv preprint arXiv:2601\.00227\.Cited by:[Benchmarks for LLM Infrastructure Engineering](https://arxiv.org/html/2609.10226#Sx2.SSx2.p1.1)\.
- Yeet al\.\(2025\)Z\. Ye, L\. Chen, R\. Lai, W\. Lin, Y\. Zhang, S\. Wang, T\. Chen, B\. Kasikci, V\. Grover, A\. Krishnamurthy, and L\. CezeFlashInfer: efficient and customizable attention engine for llm inference serving\.InProceedings of Machine Learning and Systems,Cited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p1.1)\.
- Yuet al\.\(2022\)G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. ChunOrca: a distributed serving system for\{\\\{transformer\-based\}\\\}generative models\.In16th USENIX symposium on operating systems design and implementation \(OSDI 22\),pp\. 521–538\.Cited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p2.1)\.
- Z\.AI \(2026\)Z\.AIGLM\-5: from vibe coding to agentic engineering\.External Links:2602\.15763,[Link](https://arxiv.org/abs/2602.15763)Cited by:[Introduction](https://arxiv.org/html/2609.10226#Sx1.p1.1),[Models & Scaffolds\.](https://arxiv.org/html/2609.10226#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2024\)L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez,et al\.Sglang: efficient execution of structured language model programs\.Advances in neural information processing systems37,pp\. 62557–62583\.Cited by:[LLM Infrastructure Optimization](https://arxiv.org/html/2609.10226#Sx2.SSx1.p2.1)\.