PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language

# PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language
Source: [https://arxiv.org/html/2607.09789](https://arxiv.org/html/2607.09789)
###### Abstract

We introduce PHITSBench, an execution\-scored benchmark for the Monte Carlo Particle and Heavy Ion Transport code System \(PHITS\)\. PHITSBench comprises 282 transport\-scorable tasks spanning three common workflow categories: parameter editing \(*Edit*\), syntax repair \(*Repair*\), and complete simulation generation from natural\-language descriptions \(*Reproduce*\)\. Each task is evaluated using a Composite Metric Score that combines execution success with agreement between generated and reference transport observables\. Using PHITSBench, we evaluate five GPT\-5\.4\-based configurations ranging from zero\-shot prompting to knowledge\-augmented and agentic workflows\. Without domain\-specific knowledge, the model performs well on editing and repair tasks \(95% and 70% success, respectively\) but fails to generate correct simulations from scratch \(0% success on the*Reproduce*track\)\. A structured, machine\-readable PHITS knowledge catalog, supplied alongside the user manual, raises single\-shot*Reproduce*\-task success to 57%\. Agentic execution provides a further improvement to 66\-73%, but at increased computational cost\. Failure analysis shows that the remaining errors are dominated by incorrect selection and configuration of physical observables rather than syntax generation\. These results suggest that future progress in AI\-assisted radiation\-transport modeling will depend as much on machine\-readable knowledge bases, curated domain\-training datasets, and execution\-grounded evaluation environments as on advances in foundation models themselves\.

###### keywords:

PHITS; Monte Carlo radiation transport; large language models; benchmark; input file generation; agentic systems

††articletype:Article## 1Introduction

Accurate modeling of radiation transport is essential for nuclear energy and fusion engineering, particle accelerator operation, medical physics, outer space exploration, and advanced manufacturing, which often require analysis and design of systems operating in radiation\-rich environments\. Monte Carlo \(MC\) radiation transport codes such as PHITS\[[33](https://arxiv.org/html/2607.09789#bib.bib1)\], MCNP\[[40](https://arxiv.org/html/2607.09789#bib.bib7)\], GEANT4\[[2](https://arxiv.org/html/2607.09789#bib.bib8)\], and FLUKA\[[12](https://arxiv.org/html/2607.09789#bib.bib9)\]are state\-of\-the\-art tools for predicting how neutrons, photons, electrons, and ions interact with materials in arbitrary three\-dimensional \(3D\) geometries\. These platforms encode decades of validated physics, but their simulation setup, execution, and validation of results remain challenging even for experienced designers\. Constructing a valid simulation requires extensive domain expertise in radiation physics, geometry modeling, materials, source characterization, and code\-specific input languages\. In practice, a significant fraction of the effort associated with a radiation transport study lies not in executing simulations, but in constructing, debugging, validating, and maintaining complex simulation input decks\. This creates a substantial barrier to productivity, limits rapid exploration of large design spaces, and restricts access to advanced modeling capabilities outside a relatively small community of experts\.

![Refer to caption](https://arxiv.org/html/2607.09789v1/x1.png)Figure 1:PHITS at a glance\. \(a\) A PHITS input file consists of a sequence of named keyword sections defining simulation parameters, particle sources, materials, geometry, and tally requests\. \(b\) During execution, particles are emitted from the specified source, transported through the geometry, and scored according to user\-defined\[T\-\*\]tally definitions\. In the example shown, a neutron source emits neutrons that propagate through the geometry and interact with the material region M1 \(orange\), producing secondary particles at the interaction points \(orange dots\); the simulation returns the neutron track\-length flux scored by a\[T\-Track\]tally\. \(c\) The same radiation\-transport framework underpins diverse application areas; the panels show three representative PHITS simulations from this study—medical physics \(a proton beam in water\), shielding and facility design \(a photon beam in lead\), and radiation detection and spectroscopy \(a Cs\-137 source and a NaI\(Tl\) detector\)\.Recent advances in large language models \(LLMs\) and agentic AI systems have created new opportunities to automate scientific computing workflows\. LLMs now achieve excellent performance on many conventional code\-generation benchmarks and increasingly demonstrate utility in engineering design, scientific programming, and computational modeling\[[9](https://arxiv.org/html/2607.09789#bib.bib10),[19](https://arxiv.org/html/2607.09789#bib.bib12),[21](https://arxiv.org/html/2607.09789#bib.bib13),[22](https://arxiv.org/html/2607.09789#bib.bib14),[46](https://arxiv.org/html/2607.09789#bib.bib16),[39](https://arxiv.org/html/2607.09789#bib.bib15)\]\. A growing body of work has begun to extend these capabilities to scientific simulation environments\. In radiation transport, AutoFLUKA demonstrated AI\-assisted generation of MC simulations through domain\-knowledge\-augmented agents\[[24](https://arxiv.org/html/2607.09789#bib.bib18)\]\. In finite\-element analysis, MCP\-SIM introduced a self\-correcting multi\-agent framework for FEniCS\-based simulations\[[29](https://arxiv.org/html/2607.09789#bib.bib21)\], while Shafiq et al\. evaluated LLM performance on Gmsh and Elmer workflows for geometry generation and solver setup\[[35](https://arxiv.org/html/2607.09789#bib.bib22)\]\. In computational fluid dynamics, OpenFOAMGPT employed retrieval\-augmented generation for OpenFOAM\-based simulations\[[28](https://arxiv.org/html/2607.09789#bib.bib23)\], and MetaOpenFOAM developed a specialized multi\-agent framework for CFD workflows\[[10](https://arxiv.org/html/2607.09789#bib.bib24)\], motivating the creation of the CFDLLMBench benchmark suite\[[37](https://arxiv.org/html/2607.09789#bib.bib25)\]\. Similar efforts have recently emerged for general multiphysics simulation, including MooseAgent for MOOSE\-based workflows\[[44](https://arxiv.org/html/2607.09789#bib.bib40)\], and for nuclear engineering applications, including AutoSAM for generation of input files for the SAM reactor thermal\-hydraulic code\[[1](https://arxiv.org/html/2607.09789#bib.bib41)\]\. Related benchmark efforts have also appeared in adjacent simulation domains, including network simulation\[[3](https://arxiv.org/html/2607.09789#bib.bib42)\], molecular dynamics\[[15](https://arxiv.org/html/2607.09789#bib.bib43)\], particle\-physics experiment design and simulation\[[14](https://arxiv.org/html/2607.09789#bib.bib19)\], and nuclear energy applications\[[4](https://arxiv.org/html/2607.09789#bib.bib20)\]\.

LLM performance for scientific computing is often limited by access to domain\-specific knowledge rather than by reasoning capability\. To address this challenge, recent studies have shown that retrieval\-augmented generation \(RAG\) and direct documentation injection can substantially improve code generation and simulation setup by providing structured access to technical information that is either absent from pretraining data or difficult to retrieve from long documents\[[28](https://arxiv.org/html/2607.09789#bib.bib23),[45](https://arxiv.org/html/2607.09789#bib.bib38),[30](https://arxiv.org/html/2607.09789#bib.bib39)\]\. In parallel, agentic workflows have emerged as a complementary strategy when static knowledge alone is insufficient\. Modern multi\-agent systems decompose complex tasks into specialized roles, including planning, execution, verification, and repair, as demonstrated by frameworks such as MetaGPT\[[16](https://arxiv.org/html/2607.09789#bib.bib34)\], AutoGen\[[41](https://arxiv.org/html/2607.09789#bib.bib35)\], SciAgents\[[13](https://arxiv.org/html/2607.09789#bib.bib26)\], and SWE\-agent\[[42](https://arxiv.org/html/2607.09789#bib.bib33)\]\. These systems build upon iterative reasoning and self\-correction paradigms such as ReAct\[[43](https://arxiv.org/html/2607.09789#bib.bib27)\], Self\-Refine\[[23](https://arxiv.org/html/2607.09789#bib.bib29)\], and Reflexion\[[36](https://arxiv.org/html/2607.09789#bib.bib28)\], enabling models to incorporate execution feedback and progressively improve generated outputs\. However, other studies suggest that self\-correction is not a universal solution\. Huang et al\.\[[17](https://arxiv.org/html/2607.09789#bib.bib30)\]and Olausson et al\.\[[25](https://arxiv.org/html/2607.09789#bib.bib31)\]showed that the effectiveness of self\-repair is fundamentally bounded by the quality and informativeness of external feedback, while Kamoi et al\.\[[20](https://arxiv.org/html/2607.09789#bib.bib32)\]provided a comprehensive survey of the limitations and opportunities of self\-correcting LLM systems\.

Overall, there is an emerging consensus in the scientific computing community that LLMs can assist with simulation setup and execution, particularly when augmented with retrieval mechanisms, structured documentation, and iterative self\-correction\. However, existing efforts share several important limitations\. Most target software ecosystems are extensively documented online, are represented in public training corpora, and rely on modern scripting languages and APIs\. Thus, it remains difficult to determine whether the observed performance reflects genuine reasoning about physical systems or the retrieval of previously seen examples\[[18](https://arxiv.org/html/2607.09789#bib.bib17)\]\. Furthermore, many existing benchmarks focus primarily on syntactic correctness, while placing less emphasis on the physical validity of results\.

Radiation transport simulations provide a compelling testbed for evaluating AI\-assisted scientific computing\. Successful generation of a valid simulation requires simultaneous reasoning about geometry, materials, source definitions, physics models, tally specifications, and code\-specific syntax\. Small errors may lead to failed execution, silently incorrect outputs, or physically invalid results\. As a result, radiation transport workflows offer a realistic measure of whether AI systems can bridge the gap between natural\-language scientific intent and executable, physically meaningful simulations\.

The Particle and Heavy Ion Transport code System \(PHITS\) presents an especially challenging and informative case study\. PHITS is one of the most widely used general\-purpose MC packages, which is routinely applied to reactor shielding analysis, accelerator design, detector modeling, proton therapy, medical physics, and space\-radiation assessment\. Unlike many modern software environments, PHITS relies on a specialized input language derived from legacy scientific computing workflows; its bracketed sections with column\-sensitive lines and ad\-hoc subsection grammar resemble 1980s Fortran\-card\-deck input\. Furthermore, the code, documentation, and example libraries are not broadly available on the public web, making PHITS underrepresented in the data used to train contemporary foundation models\.

Importantly, PHITS input generation requires more than code synthesis\. A valid simulation combines multiple interdependent components, including geometry definitions, material specifications, source descriptions, transport parameters, and tally requests\. Input is specified in a free\-format text file consisting of named keyword*sections*, including \[Parameters\], \[Source\], \[Material\], \[Surface\], \[Cell\], \[T\-Track\], \[T\-Deposit\], etc\. \(Fig\.[1](https://arxiv.org/html/2607.09789#S1.F1)\)\. Although the language is relatively permissive in formatting, it is unforgiving in physics\. A geometry whose outer void fails to enclose the simulation region can trigger a fatal error; a tally configured with an incorrect energy\-bin definition may execute successfully while producing physically meaningless results; and keywords that appear similar across sections may carry different numerical meanings\. Consequently, generating PHITS inputs requires simultaneous understanding of both radiation physics and parser\-specific constraints\.

To systematically evaluate the capability of LLMs to generate such inputs, we introduce PHITSBench, an execution\-scored benchmark for AI\-assisted PHITS input generation\. PHITSBench contains 282 transport\-scorable tasks spanning three categories that mirror common PHITS workflows: parameter editing, syntax repair, and complete simulation reconstruction from natural\-language descriptions\. Unlike benchmarks scored by text\-similarity metrics such as CodeBLEU\[[31](https://arxiv.org/html/2607.09789#bib.bib11)\], PHITSBench evaluates generated inputs by executing PHITS simulations and comparing the resulting transport observables against reference solutions\. This execution\-linked framework enables direct assessment of both syntactic correctness and physical fidelity\.

Using PHITSBench, we investigate three central questions\. First, to what extent can LLMs generate valid radiation transport simulations without domain\-specific knowledge? Second, how much performance can be recovered through structured knowledge injection and retrieval mechanisms? Third, can specialized agentic architectures outperform generic coding agents in the generation and repair of complex scientific simulations? The answers provide insight not only into the future of AI\-assisted radiation transport modeling, but more broadly into the role of agentic AI systems in scientific computing, nuclear engineering, and physics\-based design\.

## 2Materials and methods

### 2\.1Benchmark design

PHITSBench was developed to evaluate AI\-assisted generation of radiation transport simulations across the spectrum of common PHITS user workflows\. The benchmark contains 282 tasks derived from the official PHITS 3\.34 example library, which are organized into three task categories:*Edit, Repair*and*Reproduce*\(Fig\.[2](https://arxiv.org/html/2607.09789#S2.F2)\)\.

![Refer to caption](https://arxiv.org/html/2607.09789v1/x2.png)Figure 2:Representative examples of tasks in each benchmark track\.*Edit*\(left\): the model receives a valid PHITS input file together with a natural\-language modification request and must generate an updated input whose transport results match the modified reference simulation\.*Repair*\(center\): the model receives a deliberately corrupted PHITS input file and the corresponding PHITS error message and must generate a corrected simulation input\.*Reproduce*\(right\): the model receives only a natural\-language description of the physical problem and must generate a complete PHITS input file from scratch\.The*Edit*track includes 65 tasks \(n=65n=65\) and evaluates controlled modification of existing reference simulations\. A model receives a valid PHITS input file together with a natural\-language instruction, which requests one or more changes to the simulation setup, such as modifying source energy, material composition, geometry dimensions, or tally parameters\. Hand\-crafted modification instructions range from a single parameter change to composite edits touching source energy, material composition, and geometry simultaneously\. The generated output is expected to reproduce the behavior of a reference simulation, in which the same modifications were applied manually\. This task reflects routine parameter studies and design\-space exploration commonly performed by PHITS users\.

The*Repair*track \(n=131n=131\) evaluates error diagnosis and correction\. Each task contains a deliberately corrupted PHITS input file together with the corresponding system\-generated error message\. Corrupted input files include a single fault at a randomly chosen valid location—a misspelled keyword, a missing required parameter, a corrupted or duplicated section header, or a malformed numeric format—and record the generated PHITS error message\. The full taxonomy of edit modifications and repair faults, with task counts per category, is given in Table[2\.1](https://arxiv.org/html/2607.09789#S2.SS1)\. The model must identify the source of the error and generate a corrected simulation input\. These tasks represent common debugging activities encountered during simulation development and validation\.

The*Reproduce*track \(n=86n=86\) is the most challenging and is the primary focus of this work\. In this setting, the model receives only a natural\-language description of the physical problem, including geometry, materials, source conditions, and desired outputs\. The model must generate a complete PHITS input file from scratch\. Successful completion therefore requires simultaneous reasoning about radiation physics, geometry construction, tally selection, and PHITS syntax\.

\\tbl

Taxonomy of the*Edit*\(modification\) and*Repair*\(fault\) tracks\. Tasks are grouped into thematic categories, and the rightmost column summarizes the corresponding modification or fault represented by each category\. Counts \(nn\) correspond to the transport\-scorable subset used throughout the benchmark\.CategorynnSub\-typesEdit track\(n=65n=65\)Source16change the beam energy and its energy rangeMaterial20replace a material or change its densityGeometry5resize or reshape the geometryTally / mesh1change the resolution of the scoring meshComposite \(≥2\\geq 2edits\)23two or three of the above edits applied togetherRepair track\(n=131n=131\)Section structure25corrupted, duplicated, or missing section headersSource keywords32invalid, misspelled, or missing source parametersMaterial18malformed or missing material, density, or isotope entriesCell / surface21bad cell or region syntax, or an undefined surfaceTally keywords35unknown or missing tally settings, such as the output file, an axis, the mesh type, or the energy bins

To ensure that benchmark evaluation reflects physically meaningful transport simulations, PHITSBench is restricted to tasks that execute in transport mode \(icntl = 0\) and generate tally outputs suitable for quantitative comparison\. Non\-transport control modes like all\-vacuum mode, 2D geometry plotting mode, and source checking mode were excluded because they do not produce transport observables that can be evaluated consistently\. Every task was generated and reviewed by a domain expert\. Tasks containing prompt\-reference mismatches, spurious whitespace differences, or overly complex geometries were corrected or removed during the revision process\.

### 2\.2Composite Metric Score

Evaluating AI\-generated scientific simulations requires assessing both*syntactic validity*and*physical correctness*\. A generated PHITS input may execute successfully while producing physically incorrect outputs, whereas an otherwise valid transport model may fail because of a minor syntax or formatting error\. To capture both aspects of performance, PHITSBench employs an execution\-grounded metric termed the*Composite Metric Score*\(CMS\)\. The CMS combines an*execution score*,𝐸𝑋\\mathit\{EX\}, and a*physics\-fidelity score*,𝑃𝐹\\mathit\{PF\}:

𝐶𝑀𝑆=w𝐸𝑋​𝐸𝑋\+w𝑃𝐹​𝑃𝐹,\\mathit\{CMS\}=w\_\{\\mathit\{EX\}\}\\mathit\{EX\}\+w\_\{\\mathit\{PF\}\}\\mathit\{PF\},\(1\)where𝐸𝑋∈\{0,1\}\\mathit\{EX\}\\in\\\{0,1\\\},𝑃𝐹∈\[0,1\]\\mathit\{PF\}\\in\[0,1\], andw𝐸𝑋w\_\{\\mathit\{EX\}\},w𝑃𝐹w\_\{\\mathit\{PF\}\}are weighting coefficients\.

The execution score evaluates whether the generated input produces a valid PHITS simulation\. A task receives𝐸𝑋=1\\mathit\{EX\}=1only if PHITS terminates successfully, reports no\*\* Erroror\*\*\* ERRORentries inphits\.out, and generates all expected output files\. Otherwise,𝐸𝑋=0\\mathit\{EX\}=0\. The physics\-fidelity score quantifies agreement between the generated and reference transport solutions\. For each expected tally output, the corresponding PHITS results are parsed and converted into one\-dimensional numerical arrays\. Similarity is then evaluated using three complementary measures: \(i\) normalizedL2L^\{2\}distance, \(ii\) agreement in total integral magnitude, and \(iii\) agreement in peak position\. These metrics capture distribution shape, overall normalization, and key physical features, respectively\.

Because different tally outputs contain different amounts of information, the relative weighting of these three components depends on tally resolution\. High\-resolution distributions provide sufficient information to evaluate shape, whereas low\-resolution or integral quantities do not\. The weighting scheme used throughout the benchmark is summarized in Table[2\.2](https://arxiv.org/html/2607.09789#S2.SS2)\. Component weights were selected*a priori*based on the information content of each tally type and were not tuned using benchmark results\. For a given task, tally\-level fidelity scores are averaged across all expected outputs\. If a simulation fails to execute \(𝐸𝑋=0\\mathit\{EX\}=0\), the corresponding physics\-fidelity score is defined as𝑃𝐹=0\\mathit\{PF\}=0\.

\\tbl

Physics\-fidelity weighting scheme used in PHITSBench\.

Different benchmark tracks place different emphasis on execution success and physics fidelity\. For the*Reproduce*track, generating a runnable simulation from a natural\-language description is itself a substantial achievement\. We therefore assign equal importance to execution and physics fidelity:\(w𝐸𝑋,w𝑃𝐹\)=\(0\.5,0\.5\)\(w\_\{\\mathit\{EX\}\},w\_\{\\mathit\{PF\}\}\)=\(0\.5,0\.5\)\. For the*Edit*and*Repair*tracks, where the starting point is already a valid or nearly valid simulation, physics fidelity provides the primary measure of task completion\. Accordingly, we use\(w𝐸𝑋,w𝑃𝐹\)=\(0\.2,0\.8\)\(w\_\{\\mathit\{EX\}\},w\_\{\\mathit\{PF\}\}\)=\(0\.2,0\.8\)for*Edit*and\(0\.3,0\.7\)\(0\.3,0\.7\)for*Repair*\.

Throughout this work, a task is considered successfully solved when𝐶𝑀𝑆≥0\.95\\mathit\{CMS\}\\geq 0\.95\. This threshold corresponds to a simulation that both executes correctly and reproduces the reference transport observables with high fidelity\. Sensitivity analysis showed that perturbing the track\-specific weights by±0\.10\\pm 0\.10does not alter the qualitative ranking of methods evaluated in this study, indicating that the reported conclusions are robust to reasonable variations in scoring parameters\.

### 2\.3Knowledge Engineering for PHITS Simulation

A central objective of this work is to evaluate how different forms of domain knowledge influence AI\-assisted generation of PHITS simulations\. To this end, we constructed a machine\-readable PHITS knowledge catalog and compared multiple knowledge\-injection strategies ranging from no external knowledge to structured domain\-specific guidance\. The experiments reported in this work use three sources of PHITS\-related knowledge:

1. 1\.Model pretraining knowledge, corresponding to whatever PHITS\-related information is already present in the foundation model \(Baseline\)\.
2. 2\.The PHITS user manual in the \.pdf format, which provides comprehensive documentation of simulation syntax, physics models, and example workflows \(PDF\)\.
3. 3\.A structured PHITS knowledge catalog custom\-designed for this study \(Cat\)\.

The PHITS knowledge catalog was designed to provide a compact, machine\-readable representation of information required for simulation generation and validation\. Unlike the PHITS manual, which is optimized for human users, the catalog organizes syntax rules, parameter definitions, and parser constraints in a format intended for direct use by LLMs\. The catalog was generated through LLM\-assisted parsing of PHITS source code and subsequently reviewed, corrected, and expanded by a domain expert\. Parser routines responsible for reading and validating PHITS input files were analyzed to recover valid keyword names, accepted value ranges, default values, and inter\-keyword dependencies\. These rules were then cross\-referenced against the PHITS user manual and example library to identify undocumented behaviors and common failure modes\.

The resulting catalog consists of eleven Markdown documents comprising approximately 7,300 lines of text and approximately 69,000 tokens\. Each rule is linked to the relevant PHITS input section and, where possible, annotated with source\-code provenance for traceability\. Importantly, the catalog captures information that is difficult to infer from documentation alone, including parser\-specific defaults, context\-dependent keyword behavior, and cross\-section constraints that frequently cause simulation failures\.

### 2\.4Evaluation methods

To quantify the impact of domain knowledge on simulation generation, we evaluate three progressively richer knowledge configurations\.

- •TheBaselineconfiguration provides no PHITS\-specific information beyond the task description\. Performance under this condition reflects the model’s existing knowledge acquired during pretraining\.
- •The\+PDFconfiguration augments the model with access to the complete PHITS user manual\. This setup represents a documentation\-assisted workflow analogous to retrieval\-augmented generation approaches used in other scientific computing domains\.
- •The\+Cat\+PDFconfiguration supplements the manual with the structured knowledge catalog\. The catalog is provided directly within the model context, while the manual remains available as a reference source\. This combination allows the model to access both detailed documentation and a compact representation of parser\-level constraints\.
- •TheAgenticmethods use the same knowledge sources as the\+Cat\+PDFconfiguration, but additionally exploit execution feedback and iterative repair mechanisms\.

All methods received identical benchmark tasks and used the same underlying foundational model\. The primary differences between configurations were \(i\) the amount of PHITS\-specific knowledge available to the model and \(ii\) whether the model could execute PHITS and iteratively refine its output\.

The first three configurations perform a single inference pass and differ only in the knowledge available to the model\. TheBaselineconfiguration receives only the benchmark task description and therefore reflects whatever PHITS\-related knowledge is already present in the model’s pretraining corpus\. The\+PDFconfiguration augments the task description with the complete PHITS user manual supplied through the Responses API\. The manual remains available throughout the inference process but no execution feedback is provided\. The\+Cat\+PDFconfiguration additionally supplies the structured PHITS knowledge catalog\. The catalog is included directly within the model context, while the full manual remains available as a reference document\.

Agenticconfigurations augment knowledge injection with execution\-driven refinement\. The first agentic baseline usesCodex CLI\[[26](https://arxiv.org/html/2607.09789#bib.bib37)\], a general\-purpose coding agent capable of executing arbitrary commands and iteratively modifying generated outputs\. The agent is provided with the same PHITS manual and knowledge catalog, together with access to a local PHITS installation\. The second configuration is aCustom Multi\-Agentarchitecture, which we designed specifically for scientific simulation generation\. The multi\-agent workflow consists of three role\-specialized agents:

- •Composer: generates an initial PHITS input deck;
- •Manager: orchestrates execution, verification, and repair;
- •Repairer: modifies candidate inputs based on execution feedback\.

TheManagerimplements a ReAct\-style control loop\[[43](https://arxiv.org/html/2607.09789#bib.bib27)\], and coordinates a fixed tool set consisting of:

- •`compose\_phits\_input`
- •`static\_check\_phits`
- •`run\_phits`
- •`summarize\_failure`
- •`repair\_phits\_input`

Two additional verification tools provide domain\-specific validation:

- •`inspect\_tally\_outputs`, which identifies empty or physically implausible tally results;
- •`check\_tally\_config`, which compares generated tally configurations against task requirements\.

The workflow proceeds through iterative cycles of generation, validation, execution, diagnosis, and repair until either a valid solution is obtained or a maximum of five repair iterations is reached\. TheRepaireradditionally has retrieval access to a vector database containing both the PHITS manual and the structured catalog\. The implementation uses the OpenAI Agents SDK \(v0\.8\.4\)\[[27](https://arxiv.org/html/2607.09789#bib.bib36)\]\.

All experiments used OpenAI GPT\-5\.4 accessed through the Responses API \(April–May 2026 release\)\. Single\-shot methods employed medium reasoning effort and a maximum output length of 16,384 tokens\. No custom sampling parameters were specified\. All reported results correspond to single, non\-deterministic runs\. The agentic baselines used the same underlying model\. Codex CLI \(v0\.125\.0\) operated with xhigh reasoning effort and unrestricted iteration\. In ourCustom Multi\-Agentsystem, theManagerandComposeragents used medium reasoning effort, while theRepaireragent used xhigh reasoning\.

Inference cost was estimated from logged token usage using official GPT\-5\.4 pricing\. For the single\-shot methods, most of the manual and catalog content is prefix\-cached after the first request, reducing subsequent costs\. TheBaselineconfiguration costs approximately $0\.01 per task, while\+PDFand\+Cat\+PDFcost approximately $0\.18 and $0\.20 per task, respectively\. The unrestrictedCodex CLIworkflow incurs substantially higher cost because the agent repeatedly accesses documentation and performs multiple repair cycles\. Average cost was approximately $0\.82 per task\. The boundedCustom Multi\-Agentarchitecture reduces this cost to approximately $0\.13 per task through limited repair iterations, selective use of high\-reasoning agents, and retrieval\-based access to documentation rather than repeated inclusion in context\.

## 3Results

Table[3](https://arxiv.org/html/2607.09789#S3)summarizes benchmark performance across all evaluated methods\. We first examine the*Edit*and*Repair*tracks, which approach saturation under single\-shot prompting, before turning to the more challenging*Reproduce*track and the effects of knowledge injection and agentic execution\. The success rate is defined as the fraction of tasks executed with𝐶𝑀𝑆≥0\.95\\mathit\{CMS\}\\geq 0\.95\.

\\tbl

Performance of the evaluated methods across the three PHITSBench tracks\. Entries show the number of tasks achievingC​M​S≥0\.95CMS\\geq 0\.95out of the corresponding transport\-scorable subset \(nnshown in the column headers\)\. “n/e” denotes configurations that were not evaluated \(see Section[2\.4](https://arxiv.org/html/2607.09789#S2.SS4)\)\.

### 3\.1Edit and Repair: single\-shot prompting is sufficient

Figure[3](https://arxiv.org/html/2607.09789#S3.F3)shows the success rate of the three non\-agentic methods across all benchmark tracks\. The*Edit*and*Repair*tracks provide the model with substantial contextual information in the form of an existing PHITS input file\. Consequently, these tasks primarily evaluate localized modification and debugging rather than full simulation synthesis\.

For the*Edit*track, performance is effectively saturated even without PHITS\-specific knowledge\. TheBaselineand\+Cat\+PDFconfigurations both solve 62 of 65 tasks \(95%\), while the\+PDFconfiguration solves 60 of 65 tasks \(92%\)\. The small differences between methods suggest that modifying an existing PHITS input file is largely tractable for a frontier model once a valid simulation structure is already present\. Because both*Edit*and*Repair*approaches saturate under single\-shot prompting, agentic methods were not evaluated on these tracks\.

![Refer to caption](https://arxiv.org/html/2607.09789v1/x3.png)Figure 3:Single\-shot success rate\(𝐶𝑀𝑆≥0\.95\\mathit\{CMS\}\\geq 0\.95\) across all three benchmark tracks for the non\-agentic methods\. The*Edit*track is effectively saturated even in the zero\-shot baseline, and the*Repair*track benefits substantially from progressive knowledge injection \(\+18%\)\. In contrast, the same progression improves*Reproduce*performance only to 49 of 86 tasks \(57%\), highlighting the remaining gap that motivates the use of agentic methods\.The*Repair*track exhibits greater sensitivity to domain knowledge\. TheBaselineconfiguration successfully repairs 92 of 131 tasks \(70%\), while the addition of the PHITS manual \(\+PDF\) increases performance to 110 tasks \(84%\)\. Providing both the manual and structured catalog \(\+PDF\+Cat\) further improves performance to 116 tasks \(89%\)\. This improvement is expected because*Repair*tasks provide an explicit PHITS error message that localizes the fault\. The knowledge sources then supply the information needed to identify the correct keyword, parameter, or syntax structure\.

### 3\.2Reproduce: from natural language to executable simulation

The*Reproduce*track represents the most demanding benchmark scenario\. Unlike the*Edit*and*Repair*tasks, the model receives neither a valid PHITS template nor an error message\. Instead, it must construct an entire simulation from a natural\-language description of geometry, materials, particle sources, and desired observables\. Figure[4](https://arxiv.org/html/2607.09789#S3.F4)summarizes performance on the 86*Reproduce*tasks \(see also Table[3](https://arxiv.org/html/2607.09789#S3)\)\.

![Refer to caption](https://arxiv.org/html/2607.09789v1/x4.png)Figure 4:Success rate \(𝐶𝑀𝑆≥0\.95\\mathit\{CMS\}\\geq 0\.95\) on the 86*Reproduce*\-track tasks across the five evaluated methods \(green bars, left axis\)\. Structured knowledge injection \(\+Cat\+PDFrelative to theBaseline\) accounts for the largest overall performance improvement, whileAgenticexecution provides a further increase in success rate\. The red line \(right axis\) shows the estimated per\-task inference cost at GPT\-5\.4 list pricing \(single\-run estimate; single\-shot costs are amortized post\-cache values\)\.Codex CLIachieves the highest success rate \(73%\) but at approximately six times the per\-task cost of ourCustom Multi\-Agent, which achieves comparable performance \(66%\) at substantially lower cost\.TheBaselineconfiguration fails all 86 tasks at the CMS≥0\.95\\geq 0\.95threshold\. Although generated outputs often contain plausible PHITS\-like structures, they typically fail because of missing keywords, malformed geometry definitions, inconsistent tally specifications, or other parser\-level errors\. These results indicate that pretraining alone does not provide sufficient PHITS\-specific knowledge to generate valid simulations\.

Providing access to the PHITS user manual substantially improves performance\. The\+PDFconfiguration successfully solves 28 of 86 tasks \(33%\), demonstrating that the model can effectively utilize technical documentation during generation\. However, the improvement remains limited despite the manual containing all information required to construct valid simulations\. The greatest gain is obtained through the injection of the structured knowledge\. Adding the PHITS catalog \(\+Cat\+PDF\) increases performance from 28 to 49 tasks \(57%\), corresponding to an additional 21 successful simulations\. The catalog reorganizes information contained in the manual into a machine\-readable form and explicitly encodes parser constraints, default values, and cross\-section dependencies\. This result suggests that accessibility and organization of domain knowledge are at least as important as the volume of available documentation\.

The\+Cat\+PDFconfiguration achieves an average physics\-fidelity score of approximately 0\.67 across all*Reproduce*tasks\. Among the 37 unsuccessful cases, 23 fail to execute entirely \(C​M​S=0CMS=0\), while the remaining failures cluster between CMS values of approximately 0\.5 and 0\.95, indicating that many are near\-correct simulations rather than complete failures\. To determine whether the remaining performance gap arises from insufficient reasoning rather than insufficient knowledge, we repeated the\+Cat\+PDFevaluation using higher reasoning\-effort settings\. Increasing GPT\-5\.4 reasoning from medium to high and xhigh improves success only marginally, from 49 to 52 tasks\. In contrast, adding structured domain knowledge increases performance from 0 to 49 tasks\. These results indicate that access to the appropriate information is more important than additional reasoning depth\.

### 3\.3Agentic execution and iterative repair

The final two configurations augment knowledge injection with execution\-driven feedback and iterative repair\. The genericCodex CLIagent achieves the highest overall performance, successfully solving 63 of 86 Reproduce tasks \(73%\)\. OurCustom Multi\-Agentsystem solves 57 of 86 tasks \(66%\)\. Both substantially outperform the strongest single\-shot configuration \(Fig\.[4](https://arxiv.org/html/2607.09789#S3.F4)\)\. Importantly, both agentic methods use the same PHITS manual and structured catalog available to the\+Cat\+PDFbaseline\. The improvement therefore arises primarily from the ability to execute PHITS, inspect outputs, diagnose failures, and revise candidate solutions\.

The performance difference between the two agentic approaches should not be interpreted as a definitive ranking of architectures\.Codex CLIemploys unrestricted iteration with xhigh reasoning effort, whereas ourCustom Multi\-Agentframework uses bounded repair budgets and selective allocation of reasoning resources\. A detailed inspection shows that theRepairerin our agent issued 206file\_searchqueries across 32 repair sessions, with 67 % of those queries targeting tally syntax, which is the most error\-prone PHITS feature in our corpus\. This provides evidence that the agent is dedicating resources to address the performance bottleneck rather than rephrasing existing context\. The comparison is therefore best viewed as an operational tradeoff between performance and computational cost\.

This distinction becomes apparent when inference cost is considered\.Codex CLIrequires approximately $0\.82 per task, whereas theCustom Multi\-Agentarchitecture requires approximately $0\.13 per task\. Our multi\-agent system employs a bounded repair budget of five iterations together with a tiered reasoning strategy, using medium reasoning for theManagerandComposeragents andxhighreasoning only for theRepairerwhen needed\. As a result, it achieves a substantially lower per\-task token cost than Codex CLI, which operates with an unrestrictedxhigh\-reasoning loop\. This reduction in computational cost makes the proposed architecture a more efficient deployment strategy at comparable performance levels\.

Overall, these results suggest that execution feedback provides an additional 9–16 percentage point improvement beyond structured knowledge injection alone\. Knowledge remains the dominant contributor to performance, but agentic execution enables recovery from many errors that cannot be resolved within a single inference pass\.

### 3\.4Failure mode analysis

To understand the remaining limitations of AI\-assisted PHITS generation, we examined the 29*Reproduce*tasks that ourCustom Multi\-Agentsystem failed to solve at the CMS≥0\.95\\geq 0\.95threshold\. Figure[5](https://arxiv.org/html/2607.09789#S3.F5)summarizes the dominant failure categories\. Twenty\-five of the 29 failures successfully execute in PHITS \(E​X=1EX=1\), indicating that the generated input files are syntactically valid\. Only four failures arise from execution errors\. The majority of remaining failures therefore originate from deficiencies in physical fidelity rather than parser correctness\. The dominant failure mode is incorrect tally configuration, which accounts for 19 of 29 failures \(66%\)\. These cases typically involve incorrect particle selections, output units, scoring geometries, or energy\-bin definitions\. Execution failures account for four tasks \(14%\), prompt misinterpretation for three tasks \(10%\), and geometry\-related errors for only a single task\.Codex\-CLIfailures reveal the same qualitative pattern\.

Three representative examples illustrate the pattern of the failure stemming from the lack of domain knowledge\. In the first case, the prompt requested a current spectrum in units of1/cm2/MeV/source1/\\mathrm\{cm^\{2\}/MeV/source\}\. The agent correctly selected the\[T\-Cross\]tally and generated a valid input deck, but specifiedunit = 1\(1/cm2/source1/\\mathrm\{cm^\{2\}/source\}\) instead ofunit = 2, omitting the required per\-MeV normalization\. In the second case, the prompt requested scoring of ionization and excitation events\. The agent generated a syntactically valid\[T\-Interact\]tally but omitted theoutputkeyword, causing PHITS to revert to its default observable, namely nuclear\-reaction density\. In the third example, the agent generated a valid LET spectrum but selectedunit = 6\(dose\) rather than the requestedunit = 14, corresponding to the microdosimetric quantityL,d​\(L\)L,d\(L\)\. In all three cases, the failure arose from selecting an incorrect physical quantity, normalization, or output representation\. Such examples are largely absent from the current catalog, which functions primarily as a keyword and constraint reference\. We therefore hypothesize that future gains may require curated training examples or post\-training datasets that capture expert choices of observables and tally configurations rather than additional parser documentation alone\.

A second category of failures consists of agent\-specific regressions\. Four of the 29 failed tasks correspond to cases that are successfully solved by the strongest single\-shot configuration \(\+Cat\+PDF\) but are subsequently broken by the agentic repair process\. Inspection of theRepairertraces indicates that these failures occur when the repair process modifies portions of an already correct specification while attempting to address an unrelated warning or static\-checking issue\. This behavior appears to reflect a limitation of the repair policy rather than a limitation of the underlying model and could likely be mitigated through stricter edit constraints or localized repair strategies\.

These results indicate that current models are increasingly capable of generating syntactically valid PHITS simulations\. The remaining challenge lies in incorrect selection or configuration of physical observables, suggesting limitations in*domain judgment*rather than code generation\.

![Refer to caption](https://arxiv.org/html/2607.09789v1/x5.png)Figure 5:Dominant root causes of the 29 Reproduce\-track failures of theCustom Multi\-Agentsystem \(𝐶𝑀𝑆<0\.95\\mathit\{CMS\}<0\.95\)\. Most failures \(25 of 29\) occur after successful PHITS execution \(𝐸𝑋=1\\mathit\{EX\}=1\) and therefore reflect incorrect selection or configuration of physical observables rather than syntax generation errors\. Only four failures arise from execution errors \(𝐸𝑋=0\\mathit\{EX\}=0\)\. Incorrect tally configuration is the dominant residual failure mode, highlighting the importance of domain judgment in radiation\-transport modeling\.
### 3\.5Independent physics validation

The CMS metric evaluates agreement between generated and reference simulations within PHITSBench\. A natural question, however, is whether high benchmark scores correspond to physically meaningful radiation transport calculations rather than successful reproduction of benchmark\-specific outputs\. To address this question, we performed an independent validation study using four canonical radiation\-transport problems that do not appear anywhere in PHITSBench\. These benchmark problems were selected because they probe distinct aspects of particle transport physics and are commonly used for validation of radiation transport codes\. Specifically, we considered: \(i\) the proton Bragg peak in water, which tests charged\-particle stopping power and energy deposition; \(ii\) narrow\-beam gamma attenuation in lead, which tests photon interaction cross sections and shielding calculations; \(iii\) the practical range of fast electrons in water, which probes electron transport and energy\-loss processes; and \(iv\) the full\-energy photopeak generated by a Cs\-137 gamma source in a NaI\(Tl\) scintillation detector, which tests photon transport and detector response\.

The proton benchmark evaluates whether the model can correctly reproduce the characteristic Bragg peak observed when energetic charged particles slow down in matter \(Fig\.[6](https://arxiv.org/html/2607.09789#S3.F6)a\)\. As protons traverse water, they gradually lose kinetic energy through ionization and excitation of the medium\. Because the stopping power increases as the particle slows, most energy is deposited near the end of the proton’s range, producing a pronounced peak in deposited dose\. Accurate prediction of the Bragg peak location is essential in proton therapy and charged\-particle shielding applications\.

The gamma attenuation benchmark evaluates the exponential reduction of photon intensity as gamma rays traverse an absorbing material \(Fig\.[6](https://arxiv.org/html/2607.09789#S3.F6)b\)\. In this case, a narrow beam of photons passes through lead, where interactions such as photoelectric absorption, Compton scattering, and pair production progressively remove photons from the primary beam\. The measured attenuation coefficient therefore provides a sensitive test of photon interaction physics and shielding calculations\.

The electron\-range benchmark evaluates transport of energetic electrons in water \(Fig\.[6](https://arxiv.org/html/2607.09789#S3.F6)c\)\. Unlike heavy charged particles, electrons experience significant multiple scattering and stochastic energy\-loss processes, resulting in broader energy deposition profiles and more diffuse stopping behavior\. Accurate prediction of electron penetration depth is important for radiation dosimetry and detector modeling\.

The detector benchmark evaluates generation of the characteristic Cs\-137 full\-energy photopeak in a NaI\(Tl\) scintillation detector \(Fig\.[6](https://arxiv.org/html/2607.09789#S3.F6)d\)\. Cs\-137 emits a 662\-keV gamma ray that deposits its full energy within the detector when all secondary interactions are contained\. Reproducing the photopeak position therefore requires correct modeling of photon interactions, energy deposition, and detector geometry\.

For each benchmark,Codex CLIreceived only a natural\-language description of the physical problem together with the PHITS manual and structured knowledge catalog \(\+Cat\+PDF\)\. No benchmark\-specific templates or examples were provided\. The resulting PHITS input files were executed in PHITS 3\.34, and the principal observables were compared against established reference data \(Fig\.[6](https://arxiv.org/html/2607.09789#S3.F6)\), including NIST PSTAR and ESTAR stopping\-power tables for charged particles\[[5](https://arxiv.org/html/2607.09789#bib.bib44)\], the NIST XCOM photon cross\-section database\[[6](https://arxiv.org/html/2607.09789#bib.bib45)\], and evaluated nuclear decay data for Cs\-137\[[7](https://arxiv.org/html/2607.09789#bib.bib46)\]\.

Figure[6](https://arxiv.org/html/2607.09789#S3.F6)shows that three of the four generated simulations reproduce the corresponding reference observables within Monte Carlo statistical uncertainty\. The simulated proton Bragg peak occurs at a depth of 15\.65 cm, compared with the NIST CSDA range of 15\.77 cm\. This small offset is physically expected because the maximum deposited dose occurs slightly before the particle reaches its stopping point\. The electron practical range \(4\.91 cm versus 4\.90 cm\) and the NaI\(Tl\) photopeak energy \(0\.661 MeV versus 0\.662 MeV\) agree with reference values within simulation uncertainty\. The remaining benchmark, photon attenuation in lead, produced an attenuation coefficient approximately 14% lower than the reference value \(1\.07 versus 1\.25 cm\-1\)\. Examination of the generated input revealed that the attenuation coefficient was extracted from a track\-length tally that includes scattered photons in addition to uncollided primary photons\. Consequently, the discrepancy reflects a modeling choice rather than an error in the underlying transport calculation and could be eliminated by replacing the tally with a detector\-fluence or first\-collision scoring method\. These results suggest that successful benchmark performance corresponds to physically meaningful simulations and that the generated inputs reproduce a diverse set of canonical radiation\-transport phenomena spanning charged\-particle stopping, photon attenuation, electron transport, and detector response\.

![Refer to caption](https://arxiv.org/html/2607.09789v1/x6.png)Figure 6:Independent validation of AI\-generated PHITS simulations using four canonical radiation\-transport benchmarks not included in PHITSBench\. The left panel shows the natural language prompts\. The center panels show the simulated particle tracks and geometry, while the right panels compare the resulting observables with reference values\. Green \(a,c,d\) indicates agreement within Monte Carlo statistical uncertainty \(relative errors of about 0\.2 % for the proton Bragg peak dose, 0\.4 % for the electron depth\-dose, 1\.5 % for the NaI photopeak, and 0\.1 % for the lead fluence\)\. Amber \(b\) denotes a physically explained discrepancy arising from tally selection rather than an error in the underlying transport calculation\.

## 4Discussion

The central finding of this study is that AI\-assisted generation of radiation transport simulations is already feasible for a substantial fraction of practical PHITS workflows, provided that domain knowledge is supplied in a structured and accessible form\. Across the*Reproduce*benchmark, the largest performance gain arises not from increased reasoning effort or more sophisticated agent architectures, but from the introduction of a structured PHITS knowledge catalog \(Fig\.[4](https://arxiv.org/html/2607.09789#S3.F4)\)\. Agentic execution provides additional improvements, but its contribution is secondary to the availability of machine\-readable domain knowledge\. This observation has implications both for radiation transport modeling specifically and for AI\-assisted scientific computing more broadly\. The results suggest that many current limitations of scientific AI systems arise not because foundation models lack general reasoning ability, but because critical domain knowledge remains embedded in documentation, source code, and expert practice in forms that are difficult for AI systems to access and utilize effectively\.

### 4\.1Implications for radiation transport modeling

The benchmark tasks examined in this work closely mirror the workflows routinely performed by PHITS users during simulation development, validation, and deployment\. Consequently, the observed performance provides insight into where AI assistance may become useful in practical radiation transport studies\.

The strongest near\-term opportunity is simulation quality assurance\. The*Edit*and*Repair*tracks, which approach saturation under single\-shot prompting, correspond to common debugging and validation tasks such as identifying missing parameters, correcting malformed geometry definitions, repairing invalid tally configurations, or detecting inconsistencies between source and scoring specifications\. These activities often consume substantial time during model development, particularly because many PHITS inputs are syntactically valid yet physically incorrect\. The ability of an AI assistant to automatically flag such issues before long production runs could reduce both development time and the risk of propagating incorrect simulations through larger engineering studies\.

The benchmark also suggests a possible role for AI\-assisted verification of the radiation\-transport workflows\. Because PHITSBench evaluates both execution success and agreement with known transport observables, it provides a framework for distinguishing between simulations that merely run and simulations that produce physically meaningful results\. Such execution\-grounded validation could be particularly valuable in shielding and facility design\[[8](https://arxiv.org/html/2607.09789#bib.bib4),[38](https://arxiv.org/html/2607.09789#bib.bib2)\], detector development and medical physics\[[34](https://arxiv.org/html/2607.09789#bib.bib5)\], reactor analysis\[[11](https://arxiv.org/html/2607.09789#bib.bib3)\], and space and planetary science\[[32](https://arxiv.org/html/2607.09789#bib.bib6)\], where small configuration errors can lead to significant discrepancies in predicted particle fluxes, dose distributions, or detector responses\.

Finally, PHITSBench may serve as a useful educational resource\. Each benchmark task pairs a natural\-language physics problem with a validated reference simulation and quantitative scoring framework\. Such examples could support training of students, researchers, and designers, and help bridge the gap between understanding transport physics and implementing that understanding in executable simulation workflows\.

### 4\.2Community infrastructure opportunities

The strongest performance improvements observed in this work originate from resources that are largely external to the foundation model itself\. This suggests several opportunities for community\-driven development\.

The first opportunity is the creation of a machine\-readable PHITS knowledge base\. The single\-shot\+Cat\+PDFconfiguration produces the largest cumulative improvement over the no\-knowledge baseline in the entire ablation study \(Fig\.[4](https://arxiv.org/html/2607.09789#S3.F4), from 0% to 57 %\)\. The underlying catalog consists of 11 Markdown files totaling approximately 7,300 lines of structured content\. It was generated through LLM\-assisted parsing of the PHITS Fortran source code and then manually audited, refined, and organized by a domain expert\. This catalog serves as a proof of concept, which can be built upon to create a structured machine\-readable corpus \(in the YAML or JSON format\), updated in lock\-step with new PHITS version releases\. It should capture keyword definitions, accepted parameter ranges, default values, parser constraints, compatibility rules, and common error patterns in a format suitable for both human users and AI systems\. Because much of this information already exists within the PHITS source code and documentation, such a resource would be relatively inexpensive to maintain while providing substantial value\.

A second opportunity is the development of curated domain\-training datasets\. The remaining benchmark failures frequently involve choices that experienced users make implicitly when selecting observables, scoring methods, or tally configurations\. These decisions are rarely documented formally and, therefore, are difficult for AI systems to learn through retrieval alone\. Even a few thousand such expert\-curated examples, collected through a community submission process and reviewed by experienced PHITS users, could provide a valuable foundation for domain\-specific post\-training, including LoRA \(low\-rank adaptation\), full fine\-tuning, and DPO \(direct preference optimization\)\. Such a dataset could help move the model beyond generic code generation and retrieval\-augmented assistance toward behavior more closely resembling that of an experienced PHITS practitioner\.

A third, potentially high\-impact direction is the development of a verifiable reinforcement\-learning environment\. One of the most distinctive properties of PHITSBench as well as other Monte Carlo simulation tasks is that performance can be evaluated using a deterministic, computable reward\. Given a candidate input file and a reference simulation, the resulting CMS is a quantitative measure of correctness rather than a subjective judgment\. This is an excellent setting in which reinforcement learning from verifiable environments can be applied effectively\. A natural next step would be to package PHITSBench as a Gymnasium\-compatible environment, expose CMS as the reward signal, and enable models to improve through GRPO \(Group Relative Policy Optimization\) or related reinforcement\-learning algorithms\.

Significant engineering challenges remain, including ensuring reward stability, maintaining sufficient task diversity, preventing benchmark overfitting, and managing the licensing and computational costs associated with large\-scale training\. Although these directions are beyond the scope of this work, we believe they represent one of the most promising paths toward transforming AI\-assisted MC transport workflows from useful code\-generation tools into reliable, human\-supervised collaborators\. Such a benchmark could become a shared community resource whose value increases as multiple groups contribute data, methods, and models within a common evaluation framework\.

### 4\.3Limitations

Several limitations should be considered when interpreting these results\. First, the study evaluates a single model family \(GPT\-5\.4\) and a limited set of agent architectures\. Future models may exhibit different behavior, particularly with respect to knowledge utilization and self\-correction\. Second, the benchmark focuses exclusively on PHITS version 3\.34\. Although the methodology is broadly applicable, changes in syntax or functionality across future PHITS releases may affect absolute performance values\. Third, the knowledge catalog itself is a manually curated artifact\. While this reflects realistic domain\-engineering practice, it introduces a dependence on catalog quality that has not been systematically studied\. Future work should investigate how benchmark performance varies as a function of knowledge representation, completeness, and organization\. Finally, the benchmark evaluates simulation generation rather than scientific problem solving in the broader sense\. Producing a valid PHITS input file is only one step within a larger engineering workflow that includes model verification, uncertainty quantification, experimental validation, and interpretation of results\.

## 5Conclusions

We introduced PHITSBench, an execution\-scored benchmark for evaluating AI\-assisted generation of PHITS radiation\-transport simulations from natural\-language descriptions, and used it to evaluate five GPT\-5\.4\-based configurations ranging from zero\-shot prompting to knowledge\-augmented and agentic workflows\. The results reveal a clear hierarchy of factors limiting performance\.

Specifically, we find that \(i\) structured knowledge injection provides the largest performance gain on the most challenging benchmark track, increasing Reproduce\-task success from 0% to 57%; \(ii\) agentic execution yields a further, though smaller, improvement to 66\-73%, depending on the architecture; and \(iii\) the remaining failures can be traced to a small number of identifiable root causes that point as much to gaps in domain\-specific infrastructure as to limitations of the underlying model\.

We conclude that for highly specialized scientific simulation environments such as PHITS access to structured domain knowledge is a more important determinant of performance than additional reasoning effort alone\. We therefore view machine\-readable knowledge bases, curated domain\-training datasets, and verifiable reinforcement\-learning environments as promising directions for future development for scientific simulation ecosystems beyond PHITS\.

## Acknowledgement

The authors thank Tongge Yu \(MIT\) for contributing to the conception and early brainstorming of this project\. We are grateful to Dr\. Tatsuhiko Sato of the PHITS development team for insightful discussions and constructive feedback\. We also thank Dr\. Duo Xu and Charles Lai \(MIT\) for their contributions to the design and evaluation of the benchmark\. OpenAI API credits supporting this work were provided by OpenAI through the MIT AI Hackathon\.

## Disclosure statement

No potential conflict of interest was reported by the authors\.

## References

- \[1\]Z\. Abulawi, Z\. Ndum Ndum, E\. Cervi, R\. Hu, and Y\. Liu\(2026\)AutoSAM: an agentic framework for automating input file generation for the SAM code with multi\-modal retrieval\-augmented generation\.External Links:2603\.24736,[Link](https://arxiv.org/abs/2603.24736)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[2\]S\. Agostinelli, J\. Allison, K\. Amako, J\. Apostolakis, H\. Araujo, P\. Arce, M\. Asai, D\. Axen, S\. Banerjee, G\. Barrand,et al\.\(2003\)GEANT4—a simulation toolkit\.Nuclear Instruments and Methods in Physics Research A506\(3\),pp\. 250–303\.External Links:[Document](https://dx.doi.org/10.1016/S0168-9002%2803%2901368-8)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p1.1)\.
- \[3\]T\. Ahmed, M\. M\. Azwad, and S\. Choudhury\(2025\)SIMCODE: a benchmark for natural language to ns\-3 network simulation code generation\.External Links:2507\.11014,[Link](https://arxiv.org/abs/2507.11014)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[4\]A\. Almeldein, M\. Alnaggar, R\. Archibald, T\. Beck, A\. Biswas,et al\.\(2025\)Exploring the capabilities of the frontier large language models for nuclear energy research\.External Links:2506\.19863,[Link](https://arxiv.org/abs/2506.19863)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[5\]M\. J\. Berger, J\. S\. Coursey, M\. A\. Zucker, and J\. Chang\(2017\)ESTAR, PSTAR, and ASTAR: computer programs for calculating stopping\-power and range tables for electrons, protons, and helium ions\.Note:Version 2\.0\.1, NIST Standard Reference Database 124National Institute of Standards and Technology, Gaithersburg, MDExternal Links:[Document](https://dx.doi.org/10.18434/T4NC7P),[Link](https://physics.nist.gov/Star)Cited by:[§3\.5](https://arxiv.org/html/2607.09789#S3.SS5.p6.1)\.
- \[6\]M\. J\. Berger, J\. H\. Hubbell, S\. M\. Seltzer, J\. Chang, J\. S\. Coursey, R\. Sukumar, D\. S\. Zucker, and K\. Olsen\(2010\)XCOM: photon cross section database \(version 1\.5\)\.Note:NIST Standard Reference Database 8 \(XGAM\)National Institute of Standards and Technology, Gaithersburg, MDExternal Links:[Document](https://dx.doi.org/10.18434/T48G6X),[Link](https://physics.nist.gov/xcom)Cited by:[§3\.5](https://arxiv.org/html/2607.09789#S3.SS5.p6.1)\.
- \[7\]E\. Browne and J\. K\. Tuli\(2007\)Nuclear data sheets for A = 137\.Nuclear Data Sheets108\(10\),pp\. 2173–2318\.External Links:[Document](https://dx.doi.org/10.1016/j.nds.2007.09.002)Cited by:[§3\.5](https://arxiv.org/html/2607.09789#S3.SS5.p6.1)\.
- \[8\]Y\. Çelik, Y\. Iwamoto, S\. Hashimoto, T\. Sato, N\. Matsuda, S\. Kunieda, N\. Furutachi, and K\. Niita\(2022\)Benchmark study of particle and heavy\-ion transport code system using shielding integral benchmark archive and database for accelerator\-shielding experiments\.Journal of Nuclear Science and Technology59\(5\),pp\. 665–675\.External Links:[Document](https://dx.doi.org/10.1080/00223131.2021.1993372)Cited by:[§4\.1](https://arxiv.org/html/2607.09789#S4.SS1.p3.1)\.
- \[9\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.Note:arXiv:2107\.03374 \[cs\.LG\]External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[10\]Y\. Chen, X\. Zhu, H\. Zhou, and Z\. Ren\(2024\)MetaOpenFOAM: an LLM\-based multi\-agent framework for CFD\.Note:arXiv:2407\.21320 \[physics\.flu\-dyn\]External Links:[Link](https://arxiv.org/abs/2407.21320)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[11\]J\. R\. P\. Cruz, A\. A\. Astronomo, G\. N\. Santos, and R\. U\. Olivares\(2024\)A PHITS\-based computational model of a TRIGA\-fueled subcritical reactor for gamma dose mapping\.Nuclear Engineering and Technology56\(12\),pp\. 5339–5345\.External Links:[Document](https://dx.doi.org/10.1016/j.net.2024.07.043)Cited by:[§4\.1](https://arxiv.org/html/2607.09789#S4.SS1.p3.1)\.
- \[12\]A\. Ferrari, P\. R\. Sala, A\. Fassò, and J\. Ranft\(2005\)FLUKA: a multi\-particle transport code \(program version 2005\)\.Technical reportTechnical ReportCERN\-2005\-010, INFN/TC\-05/11, SLAC\-R\-773,CERN\.External Links:[Document](https://dx.doi.org/10.5170/CERN-2005-010),[Link](https://cds.cern.ch/record/898301)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p1.1)\.
- \[13\]A\. Ghafarollahi and M\. J\. Buehler\(2025\)SciAgents: automating scientific discovery through bioinspired multi\-agent intelligent graph reasoning\.Advanced Materials37\(22\),pp\. 2413523\.External Links:[Document](https://dx.doi.org/10.1002/adma.202413523)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[14\]J\. Hill and H\. J\. Ryoo\(2026\)GRACE: an agentic AI for particle physics experiment design and simulation\.External Links:2602\.15039,[Link](https://arxiv.org/abs/2602.15039)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[15\]E\. W\. Holbrook, J\. C\. Verduzco, and A\. Strachan\(2026\)Evaluating LLM\-generated code for domain\-specific languages: molecular dynamics with LAMMPS\.External Links:2603\.20630,[Link](https://arxiv.org/abs/2603.20630)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[16\]S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber\(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=VtmBAGCN7o)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[17\]J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou\(2024\)Large language models cannot self\-correct reasoning yet\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=IkmD3fKBPQ)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[18\]N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica\(2025\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=chfJJYC3iL)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p4.1)\.
- \[19\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[20\]R\. Kamoi, Y\. Zhang, N\. Zhang, J\. Han, and R\. Zhang\(2024\)When can LLMs actually correct their own mistakes? A critical survey of self\-correction of LLMs\.Transactions of the Association for Computational Linguistics12,pp\. 1417–1440\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00713)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[21\]Y\. Lai, C\. Li, Y\. Wang, T\. Zhang, R\. Zhong, L\. Zettlemoyer, W\. Yih, D\. Fried, S\. Wang, and T\. Yu\(2023\)DS\-1000: a natural and reliable benchmark for data science code generation\.InProceedings of the 40th International Conference on Machine Learning \(ICML\),External Links:[Link](https://proceedings.mlr.press/v202/lai23b.html)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[22\]M\. Liu, N\. Pinckney, B\. Khailany, and H\. Ren\(2023\)Invited paper: VerilogEval: evaluating large language models for Verilog code generation\.In2023 IEEE/ACM International Conference on Computer\-Aided Design \(ICCAD\),External Links:[Document](https://dx.doi.org/10.1109/ICCAD57390.2023.10323812)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[23\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.External Links:[Link](https://openreview.net/forum?id=S37hOerQLB)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[24\]Z\. Ndum Ndum, J\. Tao, J\. Ford, and Y\. Liu\(2025\)Automating Monte Carlo simulations in nuclear engineering with domain knowledge\-embedded large language model agents\.Energy and AI21,pp\. 100555\.Note:Agent framework referred to as AutoFLUKA\.External Links:[Document](https://dx.doi.org/10.1016/j.egyai.2025.100555)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[25\]T\. X\. Olausson, J\. P\. Inala, C\. Wang, J\. Gao, and A\. Solar\-Lezama\(2024\)Is self\-repair a silver bullet for code generation?\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=y0GJXRungR)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[26\]OpenAI\(2025\)Codex CLI: command\-line coding agent\.Note:[https://github\.com/openai/codex](https://github.com/openai/codex)Cited by:[§2\.4](https://arxiv.org/html/2607.09789#S2.SS4.p4.1)\.
- \[27\]OpenAI\(2025\)OpenAI Agents SDK\.Note:[https://github\.com/openai/openai\-agents\-python](https://github.com/openai/openai-agents-python)Python SDK v0\.8\.4Cited by:[§2\.4](https://arxiv.org/html/2607.09789#S2.SS4.p6.2)\.
- \[28\]S\. Pandey, R\. Xu, W\. Wang, and X\. Chu\(2025\)OpenFOAMGPT: a retrieval\-augmented large language model \(LLM\) agent for OpenFOAM\-based computational fluid dynamics\.Physics of Fluids37\(3\),pp\. 035120\.External Links:[Document](https://dx.doi.org/10.1063/5.0257555)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1),[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[29\]D\. Park, H\. Moon, and S\. Ryu\(2026\)A self\-correcting multi\-agent LLM framework for language\-based physics simulation and explanation\.npj Artificial Intelligence2\(1\),pp\. 10\.External Links:[Document](https://dx.doi.org/10.1038/s44387-025-00057-z)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[30\]S\. Pimparkhede, M\. Kammakomati, S\. Tamilselvam, P\. Kumar, A\. P\. Kumar, and P\. Bhattacharyya\(2024\)DocCGen: document\-based controlled code generation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP 2024\),pp\. 18681–18697\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1040),[Link](https://arxiv.org/abs/2406.11925)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[31\]S\. Ren, D\. Guo, S\. Lu, L\. Zhou, S\. Liu, D\. Tang, N\. Sundaresan, M\. Zhou, A\. Blanco, and S\. Ma\(2020\)CodeBLEU: a method for automatic evaluation of code synthesis\.Note:arXiv:2009\.10297 \[cs\.SE\]External Links:2009\.10297,[Link](https://arxiv.org/abs/2009.10297)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p8.1)\.
- \[32\]T\. Sato, A\. Endo, L\. Sihver, and K\. Niita\(2011\)Dose estimation for astronauts using dose conversion coefficients calculated with the PHITS code and the ICRP/ICRU adult reference computational phantoms\.Radiation and Environmental Biophysics50\(1\),pp\. 115–123\.External Links:[Document](https://dx.doi.org/10.1007/s00411-010-0330-0)Cited by:[§4\.1](https://arxiv.org/html/2607.09789#S4.SS1.p3.1)\.
- \[33\]T\. Sato, Y\. Iwamoto, S\. Hashimoto, T\. Ogawa, T\. Furuta, S\. Abe, T\. Kai, Y\. Matsuya, H\. Iwase, N\. Shigyo,et al\.\(2024\)Recent improvements of the particle and heavy ion transport code system—PHITS version 3\.33\.Journal of Nuclear Science and Technology61\(1\),pp\. 127–135\.Note:Describes PHITS version 3\.33; the present work uses version 3\.34\.External Links:[Document](https://dx.doi.org/10.1080/00223131.2023.2275736)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p1.1)\.
- \[34\]D\. Satoh and T\. Sato\(2022\)Improvements in the particle and heavy\-ion transport code system \(PHITS\) for simulating neutron\-response functions and detection efficiencies of a liquid organic scintillator\.Journal of Nuclear Science and Technology59\(8\),pp\. 1047–1060\.External Links:[Document](https://dx.doi.org/10.1080/00223131.2021.2019622)Cited by:[§4\.1](https://arxiv.org/html/2607.09789#S4.SS1.p3.1)\.
- \[35\]O\. Shafiq, A\. Rahmat, A\. Alexiadis, and B\. Ghiassi\(2025\)Evaluating the performance of large language models for geometry and simulation file generation in physics\-based simulations\.Applied Sciences15\(22\),pp\. 12114\.External Links:[Document](https://dx.doi.org/10.3390/app152212114)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[36\]N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.External Links:[Link](https://openreview.net/forum?id=vAElhFcKW6)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[37\]N\. Somasekharan, L\. Yue, Y\. Cao, W\. Li, P\. Emami, P\. S\. Bhargav, A\. Acharya, X\. Xie, and S\. Pan\(2025\)CFDLLMBench: a benchmark suite for evaluating large language models in computational fluid dynamics\.Note:arXiv:2509\.20374 \[cs\.LG\]External Links:[Link](https://arxiv.org/abs/2509.20374)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[38\]K\. Takada, T\. Sato, H\. Kumada, J\. Koketsu, H\. Takei, H\. Sakurai, and T\. Sakae\(2018\)Validation of the physical and RBE\-weighted dose estimator based on PHITS coupled with a microdosimetric kinetic model for proton therapy\.Journal of Radiation Research59\(1\),pp\. 91–99\.External Links:[Document](https://dx.doi.org/10.1093/jrr/rrx057)Cited by:[§4\.1](https://arxiv.org/html/2607.09789#S4.SS1.p3.1)\.
- \[39\]M\. Tian, L\. Gao, S\. D\. Zhang, X\. Chen, C\. Fan, X\. Guo, R\. Haas, P\. Ji, K\. Krongchon, Y\. Li,et al\.\(2024\)SciCode: a research coding benchmark curated by scientists\.InAdvances in Neural Information Processing Systems \(NeurIPS\) 37, Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=ADLaALtdoG)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[40\]C\. J\. Werner, J\. S\. Bull, C\. J\. Solomon, F\. B\. Brown, G\. W\. McKinney, M\. E\. Rising, D\. A\. Dixon, R\. L\. Martz, H\. G\. Hughes, L\. J\. Cox, A\. J\. Zukaitis, J\. C\. Armstrong, R\. A\. Forster, and L\. Casswell\(2018\)MCNP version 6\.2 release notes\.Technical reportTechnical ReportLA\-UR\-18\-20808,Los Alamos National Laboratory\.External Links:[Document](https://dx.doi.org/10.2172/1419730),[Link](https://www.osti.gov/biblio/1419730)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p1.1)\.
- \[41\]Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang\(2024\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.InConference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=BAakY1hNKS)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[42\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.37\.External Links:[Link](https://openreview.net/forum?id=mXpq6ut8J3)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[43\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.09789#S2.SS4.p5.1)\.
- \[44\]T\. Zhang, Z\. Liu, Y\. Xin, and Y\. Jiao\(2025\)MooseAgent: a LLM based multi\-agent framework for automating MOOSE simulation\.External Links:2504\.08621,[Link](https://arxiv.org/abs/2504.08621)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.
- \[45\]S\. Zhou, U\. Alon, F\. F\. Xu, Z\. Wang, Z\. Jiang, and G\. Neubig\(2023\)DocPrompting: generating code by retrieving the docs\.InThe Eleventh International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2207.05987)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p3.1)\.
- \[46\]T\. Y\. Zhuo, M\. C\. Vu, J\. Chim, H\. Hu, W\. Yu, R\. Widyasari, I\. N\. B\. Yusuf, H\. Zhan, J\. He, I\. Paul,et al\.\(2025\)BigCodeBench: benchmarking code generation with diverse function calls and complex instructions\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=YrycTjllL0)Cited by:[§1](https://arxiv.org/html/2607.09789#S1.p2.1)\.