CompMat-Bench：面向计算材料科学的 AI 智能体基准测试

# CompMat-Bench: Benchmarking AI Agents for Computational Materials Science
Source: [https://arxiv.org/html/2610.00636](https://arxiv.org/html/2610.00636)
Levi FelixJun\-Jie ZhangXingfu LiXuelian JiangAffiliation:Tao Jiang, Subhendu Mishra, Xixi Qin & Boris YakobsonAffiliation:Department of Materials Science and NanoEngineeringAffiliation:Rice University, Houston, TX 77005, USAAffiliation:zcmben2014@gmail\.com, biy@rice\.edu

###### Abstract

Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations\. In computational materials research, repeating the same expensive simulations across agents and trials can make evaluation impractical\. We introduce CompMat\-Bench, a benchmark of 94 tasks derived from recently published computational materials studies, each asking agents to complete a step toward achieving the study’s scientific goal\. We reproduce the research steps in advance and assess agents on preparing inputs and analyzing outputs for expensive simulations, so expensive simulations can be avoided during evaluation\. The reproduced inputs and results serve as ground truth for grading agents with fixed rules, without an LLM judge\. The benchmark supports four evaluation conditions: single tasks and workflows composed of related tasks, each with full or reduced methodological guidance\. With full guidance on single tasks, agents based on three LLMs demonstrate the ability to complete individual materials research steps, with pass rates of 66\.0–90\.4% across 94 tasks\. Both longer workflows and reduced guidance can limit agent performance, but in different ways for different agents: they lower the pass rates of the weaker agents, whereas the strongest agent falls only when a long workflow is combined with reduced guidance\. Failure analysis attributes most failures to scientific errors rather than to errors in software usage\. CompMat\-Bench provides a basis for comparing agents on the steps of real materials research and for analyzing agent failure modes\.

## 1Introduction

Large language models \(LLMs\) have demonstrated strong coding capabilities, including solving programming problems and fixing bugs in software repositories\([Chen et al\., 2021](https://arxiv.org/html/2610.00636#bib.bib8);[Jimenez et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib25);[Yang et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib56);[Tian et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib48)\)\. They have also demonstrated strong scientific understanding\([Rein et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib41);[Hendrycks et al\., 2021](https://arxiv.org/html/2610.00636#bib.bib17)\)\. Computational materials research brings these capabilities together: scientific understanding guides the choice of models and methods, while code implements calculations and analyzes results\. Yet separate assessments of coding and scientific understanding do not establish how well agents can combine them in materials research\. We therefore need benchmarks that directly quantify LLM agents’ capabilities in computational materials research and help assess their potential to accelerate scientific research\([Lu et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib33);[Boiko et al\., 2023](https://arxiv.org/html/2610.00636#bib.bib3);[Ghafarollahi & Buehler, 2025](https://arxiv.org/html/2610.00636#bib.bib14)\)\.

Figure 1:Overview of CompMat\-Bench, illustrated with research steps from a study of isotope effects in boron arsenide\([Wu, 2026](https://arxiv.org/html/2610.00636#bib.bib53)\)\. For an expensive step, input preparation and output analysis are separate tasks; prepared inputs are graded without running the simulation, and analysis starts from our precomputed outputs\. Lighter calculations are executed by the agent\. Tasks are evaluated singly or as a workflow of connected steps, with full or reduced method guidance\. Fixed rules grade submissions against hidden references from our reproduction\.However, evaluating agents on computational materials research is challenging because simulations can take hours, or even days\. Repeating these simulations across agents and trials is costly and makes rapid evaluation impractical\([Cao et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib5)\)\. Most existing benchmarks either exclude computationally demanding tasks, as in ScienceAgentBench\([Chen et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib9)\), or provide substantial computing resources \(e\.g\., up to 72 hours per run in AutoMat\([Huang et al\., 2026a](https://arxiv.org/html/2610.00636#bib.bib20)\)\)\. The former narrows the range of research represented, while the latter retains the cost of running simulations for each attempt\. Our aim is to evaluate LLM agents ondiverse research problemsin computational materials research,without expensive simulationsduring evaluation\.

We introduceCompMat\-Bench, a benchmark of 94 tasks derived from 15 recently published, peer\-reviewed studies in computational materials research\. We remove expensive simulations from task evaluation by \(1\) identifying the research steps that require them and reproducing these simulations in advance, \(2\) saving their inputs and results as grading references, and \(3\) assessing agents on preparing simulation inputs and analyzing the resulting data \(Fig\.[1](https://arxiv.org/html/2610.00636#S1.F1)\)\. Most computational materials tools are standard software packages with fixed interfaces\. These interfaces, together with our reproduced inputs and results, allow us to grade agents using fixed rules, including numerical tolerances for scientific outputs, without an LLM judge\.

CompMat\-Bench also provides a protocol for controlling task difficulty through methodological guidance and task span\. For each task, we provide full\- and reduced\-guidance prompts that preserve the scientific target and grading criteria\. We combine scientifically dependent tasks from the same paper into eight workflows, each requiring an agent to reach the final result from the initial inputs\. Crossing guidance with task span yields four evaluation conditions: single tasks with full \(SF\) or reduced \(SR\) guidance, and workflows with full \(WF\) or reduced \(WR\) guidance\.

We evaluate four agents built on three LLMs under the four conditions and analyze their failures\. Our contributions are:

1. \(1\)Computational materials benchmark\.We introduce CompMat\-Bench, with 94 tasks for fixed\-rule evaluation of computational materials research without expensive simulations\.
2. \(2\)Difficulty control\.We vary task difficulty along two axes, methodological guidance and task span, while keeping the scientific target and grading of each task fixed, which quantifies their individual contributions to agent performance\.
3. \(3\)Failure analysis\.We identify common failure modes of agents in computational materials research: agents do not recognize anomalies in their own results that an experienced researcher would notice, and agents based on different LLMs lack the same implicit knowledge of software interfaces\.

## 2Related Work

#### Evaluation cost\.

Computational materials research relies on simulations, such as density functional theory \(DFT\) and molecular dynamics \(MD\), that can run for hours or even days on high\-performance computing \(HPC\) clusters\([Camps et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib4)\)\. A benchmark that includes these simulations must repeat them for every agent, trial and evaluation condition\. Simulation time is an evaluation cost that token counts\([Kapoor et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib26)\)do not capture\([Cao et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib5)\)\. Most existing benchmarks either avoid this cost or pay it in every attempt\. The first approach,*exclusion*, keeps expensive computation out of the tasks\([Bogin et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib2)\)\. FIRE\-Bench selects low\-cost machine\-learning studies \(most attempts finish within one hour\)\([Wang et al\., 2026b](https://arxiv.org/html/2610.00636#bib.bib51)\)\. CORE\-Bench selects papers whose code runs in under 45 minutes\([Siegel et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib43)\)\. ReplicationBench caps the core contributions of each paper at six hours \(with intermediate data products for costlier steps\)\([Ye et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib58)\)\. Exclusion keeps attempts cheap but limits the research to light computation\. The second approach,*rerunning*, executes the computation in every attempt\([Chan et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib7);[Faroughy et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib13)\)\. AutoMat allows each run up to 96 CPU cores and 72 hours\([Huang et al\., 2026a](https://arxiv.org/html/2610.00636#bib.bib20)\)\. PaperBench executes each submission on a fresh GPU machine\([Starace et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib44)\)\. Rerunning retains the research but repeats its cost for every agent and trial\. Recent benchmarks take a third approach,*precomputation*, and supply outputs computed in advance\. gwBenchmarks builds its tasks on public gravitational\-wave simulations and catalogs\([Islam et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib22)\)\. MDArena evaluates the stages before and after MD runs, which lack a ground truth and are stochastic\([Anand et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib1)\)\.

CompMat\-Bench runs each expensive simulation once in advance and evaluates agents on preparing its inputs and analyzing its outputs\. This suits computational materials research because most simulations run in standard software packages with fixed interfaces\. Input files that agents prepare can therefore be graded by fixed rules without running them, and the outputs of one reproduction serve every agent and trial\. Table[1](https://arxiv.org/html/2610.00636#S2.T1)compares benchmarks on three properties: tasks derived from published studies, tasks involving expensive computation, and expensive computation excluded from evaluation\. Each prior benchmark lacks at least one property or holds it only in part, whereas CompMat\-Bench holds all three\.

Table 1:CompMat\-Bench and prior agent benchmarks on three properties: tasks derived from published studies, tasks involving expensive computation, and expensive computation excluded from evaluation; the last column gives the grading method\. ✓ yes, ✗ no, \(✓\) in part, – no expensive computation in the tasks\.
#### Real research tasks\.

A benchmark based on real research tasks better reflects the research capability of agents\. However, collecting and converting real research tasks is difficult, and some benchmarks define their own tasks\([Zhu et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib61)\)\. SimulCost asks agents to tune the parameters of standard physics solvers\([Cao et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib5)\)\. gwBenchmarks defines eight gravitational\-wave research tasks, from regression to physics derivation\([Islam et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib22)\)\. A better approach is to select published, peer\-reviewed research and convert it into benchmark tasks\([Qiu et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib40);[Majumder et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib34);[Deng et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib11)\)\. ScienceAgentBench adapts 102 tasks from 44 peer\-reviewed publications\([Chen et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib9)\)\. CORE\-Bench builds tasks from the released code of 90 papers\([Siegel et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib43)\)\. PaperBench asks agents to replicate 20 ICML 2024 papers\([Starace et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib44)\)\. ReplicationBench selects 20 peer\-reviewed astrophysics papers with their authors’ collaboration\([Ye et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib58)\)\. FIRE\-Bench further decomposes each paper into a tree of research problems\([Wang et al\., 2026b](https://arxiv.org/html/2610.00636#bib.bib51)\)\. CompMat\-Bench converts peer\-reviewed materials studies, mostly published between January and March 2026, into research steps that keep their scientific dependencies\. It reproduces the expensive simulations of these studies and uses the results as ground truth, so that the ground truth is consistent with the task settings and includes the details that papers omit\.

#### Materials science benchmarks\.

Although the number of materials science benchmarks keeps growing\([Zaki et al\., 2024](https://arxiv.org/html/2610.00636#bib.bib59);[Niyongabo Rubungo et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib35);[Yang et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib57)\), most of them focus on a single type of computational approach or tool\. MatTools tests knowledge and use of the pymatgen library\([Liu et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib32)\)\. INCARBench tests the preparation of VASP INCAR files against Materials Project records\([Shao et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib42)\)\. QMBench tests DFT knowledge, structure\-file generation and parameter choice without running calculations \(free text graded by an LLM\)\([Wang et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib49)\)\. DFTBench runs Quantum ESPRESSO workflows in every attempt\([Hu et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib19)\)\. MDGym reruns LAMMPS and GROMACS simulations in every attempt \(for both materials and biomolecular systems\)\([Kumar et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib29)\)\. AutoMat, the closest work, asks agents to reproduce 85 claims from 10 materials papers across several methods and scores them with an LLM judge\([Huang et al\., 2026a](https://arxiv.org/html/2610.00636#bib.bib20)\)\. Its claims are unevenly distributed across these papers, with at least 31 from a single paper \(counted from the released dataset,[Huang et al\., 2026b](https://arxiv.org/html/2610.00636#bib.bib21)\)\. CompMat\-Bench draws its 94 tasks from 15 published studies that use 12 scientific codes, with no study contributing more than 9 tasks, and grades them with fixed rules instead of an LLM judge\.

## 3Benchmark Construction

Here, we describe how CompMat\-Bench is constructed: how tasks are derived from published studies \(Section[3\.1](https://arxiv.org/html/2610.00636#S3.SS1)\), how guidance and task span are controlled \(Section[3\.2](https://arxiv.org/html/2610.00636#S3.SS2)\), and what the resulting tasks cover \(Section[3\.3](https://arxiv.org/html/2610.00636#S3.SS3)\)\.

### 3\.1From studies to tasks

#### Study selection\.

We draw studies from four journals: npj Computational Materials, Physical Review B, ACS Nano and Nano Letters\. We collected the 1,912 articles published from January to March 2026 whose titles and abstracts were available through the OpenAlex API\([Priem et al\., 2022](https://arxiv.org/html/2610.00636#bib.bib39)\)\. We keep studies whose core is a computation on a real material with a checkable result, and exclude experiment\-led, dataset and review papers\. We then require a part of the study that we can afford to reproduce and calculations that are light on the agent’s side but demand scientific reasoning\. We also exclude studies made public before August 2025 \(e\.g\., a study published in January 2026 whose preprint appeared in March 2025\)\([Jain et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib23)\)and studies that rely on closed\-source codes\. These filters leave 72 studies, from which we select 14 by hand, favoring methods and codes not yet covered\. A fifteenth study, published in September 2025, served as the pilot for these criteria\.

#### Decomposition and reproduction\.

We convert each study into tasks in two steps\. First, a language model guided by a structured prompt adapted from FIRE\-Bench\([Wang et al\., 2026b](https://arxiv.org/html/2610.00636#bib.bib51)\)decomposes the study into a directed acyclic graph of research steps, which we check against the paper\. Each step is one calculation that produces a new physical quantity shown in a main\-text figure or table, and each edge records an upstream result that the step uses\. These graphs have 6 to 18 steps per study\. Second, we select a logically connected sub\-graph in which every selected step keeps all of its upstream steps\. Each of the 15 studies contributes 3 to 7 steps, with a median of 5 \(Table[A](https://arxiv.org/html/2610.00636#A1)\)\. Of all 65 steps, 30 need HPC resources for calculations such as DFT, and 35 run within minutes on one machine\. We reproduce these sub\-graphs on HPC clusters and local workstations, and our results serve as the grading references\. They sometimes differ from the published values, mostly because papers do not report some inputs and we have to set them ourselves\.

Figure 2:A single task and a workflow from a study of magnons in antiferromagnetic CrN\([Kim et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib27)\)\. Both panels show the same three stages\. Left: in single task 013 \(SF, SR\), the first two stages \(dashed\) are precomputed, and the agent computes the spin waves from their outputs\. Right: in workflow C06 \(WF, WR\), the agent runs all three stages\. Shaded prompt text is removed in the reduced\-guidance version, and the rest of the prompt is identical\. Bottom: the hidden reference value and tolerance for one graded quantity\.
#### Tasks and grading\.

We build three kinds of tasks\. For a step that needs HPC, the agent either prepares the simulation inputs or analyzes our precomputed outputs \(Fig\.[1](https://arxiv.org/html/2610.00636#S1.F1)\)\. For a step that does not, the agent runs the calculation and analyzes the result\. Fig\.[2](https://arxiv.org/html/2610.00636#S3.F2)shows a concrete example from a study of magnons in CrN, published in Physical Review B\([Kim et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib27)\)\. Its three stages correspond to three tasks: preparing the DFT inputs, deriving exchange constants from our precomputed DFT outputs, and computing the spin\-wave dispersion, including the magnon splitting at the zone boundary\. The left panel of Fig\.[2](https://arxiv.org/html/2610.00636#S3.F2)shows the prompt of the last task\. Its full\-guidance prompt states the scientific target, the method and the required output, and the reduced\-guidance prompt omits how the magnon energies are obtained, the Holstein\-Primakoff expansion and the Bogoliubov diagonalization \(shaded\)\. The hidden reference stores our value for each graded quantity with its tolerance, here the zone\-boundary magnon splitting \(6\.1272 meV, within 10%\)\. Input files are graded on the effective value of each required setting, including software defaults, and on structures equivalent to the reference\. Analysis results must match our reference within numerical tolerances, 10% relative by default\. A task scores 1 only when every check passes\. Tasks have 1 to 55 checks, with a median of 12\. Prompts are written in our own words and structures are regenerated from public databases, so no text, figures or values are copied from the source papers\.

#### Quality control\.

Each task is reviewed by a computational materials researcher with a PhD in a related field, who grades it on four questions: \(1\) answer leakage, whether any file given to the agent reveals the answer, the method or a shortcut; \(2\) prompt clarity, whether the prompt defines the task and its valid answers without ambiguity; \(3\) reference quality, whether the reference answer or input set is physically reasonable; and \(4\) scoring fairness, whether an essentially correct answer could fail\.

### 3\.2Guidance and task span

Our construction lets us assemble variants of the benchmark\. We vary methodological guidance through the level of detail in single\-task prompts, and task span by combining single tasks into workflows\. Crossing them gives the four conditions SF, SR, WF and WR introduced in Section[1](https://arxiv.org/html/2610.00636#S1)\.

#### Full and reduced guidance\.

The full\-guidance prompt states the scientific target, the inputs, the method and the required output\. The reduced\-guidance prompt removes method guidance that the grader does not check \(Fig\.[2](https://arxiv.org/html/2610.00636#S3.F2), shaded\) and keeps everything else, so both prompts share one grader\. In 42 tasks the full prompt contains no such guidance, and the two prompts are identical\. Of the other 52 tasks, 34 lose explicit method or implementation guidance and 18 lose explanations or hints\.

#### Single tasks and workflows\.

Because the selected steps keep their scientific dependencies, we can chain single tasks from one study into a workflow that takes the agent from the initial inputs to the final result \(Fig\.[2](https://arxiv.org/html/2610.00636#S3.F2), right\)\. We build eight workflows from eight studies \(Table[C](https://arxiv.org/html/2610.00636#A3)\)\. Seven have three stages, and one has four\. Unlike single tasks, which cover one step each, a workflow requires the agent to run every calculation itself, including simulations in seven of the eight workflows, within a five\-hour limit on one node\. Each workflow has full\- and reduced\-guidance prompts, and fixed rules grade its final quantities against our reference\.

### 3\.3Composition

Figure 3:Composition of the 94 tasks\. Left: tasks by computational method\. Right: tasks by the scientific code they involve\. Shading gives the kind of task, as in Fig\.[1](https://arxiv.org/html/2610.00636#S1.F1)\.CompMat\-Bench covers different journals, computational methods and scientific codes\. The 15 source studies appeared in Physical Review B \(11\), Nano Letters \(2\) and npj Computational Materials \(2\)\. Their 94 tasks span 11 method families and 12 scientific codes \(Fig\.[3](https://arxiv.org/html/2610.00636#S3.F3)\)\. VASP\([Kresse & Furthmüller, 1996](https://arxiv.org/html/2610.00636#bib.bib28)\)\(28 tasks\), Quantum ESPRESSO\([Giannozzi et al\., 2009](https://arxiv.org/html/2610.00636#bib.bib15)\)\(11\) and LAMMPS\([Thompson et al\., 2022](https://arxiv.org/html/2610.00636#bib.bib47)\)\(10\) are the most common codes, and in 30 tasks the agent writes the calculation in Python without a named code\. The tasks comprise 30 input\-preparation, 29 output\-analysis and 35 run\-and\-analyze tasks, and no study contributes more than 9\.

## 4Evaluation Setup

#### Agents\.

We evaluate four agents built on three LLMs\. DeepSeek V4\.1 Flash\([DeepSeek\-AI, 2026](https://arxiv.org/html/2610.00636#bib.bib10)\), Gemini 3\.8 Flash\([Google DeepMind, 2026](https://arxiv.org/html/2610.00636#bib.bib16)\)and GPT\-5\.6 Sol\([OpenAI, 2026](https://arxiv.org/html/2610.00636#bib.bib37)\)each run in the MatClaw harness\([Zhang & Yakobson, 2026](https://arxiv.org/html/2610.00636#bib.bib60)\), and GPT\-5\.6 Sol also runs in OpenAI’s Codex CLI\([OpenAI, 2025](https://arxiv.org/html/2610.00636#bib.bib36)\)\. In MatClaw, all three models use high reasoning effort and, on single tasks, at most 500 agent steps \(100 for DeepSeek V4\.1 Flash\), with an 800k\-token context for DeepSeek V4\.1 Flash and 256k tokens for Gemini 3\.8 Flash and GPT\-5\.6 Sol\. Codex CLI also uses high reasoning effort and a 256k\-token context, with no step limit \(Table[B](https://arxiv.org/html/2610.00636#A2)\)\.

#### Run conditions\.

The agent works in an isolated container, with a new session for every run and no internet access except the model API\. It gets the task files and offline user manuals for the simulation codes the task uses, and cannot see reference answers or graders\. For a single task, it has 4 CPU cores, 16 GiB of memory \(6 GiB for 85 of the 94 DeepSeek V4\.1 Flash runs under SF\) and one hour, and expensive simulations are precomputed\. For a workflow, the agent runs all calculations itself on one compute node with two 64\-core AMD EPYC 7763 processors and 239 GiB of memory, and must finish within five hours and 5,000 steps\.

#### Evaluations and scoring\.

We run the following evaluations: \(1\) SF: each of the four agents runs all 94 tasks once, and each MatClaw agent runs a selected subset of 20 tasks five more times to measure the reliability of its performance; \(2\) SR: each MatClaw agent runs once the 52 tasks whose reduced prompt removes method guidance or explanations; \(3\) WF and WR: each MatClaw agent runs each of the eight workflows under both guidance levels in three rounds, 24 runs per agent and condition\. We report the pass rate, the fraction of runs whose submission meets every check of the task’s grading rules \(Section[3\.1](https://arxiv.org/html/2610.00636#S3.SS1)\); for a workflow, the checks apply to its final result\.

## 5Results and Analysis

### 5\.1All agents pass most single tasks under full guidance

All four agents pass most single tasks under full guidance, with pass rates ranging from 66\.0 to 91\.5% \(Table[2](https://arxiv.org/html/2610.00636#S5.T2)and Fig\.[C1](https://arxiv.org/html/2610.00636#A3.F1)\)\. GPT\-5\.6 Sol passes 90\.4% in MatClaw and 91\.5% in Codex CLI, so the two harnesses give nearly the same result for the same model\. Input preparation is the hardest kind of task for every agent, with pass rates of 36\.7 to 76\.7%, whereas the three stronger agents pass 96\.6 to 100% of the output\-analysis and run\-and\-analyze tasks\. At standard vendor prices, one run over the 94 tasks costs 3\.86 USD for DeepSeek V4\.1 Flash, 27\.00 USD for GPT\-5\.6 Sol and 106\.43 USD for Gemini 3\.8 Flash\. GPT\-5\.6 Sol thus matches Gemini 3\.8 Flash in pass rate at a quarter of the cost, despite a higher price per token\. The high cost of Gemini 3\.8 Flash comes from its long runs: it takes a median of 56 steps per task, against 4 for GPT\-5\.6 Sol, and uses 48 times as many input tokens\.

To test whether one run per task is enough, we also run repeated tests with the three MatClaw agents on a selected subset of tasks \(five runs on 20 tasks; Table[3](https://arxiv.org/html/2610.00636#S5.T3)\)\. The ranking of Table[2](https://arxiv.org/html/2610.00636#S5.T2)is robust to repeated runs: DeepSeek V4\.1 Flash, Gemini 3\.8 Flash and GPT\-5\.6 Sol pass 62, 88 and 93 of the 100 runs, in the same order as in their single runs\. The failures of Gemini 3\.8 Flash and GPT\-5\.6 Sol are mostly tied to specific tasks rather than to chance between runs, since only 2 to 3 of the 20 tasks give mixed results, passing some but not all five runs, whereas most failures of DeepSeek V4\.1 Flash vary between runs, with mixed results on 11 tasks\.

Table 2:Pass rates of four agents on the 94 single tasks with full guidance, one run per task\. API cost is the cost of the 94 runs at each vendor’s standard list price; the Codex CLI run was billed by subscription and has no recorded cost\.Pass rate \(%\)PrepareAnalyzeRun andAllAPI costModelHarnessinputsoutputsanalyzetasks\(USD\)n=30n=30n=29n=29n=35n=35n=94n=94DeepSeek V4\.1 FlashMatClaw36\.779\.380\.066\.0 \(62/94\)3\.86Gemini 3\.8 FlashMatClaw70\.096\.6100\.089\.4 \(84/94\)106\.43GPT\-5\.6 SolMatClaw76\.796\.697\.190\.4 \(85/94\)27\.00GPT\-5\.6 SolCodex CLI76\.7100\.097\.191\.5 \(86/94\)–Table 3:Repeatability of three agents on a selected subset of 20 single tasks with full guidance, five runs per task\. The pass rate is over the 100 runs, and the remaining columns count the tasks by how many of the five runs passed\.
### 5\.2Longer workflows limit the weaker agents, and the stronger agent falls only on workflows with reduced guidance

Fig\.[4](https://arxiv.org/html/2610.00636#S5.F4)compares the pass rates of three agents based on different LLMs under four conditions: SF, SR, WF and WR\. In WF and WR, each of the eight workflows is run three times \(24 runs per condition\)\. In SF and SR, we cover the 52 of the 94 tasks that have a reduced prompt, so that the two guidance levels are compared on the same tasks\. From Fig\.[4](https://arxiv.org/html/2610.00636#S5.F4), we find two conclusions common to all agents\. \(1\) Longer task span and reduced guidance do not raise the pass rate beyond noise, and mostly lower it\. Compared with SF, WF lowers the pass rates of DeepSeek V4\.1 Flash and Gemini 3\.8 Flash by 21 and 11 points, and SR lowers them by 9 and 7 points, whereas GPT\-5\.6 Sol stays within 4 points of SF under both\. \(2\) WR is the hardest condition, with pass rates of 50, 54 and 75% for DeepSeek V4\.1 Flash, Gemini 3\.8 Flash and GPT\-5\.6 Sol \(for DeepSeek V4\.1 Flash, the WR pass rate exceeds the WF pass rate by two runs, within statistical noise\)\.

Figure 4:Pass rates of three agents under the four evaluation conditions of Section[3\.2](https://arxiv.org/html/2610.00636#S3.SS2)\. Each cell gives the pass rate and the count behind it\. The single\-task column covers the 52 tasks whose reduced prompt differs from the full prompt, and the workflow column covers the eight workflows with three runs each\. TheΔ\\Deltacells give differences in percentage points: reduced minus full guidance \(bottom row\) and workflow minus single task \(right column, different task sets\)\.All three agents perform worse under WR than under SF, but for different reasons: the weaker agents lose on longer workflows, and the stronger agent only on the combination with reduced guidance\. For DeepSeek V4\.1 Flash, longer workflows alone reduce its performance markedly: the pass rate decreases by 21 points from SF to WF, both under full guidance\. We speculate that this agent has difficulty maintaining consistency over long trajectories \(11 of its 26 failed workflow runs reach the five\-hour limit\)\. In contrast, for GPT\-5\.6 Sol, the pass rates under WF and SR are nearly identical to that under SF, and the pass rate decreases, by 13 points, only when a long task span is combined with reduced guidance, which requires the agent to plan and carry out the multi\-step calculation itself\. Gemini 3\.8 Flash falls between these two cases: longer workflows and reduced guidance lower its pass rate by 11 and 7 points, respectively\. Their combination lowers it by 36 points, far more than either alone, and its WR pass rate \(54%\) is close to that of DeepSeek V4\.1 Flash \(50%\)\.

### 5\.3Workflow failures are mostly scientific, and all agents share some interface errors

We classify each failed run into three groups \(Fig\.[5](https://arxiv.org/html/2610.00636#S5.F5)\): scientific errors, either a wrong physical choice \(for example, a missing dipole correction for a polar slab\) or a correct choice implemented wrongly \(for example, a sign error in a Bloch phase\); operational errors, either wrong software usage \(for example, an input the code cannot parse\) or a format or rule violation \(for example, a missing answer field\); and runs that reach the time limit or remain unclassified\. In workflows, 38 of the 50 failed runs are scientific errors, and 11 of the other 12 are time limits of DeepSeek V4\.1 Flash\. In single tasks, scientific and operational errors are about equal, 42 and 44 of 90\. All 18 software\-usage errors occur in input\-preparation tasks, where the agent writes the inputs without running the software and therefore receives no error message, whereas in a workflow the agent runs each code and can correct an input that fails\.

Figure 5:Failure analysis: failed runs of three agents by cause under the four evaluation conditions of Section[3\.2](https://arxiv.org/html/2610.00636#S3.SS2)\. DeepSeek, Gemini and GPT denote DeepSeek V4\.1 Flash, Gemini 3\.8 Flash and GPT\-5\.6 Sol\. Each cell gives the number of failed runs and their share of all runs in that row\.In\-depth analysis suggests that agents fail to diagnose their errors because they do not recognize anomalies that an experienced researcher would notice\. In the CrN workflow of Fig\.[2](https://arxiv.org/html/2610.00636#S3.F2), 9 of the 15 failing runs use inappropriate settings for the Wannier construction\. Their Wannier spreads reach 105 to 215 Å2, four to eight times the 25 to 28 Å2of the passing runs, or the disentanglement does not converge, and the exchange constants computed from them deviate by up to 116%\. In each of these nine runs, the anomaly is printed in the agent’s own Wannier90 output, yet the agent continues to the exchange calculation without correcting the settings\. We speculate that such diagnostic knowledge is implicit: it is rarely documented in the literature and is instead acquired through research practice, so it is poorly represented in the training data of language models\.

Some software interfaces are difficult for all agents\. For example, one task asks for the LAMMPS inputs of a WS2/WSe2moiré bilayer with six atom types, and the line that sets the interaction between the two layers must give an element name for each of the six types\. The offline LAMMPS manual is available to the agents, but it states no rule for the number of names and its example uses two atom types, and every agent keeps this form, with two element names instead of six\. In the repeated runs, all 15 trials make the same mistake\. Because the task asks for inputs without running LAMMPS, no error message reveals the mistake\. Agents also fail when a calculation needs a setting beyond the code defaults, such as the van der Waals correction for a bilayer, which all four agents omit\. Like the diagnostic knowledge above, the knowledge these interfaces require is implicit: the documentation available to the agents does not state it, so agents based on different LLMs fail in the same way\.

## 6Limitations

CompMat\-Bench has the following limitations: the strongest current agents pass most single tasks under full guidance, which leaves little room to distinguish stronger models, and the conditions that still separate them, workflows and reduced guidance, have fewer runs; fixed\-rule grading can wrongly reject a correct answer, and although our failure analysis found and fixed several such cases, others may remain; and the tasks do not cover all computational materials research, for example studies that propose new ideas or concepts, which are excluded at selection\. We plan to add tasks to the workflow and reduced\-guidance conditions in later versions\.

## 7Conclusion

We introduce CompMat\-Bench, a benchmark of 94 tasks for evaluating AI agents on recently published computational materials research: agents are assessed on preparing the inputs and analyzing the outputs of expensive simulations, which we reproduced in advance, rather than on running them during evaluation\. With full methodological guidance, agents based on three LLMs pass 66\.0–90\.4% of the single tasks\. Varying the task span and the level of methodological guidance shows that both factors lower the pass rates of the weaker agents, whereas the strongest agent falls only when a long workflow is combined with reduced guidance\. Most failures are scientific errors, and in\-depth analysis reveals common failure scenarios: agents do not recognize anomalies in their own results that an experienced researcher would notice, and agents based on different LLMs lack the same implicit knowledge of software interfaces\. CompMat\-Bench provides a basis for comparing agents on these steps of real materials research and for analyzing their failure modes\.

## Reproducibility Statement

The tasks and workflows, with their prompts, input files, graders, reference answers and offline reference library, are provided as supplementary material\. Appendices[A](https://arxiv.org/html/2610.00636#A1)to[E](https://arxiv.org/html/2610.00636#A5)list the source studies and tasks, the agent configuration and software versions, the workflows and complete results, and the prompts\. The intermediate files of our reproductions, about 1 TB, are not released\.

## AI Use Statement

In this work, we used generative AI tools for decomposing the source studies, drafting task prompts and graders, and failure analysis\. We have not used generative AI tools for grading or for the final scientific decisions, including the acceptance of each task, its reference answer and the conclusions drawn; other tasks with required disclosure are not applicable to this work\. Additionally, we used generative AI tools for drafting and editing the manuscript\. We have reviewed all AI\-assisted work and take responsibility for its final content, including text, claims or artifacts produced with the aid of generative AI\.

## Acknowledgements

This work was supported by the U\.S\. AFOSR grant FA9550\-24\-1\-0093\.

## References

- Anand et al\. \(2026\)Nithishwer Mouroug Anand, Wei\-Tse Hsu, Kyle Vaccaro, Eden James Gage, Jonathan David Colburn, Linda Xi Phan, Minjoon Seo, Kevin Guan, and Philip C\. Biggin\.MDArena: Evaluating coding agents on realistic molecular dynamics workflows\.*arXiv preprint arXiv:2608\.02642*, 2026\.URL[https://arxiv\.org/abs/2608\.02642v1](https://arxiv.org/abs/2608.02642v1)\.
- Bogin et al\. \(2024\)Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot\.SUPER: Evaluating agents on setting up and executing tasks from research repositories\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 12622–12645, 2024\.URL[https://arxiv\.org/abs/2409\.07440v1](https://arxiv.org/abs/2409.07440v1)\.
- Boiko et al\. \(2023\)Daniil A\. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes\.Autonomous chemical research with large language models\.*Nature*, 624:570–578, 2023\.doi:10\.1038/s41586\-023\-06792\-0\.
- Camps et al\. \(2025\)Daan Camps, Ermal Rrapaj, Katherine Klymko, Hyeongjin Kim, Kevin Gott, Siva Darbha, Jan Balewski, Brian Austin, and Nicholas J\. Wright\.Quantum computing technology roadmaps and capability assessment for scientific computing: An analysis of use cases from the NERSC workload\.Technical Report LBNL\-2001699, Lawrence Berkeley National Laboratory, September 2025\.URL[https://arxiv\.org/abs/2509\.09882v1](https://arxiv.org/abs/2509.09882v1)\.
- Cao et al\. \(2026\)Yadi Cao, Sicheng Lai, Jiahe Huang, Yang Zhang, Zach Lawrence, Rohan Bhakta, Izzy F\. Thomas, Mingyun Cao, Chung\-Hao Tsai, Zihao Zhou, Yidong Zhao, Hao Liu, Alessandro Marinoni, Alexey Arefiev, and Rose Yu\.SimulCost: A cost\-aware benchmark and toolkit for automating physics simulations with LLMs\.In*International Conference on Machine Learning*, 2026\.URL[https://arxiv\.org/abs/2603\.20253v4](https://arxiv.org/abs/2603.20253v4)\.
- Chak et al\. \(2026\)Kyaw Hla Saing Chak, Bipin Bhattarai, Andrew C\. Meng, and Yijia Gu\.Integrated thermodynamic modeling of composition and strain tunable ferroelectricity in wurtzite Zn1\-xMgxO\.*npj Computational Materials*, 12:154, 2026\.doi:10\.1038/s41524\-026\-02021\-0\.URL[https://doi\.org/10\.1038/s41524\-026\-02021\-0](https://doi.org/10.1038/s41524-026-02021-0)\.
- Chan et al\. \(2025\)Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry\.MLE\-bench: Evaluating machine learning agents on machine learning engineering\.In*International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2410\.07095v6](https://arxiv.org/abs/2410.07095v6)\.
- Chen et al\. \(2021\)Mark Chen et al\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.URL[https://arxiv\.org/abs/2107\.03374](https://arxiv.org/abs/2107.03374)\.
- Chen et al\. \(2025\)Ziru Chen et al\.ScienceAgentBench: Toward rigorous assessment of language agents for data\-driven scientific discovery\.In*International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2410\.05080v3](https://arxiv.org/abs/2410.05080v3)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AI\.DeepSeek\-V4\.1\-Flash: Pushing the limits of KV cache compression\.*arXiv preprint arXiv:2609\.19969*, 2026\.URL[https://arxiv\.org/abs/2609\.19969v1](https://arxiv.org/abs/2609.19969v1)\.
- Deng et al\. \(2026\)Ken Deng, Xiangfei Wang, Guijing Duan, Chen Mo, Junkun Huang, Runqing Zhang, Ling Qian, Zhiguo Huang, Jize Han, and Di Luo\.Towards verifiable and self\-correcting AI physicists for quantum many\-body simulations\.*arXiv preprint arXiv:2604\.00149*, 2026\.URL[https://arxiv\.org/abs/2604\.00149v2](https://arxiv.org/abs/2604.00149v2)\.
- Devaraj et al\. \(2026\)Nayana Devaraj, Anumita Bose, Arindom Das, Md Afsar Reja, Arijit Mandal, Awadhesh Narayan, and B\. R\. K\. Nanda\.Unlocking doping effects on altermagnetism in MnTe: Emergence of quasi\-altermagnetism\.*Physical Review B*, 113:104438, 2026\.doi:10\.1103/k36v\-91br\.URL[https://doi\.org/10\.1103/k36v\-91br](https://doi.org/10.1103/k36v-91br)\.
- Faroughy et al\. \(2026\)Darius A\. Faroughy, Sofia Palacios Schweitzer, Ian Pang, Siddharth Mishra\-Sharma, and David Shih\.Collider\-Bench: Benchmarking AI agents with particle physics analysis reproduction\.*arXiv preprint arXiv:2605\.13950*, 2026\.URL[https://arxiv\.org/abs/2605\.13950v1](https://arxiv.org/abs/2605.13950v1)\.
- Ghafarollahi & Buehler \(2025\)Alireza Ghafarollahi and Markus J\. Buehler\.Automating alloy design and discovery with physics\-aware multimodal multiagent AI\.*Proceedings of the National Academy of Sciences*, 122:e2414074122, 2025\.doi:10\.1073/pnas\.2414074122\.
- Giannozzi et al\. \(2009\)Paolo Giannozzi, Stefano Baroni, Nicola Bonini, Matteo Calandra, Roberto Car, Carlo Cavazzoni, Davide Ceresoli, Guido L\. Chiarotti, Matteo Cococcioni, Ismaila Dabo, Andrea Dal Corso, Stefano de Gironcoli, Stefano Fabris, Guido Fratesi, Ralph Gebauer, Uwe Gerstmann, Christos Gougoussis, Anton Kokalj, Michele Lazzeri, Layla Martin\-Samos, Nicola Marzari, Francesco Mauri, Riccardo Mazzarello, Stefano Paolini, Alfredo Pasquarello, Lorenzo Paulatto, Carlo Sbraccia, Sandro Scandolo, Gabriele Sclauzero, Ari P\. Seitsonen, Alexander Smogunov, Paolo Umari, and Renata M\. Wentzcovitch\.QUANTUM ESPRESSO: A modular and open\-source software project for quantum simulations of materials\.*Journal of Physics: Condensed Matter*, 21:395502, 2009\.doi:10\.1088/0953\-8984/21/39/395502\.
- Google DeepMind \(2026\)Google DeepMind\.Gemini 3\.8 Flash model card\.[https://deepmind\.google/models/model\-cards/gemini\-3\-8\-flash/](https://deepmind.google/models/model-cards/gemini-3-8-flash/), 2026\.Model card published 2026\-09\-02; PDF at[https://storage\.googleapis\.com/deepmind\-media/Model\-Cards/Gemini\-3\-8\-Flash\-Model\-Card\.pdf](https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-8-Flash-Model-Card.pdf)\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\.Measuring massive multitask language understanding\.In*International Conference on Learning Representations*, 2021\.URL[https://arxiv\.org/abs/2009\.03300](https://arxiv.org/abs/2009.03300)\.
- Hofer et al\. \(2026\)Maximilian Hofer, Christopher Fuchs, Moritz Siebert, Christian Berger, Lena Fürst, Martin P\. Stehno, Steffen Schreyeck, Hartmut Buhmann, Tobias Kießling, Wouter Beugeling, and Laurens W\. Molenkamp\.Self\-consistent electrostatic modeling of gated narrow\-gap topological insulators\.*Nano Letters*, 26:3042–3048, 2026\.doi:10\.1021/acs\.nanolett\.5c05762\.URL[https://doi\.org/10\.1021/acs\.nanolett\.5c05762](https://doi.org/10.1021/acs.nanolett.5c05762)\.
- Hu et al\. \(2026\)Zhengding Hu, Kuntal Talit, Zhen Wang, Haseeb Ahmad, Yichen Lin, Prabhleen Kaur, Christopher Lane, Elizabeth A\. Peterson, Zhiting Hu, Elizabeth A\. Nowadnick, and Yufei Ding\.TritonDFT: Automating DFT with a multi\-agent framework\.*arXiv preprint arXiv:2603\.03372*, 2026\.URL[https://arxiv\.org/abs/2603\.03372v2](https://arxiv.org/abs/2603.03372v2)\.
- Huang et al\. \(2026a\)Ziyang Huang, Yi Cao, Ali K\. Shargh, Jing Luo, Ruidong Mei, Mohd Zaki, Zhan Liu, Wyatt Bunstine, William Jurayj, Somdatta Goswami, Tyrel McQueen, Michael Shields, Jaafar El\-Awady, Paulette Clancy, Benjamin Van Durme, Nicholas Andrews, William Walden, and Daniel Khashabi\.Can coding agents reproduce findings in computational materials science?In*Conference on Language Modeling*, 2026a\.URL[https://arxiv\.org/abs/2605\.00803v2](https://arxiv.org/abs/2605.00803v2)\.
- Huang et al\. \(2026b\)Ziyang Huang, Yi Cao, Ali K\. Shargh, Jing Luo, Ruidong Mei, Mohd Zaki, Zhan Liu, Wyatt Bunstine, William Jurayj, Somdatta Goswami, Tyrel McQueen, Michael Shields, Jaafar El\-Awady, Paulette Clancy, Benjamin Van Durme, Nicholas Andrews, William Walden, and Daniel Khashabi\.AutoMat dataset\.Hugging Face dataset jhu\-clsp/AutoMat, revision 58d2fe48,[https://huggingface\.co/datasets/jhu\-clsp/AutoMat](https://huggingface.co/datasets/jhu-clsp/AutoMat), 2026b\.
- Islam et al\. \(2026\)Tousif Islam, Digvijay Wadekar, and Zihan Zhou\.gwBenchmarks: Stress\-testing LLM agents on high\-precision gravitational wave astronomy\.*arXiv preprint arXiv:2605\.11269*, 2026\.URL[https://arxiv\.org/abs/2605\.11269v2](https://arxiv.org/abs/2605.11269v2)\.
- Jain et al\. \(2025\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\.LiveCodeBench: Holistic and contamination free evaluation of large language models for code\.In*International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2403\.07974v2](https://arxiv.org/abs/2403.07974v2)\.
- Jiang et al\. \(2026\)Xuance Jiang, Sayed Ali Akbar Ghorashi, Deyu Lu, and Jennifer Cano\.Altermagnetism induced surface Chern insulator\.*Nano Letters*, 26:1327–1333, 2026\.doi:10\.1021/acs\.nanolett\.5c05341\.URL[https://doi\.org/10\.1021/acs\.nanolett\.5c05341](https://doi.org/10.1021/acs.nanolett.5c05341)\.
- Jimenez et al\. \(2024\)Carlos E\. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan\.SWE\-bench: Can language models resolve real\-world GitHub issues?In*International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2310\.06770](https://arxiv.org/abs/2310.06770)\.
- Kapoor et al\. \(2025\)Sayash Kapoor, Benedikt Stroebl, Zachary S\. Siegel, Nitya Nadgir, and Arvind Narayanan\.AI agents that matter\.*Transactions on Machine Learning Research*, 2025\.URL[https://arxiv\.org/abs/2407\.01502v1](https://arxiv.org/abs/2407.01502v1)\.
- Kim et al\. \(2026\)Seo\-Jin Kim, Zdeněk Jirák, Jiří Hejtmánek, Karel Knížek, Helge Rosner, and Kyo\-Hoon Ahn\.Semiclassical model of magnons in double\-layered antiferromagnets\.*Physical Review B*, 113:144429, 2026\.doi:10\.1103/gj66\-4yhj\.URL[https://doi\.org/10\.1103/gj66\-4yhj](https://doi.org/10.1103/gj66-4yhj)\.
- Kresse & Furthmüller \(1996\)G\. Kresse and J\. Furthmüller\.Efficient iterative schemes for ab initio total\-energy calculations using a plane\-wave basis set\.*Physical Review B*, 54:11169–11186, 1996\.doi:10\.1103/PhysRevB\.54\.11169\.
- Kumar et al\. \(2026\)Vinay Kumar, Satyendra Rajput, Mausam, and N\. M\. Anoop Krishnan\.MDGYM: Benchmarking AI agents on molecular simulations\.*arXiv preprint arXiv:2605\.08941*, 2026\.URL[https://arxiv\.org/abs/2605\.08941v1](https://arxiv.org/abs/2605.08941v1)\.
- Li et al\. \(2025\)Xingfu Li, Sunny Gupta, and Boris I\. Yakobson\.Undulation\-induced moiré superlattices with one\-dimensional polarization domains and flat bands in two\-dimensional bilayer semiconductors\.*Physical Review B*, 112:115308, 2025\.doi:10\.1103/bxw4\-7y5c\.URL[https://doi\.org/10\.1103/bxw4\-7y5c](https://doi.org/10.1103/bxw4-7y5c)\.
- Li et al\. \(2026\)Yifei Li, Yichong Chen, Guihua Tang, Livio Gibelli, and Matthew K\. Borg\.Scattering matrix formalism for interface heat conduction based on Monte Carlo evaluation of the phonon Boltzmann transport equation\.*Physical Review B*, 113:205304, 2026\.doi:10\.1103/kz9s\-y611\.URL[https://doi\.org/10\.1103/kz9s\-y611](https://doi.org/10.1103/kz9s-y611)\.
- Liu et al\. \(2025\)Siyu Liu, Bo Hu, Beilin Ye, Jiamin Xu, David J\. Srolovitz, and Tongqi Wen\.MatTools: Benchmarking large language models for materials science tools\.*arXiv preprint arXiv:2505\.10852*, 2025\.URL[https://arxiv\.org/abs/2505\.10852v2](https://arxiv.org/abs/2505.10852v2)\.
- Lu et al\. \(2024\)Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha\.The AI scientist: Towards fully automated open\-ended scientific discovery\.*arXiv preprint arXiv:2408\.06292*, 2024\.URL[https://arxiv\.org/abs/2408\.06292v3](https://arxiv.org/abs/2408.06292v3)\.
- Majumder et al\. \(2025\)Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark\.DiscoveryBench: Towards data\-driven discovery with large language models\.In*International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2407\.01725v1](https://arxiv.org/abs/2407.01725v1)\.
- Niyongabo Rubungo et al\. \(2025\)Andre Niyongabo Rubungo, Kangming Li, Jason Hattrick\-Simpers, and Adji Bousso Dieng\.LLM4Mat\-Bench: Benchmarking large language models for materials property prediction\.*Machine Learning: Science and Technology*, 6\(2\):020501, 2025\.doi:10\.1088/2632\-2153/add3bb\.
- OpenAI \(2025\)OpenAI\.Codex CLI: A lightweight coding agent that runs in your terminal\.[https://github\.com/openai/codex](https://github.com/openai/codex), 2025\.Official GitHub repository; no CITATION\.cff or paper is provided; initial public release 2025\-04\-16\.
- OpenAI \(2026\)OpenAI\.GPT\-5\.6 system card\.[https://deploymentsafety\.openai\.com/gpt\-5\-6](https://deploymentsafety.openai.com/gpt-5-6), 2026\.Published 2026\-07\-09; covers the Sol, Terra and Luna models of the GPT\-5\.6 family\.
- Parlak et al\. \(2026\)Selçuk Parlak, Abhishek Kumar, Runhan Li, Maia G\. Vergniory, and Ion Garate\.Tight\-binding and density\-functional study of the Raman tensor in two\-dimensional massive Dirac fermion systems\.*Physical Review B*, 113:085431, 2026\.doi:10\.1103/m1p1\-w3w4\.URL[https://doi\.org/10\.1103/m1p1\-w3w4](https://doi.org/10.1103/m1p1-w3w4)\.
- Priem et al\. \(2022\)Jason Priem, Heather Piwowar, and Richard Orr\.OpenAlex: A fully\-open index of scholarly works, authors, venues, institutions, and concepts\.*arXiv preprint arXiv:2205\.01833*, 2022\.URL[https://arxiv\.org/abs/2205\.01833v2](https://arxiv.org/abs/2205.01833v2)\.
- Qiu et al\. \(2026\)Shi Qiu, Junyi Deng, Yiwei Deng, Haoran Dong, Jieyu Fu, Mao Li, Zeyu Li, Zhaolong Zhang, Huiwen Zheng, Leidong Bao, Anqi Lv, Zihan Mo, Yadi Niu, Yiyang Peng, Yu Tian, Yili Wang, Ziyu Wang, Zi\-Yu Wang, Jiashen Wei, Liuheng Wu, Aoran Xue, Leyi Yang, Guanglu Yuan, Xiarui Zhan, Jingjun Zhang, Zifan Zheng, Pengfei Liu, Linrui Zhen, Kaiyang Li, Qichang Li, Ziheng Zhou, Guo\-En Nian, Yunwei Xiao, Qing\-Hong Cao, Linjie Dai, Xu Feng, Peng Gao, Ying Gu, Chang Liu, Jia Liu, Ming\-xing Luo, Yan\-Qing Ma, Liang\-You Peng, Huichao Song, Shufeng Wang, Chenxu Wang, Tao Wang, Yi\-Nan Wang, Chengyin Wu, Pengwei Zhao, and Hua Xing Zhu\.PRBench: End\-to\-end paper reproduction in physics research\.*arXiv preprint arXiv:2603\.27646*, 2026\.URL[https://arxiv\.org/abs/2603\.27646v1](https://arxiv.org/abs/2603.27646v1)\.
- Rein et al\. \(2024\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\.GPQA: A graduate\-level Google\-proof Q&A benchmark\.In*First Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=Ti67584b98](https://openreview.net/forum?id=Ti67584b98)\.
- Shao et al\. \(2026\)Bin Shao, Jixiang Li, Xinyue Zhang, Baishun Yang, Zhiyang Liu, and Weichao Wang\.INCARBench: A benchmark for scientific configuration in VASP INCAR by large language models\.*arXiv preprint arXiv:2606\.23571*, 2026\.URL[https://arxiv\.org/abs/2606\.23571v2](https://arxiv.org/abs/2606.23571v2)\.
- Siegel et al\. \(2024\)Zachary S\. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan\.CORE\-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark\.*Transactions on Machine Learning Research*, 2024\.URL[https://arxiv\.org/abs/2409\.11363v2](https://arxiv.org/abs/2409.11363v2)\.
- Starace et al\. \(2025\)Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan\.PaperBench: Evaluating AI’s ability to replicate AI research\.In*International Conference on Machine Learning*, 2025\.URL[https://arxiv\.org/abs/2504\.01848v3](https://arxiv.org/abs/2504.01848v3)\.
- Takagi et al\. \(2026\)Tsubasa Takagi, Hikaru Watanabe, Ryutaro Yoshimi, Yuki Sato, Shingo Toyoda, Atsushi Tsukazaki, Kei S\. Takahashi, Masashi Kawasaki, Yoshinori Tokura, and Naoki Ogawa\.Quantum geometry in low\-energy linear and nonlinear optical responses of the magnetic Rashba semiconductor \(Ge,Mn\)Te\.*Physical Review B*, 113:104417, 2026\.doi:10\.1103/dp31\-y358\.URL[https://doi\.org/10\.1103/dp31\-y358](https://doi.org/10.1103/dp31-y358)\.
- Taper et al\. \(2026\)Sunil Taper, Bo Cai, Nives Strkalj, Bartomeu Monserrat, and Giuliana Di Martino\.Role of local cation environment on the formation energy of oxygen vacancies in ferroelectric Hf1\-xZrxO2\.*Physical Review B*, 113:134114, 2026\.doi:10\.1103/vz64\-n5xh\.URL[https://doi\.org/10\.1103/vz64\-n5xh](https://doi.org/10.1103/vz64-n5xh)\.
- Thompson et al\. \(2022\)Aidan P\. Thompson, H\. Metin Aktulga, Richard Berger, Dan S\. Bolintineanu, W\. Michael Brown, Paul S\. Crozier, Pieter J\. in ’t Veld, Axel Kohlmeyer, Stan G\. Moore, Trung Dac Nguyen, Ray Shan, Mark J\. Stevens, Julien Tranchida, Christian Trott, and Steven J\. Plimpton\.LAMMPS – a flexible simulation tool for particle\-based materials modeling at the atomic, meso, and continuum scales\.*Computer Physics Communications*, 271:108171, 2022\.doi:10\.1016/j\.cpc\.2021\.108171\.
- Tian et al\. \(2024\)Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, and Hao Peng\.SciCode: A research coding benchmark curated by scientists\.In*Advances in Neural Information Processing Systems \(Datasets and Benchmarks Track\)*, 2024\.URL[https://arxiv\.org/abs/2407\.13168v1](https://arxiv.org/abs/2407.13168v1)\.
- Wang et al\. \(2025\)Yanzhen Wang, Yiyang Jiang, Diana Golovanova, Kamal Das, Hyeonhu Bae, Yufei Zhao, Huu\-Thong Le, Abhinava Chatterjee, Yunzhe Liu, Chao\-Xing Liu, Felipe H\. da Jornada, Binghai Yan, and Xiao\-Liang Qi\.QMBench: A research level benchmark for quantum materials research\.*arXiv preprint arXiv:2512\.19753*, 2025\.URL[https://arxiv\.org/abs/2512\.19753v1](https://arxiv.org/abs/2512.19753v1)\.
- Wang et al\. \(2026a\)Youwen Wang, Nanya Gao, and Qingjun Tong\.Origin of moiré potentials in WS2/WSe2heterobilayers: Contributions from lattice reconstruction and interlayer charge transfer\.*Physical Review B*, 113:085408, 2026a\.doi:10\.1103/hhdc\-yb5s\.URL[https://doi\.org/10\.1103/hhdc\-yb5s](https://doi.org/10.1103/hhdc-yb5s)\.
- Wang et al\. \(2026b\)Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, and Eric P\. Xing\.FIRE\-Bench: Evaluating AI agents on the rediscovery of scientific insights\.In*International Conference on Machine Learning*, 2026b\.URL[https://arxiv\.org/abs/2602\.02905v2](https://arxiv.org/abs/2602.02905v2)\.
- Wijekoon et al\. \(2026\)Himal Wijekoon, Pierre Hirel, and Anna Grünebohm\.Ferroelectric switching at edge dislocations in BaTiO3modeled at the atomic scale\.*Physical Review B*, 113:144108, 2026\.doi:10\.1103/jrql\-63kl\.URL[https://doi\.org/10\.1103/jrql\-63kl](https://doi.org/10.1103/jrql-63kl)\.
- Wu \(2026\)Huan Wu\.Nonperturbative isotope effect on light\-matter interaction in boron arsenide\.*Physical Review B*, 113:134316, 2026\.doi:10\.1103/yqg4\-5gf7\.URL[https://doi\.org/10\.1103/yqg4\-5gf7](https://doi.org/10.1103/yqg4-5gf7)\.
- Xie et al\. \(2026\)Zhenyang Xie, Jitong Song, Yuntao Wu, Yuanji Xu, and Fuyang Tian\.Local magnetic structure in fully and partially ordered V2XAl Heusler alloys \(X = Cr, Mn, Fe, Co, Ni\)\.*Physical Review B*, 113:064410, 2026\.doi:10\.1103/4gsl\-4lxv\.URL[https://doi\.org/10\.1103/4gsl\-4lxv](https://doi.org/10.1103/4gsl-4lxv)\.
- Xu & Delin \(2026\)Qichen Xu and Anna Delin\.A general optimization framework for mapping local transition\-state networks\.*npj Computational Materials*, 12:112, 2026\.doi:10\.1038/s41524\-026\-01985\-3\.URL[https://doi\.org/10\.1038/s41524\-026\-01985\-3](https://doi.org/10.1038/s41524-026-01985-3)\.
- Yang et al\. \(2024\)John Yang, Carlos E\. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press\.SWE\-agent: Agent\-computer interfaces enable automated software engineering\.In*Advances in Neural Information Processing Systems*, 2024\.URL[https://arxiv\.org/abs/2405\.15793v3](https://arxiv.org/abs/2405.15793v3)\.
- Yang et al\. \(2026\)Penghui Yang, Zhonghan Zhang, Yue Li, Xinrun Wang, Yanchen Deng, Yuhao Lu, Bijun Tang, Zheng Liu, and Bo An\.AutoDFT: A closed\-loop multi\-agent framework for autonomous DFT calculations\.*arXiv preprint arXiv:2605\.26179*, 2026\.URL[https://arxiv\.org/abs/2605\.26179v2](https://arxiv.org/abs/2605.26179v2)\.
- Ye et al\. \(2025\)Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L\. V\. Roque, Dalya Baron, Philipp Frank, Sergio Martin\-Alvarez, Nolan Koblischke, Frank J Qu, Diyi Yang, Risa Wechsler, and Ioana Ciuca\.ReplicationBench: Can AI agents replicate astrophysics research papers?*arXiv preprint arXiv:2510\.24591*, 2025\.URL[https://arxiv\.org/abs/2510\.24591v2](https://arxiv.org/abs/2510.24591v2)\.
- Zaki et al\. \(2024\)Mohd Zaki, Jayadeva, Mausam, and N\. M\. Anoop Krishnan\.MaScQA: Investigating materials science knowledge of large language models\.*Digital Discovery*, 3:313–327, 2024\.doi:10\.1039/D3DD00188A\.
- Zhang & Yakobson \(2026\)C\. Zhang and B\. I\. Yakobson\.MatClaw: An autonomous code\-first LLM agent for end\-to\-end materials exploration\.*arXiv preprint arXiv:2604\.02688*, 2026\.
- Zhu et al\. \(2025\)Minhui Zhu, Minyang Tian, Xiaocheng Yang, Tianci Zhou, Lifan Yuan, Penghao Zhu, Eli Chertkov, Shengyan Liu, Yufeng Du, Ziming Ji, Indranil Das, Qingzhi Chen, Junyi Cao, Yufeng Du, Jiabin Yu, Peixue Wu, Jinchen He, Yifan Su, Yikun Jiang, Yujie Zhang, Chang Liu, Ze\-Min Huang, Weizhen Jia, Yunkai Wang, Farshid Jafarpour, Yong Zhao, Xinan Chen, Jessie Shelton, Aaron W\. Young, John Bartolotta, Wenchao Xu, Yue Sun, Anjun Chu, Victor Colussi, Chris Akers, Nathan Brooks, Wenbo Fu, Jinchao Zhao, Marvin Qi, Anqi Mu, Yubo Yang, Allen Zang, Yang Lyu, Peizhi Mai, Christopher Wilson, Xuefei Guo, Juntai Zhou, Daniel Inafuku, Chi Xue, Luyu Gao, Ze Yang, Yaïr Hein, Yonatan Kahn, Kevin Zhou, Di Luo, John Drew Wilson, Jarrod T\. Reilly, Dmytro Bandak, Ofir Press, Liang Yang, Xueying Wang, Hao Tong, Nicolas Chia, Eliu Huerta, and Hao Peng\.Probing the critical point \(CritPt\) of AI reasoning: a frontier physics research benchmark\.*arXiv preprint arXiv:2509\.26574*, 2025\.URL[https://arxiv\.org/abs/2509\.26574v4](https://arxiv.org/abs/2509.26574v4)\.

## Appendix ASource studies and task list

Table[A](https://arxiv.org/html/2610.00636#A1)lists the 15 source studies of Section[3\.1](https://arxiv.org/html/2610.00636#S3.SS1), and Table[A](https://arxiv.org/html/2610.00636#A1)lists the 94 single tasks derived from them\. The study numbers of Table[A](https://arxiv.org/html/2610.00636#A1)identify the source study of each task in Table[A](https://arxiv.org/html/2610.00636#A1)\. Each task is identified by the publication month of its source study and a sequence number within that month, followed by a short name\. For example, single task 013 of Fig\.[2](https://arxiv.org/html/2610.00636#S3.F2)is 202603\_013\.

Table A1:The 15 source studies\. Month is the publication month\. Methods lists the method families of Fig\.[3](https://arxiv.org/html/2610.00636#S3.F3)for the tasks of each study, shortened to their first words\. Graph steps counts the research steps in the full graph of each study, Selected counts the steps in its selected sub\-graph, and Tasks counts the single tasks built from these steps \(Section[3\.1](https://arxiv.org/html/2610.00636#S3.SS1)\)\.No\.StudyJournalMonthMethodsSelectedTasks1\([Li et al\., 2025](https://arxiv.org/html/2610.00636#bib.bib30)\)Phys\. Rev\. B2025\-09DFT, Structure10472\([Jiang et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib24)\)Nano Lett\.2026\-01Model Hamiltonians, DFT7553\([Xie et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib54)\)Phys\. Rev\. B2026\-01DFT, Structure, Spin models10584\([Devaraj et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib12)\)Phys\. Rev\. B2026\-02DFT, Model Hamiltonians, Structure12575\([Hofer et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib18)\)Nano Lett\.2026\-02Model Hamiltonians6346\([Parlak et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib38)\)Phys\. Rev\. B2026\-02Phonons, DFT10367\([Takagi et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib45)\)Phys\. Rev\. B2026\-02Model Hamiltonians13348\([Wang et al\., 2026a](https://arxiv.org/html/2610.00636#bib.bib50)\)Phys\. Rev\. B2026\-02DFT, Atomistic, Continuum18589\([Xu & Delin, 2026](https://arxiv.org/html/2610.00636#bib.bib55)\)npj Comput\. Mater\.2026\-02Atomistic, Spin models73310\([Chak et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib6)\)npj Comput\. Mater\.2026\-03Berry phase, Continuum75711\([Kim et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib27)\)Phys\. Rev\. B2026\-03Spin models, DFT75812\([Li et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib31)\)Phys\. Rev\. B2026\-03Phonon transport, Structure117913\([Taper et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib46)\)Phys\. Rev\. B2026\-03Defects, Bonding, DFT, Structure63614\([Wijekoon et al\., 2026](https://arxiv.org/html/2610.00636#bib.bib52)\)Phys\. Rev\. B2026\-03Atomistic84615\([Wu, 2026](https://arxiv.org/html/2610.00636#bib.bib53)\)Phys\. Rev\. B2026\-03Phonons, Continuum956Total1416594
Table A2:The 94 single tasks\. A dash in Codes marks a task with no named code, where the agent writes the calculation itself in Python\. Checks counts the checks in the grading rules of the task\. QE: Quantum ESPRESSO; phonopy includes phono3py\.TaskShort nameStudyKindCodesChecks202509\_001hbn\-sliding\-sweep\-inputs1input preparationVASP14202509\_002hbn\-sliding\-potential\-step1output analysisVASP4202509\_003hbn\-undulated\-sliding\-ratio1run and analyze–8202509\_004bn\-nanotube\-inputs1input preparationVASP15202509\_005bn\-nanotube\-flexoelectric\-constant1output analysisVASP1202509\_006hbn\-flexo\-sliding\-decomposition\-inputs1input preparationVASP14202509\_007hbn\-undulated\-step\-decomposition1output analysisVASP14202601\_001v2coal\-heusler\-structures3run and analyze–2202601\_002v2coal\-relax\-energy\-inputs3input preparationVASP14202601\_003v2coal\-formation\-energies\-ordering3output analysisVASP10202601\_004v2coal\-l21\-abacus\-inputs3input preparationABACUS12202601\_005v2coal\-l21\-magnetic\-moment3output analysisABACUS2202601\_006v2coal\-xa\-abacus\-inputs3input preparationABACUS13202601\_007v2coal\-xa\-site\-moments3output analysisABACUS3202601\_008v2coal\-heisenberg\-exchange3run and analyzeABACUS, TB2J5202601\_009snte\-110\-surface\-dirac\-cones2run and analyze–1202601\_010ruo2\-snte\-interface\-inputs2input preparationVASP20202601\_011coupled\-slab\-surface\-splittings2run and analyze–2202601\_012surface\-cone\-hall\-contributions2run and analyze–2202601\_013slab\-hall\-interface\-distribution2run and analyze–2202602\_001hgte\-well\-hartree\-inputs5input preparationkdotpy15202602\_002hgte\-well\-hartree\-analysis5output analysis–10202602\_003hgte\-well\-surface\-dispersion5run and analyzekdotpy5202602\_004hgte\-well\-sdh\-densities5run and analyze–10202602\_005gete\-wannier\-inputs7input preparationQE, Wannier9029202602\_006gete\-fermi\-levels7output analysisWannier905202602\_007gete\-interband\-jdos7run and analyzeWannier908202602\_008gete\-optical\-conductivity7run and analyzeWannier905202602\_009rucl2\-dftu\-bands\-inputs6input preparationVASP17202602\_010rucl2\-valley\-gaps6output analysisVASP4202602\_011rucl2\-phonon\-forces\-inputs6input preparationVASP14202602\_012rucl2\-a1p\-phonon\-mode6output analysisVASP, phonopy5202602\_013rucl2\-frozen\-phonon\-optics\-inputs6input preparationVASP17202602\_014rucl2\-magneto\-raman\-tensor6output analysisVASP6202602\_015ws2\-wse2\-moire\-lammps\-inputs8input preparationLAMMPS17202602\_016ws2\-wse2\-moire\-reconstruction\-fields8output analysisLAMMPS17202602\_017ws2\-wse2\-bilayer\-soc\-band\-inputs8input preparationVASP18202602\_018ws2\-wse2\-band\-offsets\-at\-k8output analysisVASP5202602\_019tmd\-monolayer\-strain\-sweep\-inputs8input preparationVASP13202602\_020tmd\-strain\-deformation\-potentials8output analysisVASP6202602\_021moire\-piezoelectric\-bound\-charge8run and analyzeLAMMPS7202602\_022moire\-screened\-piezo\-potential8run and analyze–6202602\_023mnte\-altermagnet\-pipeline\-inputs4input preparationQE17202602\_024mnte\-altermagnet\-spin\-splitting4output analysisQE6202602\_025doped\-mnte\-space\-group\-classification4run and analyze–2202602\_026doped\-mnte\-supercell\-band\-inputs4input preparationQE18202602\_027doped\-mnte\-momentum\-spin\-splitting4output analysisQE16202602\_028tb\-chain\-spin\-splitting4run and analyze–9202602\_029substituted\-tb\-chain\-spectrum4run and analyze–13202602\_030spin\-lattice\-hessian\-minimum\-modes9run and analyze–9202602\_031ni\-island\-eam\-saddle\-points9run and analyze–12202602\_032ni\-island\-mlp\-saddle\-certification9run and analyzeMACE8202603\_001zno\-mgo\-phase\-boundaries10run and analyze–6202603\_002zno\-finite\-strain\-inputs10input preparationQE18202603\_003znmgo\-elastic\-piezoelectric10output analysisQE28202603\_004znmgo\-berry\-phase\-inputs10input preparationQE16202603\_005znmgo\-spontaneous\-polarization10output analysisQE7202603\_006znmgo\-landau\-devonshire\-model10run and analyze–6202603\_007znmgo\-strain\-composition\-phase\-map10run and analyze–6202603\_008afm\-chain\-magnon\-spectrum11run and analyze–2202603\_009crn\-ldau\-structure\-inputs11input preparationVASP14202603\_010crn\-relaxed\-structure\-exchange\-paths11output analysisVASP15202603\_011crn\-wannier\-exchange\-inputs11input preparationVASP, Wannier90, TB2J29202603\_012crn\-nearest\-neighbour\-exchange11output analysisTB2J12202603\_013crn\-linear\-spin\-wave\-spectrum11run and analyze–2202603\_014crn\-monte\-carlo\-deck11input preparationVAMPIRE23202603\_015crn\-neel\-temperature11output analysisVAMPIRE2202603\_016si\-ge\-interface\-cells12run and analyze–4202603\_017si\-ge\-empirical\-phonons12run and analyzeLAMMPS, phonopy16202603\_018si\-ge\-phonon\-force\-inputs12input preparationVASP8202603\_019si\-ge\-abinitio\-phonons12output analysisVASP, phonopy16202603\_020si\-ge\-mode\-resolved\-transmission12run and analyze–45202603\_021si\-ge\-diffuse\-mismatch\-transmission12run and analyze–4202603\_022si\-ge\-reflection\-specularity12run and analyze–4202603\_023si\-ge\-phonon\-mc\-tables12input preparation–12202603\_024si\-ge\-slab\-thermal\-conductivity12output analysis–8202603\_025hzo\-disordered\-supercell\-inputs13input preparationVASP15202603\_026hzo\-oxygen\-environments13output analysis–17202603\_027hzo\-oxygen\-vacancy\-inputs13input preparationVASP14202603\_028hzo\-vacancy\-charge\-thermodynamics13output analysisVASP55202603\_029hzo\-bonding\-analysis\-inputs13input preparationVASP, LOBSTER17202603\_030hzo\-oxygen\-sublattice\-bonding13output analysisLOBSTER46202603\_031batio3\-hysteresis\-lammps\-inputs14input preparationLAMMPS35202603\_032batio3\-polarization\-hysteresis14output analysisLAMMPS7202603\_033batio3\-dislocation\-dipole\-inputs14input preparationLAMMPS22202603\_034batio3\-dislocation\-dipole\-relaxed14output analysisLAMMPS16202603\_035batio3\-dislocation\-strain\-maps14run and analyzeLAMMPS14202603\_036batio3\-dislocation\-core\-polarization14run and analyzeLAMMPS36202603\_037bas\-force\-constant\-inputs15input preparationQE14202603\_038bas\-harmonic\-force\-constants15output analysisQE, ALAMODE21202603\_039bas\-natural\-isotope\-dielectric15run and analyze–11202603\_040bas\-isotope\-dielectric\-dependence15run and analyze–12202603\_041bas\-surface\-phonon\-polariton\-gap15run and analyze–11202603\_042bas\-near\-field\-heat\-flux15run and analyze–19
## Appendix BAgent configuration and run environment

Table[B](https://arxiv.org/html/2610.00636#A2)gives the settings of each agent, and Table[B](https://arxiv.org/html/2610.00636#A2)the software available to it in the container\.

Table B1:Agent configuration\. Steps is the step limit per single task, Context the context window, Output the maximum output tokens per call, and Effort the reasoning effort\. For Codex CLI, none means that its native tool loop has no step cap, and the dash means its default output limit\.
MatClaw runs each attempt as a code\-first loop: one step is one model reply, a JSON object whose code field the harness executes in a persistent Python kernel, returning the printed output as the next observation\. From that code the agent can call file, search and shell tools; web and job\-submission tools are disabled\. A run ends when the agent callsfinal\_answer, reaches its step limit, or reaches the time limit\. Codex CLI receives the same task text and runs its native tool loop with web search disabled\.

Each run sees only its workspace: the task files, the offline user manuals and read\-only software\. The graders and reference values are kept outside the container and are applied to the submission after the run\. The container has no network route except to the model API\. The manuals come from one reference library, so every run of every agent reads the same documentation\.

Table B2:Software in the container\. Single tasks and Workflows state whether the agent can run each simulation code: yes, no, or source when only a read\-only source tree is staged\. Where it cannot, the agent has the code’s manual\. The Python packages are shared by every run\.

## Appendix CWorkflows and complete results

Table[C](https://arxiv.org/html/2610.00636#A3)defines the eight workflows of Section[3\.2](https://arxiv.org/html/2610.00636#S3.SS2): the source study, the steps in order, and the single task that tests the same step\.

Table C1:The eight workflows\. Study numbers refer to Table[A](https://arxiv.org/html/2610.00636#A1)and task numbers to Table[A](https://arxiv.org/html/2610.00636#A1); a range such as 202601\_004 to 007 lists the input and analysis tasks of one step\. A dash marks a step that has no single task\.WorkflowStudyStepDescriptionSingle taskC018aBuild and relax the WS2/WSe2moiré202602\_015 to 016bCompute the piezoelectric bound charge202602\_021cCompute the screened electron potential202602\_022C024aClassify Se\-pair substitutions by space group202602\_025bBuild a tight\-binding model per family–cCompute the spin splittings202602\_028 to 029C0315aCompute BAs force constants from DFT forces202603\_037 to 038bCompute the isotope\-disordered dielectric function202603\_039cCompute gap modes and near\-field transmission202603\_041C0412aRelax the Si/Ge interface and compute force constants–bCompute the phonon reflection amplitudes202603\_020cCompute the specularity of reflection202603\_022C0510aRelax five ordered Zn1\-xMgxO cells202603\_002 to 003bCompute Berry\-phase polarization paths202603\_004 to 005cFit the Landau free\-energy model202603\_006C0611aCompute the magnetic ground state and Wannier functions202603\_011bDerive the four exchange constants202603\_012cCompute the spin\-wave energies202603\_013C073aBuild the two chemical orderings of V2CoAl202601\_001bRelax both orderings with DFT202601\_002 to 003cCompute the magnetic states of both orderings202601\_004 to 007dCompute nearest\-neighbour exchange couplings202601\_008C089aFind the Ni\-island saddle point with EAM202602\_031bRefine the saddle with a machine\-learned potential–cCertify its curvature and connected minima202602\_032Figure[C1](https://arxiv.org/html/2610.00636#A3.F1)gives every attempt behind Table[2](https://arxiv.org/html/2610.00636#S5.T2), Table[3](https://arxiv.org/html/2610.00636#S5.T3)and Fig\.[4](https://arxiv.org/html/2610.00636#S5.F4): the 94 single tasks under SF and SR, the eight workflows under WF and WR, and the 20 repeated tasks\.

Figure C1:Complete results\. Each square is one run\. DeepSeek and Gemini are DeepSeek V4\.1 Flash and Gemini 3\.8 Flash; Sol and Codex CLI are GPT\-5\.6 Sol in the two harnesses of Table[B](https://arxiv.org/html/2610.00636#A2)\. \(a\) Columns are the 94 single tasks of Table[A](https://arxiv.org/html/2610.00636#A1), labelled by task number\. SR rows are blank for the 42 tasks whose prompt has no reduced form\.
## Appendix DErrors shared across agents

This appendix lists the errors that several agents make in the same way on the same task\. Section[5\.3](https://arxiv.org/html/2610.00636#S5.SS3)shows one of them, a LAMMPS line that all four agents write wrongly\. Of the 94 single tasks under full guidance, 14 are failed by two or more agents\. Table[D](https://arxiv.org/html/2610.00636#A4)lists six tasks on which the failing agents write the same wrong input\. On five of these tasks the wrong input appears in every failing run; on the sixth it appears in three of the four\.

Table D1:Errors shared across agents on single tasks under full guidance\. Task numbers refer to Table[A](https://arxiv.org/html/2610.00636#A1)\. Agents names those that fail the task with the error shown; an agent not named passes the task or fails it in another way\.
Section[5\.3](https://arxiv.org/html/2610.00636#S5.SS3)names the cause: agents based on different LLMs lack the same implicit knowledge of software interfaces\. Each row of Table[D](https://arxiv.org/html/2610.00636#A4)is such a case\. The correct form is known from practice with the code: an example in the manual does not generalise to more atom types, a default must be overridden, or a tag must be set for a class of elements\. Given the same documentation, the agents converge on the same wrong form\.

## Appendix EPrompts

### E\.1Decomposition prompt

The prompt below, given one source paper, produces the graph of research steps of Section[3\.1](https://arxiv.org/html/2610.00636#S3.SS1)\.

You are a research\-paper expert specializing in methodological analysis, problemdecomposition, and computational\-dependency analysis of scientific studies\.\*\*GOAL\*\*Produce, in a SINGLE response, a directed acyclic graph \(DAG\) of the paper’sresearch\-problem decomposition\. Work in two INTERNAL phases, then output once\.PHASE 1 \-\- Construct the research\-problem tree \(do not output yet\):\- Fully comprehend the paper; understand its core research problems and experiments\.\- Build a hierarchical research\-problem tree that mirrors the authors’ logic:\* Root node \-\- the single, broadest research problem tackled by the paper\.\* Intermediate nodes \-\- progressively narrower sub\-problems / objectives\.\* Leaf nodes \-\- fully specified experimental tasks \(datasets, models, metrics,protocols\) that each map to a MAIN\-TEXT Figure or Table\.\- Continue decomposing until every branch ends in such a leaf\. There is no depth limit\.PHASE 2 \-\- Annotate the COMPLETE tree into a DAG \(only after Phase 1 is finished, so thefull set of leaves is already known\):\- For each leaf node, append: \`id\`, \`cost\_class\`, \`consumes\`\.\- For each intermediate node \(root included\), append: \`id\`\.\- Because the whole tree already exists before you annotate, every \`consumes\` referencemust point to a leaf you have already identified in Phase 1\.Output ONLY the final annotated JSON \(the Phase 1 tree is internal scratch work\)\.\-\-\-\#\#\# Phase 1 \-\- Reading & Extraction Rules1\. \*\*Locate the root\*\* in the title, abstract, introduction, or discussion\.2\. \*\*Recursively decompose\*\* each problem by following explicit textual cues \(headings,"first\.\.\. second\.\.\.", "to this end\.\.\.", method overviews, figure/table captions,bullet lists, etc\.\)\.3\. \*\*Identify leaves\*\*: a node is a leaf \*only if\* it describes a concrete experiment andits \`evidence\.figure\` or \`evidence\.table\` field cites a \*\*main\-text\*\* Figure or Table\.A section reference alone is not sufficient\. Supplementary material \(SI, SM, Appendix,Supplementary Figs\./Tables\) may appear only inside a leaf’s \`protocol\_or\_setup\` field\.4\. \*\*Not every main\-text figure is a leaf\.\*\* A leaf produces a new physical quantity byrunning a calculation, a measurement, or its own equation on upstream output\. Thefollowing are never leaves; fold each into the leaf whose output it reads, by addingthe figure or table to that leaf’s \`evidence\` and the quantity to its \`metrics\` and\`conclusion\`:\(1\) a figure or panel that only re\-plots, visualizes, or takes summary statistics ofanother leaf’s output;\(2\) a comparison of the paper’s own results with literature or experimental valuesthat the paper did not produce;\(3\) computing time, cost, or software\-performance reports\.5\. \*\*Capture all layers\*\* \-\- do \*\*not\*\* skip intermediate hypotheses, objectives, oranalysis steps the paper explicitly discusses\.6\. \*\*Stay faithful\*\* to the paper’s wording for technical terms; paraphrase only for brevity\.7\. \*\*No outside invention\*\* \-\- derive every node from the paper alone\. If information ismissing, mark the node with \[uncertain\]\.\#\#\# Phase 2 \-\- Annotation Rules1\. \*\*Assign stable IDs\.\*\* Leaves receive \`Lk\` numbered top\-to\-bottom in tree order\(\`L1\`, \`L2\`, \.\.\.\)\. Intermediate nodes receive \`Ik\` \(\`I1\`, \`I2\`, \.\.\.\)\. The root nodereceives \`id = "root"\`\.2\. \*\*Tag \`cost\_class\` for each leaf\.\*\* Choose \`"heavy"\` if the leaf describes a first\-principles calculation \(DFT, GW/BSE, AIMD, TDDFT\), large\-scale classical MD \(˜10ˆ4atoms or longer than ˜1 ns\), Monte Carlo with non\-trivial sampling, or ML\-potentialtraining \-\- anything taking more than ˜1 hour on a single GPU/CPU node\. Choose\`"light"\` otherwise: post\-processing, analytical fits, geometry construction,structure mapping, visualization, or 1D ODE/PDE solving\.3\. \*\*Write \`consumes\` for each leaf\.\*\* List the IDs of upstream leaves whose outputs thisleaf needs, e\.g\. \`\["L1", "L3"\]\`\. Empty list \`\[\]\` is allowed for foundational leaveswith no upstream\. Dependencies may cross intermediate branches; do not inventdependencies that are not stated or implied in the paper\.4\. \*\*Stay acyclic and faithful\.\*\* The graph must be acyclic; a \`consumes\` reference thatwould create a cycle is the result of a misread and must be corrected\. If a cost classor dependency is genuinely unclear, append \`\[uncertain\]\` to the relevant field\.Strictly output the annotated tree in a JSON format\. Extraction fields come first; thethree annotation fields are appended within the same node object:\`\`\`\{ "paper": \{ "title": "", "authors": \[\], "venue": "", "year": "" \},"problem\_tree": \{"node": "Root: broadest research problem tackled by the paper","type": "root node","description": "a detailed description of the research problem in this node","evidence": "references back to the original paper","id": "root","children": \[\{"node": "Intermediate sub\-problem / objective 1","type": "depth\-1 node","description": "a detailed description of the research problem in this node","evidence": "references back to the original paper","id": "I1","children": \[\{"type": "leaf node","task": "Concrete experimental task \(as phrased by paper\)","dataset": \["\.\.\."\],"model\_or\_method": \["\.\.\."\],"metrics": \["\.\.\."\],"protocol\_or\_setup": "key settings/splits/hyperparams if stated","evidence": \{"figure": "Fig\. X", "table": "Table Y", "section": "Sec\. Z"\},"conclusion": "explicit, detailed conclusions from this leaf’s experiment","status": "","id": "L1","cost\_class": "heavy","consumes": \[\]\}\]\}\]\}\}\`\`\`

### E\.2Time notice

In a single\-task run, MatClaw appends the notice below to the task and to the observation returned after every step, with the remaining time updated; the example is taken with 1,008 seconds left\. Codex CLI receives a similar notice\.

\[Time budget\] Total task wall\-clock budget: 3600 seconds \(60 minutes\),including API requests, reasoning, tool execution and waiting\. Remaining atthis observation: 1008 seconds \(16m 48s\)\. Complete the required submissionfiles and call final\_answer before the deadline\.

### E\.3One task under reduced and full guidance

The two prompts below are the reduced \(SR\) and full \(SF\) forms of task 202603\_020, the mode\-resolved phonon transmission through a Si/Ge interface\. The reduced prompt keeps the input file, the required accuracy, the definitions of the reported quantities and the output format\. The full prompt adds the setting of the problem, the contents of the input file, and the method: the atomistic Green’s function approach, the criterion for a propagating channel, and the conventions of the dynamical matrix and of the normalised scattering matrix\.

Reduced guidance \(SR\)\# Task: mode\-resolved phonon transmission through a Si/Ge interface from harmonic force constantsCompute the harmonic phonon transmission of a planar Si/Ge\(100\) interfacebetween a semi\-infinite Si lead on the left and a semi\-infinite Ge lead onthe right\.\`device\_force\_constants\.hdf5\` contains the device’s principal\-layerforce\-constant blocks and atomic masses\. Its attributes specify units,block ordering, coupling directions, layer period and transverse periodicity\.Use these blocks as given, without symmetrisation or sum\-rule corrections\.No molecular\-dynamics or electronic\-structure calculation is required\.Python with numpy, scipy and h5py is available\.\#\# Quantities and accuracyLet $t$ be the flux\-normalized Si\-to\-Ge transmission matrix, with incomingpropagating Si channels as rows and outgoing propagating Ge channels as columns\.For every incoming channel, transmission plus reflection must sum to onewith absolute error below $10ˆ\{\-6\}$\.At ordinary frequencies $\\nu = 1\.00, 2\.00, 4\.00, 5\.05, 6\.00, 8\.00,10\.00, 11\.50, 12\.00$ THz, report:\- \`n\_Si\`, \`n\_Ge\`: the number of propagating channels in each lead travellingtowards the interface;\- \`Xi\`: the sum of transmission probabilities over all incoming Si channels;\- \`mean\_T\_Si\`: the mean transmission probability over all incoming Si channels;\- \`std\_tau\_Si\`: the population standard deviation of the full spectrum of$t tˆ\\dagger$ \(all $n\_\{\\mathrm\{Si\}\}$ eigenvalues, including zeros\)\.\#\# Output formatWrite a single \`answer\.json\` in the workspace root with exactly this layout:one entry per requested frequency, in the order listed above, the frequencyechoed as given \(counts are integers; the other three fields are dimensionlessfloats\):\`\`\`json\{"results": \[\{"frequency\_THz": 1\.00,"n\_Si": <int\>,"n\_Ge": <int\>,"Xi": <float\>,"mean\_T\_Si": <float\>,"std\_tau\_Si": <float\>\},\.\.\.\]\}\`\`\`All values must be computed from the staged file; do not guess or hardcode\.

Full guidance \(SF\)\# Task: mode\-resolved phonon transmission through a Si/Ge interface from harmonic force constantsA planar \(100\) interface between crystalline Si and crystalline Ge is studiedas a phonon scatterer\. The interface sits in a device made of a semi\-infiniteSi lead on the left, a semi\-infinite Ge lead on the right and a central regionthat contains the junction\. All three parts are described by harmonic forceconstants in a principal\-layer basis\. The question is how many propagatingphonon channels each lead carries towards the interface at a given frequency,how much of the incident flux the interface transmits, and how unevenly thetransmission is distributed over the channels of the Si side\.\#\# What you are given\- \`device\_force\_constants\.hdf5\` \-\- the harmonic force\-constant blocks of thedevice and the per\-atom masses: for each lead, the on\-site block of oneprincipal layer and the coupling block to the next layer; for the centralregion, the on\-site block of each of its slices, the couplings betweenconsecutive slices, and the couplings of the first and last slice to theirleads\. Units, the atom\-major ordering inside the blocks and the directionconvention are recorded in the file’s attributes\. For both leads the couplingblock connects a principal layer to the next one along $\+x$, the directionfrom the Si lead towards the Ge lead, and in every coupling block the rowblock precedes the column block along $\+x$\. Both leads share the one layerperiod recorded in the file\. The blocks already contain the full transverseperiodic cell of the device, so no transverse wave\-vector sampling isneeded\. Use the blocks as given, without further symmetrisation or sum\-rulecorrections\.No molecular\-dynamics or electronic\-structure calculation is required\. Theenvironment provides Python with numpy, scipy and h5py\. Solving one frequencytakes of the order of a minute on one core; the frequencies are independent\.\#\# MethodUse the atomistic Green’s function method for a lead\-device\-lead geometry withthe leads treated as semi\-infinite, extended to a mode\-resolved scatteringmatrix: at each frequency, the propagating Bloch channels of each lead areobtained from the lead’s own blocks, the scattering matrix between the incomingand outgoing propagating channels of the two leads is formed from the deviceGreen’s function, and each entry is normalised by the channel group velocitiesso that its squared modulus is a probability\. A lead channel counts aspropagating when its Bloch factor across one layer period has unit modulus\(real wave vector\); evanescent channels are removed before the scatteringmatrix is formed\. At the requested frequencies no channel lies at a band edge,so the propagating and evanescent sets are well separated\.Conventions \(fixed for this task\):\- Frequencies are ordinary frequencies $\\nu$ in THz; the dynamical problem isposed in the angular frequency $\\omega = 2\\pi\\nu$\.\- The dynamical blocks are the mass\-weighted force constants$\\Phi\_\{ij\}/\\sqrt\{m\_i m\_j\}$, converted to SI angular\-frequency\-squared units\.\- The small positive imaginary part added to $\\omegaˆ2$ for the retardedGreen’s functions must be chosen small enough that, for every incomingchannel, transmission plus reflection sums to one to better than $10ˆ\{\-6\}$\.\- Let $t$ be the normalised transmission block of the scattering matrix fromthe incoming propagating channels of the Si lead \(rows\) to the outgoingpropagating channels of the Ge lead \(columns\)\. For an incoming Si channel$l$, its transmission is $\\Gamma\_l = \\sum\_m \|t\_\{lm\}\|ˆ2$, and the \*\*totaltransmission\*\* is $\\Xi\(\\nu\) = \\sum\_l \\Gamma\_l$ over all incoming propagatingchannels of the Si lead\.\- The \*\*transmission eigenvalues\*\* $\\tau\_k$ are the eigenvalues of the Hermitianmatrix $t\\,tˆ\\dagger$; there are as many as incoming Si channels, and theirsum equals $\\Xi$\. Their standard deviation is the population one \(thevariance normalised by the number of channels\)\. Unlike the individual$\\Gamma\_l$, the $\\tau\_k$ do not depend on how a degenerate multiplet ofchannels is resolved into a basis\.\#\# What to determineAt each of the nine frequencies $\\nu \\in \\\{1\.00, 2\.00, 4\.00, 5\.05, 6\.00, 8\.00,10\.00, 11\.50, 12\.00\\\}$ THz:\(1\) $n\_\{\\mathrm\{Si\}\}$ and $n\_\{\\mathrm\{Ge\}\}$, the number of propagating channelsof each lead that travel towards the interface;\(2\) $\\Xi$, the total transmission from the Si side;\(3\) the mean of $\\Gamma\_l$ over the incoming Si channels, and the standarddeviation of the transmission eigenvalues $\\tau\_k$\.\#\# Output formatWrite a single \`answer\.json\` in the workspace root with exactly this layout:one entry per requested frequency, in the order listed above, the frequencyechoed as given \(counts are integers; the other three fields are dimensionlessfloats\):\`\`\`json\{"results": \[\{"frequency\_THz": 1\.00,"n\_Si": <int\>,"n\_Ge": <int\>,"Xi": <float\>,"mean\_T\_Si": <float\>,"std\_tau\_Si": <float\>\},\.\.\.\]\}\`\`\`\- \`n\_Si\`, \`n\_Ge\`: item \(1\)\.\- \`Xi\`: item \(2\)\.\- \`mean\_T\_Si\`, \`std\_tau\_Si\`: item \(3\)\.All values must be computed from the staged file; do not guess or hardcode\.