BenchBench-Cloning: Evaluating Molecular Cloning Design and Protocol Development

# BenchBench-Cloning:Evaluating Molecular Cloning Designand Protocol Development
Source: [https://arxiv.org/html/2610.33355](https://arxiv.org/html/2610.33355)
Peter Ciaccia\*DNAForgeHassaan Qadir\*DNAForgeIan Hsu\*BenchlingAshu SinghalBenchling

###### Abstract

Molecular cloning is one of the most routine wet\-lab workflows, yet existing evaluations of language models in biology mostly test knowledge recall or isolated steps rather than end\-to\-end experimental planning\. We introduce BenchBench\-Cloning, a benchmark for large language models comprising 65 molecular cloning tasks grounded in sequence records, laboratory notebooks, and published experimental studies\. Planning a molecular cloning experiment requires defining a set of constructs needed for a scientific objective and a construction route to produce them\. BenchBench\-Cloning contains 50 protocol\-development tasks, in which agents produce an executable protocol for a specified construct, and 15 construct\-design tasks, in which agents determine the constructs a scientific objective calls for\. Protocol\-development tasks are evaluated by execution in a cloning simulator; construct\-design tasks are evaluated with task\-specific deterministic checks and, where needed, rubric\-based LLM\-as\-a\-judge evaluation\. Across eight systems with repeated attempts per task, GPT\-6 Astra scores highest at 30\.6%, with the others between 8\.6% and 22\.2%, and the benchmark remains unsaturated under best\-of\-ten sampling\. As integrating individual cloning decisions into complete, valid plans remains challenging even for frontier models, we expect BenchBench\-Cloning to serve as a useful benchmark for tracking progress in experimental planning\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author:aditya\.sivakumar@benchling\.com\.## 1Introduction

Designing and constructing plasmids is a common task in molecular biology\. A scientist must determine the required sequence, identify the requisite starting materials, choose a construction strategy, and specify the reactions needed to produce the desired product\. The choice of assembly method depends on the starting materials, and the products of each reaction determine which steps can follow\.

Language models are commonly used to assist with individual steps of this process, including sequence analysis, primer design, and selection of cloning methods\. Evaluating capabilities separately, however, leaves open whether models can successfully plan and execute a full experiment\. For example, a proposed sequence may satisfy the requested design while an accompanying protocol may fail to produce it\. Similarly, executable protocols may fail to address the scientist’s objective\. This planning process is time\-consuming even for experienced scientists, with human errors frequent and surfacing only after several days \(or weeks\) of work\. A model that can reliably reason through entire cloning workflows stands to provide enormous scientific and commercial benefit for scientists\.

To measure progress toward this goal, we introduce BenchBench\-Cloning, a benchmark of 65 molecular cloning tasks designed to evaluate a model’s performance on correct and efficient cloning strategies\. The protocol\-development set contains 50 tasks where scientists specify the desired construct and the agent develops an executable protocol from the supplied inventory, and the construct\-design set contains 15 tasks where the agent must design a construct that satisfies a scientific objective based on experimental motivations\.

We contribute:

- •BenchBench\-Cloning:a benchmark of 65 tasks testing AI capability in reasoning through complex cloning tasks\.
- •Baseline resultsfor current frontier models, indicating that current models struggle to develop complete, valid cloning plans\.

We describe the design and construction of the benchmark, present results from evaluating current models against it, and discuss what these results suggest about the current state and future trajectory of AI\-assisted experimental planning\.

## 2Related Work

Life\-science benchmarks increasingly evaluate tasks that resemble parts of a scientist’s workflow\. LAB\-Bench and LABBench2 cover literature use, data interpretation, and sequence manipulation\([Laurent et al\., 2024](https://arxiv.org/html/2610.33355#bib.bib1);[Laurent et al\., 2026](https://arxiv.org/html/2610.33355#bib.bib2)\)\. LifeSciBench evaluates expert\-authored research tasks using detailed rubrics across multiple scientific workflows\([Liu et al\., 2026](https://arxiv.org/html/2610.33355#bib.bib3)\)\. These evaluations broaden the assessment of scientific assistance beyond factual recall to evidence interpretation and conclusion formulation\. BenchBench\-Protocol evaluates modifications to published laboratory protocols using tasks recovered from changes scientists made to experimental work\([Sivakumar et al\., 2026](https://arxiv.org/html/2610.33355#bib.bib5)\)\.

Agentic benchmarks such as BixBench\([Mitchener et al\., 2025](https://arxiv.org/html/2610.33355#bib.bib4)\)and Terminal\-Bench\-Science\([Terminal\-Bench\-Science Team, 2026](https://arxiv.org/html/2610.33355#bib.bib6)\)evaluate models on multistep computational analyses in biology and other scientific domains\. LABBench2 CloningQA is the closest prior work, with 14 tasks requiring complete cloning protocols including methods like DNA and enzyme reagents, restriction\-ligation cloning, Gibson assembly, and Golden Gate assembly\. Submissions use a domain\-specific language and are evaluated using component checks and end\-to\-end in silico validation\([Laurent et al\., 2026](https://arxiv.org/html/2610.33355#bib.bib2)\)\. BenchBench\-Cloning expands this formulation to 50 protocol\-development tasks in an environment supporting 42 operations\. Our verification requirements include method\-specific primer constraints, molecular\-state and end\-chemistry checks, and the ability to produce and isolate multiple products\.

## 3Benchmark Design and Construction

BenchBench\-Cloning contains 65 tasks divided into two sets: 50 protocol\-development tasks and 15 construct\-design tasks\. In protocol\-development tasks, the desired construct is specified and the agent must determine how to produce it from the available materials\. In construct\-design, the scientific objective is specified and the agent must determine appropriate construct\(s\), specify their exact sequences, and provide a construction plan\.

Both sets of tasks provided a fixed request and supporting files\. Agents inspect these files, use analysis tools, and revise their work during an attempt\. The two task sets differ in their inputs, outputs, and evaluation procedure \(Figure[1](https://arxiv.org/html/2610.33355#S3.F1)\)\.

Figure 1:The two task formats\. In protocol development \(A\) the agent writes a protocol that the simulator executes; in construct design \(B\) the agent returns constructs and a plan that deterministic checks and, on ten tasks, a model judge score\.### 3\.1Protocol\-Development Tasks

Each protocol\-development task provides a scientist’s request describing the desired construct and an inventory of starting materials\. Products are circular plasmids between 2\.8kb and 15\.2kb long, with a median length of 5\.4kb\. All 50 protocol\-development tasks are derived from laboratory notebooks created by contributors on Benchling, drawing on real construct requests and experimental context\. An inventory containing up to eight starting plasmid constructs is supplied as Genbank or FASTA files with accompanying sequence annotations\. Notes of material properties \(such as methylation sensitivity at restriction sites, GC\-rich regions, end chemistry, acceptor vectors with ccdB counter\-selection cassettes, host strain\) are also provided\. The task set spans nine experimental contexts, each defined by the dominant constraint the task imposes \(Table[1](https://arxiv.org/html/2610.33355#S3.T1)\)\.

Table 1:Protocol\-development task families\. Task counts and shares are shown for each dominant constraint with the reasoning it requires\.The environment includes documentation for a custom cloning protocol language supporting 42 operations\. These cover amplification, partial and complete digestion, Gibson and Golden Gate assembly, oligonucleotide annealing and ligation, end modification, CRISPR\-based operations, and other relevant techniques\. The environment also includes tools such as Python, Biopython, Primer3\-py, and plotting libraries for inspection and design\.

During evaluation, the agent receives the request, inventory, operation documentation, and analysis tools\. It submits a protocol that is then passed through the simulator and assigned a binary score\. A protocol passes if it successfully executes and ends with one isolated product that matches the hidden target sequence\. Invalid operations, incorrect sequences, or unresolved mixtures result in failure\. Alternative construction routes are accepted when they pass through the simulator and produce the same final product as the reference protocol\.

### 3\.2Construct\-Design Tasks

The construct\-design set contains 15 tasks derived from published scientific studies\.111Tasks, graders, and reference solutions for the construct\-design set are available at[https://github\.com/benchling/benchbench\-cloning](https://github.com/benchling/benchbench-cloning)\.Each task provides a scientist’s objective alongside sequence records and supporting materials\. Six tasks additionally provide experimental records such as sequencing reads or gel images\. Agents return an annotated construct sequence, orderable primers when required, a construct plan, and a written justification\.

These tasks require the agent to determine which constructs are needed to address a scientific objective\. Depending on the task, a successful response may require identifying a relevant sequence feature, applying a repair across related constructs, or including controls needed to interpret an experiment\.

Each task has a deterministic grader that checks the returned sequences, their source, and the construction route\. Additionally, 10 tasks use a rubric\-based LLM\-as\-a\-judge to assess the written reasoning and construction plan\. For these tasks, the final score is the product of the deterministic grader and the rubric score\. Task sources are listed inAppendix[A](https://arxiv.org/html/2610.33355#A1)\.

### 3\.3Model Setup

We evaluate eight systems using Claude Opus 5\.5, Claude Mythos 5\.1, GPT\-6 Astra, GPT\-6\.1 Sol, Gemini 3\.8 Flash, Gemini 4 Argon, Grok 4\.7, and Kimi K3\. Models were tested in their native agent harnesses where available, and Kimi K3 was tested using the Pi harness\. Each system completed ten individual attempts per protocol\-development task and five per construct\-design task\.Appendix[B](https://arxiv.org/html/2610.33355#A2)shows the specific settings of each run\.

Runs were orchestrated using Harbor in Modal sandboxes with two vCPUs and 6 GiB of memory\. Network access was restricted to calls to the model provider\. Each agent received the task instructions and supporting files, used the available tools, and wrote its submission to a designated answer file\. Attempts were limited to two hours\. GPT\-6 Astra served as the model judge for the ten construct\-design tasks requiring rubric\-based evaluation\. We also evaluated the tasks with Claude Mythos 5\.1 and report agreement metrics inAppendix[C](https://arxiv.org/html/2610.33355#A3)\.

### 3\.4Evaluation Metrics

For the protocol\-development tasks, we report pass rate, defined as the proportion of attempts whose submitted protocol executes successfully and produces the isolated target sequence\. We average the ten attempt outcomes within each task and average across the fifty tasks\. We estimate 95% confidence intervals using 20,000 bootstrap resamples of the tasks, retaining all attempts associated with each sampled task\. For construct\-design tasks, an attempt is classified as successful when its final score is at least 0\.9\. The pass rate is computed by averaging the five attempt outcomes within each task and then averaging across the 15 tasks\. The aggregate score is a weighted average of the protocol\-development and construct\-design scores, with weights of 50/65 and 15/65, respectively, giving each task equal weight\.

For protocol\-development tasks, we additionally report pass@kkfor up to ten attempts to measure the probability that at least one ofkkattempts succeeds\. This measures the availability of a valid protocol among repeated attempts\.

## 4Results

Performance on BenchBench\-Cloning varies substantially across systems\. GPT\-6 Astra is the strongest overall system, with an aggregate score of 30\.6%\. Scores for the other evaluated systems range from 8\.6% to 22\.2% \(Figure[2](https://arxiv.org/html/2610.33355#S4.F2)\)\.

Figure 2:Aggregate score by system\. Task\-weighted means at top effort are shown with 95% confidence intervals\.### 4\.1Performance by Task Set

On the protocol\-development set, GPT\-6 Astra achieves the highest pass rate at 27\.0% \(95% CI 19\.8–34\.6%\), followed by Claude Opus 5\.5 at 19\.0% and GPT\-6\.1 Sol at 16\.4% \(Figure[3](https://arxiv.org/html/2610.33355#S4.F3)\)\. Grok 4\.7, Kimi K3, and Gemini 3\.8 Flash score 6\.2%, 5\.4%, and 4\.0%, respectively \(Table[2](https://arxiv.org/html/2610.33355#S4.T2)\)\.

Figure 3:Pass rate by task set\. Means at top effort are shown with 95% confidence intervals for protocol development \(A\) and construct design \(B\)\.Table 2:Performance by task set at top effort\. Pass rates in percent are shown with 95% confidence intervals\.All evaluated systems produce successful protocols, but no system consistently completes the protocol\-development tasks\. Analysis of submitted designs shows that unsuccessful attempts can contain substantial correct work, with a small number of decisions preventing completion\.

### 4\.2Performance Under Repeated Sampling

Because a scientist can compare several candidate answers before acting on one, expected best\-of\-kkperformance is reported as an upper bound on single\-system performance\. System performance ranking is not stable inkk\(Figure[4](https://arxiv.org/html/2610.33355#S4.F4)\)\. Opus performs better than Sol on individual attempts, but Sol performs better atk=10k=10\. Opus’s successful attempts are concentrated on fewer tasks, while Sol succeeds less consistently across a broader set of tasks\.

Figure 4:Protocol\-development pass@kk\. Probability that at least one ofkkattempts passes is shown with 95% confidence bands for the three leading systems\.At ten attempts, Astra solves 36 of the 50 protocol\-development tasks at least once, corresponding to a pass@10 of 72% \(Table[G1](https://arxiv.org/html/2610.33355#A7.T1)\)\. Sol solves 33 tasks and Opus solves 29, corresponding to 66% and 58%, respectively\. The strongest system still leaves 14 tasks unsolved, indicating that the benchmark is not saturated within ten attempts\.

### 4\.3Reasoning Effort and Cost

Average cost per attempt varies by more than an order of magnitude across systems at comparable performance \(Figure[5](https://arxiv.org/html/2610.33355#S4.F5)\)\. The pareto frontier identifies the system that offers the best observed trade\-off between performance and cost\. Token counts for each run are reported inAppendix[D](https://arxiv.org/html/2610.33355#A4)\.

Figure 5:Aggregate score against cost per attempt\. Each system is shown at its three reasoning settings with the top setting filled; the dashed line is the cost frontier\.

## 5Analysis of Model Behavior

### 5\.1Successful Work

Systems perform well on several tasks where the expected molecular outcome is spelled out clearly\. Six of the fifteen construct\-design tasks were passed in every attempt by at least two systems, and four systems passed the minimal\-edit recoding task every time \(Figure[6](https://arxiv.org/html/2610.33355#S5.F6);Table[G2](https://arxiv.org/html/2610.33355#A7.T2)\)\. Many failed submissions also contained correct work\. In the sodium\-channel task, every scored attempt placed both introns at the junctions and kept the full channel protein intact, even though none of them passed the task\.

Figure 6:Construct\-design solves by task and system\. Cells give the number of five attempts that passed at top effort\.The dual\-luciferase task makes this especially clear\. Across the eight systems, 38 of the 39 submitted panels had correctly built reference, frame 0, frame\+1\+1and frame\+2\+2constructs, and every construction step in those panels ran as written\. None of them included the fifth construct, a control with the Firefly reading frame closed by a stop codon, which the experiment needs in order to separate readthrough from re\-initiation \(Figure[7](https://arxiv.org/html/2610.33355#S5.F7)\)\. Several systems even mentioned re\-initiation in their written answers, but they treated it as background shared by all constructs rather than something the panel had to measure\.

Figure 7:The dual\-luciferase frame\-panel task\. The request, the five reference constructs, and the number of the 39 submitted panels containing each construct are shown\.Taken together, these results suggest that agents can usually carry out the individual sequence edits a task requires\. Where they fall short is in reasoning from the experimental question to the set of constructs required to exclude alternative explanations\.

### 5\.2Protocol\-Development Failure Modes

Analysis of protocol failures identifies recurring errors in primer design, backbone preparation, and methylation state\. These errors often arise in systems that correctly identify the intended sequence changes but do not maintain compatibility between the materials and operations used to produce them\.

In several cases, we found that systems would frequently select primers which target the correct sequence while remaining unsuitable for the proposed reaction conditions\. Primer failures frequently involve pairs that cannot be used together under the simulated annealing conditions\. For example, primer pairs were found to dimerize, or to have annealing points which were too far apart to be feasibly used in a reaction together\. In these cases, while the system succeeded in designing single appropriate primers, it failed to evaluate whether the primer could be appropriately used as a pair\. We also identified primer design failures involving secondary structure issues \(e\.g\. hairpin loops\), length limits, or homopolymer runs\.

Backbone preparation and methylation produced additional failures\. Systems ligated vectors with self\-compatible ends without dephosphorylating them, leading to backbones recircularizing or dimerizing\. In other cases, systems treated compatible or blunt ends as directional, carrying inserts forward in one orientation when the reaction yields both, or ignored methylation\-sensitive sites in plasmids from a Dam/Dcm\-methylating host, leaving the acceptor circular\. These are mistakes a molecular biologist would ordinarily anticipate and prevent when designing the protocol\. Each attempt is classified by the first error that blocks execution, reported by system inAppendix[E](https://arxiv.org/html/2610.33355#A5)\.

### 5\.3Construct\-Design Failure Modes

Most construct\-design failures were incomplete designs\. Systems usually made the change that was asked for, but they often missed a second change that the experiment needed\. The sodium\-channel task is a good example \(Table[G2](https://arxiv.org/html/2610.33355#A7.T2)\)\. Two related expression constructs each carry a cryptic bacterial promoter that has to be removed\. GPT\-6 Astra and GPT\-6\.1 Sol fixed the first construct in all of their attempts, yet no system ever fixed the second one\. The dual\-luciferase task shows a similar pattern\. Nearly all submitted panels had correct reference and frame\-shifted reporters, but none of them included the closed\-frame control\.

Some designs went further than the task asked\. The matched\-panel task asks for the smallest set of constructs that can resolve a single discrepancy, and the grader checks for this\. Astra and Sol built an eight\-construct factorial panel in every attempt and so failed the check\. Most of the other systems submitted the four\-construct panel and passed\.

When the grader also checked the construction route, many failures came from the construction record and not from the sequences themselves\. We saw source references that could not be resolved, the same identifier used for different sequences, missing oligonucleotide tables, and fragments that no listed reaction actually produces\. The designed sequences in these submissions were sometimes correct, but the grader had no way to confirm that the plan could build them from the materials provided\.

Ten of the tasks also use a model judge, so passing the deterministic checks does not guarantee a pass\. The judge scores the written plan, and an attempt needs at least 0\.9 to pass\. At the top effort setting, 34 of the 142 attempts that cleared the deterministic checks fell below this threshold \(Table[F3](https://arxiv.org/html/2610.33355#A6.T3)\)\. Most often the judge’s main concern was that the plan did not lay out an orderable path from the supplied sources to the final product; in two cases it was a misreading of the experimental evidence\. We list the reviewed submissions inAppendix[F](https://arxiv.org/html/2610.33355#A6)\.

## 6Discussion

### 6\.1Implications and Future Work

BenchBench\-Cloning shows that frontier models can simulate many of the individual operations required by molecular cloning, but they do not yet reliably assemble those operations into a complete, executable experimental design\. The strongest system, GPT\-6 Astra, reaches an aggregate score of 30\.6% and a protocol\-development pass rate of 27\.0%, and even at ten attempts it leaves 14 of 50 protocol\-development tasks unsolved\. However, it comes with significant costs; under the pareto frontier, it is observed that mid\-range systems like GPT\-6\.1 Sol can achieve similar results at lower cost\. Notably, while repeated sampling produces a greater likelihood of valid solutions, its practical value depends on identifying those solutions\. Comparing individual attempts with multiple candidates and a selection procedure under the same inference budget would help determine how additional computation is best used\.

Failures are rarely a matter of missing basic knowledge\. Unsuccessful submissions are often the result of flawed critical thinking and understanding of the deeper underlying motivations, such as an omitted control\. The failures are not at any single subproblem, but rather in the ability to maintain context across longer horizon tasks, such as understanding that materials have constraints, or that a primer pair cannot work together under the stated reaction conditions\.

These findings bear on how such systems are used today\. A failing submission typically reads as a complete and well\-reasoned plan, with correct sequence edits, plausible primers, and a confident justification, and any mistakes are only visible to someone who already knows what the experiment demands\. Current systems are therefore convincing enough to mislead the users most likely to rely on them\. The near\-miss character of these failures also points to the fix, with blocking errors being checks an experienced molecular biologist makes as a matter of course that a system should make on its own plan before presenting it\. Until systems do so reliably, expert review of model\-generated protocols remains necessary; the benchmark measures progress toward systems that reliably and correctly plan complete cloning workflows\. Further analysis can assess whether explicit self\-verification and candidate selection improve complete\-task success\.

For benchmark design, our results also show that grading choices matter: of 142 submissions that passed the deterministic gate, 34 fell below the pass threshold on the model judge, most often because the plan lacked an orderable construction route from the supplied sources\. Correct sequences thus do not guarantee a usable plan, and some failures reflect incomplete documentation rather than flawed biology\.

### 6\.2Limitations

While BenchBench\-Cloning was designed as a single\-turn benchmark evaluating whether a model could respond correctly to a fixed scientist request, this likely does not reflect how a scientist would collaborate with a model\. Scientists often use LLMs in a multi\-turn capacity, which would allow models to ask clarifying questions about available materials, revise plans after feedback, respond to experimental results obtained during construction, and emphasize practical limitations on suggested protocols\. Future work can focus on a format that more closely reflects this reality\.

Further, while BenchBench\-Cloning measures planning for molecular cloning, it does not directly measure whether models improve experimental outcomes at the bench\. The simulator cannot capture every source of variation in reagents, handling, growth, or reaction outcomes\. As LLMs move into the physical world, future iterations of BenchBench\-Cloning could involve actually conducting experiments designed by the models to test out the protocols designed\.

Another limitation in analysis of traces is that our simulator inherently reports the first blocking error, so these diagnostics do not necessarily establish whether later steps would have succeeded\. Failure frequencies also depend on the routes models choose\. Models using PCR more frequently encounter primer\-related failures, which may have skewed the occurrences of primer design failures amongst the reasoning traces\.

Finally, a limitation of our construct\-design tasks was the use of LLM\-as\-a\-judge to assess written reasoning and construction plans\. Because this kind of model grading is inherently stochastic, there were occasional inconsistencies in grading results\. Notably, agreement between graders does not establish independent scientific validity\.

## 7Conclusion

BenchBench\-Cloning measures whether models can determine the constructs addressing a scientific objective and plan a construction route from the available materials\. GPT\-6 Astra scores highest at 30\.6%, with other systems’ aggregate scores ranging from 8\.6% to 22\.2%\. Repeated sampling improves protocol\-development performance, but the strongest system still leaves 14 out of 50 tasks unsolved after ten attempts\.

A closer look at successes and failures identified trends\. Systems demonstrate surprisingly skilled sequence work and successful construction planning on part of the benchmark\. However, their failures frequently concern compatibility between operations, incomplete construct plans, or lack of consideration towards any changes needed for a broader scientific objective\. These failures matter in practice because the scientists most likely to rely on a system for cloning judgment are those with the least bench experience, and the least able to catch its errors\. Evaluating molecular cloning assistance therefore requires checking both whether a design addresses the experiment and whether the proposed operations can produce it\. Closing the gap between individual cloning decisions and a complete experimental plan is the next step toward systems that scientists can rely on, and BenchBench\-Cloning offers a way to measure progress toward it\.

## References

- Laurentet al\.\(2026\)J\. M\. Laurent, A\. Bou, M\. Pieler, C\. Igoe, A\. Andonian, S\. Narayanan, J\. Braza, A\. Sanchez Vassopoulos, J\. L\. Steenwyk, B\. Lash, A\. D\. White, and S\. G\. RodriquesLABBench2: an improved benchmark for AI systems performing biology research\.External Links:2604\.09554Cited by:[§2](https://arxiv.org/html/2610.33355#S2.p1.1),[§2](https://arxiv.org/html/2610.33355#S2.p2.1)\.
- Laurentet al\.\(2024\)J\. M\. Laurent, J\. D\. Janizek, M\. Ruzo, M\. M\. Hinks, M\. J\. Hammerling, S\. Narayanan, M\. Ponnapati, A\. D\. White, and S\. G\. RodriquesLAB\-Bench: measuring capabilities of language models for biology research\.External Links:2407\.10362Cited by:[§2](https://arxiv.org/html/2610.33355#S2.p1.1)\.
- Liuet al\.\(2026\)A\. Liu, A\. Ho, A\. M\. Droste, D\. Martin, E\. Wong, E\. Zhou, I\. Zhou, J\. Park, J\. Jiao, K\. Skelly, K\. Kim, K\. Rao, M\. Uehara, M\. Marion, N\. Fitzgerald, R\. Dias, S\. Shringarpure, Y\. Yuan, and Y\. WangLifeSciBench: evaluating language models on realistic, expert\-level tasks in the life sciences\.Note:PreprintCited by:[§2](https://arxiv.org/html/2610.33355#S2.p1.1)\.
- Mitcheneret al\.\(2025\)L\. Mitchener, J\. M\. Laurent, B\. Tenmann, S\. Narayanan, G\. P\. Wellawatte, A\. D\. White, L\. Sani, and S\. G\. RodriquesBixBench: a comprehensive benchmark for LLM\-based agents in computational biology\.External Links:2503\.00096Cited by:[§2](https://arxiv.org/html/2610.33355#S2.p2.1)\.
- Sivakumaret al\.\(2026\)A\. Sivakumar, A\. Singhal, N\. Larus\-Stone, and N\. ParsanBenchBench\-Protocol: evaluating real\-world wet\-lab protocol reasoning and modification\.External Links:2608\.23898Cited by:[§2](https://arxiv.org/html/2610.33355#S2.p1.1)\.
- Terminal\-Bench\-Science Team \(2026\)Terminal\-Bench\-Science TeamTerminal\-Bench\-Science: evaluating AI agents on research workflows across scientific domains\.Note:Version v0\.1\.0[https://github\.com/harbor\-framework/terminal\-bench\-science](https://github.com/harbor-framework/terminal-bench-science)External Links:[Document](https://dx.doi.org/10.5281/zenodo.22110254)Cited by:[§2](https://arxiv.org/html/2610.33355#S2.p2.1)\.

## Appendix AConstruct\-Design Tasks and Sources

Each construct\-design task gives the agent a scientist’s objective, sequence records and supporting materials\. Six tasks also supply experimental records from the source study\. Every task has a deterministic grader that checks the returned sequences, their sources and the construction route; ten tasks also use a rubric\-based model judge to assess the written reasoning and construction plan, and for these the final score is the product of the deterministic result and the rubric score\. An attempt passes at a final score of at least 0\.9\.Table[A1](https://arxiv.org/html/2610.33355#A1.T1)lists the tasks;Figures[1](https://arxiv.org/html/2610.33355#S3.F1)and[7](https://arxiv.org/html/2610.33355#S5.F7)show the dual\-luciferase task in detail\.

Table A1:Construct\-design tasks\. Source, experimental records supplied, and grading route are listed for each task\.
## Appendix BModel and Harness Settings

All attempts ran through Harbor in Modal sandboxes \(2 vCPU, 6 GiB\) with network access limited to the model provider\. Each system completed 10 attempts per protocol\-development task and 5 attempts per construct\-design task at every reasoning setting\.

Table B1:Model and harness configurations\. Model identifier, harness, access route, and reasoning settings are listed per system\.
## Appendix CJudge Agreement

To check the rubric\-based outcomes, every construct\-design attempt from the first six systems evaluated was graded a second time by Claude Mythos 5\.1 acting as an independent judge\. The judge received the public task text, the private reference material and the submission files, and returned a score on the same 0 to 1 scale; agreement is measured on the binary pass outcome\.

Table C1:Agreement between the Claude Mythos 5\.1 judge and reported outcomes\. Trials graded, agreement, and Cohen’s kappa are shown per slice\.Agreement between Claude Mythos 5\.1, used as an independent judge, and the outcomes reported in the paper \(pass = final score of at least 0\.9 under both\)\. The comparison covers the 450 construct\-design attempts of the first six systems evaluated \(15 tasks, 5 attempts each\)\. The Mythos judge saw the public task text, the private reference material and the submission files, but not the reported outcome\. It declined to grade 63 attempts under its own safety policy; those are excluded\.

## Appendix DToken Usage and Cost

Table D1:Token usage, cost, and runtime per protocol\-development attempt\. Mean tokens, cost, median minutes, and time\-cap hits are shown per lane with the construct\-design cost per attempt\.Table D2:Complete performance metrics across evaluated lanes\. Pass rates in percent with 95% confidence intervals and cost per attempt are shown for every system and reasoning setting\.
## Appendix EFailure Categories and Attempt Outcomes

Figure E1:First blocking failure of protocol\-development attempts at top effort\. Shares of graded attempts are shown per system, with counts inTable[E1](https://arxiv.org/html/2610.33355#A5.T1)\.Table E1:First blocking failures by system for protocol\-development tasks at top effort\.Table E2:Breakdown of execution and grading statuses for protocol\-development attempts\.
## Appendix FReviewed Construct\-Design Submissions

Adjudication outcomes are summarized inTable[F1](https://arxiv.org/html/2610.33355#A6.T1), and every reviewed attempt at the top reasoning setting is listed inTable[F2](https://arxiv.org/html/2610.33355#A6.T2)\.Table[F3](https://arxiv.org/html/2610.33355#A6.T3)gives, for the ten judged tasks, how many top\-effort attempts passed the deterministic checks and how many of those the judge scored below 0\.9\. Review decides only whether a submission passes the deterministic checks\. On the ten judged tasks the final score also includes the rubric judge, so a reviewed attempt can pass the deterministic checks and still fall below the 0\.9 threshold; five top\-effort attempts did \(four by Gemini 4 Argon and one by Gemini 3\.8 Flash\), and they are counted as failures inTables[2](https://arxiv.org/html/2610.33355#S4.T2)and[G2](https://arxiv.org/html/2610.33355#A7.T2)\.

Table F1:Adjudication outcomes for construct\-design submissions\. Counts are shown for the eight top\-effort lanes and for all 24 lanes\.Table F2:Reviewed construct\-design attempts at top effort\. Deterministic results after review and final results at the 0\.9 threshold are listed with the basis for each decision\.Table F3:Judge threshold on the ten rubric\-judged tasks at top effort\. Attempts that passed the deterministic checks and, of those, the number the judge scored below 0\.9 are shown per system\.
## Appendix GRepeated Sampling and Per\-Task Results

Confidence intervals inTable[G1](https://arxiv.org/html/2610.33355#A7.T1)use the same bootstrap draws asTable[2](https://arxiv.org/html/2610.33355#S4.T2)\.

Table G1:Protocol\-development pass@kkand task coverage at top effort\. Pass rates in percent with 95% confidence intervals are shown atk=1k=1,55, and1010with the number of tasks solved at least once and in every attempt\.Table G2:Construct\-design solves per task and system at top effort\. Each cell gives the number of five attempts that passed\.