评估和提升 AI 智能体对测序方案的理解能力

# Evaluating and improving AI agents for sequencing protocol understanding
Source: [https://arxiv.org/html/2610.37851](https://arxiv.org/html/2610.37851)
\\workshoptitle

NeurIPS 2026 Workshops

Chi\-Lam Poon††thanks:Corresponding author:poonc2@mskcc\.orgMichael GoneosAffiliation:Weill Cornell MedicineAffiliation:NASA Ames Research CenterZezheng SongAffiliation:University of Maryland, College ParkXi ChenAffiliation:Southern University of Science and Technology

###### Abstract

Existing scientific\-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays\. We introduce LibStructBench, a document\-grounded benchmark for reconstructing sequencing\-library structures and molecular workflows\. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular\-state representation and workflow connectivity\. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular\-state reconstruction\. Additional knowledge improved oligo\-family recovery in a small pilot without improving mean molecular\-workflow scores\. Autonomous accumulation of procedural rules and cross\-protocol examples showed non\-monotonic validation performance and no mean held\-out molecular\-workflow improvement in the tested trajectory\. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols\.

## 1Introduction

Correct analysis of sequencing data depends on knowing the molecular structure of the library being sequenced\. Adapter sequences, cell and sample barcodes, unique molecular identifiers \(UMIs\), their positions and strand orientation determine how raw sequencing reads should be extracted, trimmed, demultiplexed, and quantified\. This is particularly important for customized or newly developed single\-cell assays, where barcode and UMI layouts may differ substantially from standard commercial protocols\. Machine\-readable specifications such asseqspec\[[6](https://arxiv.org/html/2610.37851#bib.bib1)\]demonstrate that explicit library descriptions can support standardized preprocessing across genomics assays\. However, such approaches assume that the underlying library structure has already been established\.

For new or customized assays, establishing this structure can itself require substantial manual interpretation of the experimental design, followed by custom code for cell and molecule recovery and assessment of assay sensitivity and specificity\. Moreover, diagnosing a failed sequencing run is even harder: it requires not only the final read structure but also an understanding of all molecular intermediates and experimental operation steps\. AI agents based on frontier large language models \(LLMs\) could potentially automate this process by reconstructing the library, generating assay\-specific preprocessing logic, inspecting raw sequencing data, and tracing quality\-control failures back to candidate experimental steps\. Such capability could substantially accelerate the development and debugging of new sequencing technologies\.

Sequencing protocols distribute relevant evidence across text, supplementary tables and files, and figures\. An agent must therefore identify which components are relevant, assign them to the correct molecular roles and orientations, and track how intermediate molecules are transformed into the final library\. This makes sequencing\-protocol reconstruction a stringent test of evidence\-based domain\-specific reasoning rather than simply document retrieval\. In this work, we introduce LibStructBench to measure AI agent’s ability to extract and reconstruct the molecular logic of sequencing protocols from their source documents\. The benchmark decomposes this problem into three linked tasks that model the physical construction of sequencing libraries \(Figure[1](https://arxiv.org/html/2610.37851#S2.F1)\): Task 1 reconstructs the final sequencing\-library structure; Task 2 recovers workflow\-linked oligo families and their roles; and Task 3 reconstructs library generation as a graph of molecular states and transitions\. Human curated ground\-truth records were audited against protocol evidence\.

Our experiments show that frontier LLMs often reconstruct final\-library sequences approximately but remain unreliable at exact reconstruction\. For molecular\-workflow reconstruction, no native system dominated all metrics, and model\-associated differences were most pronounced for molecular\-state reconstruction\. Failure analysis identified frequent discrepancies in molecular\-state assembly, workflow connectivity, and missing or unsupported components\. The broad baseline findings remained largely robust to alternative scoring assumptions, while repeated executions revealed variability at the individual\-protocol level\. Protocol\-name\-only controls further demonstrated substantial reconstruction capability without supplied documents, highlighting the importance of verifying protocol\-specific information against primary evidence\. Neither additional domain knowledge nor cumulative learning consistently improved molecular\-workflow reconstruction in the tested settings\. These observations motivate the development of agents that can explicitly ground and verify molecular structures and transformations\.

## 2Benchmark construction and validation

LibStructBench was built fromscg\_lib\_structs, an open resource of manually reconstructed sequencing\-library primer and adapter elements \(hereafter, oligos\) and step\-by\-step library\-generation procedures for single\-cell genomics methods \([https://teichlab\.github\.io/scg\_lib\_structs/](https://teichlab.github.io/scg_lib_structs/)\)\. The resource spans diverse assay classes, including gene expression, chromatin accessibility, DNA methylation, genomic DNA, and multi\-omic protocols\. We selected more than 70 curated commercial and academic sequencing protocols with diverse library architectures as the initial legacy records for LibStructBench\.

For each protocol, we assembled an agent\-visible evidence bundle containing available experimental protocol or technical documentation, primary journal article, supplementary files, oligo tables, and molecular figures\. These bundles were frozen and content\-addressed before evaluation and defined the evidence boundary for reconstruction\. Agents could inspect and process the supplied files but were not permitted to search the web or retrieve external protocol\-specific information\. Final ground\-truth records were produced only after the agent\-audit and human\-adjudication procedure described below\.

The benchmark datasets and implementation are publicly available:

- •
- •
- •

The ground\-truth release manifests retain recorded source URLs, document versions where available, and content hashes\. These manifests do not grant redistribution rights for the underlying documents, which retain their original licensing restrictions\.

### 2\.1Benchmark tasks and output representations

![Refer to caption](https://arxiv.org/html/2610.37851v1/fig1_benchmark.png)Figure 1:LibStructBench overview\.Given a protocol bundle containing primary source evidence, an AI agent reconstructs three linked representations of sequencing\-library construction, corresponding to Tasks 1\-3\. The schematic illustrates these tasks using a simplified workflow for 10x Chromium 3′Gene Expression v4\[[1](https://arxiv.org/html/2610.37851#bib.bib33)\]\.#### Task 1: final sequencing\-library reconstruction\.

The final library structure is a key piece of information for raw sequencing\-data processing and sequencing run quality assessment\. Given a protocol bundle, the agent must reconstruct the canonical5′→3′5^\{\\prime\}\\rightarrow 3^\{\\prime\}structure of each defined final library\. Fixed nucleotide segments and typed variable regions are normalized before scoring\. Reconstruction accuracy is evaluated using normalized sequence similarity based on Levenshtein distance \(edit distance\),

s⁡\(x,y\)=1−dLev​\(x,y\)max⁡\(\|x\|,\|y\|\)\.s\(x,y\)=1\-\\frac\{d\_\{\\mathrm\{Lev\}\}\(x,y\)\}\{\\max\(\|x\|,\|y\|\)\}\.\(1\)

#### Task 2: workflow\-linked oligo recovery\.

Oligos are the physical reagents that include adapters, priming sites, barcodes, and other technical elements into a sequencing library\. Extracting them therefore tests whether the reconstructed molecular workflow is grounded in the correct building blocks, and the ground truth of this task is mainly the "Adapter and primer sequences" section inscg\_lib\_structs\. The agent returns one structured record per oligo family, including its sequence, name and alias, role, molecular kind, orientation, ordered components, and modifications\. Ordered components are used for assembled, double\-stranded, or hairpin oligos that cannot be represented faithfully as a single flat sequence\.

Required ground\-truth oligo families are those linked to Task 3 with source\-recoverable sequences\. Agent\-extracted oligos with compatible sequence scaffolds and metadata are collapsed into predicted families and matched one\-to\-one to required families, with eligible optional or non\-recoverable matches neutralized\. For each extraction\-ground\-truth pair, we compute a weighted similarity

wi​j=0\.65​sseq\+0\.15​smod\+0\.10​skind\+0\.05​sorient\+0\.05​srole\.w\_\{ij\}=0\.65\\,s\_\{\\mathrm\{seq\}\}\+0\.15\\,s\_\{\\mathrm\{mod\}\}\+0\.10\\,s\_\{\\mathrm\{kind\}\}\+0\.05\\,s\_\{\\mathrm\{orient\}\}\+0\.05\\,s\_\{\\mathrm\{role\}\}\.\(2\)
The weights summarize sequence, modification, molecular kind, orientation, and role similarity, and are renormalized over supported components\. Ifℳ∗\\mathcal\{M\}^\{\*\}denotes the resulting one\-to\-one matching, the primary Task 2 score is

F​1oligo=2​∑\(i,j\)∈ℳ∗wi​jNouteff\+Nreq\.F1\_\{\\mathrm\{oligo\}\}=\\frac\{2\\sum\_\{\(i,j\)\\in\\mathcal\{M\}^\{\*\}\}w\_\{ij\}\}\{N\_\{\\mathrm\{out\}\}^\{\\mathrm\{eff\}\}\+N\_\{\\mathrm\{req\}\}\}\.\(3\)
Here,ℛ\\mathcal\{R\}denotes required, source\-recoverable ground\-truth families,𝒫\\mathcal\{P\}all collapsed predicted families, and𝒵⊆𝒫\\mathcal\{Z\}\\subseteq\\mathcal\{P\}predictions neutralized by eligible optional or non\-recoverable matches\. Neutralization uses exact matches or a correspondence threshold of 0\.50 for used non\-recoverable families, while preserving required matches\. Thus,Nreq=\|ℛ\|N\_\{\\mathrm\{req\}\}=\|\\mathcal\{R\}\|andNouteff=\|𝒫∖𝒵\|N\_\{\\mathrm\{out\}\}^\{\\mathrm\{eff\}\}=\|\\mathcal\{P\}\\setminus\\mathcal\{Z\}\|, counting both matched and unmatched non\-neutralized predictions\. WhenNreq=0N\_\{\\mathrm\{req\}\}=0, F1 is 1 ifNouteff=0N\_\{\\mathrm\{out\}\}^\{\\mathrm\{eff\}\}=0, and 0 otherwise\. These protocols are excluded from primary Task 2 macro\-F1\.

#### Task 3: molecular library\-generation workflow\.

Task 3 reconstructs the “Step\-by\-step library generation” section inscg\_lib\_structsas a directed state\-transition graph\. Shared upstream steps are represented once, with modality\-specific or alternative routes branching toward their terminal states\. Each state records the physical molecular strands and their5′→3′5^\{\\prime\}\\\!\\rightarrow\\\!3^\{\\prime\}architecture, while each transition records the input states, molecular operation, oligos, products, and whether those products are carried forward or discarded\. For example, reverse transcription \(operation\) converts an RNA template \(substrate\) using an RT primer \(oligo\) into an RNA–cDNA intermediate \(product\)\. And in the fragmentation example shown in Figure[1](https://arxiv.org/html/2610.37851#S2.F1), fragmentation generates multiple product classes from double\-stranded cDNA \(substrate\), with the barcode\-bearing cDNA fragment carried forward and internal or TSO\-end fragments discarded\.

A transition is represented asτ=\(Sin,O,Q,Sout,Scarried,Sdiscarded\),\\tau=\\left\(S\_\{\\mathrm\{in\}\},O,Q,S\_\{\\mathrm\{out\}\},S\_\{\\mathrm\{carried\}\},S\_\{\\mathrm\{discarded\}\}\\right\),whereSinS\_\{\\mathrm\{in\}\}andSoutS\_\{\\mathrm\{out\}\}are the input and output molecular states \(e\.g\. RNA and RNA\-cDNA intermediate\),OOis the molecular operation, andQQis the set of linked Task 2 oligo families\. Agent\-reconstructed and ground\-truth workflows are first matched using their terminal modalities, states, and transitions\. Within each matched workflow, states and transitions are aligned through global partial one\-to\-one matching, with candidate pairs below a context\-adjusted assignment score of0\.250\.25left unmatched\. For each matched transition pair, scientific similarity is

si​jτ=0\.30​soperation\+0\.15​ssubstrates\+0\.20​sproducts\+0\.15​sdisposition\+0\.20​soligos,s^\{\\tau\}\_\{ij\}=0\.30s\_\{\\mathrm\{operation\}\}\+0\.15s\_\{\\mathrm\{substrates\}\}\+0\.20s\_\{\\mathrm\{products\}\}\+0\.15s\_\{\\mathrm\{disposition\}\}\+0\.20s\_\{\\mathrm\{oligos\}\},\(4\)which compares the operation, aligned input and product states, carried\-versus\-discarded disposition, and transition\-linked oligos\.

For entity typeX∈\{τ,S\}X\\in\\\{\\tau,S\\\}, representing transitions or states, letℳX∗\\mathcal\{M\}^\{\*\}\_\{X\}denote the matched pairs,NoutX,effN\_\{\\mathrm\{out\}\}^\{X,\\mathrm\{eff\}\}the number of scored agent\-output entities, andNgtX,scoreN\_\{\\mathrm\{gt\}\}^\{X,\\mathrm\{score\}\}the number of scorable ground\-truth entities\. Transition and state F1 are computed using the same soft\-F1 form:

F​1X=2​∑\(i,j\)∈ℳX∗si​jXNoutX,eff\+NgtX,score,X∈\{τ,S\}\.F1\_\{X\}=\\frac\{2\\sum\_\{\(i,j\)\\in\\mathcal\{M\}^\{\*\}\_\{X\}\}s^\{X\}\_\{ij\}\}\{N\_\{\\mathrm\{out\}\}^\{X,\\mathrm\{eff\}\}\+N\_\{\\mathrm\{gt\}\}^\{X,\\mathrm\{score\}\}\},\\qquad X\\in\\\{\\tau,S\\\}\.\(5\)Matched entities receive fractional credit, unmatched scored outputs reduce precision, and missing ground\-truth entities reduce recall\. For states,si​jSs^\{S\}\_\{ij\}weights reference\-strand sequence0\.500\.50, molecular architecture0\.150\.15, ordered strand and segment structure0\.200\.20, and pairing or discontinuities0\.150\.15\.

To evaluate graph topology separately, we compare typed substrate, carried\-product, and discarded\-product edges between aligned states\. LetmEm\_\{E\}denote the number of matching typed edges\. Using the same effective output and scorable ground\-truth counts, edge F1 is

F​1edge=2​mENoutE,eff\+NgtE,score\.F1\_\{\\mathrm\{edge\}\}=\\frac\{2m\_\{E\}\}\{N\_\{\\mathrm\{out\}\}^\{E,\\mathrm\{eff\}\}\+N\_\{\\mathrm\{gt\}\}^\{E,\\mathrm\{score\}\}\}\.\(6\)

### 2\.2Ground\-truth construction, audit, and adjudication

Human\-curated records inscg\_lib\_structsmay contain reconstruction errors, while the underlying protocol documents may themselves be ambiguous or internally inconsistent\. We therefore audited every candidate benchmark record before evaluation\. For each protocol, the legacy curation was first converted into the canonical Task 1–3 ground\-truth schema\. An AI agent then compared this candidate record against the frozen source bundle and generated proposed corrections for human review\.

Codex implemented the deterministic audit framework, including source manifests, schemas, Task 1–3 validation, review checkpoints, and application of approved changes\. Claude Code performed the protocol\-level audit, checking each candidate against the protocol sources and flagging unsupported claims, ambiguities, protocol\-version conflicts, and proposed corrections\. A human expert adjudicated every substantive finding by accepting, revising, or rejecting the proposed change and directly reviewed the oligos, molecular states, and transitions against the primary sources\. Only human\-approved changes were applied to the benchmark record \(Figure[1\(a\)](https://arxiv.org/html/2610.37851#A2.F1.sf1)\)\. In the summarized audit, 54 of 69 confirmed findings \(78\.3%\) were categorized as legacy\-curation errors \(Figure[1\(b\)](https://arxiv.org/html/2610.37851#A2.F1.sf2)\)\. This statistic describes error causes rather than the origin of accepted corrections, which could not be reliably determined from the available adjudication records\. Future ground\-truth audit updates and corrections will be documented at[https://acolytics\.com/proofread](https://acolytics.com/proofread)\.

## 3Experiments and results

### 3\.1Interface and PDF\-conversion controls

#### Direct LLM API and coding\-agent comparison\.

We first assessed final\-library reconstruction accuracy and whether native coding\-agent interfaces improved performance over direct LLM APIs\. Three models were evaluated across[20 sequencing protocols](https://huggingface.co/datasets/sequencing/scg-protocols-v1-groundtruth): Claude Opus 5, GPT\-5\.6 Sol, and Gemini 3\.7 Flash, using their direct APIs and corresponding native agent interfaces \(Claude Code, Codex, and Antigravity CLI\) with high reasoning effort via Harbor\[[16](https://arxiv.org/html/2610.37851#bib.bib16)\]\. Direct APIs received protocol PDFs without tools, whereas agents could use native file, shell, and document\-processing tools\. Excel sheets were excluded in inputs here because direct APIs do not accept this format\.

Mean paired similarity differences \(agent minus API\) were \+0\.033, \+0\.005, and \-0\.030 for Claude, GPT, and Gemini, respectively\. None was statistically significant in two\-sided paired Wilcoxon signed\-rank tests with Bonferroni correction across the three model comparisons \(pBonf=1\.000p\_\{\\mathrm\{Bonf\}\}=1\.000for all; Figure[2](https://arxiv.org/html/2610.37851#S3.F2)\)\. Exact final\-library reconstruction remained uncommon, with only 3\-5 of 20 protocols \(15\-25%\) reconstructed exactly across configurations; API/agent win counts were 4/3 \(13 ties\), 1/1 \(18 ties\), and 3/1 \(16 ties\), respectively \(Figure[2](https://arxiv.org/html/2610.37851#S3.F2)\)\. Therefore, native coding agents showed no statistically detectable improvement in Task 1 accuracy\.

As final\-library similarity alone cannot characterize errors in molecular components and workflow reconstruction, we further evaluated Tasks 2 and 3 using coding agents, whose tool\-calling and structured\-execution capabilities support more complex evaluations and improvement\.

Paired Task 1 performance

Figure 2:Task 1 performance for direct APIs versus native coding agents\.Each point represents one sequencing protocol completed by both interfaces\.Δ\\Deltadenotes the mean paired similarity difference \(agent minus API\), andpadjp\_\{\\mathrm\{adj\}\}denotes the adjustedpp\-value for two\-sided paired Wilcoxon signed\-rank test\. The inset table reports mean normalized sequence similarity and exact reconstruction counts for each interface\.
#### Text\-layer nucleotide sequences are reliably recovered from PDFs\.

Since sequence reconstruction errors could arise from technical PDF text conversion rather than downstream protocol reasoning, we tested sequence recovery across five representative protocols containing 80 human\-verified, source\-visible nucleotide strings\. Curated human text, PyMuPDF\[[3](https://arxiv.org/html/2610.37851#bib.bib12)\], pypdf\[[11](https://arxiv.org/html/2610.37851#bib.bib13)\], Docling without OCR\[[20](https://arxiv.org/html/2610.37851#bib.bib14)\], and MinerU OCR\[[36](https://arxiv.org/html/2610.37851#bib.bib15)\]were compared, and a fixed Codex independently recovered sequence strings\. Each ground\-truth sequenceyjy\_\{j\}was scored by its best match among the recovered candidates,maxi⁡s⁡\(xi,yj\)\\max\_\{i\}s\(x\_\{i\},y\_\{j\}\), using the normalized Levenshtein sequence similarity defined in Eq\. \([1](https://arxiv.org/html/2610.37851#S2.E1)\)\.

Text from human, PyMuPDF, pypdf, and Docling recovered all 80 sequences exactly \(Figure[S2](https://arxiv.org/html/2610.37851#A3.F2)\), but MinerU OCR recovered only 71 of 80 exact sequences\. The OCR errors included corruption of repetitive character strings such as "GGGGGG"\. These results suggest that text\-layer nucleotide extraction alone is unlikely to explain the observed reconstruction errors\. However, this does not assess sequences embedded in figures or the interpretation of graphical molecular structures\.

### 3\.2Baseline molecular\-workflow reconstruction

#### Baseline performance across models and harnesses\.

To assess how reliably coding agents could reconstruct the underlying molecular workflow and components, we benchmarked four native coding\-agent systems on the same 20\-protocol panel: GPT\-5\.6 Sol Max with Codex, Claude Opus 5 Max with Claude Code, Gemini 3\.7 Flash High with Antigravity CLI, and Kimi K3 Max with Kimi Code\. Each system received the same frozen protocol bundles, task instructions, output schema, and deterministic verifier similarly through Harbor\[[16](https://arxiv.org/html/2610.37851#bib.bib16)\]\. For Task 2, ddSEQ was excluded from the macro\-average because it contained no required, source\-recoverable oligo families \(Nreq=0N\_\{\\mathrm\{req\}\}=0\)\. Mean Task 2 oligo\-family F1 ranged from 0\.724 to 0\.774 with required recoverable oligo families\. Task 3 molecular\-transition, molecular\-state, and typed\-edge F1 ranged from 0\.767–0\.814, 0\.495–0\.723, and 0\.721–0\.754, respectively, across all 20 protocols \(Figure[3\(a\)](https://arxiv.org/html/2610.37851#S3.F3.sf1)\)\. No single system dominated all metrics\. Kimi obtained the highest Task 2 F1, Gemini the highest transition F1, Claude the highest state F1, and GPT the highest typed\-edge F1\. Molecular\-state F1 showed the largest between\-system variation\.

We then asked whether the agent harness altered performance for the same underlying model\. GPT, Claude, and Gemini were additionally paired with Pi\[[39](https://arxiv.org/html/2610.37851#bib.bib17)\]and mini\-SWE\-agent\[[38](https://arxiv.org/html/2610.37851#bib.bib18)\], yielding nine model\-harness systems when combined with their native interfaces\. These systems were compared on 10 protocols common to all configurations, drawn from the 20\-protocol baseline panel\. Since molecular\-transition F1 jointly evaluates the operation, substrates, products, linked oligos, and product disposition, we used it as the primary Task 3 metric for this comparison\. Mean molecular\-transition F1 ranged from 0\.714 to 0\.805 across the nine model\-harness systems \(Figure[3\(b\)](https://arxiv.org/html/2610.37851#S3.F3.sf2)\)\. Hierarchical clustering did not clearly separate systems by model or harness, although molecular\-state performance showed more pronounced model\-associated patterns\. \(Supplementary Figure[S4](https://arxiv.org/html/2610.37851#A6.F4)\)\.

Estimated costs varied across systems \(Figure[5\(a\)](https://arxiv.org/html/2610.37851#A7.F5.sf1)\) and were not monotonically related to reconstruction performance \(Figure[5\(b\)](https://arxiv.org/html/2610.37851#A7.F5.sf2)\)\. To quantify model and harness effects, we performed a protocol\-paired analysis across three models and two shared harnesses \(Pi and mini\-SWE\-agent\)\. Model\-switch differences exceeded harness\-switch differences on average across all four metrics, although only molecular\-state F1 remained statistically significant after Bonferroni correction \(Δ=0\.079\\Delta=0\.079, pointwise 95% CI \[0\.038, 0\.124\],pBonf=0\.016p\_\{\\mathrm\{Bonf\}\}=0\.016; Supplementary Figure[S6](https://arxiv.org/html/2610.37851#A7.F6)\)\.

\(a\)\(b\)
Figure 3:Baseline agent comparison\.\(a\)Mean Task 2 oligo\-family and Task 3 transition, state, and typed\-edge F1 for native CLI systems\.\(b\)Task 3 molecular\-transition F1 across 10 common protocols and nine model\-harness systems, clustered by protocol\-level performance\.
#### Variability across repeated executions\.

As our baseline comparisons used a single execution per model\-harness system and protocol, we assessed how aggregate performance and individual\-protocol scores varied across repeated executions of the same agent configuration\. We evaluated GPT\-Codex across three executions per protocol on the 20\-protocol baseline panel\. Averaging the three executions changed the original oligo\-family, molecular\-transition, molecular\-state, and typed\-edge macro\-F1 scores by−0\.005\-0\.005,−0\.014\-0\.014,−0\.012\-0\.012, and−0\.024\-0\.024, respectively\. Despite these relatively small aggregate differences, within\-protocol F1 ranges exceeded 0\.10 for 5/19 informative oligo protocols and 9/20, 8/20, and 12/20 protocols for the three Task 3 metrics, respectively \(Appendix[D](https://arxiv.org/html/2610.37851#A4)\)\. Although individual protocols showed execution\-to\-execution variability, the aggregate performance estimates remained broadly consistent across the three executions, with substantial room for improvement in molecular\-workflow reconstruction\. Given the computational cost of repeated evaluations across multiple models and harnesses, we retained single\-execution comparisons for the broader benchmark\.

#### Sensitivity to scoring assumptions\.

Similarity weights and matching thresholds may influence reported scores and system rankings, we therefore recomputed matching and scoring across 640 configurations \(Appendix[K](https://arxiv.org/html/2610.37851#A11)\)\. The leading native systems for oligo\-family, molecular\-state, and typed\-edge F1 remained unchanged\. Typed\-edge F1 was also more sensitive to matching thresholds than transition or state F1\. Overall, the broad baseline findings were robust to the tested scoring assumptions, although absolute scores and close system rankings warrant cautious interpretation\.

#### Reconstruction without protocol source bundles\.

Many sequencing protocols andscg\_lib\_structsare publicly available and may have appeared in LLM training data, we further evaluated protocol\-name\-only controls on the 20\-protocol panel \(Appendix[E](https://arxiv.org/html/2610.37851#A5)\)\. In a separate Task 1 API experiment, name\-only prompting increased mean sequence similarity for all three models but yielded fewer exact reconstructions\. Only Claude’s similarity improvement remained significant after Bonferroni correction \(padj=0\.0215p\_\{\\mathrm\{adj\}\}=0\.0215\)\. For native Tasks 2\-3, name\-only prompting yielded lower mean F1 across all three systems and four metrics, although none of the 12 paired comparisons remained significant after Bonferroni correction\. These results indicate substantial reconstruction ability without supplied evidence but do not establish training\-data memorization or that source bundles are unnecessary\.

#### Failure modes\.

To characterize where agent reconstructions disagreed with the ground truth, we applied deterministic error analysis to 100 outputs from ten systems across ten common protocols\. For each output, the verifier recorded whether at least one discrepancy occurred in each of six predefined categories \(Table[S3](https://arxiv.org/html/2610.37851#A8.T3)\)\. Molecular\-state assembly and workflow\-topology/disposition discrepancies were identified in all 100 outputs \(Figure[7\(a\)](https://arxiv.org/html/2610.37851#A8.F7.sf1)\)\. Missing recoverable information occurred in 88/100 outputs, unsupported additions in 77/100, operation or target mismatches in 24/100, and strand or orientation errors in only 8/100\. Exploratory clustering of system\-level discrepancy profiles tended to group configurations by underlying LLM \(Figure[7\(a\)](https://arxiv.org/html/2610.37851#A8.F7.sf1)\), while protocol\-level profiles showed variation in missing information, unsupported additions, and operation or target mismatches \(Figure[7\(b\)](https://arxiv.org/html/2610.37851#A8.F7.sf2)\)\. These categories are nonexclusive, as molecular\-state alignment errors can propagate into topology discrepancies; their frequencies therefore describe disagreement with the canonical reconstruction rather than independent failure mechanisms\.

### 3\.3Improving molecular\-workflow reconstruction

Accurate protocol understanding is important for downstream sequencing quality control, where errors in library structure and oligo definitions can directly affect read parsing and quantitative analyses\. Baseline results showed that none of the native model\-harness systems achieved consistently accurate reconstruction, we next explored whether supplying additional domain knowledge or accumulating reviewed experience could improve agent performance\.

\(a\)\(b\)\(c\)\(d\)
Figure 4:Improving molecular\-workflow reconstruction\.\(a\)Mean Task 2 and Task 3 scores under the baseline, general\-knowledge, cross\-protocol memory, and combined conditions\.\(b\)Autonomous cumulative\-improvement framework\.\(c\)Validation performance across autonomous\-training checkpoints\.\(d\)Held\-out test performance of baseline and the validation\-selected checkpoint with and without knowledge base \(KB\) supplement\. For panels \(a\) and \(c\), Task 2 includes four informative validation protocols and Task 3 includes five; the two experiments use different validation panels\. Panel \(d\) includes nine informative held\-out protocols for Task 2 and ten protocols for Task 3\.#### External knowledge had task\-dependent effects in this pilot\.

GPT\-5\.6 Sol Max with Codex was used to examine whether external domain knowledge or cross\-protocol examples could improve Tasks 2 and 3 performance on a fixed five\-protocol validation panel \(sci\-ATAC\-seq, scRRBS, SMART\-seq, SHARE\-seq, ddSEQ\)\. Each protocol was tested under four paired conditions: 1\) the protocol bundle alone \(baseline\); 2\) a general molecular\-method primer derived from textbooks and review articles \(Table[S4](https://arxiv.org/html/2610.37851#A9.T4)\); 3\) ground\-truth records from five separate donor protocols, reformatted as example agent outputs and supplied as cross\-protocol exemplar memory \(Table[S5](https://arxiv.org/html/2610.37851#A9.T5)\); and 4\) both knowledge sources\. The baseline achieved mean oligo\-family, molecular\-transition, molecular\-state, and typed\-edge F1 scores of 0\.826, 0\.800, 0\.636, and 0\.673, respectively \(Figure[4\(a\)](https://arxiv.org/html/2610.37851#S3.F4.sf1)\)\. Combining general knowledge with cross\-protocol memory increased oligo\-family F1 to 0\.877, a paired improvement of 0\.050 \(pointwise 95% protocol\-bootstrap CI \[0\.022, 0\.079\]\)\. General knowledge alone and memory alone achieved oligo\-family F1 scores of 0\.777 and 0\.814, respectively\. All three augmented conditions had lower Task 3 mean scores than baseline, although their paired confidence intervals included zero\. These results show a task\-dependent response in the saved outputs: combined knowledge improved informative\-protocol oligo recovery, while no augmentation exceeded baseline Task 3 point estimates\.

#### Autonomous cumulative learning\.

We tested whether an agent could improve molecular\-workflow reconstruction by accumulating experience across previously verified protocols\. The 73 usable protocols were split into 58 training, 5 validation, and 10 held\-out test protocols\. Validation panel includes CEL\-seq, chATAC\-seq, ddSEQ, HyDrop\-RNA, scRRBS\. Training proceeded in batches of five protocols\. We denote the initial agent as checkpointC0C\_\{0\}\(baseline\), and each subsequent checkpointCkC\_\{k\}as the agent state after learning fromkktraining protocols\. After each batch, the agent autonomously updated two components: 1\) a protocol\-neutral procedural rule set derived from observed agent errors and 2\) a knowledge base of oligos and molecular graphs derived from completed training ground truth\. These components were frozen as the next checkpoint, and each checkpoint was evaluated on the same five validation protocols without using validation outputs for learning \(Figure[4\(b\)](https://arxiv.org/html/2610.37851#S3.F4.sf2)\)\.

Validation performance was non\-monotonic across checkpoints\. Using the aggregate validation criterion,C30C\_\{30\}was selected, whereas performance declined at later checkpoints, with substantial changes appearing byC30C\_\{30\}\-C40C\_\{40\}\. The final checkpointC58C\_\{58\}scored belowC0C\_\{0\}on all four validation metrics \(Figure[4\(c\)](https://arxiv.org/html/2610.37851#S3.F4.sf3)\)\. We then evaluated the selectedC30C\_\{30\}procedural rules on the 10 held\-out protocols, both alone and supplemented with the accumulated training knowledge base\. Relative toC0C\_\{0\}, theC30C\_\{30\}rules alone did not improve any Task 3 metric\. Adding the knowledge base increased the held\-out oligo\-family F1 point estimate from 0\.560 to 0\.621 \(paired difference 0\.061; pointwise 95% protocol\-bootstrap CI \[\-0\.024, 0\.159\]\)\. Transition, state, and typed\-edge F1 means remained below baseline \(0\.684 vs\. 0\.721, 0\.523 vs\. 0\.592, and 0\.588 vs\. 0\.674, respectively; Figure[4\(d\)](https://arxiv.org/html/2610.37851#S3.F4.sf4)\)\. Thus, this single learning trajectory showed non\-monotonic validation performance and no held\-out Task 3 point\-estimate gain\. These findings are conditional on the saved outputs and canonical graph representation; they do not establish that cumulative learning generally fails to improve molecular understanding\.

## 4Discussion

Our results highlight a gap between recovering sequencing\-library components and faithfully reconstructing molecular workflows\. Agents often identified experimental operations but struggled with intermediate molecular states and workflow connectivity, explaining relatively high transition F1 despite lower state F1\. Task 2 and Task 3 scores measure agreement with canonical references rather than independently validated biological correctness or downstream usability\. The broad findings were robust to tested scoring assumptions, while repeated GPT\-Codex executions revealed protocol\-level variability despite similar aggregate performance\.

#### Model and harness effects\.

No native model\-harness system consistently outperformed the others across all reconstruction metrics\. In our paired comparison across three models and two shared harnesses, switching the underlying language model produced larger average performance differences than switching harnesses, with a statistically significant difference only for molecular\-state reconstruction\. This suggests that representing intermediate molecular architectures is particularly sensitive to the underlying model in the tested configurations\.

#### Additional knowledge and autonomous learning had task\-dependent effects\.

Combining general biological knowledge with cross\-protocol examples improved oligo\-family recovery in our small pilot without improving mean molecular\-workflow scores\. Similarly, accumulated knowledge increased the held\-out oligo\-family F1 point estimate, with uncertainty spanning zero, but did not improve Task 3 mean performance\. These findings suggest that recovering additional molecular components does not necessarily translate into correctly integrating them into a molecular workflow\.

One possible explanation is that models already possess substantial general sequencing knowledge, as suggested by the name\-only controls\. Additional context may therefore offer limited benefit while complicating the selection and integration of protocol\-specific evidence\. Moreover, cumulative learning showed non\-monotonic validation performance, with some learned rules associated with omitted molecular intermediates or altered workflow connectivity\. Some discrepancies may also reflect differences in representation granularity, as chemically valid intermediate steps can be penalized by canonical graph matching\. Furthermore, the verifier may penalize chemically valid reconstructions when their molecular\-state or workflow representations differ from the canonical ground truth\.

#### Future improvement should emphasize controlled and verifiable learning\.

Our document\-grounded setting reflects the intended application to new or unpublished sequencing protocols, for which protocol\-specific information may not be publicly available\. We therefore use publicly documented protocols fromscg\_lib\_structsas an auditable benchmark, while restricting agents to frozen source bundles rather than external retrieval\.

The name\-only controls suggest that agents can leverage prior knowledge to generate candidate reconstructions, which could then be verified against protocol\-specific evidence\. Future agents could first propose a provisional molecular workflow, then extract evidence from supplied documents to verify, revise, or reject individual components and transitions\. Model Context Protocol \(MCP\)\-accessible tools could support document parsing, oligo validation, molecular\-state construction, and graph\-consistency checking, while cross\-protocol memory could enable iterative improvement\. Learned rules should be reversible and validated against previously solved protocols before retention\. Future work should compare this prior\-guided approach with direct document\-grounded reconstruction and accommodate biologically equivalent workflow representations\.

## Acknowledgments

We thank Ruslan Soldatov \(Memorial Sloan Kettering Cancer Center\) and Xinming Tu \(Phylo Bio\) for valuable discussions and guidance\. We also acknowledge the Google Cloud Research Credits Program for providing Google Cloud and Gemini API credits, and the MIT–Anthropic AI for Science program and Claude hackathon for providing Claude usage and API credits\.

## References

- \[1\]10x Genomics\(2024\)Chromium gem\-x single cell 3’ v4 gene expression user guide\.10x Genomics\.External Links:[Link](https://www.10xgenomics.com/support/universal-three-prime-gene-expression/documentation/steps/library-prep/chromium-gem-x-single-cell-3-v4-gene-expression-user-guide)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.3.3.1.1),[Figure 1](https://arxiv.org/html/2610.37851#S2.F1),[Figure 1](https://arxiv.org/html/2610.37851#S2.F1.6.1)\.
- \[2\]A\. F\. Akyürek, X\. Tu, A\. Gutmanstein, J\. Qin, D\. Agarwal, S\. Monasdotter, S\. Chekhov, B\. Hernandez Villegas, K\. Chugunov, J\. Engel, V\. Chatrath, O\. Kavanagh, G\. Boo, E\. Hernandez, Y\. Liu, Y\. Xue, A\. Sabharwal, D\. Y\. Zhang, Z\. Doctor, Y\. Qu, Y\. He, and S\. Hassaan\(2026\)DrugDiscoveryBench: can coding agents assist early\-stage drug discovery?\.Note:Scale Labs and Phylo research reportExternal Links:[Link](https://labs.scale.com/leaderboard/drugdiscoverybench)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px2.p2.1)\.
- \[3\]Artifex Software, Inc\.\(2026\)PyMuPDF documentation\.External Links:[Link](https://pymupdf.readthedocs.io/en/latest/)Cited by:[§3\.1](https://arxiv.org/html/2610.37851#S3.SS1.SSS0.Px2.p1.1)\.
- \[4\]J\. M\. S\. Bartlett and D\. Stirling \(Eds\.\)\(2003\)PCR protocols\.2 edition,Methods in Molecular Biology, Vol\.226,Humana Press,Totowa, NJ\.External Links:ISBN 978\-0\-89603\-642\-0Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.3.2.1.1)\.
- \[5\]Bio\-Rad Laboratories\(2024\)DdSEQ single\-cell 3’ rna\-seq kit user guide\.Bio\-Rad Laboratories\.Note:Version 1\.0, Ver\. BExternal Links:[Link](https://www.bio-rad.com/sites/default/files/2024-06/10000167449.pdf)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.6.3.1.1)\.
- \[6\]A\. S\. Booeshaghi, X\. Chen, and L\. Pachter\(2024\)A machine\-readable specification for genomics assays\.Bioinformatics40\(4\),pp\. btae168\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btae168)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.37851#S1.p1.1)\.
- \[7\]J\. Cao, J\. S\. Packer, V\. Ramani, D\. A\. Cusanovich, C\. Huynh, R\. Daza, X\. Qiu, C\. Lee, S\. N\. Furlan, F\. J\. Steemers, A\. Adey, R\. H\. Waterston, C\. Trapnell, and J\. Shendure\(2017\)Comprehensive single\-cell transcriptional profiling of a multicellular organism\.Science357\(6352\),pp\. 661–667\.External Links:[Document](https://dx.doi.org/10.1126/science.aam8940)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.6.3.1.1)\.
- \[8\]Z\. Chen, S\. Chen, Y\. Ning, Q\. Zhang, B\. Wang, B\. Yu, Y\. Li, Z\. Liao, C\. Wei, Z\. Lu, V\. Dey, M\. Xue, F\. N\. Baker, B\. Burns, D\. Adu\-Ampratwum, X\. Huang, X\. Ning, S\. Gao, Y\. Su, and H\. Sun\(2025\)ScienceAgentBench: toward rigorous assessment of language agents for data\-driven scientific discovery\.InInternational Conference on Learning Representations,External Links:2410\.05080,[Link](https://arxiv.org/abs/2410.05080)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px2.p2.1)\.
- \[9\]D\. A\. Cusanovich, R\. Daza, A\. Adey, H\. A\. Pliner, L\. Christiansen, K\. L\. Gunderson, F\. J\. Steemers, C\. Trapnell, and J\. Shendure\(2015\)Multiplex single cell profiling of chromatin accessibility by combinatorial cellular indexing\.Science348\(6237\),pp\. 910–914\.External Links:[Document](https://dx.doi.org/10.1126/science.aab1601)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.2.3.1.1)\.
- \[10\]F\. Erhard, A\. Saliba, A\. Lusser, C\. Toussaint, T\. Hennig, B\. K\. Prusty, D\. Kirschenbaum, K\. Abadie, E\. A\. Miska, C\. C\. Friedel, I\. Amit, R\. Micura, and L\. Dölken\(2022\)Time\-resolved single\-cell RNA\-seq using metabolic RNA labelling\.Nature Reviews Methods Primers2,pp\. 77\.External Links:[Document](https://dx.doi.org/10.1038/s43586-022-00157-z),[Link](https://doi.org/10.1038/s43586-022-00157-z)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.12.2.1.1)\.
- \[11\]M\. Fenniak, M\. Stamy, pubpub\-zz, M\. Thoma, M\. Peveler, exiledkingcc, and pypdf Contributors\(2024\)The pypdf library\.Note:See the pypdf documentation for the complete contributor listExternal Links:[Link](https://pypi.org/project/pypdf/)Cited by:[§3\.1](https://arxiv.org/html/2610.37851#S3.SS1.SSS0.Px2.p1.1)\.
- \[12\]C\. Gawad, W\. Koh, and S\. R\. Quake\(2016\)Single\-cell genome sequencing: current state of the science\.Nature Reviews Genetics17,pp\. 175–188\.External Links:[Document](https://dx.doi.org/10.1038/nrg.2015.16),[Link](https://doi.org/10.1038/nrg.2015.16)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.5.2.1.1)\.
- \[13\]A\. E\. Ghareeb, B\. Chang, L\. Mitchener,et al\.\(2026\)A multi\-agent system for automating scientific discovery\.Nature655,pp\. 497–505\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10652-y)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px2.p1.1)\.
- \[14\]J\. Gottweis, W\. Weng, A\. Daryin,et al\.\(2026\)Accelerating scientific discovery with co\-scientist\.Nature655,pp\. 487–496\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10644-y)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px2.p1.1)\.
- \[15\]H\. Guo, P\. Zhu, F\. Guo, X\. Li, X\. Wu, X\. Fan, L\. Wen, and F\. Tang\(2015\)Profiling dna methylome landscapes of mammalian cells with single\-cell reduced\-representation bisulfite sequencing\.Nature Protocols10,pp\. 645–659\.External Links:[Document](https://dx.doi.org/10.1038/nprot.2015.039)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.3.3.1.1)\.
- \[16\]Harbor Framework Team\(2026\)Harbor: a framework for evaluating and optimizing agents and models in container environments\.Note:Software frameworkExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.20953922),[Link](https://github.com/harbor-framework/harbor)Cited by:[§3\.1](https://arxiv.org/html/2610.37851#S3.SS1.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.37851#S3.SS2.SSS0.Px1.p1.1)\.
- \[17\]A\. J\. Harwood \(Ed\.\)\(1994\)Protocols for gene analysis\.Methods in Molecular Biology, Vol\.31,Humana Press,Totowa, NJ\.External Links:ISBN 978\-0\-89603\-258\-3,[Document](https://dx.doi.org/10.1385/0896032582),[Link](https://doi.org/10.1385/0896032582)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.4.2.1.1)\.
- \[18\]K\. Huang, S\. Zhang, H\. Wang, Y\. Qu, Y\. Lu, R\. Li, Y\. Roohani, L\. Qiu, S\. Cao, G\. Li, J\. Zhang, D\. Yin, R\. Wierenga, D\. Kavi, S\. Liu, T\. She, S\. Marwaha, J\. N\. Carter, X\. Zhou, M\. T\. Wheeler, J\. A\. Bernstein, M\. Wang, P\. He, J\. Zhou, M\. P\. Snyder, L\. Cong, A\. Regev, and J\. Leskovec\(2026\)Autonomous biomedical research with an artificial intelligence agent\.Science393\(6813\),pp\. eadz4351\.External Links:[Document](https://dx.doi.org/10.1126/science.adz4351)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px2.p1.1)\.
- \[19\]J\. M\. Laurent, J\. D\. Janizek, M\. Ruzo, M\. M\. Hinks, M\. J\. Hammerling, S\. Narayanan, M\. Ponnapati, A\. D\. White, and S\. G\. Rodriques\(2024\)LAB\-Bench: measuring capabilities of language models for biology research\.arXiv preprint arXiv:2407\.10362\.Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px2.p2.1)\.
- \[20\]N\. Livathinos, C\. Auer, M\. Lysak, A\. Nassar, M\. Dolfi, P\. Vagenas, C\. Berrospi Ramis, M\. Omenetti, K\. Dinkla, Y\. Kim, S\. Gupta, R\. Teixeira de Lima, V\. Weber, L\. Morin, I\. Meijer, V\. Kuropiatnyk, and P\. W\. J\. Staar\(2025\)Docling: an efficient open\-source toolkit for ai\-driven document conversion\.arXiv preprint arXiv:2501\.17887\.Cited by:[§3\.1](https://arxiv.org/html/2610.37851#S3.SS1.SSS0.Px2.p1.1)\.
- \[21\]H\. Lodish, A\. Berk, P\. Matsudaira, C\. A\. Kaiser, M\. Krieger, M\. P\. Scott, S\. L\. Zipursky, and J\. Darnell\(2003\)Molecular cell biology\.5 edition,W\. H\. Freeman,New York\.External Links:ISBN 978\-0\-7167\-4366\-8Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.2.2.1.1)\.
- \[22\]S\. Ma, B\. Zhang, L\. M\. LaFave, A\. S\. Earl, Z\. Chiang, Y\. Hu, J\. Ding, A\. Brack, V\. K\. Kartha, T\. Tay, T\. Law, C\. Lareau, Y\. Hsu, A\. Regev, and J\. D\. Buenrostro\(2020\)Chromatin potential identified by shared single\-cell profiling of rna and chromatin\.Cell183\(4\),pp\. 1103–1116\.e20\.External Links:[Document](https://dx.doi.org/10.1016/j.cell.2020.09.056)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.5.3.1.1)\.
- \[23\]E\. Z\. Macosko, A\. Basu, R\. Satija, J\. Nemesh, K\. Shekhar, M\. Goldman, I\. Tirosh, A\. R\. Bialas, N\. Kamitaki, E\. M\. Martersteck, J\. J\. Trombetta, D\. A\. Weitz, J\. R\. Sanes, A\. K\. Shalek, A\. Regev, and S\. A\. McCarroll\(2015\)Highly parallel genome\-wide expression profiling of individual cells using nanoliter droplets\.Cell161\(5\),pp\. 1202–1214\.External Links:[Document](https://dx.doi.org/10.1016/j.cell.2015.05.002)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.4.3.1.1)\.
- \[24\]L\. Minnoye, G\. K\. Marinov, T\. Krausgruber, L\. Pan, A\. P\. Marand, S\. Secchia, W\. J\. Greenleaf, E\. E\. M\. Furlong, K\. Zhao, R\. J\. Schmitz, C\. Bock, and S\. Aerts\(2021\)Chromatin accessibility profiling methods\.Nature Reviews Methods Primers1,pp\. 10\.External Links:[Document](https://dx.doi.org/10.1038/s43586-020-00008-9),[Link](https://doi.org/10.1038/s43586-020-00008-9)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.10.2.1.1)\.
- \[25\]R\. M\. Mulqueen, D\. Pokholok, B\. L\. O’Connell, C\. A\. Thornton, F\. Zhang, B\. J\. O’Roak, J\. Link, G\. G\. Yardimci, R\. C\. Sears, F\. J\. Steemers, and A\. C\. Adey\(2021\)High\-content single\-cell combinatorial indexing\.Nature Biotechnology39,pp\. 1574–1580\.External Links:[Document](https://dx.doi.org/10.1038/s41587-021-00962-z)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.2.3.1.1)\.
- \[26\]F\. Nasri, S\. Gurev, P\. Varilly, K\. Ramesh, N\. A\. O’Leary, J\. Cool, B\. Y\. Renard, P\. C\. Sabeti, and L\. Luebbert\(2026\)Deterministic access to global viral sequence data enables robust agentic scientific discovery\.External Links:2606\.06749Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px3.p1.1)\.
- \[27\]S\. M\. Prakadan, A\. K\. Shalek, and D\. A\. Weitz\(2017\)Scaling by shrinking: empowering single\-cell ‘omics’ with microfluidic devices\.Nature Reviews Genetics18,pp\. 345–361\.External Links:[Document](https://dx.doi.org/10.1038/nrg.2017.15),[Link](https://doi.org/10.1038/nrg.2017.15)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.6.2.1.1)\.
- \[28\]Y\. Qu, Y\. Lu, X\. Tu, S\. Zhang, T\. She, A\. G\. Shaw,et al\.\(2026\)BiomniBench: process\-level evaluation of llm agents for real\-world biomedical research\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.64898/2026.05.12.724604)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px2.p2.1)\.
- \[29\]D\. Ramsköld, S\. Luo, Y\. Wang, R\. Li, Q\. Deng, O\. R\. Faridani, G\. A\. Daniels, I\. Khrebtukova, J\. F\. Loring, L\. C\. Laurent, G\. P\. Schroth, and R\. Sandberg\(2012\)Full\-length mrna\-seq from single\-cell levels of rna and individual circulating tumor cells\.Nature Biotechnology30,pp\. 777–782\.External Links:[Document](https://dx.doi.org/10.1038/nbt.2282)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.4.3.1.1)\.
- \[30\]A\. B\. Rosenberg, C\. M\. Roco, R\. A\. Muscat, A\. Kuchina, P\. Sample, Z\. Yao, L\. T\. Graybuck, D\. J\. Peeler, S\. Mukherjee, W\. Chen, S\. H\. Pun, D\. L\. Sellers, B\. Tasic, and G\. Seelig\(2018\)SPLiT\-seq reveals cell types and lineages in the developing brain and spinal cord\.Science360\(6385\),pp\. 176–182\.External Links:[Document](https://dx.doi.org/10.1126/science.aam8999)Cited by:[Table S5](https://arxiv.org/html/2610.37851#A9.T5.3.5.3.1.1)\.
- \[31\]O\. Schwartzman and A\. Tanay\(2015\)Single\-cell epigenomics: techniques and emerging applications\.Nature Reviews Genetics16,pp\. 716–726\.External Links:[Document](https://dx.doi.org/10.1038/nrg3980),[Link](https://doi.org/10.1038/nrg3980)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.7.2.1.1)\.
- \[32\]R\. Stark, M\. Grzelak, and J\. Hadfield\(2019\)RNA sequencing: the teenage years\.Nature Reviews Genetics20,pp\. 631–656\.External Links:[Document](https://dx.doi.org/10.1038/s41576-019-0150-2),[Link](https://doi.org/10.1038/s41576-019-0150-2)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.8.2.1.1)\.
- \[33\]D\. T\. P\. Tang, C\. Plessy, M\. Salimullah, A\. M\. Suzuki, R\. Calligaris, S\. Gustincich, and P\. Carninci\(2013\)Suppression of artifacts and barcode bias in high\-throughput transcriptome analyses utilizing template switching\.Nucleic Acids Research41\(3\),pp\. e44\.External Links:[Document](https://dx.doi.org/10.1093/nar/gks1128),[Link](https://doi.org/10.1093/nar/gks1128)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.11.2.1.1)\.
- \[34\]K\. Vandereyken, A\. Sifrim, B\. Thienpont, and T\. Voet\(2023\)Methods and applications for single\-cell and spatial multi\-omics\.Nature Reviews Genetics24\(8\),pp\. 494–515\.External Links:[Document](https://dx.doi.org/10.1038/s41576-023-00580-2),[Link](https://doi.org/10.1038/s41576-023-00580-2)Cited by:[Table S4](https://arxiv.org/html/2610.37851#A9.T4.3.9.2.1.1)\.
- \[35\]Y\. Wan, T\. Fang, Z\. Li, Y\. Huo, W\. Wang, H\. Mi, D\. Yu, and M\. R\. Lyu\(2026\)Inference\-time scaling of verification: self\-evolving deep research agents via test\-time rubric\-guided verification\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 24822–24835\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1243)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px3.p1.1)\.
- \[36\]B\. Wang, C\. Xu, X\. Zhao, L\. Ouyang, F\. Wu, Z\. Zhao, R\. Xu, K\. Liu, Y\. Qu, F\. Shang, B\. Zhang, L\. Wei, Z\. Sui, W\. Li, B\. Shi, Y\. Qiao, D\. Lin, and C\. He\(2024\)MinerU: an open\-source solution for precise document content extraction\.arXiv preprint arXiv:2409\.18839\.Cited by:[§3\.1](https://arxiv.org/html/2610.37851#S3.SS1.SSS0.Px2.p1.1)\.
- \[37\]J\. Wang, Q\. Yan, Y\. Wang, Y\. Tian, S\. S\. Mishra, Z\. Xu, M\. Gandhi, P\. Xu, and L\. L\. Cheong\(2026\)Reinforcement learning for self\-improving agent with skill library\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,pp\. 1529–1550\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.69)Cited by:[Appendix A](https://arxiv.org/html/2610.37851#A1.SS0.SSS0.Px3.p1.1)\.
- \[38\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. R\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-1601)Cited by:[§3\.2](https://arxiv.org/html/2610.37851#S3.SS2.SSS0.Px1.p2.1)\.
- \[39\]M\. Zechner\(2026\)Pi: a minimal coding\-agent harness and agent runtime\.Note:[https://github\.com/badlogic/pi\-mono](https://github.com/badlogic/pi-mono)SoftwareCited by:[§3\.2](https://arxiv.org/html/2610.37851#S3.SS2.SSS0.Px1.p2.1)\.

## Appendix ARelated work

#### Sequencing\-library specifications\.

Explicit descriptions of sequencing\-library structure are important for reproducible upstream preprocessing and interpretation of single\-cell sequencing data\. Community resources such asscg\_lib\_structs\([https://teichlab\.github\.io/scg\_lib\_structs/](https://teichlab.github.io/scg_lib_structs/)\) manually curate primers, adapters, barcodes, final\-library structures, and library\-generation steps across single\-cell protocols\.seqspec\[[6](https://arxiv.org/html/2610.37851#bib.bib1)\]formalizes this idea with a machine\-readable specification for genomics assays, representing ordered library regions, technical sequences, and their relationships to sequencing reads\. This enables more standardized preprocessing, assay comparison, and reanalysis across various sequencing technologies\. LibStructBench addresses a complementary problem: rather than assuming that a structured assay description is already available, it evaluates whether an AI agent can autonomously reconstruct such information accurately from heterogeneous sequencing\-protocol documents\.

#### Biomedical agents and benchmarks\.

Recent agentic systems increasingly combine language\-model reasoning with scientific tools, retrieval, and structured execution environments\. Co\-Scientist\[[14](https://arxiv.org/html/2610.37851#bib.bib3)\]uses a multi\-agent architecture to generate, critique, and refine scientific hypotheses, and Robin\[[13](https://arxiv.org/html/2610.37851#bib.bib4)\]combines literature\-search and data\-analysis agents in an iterative lab\-in\-the\-loop discovery workflow\.BIOMNI\[[18](https://arxiv.org/html/2610.37851#bib.bib2)\]generalizes this idea to a broad biomedical environment spanning database retrieval, code execution, and other domain tools\.

On the evaluation side, LAB\-Bench\[[19](https://arxiv.org/html/2610.37851#bib.bib5)\], ScienceAgentBench\[[8](https://arxiv.org/html/2610.37851#bib.bib6)\], BiomniBench\[[28](https://arxiv.org/html/2610.37851#bib.bib7)\], and DrugDiscoveryBench\[[2](https://arxiv.org/html/2610.37851#bib.bib8)\]evaluate increasingly realistic biological and scientific tasks, from literature and sequence reasoning to executable multi\-step workflows\. LibStructBench differs by focusing on a narrow but exact reconstruction problem: recovering sequencing\-library components and the molecular workflow from primary protocol files\.

#### Deterministic verification and iterative improvement\.

Recent agent systems increasingly combine generative reasoning with deterministic feedback and persistent adaptation\. VirBench\[[26](https://arxiv.org/html/2610.37851#bib.bib9)\]shows that coupling scientific agents to a deterministic programmatic retrieval layer substantially improves the reliability of viral\-sequence retrieval\. SAGE\[[37](https://arxiv.org/html/2610.37851#bib.bib10)\]develops a self\-improving agent with a persistent skill library, in which skills learned from earlier tasks accumulate\. DeepVerifier\[[35](https://arxiv.org/html/2610.37851#bib.bib11)\]instead uses rubric\-based verification to generate feedback for iterative refinement at inference time\. Together, these studies motivate closed\-loop agent systems where deterministic tools and prior experience are converted into reusable capabilities\. Our work follows this direction by testing whether self\-improvement loops that learn from discrepancies on reviewed training protocols can improve molecular\-workflow reconstruction of unseen sequencing protocols\.

## Appendix BHuman\-agent ground\-truth audit and adjudication

![Refer to caption](https://arxiv.org/html/2610.37851v1/supp1a_audit_loop.png)\(a\)
\(b\)
Figure S1:Human\-agent ground\-truth audit and adjudication\.\(a\)Existing human curation is converted to the canonical representation and verified against primary evidence\. Claude Code proposes corrections, a human curator adjudicates substantive findings, and only approved decisions are applied by deterministic code\.\(b\)Distribution of confirmed audit findings of errors from 73 usable protocols collected inscg\_lib\_structs\.#### Representative errors\.

Human curation error \(dsciATAC\-seq\)\.The legacy record labeled the 6\-nt sequence between the s5 adapter and mosaic end as a UMI\. The Methods and oligo table identify it as one of 96 Tn5 well barcodes, so the placeholder was corrected from\[UMI:6\]to\[TN5\_INDEX:6\]in Tasks 1 and 3\.

Protocol ambiguity \(scDamID\)\.The supplement prints a 57\-nt P5 primer beginningATG, whereas the final\-library structure in the same protocol beginsAATGand requires the 58\-nt primer\. Direct PDF re\-extraction confirmed that the source itself omits the leading A; we retained the 58\-nt form and recorded the conflict\.

## Appendix CPDF\-parser\-control detail

Figure S2:PDF\-parser\-control sequence recovery\.Sequence recovery was evaluated across 80 verified nucleotide strings from 5 protocols\. For each reference sequence, we report the highest normalized Levenshtein similarity to any sequence extracted\. The table summarizes exact recovery and mean similarity\.
## Appendix DVariability across repeated agent executions

We assessed execution variability for GPT\-5\.6 Sol Max with Codex by comparingn=1n=1versusn=3n=3executions per protocol across the 20\-protocol baseline panel \(60 executions total\)\. Then=3n=3analysis included the original execution\. All outputs were evaluated using the same scoring rules and approved ground\-truth references as the main experiments\. For protocolpp, metricmm, and executionrr, letFp​r​mF\_\{prm\}denote the corresponding F1 score\. Within\-protocol variability was measured asRp​m=maxr⁡Fp​r​m−minr⁡Fp​r​mR\_\{pm\}=\\max\_\{r\}F\_\{prm\}\-\\min\_\{r\}F\_\{prm\}across the three executions\. We summarized these ranges across protocols using the median and interquartile range \(linearly interpolated empirical quartiles\), and counted protocols withRp​m\>0\.10R\_\{pm\}\>0\.10\. Execution\-specific protocol\-macro F1 was calculated asF¯r​m=1Pm​∑p=1PmFp​r​m\\bar\{F\}\_\{rm\}=\\frac\{1\}\{P\_\{m\}\}\\sum\_\{p=1\}^\{P\_\{m\}\}F\_\{prm\}, and the three\-execution mean asF¯⋅m=13​∑r=13F¯r​m\\bar\{F\}\_\{\\cdot m\}=\\frac\{1\}\{3\}\\sum\_\{r=1\}^\{3\}\\bar\{F\}\_\{rm\}, wherePmP\_\{m\}denotes the number of informative protocols\. Task 2 excludes ddSEQ becauseNreq=0N\_\{\\mathrm\{req\}\}=0\(Pm=19P\_\{m\}=19\), whereas Task 3 includes all 20 protocols \(Pm=20P\_\{m\}=20\)\.

\(a\)

\(b\)

Figure S3:GPT\-Codex variability across three executions per protocol\.\(a\)Within\-protocol F1 ranges\. Points represent protocols; diamonds and intervals indicate medians and interquartile ranges\. The dashed line marks a range of 0\.10, with annotations indicating counts above this threshold\.\(b\)Protocol\-macro F1 for each execution and their three\-execution mean\.Table S1:Observed variability across three GPT\-Codex executions per protocol\.Macro\-F1 averages 19 informative protocols for oligo recovery and 20 for each Task 3 metric\.The three\-execution macro\-F1 estimates differed from the original scores by only 0\.005\-0\.024 across metrics \(Table[S1](https://arxiv.org/html/2610.37851#A4.T1)\)\. However, within\-protocol F1 ranges exceeded 0\.10 for 5/19 oligo\-family, 9/20 transition, 8/20 state, and 12/20 typed\-edge comparisons \(Figure[S3](https://arxiv.org/html/2610.37851#A4.F3)\)\. The largest observed range was 0\.471 for typed\-edge F1 in s3\-ATAC\. Thus, aggregate performance remained broadly consistent despite variability in individual protocols\.

This analysis was limited to GPT\-Codex; Figure[3\(a\)](https://arxiv.org/html/2610.37851#S3.F3.sf1)and Figure[3\(b\)](https://arxiv.org/html/2610.37851#S3.F3.sf2)retain one selected execution per system and protocol\. Protocol\-bootstrap intervals in the main results reflect across\-protocol uncertainty conditional on these outputs, rather than execution variability\. The results therefore do not quantify variability for other systems or for the knowledge and learning interventions, including matchedC0C\_\{0\}\-C30C\_\{30\}comparisons\.

## Appendix EReconstruction without protocol source bundles

Publicly available sequencing protocols and resources such asscg\_lib\_structsmay have appeared in LLM training data, we evaluated whether models could reconstruct sequencing libraries using only protocol names and versions, without access to source documents\. Generic instructions, output schemas, and local tools remained available, but source documents and retrieval access were withheld\. External network access was restricted except for model\-provider connections, and agent worked within an isolated docker image\. This test assesses performance without supplied evidence, rather than distinguishing memorization from general learned knowledge or reasoning\.

#### Task 1 API control\.

We compared the PDF\-input API predictions from Figure[2](https://arxiv.org/html/2610.37851#S3.F2)with one independent name\-only API execution per model and protocol\. Interestingly, name prompting generally yielded increased sequence similarity for all models, although only Claude remained significant after Bonferroni correction across three comparisons \(padj=0\.0215p\_\{\\mathrm\{adj\}\}=0\.0215\)\. Exact reconstructions decreased from 4 to 2 for GPT, 5 to 3 for Claude, and 4 to 3 for Gemini, out of 20 protocols per model\.

#### Native\-agent comparison \(Tasks 2\-3\)\.

For Codex, Claude Code, and Antigravity, we compared one Name execution with the original Bundle execution used in Figure[3\(a\)](https://arxiv.org/html/2610.37851#S3.F3.sf1)\(n=1n=1per condition\)\. All outputs were scored against the same approved references\. All 20 protocols had valid pairs, with ddSEQ excluded from Task 2 \(n=19n=19\)\. For each system and metric, we calculated the mean paired differenceΔ=1n​∑p\(Fp,Name−Fp,Bundle\)\\Delta=\\frac\{1\}\{n\}\\sum\_\{p\}\(F\_\{p,\\mathrm\{Name\}\}\-F\_\{p,\\mathrm\{Bundle\}\}\)\. We used two\-sided paired sign\-flip tests with Bonferroni correction across 12 comparisons\. In contrast to the Task 1 similarity results, Name prompting yielded lower mean F1 across all systems and metrics, although none of the differences remained significant after correction \(allpadj\>0\.05p\_\{\\mathrm\{adj\}\}\>0\.05\)\.

Task 1Task 2Task 3SimilarityOligo F1Transition F1State F1Edge F1ProtocolModelPDFNameBundleNameBundleNameBundleNameBundleName10x ATAC v2GPT1\.0000\.9470\.7910\.6730\.7820\.8480\.8750\.8630\.7141\.000Claude1\.0001\.0000\.7130\.6870\.7770\.7610\.7760\.8120\.7140\.714Gemini1\.0001\.0000\.7010\.6730\.9560\.7070\.7200\.3961\.0000\.66710x 3’ GEX v4GPT1\.0001\.0000\.7240\.8670\.8810\.9500\.8680\.8740\.8461\.000Claude1\.0000\.9700\.6580\.8490\.8810\.9990\.8470\.8750\.8461\.000Gemini1\.0001\.0000\.8110\.8450\.8580\.8690\.5780\.4620\.6670\.81810x 3’ Feature BarcodingGPT1\.0000\.8530\.8650\.7030\.6950\.7510\.7200\.5500\.6500\.600Claude1\.0000\.8810\.8520\.7040\.7700\.7520\.7170\.6660\.6500\.650Gemini0\.9950\.9030\.8170\.7120\.7590\.6500\.5130\.4690\.6840\.684Drop\-seqGPT0\.8970\.9890\.6750\.7330\.7660\.9180\.5340\.7390\.7500\.960Claude0\.8860\.9890\.5280\.7460\.8530\.9420\.7700\.8000\.8891\.000Gemini0\.8860\.9890\.5110\.7350\.8200\.6170\.5580\.3290\.7500\.364Seq\-Well S3GPT0\.6811\.0000\.8340\.7330\.8290\.6760\.8010\.6430\.7780\.643Claude0\.6811\.0000\.7670\.7230\.8500\.9090\.8050\.7500\.8240\.848Gemini1\.0001\.0000\.8550\.6980\.5640\.8740\.3210\.5110\.1330\.848inDrop v1GPT1\.0000\.5860\.6720\.4480\.8950\.7660\.7060\.5050\.8460\.720Claude1\.0000\.8710\.6750\.4770\.6410\.6560\.4910\.5120\.5950\.500Gemini1\.0000\.8600\.6550\.4900\.7210\.7780\.2310\.4790\.4620\.692s3\-ATACGPT0\.1040\.7680\.9270\.6320\.9380\.4890\.9090\.5290\.9410\.400Claude0\.0980\.7930\.8970\.6180\.9880\.4860\.8400\.5530\.9410\.400Gemini0\.0000\.8170\.8840\.6110\.9860\.4880\.4740\.2390\.9410\.400CEL\-seqGPT0\.3910\.9380\.6000\.6580\.8780\.8960\.5440\.6730\.8150\.846Claude0\.3910\.9930\.6000\.5180\.9000\.8760\.7180\.7510\.8150\.846Gemini0\.3840\.9680\.3530\.6830\.8350\.8320\.3150\.3330\.6920\.692SPLiT\-seqGPT0\.0260\.6270\.8970\.5890\.9000\.6920\.6360\.4620\.8670\.643Claude0\.1320\.7560\.9010\.7310\.7710\.7520\.8160\.7910\.7140\.857Gemini0\.1700\.6670\.8780\.5460\.8000\.7850\.4060\.4170\.7140\.846sci\-RNA\-seqGPT0\.9000\.8740\.7710\.7540\.6640\.8240\.8250\.8290\.6360\.778Claude1\.0001\.0000\.7970\.7300\.6980\.8250\.8180\.7270\.6360\.778Gemini0\.9000\.5420\.7430\.7600\.8110\.8240\.4660\.4610\.7780\.778sci\-ATAC\-seqGPT0\.0490\.6630\.6610\.6750\.5640\.6770\.3730\.3290\.2350\.471Claude0\.0980\.7410\.8040\.6470\.6940\.4920\.4630\.4020\.3160\.211Gemini0\.0000\.7410\.7290\.6620\.6940\.5900\.1790\.1880\.4710\.235Microwell\-seqGPT0\.5780\.8260\.5270\.7390\.9000\.9440\.6480\.7140\.9091\.000Claude0\.5780\.8530\.5160\.4750\.8080\.5440\.7320\.2400\.7830\.182Gemini0\.5090\.8120\.5450\.6610\.9400\.6660\.7340\.3621\.0000\.600scRRBSGPT0\.9530\.9530\.8380\.7100\.9490\.9260\.7410\.6450\.9090\.909Claude0\.9530\.9530\.7900\.4180\.8130\.9250\.8450\.7540\.7500\.909Gemini0\.9450\.9530\.8160\.5280\.8090\.9150\.7180\.3710\.8330\.909PIP\-seq v4GPT0\.8290\.8420\.5750\.7130\.7650\.8880\.3160\.6370\.7500\.960Claude0\.8290\.8380\.6970\.7310\.7710\.7470\.5890\.6580\.6860\.720Gemini0\.8290\.8200\.7670\.6350\.7750\.6820\.3630\.3820\.7830\.720scDamIDGPT0\.9940\.5950\.6430\.4960\.7830\.5470\.6440\.5460\.6920\.480Claude0\.9940\.6370\.6290\.7280\.8810\.7890\.6720\.6270\.8460\.750Gemini0\.9940\.6190\.6610\.6070\.9920\.4900\.4520\.2551\.0000\.364ddSEQGPT0\.0630\.401––0\.9360\.7010\.8980\.6690\.9380\.519Claude0\.1810\.382––0\.9710\.7000\.9140\.6491\.0000\.583Gemini0\.0630\.384––0\.6640\.7000\.6480\.4900\.5190\.583SMART\-seqGPT0\.0000\.3420\.8590\.8110\.7910\.6040\.5730\.4990\.7220\.643Claude0\.2400\.5000\.6620\.8330\.8790\.8000\.6300\.6310\.8330\.800Gemini0\.0000\.5000\.7570\.8660\.8030\.5570\.5010\.2760\.7650\.462SHARE\-seqGPT0\.0830\.4570\.9480\.5420\.7600\.6740\.5960\.3990\.5620\.536Claude0\.0830\.7700\.9550\.6530\.8210\.7890\.7540\.5980\.8120\.714Gemini0\.1670\.6090\.9500\.5340\.7840\.3490\.5910\.2430\.5830\.231DR\-seqGPT0\.1090\.6300\.8500\.4250\.5650\.3230\.4350\.4650\.5110\.300Claude0\.1090\.7710\.4970\.3380\.4820\.4220\.4720\.4840\.2260\.213Gemini0\.1090\.8820\.6000\.4440\.7220\.4830\.3290\.3360\.6430\.426PETRI\-seqGPT0\.5680\.6680\.7820\.4240\.9390\.7650\.7930\.6381\.0000\.818Claude0\.5630\.5690\.8140\.5250\.8690\.6870\.7960\.5890\.8460\.667Gemini0\.9850\.5960\.8970\.0920\.9830\.8440\.8110\.5231\.0001\.000Macro meanGPT0\.5610\.7480\.7600\.6490\.8090\.7430\.6720\.6100\.7540\.711Claude0\.5910\.8130\.7240\.6380\.8060\.7430\.7230\.6440\.7360\.667Gemini0\.5970\.7830\.7330\.6200\.8140\.6850\.4950\.3760\.7210\.616Δ\\Delta\(Name−\-source\)GPT\+0\.187\+0\.187−0\.111\-0\.111−0\.066\-0\.066−0\.061\-0\.061−0\.042\-0\.042Claude\+0\.223\+0\.223−0\.085\-0\.085−0\.063\-0\.063−0\.080\-0\.080−0\.069\-0\.069Gemini\+0\.186\+0\.186−0\.113\-0\.113−0\.129\-0\.129−0\.119\-0\.119−0\.105\-0\.105padjp\_\{\\mathrm\{adj\}\}GPT0\.05660\.25351\.00001\.00001\.0000Claude0\.02150\.54490\.98890\.13311\.0000Gemini0\.13290\.72610\.14200\.11231\.0000Task 1 exactGPT4/202/20Claude5/203/20Gemini4/203/20Table S2:Protocol\-name\-only reconstruction across Tasks 1–3\.Task 1 reports API sequence similarity \(PDF vs\. Name\); Tasks 2–3 report native\-agent F1 \(Bundle vs\. Name\), using the Figure[3\(a\)](https://arxiv.org/html/2610.37851#S3.F3.sf1)bundle executions\. Bold indicates the higher score within each pair; ties are unbolded\.Δ\\Deltadenotes Name minus source\-input score\.padjp\_\{\\mathrm\{adj\}\}denotes Bonferroni\-adjustedpp\-values, correcting separately across three Task 1 and 12 Tasks 2–3 comparisons\.

## Appendix FComplementary model\-harness performance metrics

\(a\)\(b\)\(c\)
Figure S4:Complementary performance metrics across models and harnesses\.Heatmaps show performance across the ten protocols and nine model–harness systems used in Figure[3\(b\)](https://arxiv.org/html/2610.37851#S3.F3.sf2)\. System and protocol orders are fixed to the molecular\-transition\-F1 clustering in the main figure to permit direct comparison across metrics\. All color scales range from 0 to 1\.
## Appendix GCost analysis and model\-harness effects

\(a\)\(b\)
Figure S5:Estimated API\-equivalent cost and reconstruction performance\.\(a\)Per\-protocol costs and system means across ten protocols\.\(b\)Cost versus mean performance across four F1 metrics\.#### Model\-harness effect analysis\.

To test the differences of model and harness selection on reconstruction performance, we compared GPT, Claude, and Gemini in a fully crossed design with Pi and mini\-SWE\-agent, using one execution per configuration on 10 common protocols \(excluding ddSEQ for Task 2\) \. For protocolpp, modelmm, and harnesshh, letyp​m​hy\_\{pmh\}denote F1\. Model\- and harness\-switch magnitudes wereMp=16​∑h∑m<m′\|yp​m​h−yp​m′​h\|M\_\{p\}=\\frac\{1\}\{6\}\\sum\_\{h\}\\sum\_\{m<m^\{\\prime\}\}\|y\_\{pmh\}\-y\_\{pm^\{\\prime\}h\}\|,Hp=13​∑m\|yp​m,mini−yp​m,Pi\|H\_\{p\}=\\frac\{1\}\{3\}\\sum\_\{m\}\|y\_\{pm,\\mathrm\{mini\}\}\-y\_\{pm,\\mathrm\{Pi\}\}\|\. The factors1/61/6and1/31/3average over the three unique model pairs within each of two harnesses, and the single harness pair within each of three models, respectively\. We estimated the mean ofDp=Mp−HpD\_\{p\}=M\_\{p\}\-H\_\{p\}and its pointwise 95% CI using 10,000 paired protocol\-bootstrap resamples\. Mean model\-switch magnitudes exceeded harness\-switch magnitudes across all four metrics, but only molecular\-state F1 showed a significant difference \(D¯=0\.0793\\bar\{D\}=0\.0793, 95% CI\[0\.038,0\.124\]\[0\.038,0\.124\],pBonf\.=0\.0156p\_\{\\mathrm\{Bonf\.\}\}=0\.0156; Figure[S6](https://arxiv.org/html/2610.37851#A7.F6)\)\.

#### Secondary analyses and results\.

For each metric, we fitted a linear mixed model with model, harness, and their interaction as fixed effects and a protocol random intercept\. Protocol\-blocked repeated\-measures ANOVA with Greenhouse\-Geisser correction served as a robustness check\. Bonferroni correction covered 12 omnibus tests \(three effects×\\timesfour metrics\) within each analysis\. Only molecular\-state F1 showed a detectable model main effect \(mixed model:F⁡\(2,45\)=23\.105F\(2,45\)=23\.105,padj=1\.50×10−6p\_\{\\mathrm\{adj\}\}=1\.50\\times 10^\{\-6\}; blocked analysis:padj=0\.0010p\_\{\\mathrm\{adj\}\}=0\.0010\)\. State F1 averaged over the two harnesses was 0\.655 for Claude, 0\.612 for GPT, and 0\.459 for Gemini\. Neither analysis detected a harness main effect or model\-harness interaction for any metric\. These secondary tests concern average scores; nonsignificance does not establish equivalence or negligible harness effects\.

Figure S6:Model\-harness effects\.Mean absolute switch magnitudes \(left\) and paired model\-minus\-harness contrasts \(right\)\. Intervals are pointwise 95% protocol\-bootstrap CIs, conditional on the selected executions\. The state annotation givespBonfp\_\{\\mathrm\{Bonf\}\}for the primary four\-metric comparison\.

## Appendix HBaseline output failure modes

\(a\)LLM\-harness system profiles\(b\)Protocol\-centric profiles
Figure S7:Baseline output\-discrepancy profiles\.\(a\)The fraction of the ten protocols with at least one discrepancy\.\(b\)The fraction of the ten systems with at least one discrepancy\. Values range from 0 to 1, where 1 indicates this error appeared in 100% protocols or LLM\-harness runs\.
Table S3:Deterministic output\-discrepancy categories used in the baseline error analysis\.

## Appendix IExternal knowledge for agent improvement

Table S4:Sources for the general molecular\-method knowledge primer\.The primer summarizes general molecular operations instead of protocol\-specific answers\.Table S5:Protocols used in the cross\-protocol memory pilot\.Donor protocols were selected as conceptual matches to the validation protocols based on shared molecular features\.
## Appendix JTrajectory\-based behavioral changes across autonomous\-learning checkpoints

To interpret the non\-monotonic learning curve, we compared the agent\-visible context, execution trajectories, and predictions at the baseline,C30C\_\{30\},C40C\_\{40\}, andC58C\_\{58\}checkpoints on the same frozen five\-protocol validation panel\. For each checkpoint, we analyzed one selected execution per validation protocol\. The observed changes support behavioral interpretation but do not alone establish the causal effect of an individual rule or retrieved example\.

Table S6:Behavioral comparison of selected autonomous\-learning checkpoints\.Learned rules are protocol\-neutral instructions accumulated from previous training protocols; exemplar records are ground\-truth\-derived cross\-protocol examples available for retrieval\. Task 2 scores are macro\-averaged over four informative protocols; ddSEQ has no required recoverable oligo families and is reported separately\. Task 3 scores include all five validation protocols\.The Task 3 deterioration visible atC40C\_\{40\}was already apparent atC35C\_\{35\}, with pronounced losses for ddSEQ and HyDrop\-RNA\. The decline in informative Task 2 F1 also involved CEL\-seq, whose score decreased from 0\.600 atC30C\_\{30\}to 0\.450 atC40C\_\{40\}\.

For ddSEQ, a rule learned from a previous multi\-omic protocol instructed the agent to keep co\-processed but physically distinct molecules in separate workflows\. In the observed trajectory, the agent separated the cDNA and deconvolution\-oligo branches into two workflows, whereas the benchmark connects them through a shared fraction\-separation transition\. One predicted workflow was therefore left unmatched\. FromC30C\_\{30\}toC40C\_\{40\}, ddSEQ transition F1 decreased from 1\.000 to 0\.493, state F1 from 0\.830 to 0\.449, and typed\-edge F1 from 1\.000 to 0\.364\. Because ddSEQ has no required recoverable oligo families, it is excluded from the Task 2 macro\-average\. Both checkpoints produced zero effective spurious oligo families;C58C\_\{58\}produced one\.

For HyDrop\-RNA, a learned rule instructed the agent to begin the molecular workflow after oligo\-reagent fabrication\. In the observed reconstruction, the agent represented a completed bead construct instead of the separate Acrydite and split\-pool barcode oligos, omitting early bead\-construction states and transitions\. The benchmark, however, explicitly scores these oligos and construction steps\. FromC30C\_\{30\}toC40C\_\{40\}, HyDrop\-RNA Task 2 F1 decreased from 0\.915 to 0\.738, state F1 from 0\.401 to 0\.159, transition F1 from 0\.666 to 0\.587, and typed\-edge F1 from 0\.609 to 0\.526\. These cases are consistent with negative transfer or a mismatch between learned representation conventions and the canonical benchmark\. The trajectories do not isolate the causal effect of an individual rule from execution variability\.

AtC58C\_\{58\}, the agent recovered some molecular\-state detail but frequently used a finer event granularity than the canonical benchmark\. For example, it represented MspI\-end fill\-in, dA addition, bisulfite denaturation, and bisulfite conversion as separate scRRBS events\. This representation was associated with higher scRRBS state F1, increasing from 0\.521 atC40C\_\{40\}to 0\.712 atC58C\_\{58\}, while transition F1 decreased from 0\.883 to 0\.741 and typed\-edge F1 from 0\.696 to 0\.615, reflecting additional unmatched transitions and edges\. Similarly, theC58C\_\{58\}chATAC\-seq output represented adapter reagents as independent molecular states, coinciding with lower state precision and typed\-edge agreement\.

## Appendix KSensitivity analysis

Here we assessed the sensitivity of benchmark scores and system comparisons to similarity weights and matching thresholds used in Tasks 2 and 3\. In total, we evaluated 640 distinct scoring configurations: one default configuration, three uniform\-weight alternatives, 28 individual weight perturbations, 600 Dirichlet\-sampled weight vectors, and eight additional threshold combinations\. Weight and threshold variations were evaluated separately, rather than through a full factorial combination\.

For each configuration, we recomputed the complete matching and scoring pipeline using the same saved agent predictions and ground\-truth records, without additional agent executions\. This captured changes in both partial\-credit similarity scores and molecular\-state or transition alignments\. The resulting ranges characterize sensitivity to scoring assumptions rather than statistical confidence intervals\.

#### Sensitivity to similarity weights\.

We examined three sets of scientific similarity weights for oligo\-family, molecular\-transition, and molecular\-state similarity\. Table[S7](https://arxiv.org/html/2610.37851#A11.T7)summarizes their default values and scientific rationale\. The weights were chosen heuristically to prioritize molecular identity and relevant structural information, rather than estimated from experimental data\.

Table S7:Default component weights for pairwise scientific similarity\.These similarities contribute to entity matching and subsequent F1 calculation\. For oligo\-family and molecular\-state scoring, weights are renormalized over components enabled by the ground\-truth support rules\.For each default weight vector𝒘0\\boldsymbol\{w\}\_\{0\}\(Table[S7](https://arxiv.org/html/2610.37851#A11.T7)\), we evaluated three types of alternatives: i\) uniform weights \(1/d1/dfor each ofddcomponents\); ii\) one\-at\-a\-time±20%\\pm 20\\%relative perturbations of individual components, followed by renormalization; and iii\) 200 joint perturbations of all components, sampled from𝒘∼Dirichlet⁡\(λ​𝒘0\)\\boldsymbol\{w\}\\sim\\operatorname\{Dirichlet\}\(\\lambda\\boldsymbol\{w\}\_\{0\}\), withλ=100\\lambda=100\. Here,λ\\lambdacontrols the dispersion of sampled weights around the default vector𝒘0\\boldsymbol\{w\}\_\{0\}\. The concentration parameter was chosen heuristically to explore moderate departures from the default weights\. Each of the oligo, transition, and state weight vectors was varied separately, with all other weights and matching thresholds held fixed\.

Figure[S8](https://arxiv.org/html/2610.37851#A11.F8)shows that absolute F1 scores varied with component weighting\. For example, uniform molecular\-state weights increased Gemini’s state F1 from 0\.495 to 0\.581 and Kimi’s from 0\.500 to 0\.594, while Claude remained the highest\-scoring system for molecular\-state reconstruction\. The leading systems for oligo\-family and molecular\-state F1 remained unchanged across the tested weight configurations\. Molecular\-transition rankings were less stable: Gemini ranked first in 179 of 200 sampled transition\-weight configurations, while GPT ranked first in the remaining 21\. Uniform transition weights also favored GPT\. These results indicate that the principal leading\-system comparisons were largely robust to alternative similarity weights, although absolute F1 scores and the close Gemini\-GPT transition comparison were sensitive to component weighting\. The analysis supports the stability of our comparative findings without establishing that the default weights are biologically optimal\.

Figure S8:Sensitivity of native\-system performance to similarity weights\.Macro\-averaged oligo\-family, molecular\-transition, and molecular\-state F1 scores for four native coding\-agent systems\. The displayed ranges characterize sensitivity to scoring assumptions, not statistical confidence intervals\.
#### Sensitivity to matching thresholds\.

Next we assessed sensitivity to molecular\-state and transition assignment thresholds, both set to 0\.25 by default\. Each threshold was independently varied over\{0\.15,0\.25,0\.40\}\\\{0\.15,0\.25,0\.40\\\}, yielding nine combinations\. Thresholds were applied to context\-adjusted assignment scores rather than raw scientific similarity scores\. All similarity weights were kept at their default values, and matching and scoring were recomputed for each combination\.

Table[S8](https://arxiv.org/html/2610.37851#A11.T8)summarizes the maximum absolute changes in macro\-averaged F1 across the tested thresholds and systems or conditions within each experiment\. For the native\-system comparison \(Figure[3\(a\)](https://arxiv.org/html/2610.37851#S3.F3.sf1)\), the maximum changes in transition, state, and typed\-edge F1 were 0\.035, 0\.031, and 0\.069, respectively\.

Table S8:Sensitivity to state and transition assignment thresholds\.Maximum absolute change in protocol\-macro F1 relative to the default thresholds\(0\.25,0\.25\)\(0\.25,0\.25\), across nine threshold combinations and all systems or conditions within each experiment\.Figure S9:Sensitivity of molecular\-workflow scores to matching thresholds\.Heatmaps show the maximum absolute changes in molecular\-transition, molecular\-state, and typed\-edge F1 across autonomous\-learning validation checkpoints relative to their respective scores under the default thresholds\. State and transition assignment thresholds were independently varied over\{0\.15,0\.25,0\.40\}\\\{0\.15,0\.25,0\.40\\\}\. Each cell represents the maximum across checkpoints, rather than the average change\. The default combination\(0\.25,0\.25\)\(0\.25,0\.25\)is outlined\.Figure[S9](https://arxiv.org/html/2610.37851#A11.F9)illustrates threshold sensitivity in the autonomous\-learning validation experiment \(Figure[4\(c\)](https://arxiv.org/html/2610.37851#S3.F4.sf3)\), which exhibited the largest changes in transition and typed\-edge F1 \(0\.055 and 0\.132, respectively\)\. Notably, increasing only the state\-matching threshold from 0\.25 to 0\.40 produced a maximum typed\-edge F1 change of 0\.117, compared with 0\.025 for state F1\.

Overall, transition and state F1 were less sensitive to the tested matching thresholds than typed\-edge F1\. This likely reflects the dependence of graph\-connectivity scoring on molecular\-state alignment, where changes in matched states can propagate to the evaluation of associated edges\. These findings suggest that graph\-topology scores should be interpreted alongside molecular\-state alignment and the canonical graph representation\.

Figure S10:Robustness of knowledge augmentation and autonomous\-learning results\.Differences in F1 relative to baseline for combined external knowledge versus baseline; final checkpointC58C\_\{58\}versusC0C\_\{0\}on the learning\-validation panel; andC30C\_\{30\}with knowledge\-base supplementation versusC0C\_\{0\}on held\-out protocols\.
#### Robustness of knowledge augmentation and cumulative learning\.

We further investigated whether the conclusions from the external\-knowledge and autonomous\-learning experiments were preserved under alternative scoring assumptions \(Figure[S10](https://arxiv.org/html/2610.37851#A11.F10)\)\. In the external\-knowledge pilot \(Figure[4\(a\)](https://arxiv.org/html/2610.37851#S3.F4.sf1)\), combining general knowledge with cross\-protocol examples consistently increased oligo\-family F1 relative to baseline \(range: 0\.022–0\.129\) across the tested scoring configurations\. However, baseline retained the highest mean scores across all three Task 3 metrics\. In the cumulative\-learning validation experiment \(Figure[4\(c\)](https://arxiv.org/html/2610.37851#S3.F4.sf3)\), performance remained non\-monotonic across all tested configurations\. The final\-checkpoint oligo\-family F1 difference \(C58−C0C\_\{58\}\-C\_\{0\}\) ranged from−0\.021\-0\.021to0\.0150\.015, indicating sensitivity to component weighting\. In contrast, all three Task 3 endpoint differences remained negative\. On the held\-out protocols \(Figure[4\(d\)](https://arxiv.org/html/2610.37851#S3.F4.sf4)\), supplementing the selectedC30C\_\{30\}checkpoint with the knowledge base produced consistently positive oligo\-family F1 differences relative to baseline \(range: 0\.049–0\.075\)\. However, the pointwise 95% protocol\-bootstrap confidence interval under default scoring included zero \(\[\-0\.024, 0\.159\]\)\. Both evaluatedC30C\_\{30\}conditions remained below baseline in mean transition, state, and typed\-edge F1 across the tested scoring configurations\.

Together, these results indicate that the absence of improvement in canonical molecular\-workflow reconstruction was robust to the tested scoring assumptions\. However, the final\-checkpoint oligo\-family F1 comparison was sensitive to component weighting\.

#### Limitations\.

These sensitivity analyses used fixed agent predictions and therefore assess robustness to scoring assumptions, rather than execution\-to\-execution variability \(Appendix[D](https://arxiv.org/html/2610.37851#A4)\)\. Robustness across the tested configurations does not establish that the default weights are biologically optimal or that canonical graph matching accommodates all chemically valid representations\. Furthermore, the model\-versus\-harness statistical tests were not repeated across all scoring configurations\.