Sci$\Pi$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

# Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
Source: [https://arxiv.org/html/2610.21082](https://arxiv.org/html/2610.21082)
Xingyu RenChugang YiAffiliation:University of Maryland, College ParkKejia ZhangAffiliation:University of Maryland, College ParkDi WuAffiliation:University of Maryland, College ParkYingkang CaoAffiliation:University of Maryland, College ParkAllen ZhouAffiliation:Gradient NetworkJianping YeAffiliation:University of Maryland, College ParkSuying LiuAffiliation:University of Maryland, College ParkJiaxuan GuoAffiliation:Stanford UniversityJianda DuAffiliation:University of Maryland, College ParkWeicheng YeAffiliation:The Chinese University of Hong KongKecheng LiuAffiliation:University of Maryland, College ParkHefan ZhangAffiliation:Dartmouth CollegeHaixin WangAffiliation:University of California, Los AngelesJie XiaoAffiliation:Gradient NetworkEric YangHaizhao Yang

###### Abstract

Large language model \(LLM\) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment\. Evaluating scientific judgment is difficult because research decisions are open\-ended and depend on project context\. We construct SciΠ\\Pivot, a 248\-case benchmark of scientific judgment based on human corrections\. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents’ research judgments\. In each case, an agent chooses between the original LLM plan and the human\-corrected alternative, using only the project record available before the correction as context\. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration\. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human\-corrected alternative\. Four of five frontier models perform below chance on the selected cases\. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63\.5%, compared with 73\.4% for fresh researchers\. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy\. Even with a human\-corrected alternative available, frontier models struggle with the selected research decisions, while SciΠ\\Pivot supports systematic evaluation of scientific judgment for LLM agents\.

\*\*footnotetext:Equal contribution\.$\{\\dagger\}$$\{\\dagger\}$footnotetext:Corresponding author:ey@gradientdata\.ai$\{\\ddagger\}$$\{\\ddagger\}$footnotetext:Corresponding author:hzyang@umd\.edu###### Contents

1. [1Introduction](https://arxiv.org/html/2610.21082#S1)
2. [2Scientific Judgment as a Bottleneck](https://arxiv.org/html/2610.21082#S2)1. [2\.1What We Require of AutoResearch](https://arxiv.org/html/2610.21082#S2.SS1) 2. [2\.2What Scientific Quality Depends On](https://arxiv.org/html/2610.21082#S2.SS2) 3. [2\.3Two Examples of Human Scientific Judgment](https://arxiv.org/html/2610.21082#S2.SS3)
3. [3From Human Corrections to Evaluation](https://arxiv.org/html/2610.21082#S3)1. [3\.1What Should the Evaluation Preserve?](https://arxiv.org/html/2610.21082#S3.SS1) 2. [3\.2Recovering a Decision from Its History](https://arxiv.org/html/2610.21082#S3.SS2) 3. [3\.3Recurring Failure Modes](https://arxiv.org/html/2610.21082#S3.SS3) 4. [3\.4Relation to Existing Benchmarks](https://arxiv.org/html/2610.21082#S3.SS4)
4. [4Can Models Recognize Human Corrections?](https://arxiv.org/html/2610.21082#S4)1. [4\.1Evaluation Setup](https://arxiv.org/html/2610.21082#S4.SS1) 2. [4\.2Models Often Reject the Human\-Corrected Plan](https://arxiv.org/html/2610.21082#S4.SS2) 3. [4\.3Different Judgments Reveal Different Gaps](https://arxiv.org/html/2610.21082#S4.SS3) 4. [4\.4Lower Accuracy with the Project Record](https://arxiv.org/html/2610.21082#S4.SS4) 5. [4\.5More Thinking Can Lead Back to the Same Mistake](https://arxiv.org/html/2610.21082#S4.SS5)
5. [5From Diagnosis to Better AutoResearch Agents](https://arxiv.org/html/2610.21082#S5)
6. [References](https://arxiv.org/html/2610.21082#bib)
7. [AModel Results and Statistical Details](https://arxiv.org/html/2610.21082#A1)1. [A\.1Project\-Reading Clients](https://arxiv.org/html/2610.21082#A1.SS1) 2. [A\.2Scoring and Uncertainty](https://arxiv.org/html/2610.21082#A1.SS2) 3. [A\.3Project Coverage and Effort Settings](https://arxiv.org/html/2610.21082#A1.SS3) 4. [A\.4Option Order and Reconsideration](https://arxiv.org/html/2610.21082#A1.SS4) 5. [A\.5Writer Sensitivity](https://arxiv.org/html/2610.21082#A1.SS5)
8. [BTwo Illustrative Scientific Decisions](https://arxiv.org/html/2610.21082#A2)1. [B\.1Checking Evaluation Before Changing Model Size](https://arxiv.org/html/2610.21082#A2.SS1) 2. [B\.2Testing a Restrictive Physical Model](https://arxiv.org/html/2610.21082#A2.SS2)
9. [CJudgment Categories and Illustrative Cases](https://arxiv.org/html/2610.21082#A3)1. [C\.1Coding the Historical Decision](https://arxiv.org/html/2610.21082#A3.SS1) 2. [C\.2Boundaries and Review Procedure](https://arxiv.org/html/2610.21082#A3.SS2) 3. [C\.3Profiles by Judgment Category](https://arxiv.org/html/2610.21082#A3.SS3) 4. [C\.4Additional Illustrations](https://arxiv.org/html/2610.21082#A3.SS4)
10. [DHuman Baseline Protocol](https://arxiv.org/html/2610.21082#A4)1. [D\.1Coverage and Comparison](https://arxiv.org/html/2610.21082#A4.SS1)
11. [EEvaluation Prompts and Available Materials](https://arxiv.org/html/2610.21082#A5)1. [E\.1Twelve Evaluation Conditions](https://arxiv.org/html/2610.21082#A5.SS1) 2. [E\.2Prompts and Available Materials](https://arxiv.org/html/2610.21082#A5.SS2)

## 1Introduction

Scientific research tackles problems with unknown answers\. Formulating useful questions, designing experiments, and revising assumptions as evidence arrives make scientific research a demanding test of general intelligence\. An AI system with this ability could study and improve its own models and methods\. Such progress could enable recursive self\-improvement, as better systems conduct better research that helps create more capable systems\([Wijk et al\., 2025](https://arxiv.org/html/2610.21082#bib.bib30)\)\.

We began with a practical question\. Can frontier models conduct high\-quality scientific research autonomously? Using frontier models with frontier agentic systems, we ran*scientific AutoResearch agents*on original, PhD\-level projects at scale across disciplines\. Over roughly half a year, individual projects ran for weeks to months with publication as the goal\. They involved theory, simulation, computation, and analysis of existing scientific data using CPUs and GPUs\. Keeping experiments running proved easier than choosing experiments that advanced the scientific question\. Automated checks and review still depended on models to notice mistaken assumptions and decide how to test them\. Failed judgments could stall projects or produce results that left the underlying problem unresolved\. Senior researchers had to diagnose the problem and steer the next experiment\.

These interventions required anticipating what an experiment could establish\. Should an AutoResearch agent scale up, or first check its measurements? Does a failed attempt undermine the idea, or only its implementation? Such decisions repeatedly required human intervention \(Figure[2](https://arxiv.org/html/2610.21082#S2.F2)\)\. We call this scientific planning and decision\-making*Principal Investigator \(PI\) Judgment*\.

Figure 1:How a case is built and used\. The top row shows an original LLM plan \(A\), a senior researcher’s intervention, and the project’s subsequent progress\. We use the completed history to reconstruct the human\-corrected plan \(B\) and the evidence supporting it\. Evaluators receive the pre\-intervention project record and both plans, then choose which to pursue\. Choosing B counts as correct\.This dependence raises our next question\. How can we improve scientific AutoResearch agents so that fewer decisions require human intervention? Agents can readily propose plausible plans, but executing an uninformative one can waste weeks of research\. The bottleneck is judging which plan merits those resources\. Human corrections expose these decisions and offer a way to teach them\.

We introduce SciΠ\\Pivot to make these judgments measurable\. Its 248 cases from 26 projects pair original LLM plans with human\-corrected alternatives reconstructed from completed project histories\. The model chooses between them using the project record available before the human intervention, including its code, experimental logs, and progress notes\.

At medium reasoning effort with the project record, four of five models choose the original LLM plan more often than the human\-corrected plan on this selected set\. Further deliberation can undo a correct choice and return to the same error, a failure we call*re\-trapping*\.

SciΠ\\Pivot also connects to core NLP problems\. The case pairs provide preference supervision for scientific corrections rather than stylistic rewrites\. Scoring a choice evaluates the judgment of language agents, and the protocol exercises how agents use language to read a project record, compare two plans, and justify a choice\. These connections make SciΠ\\Pivot a testbed for NLP training and evaluation methods\.

## 2Scientific Judgment as a Bottleneck

Figure 2:Two human corrections from frontier research projects\. In \(A\), a small language model scores zero on three LiveCodeBench groups\([Jain et al\., 2024](https://arxiv.org/html/2610.21082#bib.bib11)\), and the original LLM plan is to retest with a larger model\. The senior researcher retains the small model, and a subsequent audit finds that the scorer had skipped answer extraction\. In \(B\), a cloth fit reaches the search limits while four parameters stay fixed at zero, a restriction the agent had chosen in every earlier run\. The senior researcher asks to fit all six parameters\. Models receive the project record before the intervention and the two plans, without correctness labels or the explanations shown here\. Appendix[B](https://arxiv.org/html/2610.21082#A2)gives the source evidence and model responses\.### 2\.1What We Require of AutoResearch

Our AutoResearch program has three joint requirements\.

1. 1\.Autonomy\.AutoResearch agents should work around the clock with minimal human intervention\.
2. 2\.Scale and scope\.Many projects should run in parallel across scientific disciplines\.
3. 3\.Scientific quality\.Projects should pursue original, PhD\-level frontier scientific research worthy of publication\.

Frequent human intervention limits how many projects a human scientist can supervise, so autonomy and scale depend on better scientific judgment\.

### 2\.2What Scientific Quality Depends On

Scientific quality depends on three contributors\.

1. 1\.Model capability\.Models must interpret evidence, adapt to experimental results, and choose useful next steps\.
2. 2\.The agentic system\.Systems must organize experiments, preserve project knowledge, and support review across long runs\.
3. 3\.Human scientific judgment\.Human scientists currently diagnose scientific errors and steer experiments toward the research question\.

Agentic systems could reduce dependence on human judgment by adding rules and automated checks for observed failures\. However, a growing set of instructions becomes harder for models to follow reliably\. Extra instructions also consume tokens and can impair task performance\. Controlled report\-writing experiments show declining adherence and, for some models, declining report quality as requirements accumulate\([Jaroslawicz et al\., 2025](https://arxiv.org/html/2610.21082#bib.bib12)\)\.

Rules also leave scientific questions for the model to resolve\. An instruction to check a measurement does not specify which assumption is wrong or what evidence would expose it\. In our projects, existing checks and review did not reliably supply these judgments\. Senior researchers reviewed projects several times daily to identify stalled or misdirected work and choose the next test\. Without these corrections, continued execution could consume computation without resolving the scientific problem\. A capable AutoResearch agent must learn to supply this diagnosis and change of direction on its own\.

### 2\.3Two Examples of Human Scientific Judgment

Figure[2](https://arxiv.org/html/2610.21082#S2.F2)shows how a human correction changes what a project can learn from its next experiment\.

#### Check the measurement before scaling up\.

The Transformer project asks whether reusing a pretrained model’s layers, with small trainable adapters, can improve performance\. It first needs tasks on which the small base model can be evaluated meaningfully\. After observing zero scores on three LiveCodeBench groups, the agent proposes moving to a larger model\. The senior researcher keeps the small model and asks the agent to find suitable tasks\. A later audit finds that the scorer failed to extract answers from generated text\. Changing model size with the same scorer would leave this error in place\.

#### Question the fit before refining the search\.

The cloth project asks how simulation parameters should change with mesh resolution to preserve realistic motion\. Its agent fixes four of six parameters at zero, a restriction carried forward from its own earlier runs, and proposes refining the two\-parameter search after the best sampled fits reach the boundary\. The senior researcher points out that missing fitted values do not justify setting the corresponding physical parameters to zero and calls for fitting all six with the restricted fit as a comparison\.

Both workflows could keep running without testing the mistaken assumptions that the corrections address\. We evaluate whether models recognize these scientific decisions from the project record\.

## 3From Human Corrections to Evaluation

Table 1:Scientific fields represented in the benchmark, grouped by each project’s primary objective\. The construction procedure does not depend on these area labels\.
- Planning\(123 cases\)\. The agent starts costly runs before basic checks, or runs experiments too small to answer the question\.
- Adapting\(65 cases\)\. It keeps the same hypothesis or search strategy even when earlier experiments have shown what should change\.
- Anomaly awareness\(88 cases\)\. It overlooks suspicious results that should trigger checks for bugs, faulty measurements, or confounding factors\.
- Evidence sufficiency\(47 cases\)\. It declares an idea a success or a failure before the experiments justify that conclusion\.
- Repetition\(4 cases\)\. It repeats an experiment or computation whose answer is already available\.
- Focus\(122 cases\)\. It tests a different claim or optimizes a proxy, leaving the project’s actual research question unanswered\.

Table 2:Judgment categories, the number of cases involving each, and what the agent does wrong\. Counts include primary and secondary judgment labels across all 248 cases\. Categories can overlap\. Appendix[C](https://arxiv.org/html/2610.21082#A3)distinguishes judgment labels from historical\-error coding\.

Figure 3:Accuracy by \(a\) scientific field \(Table[3](https://arxiv.org/html/2610.21082#S3.F3)\) and \(b\) judgment category \(Table[3](https://arxiv.org/html/2610.21082#S3.F3)\)\. Models run with the project record at medium reasoning effort\. Section[4\.1](https://arxiv.org/html/2610.21082#S4.SS1.SSS0.Px1)describes the human baseline\. Categories in \(b\) can overlap\.### 3\.1What Should the Evaluation Preserve?

We build cases retrospectively from completed scientific research projects\. Each project’s senior researcher supervised it from the initial idea through the final write\-up\. The full history links human interventions to subsequent investigation and provides evidence of their scientific value\. We use this evidence to recover useful corrections, then evaluate them at the earlier decision point\. Evaluators see only the earlier project record and two plans and judge which is more likely to advance the scientific question while the outcome is still unknown\. Providing the same plans isolates this judgment from generating plausible alternatives\. Figure[1](https://arxiv.org/html/2610.21082#S1.F1)summarizes construction and evaluation\.

### 3\.2Recovering a Decision from Its History

Candidate decisions come from human messages and changes to the project’s human\-instruction section\. The case writer inspects the full project history, recovers the original LLM plan, and selects a commit before the human intervention\. It reconstructs the human\-corrected plan using the intervention and subsequent scientific investigation\. The evaluator receives the project record at that commit and the two plans, without the later history\. Accuracy measures how often the evaluator selects the reference plan\.

Grok 4\.6 wrote the cases\. Appendix[A](https://arxiv.org/html/2610.21082#A1)reports a controlled pilot in which the writer gains no advantage over the other models on its own cases\. The writing instructions exclude administrative actions, unrecoverable states or decisions, and human corrections that cannot be linked to an error\. Both options follow the same EnglishGoal:andPlan:format\. The prompt requests comparable detail and length without wording that identifies the reference plan\. The evaluator does not see the writer’s explanations\. Each case is linked to its commit and exact option text\.

We selected 248 cases from 26 projects using 120 model responses per case collected during construction\. These responses cover five models, twelve conditions, and both option orders\. We retained cases whose model responses vary across the two plans\. Each retained case keeps a minimum share of responses on both plans\. Four one\-sided cases are kept only after checking their reference plans against documentation at the case commit\. This selection focuses the comparison on cases where model responses vary\. Model\-in\-the\-loop selection follows challenge\-set precedent\([Kiela et al\., 2021](https://arxiv.org/html/2610.21082#bib.bib13)\)\. The precedent applies to construction, and the resulting scores are diagnostic and measure recognition on decisions where models disagree rather than accuracy on research decisions in general\. Case review distinguishes evidence available at the decision point from later human corrections, outcomes, and changes in the senior researcher’s goals\.

### 3\.3Recurring Failure Modes

The human corrections reveal recurring failures across scientific research projects \(Table[3](https://arxiv.org/html/2610.21082#S3.F3)\)\. We name each category after the judgment involved and describe what goes wrong\. These judgments range from adapting to experimental results to deciding whether the evidence suffices for a conclusion\. The focus category covers experiments that can run correctly yet leave the research question unanswered\. For example, changing a cloth’s physical area does not answer how its parameters should change with mesh resolution on the same cloth\. The coverage counts and radar scores group cases by these judgments\. All 248 cases have a primary judgment label, and some carry secondary labels\. Historical\-error coding assigns 243 cases a primary error label and up to two secondary labels\. Coding uses the original LLM plans and human corrections, without the evaluated responses \(Appendix[C](https://arxiv.org/html/2610.21082#A3)\)\.

Construction requires a recoverable project state and documented human correction, so it applies wherever these records exist\. Table[3](https://arxiv.org/html/2610.21082#S3.F3)groups projects into six scientific fields by primary objective\. Biological research using neural models is therefore classified as a biological application\.

SciΠ\\Pivot:Frontier scientific research→\\rightarrowexperimental evidence→\\rightarrowhuman corrections→\\rightarrowPI Judgment tests

Table 3:Representative scientific research benchmarks and how they establish good answers\. SciΠ\\Pivot tests scientific planning and decision\-making using experimental records and senior researchers’ corrections from real frontier scientific research\. Some benchmarks evaluate the listed activities in separate tasks\. Multiple means coverage across broad scientific fields\. AI/ML subfields count as one field\.
### 3\.4Relation to Existing Benchmarks

Existing benchmarks assess different aspects of scientific research \(Table[3](https://arxiv.org/html/2610.21082#S3.T3)\)\. PaperBench tests replication, while RE\-Bench and MLE\-bench test progress against defined research or engineering objectives\([Starace et al\., 2025](https://arxiv.org/html/2610.21082#bib.bib26);[Wijk et al\., 2025](https://arxiv.org/html/2610.21082#bib.bib30);[Chan et al\., 2025](https://arxiv.org/html/2610.21082#bib.bib4)\)\. ScienceAgentBench tests executable scientific workflows\([Chen et al\., 2025](https://arxiv.org/html/2610.21082#bib.bib5)\)\. The SDE benchmark includes scenarios from ongoing and past scientific research\([Song et al\., 2026](https://arxiv.org/html/2610.21082#bib.bib25)\); Corral supports scientific environments, trajectory analysis, and replay\([Ríos\-García et al\., 2026](https://arxiv.org/html/2610.21082#bib.bib22)\)\. BLADE uses analysis decisions from human experts\([Gu et al\., 2024b](https://arxiv.org/html/2610.21082#bib.bib7)\); TruthInsightBench assesses evidence for scientific claims\([Yang et al\., 2026](https://arxiv.org/html/2610.21082#bib.bib33)\); SoundnessBench uses human reviewers’ judgments of proposals\([Ho et al\., 2026](https://arxiv.org/html/2610.21082#bib.bib9)\); and ForeSci evaluates plans against later literature\([Tian et al\., 2026](https://arxiv.org/html/2610.21082#bib.bib28)\)\.

SciΠ\\Pivot evaluates whether models identify the better\-supported scientific decision using information available before a human intervention\. Its reference plans are grounded in human corrections and the experimental records that explain them\. The AI Scientist systems and Agon develop workflows for conducting experiments and producing scientific manuscripts\([Lu et al\., 2024](https://arxiv.org/html/2610.21082#bib.bib15);[Yamada et al\., 2025](https://arxiv.org/html/2610.21082#bib.bib32);[Sun et al\., 2026](https://arxiv.org/html/2610.21082#bib.bib27)\)\. SciΠ\\Pivot tests the scientific judgments needed to guide such workflows\.

## 4Can Models Recognize Human Corrections?

Table 4:Model accuracy on the same 248 cases, averaged over both option orders\. Project and Options only use medium effort\. Best observed project is the highest score among seven conditions\. These conditions are the five initial effort settings and the first and second review at medium effort\. Complete 95% intervals are in Appendix[A](https://arxiv.org/html/2610.21082#A1)\. Appendix[D](https://arxiv.org/html/2610.21082#A4)describes the human baseline\. The options\-only condition measures model context sensitivity\.### 4\.1Evaluation Setup

We evaluate GPT\-6 Astra\([OpenAI, 2026b](https://arxiv.org/html/2610.21082#bib.bib19)\), Claude Opus 5\([Anthropic, 2026](https://arxiv.org/html/2610.21082#bib.bib1)\), Grok 4\.6\([xAI, 2026](https://arxiv.org/html/2610.21082#bib.bib31)\), and GPT\-5\.6 Sol and Terra\([OpenAI, 2026a](https://arxiv.org/html/2610.21082#bib.bib18)\), abbreviated as Astra, Opus, Grok, Sol, and Terra\. In the*project*condition, the model receives two plans and the project record\. We ask it to start with the research idea, proposal, and progress notes, then consult other files as needed\. The project record excludes Git history and later project state\. Network access and research execution are prohibited\. The model returns a choice and an explanation\. The*options\-only*condition presents the same plans without the project record\. Client details appear in Appendix[A](https://arxiv.org/html/2610.21082#A1)\. Appendix[E](https://arxiv.org/html/2610.21082#A5)gives the exact prompts, all twelve conditions, and available materials\.

The main comparison requests medium reasoning effort and evaluates every case in two separate sessions with reversed option order\. We average correctness over the two orders, then give each case equal weight\. Chance accuracy is 50%\. Completed responses without a parseable choice count as incorrect, although all responses in the main comparison have a parsed choice\. We estimate 95% intervals from 10,000 paired bootstrap samples of cases and report project\-cluster intervals in the appendix\.

#### Human baseline\.

The human baseline comes from fresh researchers with no prior exposure to these projects and no involvement in supervising them\. They answered through an interface for reading the project, with an agent assistant available\. Each learned the project from the project record and judged a case, making the final choice\. Thirteen PhD students and postdocs took part, and all were informed about the study, gave consent, and are reported under anonymous identifiers\. Appendix[D](https://arxiv.org/html/2610.21082#A4)describes the human baseline protocol\.

### 4\.2Models Often Reject the Human\-Corrected Plan

At medium effort with the project record, four of the five models select the rejected plan more often than the reference plan on this selected set \(Table[4](https://arxiv.org/html/2610.21082#S4.T4)\)\. For all four, the upper bounds of both case and project bootstrap intervals are below 50%\. Astra reaches 59\.5%, and the five\-model average is 39\.2%\. The highest observed project score across all tested effort settings is Astra’s 63\.5% at xhigh, compared with 73\.4% for the human baseline\.

On 49 cases, all five models choose the rejected plan in both orders; on 16, they all choose the reference plan\. The original LLM plan is on\-policy text, while the human\-corrected plan is off\-policy\. A systematic preference for the original plan is the structural signature of this asymmetry, not a coincidence\.

Figure 4:Accuracy difference with the project record, relative to the options\-only condition\. Comparisons pair the same cases at medium effort\. Bars show 95% case\-bootstrap intervals\.Figure 5:More deliberation can restore a rejected plan after a correct choice\. \(a\) Accuracy at each requested effort setting\. \(b\) The initial answer at medium effort and its two recorded continuations\. \(c\) Repairs and new errors across five models and both orders\. Each review contains 2,480 response transitions on 248 cases\. Panels \(a\) and \(b\) use a 25–67% accuracy range\. The dashed line marks 50%\. Effort settings are not matched token budgets\. Repeated responses are not independent cases\.
### 4\.3Different Judgments Reveal Different Gaps

Figure[3](https://arxiv.org/html/2610.21082#S3.F3)breaks down the variation across types of scientific decisions by scientific field \(a\) and by the judgment each case requires \(b\)\. Each model point averages the two orders within a case and weights cases equally within its group\. Astra reaches 72\.2% on anomaly\-awareness cases, but 56\.1% on planning cases and 54\.9% on focus cases\.

Two cases illustrate the judgment needed to choose an experimental scale \(Appendix[C](https://arxiv.org/html/2610.21082#A3)\)\. In case 21, an untested setup calls for a pilot, yet models retain a 200\-item\-per\-cell batch in seven of ten responses\. In case 13, testing the hypothesis requires a full\-length training run, yet models retain three short gated runs in six of ten responses\. Models must allocate effort according to what the investigation requires, rather than consistently prefer larger or smaller experiments\.

### 4\.4Lower Accuracy with the Project Record

The project record contains observations alongside the original agent’s explanations, including mistaken interpretations\. On this selected set, every model nevertheless scores higher without the project record at medium effort\. Mean accuracy is 39\.2% with the project record and 55\.8% without it \(Figure[4](https://arxiv.org/html/2610.21082#S4.F4)\)\. For four models, the paired 95% interval of this decrease excludes zero, and Astra’s includes zero\. Long contexts can lower task accuracy even when the added material is informative\([Liu et al\., 2023](https://arxiv.org/html/2610.21082#bib.bib14)\)\. The comparison evaluates the complete protocol of the project condition, including retrieval, interpretation, and the client\-specific reading interface\.

The Transformer case illustrates how an inherited explanation can make the rejected plan appear justified\. Opus accepts the progress note’s claim that scoring faults were excluded and uses the record to support the original mistake by treating that claim as established evidence\. PI Judgment requires checking the claim against the observations and code\.

### 4\.5More Thinking Can Lead Back to the Same Mistake

Test\-time scaling can improve model performance by allocating more computation to reasoning\([Snell et al\., 2024](https://arxiv.org/html/2610.21082#bib.bib24)\)\. Can further deliberation help models question these explanations and retain a correction? We vary requested effort from low through max and continue the original medium\-effort session twice\. Each continuation permits further project reading and a changed answer, without correctness feedback\. Refinement methods that supply external feedback can improve outputs\([Madaan et al\., 2023](https://arxiv.org/html/2610.21082#bib.bib16);[Shinn et al\., 2023](https://arxiv.org/html/2610.21082#bib.bib23)\), while reconsideration here receives no feedback, and self\-correction without external feedback can reduce accuracy\([Huang et al\., 2023](https://arxiv.org/html/2610.21082#bib.bib10)\)\. We pair each answer with its recorded parent to count both repaired and newly introduced errors\.

Higher effort sometimes helps, but the gains are inconsistent \(Figure[5](https://arxiv.org/html/2610.21082#S4.F5)a\)\. From low to max, four models improve while Sol declines from 36\.1% to 29\.2%\. Terra rises from 29\.2% to 52\.0% after the first review, then falls to 42\.7% after the second\. Opus also loses accuracy on the second review, and both declines exclude zero under paired case and project intervals\.

Across the five models, the first review repairs 422 errors and introduces 302\. The second repairs 264 and introduces 305, partially undoing the earlier gain\. Mean accuracy across the three stages is 39\.2%, 44\.0%, and 42\.3%, with paired intervals in Table[10](https://arxiv.org/html/2610.21082#A1.T10)\.

In the Transformer case, Opus first chooses the larger model, then questions the scorer and chooses the reference plan\. On its second review, it returns to the larger model with a premise that the frozen code contradicts\.

## 5From Diagnosis to Better AutoResearch Agents

Sustained scientific research exposed decisions that frontier models and agentic systems still left to senior researchers\. SciΠ\\Pivot turns these human corrections into tests of PI Judgment, such as adapting to experimental results and keeping experiments on the research question\. Even with the project record and further deliberation, models often return to the decisions senior researchers corrected\.

Each case supplies a project record, a preferred human\-corrected plan, and a rejected original LLM plan\. These comparisons match the training format of direct preference optimization\([Rafailov et al\., 2023](https://arxiv.org/html/2610.21082#bib.bib20)\)\. As future work, a preference scorer trained on these comparisons could supply rewards for research actions in agentic reinforcement learning\. The next test is whether such training improves decisions in new projects and reduces human intervention\.

Learning these shared judgments could support more autonomous scientific AutoResearch agents across disciplines, including research aimed at improving AI systems themselves\.

## Limitations

SciΠ\\Pivot covers theoretical and computational research using CPUs and GPUs, including simulation and analysis of existing data such as biological images\. Wet\-lab execution and workflows using specialized commercial scientific software such as COMSOL remain outside this evaluation\. The cases test judgments central to PhD research across disciplines, including experiment selection, anomaly diagnosis, and revision of conclusions\. Evaluation in other settings would test how these judgments transfer alongside different tools and experimental procedures\.

SciΠ\\Pivot measures recognition of corrective scientific decisions drawn from projects supervised by senior researchers and selected using the procedure in Section[3](https://arxiv.org/html/2610.21082#S3)\. Generating plans and carrying out scientific research require separate evaluation\. The selected cases do not estimate how often these failures occur across all scientific research activity\. Several cases come from each project, so we report project\-cluster intervals alongside case\-level intervals\.

The evaluated clients expose different reading tools, and requested effort settings do not specify equal token budgets\. Our effort and reconsideration experiments test these available controls; they do not test controlled compute scaling or verifier\-guided search\.

## Ethics Statement

The source records include scientific research conversations and project histories\. Public materials should preserve the evidence needed to understand each decision while removing personal identifiers, account information, and unrelated private discussion\. All fresh researchers were informed about the study, gave consent, and are reported under anonymous identifiers; Appendix[D](https://arxiv.org/html/2610.21082#A4)describes the evaluation setting\.

Language models assisted with drafting, organizing the literature, and writing analysis code\. The authors remain responsible for checking the claims, references, and reported results\.

#### Data availability\.

The complete question set of 248 cases with anonymized case identifiers, the two options per case in their exact wording, and reference answers is released in the SciΠ\\Pivot repository\([AutoResearch\-Factory, 2026](https://arxiv.org/html/2610.21082#bib.bib3)\)\. The repository also hosts the exact evaluation and case\-writing prompts and aggregate model results for all twelve conditions\. Individual model response logs are not part of this release\. The frozen project records are too large to upload, and the same repository will host them\. All case details will be released under an open license that permits research and training use, in JSON and plain\-text formats\.

## References

- Anthropic \(2026\)Anthropic\. 2026\.[System card: Claude Opus 5](https://www.anthropic.com/claude-opus-5-system-card)\.[https://www\.anthropic\.com/claude\-opus\-5\-system\-card](https://www.anthropic.com/claude-opus-5-system-card)\.Published July 24, 2026\. Updated August 19, 2026\.
- Austin et al\. \(2021\)Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton\. 2021\.[Program synthesis with large language models](https://arxiv.org/abs/2108.07732)\.*Preprint*, arXiv:2108\.07732\.
- AutoResearch\-Factory \(2026\)AutoResearch\-Factory\. 2026\.[SciPIvot](https://github.com/AutoResearch-Factory/SciPIvot)\.[https://github\.com/AutoResearch\-Factory/SciPIvot](https://github.com/AutoResearch-Factory/SciPIvot)\.
- Chan et al\. \(2025\)Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry\. 2025\.[Mle\-bench: Evaluating machine learning agents on machine learning engineering](https://arxiv.org/abs/2410.07095)\.*Preprint*, arXiv:2410\.07095\.
- Chen et al\. \(2025\)Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N\. Baker, Benjamin Burns, Daniel Adu\-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun\. 2025\.[Scienceagentbench: Toward rigorous assessment of language agents for data\-driven scientific discovery](https://arxiv.org/abs/2410.05080)\.*Preprint*, arXiv:2410\.05080\.
- Gu et al\. \(2024a\)Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar\-Lezama, Gabriel Synnaeve, and Sida I\. Wang\. 2024a\.[Cruxeval: A benchmark for code reasoning, understanding and execution](https://arxiv.org/abs/2401.03065)\.*Preprint*, arXiv:2401\.03065\.
- Gu et al\. \(2024b\)Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard\-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M\. Zhang, Lanyi Zhu, Mike A Merrill, Jeffrey Heer, and Tim Althoff\. 2024b\.[BLADE: Benchmarking language model agents for data\-driven science](https://doi.org/10.18653/v1/2024.findings-emnlp.815)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 13936–13971, Miami, Florida, USA\. Association for Computational Linguistics\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt\. 2021\.[Measuring mathematical problem solving with the math dataset](https://arxiv.org/abs/2103.03874)\.*Preprint*, arXiv:2103\.03874\.
- Ho et al\. \(2026\)Sy\-Tuyen Ho, Minghui Liu, Huy Nghiem, and Furong Huang\. 2026\.[Soundnessbench: Can your ai scientist really tell good research ideas from bad ones?](https://arxiv.org/abs/2605.30329)*Preprint*, arXiv:2605\.30329\.
- Huang et al\. \(2023\)Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou\. 2023\.[Large language models cannot self\-correct reasoning yet](https://arxiv.org/abs/2310.01798)\.*Preprint*, arXiv:2310\.01798\.
- Jain et al\. \(2024\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\. 2024\.[Livecodebench: Holistic and contamination free evaluation of large language models for code](https://arxiv.org/abs/2403.07974)\.*Preprint*, arXiv:2403\.07974\.
- Jaroslawicz et al\. \(2025\)Daniel Jaroslawicz, Brendan Whiting, Parth Shah, and Karime Maamari\. 2025\.[How many instructions can LLMs follow at once?](https://arxiv.org/abs/2507.11538)*Preprint*, arXiv:2507\.11538\.
- Kiela et al\. \(2021\)Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, P\. Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Talat, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams\. 2021\.[Dynabench: Rethinking benchmarking in nlp](https://arxiv.org/abs/2104.14337)\.*Preprint*, arXiv:2104\.14337\.
- Liu et al\. \(2023\)Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, F\. Petroni, and Percy Liang\. 2023\.[Lost in the middle: How language models use long contexts](https://arxiv.org/abs/2307.03172)\.*Preprint*, arXiv:2307\.03172\.
- Lu et al\. \(2024\)Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha\. 2024\.[The AI Scientist: Towards fully automated open\-ended scientific discovery](https://arxiv.org/abs/2408.06292)\.*Preprint*, arXiv:2408\.06292\.
- Madaan et al\. \(2023\)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark\. 2023\.[Self\-refine: Iterative refinement with self\-feedback](https://arxiv.org/abs/2303.17651)\.*Preprint*, arXiv:2303\.17651\.
- Miao et al\. \(2020\)Shen\-Yun Miao, Chao\-Chun Liang, and Keh\-Yih Su\. 2020\.[A diverse corpus for evaluating and developing English math word problem solvers](https://aclanthology.org/2020.acl-main.92/)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*\.
- OpenAI \(2026a\)OpenAI\. 2026a\.[GPT\-5\.6 System Card](https://deploymentsafety.openai.com/gpt-5-6)\.[https://deploymentsafety\.openai\.com/gpt\-5\-6](https://deploymentsafety.openai.com/gpt-5-6)\.Published July 9, 2026\. Accessed October 11, 2026\.
- OpenAI \(2026b\)OpenAI\. 2026b\.[GPT\-6 Astra System Card](https://deploymentsafety.openai.com/gpt-6-astra)\.[https://deploymentsafety\.openai\.com/gpt\-6\-astra](https://deploymentsafety.openai.com/gpt-6-astra)\.Published September 3, 2026\. Accessed October 11, 2026\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\. 2023\.[Direct preference optimization: Your language model is secretly a reward model](https://arxiv.org/abs/2305.18290)\.*Preprint*, arXiv:2305\.18290\.
- Rein et al\. \(2023\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\. 2023\.[Gpqa: A graduate\-level google\-proof q&a benchmark](https://arxiv.org/abs/2311.12022)\.*Preprint*, arXiv:2311\.12022\.
- Ríos\-García et al\. \(2026\)Martiño Ríos\-García, Nawaf Alampara, Chandan Gupta, Indrajeet Mandal, Sajid Mannan, Ali Asghar Aghajani, N\. M\. Anoop Krishnan, and Kevin Maik Jablonka\. 2026\.[Ai scientists produce results without reasoning scientifically](https://arxiv.org/abs/2604.18805)\.*Preprint*, arXiv:2604\.18805\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\. 2023\.[Reflexion: Language agents with verbal reinforcement learning](https://arxiv.org/abs/2303.11366)\.*Preprint*, arXiv:2303\.11366\.
- Snell et al\. \(2024\)Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar\. 2024\.[Scaling LLM test\-time compute optimally can be more effective than scaling model parameters](https://arxiv.org/abs/2408.03314)\.*Preprint*, arXiv:2408\.03314\.
- Song et al\. \(2026\)Zhangde Song, Jieyu Lu, Yuanqi Du, Botao Yu, Thomas M\. Pruyn, Yue Huang, Kehan Guo, Xiuzhe Luo, Yuanhao Qu, Yi Qu, Yinkai Wang, Haorui Wang, Jeff Guo, Jingru Gan, Parshin Shojaee, Di Luo, Steven Dillmann, Andres M Bran, Gen Li, and 38 others\. 2026\.[Evaluating large language models in scientific discovery](https://arxiv.org/abs/2512.15567)\.*Preprint*, arXiv:2512\.15567\.
- Starace et al\. \(2025\)Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan\. 2025\.[Paperbench: Evaluating ai’s ability to replicate ai research](https://arxiv.org/abs/2504.01848)\.*Preprint*, arXiv:2504\.01848\.
- Sun et al\. \(2026\)Youran Sun, Xingyu Ren, Chugang Yi, Jiaxuan Guo, Kejia Zhang, Jianda Du, and Haizhao Yang\. 2026\.[Agon: An autonomous large\-scale omnidisciplinary research system built on prompt economy](https://arxiv.org/abs/2606.24177)\.*Preprint*, arXiv:2606\.24177\.
- Tian et al\. \(2026\)Qiuyu Tian, Haojie Yin, Yingce Xia, Youyong Kong, and Zequn Liu\. 2026\.[Foresci: Evaluating llm agents for forward\-looking ai research judgment](https://arxiv.org/abs/2606.00644)\.*Preprint*, arXiv:2606\.00644\.
- Wang et al\. \(2023\)Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R\. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang\. 2023\.[SciBench: Evaluating college\-level scientific problem\-solving abilities of large language models](https://arxiv.org/abs/2307.10635)\.*Preprint*, arXiv:2307\.10635\.
- Wijk et al\. \(2025\)Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, and 4 others\. 2025\.[Re\-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts](https://arxiv.org/abs/2411.15114)\.*Preprint*, arXiv:2411\.15114\.
- xAI \(2026\)xAI\. 2026\.[Model card: Grok 4\.6](https://media.x.ai/v1/website/card-4p6-4cd2dc57.pdf)\.[https://media\.x\.ai/v1/website/card\-4p6\-4cd2dc57\.pdf](https://media.x.ai/v1/website/card-4p6-4cd2dc57.pdf)\.Published August 12, 2026\. Revised August 17, 2026\.
- Yamada et al\. \(2025\)Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha\. 2025\.[The AI Scientist\-v2: Workshop\-level automated scientific discovery via agentic tree search](https://arxiv.org/abs/2504.08066)\.*Preprint*, arXiv:2504\.08066\.
- Yang et al\. \(2026\)Zhibo Yang, Chen Zhang, Yuewei Zhang, and Hao Wang\. 2026\.[Truthinsightbench: An evidence\-grounded benchmark for automated evaluation of open\-ended scientific discovery agents](https://arxiv.org/abs/2609.05079)\.*Preprint*, arXiv:2609\.05079\.

## Appendix AModel Results and Statistical Details

### A\.1Project\-Reading Clients

The requested model identifiers aregpt\-6\-astra,claude\-opus\-5,grok\-4\.6,gpt\-5\.6\-sol, andgpt\-5\.6\-terra\. Opus and Grok use dedicated file\-reading and search tools through one shared client harness, the Claude client\. Shell execution, editing, and web tools are disabled in that harness\. The GPT models use shell commands through Codex, with a read\-only project and web search disabled\. All project prompts prohibit network access and reading outside the project record\. The options\-only prompt removes the project record and its reading instructions while keeping the same plans\. The comparison between conditions therefore includes retrieval, interpretation, and the client\-specific tool interface\.

### A\.2Scoring and Uncertainty

The benchmark contains 248 cases from 26 projects, with two option orders per case\. Each medium\-effort Project or Options only cell therefore contains 496 completed responses\. We retain the last logged record for each combination of case, option hash, condition, engine, requested model, effort, prompt hash, and repeat identifier\. This rule replaces a failed or superseded attempt rather than counting a retry as another observation\. All 29,760 retained runs completed without a transport error\. Eight responses could not be parsed and receive a score of zero\. These are four Sol responses at xhigh, one Sol response at max, one Terra response at high, one Grok response at xhigh, and one Grok response after the second review\. All eight unparsed responses occur in project\-condition runs\. Both medium\-effort primary conditions are fully parsed\. Excluding unparsed responses changes any affected cell by at most 0\.23 percentage points\.

For each case, we average the two binary scores before estimating uncertainty\. The main intervals use 10,000 percentile bootstrap samples of 248 cases drawn with replacement\. The random seed is 20261004\. We resample the same case indices in both conditions when estimating their difference\. The two option orders are repeated measurements, not independent cases\. Table[5](https://arxiv.org/html/2610.21082#A1.T5)reports the accuracy of both conditions and their paired difference, with complete case\-bootstrap intervals\.

Table 5:Medium\-effort accuracies and paired Project minus Options only differences on 248 cases\. Brackets give 95% percentile case\-bootstrap confidence intervals after averaging the two option orders\.Several cases come from the same project and may describe related human interventions\. We therefore sample 26 projects with replacement in each bootstrap replicate and include every case from each sampled project\. A project selected more than once contributes its complete case set each time\. No additional sampling occurs within projects\. We continue to weight cases equally and use the same project draws in both conditions\. Table[6](https://arxiv.org/html/2610.21082#A1.T6)reports project\-cluster intervals for project accuracy and the difference between conditions\.

Table 6:Project\-cluster 95% intervals for project accuracy \(%\) and the Project minus Options only difference \(percentage points\)\. Both estimates retain equal case weights\.
### A\.3Project Coverage and Effort Settings

Tables[7](https://arxiv.org/html/2610.21082#A1.T7)and[8](https://arxiv.org/html/2610.21082#A1.T8)list each project’s identifier, research question, area, and case count\. We classify projects by their primary scientific objectives\. The construction procedure does not depend on these area labels\. Project size ranges from one to 43 cases, so averaging project means differs from averaging cases\. The effort labels in Table[9](https://arxiv.org/html/2610.21082#A1.T9)are the requested effort settings that the evaluated model interfaces expose\. We do not interpret them as equal token budgets or equal compute across providers\. Every entry uses the same 248 cases and two orders per case\. PDE abbreviates partial differential equation\.

Table 7:Benchmark projects \(1/2\)\. The questions describe scientific objectives rather than established findings\. Area labels describe the current sample\. We do not count secondary methodological links separately\.Table 8:Benchmark projects \(2/2\)\. The questions describe scientific objectives rather than established findings\. Area labels describe the current sample\. We do not count secondary methodological links separately\.Table 9:Accuracy \(%\) at every requested effort setting\. Each cell reports Project / Options only\. Unparsed completed responses count as incorrect\.
### A\.4Option Order and Reconsideration

The two orders place the reference plan in position A and position B, respectively\. Table[12](https://arxiv.org/html/2610.21082#A1.T12)reports each order separately at medium effort\. The main score gives the two orders equal weight instead of selecting the better order\. The A/B differences can reflect a position preference or an interaction between option order and reasoning\.

The reconsideration experiment follows the project\-condition answer through two continuation stages at medium effort\. We join each child to the logged parent run and attempt, then verify that parent and child runs agree on the case, option hash, commit, model, engine, effort, repeat, and order\. Each model has 496 parent\-child pairs at each stage\. The change at a stage is the number of corrected errors minus the number of newly introduced errors, divided by 496\. The accuracy trajectories in Figure[5](https://arxiv.org/html/2610.21082#S4.F5)use these actual chains rather than independently regenerated initial answers\. We count changes in the selected plan rather than changes in the underlying scientific reasoning\. Table[10](https://arxiv.org/html/2610.21082#A1.T10)reports paired changes after the second review, with uncertainty estimated from 100,000 resamples\. For the five\-model mean, we average the fixed models within each case before resampling cases or whole projects\.

Table 10:Accuracy change after the second review, relative to the first, in percentage points\. We average both option orders within each of 248 cases before computing paired differences\. Intervals use 100,000 paired percentile bootstrap samples\. The project bootstrap resamples 26 whole projects with replacement and retains case weights\. The mean averages the fixed set of five evaluated models within each case\.
### A\.5Writer Sensitivity

One writer model wrote all released cases under one instruction set, so writer identity is constant across every model in the main comparison\. A controlled pilot checks whether this choice favors the writer model\. The pilot uses three writers, Astra, Opus, and Grok\. Each writer produced options for 20 shared cases, with two option orders and three answering models\. Seven shared cases have project records frozen at different commits across writers\. We therefore analyze the 13 cases whose frozen commits and option hashes match across all three writers \(Table[12](https://arxiv.org/html/2610.21082#A1.T12)\)\. Every cell contains 26 responses\.

Table 11:Accuracy \(%\) when the reference plan appears first \(A\) or second \(B\), with 248 responses per order\.
Table 12:Controlled writer pilot on 13 cases whose project records are frozen at the same commit for all three writers, with 26 responses per cell\. Columns identify the answering model, and entries are accuracy \(%\)\. Options denotes the options\-only condition\.

In the pilot, the writer of the released set has no advantage over its peers\. Grok, the writer of the released set, is never the top scorer on its own cases\. Opus reaches 34\.6% on Grok\-written cases with the project record, against 23\.1% for Grok itself, and Opus also leads the options\-only cells\. The largest single gain runs across writers rather than along them\. Astra reaches its highest accuracy, 76\.9%, on Opus\-written cases\. Writer identity changes absolute case difficulty, but the writer of the released set never outscores its peers on its own text\. The main comparison holds the writer and option texts fixed across all answering models, so writer identity does not confound the cross\-model comparison\.

## Appendix BTwo Illustrative Scientific Decisions

These examples show the decisions that human interventions changed and quote evaluation responses that repeated the original mistake\. Table[13](https://arxiv.org/html/2610.21082#A2.T13)reports the reference\-plan choices of all five models in both option orders for each example\. The reference and rejected plans below reproduce the benchmark task wording\.

### B\.1Checking Evaluation Before Changing Model Size

The project studied whether repeated passes through a language\-model block improve performance\. It was building an evaluation set with several difficulty levels for the selected 0\.8B base model\. At the frozen decision, three LiveCodeBench groups\([Jain et al\., 2024](https://arxiv.org/html/2610.21082#bib.bib11)\)scored 0/480\. A project note treated earlier checks as having ruled out an evaluation fault and proposed rerunning the tasks on a 2B model\. The frozen code, however, passed raw generated text directly to the new tasks’ graders without the official answer postprocessing\. Its preflight also did not reject individual benchmark groups with all\-zero scores\. The senior researcher chose to retain 0\.8B and find suitable official tasks\. A subsequent audit identified the scoring fault, which the reference plan addressed before further calibration\. Later regrading used the same 608 affected completions, verified byte for byte, and recovered 70 correct answers among the 480 code items\. The same four difficulty cells, however, remained unfilled\. In one response under the project condition, Opus nevertheless selected the larger\-model route and wrote, “The blocker is not an evaluation bug, it is a size/tier conflict\.” Holding tasks fixed was a reasonable control in principle\. But accepting the unverified claim that scoring faults had been excluded made the original zero scores an unreliable basis for the size decision\.

For this option order, two subsequent reconsiderations moved the choice from the rejected plan to the reference plan and back to the rejected plan\. The project record and option order were unchanged throughout\. The first reconsideration acknowledged, “I previously trusted STATE\.md’s assertion that the evaluator was ruled out,” and selected the reference plan after examining the pattern of zero scores\. That selection was correct, but the explanation behind it was only partly correct\. It relied in part on a comparison of free\-generation scores with forced\-choice scores, rather than directly identifying the missing postprocessing in the code\. The second reconsideration noticed this difference in evaluation format\. It nevertheless returned to the rejected plan, claiming that the nonzero results on MBPP, the Mostly Basic Python Problems benchmark\([Austin et al\., 2021](https://arxiv.org/html/2610.21082#bib.bib2)\), used an “identical generate/extract/execute pipeline\.” The frozen code contradicts that equivalence\. The old\-pool path used for MBPP truncates generated text before grading, whereas the new\-pool path used for LiveCodeBench passes raw text to its graders\. Success on the MBPP path therefore did not establish that the LiveCodeBench path was correct\. The sequence connects both answer changes to the evidence Opus cited\. Further review repaired the choice once, then reinstated the rejected plan using a faulty comparison\. The quoted plans name the following evaluation suites\. LiveCodeBench covers code generation\([Jain et al\., 2024](https://arxiv.org/html/2610.21082#bib.bib11)\), MATH covers arithmetic\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.21082#bib.bib8)\), and CRUXEval covers code reasoning and execution\([Gu et al\., 2024a](https://arxiv.org/html/2610.21082#bib.bib6)\)\. ASDiv covers arithmetic word problems\([Miao et al\., 2020](https://arxiv.org/html/2610.21082#bib.bib17)\), and SciBench covers scientific problem solving\([Wang et al\., 2023](https://arxiv.org/html/2610.21082#bib.bib29)\)\. GPQA abbreviates Graduate\-Level Google\-Proof QA\([Rein et al\., 2023](https://arxiv.org/html/2610.21082#bib.bib21)\)\.

#### Reference plan\.

Goal:Finish the real\-task tier asset after the 0\.8B Base\-K1 calibration stopped at 8 of 12 cells\.

Plan:Keep 0\.8B as the selected size, and keep the 25, 50, and 75 percent accuracy layers, the quotas, and the official metrics\. Grade generations with the official LiveCodeBench stop and extract functions, and stop a panel when any benchmark group scores all zeros\. Regrade or regenerate the LiveCodeBench and word\-sorting groups\. Add official CRUXEval, ASDiv, and SciBench sources on this 0\.8B Base\. Write the 12\-cell asset after all 12 cells are filled\. Start the four matched\-depth micro\-runs after that asset exists\.

#### Rejected plan\.

Goal:Finish the real\-task tier asset after the 0\.8B Base\-K1 calibration stopped at 8 of 12 cells\.

Plan:Keep the current LiveCodeBench, MATH, and GPQA items\. Score the same registered code, math, and science subset on the 2B Base with the same calibration plan and the same method that graded the 0\.8B outputs, using about 1700 rows\. If 2B fills the missing cells, set the selected experiment size to 2B and write the 12\-cell asset\. If 2B still misses cells, widen the accuracy layers on 0\.8B or change the official tasks after that measurement, then write the asset\. Start the four matched\-depth micro\-runs after that asset exists\.

### B\.2Testing a Restrictive Physical Model

The project aimed to calibrate a cloth simulator against recorded motion before studying how its fitted parameters change with simulation resolution\. Its frozen proposal specified six positive parameters and a nonzero reference vector, but the working plan fixed four parameters to zero and searched only the remaining two\. All ten sampled minima reported in the project record lay at the same corner of their search domains\. The senior researcher clarified that two reference values came from published fits and four from simulator defaults, so missing fitted material values did not justify setting the corresponding physical parameters to zero\. The reference plan therefore reopened the six\-parameter fit while retaining the original two\-parameter slice as a comparison on the same objective\. The defaults supplied a search center for testing the fuller physical model\. Opus described the restricted search as a “standing, human\-authorized restriction” and selected it in both option orders under the project condition\. In one response, it also stated, “Option B is the direct continuation of the actual live plan\.” Its concerns about previous optimization instability and cost were relevant, but the continuing boundary results called for testing the restriction itself\. RG abbreviates renormalization group\.

#### Reference plan\.

Goal:Identify the six simulator parameters on one time\-step cell, using the six\-parameter reference vector as the search center\.

Plan:Keep density at 0\.3046 and search delta, alpha, kappa, kappa\_bnd, stretch, and shear together\. Center the search on phi\_ref = \(0\.3046, 0\.6092, 1\.523e\-4, 1\.523e\-5, 1e\-6, 1e\-3\)\. Use the four full marker recordings as one joint Eq\. \(10\) objective on the native G0 mesh with eight substeps\. Alternate two\-dimensional blocks in the order dynamic, bending, then in\-plane, run that cycle twice, and then run a bounded joint Nelder\-Mead polish\. Fit a second arm that searches only alpha and delta with the other four at zero, on the same objective\. Start cross\-scale RG work when the six\-parameter endpoints sit off the box walls and agree across starts\.

#### Rejected plan\.

Goal:Map how the joint two\-parameter loss surface for alpha and delta changes across allowed time steps, with the other four physical parameters held at zero\.

Plan:Keep density at 0\.3046 and set kappa, kappa\_bnd, stretch, and shear to zero\. Use the four full marker recordings as one joint Eq\. \(10\) objective on the native G0 mesh\. After reading the five\-scale coarse surfaces and the external corner bridge, add the remaining allowed time\-step levels and extra shifted grids on the same two\-parameter box of alpha and delta\. Report sampled minima, edge profiles, and the direction of the surface at the walls\. Start a lowest\-order RG fit when at least five time\-step levels have interior minima in this two\-parameter slice\.

Table 13:Reference\-plan choices across the two option orders at medium effort\. “Options” denotes the options\-only condition\. Each model answers the same case twice, with the option order reversed\. The ten choices per condition are not ten independent cases\.

## Appendix CJudgment Categories and Illustrative Cases

Table 14:Reference\-plan choices at medium effort for the four examples in Appendix[C](https://arxiv.org/html/2610.21082#A3)\. We selected these examples for their decision structure before inspecting model scores\. Each count is out of ten responses from five fixed models and two option orders, not ten independent cases\. Case numbers are zero\-based indices in the benchmark\.### C\.1Coding the Historical Decision

The codebook describes the original LLM plan and the decision that the human intervention changed, without using the evaluated models’ answers\. We organize the codebook around three overlapping demands, namely identifying the research goal, interpreting evidence, and choosing experiments and resource commitments\. The six categories below name the judgment involved and specify what went wrong in a decision\.

- Planning\.The scale, training duration, sequence, or parallelization of experiments does not fit the question that must be resolved next\. This includes both launching a full experiment before checking its pipeline and using a small experiment that cannot test the proposed explanation\. Annotators judge scale against the scientific question, so a larger experiment can be the appropriate correction\.
- Adapting\.A relevant result is already available, but the plan does not use it to revise a hypothesis, configuration, or priority\. The annotator must identify the result that should have changed the next decision\. If the central issue is whether that result is valid, anomaly awareness is the more direct category\.
- Anomaly awareness\.The plan treats an implementation problem, measurement defect, confound, or anomalous output as usable scientific evidence without an adequate diagnostic step\. The issue is whether to check the experiment before interpreting its result, rather than whether the agent can implement the repair\. A correctly executed experiment that answers the wrong research question belongs under focus\.
- Evidence sufficiency\.The available evidence does not support the decision to declare success, failure, completion, or closure\. Examples include rejecting several claims after one immature experiment and treating a noisy result near a threshold as a decisive verdict\. A justified negative conclusion or a stop required by an actual budget limit is not a failure under this category\.
- Repetition\.The proposed work reproduces information already available or presents reused information as independent new evidence\. The annotator must explain what remains scientifically unchanged, such as the predictions, experimental cell, or estimated quantity\. This category excludes planned independent replication and a necessary rerun under a changed protocol\.
- Focus\.Even if executed correctly, the proposed experiment would answer a different question from the one the project is trying to resolve\. This includes substituting a proxy for the target, changing the wrong experimental variable, or treating an agent’s own assumption as a binding human constraint\. When a senior researcher introduces a new goal, annotators must mark it as a later change rather than attribute it to the earlier project state\.

### C\.2Boundaries and Review Procedure

The primary error label follows the smallest decision change needed to correct the original LLM plan\. Annotators may add up to two secondary error labels when these describe distinct errors rather than restating the same error’s cause and consequence\. The six categories have no fixed precedence\. For example, adapting concerns an omitted update, repetition concerns whether proposed work would add information, and evidence sufficiency concerns a conclusion or stop already made\. A case can receive a judgment label without a primary error label\.

The project record, an agent’s interpretation of it, and the original human message are separate evidence sources\. A project note headed “human rule” does not by itself establish that a senior researcher imposed the rule\. Outcomes after a human intervention can explain what happened after the correction, but annotators must not treat them as evidence visible at the decision point\. Related decisions from one human intervention need not be independent instances of an error\.

Annotation status\.The historical\-error coding assigns a primary error label to 243 cases\. The remaining five cases carry a judgment label without one\. All 248 cases have a judgment label, which names the category of judgment the case concerns\.

### C\.3Profiles by Judgment Category

Figure[3](https://arxiv.org/html/2610.21082#S3.F3)\(b\) groups all 248 cases by the scientific judgment they concern\. For each model, a spoke shows accuracy in recognizing the reference plan, averaging both option orders within each case and then weighting cases with that label equally\. Categories can overlap, and the scores describe performance on these cases rather than general abilities independent of this set of cases\. Both panels of Figure[3](https://arxiv.org/html/2610.21082#S3.F3)use the same linear 0–100% radial scale and display all observed values\. The repetition category has only four cases\. Restricting this analysis to the 243 cases with a primary error label changes any model score by at most 1\.6 percentage points\. For Figure[3](https://arxiv.org/html/2610.21082#S3.F3)\(a\), the scientific fields partition the 248 cases, so weighting field scores by case counts recovers each model’s overall score\.

### C\.4Additional Illustrations

We selected the examples below for their decision structure, without using the evaluated models’ error rates as a selection criterion\. Case numbers refer to the zero\-based indices of the benchmark\. Each description summarizes the source material we checked\. Table[14](https://arxiv.org/html/2610.21082#A3.T14)reports the corresponding model choices\.

#### A measurement that barely responds to a stiffness change \(case 206\)\.

In a soft\-support simulation, a stiffness sweep from0\.25×0\.25\\timesto8×8\\timeschanged the center support’s static deflection only from roughly 212 to 222 mm\. All six runs recorded activation of the upper travel limit\. The original LLM plan proposed reducing the number of springs to find the point where visible indentation disappeared and standing performance recovered\. The senior researcher instead asked for an order\-of\-magnitude estimate of how many springs supported the ball and a check of the effective stiffness and travel limit while retaining the fixed grid\. This case illustrates an anomaly\-awareness decision, which is to diagnose whether the simulated response is informative before interpreting the effect of substrate stiffness\. The quoted displacement is the center support’s static deflection, not the ball\-center displacement\.

#### A gate that delays the main measurement \(case 3\)\.

The original LLM plan kept a3×33\\times 3resolution matrix closed until renormalization\-group predictions beat the Transferred baseline on both contact treatments under a statistical admission rule\. The measurement that answers the project question is the grid of effective parameters itself, that is, how cloth parameters change with spatial and time resolution\. The senior researcher’s note of 2026\-08\-02 required starting the matrix on the five cross cells\. Under the note, the baseline comparison serves to assess the scale of the error rather than to decide when the matrix may start\. We assign focus because the inherited gate delayed measuring the quantity the project set out to study\.

#### When to shrink an experiment and when to enlarge it \(cases 21 and 13\)\.

For a new batch\-API measurement, the original LLM plan took the first reading at 200 items per cell, the size used for the paper tables\. It did so even though an earlier full\-size wave had to be discarded after a pipeline bug\. The reference plan ran 20 items per cell on the full grid first and scheduled the 200\-item tables after review of those numbers\. In a reinforcement\-learning project using proximal policy optimization, the opposite direction was needed\. Three seeds gated at 1,000 iterations could only repeat a known failure with insufficient training and could not distinguish that problem from sensitivity to the random seed\. The reference plan trained one seed for the full 8,000 iterations of the historical protocol before adding more seeds\. Both cases illustrate planning decisions, but the appropriate direction of the scale change is opposite\. The 20\-then\-200 item protocol and the 8,000\-iteration budget are local project requirements, not general optima\.

## Appendix DHuman Baseline Protocol

Fresh researchers with no prior exposure to the source projects answer through an interface that presents the two plans and the associated project record\. They learn the project from that record; none of them supervised the source projects\. Thirteen PhD students and postdocs took part\. Each was informed about the study and gave consent\. All are reported under anonymous identifiers\. The baseline therefore measures what a researcher can recover about the senior researcher’s correction from the project record, as the models do\. The interface randomizes option order\. An agent assistant is available to explain the project record and differences between the plans; the fresh researcher makes the final choice\. The default assistant runsgpt\-5\.6\-terrathrough Codex, with Astra and Claude assistants also available\.

### D\.1Coverage and Comparison

All 248 cases have recorded human responses\. The fresh researchers answer 182 of the 248 cases correctly, an accuracy of 73\.4%\. The radar plots apply this same measure within each scientific field or judgment category\.

## Appendix EEvaluation Prompts and Available Materials

### E\.1Twelve Evaluation Conditions

For each model and case, the evaluation contains ten initial conditions\. These combine the project and options\-only conditions with each of the low, medium, high, xhigh, and max requested effort settings\. Two further conditions continue the medium\-effort project session through a first and then a second reconsideration\. Each continuation uses the same reconsideration prompt below and retains the preceding conversation, without correctness feedback\. We evaluate every condition in both option orders, giving 24 responses per model and 120 across the five models for each case\.

### E\.2Prompts and Available Materials

The templates below reproduce the evaluation instructions with display wrapping; the Markdown files preserve the original whitespace\. The placeholders\{a\}and\{b\}hold the two plans, swapped for the second option order\. The source archive provides the manuscript, figures, bibliography, prompts, and aggregate model results for all twelve conditions\. The question archive, provided as supplementary material, contains the complete 248\-case set with anonymized case identifiers, the two options per case in their exact wording, and reference answers\. Appendix[A](https://arxiv.org/html/2610.21082#A1)documents the evaluation clients and scoring rules, and this appendix provides the prompts\.

#### Project condition\.

```
‘/project‘ is the current state of a research
project. You need to think about what to do
next.

Read idea.md first, then proposal.md, then
STATE.md, and then other files if you think it
is necessary.

Below are two candidate plans. Please choose the
one you think is better.

<option_A>
(A) {a}
</option_A>

<option_B>
(B) {b}
</option_B>

Output format:

‘‘‘
{
  "argument": "The reason for your choice",
  "response": "A or B"
}
‘‘‘

Note:

- Only read the contents of ‘/project‘. Do not
go to parent directories. Do not access the
network.
```

#### Options\-only condition\.

```
Below are two candidate next-step plans for a
research project in a particular state. Please
choose the one you think is better.

<option_A>
(A) {a}
</option_A>

<option_B>
(B) {b}
</option_B>

Output format:

‘‘‘
{
  "argument": "The reason for your choice",
  "response": "A or B"
}
‘‘‘
```

#### Reconsideration, used for both continuations\.

```
Please reconsider the original question and both
options carefully, then answer again. You may
review the project files if needed. You may keep
or change your previous answer.

Return your answer in the same JSON format as
before:

‘‘‘json
{
  "argument": "The reason for your choice",
  "response": "A or B"
}
‘‘‘
```