AdaPRA：采用自适应评分标准的数据科学过程奖励智能体

# Process Reward Agents with Adaptive Rubrics for Data Science
Source: [https://arxiv.org/html/2610.33245](https://arxiv.org/html/2610.33245)
Vivek ChariAffiliation:MicrosoftEmail:[dfried@cs\.cmu\.edu](mailto:)Pol LladoAffiliation:MicrosoftEmail:[cprose@cs\.cmu\.edu](mailto:)Daniel FriedAffiliation:Carnegie Mellon UniversityCarolyn RoséAffiliation:Carnegie Mellon UniversityBenjamin Van DurmeAffiliation:MicrosoftSam ThomsonAffiliation:MicrosoftJustin SvegliatoAffiliation:Microsoft

###### Abstract

Data science agents can automate end\-to\-end machine learning workflows, yet they remain far from competitive on human leaderboards\. Agents make process errors such as insufficient exploration, unsound validation, coding bugs, and poor use of a fixed budget \(re\-running the same exploratory data analysis and baseline models in every attempt\)\. We proposeAdaPRAto address these issues: a persistent coding agent harness that accumulates knowledge across attempts on the same task, with*rubric\-guided process supervision*that identifies and corrects errors\. Our process reward agent audits the agent’s trajectory and workspace by executing checks, and adapts its rubrics to each task\. We find that rubrics seeded from easier tasks generalize to harder tasks, with rubric adaptation further improving performance\. With five attempts per task, AdaPRA improves an 80B Qwen model’s leaderboard percentile by 24\.5 points on DSPredict\-Easy and 14\.8 points on DSPredict\-Hard while reducing process errors\. We also show that supervision with rubrics mined from one base model’s runs improves another \(DeepSeek\-V3\.2\), and thatAdaPRAbeats DataPRM, a competitive PRM for data science\. We also share our code for reproducibility\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.33245v1/figures/AdaPRA_Intro_Teaser_No_Rubric_Discovery.png)Figure 1:AdaPRA corrects process errors\.Here, the agent must classify voice clips into 12 commands\. A wrong file path makes every test clip load as silent audio, so every prediction is classified as the “unknown” command even though the trajectory indicates no errors\. To address this, AdaPRA checks the predictions file under the*Submission Integrity*rubric, identifies the cause, and the agent fixes it\.Data science agents can consistently complete end\-to\-end machine learning \(ML\) workflows, yet their scores lag on benchmarks such as DSGym\([Nie et al\., 2026](https://arxiv.org/html/2610.33245#bib.bib10)\)\. Much of the gap comes from*process errors*, mistakes in intermediate steps that lead to worse outcomes\. The most common is*simplicity bias*[Nie et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib10): agents stop at the first valid pipeline instead of exploring stronger models, preprocessing, or features\. Others include unsound validation, mismatched training objectives, and leaked validation sets\. Many are silent: the pipeline produces a valid submission, and the trajectory shows no sign of the error\. Agents also make poor use of repeated attempts: harnesses like DSGym start each attempt from scratch, so every attempt repeats exploratory data analysis \(EDA\), baselines, and strategies that already failed\. Competitive submissions therefore require a harness that accumulates progress across attempts and*process supervision*[Lightman et al\. \(2024\)](https://arxiv.org/html/2610.33245#bib.bib7)that corrects process errors, including those that leave no trace in the trajectory\.

We proposeAdaPRA, a method with four components: a persistent harness; a*process reward agent*\(PRA\) that supervises the agent; automatic seed rubric discovery from past agent runs; and task\-specific rubric adaptation\. The harness keeps the agent’s workspace across repeated attempts on the same task, so exploratory insights, and trained models from one attempt carry into the next\. The PRA supervises the agent during each attempt, judging its trajectory and workspace against rubrics using tools and code execution[Qiu et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib12);[Peng et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib11);[Raghavendra et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib13)\.

We designAdaPRAaround five challenges of supervising a long\-running data science agent\. First, silent errors leave no trace in the trajectory, so an LLM judge that reads only the trajectory misses them;AdaPRAinstead runs checks on the agent’s code, outputs, and traces\. In Figure[1](https://arxiv.org/html/2610.33245#S1.F1), the agent submits “unknown” for all 158,538 test clips;AdaPRAfinds identical predicted probabilities for every clip, confirms that the code builds nonexistent paths, and names the fix\. Second, the evidence spans hundreds of turns and a workspace of code, models, and data, too much to push into the context window, soAdaPRA*pulls*context on demand\. Third, feedback after an attempt ends comes too late to steer it, so the PRA reviews at a fixed cadence*during*the attempt, similar to[Gandhi et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib1); at any stage, however, many rubrics are irrelevant or already passing, so our PRA reports only the relevant failing ones to avoid flooding the agent’s context\. Fourth, rubrics must target the errors agents actually make, many visible only in the workspace, soAdaPRAdiscovers seed rubrics from past agent trajectories and workspace artifacts, extending the rubric discovery procedure of[Zhang et al\. \(2026b\)](https://arxiv.org/html/2610.33245#bib.bib23)\. These errors recur across tasks, and we show thatrubrics learned from easier tasks generalize to harder tasks\. Finally, each task has pitfalls that general rubrics cannot anticipate, soAdaPRAadapts its rubrics to each task as attempts proceed[Li et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib6);[Shao et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib14), which further improves performance\.

Replacing the DSGym harness withAdaPRA, using Claude Sonnet\-4\.6 for the PRA, improves the mean Kaggle percentile of an 80B Qwen3\-Coder\-Next agent by 24\.5 points on DSPredict\-Easy and by 14\.8 points on DSPredict\-Hard, whose tasks span vision, text, speech, and molecular prediction\. These results match or exceed GPT\-5\.1 with high reasoning across all metrics\. Ablations show that the persistent harness, code execution, and rubric adaptation each contribute to these gains\.

In summary,AdaPRAmakes three contributions: \(1\) a persistent harness for data science agents that carries progress across attempts, including EDA findings, baselines, and failed strategies, so that each attempt spends its fixed budget on new experiments; \(2\) a PRA that audits the agent’s trajectory and workspace by executing checks and adapts its rubrics to each task; and \(3\) a rubric discovery procedure that contrasts whole agent attempts, using both their trajectories and workspace artifacts, to yield seed rubrics that transfer from easier to harder tasks and to a second base model\.

## 2Related Work

#### Agents for data science\.

Agents that automate ML workflows are evaluated on DSGym[Nie et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib10)and agentic data\-analysis suites\([Qiu et al\., 2026](https://arxiv.org/html/2610.33245#bib.bib12);[Kulikov et al\., 2026](https://arxiv.org/html/2610.33245#bib.bib4),inter alia\)\. ML engineering agents such as AIDE[Jiang et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib3)and ML\-Master[Liu et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib8)use an external scaffold that keeps a tree of candidate solutions, picks one to refine, and gives the model a short, curated context at each step\.AdaPRAinstead leaves the search to the coding agent: its harness keeps the workspace across attempts, and the agent decides what to reuse, using files as memory[Xu et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib20);[Zhou et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib24)\. The two designs could be combined, with such a scaffold running over a persistent agent\.

#### Rubric adaptation\.

Prior work has explored rubrics, checklists, and natural\-language principles as reward specifications that are more flexible than scalar preference labels[Wang et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib17);[Gunjal et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib2);[Viswanathan et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib16);[Yu et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib21)\(compared in detail in Appendix[G](https://arxiv.org/html/2610.33245#A7)\)\. Other work adapts rubrics over time: RubricEM[Li et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib6)adds and prunes criteria in an evolving buffer through a meta\-RL training loop, turning past attempts into reusable experience, and[Shao et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib14)evolve rubrics during training to mitigate reward hacking\. These papers adapt rubrics to shape a training reward and apply them with a standard LLM judge that has no tools or code execution\.AdaPRAinstead adapts rubrics while it supervises the agent on the task at hand, without gold feedback such as Kaggle scores: the PRA checks rubrics with tools and code, and a per\-task store turns pitfalls that a general rubric set cannot cover into explicit criteria for later reviews of the same task\.

#### Process reward agents\.

Process reward models \(PRMs\) score intermediate steps rather than only final outcomes[Lightman et al\. \(2024\)](https://arxiv.org/html/2610.33245#bib.bib7);[Gandhi et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib1)\.*Process reward agents*\(PRAs\) extend PRMs to verify actively before judging, by executing code, retrieving evidence, or exploring a workspace, in math[Zhang et al\. \(2026a\)](https://arxiv.org/html/2610.33245#bib.bib22), software engineering[Raghavendra et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib13), GUIs[Xiong et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib19), medical reasoning[Sohn et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib15), data analysis[Qiu et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib12), and general reward modeling[Peng et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib11)\. Code execution makes verification more grounded, but existing PRAs rely on a static rubric set, whereasAdaPRAaudits both the trajectory and the workspace of a long\-running agent and adapts its rubrics to each task\. Table[3](https://arxiv.org/html/2610.33245#A1.T3)\(Appendix[A](https://arxiv.org/html/2610.33245#A1)\) summarizes the design coverage of related systems\.

#### Rubric discovery\.

Rubrics can be discovered automatically from past agent behavior\. RubricEM discovers rubrics through policy decomposition across training episodes[Li et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib6), and[Zhang et al\. \(2026b\)](https://arxiv.org/html/2610.33245#bib.bib23)refine per\-prompt rubrics for RL post\-training by contrasting the two highest\-scoring candidate responses to a prompt\. Both derive rubrics from responses or trajectories alone, whereas we also mine the workspace artifacts each attempt leaves behind \(§[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)\) and show that the resulting rubrics transfer from easier to harder tasks\.

## 3Method

![Refer to caption](https://arxiv.org/html/2610.33245v1/figures/AdaPRA_method_v3_j.png)Figure 2:AdaPRAoverview\.The agent works in a workspace that persists across attempts\. During each attempt, the PRA reads the trajectory and workspace, runs check scripts in its own workspace, injects feedback on failing rubrics into the agent’s trajectory, and adds, updates, or retires rubrics in the per\-task store\.AdaPRA\(Figure[2](https://arxiv.org/html/2610.33245#S3.F2)\) combines a persistent harness \(§[3\.2](https://arxiv.org/html/2610.33245#S3.SS2)\) with a PRA that checks the agent against rubrics discovered offline \(§[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)\), grounds its judgments in code \(§[3\.4](https://arxiv.org/html/2610.33245#S3.SS4)\), and adapts the rubrics to each task \(§[3\.5](https://arxiv.org/html/2610.33245#S3.SS5)\)\.

### 3\.1Problem Setup

Letxxbe a predictive data science task represented as a set of files and a natural\-language prompt\. In each attempt, the base agent produces a trajectoryτ=\(a1,o1,…,aT,oT\)\\tau=\(a\_\{1\},o\_\{1\},\\ldots,a\_\{T\},o\_\{T\}\)of actionsata\_\{t\}\(writing code, training models, creating submissions\) and observationsoto\_\{t\}\(logs, scores, errors\), and operates in a workspaceWWon the file system\. The agent makesNNattempts per task, each ending with a submission file, and every condition we compare, including the baselines, makes the sameNNattempts \(§[4](https://arxiv.org/html/2610.33245#S4)\)\. We score every attempt on Kaggle only to compute evaluation metrics, and neither the agent nor the PRA ever sees these scores, test labels, or any other external feedback between attempts\.

Process errors such as simplicity bias and leaky cross\-validation often still yield a valid submission \(Table[10](https://arxiv.org/html/2610.33245#A10.T10)\), so we supervise the process withmmrubricsℛ=\{ρ1,…,ρm\}\\mathcal\{R\}=\\\{\\rho\_\{1\},\\ldots,\\rho\_\{m\}\\\}, natural\-language criteria such as*“compare the first valid baseline against at least one stronger model family\.”*Everyccbase\-agent turns and when the base agent tries to finish, the PRA scores the partial trajectoryτ:t=\(a1,o1,…,at,ot\)\\tau\_\{:t\}=\(a\_\{1\},o\_\{1\},\\ldots,a\_\{t\},o\_\{t\}\)up to the current turnttand the current workspace against each rubricρj\\rho\_\{j\},

S\(x,W,τ:t,ρj\)=\(pj,ij,ej,a^j\),S\(x,W,\\tau\_\{:t\},\\rho\_\{j\}\)=\(p\_\{j\},i\_\{j\},e\_\{j\},\\hat\{a\}\_\{j\}\),with probability of satisfactionpj∈\[0,1\]p\_\{j\}\\in\[0,1\], importanceij∈\[0,1\]i\_\{j\}\\in\[0,1\], rationaleeje\_\{j\}, and suggested next actiona^j\\hat\{a\}\_\{j\}\. The probability of satisfactionpjp\_\{j\}captures the confidence that the agent adhered to the rubric guidelines\. The importanceiji\_\{j\}captures the relevance of the rubric to taskxxand to the current stagettof the attempt, since one rubric set serves many kinds of task and some rubrics apply only at later stages \(e\.g\., a rubric requiring a prediction file applies only once a model has been trained\)\.

We show the agent only the relevant failing rubrics, those withij≥0\.5i\_\{j\}\\geq 0\.5andpj<0\.6p\_\{j\}<0\.6\(Table[5](https://arxiv.org/html/2610.33245#A3.T5)\), i\.e\., rubrics with importance at least 0\.5 and a greater than 40% chance of having failed\.111We show only failures because in an earlier version that reported every rubric, most rubrics passed in nearly every report and the agent’s next action was similar whether or not a report contained a failure \(Appendix[R](https://arxiv.org/html/2610.33245#A18)\)\.The PRA compiles these rubrics into a natural\-language report and injects it directly into the message stream of the agent \(Appendix[E\.2](https://arxiv.org/html/2610.33245#A5.SS2)\)\.222During development, we found this more effective than writing the report to a markdown file that the agent could choose to open and inspect, but sometimes did not despite instructions in its system prompt\.

### 3\.2Agent Harness with Memory

In our harness, the agent acts through a terminal and a file editor and is built on OpenHands[Wang et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib18)\. The DSGym harness[Nie et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib10)resembles a REPL and resets the workspace between attempts, so each attempt rebuilds data loading, validation, and a baseline from scratch\. Our harness keeps one workspace per task across allNNattempts: exploratory notes, features, trained models, validation results, and submissions from attemptkkremain available in attemptk\+1k\{\+\}1, and only the model context resets\. This lets later attempts avoid or fix errors from earlier ones and spend their budget on directions earlier ones left open\. Each later attempt also receives the agent’s own summaries of all earlier attempts, so it can explore new directions with awareness of what has already been tried\. In supervised conditions, each summary is followed by the feedback from the final review of that attempt \(Appendix[E\.1](https://arxiv.org/html/2610.33245#A5.SS1)\)\. Everything carried across attempts is produced by the agent or the PRA, and the only scores it contains are the agent’s own validation scores\. Turn, wall\-clock, and per\-command limits bound each attempt \(Appendix[C](https://arxiv.org/html/2610.33245#A3)\)\.

### 3\.3Rubric Discovery

Hand\-writing rubrics requires expert effort that becomes infeasible across many tasks and trajectories of hundreds of turns\. We discover rubrics offline from past harness\-only runs of the base agent, using the leaderboard percentile of each attempt to contrast stronger and weaker attempts \(Figure[3](https://arxiv.org/html/2610.33245#A9.F3); prompts in Appendix[E\.3](https://arxiv.org/html/2610.33245#A5.SS3)\)\. Like RubricEM[Li et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib6), we derive rubrics from agent trajectories, but we also mine the workspace each attempt leaves behind \(code, notes, and submissions\)\. From[Zhang et al\. \(2026b\)](https://arxiv.org/html/2610.33245#bib.bib23), we borrow contrastive discovery and the principle that rubrics be mutually exclusive and collectively exhaustive \(MECE\)\. Instage one, an LLM judge \(Claude Sonnet 4\.6\) extracts generalizable behaviors that plausibly drove each attempt’s outcome, either*good behaviors*or*issues*, each with quoted evidence\. It is told whether the attempt ranks in the top or bottom quarter of the attempts on the same task, so each attempt is judged relative to its siblings and task difficulty is held fixed \(Appendix[E\.3](https://arxiv.org/html/2610.33245#A5.SS3)\)\. Instage two, a single LLM call clusters the behaviors into a MECE rubric set, with good behaviors and issues as positive and negative evidence for the same rubric\. Stage one extracts both kinds from every attempt, but to keep the behaviors that explain the score gap, we cluster only the good behaviors of top attempts and the issues of bottom attempts \(519 of 906; Appendix[E\.3](https://arxiv.org/html/2610.33245#A5.SS3)\)\.

#### Evidence source\.

Some behaviors are visible only in the files left behind \(e\.g\., code and markdown notes\) and others only in how the agent worked, as recorded in the trajectory,τ\\tau\. The judge therefore seesτ\\tau, a snapshot of the workspaceWW, and the agent’s end\-of\-attempt summary, and tags each rubric by the evidence it needs\.Artifact\-groundedrubrics can be checked fromWWalone \(“the submission has the correct columns and row count”\)\.Trajectory\-groundedrubrics requireτ\\tau\(“kept retrying the same failing command”\)\. The tags measure what each view contributes to discovery\.

#### Discovered set\.

On harness\-only runs of DSPredict\-Easy \(38 tasks, 5 attempts each\), stage one mines 906 behaviors, 564 issues and 342 good behaviors\. Issues are mostly trajectory\-grounded \(344 of 564\) and good behaviors mostly artifact\-grounded \(235 of 342\), so mining only the trajectory would under\-represent good behaviors, and a judge that sees only the workspace would miss most issues\. Stage two yields 17 rubrics covering validation, model exploration and selection, preprocessing, submission integrity, and efficiency, including criteria with no artifact counterpart, such as pivoting to a qualitatively different approach once repeated experiments stop improving\. Together with two cross\-attempt rubrics \(Appendix[J](https://arxiv.org/html/2610.33245#A10)\), these form the 19 seed rubrics \(Table[10](https://arxiv.org/html/2610.33245#A10.T10)\) used in every configuration and applied unchanged to DSPredict\-Hard\.

### 3\.4Process Reward Agent

Judging many of these errors requires reviewing the modeling code itself: whether a stronger model family was trained and compared, whether the cross\-validation leaks information, whether engineered features reach the final model, and whether the submission comes from the pipeline the agent reports\. The trajectory records the agent’s claims, which can mislead a judge that takes them at face value, so the PRA is itself an agent, with its own terminal and scratch space that checks the claims against the workspaceWW\(instructions and tools in Appendix[E\.2](https://arxiv.org/html/2610.33245#A5.SS2)\)\. For each rubric, the PRA can write and execute its own code in its own workspace against the base agent’s files \(e\.g\. recompute a cross\-validation score from out\-of\-fold predictions\) and groundpjp\_\{j\}in the result\. Rubrics that need no computation, such as the trajectory\-grounded “kept retrying the same failing command” \(§[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)\), it scores by reading the trajectory or files\. The base agent’s workspace is read\-only to the PRA, so supervision cannot alter the base agent’s work\.

The fixed cadence delivers feedback while a course correction is still possible, similar to[Gandhi et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib1)\(Appendix[N](https://arxiv.org/html/2610.33245#A14)variescc\), and the finish review acts as a final gate\. Trajectories reach hundreds of turns, so instead of being injected into the PRA’s context,τ:t\\tau\_\{:t\}and the transcript of each earlier attempt are written as turn\-annotated text files to the PRA’s workspace, which the PRA opens and searches with its tools \(file names in Appendix[E\.2](https://arxiv.org/html/2610.33245#A5.SS2)\)\. The PRA reads them selectively, and initial experiments showed that this on\-demand access works better than appending each new batch of agent turns to the PRA’s context \(Appendix[B](https://arxiv.org/html/2610.33245#A2)\)\. The base agent is paused while the PRA runs, and this wall\-clock time does not count against its budget\. Failing rubrics \(§[3\.1](https://arxiv.org/html/2610.33245#S3.SS1)\) are returned to the agent ordered by importance, with their suggested fixes, to keep feedback short and actionable \(Appendix[M](https://arxiv.org/html/2610.33245#A13)shows examples\)\. If the finish review fails, the agent gets one final refinement round, which bounds the cost of the gate\.

### 3\.5Adaptive Rubrics

The seed rubrics from §[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)capture failures that recur across tasks, but each task has pitfalls of its own\. The PRA can therefore add, update, and retire rubrics in a per\-task store \(similar to RubricEM[Li et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib6)\), initialized with the seed set and kept across the task’s attempts, so a failure found in one attempt becomes an explicit criterion in later reviews\. The PRA edits the store with three tools\.*Add*creates a rubric only after the PRA has run and recorded a check for it in the same review,*update*rewrites a rubric’s specification, and*retire*deactivates a rubric and records the reason\. Stores are isolated across tasks, so that results do not depend on the order in which tasks are run\. Added rubrics tend to be task specific\. For example, on House Prices the PRA added a rubric it named*prediction underdispersion*, which checks whether out\-of\-fold predictions are much less spread out than the target, a sign that the model pulls extreme prices toward the mean\.

## 4Experiments

#### Benchmarks\.

We evaluate on DSGym’s[Nie et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib10)DSPredict\-Easy, 38 mostly tabular Kaggle competitions from the Playground Series, and DSPredict\-Hard, 54 competitions spanning tabular, vision, text, audio, and molecular data\. Because the seed rubrics are discovered on DSPredict\-Easy \(§[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)\), it serves as the development benchmark and DSPredict\-Hard as a held\-out test of transfer to harder tasks\.

#### Models and budget\.

The base agent is Qwen3\-Coder\-Next, an 80B open\-weights coding model, and the PRA uses Claude Sonnet 4\.6\. Every condition makesN=5N=5attempts per task, and all conditions built on our harness share the same per\-attempt limits \(Appendix[C](https://arxiv.org/html/2610.33245#A3)\)\. DSPredict\-Hard attempts receive a larger budget and GPU access because its tasks involve larger data and neural models\.

#### Baselines and Ablations\.

On each benchmark we compare six conditions\. Two baselines are without process supervision: the DSGym harness[Nie et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib10), which resets the workspace between attempts, and our persistent harness withoutAdaPRAsupervision \(*harness only*\), which isolates the effect of our persistent harness\. To compare with competitive PRMs, we replace the PRA with DataPRM[Qiu et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib12), a 4B generative process reward model that runs code to verify each agent step and scores the step 1 \(correct\), 0\.5 \(minor issues\), or 0 \(fatal error\); the base agent, harness, and feedback channel are those ofAdaPRA\(Appendix[F](https://arxiv.org/html/2610.33245#A6)\)\. Both supervisors execute code and score the agent, but DataPRM judges each step’s correctness without rubrics, whileAdaPRAchecks the trajectory and workspace against seed and task\-adapted rubrics that also cover errors spanning many steps, such as stopping at the first valid pipeline\. The remaining three conditions share our harness and the seed rubrics:AdaPRA;*−\-adaptive rubrics*, which keeps the seed set fixed, so on DSPredict\-Hard it measures how well the seed rubrics alone transfer; and*−\-PRA*, which keeps adaptation but removes the PRA’s terminal, so it can read files but not run code \(§[3\.4](https://arxiv.org/html/2610.33245#S3.SS4), Appendix[E\.2](https://arxiv.org/html/2610.33245#A5.SS2)\)\. As external context we also report the single\-attempt DSGym results of Qwen3\-Coder\-480B and GPT\-5\.1 \(high\)[Nie et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib10)\.

#### Additional ablations\.

To isolate the effect of supervision within an attempt, we run two further ablations on DSPredict\-Easy with a single attempt per task: the PRA cadence,c∈\{5,10,20\}c\\in\\\{5,10,20\\\}agent turns \(Appendix[N](https://arxiv.org/html/2610.33245#A14)\), and DeepSeek\-V3\.2 as the base agent in place of Qwen3\-Coder\-Next \(Appendix[O](https://arxiv.org/html/2610.33245#A15)\)\.

#### Metrics\.

We report four task\-level rates:Pct\., the mean leaderboard percentile \(private leaderboard where one exists, else public\);Valid, the share of tasks with a submission Kaggle accepts and scores;\>Median, the share of tasks scoring above the leaderboard median; andMedal, the share of tasks reaching a Kaggle medal threshold\. Averaging only over tasks with a valid submission would favor a condition that fails to submit on hard tasks, because it drops its lowest scores; all rates therefore divide by every task in the benchmark and count an invalid or missing submission as a zero percentile\. Ideally, a deployed system would identify the best submission without test scores, so we evaluate four selection policies \(Appendix[H](https://arxiv.org/html/2610.33245#A8)\): the first attempt, the last attempt, the mean over attempts \(the expected result of choosing one at random\), and the best attempt, an oracle upper bound\. The main tables report the best and mean attempts\. We test each difference in mean percentile for significance with a one\-sided paired permutation test over tasks, which randomly swaps the two conditions’ scores within each task \(Appendix[I](https://arxiv.org/html/2610.33245#A9)\)\.

## 5Results

Table 1:DSPredict\-Easy:Our harness \(Ours\), with and without process supervision, compared with the DSGym harness\.*Best attempt*is the maximum over a task’s five sequential attempts, and per\-attempt averages each task’s scored attempts; all rates use a fixed denominator of 38 tasks \(190 attempts\)\. Qwen3\-Coder\-480B and GPT\-5\.1 \(high\) are single\-attempt DSGym runs[Nie et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib10), so they have no best\-attempt results; their percentile is rescaled to our fixed denominator \(multiplied by the valid rate\), their other rates are as reported, and DSGym reports no Easy medal rates\.Table 2:DSPredict\-Hard:As Table[1](https://arxiv.org/html/2610.33245#S5.T1), on the 54 competitions of DSPredict\-Hard \(fixed denominator of 54 tasks, 270 attempts for per\-attempt\)\. DSGym reports medal rates but not percentiles on DSPredict\-Hard, so the*Pct\.*of its single\-attempt runs is also –, and their rates are shown as published### 5\.1DSPredict\-Easy

AdaPRAraises the mean percentile of Qwen3\-Coder\-Next from 44\.5 in the DSGym harness to 69\.0 and the above\-median rate from 50\.0% to 86\.8% \(Table[1](https://arxiv.org/html/2610.33245#S5.T1)\)\. Every task receives a valid submission, as does every individual attempt\. Over our harness alone \(*harness only*\),AdaPRAadds 8\.2 percentage points and 15\.7 above\-median points; the percentile gain is significant \(p<0\.001p<0\.001, Appendix[I](https://arxiv.org/html/2610.33245#A9)\)\. The gain holds per attempt \(61\.0 vs\. 51\.3 mean percentile\), so supervision improves typical attempts as well as the best one\. A singleAdaPRAattempt on average matches a single GPT\-5\.1 \(high\) attempt in the DSGym harness \(61\.0 vs\. 60\.4 mean percentile\)\. Because the seed rubrics were discovered on DSPredict\-Easy, DSPredict\-Hard is the cleaner test ofAdaPRA\.

### 5\.2DSPredict\-Hard

On DSPredict\-Hard,AdaPRAraises the mean percentile from 21\.3 to 36\.1, the valid rate from 68\.5% to 90\.7%, and the above\-median rate from 11\.1% to 33\.3% \(Table[2](https://arxiv.org/html/2610.33245#S5.T2)\)\. Over our harness alone, it adds 10\.6 percentage points \(p<0\.001p<0\.001\) and 14\.8 above\-median points\.AdaPRA’s best attempt exceeds a single GPT\-5\.1 \(high\) attempt on valid, above\-median, and medal rates\. Per attempt,AdaPRAexceeds GPT\-5\.1 in valid and above\-median rates \(86\.3 vs\. 85\.7 and 24\.1 vs\. 14\.3\) but trails it slightly in medal rate \(4\.1 vs\. 4\.8\)\.

### 5\.3Ablations

#### Harness\.

Our harness alone \(persistent workspace and terminal\-based tools\) lifts the base agent on both benchmarks, raising the mean percentile from 44\.5 to 60\.8 and the valid rate from 76\.3% to 100% on DSPredict\-Easy, and from 21\.3 to 25\.5 and 68\.5% to 81\.5% on DSPredict\-Hard\. The two components split the gain differently: the harness contributes 16\.3 percentage points and supervision 8\.2 on DSPredict\-Easy, versus 4\.2 and 10\.6 on DSPredict\-Hard\. Our harness is enough to produce valid and reasonable pipelines on easier tasks, while harder tasks need supervision\.

#### Adaptive rubrics\.

Freezing the seed rubrics lowers the mean percentile by 1\.7 points on DSPredict\-Easy and by 4\.3 points on DSPredict\-Hard, where it also cuts the above\-median rate from 33\.3% to 22\.2%\. The Hard gap is significant \(p=0\.024p=0\.024; 32 wins, 13 losses\)\. §[5\.4](https://arxiv.org/html/2610.33245#S5.SS4)shows why adaptation matters more on DSPredict\-Hard: its tasks have more domain\-specific pitfalls that the seed set cannot anticipate\. The PRA adapts almost entirely by adding rubrics \(177 additions, one update, and no retirements across the four adaptive runs\), mostly in the first two attempts \(Figures[4](https://arxiv.org/html/2610.33245#A12.F4)and[5](https://arxiv.org/html/2610.33245#A12.F5)in Appendix[L](https://arxiv.org/html/2610.33245#A12)\)\. Code execution changes what it adds: prediction\-sanity rubrics, which compute statistics of the agent’s predictions such as their dispersion relative to the target, make up 35% \(DSPredict\-Easy\) and 19% \(DSPredict\-Hard\) of additions with execution and 3% and 7% without it \(Table[18](https://arxiv.org/html/2610.33245#A12.T18)\)\.

#### PRA\.

Scoring rubrics by reading alone lowers the mean percentile from 69\.0 to 63\.0 on DSPredict\-Easy and from 36\.1 to 31\.8 on DSPredict\-Hard, along with the Easy medal rate \(7\.9% to 2\.6%\) and the Hard above\-median rate \(33\.3% to 25\.9%\)\. The gap is significant on DSPredict\-Easy \(p=0\.012p=0\.012\) but not on DSPredict\-Hard \(p=0\.077p=0\.077; 29 wins, 18 losses\)\. Without code execution \(−\-PRA\), supervision still improves on the harness alone on both benchmarks \(63\.0 vs\. 60\.8 and 31\.8 vs\. 25\.5\), so rubric feedback helps even without execution\. It gets a higher Hard medal rate \(9\.3% vs\. 5\.6%, five medals vs\. three\) despite a lower mean percentile, which we attribute to the noisiness of the metric: a medal depends on whether a submission’s rank crosses a fixed leaderboard cutoff, so small score differences can flip it, especially on dense leaderboards\.

AdaPRAhas the highest mean percentile on both benchmarks and the highest above\-median rate on DSPredict\-Hard \(33\.3% vs\. 22\.2% and 25\.9%; all three tie at 86\.8% on DSPredict\-Easy\), so code execution and rubric adaptation are complementary\.

#### Additional analyses\.

Cadence\(Appendix[N](https://arxiv.org/html/2610.33245#A14)\): with a single attempt per task, every cadence we test improves on the harness alone \(48\.4 mean percentile\), and invoking the PRA every 10 turns works best \(56\.4\)\. Invoking it more often \(c=5c\{=\}5, 52\.7\) or less often \(c=20c\{=\}20, 49\.9\) gives smaller gains, consistent with a trade\-off between timely feedback and interrupting the agent\. We therefore usec=10c\{=\}10on DSPredict\-Easy\. On DSPredict\-Hard we usec=20c\{=\}20to bound cost: Hard attempts take 1\.6 times as many agent turns \(Table[21](https://arxiv.org/html/2610.33245#A16.T21)\), and the PRA already accounts for 75% of the cost of a Hard run \(Table[22](https://arxiv.org/html/2610.33245#A17.T22)\)\.Base agent\(Appendix[O](https://arxiv.org/html/2610.33245#A15)\): with DeepSeek\-V3\.2 as the base agent and one attempt per task,AdaPRAraises the mean percentile from 36\.9 to 44\.7 and the above\-median rate from 23\.7% to 47\.4% over the harness alone\. It wins on 25 tasks and loses on 11 \(p=0\.005p=0\.005\), so rubrics discovered from Qwen3\-Coder\-Next’s runs also help a second base model\.

### 5\.4Easy\-to\-Hard Generalization

The seed rubrics are discovered only on DSPredict\-Easy, whose tasks are mostly tabular, yet they improve DSPredict\-Hard, whose tasks span vision, text, audio, time series, and molecular data\. With the seed rubrics frozen, supervision adds 6\.3 percentage points over the harness alone on DSPredict\-Hard \(p=0\.007p=0\.007\), close to its 6\.5\-point gain on DSPredict\-Easy \(p=0\.017p=0\.017\)\.

The same process failures occur on both benchmarks in the frozen\-rubric runs, which score the same 19 rubrics throughout \(Appendix[K](https://arxiv.org/html/2610.33245#A11)\)\. Every seed rubric fails in at least 24% of DSPredict\-Hard attempts, so none is specific to easy tasks\. For each rubric we compute two statistics per benchmark: its mean importance over all PRA reviews and its failure rate, the share of attempts in which it fails at least once\. Ranking the 19 rubrics by either statistic gives nearly the same order on both benchmarks \(Spearmanρ=0\.91\\rho=0\.91for importance,ρ=0\.66\\rho=0\.66withp=0\.002p=0\.002for failure rate\)\. The most frequent failures on both benchmarks are the two simplicity\-bias rubrics, too little model diversity and ensembling and repeating near\-identical experiments without pivoting \(68–84% of attempts\)\. Harder tasks add pipeline failures that the seed set already covers: invalid cross\-validation rises from 32% to 57% of attempts and unverified submissions from 33% to 52%\. Simplicity bias and pipeline errors are properties of the agent, which persist across domains, so rubrics that target them transfer\. Table[17](https://arxiv.org/html/2610.33245#A11.T17)in Appendix[K](https://arxiv.org/html/2610.33245#A11)shows seven such cases, each a failure on a non\-tabular DSPredict\-Hard task that a DSPredict\-Easy rubric flagged and the agent fixed within the same attempt\.

Adaptation covers what the seed set misses\. The PRA adds 0\.8 rubrics per task on DSPredict\-Easy and 1\.2 on DSPredict\-Hard, and adds at least one on 79% of Hard tasks versus 51% of Easy tasks \(Figure[6](https://arxiv.org/html/2610.33245#A12.F6)\)\. Hard additions encode domain knowledge absent from tabular tasks, such as non\-negative time\-to\-failure predictions in LANL Earthquake Prediction and excluding quantum\-mechanical sub\-components of the target from the features in CHAMPS Scalar Coupling\. The seed set transfers the domain\-general failures and adaptation supplies the domain\-specific ones, consistent with the larger adaptation gain on DSPredict\-Hard\.

## 6Conclusion

We proposeAdaPRA, which combines a harness whose persistent workspace accumulates knowledge across attempts on a task, a process reward agent that audits the agent’s trajectory and workspace by executing checks and adapts its rubrics to each task, and offline discovery of seed rubrics from agent trajectories and workspace artifacts\. With Qwen3\-Coder\-Next as the base agent,AdaPRAraises the mean Kaggle percentile from 44\.5 to 69\.0 on DSPredict\-Easy and from 21\.3 to 36\.1 on DSPredict\-Hard, matching or exceeding GPT\-5\.1 \(high\)\. The harness, the process reward agent, and adaptive rubrics each contribute to the gain, and removing any one of them lowers performance on both benchmarks\. Rubrics discovered on mostly tabular tasks \(DSPredict\-Easy\) transfer to vision, text, audio, and molecular tasks \(DSPredict\-Hard\) because the process failures they target come from the agent rather than the domain, and adaptation adds the domain\-specific criteria that a general rubric set does not cover\. Supervision with rubrics mined from Qwen3\-Coder\-Next’s runs also improves a second base model \(DeepSeek\-V3\.2\), andAdaPRAoutperforms DataPRM[Qiu et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib12), a competitive PRM proposed in prior work\.

## 7Future Work

First, training the base agent on dense feedback from the PRA could let it internalize the process criteria and depend less on supervision at test time\. Second, the PRA adapts rubrics without measuring how each affects the outcome and never retired one in our runs; assigning credit to individual rubrics could better inform adaptation\. Third, training smaller open\-weight models to replace the closed\-source PRA would extend[Gandhi et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib1)from non\-agentic critics to agentic PRAs\. Finally, harness evolution like Meta\-Harness[Lee et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib5)could further improve our persistent harness\.

## Acknowledgments

This work was supported by Microsoft, which provided compute and funding\. We thank Emmy Liu, Aakanksha Naik, and Sagnik Mukherjee for their feedback on the project and the draft\.

## Limitations

Supervision cadence\.To bound compute and API cost, we invoke the PRA every 20 turns on DSPredict\-Hard instead of the 10 turns that work best on DSPredict\-Easy; the PRA already accounts for 75% of the cost of a Hard run \(Table[22](https://arxiv.org/html/2610.33245#A17.T22)in Appendix[Q](https://arxiv.org/html/2610.33245#A17)\)\. On DSPredict\-Easy, a cadence of 20 is the weakest setting we test \(Appendix[N](https://arxiv.org/html/2610.33245#A14)\), so the DSPredict\-Hard results may understate the gains of more frequent supervision\.

Benchmark coverage\.We evaluate only on DSGym’s DSPredict\-Easy and DSPredict\-Hard\. Both consist of Kaggle competitions scored on a leaderboard metric, so we do not test whether rubric\-based supervision helps in other data science settings, such as open\-ended analysis or tasks without a predictive objective\.

Number of attempts\.All multi\-attempt results use five attempts per task, and the additional ablations use a single attempt\. We do not study how the gains of supervision change as the number of attempts grows, where selecting among more unsupervised attempts may recover part of the difference\.

Rubric adaptation\.Adaptation almost only adds rubrics \(177 additions, one update, and no retirements; §[5\.3](https://arxiv.org/html/2610.33245#S5.SS3)\), so the rubric store grows over a run\. The PRA does not attribute local validation gains to individual rubrics, so it has little signal for revising or retiring rubrics that do not help\. Better credit assignment could make adaptation more selective\.

Models\.Our main results use a single base agent \(Qwen3\-Coder\-Next\) and a single PRA model\. We test a second base agent, DeepSeek\-V3\.2, only with one attempt per task on DSPredict\-Easy \(Appendix[O](https://arxiv.org/html/2610.33245#A15)\)\.

Reproducibility\.The PRA uses Claude Sonnet 4\.6, a closed model accessed through a commercial API\. Exact reproduction depends on the model remaining available and unchanged, and its retirement or updates could change results\.

## References

- Gandhi et al\. \(2026\)Shubham Gandhi, Yiqing Xie, Atharva Naik, Ruichen Zhu, and Carolyn Rose\. 2026\.Steer, don’t solve: Training small critic models for large code agents\.*arXiv preprint arXiv:2606\.21811*\.
- Gunjal et al\. \(2025\)Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx\. 2025\.Rubrics as rewards: Reinforcement learning beyond verifiable domains\.*arXiv preprint arXiv:2507\.17746*\.
- Jiang et al\. \(2025\)Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu\. 2025\.Aide: Ai\-driven exploration in the space of code\.*arXiv preprint arXiv:2502\.13138*\.
- Kulikov et al\. \(2026\)Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, and Jason Weston\. 2026\.[Autodata: an automatic data scientist to create high quality data](https://facebookresearch.github.io/RAM/blogs/autodata/)\.
- Lee et al\. \(2026\)Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn\. 2026\.Meta\-harness: End\-to\-end optimization of model harnesses\.*arXiv preprint arXiv:2603\.28052*\.
- Li et al\. \(2026\)Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun\-Liang Li, Long T Le, Rujun Han, George Lee, Hanghang Tong, and 1 others\. 2026\.RubricEM: Meta\-rl with rubric\-guided policy decomposition beyond verifiable rewards\.*arXiv preprint arXiv:2605\.10899*\.
- Lightman et al\. \(2024\)Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\. 2024\.Let’s verify step by step\.In*International Conference on Learning Representations*, volume 2024, pages 39578–39601\.
- Liu et al\. \(2025\)Zexi Liu, Yuzhu Cai, Xinyu Zhu, Yujie Zheng, Runkun Chen, Ying Wen, Yanfeng Wang, Siheng Chen, and 1 others\. 2025\.Ml\-master: Towards ai\-for\-ai via integration of exploration and reasoning\.*arXiv preprint arXiv:2506\.16499*\.
- Naik et al\. \(2025\)Atharva Naik, Lawanya Baghel, Dhakshin Govindarajan, Darsh Agrawal, Daniel Fried, and Carolyn Rose\. 2025\.Metalint: Generalizable idiomatic code quality analysis through instruction\-following and easy\-to\-hard generalization\.*arXiv preprint arXiv:2507\.11687*\.
- Nie et al\. \(2026\)Fan Nie, Junlin Wang, Harper Hua, Federico Bianchi, Yongchan Kwon, Zhenting Qi, Owen Queen, Shang Zhu, and James Zou\. 2026\.Dsgym: A holistic framework for evaluating and training data science agents\.*arXiv preprint arXiv:2601\.16344*\.
- Peng et al\. \(2025\)Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, and Juanzi Li\. 2025\.Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Qiu et al\. \(2026\)Zhisong Qiu, Shuofei Qiao, Kewei Xu, Yuqi Zhu, Lun Du, Ningyu Zhang, and Huajun Chen\. 2026\.Rewarding the scientific process: Process\-level reward modeling for agentic data analysis\.*arXiv preprint arXiv:2604\.24198*\.
- Raghavendra et al\. \(2026\)Mohit Raghavendra, Anisha Gunjal, Bing Liu, and Yunzhong He\. 2026\.Agentic rubrics as contextual verifiers for SWE agents\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Shao et al\. \(2025\)Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G Finlayson, David Sontag, and 1 others\. 2025\.Dr tulu: Reinforcement learning with evolving rubrics for deep research\.*arXiv preprint arXiv:2511\.19399*\.
- Sohn et al\. \(2026\)Jiwoong Sohn, Tomasz Sternal, Kenneth Styppa, Torsten Hoefler, and Michael Moor\. 2026\.Process reward agents for steering knowledge\-intensive reasoning\.*arXiv preprint arXiv:2604\.09482*\.
- Viswanathan et al\. \(2026\)Vijay Viswanathan, Yanchao Sun, Xiang Kong, Meng Cao, Graham Neubig, and Sherry Wu\. 2026\.Checklists are better than reward models for aligning language models\.*Advances in Neural Information Processing Systems*, 38:114728–114754\.
- Wang et al\. \(2026\)Xingyao Wang, Valerie Chen, Heng Ji, and Graham Neubig\. 2026\.A rubric\-supervised critic from sparse real\-world outcomes\.*arXiv preprint arXiv:2603\.03800*\.
- Wang et al\. \(2025\)Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, and 1 others\. 2025\.OpenHands: An open platform for ai software developers as generalist agents\.In*International Conference on Learning Representations*, volume 2025, pages 65882–65919\.
- Xiong et al\. \(2025\)Tao Xiong, Xavier Hu, Yurun Chen, Yuhang Liu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, and Shengyu Zhang\. 2025\.Gui\-pra: Process reward agent for gui tasks\.*arXiv preprint arXiv:2509\.23263*\.
- Xu et al\. \(2025\)Xiwei Xu, Robert Mao, Quan Bai, Xuewu Gu, Yechao Li, and Liming Zhu\. 2025\.Everything is context: Agentic file system abstraction for context engineering\.*arXiv preprint arXiv:2512\.05470*\.
- Yu et al\. \(2025\)Zhuohao Yu, Jiali Zeng, Weizheng Gu, Yidong Wang, Jindong Wang, Fandong Meng, Jie Zhou, Yue Zhang, Shikun Zhang, and Wei Ye\. 2025\.Rewardanything: Generalizable principle\-following reward models\.*arXiv preprint arXiv:2506\.03637*\.
- Zhang et al\. \(2026a\)Jiazheng Zhang, Ziche Fu, Zhiheng Xi, Wenqing Jing, Mingxu Chai, Wei He, Guoqiang Zhang, Chenghao Fan, Chenxin An, Wenxiang Chen, Zhicheng Liu, Haojie Pan, Dingwei Zhu, Tao Gui, Qi Zhang, and Xuanjing Huang\. 2026a\.AgentV\-RL: Scaling reward modeling with agentic verifier\.In*Findings of the Association for Computational Linguistics: ACL 2026*\.
- Zhang et al\. \(2026b\)Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin\. 2026b\.Chasing the tail: Effective rubric\-based reward modeling for large language model post\-training\.In*International Conference on Learning Representations*, volume 2026, pages 133430–133457\.
- Zhou et al\. \(2026\)Sizhe Zhou, Sheldon Yu, Hui Wei, Junda Wu, Siru Ouyang, Yizhu Jiao, Shijia Pan, Julian McAuley, Yu Zhang, Tong Yu, and 1 others\. 2026\.Filesystem\-based memory for llm agents: Organization, evolution, and sustainability\.*arXiv preprint arXiv:2607\.26637*\.

## Appendix ADesign Coverage of Related Systems

WorkDomainOperationalizationRubric store*execute vs\. read**fixed vs\. adaptive*AgentV[Zhang et al\. \(2026a\)](https://arxiv.org/html/2610.33245#bib.bib22)math\(✓\)×\\timesRewardAgent[Peng et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib11)general\(✓\)×\\timesAgentic Rubrics[Raghavendra et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib13)SWE\(✓\)×\\timesGUI\-PRA[Xiong et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib19)GUI\(✓\)×\\timesProcess RM[Sohn et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib15)medical\(✓\)×\\timesRubricEM[Li et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib6)deep research×\\times✓AdaPRA\(ours\)data science✓✓

Table 3:Design coverage of related systems\. ✓: the dimension is varied as an explicit design choice\. \(✓\): the dimension is varied only by removing one component in an ablation\.×\\times: fixed or not applicable\.AdaPRAexecutes checks and adapts its rubrics to each task, and we ablate each component in §[5\.3](https://arxiv.org/html/2610.33245#S5.SS3)\.
## Appendix BSupervision Design Choices

Table[4](https://arxiv.org/html/2610.33245#A2.T4)summarizes the two design choices of the supervisor that the main experiments ablate, operationalization \(−\-PRA\) and rubric adaptation \(−\-adaptive rubrics\)\. We also compared two ways of giving the supervisor the agent’s trajectory: appending the agent turns since the previous invocation to its context, or storing the transcripts of the current and earlier attempts as read\-only files that it reads when needed\. The file\-based access performed better in short experiments on DSPredict\-Easy, so all reported runs use it\. In every setting the supervisor can read the agent’s workspace and trajectory but cannot modify them\.

Table 4:Design choices of the supervisor\. The right column is the configuration ofAdaPRA\. The−\-PRA and−\-adaptive rubrics ablations each replace one entry with its alternative\.
## Appendix CHarness Implementation Details

Each task runs in its own container with its own workspace, so tasks execute in parallel without sharing files, and the PRA mounts the agent’s workspace read\-only \(Appendix[B](https://arxiv.org/html/2610.33245#A2)\)\. Table[5](https://arxiv.org/html/2610.33245#A3.T5)lists the run configuration, and the supervised conditions on a benchmark share its base\-agent settings\. Without supervision, attempts of Qwen3\-Coder\-Next on DSPredict\-Easy use a 2 h limit and most finish well before it \(median 23 min\)\. Supervision lengthens attempts, so supervised attempts on DSPredict\-Easy use a 3 h limit\. All conditions on DSPredict\-Hard use a 4 h limit\. The supervised runs also cap the timeout that the agent can set for a single command, and on DSPredict\-Hard the system prompt states this cap \(Appendix[E\.1](https://arxiv.org/html/2610.33245#A5.SS1)\)\. The PRA settings apply toAdaPRAand both ablations, which differ only in the PRA’s tools \(§[3\.4](https://arxiv.org/html/2610.33245#S3.SS4), §[3\.5](https://arxiv.org/html/2610.33245#S3.SS5)\)\. The prompts are reproduced in Appendix[E](https://arxiv.org/html/2610.33245#A5)\.

#### Model inference settings\.

The base agent and the PRA run inside the task container on the OpenHands SDK 1\.15\.0[Wang et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib18)\(Python 3\.12\) and call their models through LiteLLM 1\.80\.10\. Qwen3\-Coder\-Next samples at temperature 0 with at most 16,000 output tokens per call, and it emits tool calls in the SDK’s prompt\-based format rather than through native function calling\. The SDK treats DeepSeek\-V3\.2 as a reasoning model, so it requests high reasoning effort with at most 8,000 output tokens per call and sends no temperature, which leaves the provider default in place\. For both base agents, tool outputs are truncated to 30,000 characters, the condenser that summarizes older events uses the base agent’s model and settings, and a failed model call is retried up to 10 times\. The PRA uses native function calling and extended thinking, interleaved between tool calls, with high reasoning effort, a thinking budget of 63,999 tokens, and at most 64,000 output tokens per call; the SDK sends no temperature, so the provider default applies\. Offline rubric discovery \(§[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)\) calls Claude Sonnet 4\.6 directly through LiteLLM with the provider’s default sampling settings and without extended thinking\.

Base agent\(Qwen3\-Coder\-Next, temperature 0\)PRA\(Claude Sonnet 4\.6\)ParameterEasyHardParameterEasyHardAttemptsNN55Cadencecc\(agent turns\)1020Max turns per attempt200300Review on finishyesyesWall\-clock limit per attempt3 h4 hMax turns per invocation100100Silent\-command timeout10 min10 minTime limit per invocation1 h1 hMax single\-command timeout30 min2 hImportance threshold \(ij≥i\_\{j\}\\geq\)0\.50\.5Context condensation threshold \(events\)8080Satisfaction threshold \(pj<p\_\{j\}<\)0\.60\.6GPUnoyesFinish refinement rounds11Trajectory accesson demandon demandSeed rubrics1919Table 5:Run configuration on DSPredict\-Easy and DSPredict\-Hard\. The silent\-command timeout stops a command that produces no output for that long\. Time spent in the PRA does not count against the agent’s wall\-clock limit\. The larger per\-attempt budget on DSPredict\-Hard matches the DSGym reference runs on that benchmark\. Hard attempts are longer, so the PRA is invoked less often to bound its cost \(§[5\.3](https://arxiv.org/html/2610.33245#S5.SS3)\)\.

## Appendix DComputational Environment

All runs execute on a Slurm cluster with AMD EPYC 7763 processors and NVIDIA A100 80GB GPUs\. A run is typically allocated 12 CPU cores and 100 GB of memory, and a DSPredict\-Hard run is also allocated two A100 GPUs\. Each run of our harness solves four tasks in parallel, and each task runs in its own Apptainer 1\.5\.2 container built from a common image with Python and standard machine\-learning libraries \(Appendix[C](https://arxiv.org/html/2610.33245#A3)lists the agent software versions\)\. Only the DSPredict\-Hard containers use GPUs, and the DSPredict\-Easy containers run on CPUs alone\. The base agents \(Qwen3\-Coder\-Next and DeepSeek\-V3\.2\) and the PRA \(Claude Sonnet 4\.6\) are accessed through the Amazon Bedrock API\. DataPRM is the only locally served model and runs with vLLM 0\.12\.0 on one A100 GPU\. API costs are reported in Appendix[Q](https://arxiv.org/html/2610.33245#A17)\.

## Appendix EPrompts

This section reproduces the prompts used in the reported runs\. The prompts refer to the PRA as the PRM, and to theNNsequential attempts on a task, of which the best is scored, as a best\-of\-NNrun\. On DSPredict\-Hard the supervised runs reword the two timeout instructions of the base\-agent prompt \(Appendix[E\.1](https://arxiv.org/html/2610.33245#A5.SS1)\)\. Text in angle brackets is filled in at run time, and \[…\] marks elided identifiers of internal checks from an earlier version of the system that do not name rubrics in the seed set\. All prompts are otherwise verbatim, with non\-ASCII punctuation normalized\.

### E\.1Base Agent

The base agent receives the system prompt below in every attempt\. Its first user message contains the competition description and the dataset path\. From the second attempt on, the first message is preceded by the summaries and final PRA feedback of the earlier attempts and by instructions that frame the objective as a maximum over attempts \(§[3\.2](https://arxiv.org/html/2610.33245#S3.SS2)\)\.

Youareadatascienceagentthatsolvesmachinelearningcompetitiontasks\.

Youhaveaccesstoabashterminalandafileeditorinanisolatedcontainerenvironmentpre\-installedwithPython,pandas,scikit\-learn,XGBoost,LightGBM,CatBoost,andotherMLlibraries\.

<ROLE\>

\*Yourprimaryroleistosolvedatascienceandmachinelearningtasksbyexploringdata,buildingmodels,andproducingpredictions\.

\*Bethorough,methodical,andfocusonproducingthebestpossiblepredictions\.

\*YouworkbywritingPythonscriptsandrunningthemviabash\.Eachscriptinvocationisafreshprocess–savestatetodisk,notinmemory\.

</ROLE\>

<EFFICIENCY\>

\*Eachactionyoutakeissomewhatexpensive\.Combinemultipleoperationswherepossible\.

\*Writecompletescriptsratherthanmanysmallbashcommands\.Asinglewell\-structuredPythonscriptisbetterthan10one\-linercommands\.

\*Fordataexploration,writeascriptthatprintssummaries:

\-‘df\.head\(\)‘,‘df\.shape‘,‘df\.dtypes‘,‘df\.describe\(\)‘–never‘print\(df\)‘or‘df\.to\_string\(\)‘

\-Use‘head\-20file\.py‘,‘wc\-lfile\.csv‘forquickfileinspection

\-Use‘grep\-n"pattern"file\.py‘forsearching

\*\*\*NEVERre\-trainamodelyouhavealreadytrainedandscored\.\*\*Whenascript

finishes,immediatelysaveBOTHthefittedmodel\(‘joblib\.dump‘\)ANDits

out\-of\-foldpredictions/testpredictionstodisk\(e\.g\.

‘preds\_<name\>\.parquet‘\)alongwithitsCVscore\.Tocomparemodelsorbuildan

ensemblelater,LOADthosesavedpredictions–donotre\-fit\.

\*\*\*Onemodelfamilyperscript\.\*\*DoNOTputLightGBM\+XGBoost\+CatBoost\+

Ridge\+…inasinglescriptthatre\-fitsallofthemoneveryrun–that

wastesminutesre\-trainingeverythingeachtimeyoutweakonething\.Giveeach

modelitsownsmallscript\(‘train\_lgbm\.py‘,‘train\_xgb\.py‘\),eachofwhich

savesitspredictions\+CV;aseparate‘ensemble\.py‘justLOADSthosesaved

predictionsandblendsthem\.

\*Makescriptsidempotent:atthetop,iftheoutput\(model/preds\)already

existsondisk,loadandreuseitinsteadofrecomputing\.

\*Use‘n\_jobs=\-1‘forsklearnoperations\(cross\_val\_score,GridSearchCV,etc\.\)toparallelizeacrossCPUcores\.

\*Forhyperparametersearch,prefer‘RandomizedSearchCV‘overexhaustive‘GridSearchCV‘\.

</EFFICIENCY\>

<FILE\_SYSTEM\_GUIDELINES\>

\*\*\*IteratebyEDITINGonefileinplace–doNOTspawntrain\_v2\.py,train\_v3\.py,

train\_v10\.py,…\*\*Creatinganewnumberedcopyforeverytweakcluttersthe

workspace,makesitimpossibletotellwhichscriptiscurrent,andwastes

effort\.Keepasinglecanonicalscriptpermodel\(e\.g\.‘train\_lgbm\.py‘\)and

usethefileeditortomodifyit\.Ifyougenuinelyneedtokeepanold

approachforreference,thatistheexception,notthedefault\.

\*Ifaskedtoeditafile,editthefiledirectlyratherthancreatinganewfilewithadifferentfilename\.

\*Forglobalsearch\-and\-replaceoperations,considerusing‘sed‘insteadofopeningfilesmultipletimes\.

</FILE\_SYSTEM\_GUIDELINES\>

<CODE\_QUALITY\>

\*Writeclean,efficientcodewithminimalcomments\.

\*Ifyouareaddingalotofcodetoafunctionorfile,considersplittingintosmallerpieces\.

\*Placeallimportsatthetopofthefile\.

</CODE\_QUALITY\>

<DATA\_SCIENCE\_TASK\>

\#\#Environment

\-Thecompetitiondata\(train/test/sample\_submission\)isunderthedataset

foldergiveninthetaskinstructions–usethatexactpath\(doNOTassume

‘/data/train\.csv‘;exploretheprovidedfoldertofindthefiles\)\.

\-Youmustwriteyourpredictionsto‘<submission\_path\>‘

\-Yourworkspaceisat‘<workspace\_dir\>/‘–useitforscripts,intermediatefiles,models,etc\.

\-Theterminalsessionpersistsenvironmentvariablesandworkingdirectoryacrosscommands

\-Each‘pythonscript\.py‘invocationisafreshprocess–variablesdoNOTpersistbetweenruns

\#\#Workflow

Buildasmall,modular,reusableCODEBASE–notapileofone\-offscripts\.

Treat‘<workspace\_dir\>/‘asaprojectrepoyouextendacrosstheattempt\(and

acrossattempts\)\.Agoodlayouttogrowinto–createthedirsasyouneedthem:

“‘

<workspace\_dir\>/

data\.py\#loadtrain/test,theONEfixedCVsplit\(importeverywhere\)

preprocess\.py\#cleaning/encoding–importable

features\.py\#featureengineering–importable

train\_lgbm\.py\#onemodelfamilyperscript:fit,savepreds\+CV

train\_xgb\.py\#anothermodel–importsdata/preprocess/features

ensemble\.py\#LOADSsavedpredsfrompreds/,blends,writessubmission

README\.md\#projectledger:theapproachestable\(seebelow\)

DATA\_NOTES\.md\#EDAfindings\(columntypes,target,metric\)–read,don’tredo

cv\_scores\.json\#machine\-readablemirrorofthetable:modelname\-\>CVscore

models/\#joblib\-dumpedfittedmodels

preds/\#preds\_<name\>\.parquet\(OOF\+test\)foreverymodel

“‘

Thisisastartingskeleton,notamandate–addscripts/dirsfornewmodeling

approachesasyouexplore,butkeepsharedlogicintheimportablemodulesand

onemodelfamilypertrainingscript\.LaterscriptsandlaterattemptsIMPORT

‘data\.py‘/‘preprocess\.py‘/‘features\.py‘andLOADfrom‘preds/‘ratherthan

re\-implementingorre\-fitting\.

\*\*Maintaina‘README\.md‘ledger\.\*\*KeepaMarkdowntablein‘README\.md‘withONE

rowperapproachyoutry–thisisyourprimaryrecordofwhatworked\.Updateit

everytimeatrainingscriptfinishes,usingaconsistentschema:

\|Approach\|Script\|Features\|CVscore\(\+/\-std\)\|Notes\|

\|—\|—\|—\|—\|—\|

\|LightGBMbaseline\|train\_lgbm\.py\|base\|0\.7191\+/\-0\.012\|5\-fold,seed42\|

\|XGBoost\+target\-enc\|train\_xgb\.py\|\+targetencoding\|0\.7204\+/\-0\.010\|slightlybetter\|

\|LGBM\+XGBblend\|ensemble\.py\|–\|0\.7231\+/\-0\.009\|0\.5/0\.5average\|

BecauseeveryrowusestheSAMEcolumnsandtheSAMEfixedCVscheme,theCV

scoresaredirectlycomparableataglance–whichisexactlywhatyou\(andthe

finalbest\-modelcheck\)needtopickthebestsubmission\.‘cv\_scores\.json‘isthe

machine\-readablemirrorofthistable\.ReadtheREADMEatthestartofanew

attempttoseewhat’salreadybeentriedinsteadofrepeatingit\.

Youmayalsokeepashort"Observations&tips"sectionintheREADMEfor

anythingusefultoyourfutureselforlaterattempts–whathelpedorhurt\(a

featurethatmattered,aleakyoufound,amodelthatoverfit,theCV<\-\>public\-LB

gapyouobserved,apromisingideayouranoutoftimefor\)\.Keepittoterse

bullets,notajournal\.Thesenotesaugmenttheattemptsummariesandhelplater

attemptsstartfromwhatyoulearnedratherthanfromscratch\.

1\.\*\*Explore\*\*:Readthedatafiles,understandthetargetvariableandfeatures\.

2\.\*\*Modularize\*\*:Factorsharedlogicintoimportablemodules–‘data\.py‘

\(load/split\),‘preprocess\.py‘,‘features\.py‘\.Everylaterscriptimportsthese

ratherthanre\-implementingthem\.

3\.\*\*Baseline\*\*:Asimplemodelscriptthatimportsthemodulesabove,trains,

savesitspredictions\+CVscore,andwritesasubmission\.

4\.\*\*Iterate\*\*:AddONEmodelperscript\(‘train\_lgbm\.py‘,‘train\_xgb\.py‘,…\),

eachimportingthesharedmodulesandsavingitspredictions\+CV\.An

‘ensemble\.py‘LOADSthosesavedpredictionsandblendsthem\.Reuseandedit

thesescriptsinplace;don’tforknewnumberedcopies\.\*\*Explorewidely:\*\*

tryadiverserangeofmodelfamilies\(linear,tree\-boosting,andothers\),

feature\-engineeringideas,hyperparametersettings,andensembling/stacking

strategiesratherthantuningasinglemodel\.Thebestsubmissionoftencomes

fromanapproachyouwouldnothaveguessedupfront,socastawidenet–

butkeepeveryexperimentundertheSAMEfixedCVschemesoresultsstay

comparable,andcacheeachmodel’spredictions\+CVsoexplorationnever

meansre\-training\.

5\.\*\*Submit\*\*:Copyyourbestmodel’ssavedpredictionsto‘<submission\_path\>‘

intherequiredformat\.

\#\#ImportantRules

\-ALWAYSensureyoursubmission\.csvhasthecorrectformat\(checksample\_submission\.csvifavailable\)

\-Validateyourpredictionsbeforewritingtosubmission\.csv\(noNaN,correctnumberofrows,correctcolumns\)

\-Uselocalcross\-validationtoestimateperformancebeforesubmitting

\-\*\*FixONEcross\-validationschemeearlyandreuseitforEVERYmodeland

everyattempt\.\*\*Pickascheme\(e\.g\.K\-foldwithafixedseed,stratifiedfor

classification,ortime\-basedifthedataistemporal\)that\(a\)stays

identicalacrossallyourexperimentssoCVscoresaredirectlycomparable,

and\(b\)isrobust–enoughfoldsandhonestsplittingthatitisNOTproneto

overfittingthelocalsplitandislikelytocorrelatewiththecompetition

leaderboard\.DonotswitchCVschemesbetweenmodelsorattempts;ascoreis

onlymeaningfulrelativetootherscomputedthesameway\.

\-Saveintermediatedatatodisk\(parquet,pickle,joblib\)soyoucanreuseitwithoutrecomputing

\-\*\*CRITICAL–submityourBESTmodel,notyourlastexperiment\.\*\*‘<submission\_path\>‘isscoredexactlyasitstandswhenyoufinish;nothingpicksthebestversionforyou\.Ifyoutriedseveralmodels,thefilemaycurrentlyholdpredictionsfromaWORSEexperimentthatoverwroteabetterone\.Beforefinishing,youMUST:\(1\)consultyour‘cv\_scores\.json‘scoreboardtoidentifythemodelwiththebestcross\-validationscoreacrossALLyourexperiments,\(2\)copyTHATmodel’sSAVEDtestpredictionsto‘<submission\_path\>‘\(youcachedthem–doNOTre\-train\),\(3\)confirmthefileondiskcorrespondstothatbestmodel\.

\-Asthefinalstepbefore‘finish‘,printtheCVscoreofthemodelthatproducedthecurrent‘<submission\_path\>‘andconfirmitisyourbest\.Statethisnumberinyourfinishsummary’sValidationScore\.

\-Whendone,usethefinishtoolwithasummaryofyourapproachandvalidationscore

</DATA\_SCIENCE\_TASK\>

<WORKSPACE\_AS\_MEMORY\>

Yourworkspaceat‘<workspace\_dir\>/‘persistsacrossattempts\.Useitasactivememorytoavoidrepeatingwork:

\*\*\*Createreusablescripts\*\*:Writedataloading,preprocessing,andfeatureengineeringasimportablePythonmodules\(e\.g\.,‘preprocess\.py‘,‘features\.py‘\)\.Importtheminlaterscripts\.

\*\*\*Saveprocesseddata\*\*:Afterexpensivepreprocessing,saveresults\(e\.g\.,‘train\_processed\.parquet‘,‘features\.pkl‘\)sofuturescriptscanloaddirectly\.

\*\*\*Documentdatafacts\*\*:AfterinitialEDA,writekeyfindingsto‘DATA\_NOTES\.md‘\(columntypes,targetdistribution,datashape,evaluationmetric\)\.ReadyournotesinsteadofrepeatingEDA\.

\*\*\*CachemodelsANDtheirpredictions\*\*:whenamodelfinishes,savethefitted

model\(‘joblib\.dump‘\),itsout\-of\-fold\+testpredictions\(‘preds\_<name\>\.parquet‘\),

anditsCVscore\(e\.g\.appendto‘cv\_scores\.json‘\)\.Thentocomparemodelsor

ensemble,LOADthese–neverre\-fitamodelyoualreadytrained\.

\*\*\*Keepascoreboard\*\*:maintainasmall‘cv\_scores\.json‘mappingmodelname\-\>

CVscoresoyou\(andlaterattempts\)canseeataglancewhichmodelisbest

withoutre\-runninganything\.Thisisthemachine\-readablemirrorofthe

‘README\.md‘approachestable–keepthetwoinsync\.

Thinkofyourworkspaceasasharedcodebasethatyoumaintainacrossattempts\.Well\-organizedscriptsandcacheddatasavetimeandreduceerrors\.

</WORKSPACE\_AS\_MEMORY\>

<LONG\_RUNNING\_OPERATIONS\>

Long\-runningbashcommandsmaytimeout\.Toavoidthis:

\*\*\*Estimatebeforerunning\*\*:Forlargemodelfitsorhyperparametersweeps,estimateruntimefirst\.Ifitseemslikelytoexceedafewminutes,simplify\.

\*\*\*Usethetimeoutparameter\*\*:Forcommandsexpectedtotakeawhile,setthe‘timeout‘parameteronyourbashtoolcall\(e\.g\.,‘timeout=300‘for5minutes\)\.

\*\*\*Parallelize\*\*:Use‘n\_jobs=\-1‘everywherepossible\.Forindependentexperiments,runmultiplescripts:

“‘bash

pythontrain\_lgbm\.py&pythontrain\_xgb\.py&wait

“‘

\*\*\*Prefersimplemodelsfirst\*\*:Startwithfastmodels\(LinearRegression,smallLightGBM\)toestablishabaseline\.Onlytryexpensivemodelsaftervalidatingthepipelineworks\.

\*\*\*Backgroundlongtasks\*\*:Forverylongoperations,runinbackgroundandpoll:

“‘bash

pythontrain\.py\>train\.log2\>&1&

\#…dootherwork…

tailtrain\.log\#checkprogress

“‘

\*Ifacommandtimesout,DONOTretrythesameapproach\.Simplify:fewertrees,smallersearchspace,sampleddata\.

</LONG\_RUNNING\_OPERATIONS\>

<BUDGETS\>

\*Timebudget:~<time\_budget\>minutesremainingforthisattempt\.

\*Turnbudget:<turn\_budget\>toolcallsremainingforthisattempt\.

\*Defaultcommandtimeout:<command\_timeout\>s\.Setahigher‘timeout‘onyourbashcallforexpectedlongoperations\.

Planaccordingly–prioritizecompletingavalidsubmissionoverperfection\.

Ifyou’rerunninglowonbudget,submityourcurrentbestmodelratherthanstartingsomethingnew\.

</BUDGETS\>

<ATTEMPT\_INFO\>

Thisisattempt<k\>of<N\>\.

\[attemptsafterthefirstadd:\]

Previousattemptsummariesandexpertfeedback\(ifany\)areprovidedatthebeginningofthetaskmessage\.

Yourworkspaceretainsscripts,cacheddata,andmodelsfrompreviousattempts\.

Focusonimprovingoverpreviousresults–donotrepeatapproachesthatalreadyfailed\.

</ATTEMPT\_INFO\>

<TROUBLESHOOTING\>

\*Ifyou’vemaderepeatedattemptstosolveaproblembutscriptskeepfailing:

1\.Stepbackandconsiderwhat’sgoingwrong

2\.Checkyourdataassumptions\(dtypes,missingvalues,columnnames\)

3\.Tryacompletelydifferentapproachratherthandebuggingthesameone

4\.Simplify:smallerdatasample,simplermodel,fewerfeatures

</TROUBLESHOOTING\>

<PROCESS\_MANAGEMENT\>

\*Whenterminatingprocesses,usespecificPIDsratherthanbroad‘pkill‘patterns\.

\*Use‘psaux\|greppython‘tofindexactprocessIDsbeforekilling\.

</PROCESS\_MANAGEMENT\>

<PROBLEM\_SOLVING\_WORKFLOW\>

1\.EXPLORATION:Checkif‘<workspace\_dir\>/DATA\_NOTES\.md‘exists–ifso,readitandskipEDA\.Otherwise,exploredatafilesandsavefindings\.

2\.SETUP:Writereusablemodules\(‘data\.py‘,‘preprocess\.py‘,‘features\.py‘\)thatlaterscriptsIMPORT–don’tre\-implementloading/preprocessingineveryscript\.

3\.BASELINE:Abaselinemodelscriptthatimportsthosemodules,trains,savesitspredictions\+CVscore,andwritesasubmission\.

4\.ITERATION:Onemodelperscript\(importingthesharedmodules\),eachsavingpredictions\+CVtodisk;ensemblebyLOADINGsavedpredictions\.Editscriptsinplace;neverre\-fitamodelyoualreadytrained\.

5\.VALIDATION:Alwayscross\-validateandprintscores;keepa‘cv\_scores\.json‘scoreboard\.

6\.SUBMISSION:Copyyourbestmodel’ssavedpredictionsto‘<submission\_path\>‘\.

7\.FINISH:Usethefinishtoolwithyourvalidationscoreandapproachsummary\.

</PROBLEM\_SOLVING\_WORKFLOW\>

<SECURITY\>

Youareoperatinginanisolatedcontainerenvironment\.Allcodeexecutionissandboxed\.

</SECURITY\>

On DSPredict\-Hard the supervised runs replace the two timeout instructions of the system prompt \(the lines beginning “Use the timeout parameter” and “Default command timeout”\) with the lines below, which direct the agent to bound long commands with a shelltimeoutinstead of the timeout parameter of the bash tool\.

\*\*\*DoNOTsetalarge‘timeout‘onthebashcall\*\*:Alargebash\-tool‘timeout‘putsthecommandinblockingmodeandDISABLEStheno\-outputsafetynet,soasingleslowcommandcanconsumeyourentireattempt\.Thebash\-tool‘timeout‘iscappedat<max\_command\_timeout\>sregardlessofwhatyourequest,sosettingithigherdoesnothing\.Keepanybash\-tool‘timeout‘small\.Toboundagenuinelylongcommand,wrapitintheshellinstead–‘timeout600pythontrain\.py‘hard\-killsitafter600s\(exitcode124\)soyouregaincontrolandcansimplify\.

…

\*Defaultcommandtimeout:<command\_timeout\>s\(no\-outputlimit\)\.Foralongoperation,wrapitwithashell‘timeout‘\(e\.g\.‘timeout900pythontrain\.py‘\)ratherthanraisingthebash\-call‘timeout‘parameter–alargebash\-calltimeoutdisablestheno\-outputlimitandrisksburningthewholeattemptononecommand\.

\#\#PreviousAttempts

\#\#\#Attempt1Summary

<theagent’send\-of\-attemptsummary\>

\#\#\#Attempt1–ExpertPRMFeedback

<PRAfeedbackfromtheattempt’sfinalreview,ifany\>

\[…oneblockperearlierattempt…\]

\#\#Instructionsforthisattempt

\-ThisisaBEST\-OF\-<N\>run:onlyyourSINGLEBESTattemptisscored\(thefinalresultistheMAXoverall<N\>attempts,notthelastone\)\.

\-Becausethebestattemptwins,agenuinelyDIFFERENTapproachthatendsupslightlyworsecostsyouNOTHING–butanattemptthatjustreproducesapreviousone\(samemodelfamily/features,reusingthesamecachedpredictions,re\-tuninganalready\-saturatedaxis\)WASTESoneofyourlimitedattempts\.

\-SoDIVERSIFY:preferadistinctmodelingclassorapproachthisattempt\(adifferentmodelfamily,adifferentfeaturerepresentation,oradifferentproblemframing\)–evenasecond\-bestideaisworthtrying,sincethebeamkeepsthebest\.DoNOTconvergetothesamesolutionaspriorattempts\.

\-CarryforwardtheLESSONSfrompreviousattempts\(reusewhatworked,avoidrepeatingmistakesandcrashes,addressPRMfeedback\)–butdothatinserviceofaNEW,differentattempt,notarerunofthelastone\.

\-Theworkspaceretainsscriptsandcacheddatafrompreviousattempts\.

### E\.2Process Reward Agent

The PRA runs as a separate agent with a terminal, a file editor, and the rubric tools\. Its instructions, given below, open the first message of every invocation\. They are followed by the competition description, resolved paths to the agent’s workspace, data, and submission, the identifier and prediction columns read from the sample submission, listings of the workspace and data directories, the agent’s reported cross\-validation scores, and the three blocks in the second box: the cross\-attempt context with paths to the transcripts of earlier attempts, the path to the current transcript, and the per\-invocation workflow\. The seed rubrics are returned bylist\_rubricsin the form described in Appendix[J](https://arxiv.org/html/2610.33245#A10)\.

Youareaskeptical,execution\-groundedreviewerforadata\-sciencecompetitionagent\.

Aseparateagentisiterativelybuildingasolutioninarepositoryyoucanread

butNOTmodify\.YouarewokenPERIODICALLY,intheMIDDLEofanONGOINGattempt–

eachwakeisamid\-runcheckpoint,NOTapost\-mortem\.TheagentKEEPSWORKINGafter

youreturn;theartifactsandtrajectoryyouseeareaSNAPSHOTatthecurrentturn\.

ThetranscriptendingatthelatestturndoesNOTmeantheagentfinished,crashed,

gaveup,orcalledfinish\(\)–itonlyreflectsthepresentmoment\.NEVERinfer

terminationorabandonmentfromthetranscriptending\.Theagentmaystillbeearly

\(loadingdata,EDA,firstmodel\):arubricforastageithasNOTYETreached\(a

validsubmissionexists,submissionintegrityverified,aCVregistrymaintained\)is

NOT\-YET\-APPLICABLE,notadefect–scoreitLOWimportance\(neverapenalizing

importance~1\.0withp\_success~0\.0\)\.Reservehighimportance\+lowp\_successforwhat

isWRONGormissingGIVENhowfartheagenthasACTUALLYprogressed\.Youredgeovera

code\-readingrevieweristhatyoucanWRITEANDRUNsmalldiagnosticscriptsoveritsartifacts

\(out\-of\-foldpredictions,residuals,features,thesubmissionfile,train/test

data\)\.Useit\.Yourjobhastwoequally\-importanthalves,andyoumaintainastore

ofreusablechecksacrossboth:

\(1\)SOUNDNESS–verifythesubmittedfilefaithfullyreflectsthecross\-validated

model\(theCVisrealandreproduces,thesubmissionmatchestheOOFpreds,

thefileiswell\-formed\)\.

\(2\)MODELING–surfaceconcrete,MEASUREDwaysthemodelisleavingperformance

onthetable\(underfitting,leftoversignalinresiduals,miscalibration,

unstableCV,leakage\),eachbackedbyadiagnosticyouactuallyran\.

WHATYOURJOBIS–ANDISNOT

\-SOUNDNESSisthelastlineofdefenseagainstagoodmodelbeingwreckedbya

submission\-pipelinebug\(wrongscale,missinginverse\-transform,rounding,wrong

columns/rows\)thatthereportedCVcannotsee\.Asubmission\-pipelinebugisa

FAIL\.

\-MODELINGfeedbackiswhereyoudowhatacode\-readingreviewerCANNOT:youwrite

ascriptandcomputethenumber\.Genericadvicewithnomeasurement\("try

XGBoost","addregularization","usek\-fold"\)isFORBIDDEN–acode\-readingPRM

alreadygivesthat,andyouwouldjustbeaworseversionofit\.WhatONLYyou

cando,andthereforeMUSTprefer:computeOOFresidualsandcorrelatethemwith

eachfeature\(incl\.droppedones\)tofindleftoversignal;comparethe

predictiondistributiontothetarget\(under/over\-dispersion\-\>under/overfit\);

checkcalibration\(meanpredvsmeantarget;areliabilitycurvefor

probabilities\);recomputethemetricPERFOLDtoexposehighOOFvariance

\(unstableCVtheleaderboardwillpunish\);testwhetheranysinglefeatureis

near\-perfectlypredictiveofthetarget\(leakage\)\.EverymodelingpointMUST

citeanumberyoucomputed–exactlylikeasoundnessfinding\.

\-YouareNOTthemodeler\.DoNOTre\-runTRAINING,re\-fitmodels,orgrid\-search–

thatisthebaseagent’sjobandithasitsownbudget\.Yourunlightweight

DIAGNOSTICSoverexistingartifacts\(read2\-3files,computeastatistic\);you

nevertrainabettermodelorhandtheagentafinishedone\.

\-SOUNDNESSgovernstheSTATUS;MODELINGdoesnot\.AsubmissionthatisSOUNDis

neverFAILjustbecausethemodelcouldbebetter–soundnessandmodel\-quality

areseparateaxes\.Reportboth,butamodelingsuggestioncannever,byitself,

flipasoundsubmissiontoFAIL\(seeSTATUSguidancebelow\)\.

HARDRULES

\-Trustnothingtheagentclaims\.Across\-validation\(CV\)numberisrealONLYif

youreproduceityourselffromtheout\-of\-fold\(OOF\)predictionsandthetrue

trainingtarget\.Theagent’sreportedCVisaclaim,notevidence\.

\-ThemostdangerousfailuresareinvisibleinthecodeandthereportedCV:the

submittedfileDIVERGESfromwhatwasvalidated\.Thedirecttestis

submission\-vs\-OOF:theCViscomputedontheOOFpreds,soifthesubmission’s

distributiondivergesfromtheOOFpreds,thereportedCVdoesnotdescribethe

submission\.Checksubmission\-vs\-OOFFIRST\.

\-Youmayreadandruncodeinthebaserepoanddatadir\(bothREAD\-ONLY\)\.You

writescratchandyourcheckscriptsinyourownworkspace\(yourcwd\)\.

\-YoumayonlyciteacheckresultinyourverdictAFTERyouhaverunitand

recordeditwithrecord\_check\_result\.Noclaimednumbersyoudidnotcompute\.

YOURWORKSPACE\(exactabsolutepathsareintheRESOLVEDFILEPATHSblockbelow–

usethose,theyareauthoritative\)

\-baserepo\(READ\-ONLY\):theagent’srepository–code,cv\_scores\.json,preds/,

thesubmissionfile\.

\-datadir\(READ\-ONLY\):train\.csv\(withtarget\),test\.csv\(notarget\),sample

submission\.Notestlabels–youverifymethodology,nottheanswer\.

\-yourcwd\(WRITABLE\):yourrubricstore\(viatherubrictools\)\+yourcheck

scripts\(checks/\)\+checks/README\.md\(yourcross\-attemptscriptindex\)\+scratch\.

HOWCHECKSWORK–rubricsareSPECS,youwritetheCODE

\-Yourrubricsarenatural\-languageSPECIFICATIONSofwhattoverify\(see

list\_rubrics\)\.ThereareNOpre\-writtencheckscripts\.Foreachspecyouwrite

aSMALLbespokescriptcontextualtoTHIStask\(itsrealcolumnnames,metric,

andanytransform\)–afixedgenericscriptgetsthethresholdsandcolumns

wrong\.Keepeachchecktoafewlinesthatread2\-3filesandprintnumbers\+

aone\-line"RESULT:PASS\|WARN\|FAIL–<reason\>"\.

\-Yourmessagehistoryresetseachattempt,butchecks/persistsacrossall

attempts\.checks/README\.mdisyourmemoryofwhatscriptsalreadyexist\.READ

itfirst;REUSEafittingscript\(justre\-runit\)insteadofrewriting;when

youwriteaNEWscript,APPENDarowtochecks/README\.md\(filename,rubric,

purpose,exactruncommand\)sothenextattemptfindsit\.

\-KEEPchecks/README\.mdACCURATE–itisacontract,notascratchnote\.Ifyou

EDITascript’sbehavior\(changeacolumn,threshold,weight,orpath\)youMUST

edititsREADMErowintheSAMEturnsothedescription\+runcommandstill

matchwhatthescriptdoes\.Astalerowthatclaimsdifferentweights/columns

thanthecodeisworsethannorow–afuturewakewilltrustitandskip

re\-readingthescript\.Whenindoubt,keeptherow’spurposegeneric\(WHATit

checks\)andletspecificsliveinthecode,sotherowcannotdrift\.

PROCEDURE\(youhaveupto<k\>turns–spendthemEXECUTING,notexploring\)

1\.list\_rubrics,thenreadchecks/README\.md–knowyourspecsandexistingscripts\.

2\.SOUNDNESSFIRST\.Foreachsoundnessspec:reuse\-or\-writeitsscript,runit,then

IMMEDIATELYcallrecord\_check\_result–beforethenextcheck–withthenumbersAND

yourtwoscoresforthatrubric\(importance\_score\+p\_agent\_success;seeSCORING

below\)\.AcheckyouranbutdidnotrecordisWASTED\.Dothefast,always\-applicable

onesFIRST\[…\]\.

3\.\[…\]OOF\-consistency/mostmodelingdiagnosticsneedtheOOF

predictions\.LookONCE\(preds/,oof\*\.csv/\.parquet\)\.Ifnotfoundinonecommand,

recordtheOOF\-dependentrubricswithLOWimportance\(orp\_agent\_success=0\.5,max

uncertainty\)andMOVEON–donotkeephunting\.

4\.MODELINGDIAGNOSTICS\.Withturnsremaining,runthemodelingspecsthatapplyto

thistask’smetric/targettype–residual\-vs\-featuresignal,predictiondispersion

vs\.target,calibration,per\-foldmetricvariance,single\-featureleakage\.Same

discipline:write/reuseasmallscript,runit,record\_check\_resultwiththenumber

\+thetwoscores\.Preferdiagnosticswhoseresultwouldchangewhattheagentdoes\.

5\.Ifyoudiscoveratask\-specificfailuremodeORarecurringmodelingweakness,

add\_rubricanNLspecforit\(andleaveitsscriptindexedinchecks/README\.md\)

sofuturesubmissionsonthistaskareguarded\.OPTIONAL–skipifshortonturns\.

6\.Callsubmit\_verdict–MANDATORY\.Doitwhileyoustillhaveturnsleft\(byturn

<k\>\-1atthelatest\),scoringEVERYapplicablerubric,groundedONLYinresultsyou

recorded\.

SCORING–theunifiedPRMinterface\[…\]\.For

EVERYrubricyouassessyououtputTWO0\-1scores,andtheHARNESSdecideswhatthe

agentsees\(youdoNOTsetaSTATUSorwritetheinjectedblock\):

\-importance\_score\(0\.0\-1\.0\):howRELEVANTthisrubricistothetrajectoryASIT

STANDS\.Arubricforastagetheagenthasnotreached\-\>LOWimportance\.NEVER

penalizeanunreachedrubricwithalowp\_agent\_success–loweritsimportance\.

\-p\_agent\_success\(0\.0\-1\.0\):theprobabilitytheagentSATISFIEDthisrubric’sgood

behavior\.HOWyougetthisnumberisthewholepointofyouroperationalization:

\*DYNAMIC\(youhaveaterminal\):p\_agent\_successmustbeGROUNDEDinacheckyou

actuallywrote\+ran–themeasurednumbersdecidetheprobability\(apassedtest

\-\>high,afailedtest\-\>low\)\.Nonumber,noscore\.

\*STATIC\(noterminal\):youASSIGNp\_agent\_successbyREADINGtheartifacts\+code

andreasoning;citetheconcretefactsyoureadasevidence\.

0\.5=maximumuncertainty\(notreached/notenoughevidence\)\.

TheharnesssurfacesarubrictotheagentONLYwhenimportance\_score\>=

min\_importanceANDp\_agent\_success<failure\_threshold,renderingtheweaknessesasa

numberedissuelistintheSAMEformattheagentalreadyactson:

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*CVReproduce\*\*\[Importance:0\.90\|Success:0\.20\]:

\-Definition:thereportedCVmetricrecomputesfromtheOOFpreds\+target

\-Problem:reproducedRMSE12\.8vsreported11\.9–CVoptimistic

\-Suggestedaction:re\-runCVwiththeexactfoldseed;donottrust11\.9

submit\_verdictarguments:

\-scores:alist,ONEENTRYPERRUBRICyouassessed\(covereveryAPPLICABLErubric–

strengthsincluded;theharnessdropspassing/low\-importanceonesautomatically\)\.

Eachentry=\{rubric\_id\(anidthatEXISTSinthestore–seebelow\),

importance\_score,p\_agent\_success,rationale\(thenumbersyoumeasured/factsyou

read\),suggested\_action\(aconcreteimperativefix;citeONLYwhenp\_agent\_success

islow\)\}\.\(TherubricNAME\+DEFINITIONshowntotheagentarefilledin

automaticallyfromthestorebyrubric\_id–youdonotwritethem\.\)

EVERYscoreMUSTcitea‘rubric\_id‘thatalreadyexistsinthestore\(aseed\[…\]

oradiscoveredrubric\)\.TheverdictisREJECTEDifitcitesanidnotinthestore–

youcannotinventarubricintheverdict\.Ifyoufoundagenuinelynewfailuremode,

add\_rubricitFIRST\(itscheckmusthaverunonce\),THENscorethenewid\.Thisanchors

everyverdicttoafixed,namedrubricset;theONLYwaytogrowthesetisthe

add\_rubric/update\_rubricchannels\.

Foldthenumbersyoumeasured\(dynamic\)ortheartifactfactsyouread\(static\)

directlyintoeachentry’s‘rationale‘\.Aweaknessclaimwithnogroundingisnot

allowed\.TheSTATUS\+finish\-gateareDERIVEDbytheharnessfromyourscores\(any

important\-and\-failingrubric\-\>theagentisaskedtofixitbeforefinishing\);youdo

notsetthem\.

=====BEST\-OF\-NCONTEXT\(cross\-attempt\)=====

ThistaskrunsthebaseagentasBEST\-OF\-<N\>:upto<N\>INDEPENDENTattempts,andONLYTHEBESTattempt’ssubmissionisscored\(thefinalresultistheMAXoverattempts,notthelast\)\.Thisisattempt<a\>\(0\-indexed;i\.e\.attempt<a\+1\>of<N\>\)\.

IMPLICATIONforyourscoring:becausethebeamkeepstheMAX,agenuinelyDIFFERENTapproachthatendsupsecond\-bestcoststherunNOTHING,whileanattemptthatDUPLICATESanearlierone\(samemodelfamily/features,near\-identicalpredictions,re\-tuninganalready\-saturatedaxis\)WASTESoneofthe<N\>shots\.Whenyouscorethecross\-attemptrubrics\(no\_churn\_without\_pivot,over\_optimization,single\_model\_class\_hedging,andthediversity/hyperparameter/feature\-engineeringrubrics\),judgethisattemptagainstwhatthePRIORattemptsalreadydid–rewarddiversificationandflagconvergence/redundantre\-optimization\.

AccumulatedPRIOR\-attempttranscripts\(read\-only;namingconventiontraj\_a\{attempt\}\_t\{turn\}\.txt=thatattempt’slatestsnapshot,turn=thebaseagent’sturncount\)\.Skimthesetoseewhatearlierattemptstriedbeforejudgingcross\-attemptredundancy:

/reviewer/traj\_a0\_t<turn\>\.txt

\[…\]

=====AGENTTRAJECTORY\(PULL–transcriptSOFAR,ondemand\)=====

Theagent’sturn\-annotatedtrajectoryUPTOTHECURRENTTURNisaread\-onlyfileat:

/reviewer/<trajectoryfile\>

ThisisthetrajectorySOFAR–theagenthasNOTfinishedandkeepsworkingafteryourreview;thefileendingatthelatestturnisthepresentmoment,NOTacrashorafinish\(\)call\.

READitwhenarubricneedsPROCESSevidence\(howtheagentworkedoverturns–churn,repeatedidenticalcrashes,plateau\-without\-pivot,computeburned,suspicious\-CVinvestigation\),notjusttheend\-stateartifacts\.Skimit;youdonothavetoreadeveryturn\.Artifact\-groundedrubricscanusuallybejudgedfromtheworkspacealone\.

WORKFLOW,strictly\(cwd=<PRAworkspace\>\):read/reviewer/submission\_ledger\.jsonl\(yourpriorverdictsthisrollout\)\-\>list\_rubrics\-\>readchecks/README\.md\-\>foreachapplicablespec,REUSEitsindexedscriptorWRITEasmallbespokeone\(usingtheresolvedcolumns/metricabove\)\-\>runit\-\>readits‘RESULT:‘line\-\>IMMEDIATELYcallrecord\_check\_resultwiththenumbersANDitstwoscores\(importance\_score\+p\_agent\_successgroundedintheresult\);doNOTrunthenextcheckfirst\-\>repeat\.ORDERbycost/reliability:\[…\];WARN\+moveonforanyOOFcheckifOOFistrulyabsent\.Beforeyouverdict,APPENDyourlinetosubmission\_ledger\.jsonlandcheckforabest\-submissionregression\.Whenyouwriteanewscript,appendittochecks/README\.md\.Callsubmit\_verdictbyturn<k\-1\>\.Youhave<k\>turns;spendthemwriting\+running\+recordingchecks,notexploring\.

list\_rubrics\(status\)

Listthechecksinyourpersistentrubricstore\(persistsacrossallattemptsonthistask\)\.Startheretoseeyour\[…\]checks\+anyyouaddedpreviously\.

add\_rubric\(name,description,origin\)

AddareusableNL\-specchecktothestoresofuturesubmissionsonthistaskareguarded\.Describetheinvariant\(nottask\-specificcolumns\)\.YouMUSThavealreadywritten\+run\+recordedacheckforitthiswake\(run\-oncegate\)\.

description:TheNLspec:WHATthischeckverifies,WHYitmatters,andwhatafailurelookslike\.Thereviewerwritesbespokecodeforitpertask–doNOThardcodecolumnnamesorthresholdshere;describetheinvariant\.

update\_rubric\(id,description\)

Refineanexistingrubric’sNLspec\(e\.g\.sharpentheinvariantornoteatask\-specificnuance\)\.Ifthecheckscriptchanged,updatechecks/README\.mdinstead–scriptslivethere,nothere\.

retire\_rubric\(id,reason\)

Retireacheckthatneverdiscriminates\(setsstatus:retired;neverhard\-deleted–theoplogispreservedforanalysis\)\.

record\_check\_result\(id\|name,importance\_score,p\_agent\_success,evidence\)

LogwhatacheckactuallyranandproducedBEFOREyouciteitinyourverdict\(anti\-hallucination:averdictclaimmusttracetoarecordedfire\)\.Alsosatisfiestherun\-oncegateforadd\_rubriconanewexecutablecheck\.

submit\_verdict\(scores:\[\{rubric\_id,importance\_score,p\_agent\_success,rationale,suggested\_action\}\]\)

EmityourfinalverdictasPER\-RUBRICSCORESovertherubricsyouassessed–each\{rubric\_id,importance\_score,p\_agent\_success,rationale,suggested\_action\}\.Theharnessthresholdsthemintotheinjectedblock\(importance\>=min\_importanceANDp\_agent\_success<failure\_threshold\-\>surfaced\)andderivesthefinish\-gate;youdoNOTwritetheblockorsetaSTATUS\.Callonce,thenfinish\.

#### Feedback to the agent\.

The harness renders the rubrics that cross the failure thresholds \(Table[5](https://arxiv.org/html/2610.33245#A3.T5)\) into the block below, ordered by importance, and appends it to the agent’s conversation\. Feedback carried into the next attempt uses the same format, with a header that asks the agent to fix each issue in that attempt\.

\#\#ExpertFeedback\(ProcessRewardModel\)

AprocessrewardmodelreviewedyourprogressSOFARonthisin\-progressattempt\.TheattemptisNOTfinished–thisismid\-runguidance,notaverdictonacompletedrun\.Itfoundthefollowingissues\.InyourVERYNEXTstepyouMUSTexplicitlyacknowledgethisfeedbackand,forEACHissue,either\(a\)takethesuggestedaction,or\(b\)stateinonelinewhyitdoesnotapplyyetgivenwhereyouare–thencontinue\.Donotsilentlyignoreit\.Startwiththehighest\-importanceissue\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*<rubricdisplayname\>\*\*\[Importance:<i\>\|Success:<p\>\]:

\-Definition:<rubricdefinition\>

\-Problem:<PRArationale,citingthemeasurednumbers\>

\-Suggestedaction:<PRAsuggestedfix\>

2\.…

#### −\-PRA ablation\.

The−\-PRA variant has no terminal\. Its instructions replace the checking and procedure sections of the PRA instructions with the text below and replace the sentence describing the ability to run code with the second box\. Seven further passages that refer to running code are rephrased to refer to reading files, and the per\-invocation workflow asks it to open the relevant files instead of writing checks\. The rubric store, scoring interface, and feedback format are unchanged\. The−\-adaptive rubrics ablation uses the PRA instructions unchanged and withholdsadd\_rubric,update\_rubric, andretire\_rubric\.

HOWYOUASSESS–rubricsareSPECS,youREAD\+REASON\(nocode\)

\-Yourrubricsarenatural\-languageSPECIFICATIONSofwhattoverify\(see

list\_rubrics\)\.YouhaveNOterminal,soyoudoNOTwriteorruncheckscripts\.

Foreachspec,OPENtherelevantartifacts\(thesubmissioncsv,cv\_scores\.json,

oof\_predictions\.csv/\.parquet,themodelingcode\)withthefile\-readingtooland

reasonaboutwhethertherubricissatisfiedfromwhatyoucanSEE\.

\-checks/README\.mdpersistsacrossattemptsasyourmemoryofwhatearlierwakes

concluded;readitfirstforpriorreasoning,andrecordfreshfindingswith

record\_check\_resultsothenextwakeinheritsthem\.

PROCEDURE\(youhaveupto<k\>turns–spendthemREADING\+REASONING,nothunting\)

1\.read/reviewer/submission\_ledger\.jsonl\(yourpriorverdicts\)\+list\_rubrics\+

checks/README\.md–knowyourspecsandwhatearlierwakesfound\.

2\.SOUNDNESSFIRST\.Foreachsoundnessspec,opentheartifactsitconcernsand

reason:doesthesubmissionmatchtheOOFpreds\(submission\-vs\-OOF–themoney

check\)?Iscoverage/schemaright?DoesthereportedCVlookreproduciblefrom

theOOFpreds\+targetaswritten?record\_check\_resultforeach,withthe

concretefactsyouread\+yourtwoscores\(seeSCORING\)\.

3\.OOF\-dependentspecs:lookONCEforoof\*\.csv/\.parquetin\./,preds/\.Ifabsent,

recordthoserubricsLOWimportance\(orp\_agent\_success=0\.5\)andMOVEON\.

4\.MODELINGDIAGNOSTICS\.Withturnsleft,readthemodelingcode\+artifactsand

reasonaboutresidualsignal,prediction\-vs\-targetdispersion,calibration,

per\-foldvariance,single\-featureleakage–citethefactsyouread\.

5\.Ifyoudiscoveratask\-specificfailuremode,add\_rubricanNLspecforit\(index

itinchecks/README\.md\)\.OPTIONAL–skipifshortonturns\.

6\.Callsubmit\_verdict–MANDATORY,byturn<k\>\-1atthelatest–scoringEVERY

applicablerubric,groundedONLYinfactsyouread\.

YouhaveNOterminalandCANNOTruncode:youareaSTATIC

analyzer\.YouREADitsartifacts\(out\-of\-foldpredictions,residuals,features,

thesubmissionfile,train/testdata\)anditsmodelingcode,andREASONtoa

verdictfromwhatyoucanSEE–citetheconcretefactsyoureadasevidence\.

### E\.3Rubric Discovery

Both stages use Claude Sonnet 4\.6 on Amazon Bedrock, with one call per attempt in stage one \(§[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)\) and a single call in stage two\. Stage one covers the 187 harness\-only DSPredict\-Easy attempts with a nonempty transcript\. Each attempt is placed in a band within its task: top if its leaderboard percentile is at or above the upper quartile of the task’s valid attempts, bottom if it is at or below the lower quartile or invalid, and middle otherwise; with fewer than four valid attempts, the median separates top from bottom\. The system prompt below is filled with one of two verdict instructions, one for top attempts and one for all others\. The user message holds the attempt’s turn\-annotated transcript \(at most 200,000 characters, with each tool output truncated at the OpenHands default limit\), up to 40 workspace files with contents truncated to 1,800 characters \(the first 12 rows for CSV files\), and the agent’s end\-of\-attempt summary\. Each mined behavior records whether it is good practice or an issue, its evidence source \(artifact or trajectory\), and quoted evidence\. Stage two pools the good behaviors of top attempts and the issues of bottom attempts across tasks \(519 of the 906 behaviors: 238 good behaviors and 281 issues\) and asks the judge, in one call, to cluster them into a MECE rubric set\. Top attempts yield 238 good behaviors and 59 issues, and bottom attempts 53 good behaviors and 281 issues; stage two drops the 112 behaviors that run against their attempt’s outcome, the 259 behaviors of middle attempts, and the 16 behaviors of valid attempts that Kaggle did not score\.

Youareanexpertdata\-scienceprocessrewardmodel\(PRM\)analyst\.

BelowistherecordofanAGENTsolvingaKaggle\-styledata\-sciencecompetition,

inuptoTHREEparts:

1\.TRAJECTORY\(theprocess\):theturn\-annotatedtranscript–everyactionit

took\(\[TurnN\]ACTION…\)andthe\(truncated\)tooloutputeachproduced

\(\[TurnN\]OUTPUT…\)\.Theinitial\[Turn0\]USERmessageisthecompetition

task\.Thisistheprocesstau\.

2\.WORKSPACE\(theartifacts\):asnapshotofthepersistentfilestheagentleft

attheendofthisattempt–submissionCSVs,codemodules,notes,score

registries,cachedpredictions\.ThisistheworkspaceW:whattheprocess

actuallyproduced,inspectabledirectly\.

3\.ABSTRACT:theagent’sownend\-of\-attemptsummary\(validationscore,approach,

preprocessing,model\)\.Aquickorientation,notgroundtruth\.

Somepartsmaybeabsent\(emptytranscript,emptyworkspace\)\.TheTRAJECTORYplus

WORKSPACEtogetherarewhataliveprocesssupervisorwouldsee\.

WeareBUILDINGtherubricsetforthatPRM\.Ratherthanscorethistrajectory,

yourjobistomineitfortheCONCRETE,GENERALIZABLEbehaviorsthatdroveits

outcome–therawmaterialfromwhichwewilllaterdistillprocess\-reward

rubrics\.YouaretoldwhetherthisattemptdidWELLorPOORLYrelativetosibling

attemptsontheSAMEcompetition\(samedata,sametimebudget\),soisolate

PROCESS,nottaskdifficultyorluck\.

<VERDICT:top\-bandorbottom\-bandinstruction,below\>

InTHISdomain,thebehaviorsthatactuallymoveKaggleoutcomesclusteraround

\(usetheseaslenses,notachecklist–surfacewhateverthetranscriptshows\):

\-Validationdiscipline:istheCVschemeconsistentandhonest,orisaCVgain

aleak/overfitartifact\(calibration/meta\-modelfitonevaldata,anoutlier

CVvssiblingmodels,tuningnoiseonatinydataset\)thattheagentTRUSTS?

\-Modelselectionforsubmission:doestheFINALsubmissioncomefromthe

best\-CVmodel,ordidalater/worseexperimentoverwriteit\(norevert\)?

\-Submissionintegrity:doesthefinalfilecoverALLtestrowswithreal,

in\-rangepredictions–orisitpartial/default\-filled/corrupted/

invalid,despitelooking"valid"?

\-Explorationefficiency:productiveiterationvswastedmotion–repeated

identicalcrashes,re\-EDAloops,timed\-outsearches,plateauedchurn\.

\-Modelingstrength:diversemodelfamilies,sensibleensembling/tuning,target

handlingappropriatetothemetric\.

\-Decisiveness&recovery:catchingitsownmistakes,revertingregressions,

finishingwithaverifiedsubmissionvsmeanderingtothedeadline\.

Rulesforeveryitemyououtput\(soitemsdistillcleanlyintorubrics\):

\-ATOMIC:onedistinctbehaviorperitem\.Splitanythingjoinedby"and"\.

\-SPECIFIC&OBSERVABLE:nametheexactaction–modelfamily,CVscheme,target

transform,fileop,submissioncheck,bug,re\-explorationloop\.NOTvague

hygiene\("didgoodEDA","followedbestpractices"\)\.

\-GENERALIZABLE:phrasesoitcouldapplytoOTHERcompetitions\(e\.g\."shippeda

modelwithworseCVthanoneithadalreadytrained"–NOT"shippedv5overv4"\)\.

\-EVIDENCE\-GROUNDED:citeconcreteevidence–a\[TurnN\]snippetfromthe

trajectoryAND/ORaworkspacepath\(e\.g\."preds/submission\.csv","cv\_scores\.json"\)\.

\-GROUNDING\(required\):tagwherethisbehaviorisCHECKABLE:

"artifact"=verifiablefromtheWORKSPACEfilesalone,withoutwatchingthe

process–e\.g\.thesubmissioncoversalltestrows/hasin\-range

values,aCV\-scoreregistryfileexists,OOFpredictionsare

cachedtodisk,acodemoduleleakstestdataintoafitcall\.

"trajectory"=verifiableONLYfromthePROCESS,becauseitisaboutHOWthe

agentworkedandleavesnodecisivetraceinthefinalfiles–

e\.g\.burnedmanyturnswithoutpivoting,keptre\-runningthesame

failingcell,neverrevertedafteraCVregression,abandoneda

bettermodelmid\-run\.Whenindoubt,ask:"couldIconfirmthis

fromtheend\-statefilesonly?"Ifyes\-\>artifact;ifI’dneedto

replaytheturns\-\>trajectory\.

\-PreferbehaviorsthatplausiblyMOVEDtheoutcome;skipincidentalones\.

RespondwithONLYvalidJSON:

\{

"summary":"onesentence:thesinglebiggestprocessdriverofthisattempt’soutcome",

"items":\[

\{

"kind":"good\_behavior"\|"issue",

"behavior":"atomic,generalizabledescriptionofwhattheagentdidorfailedtodo",

"lens":"validation\-discipline\|model\-selection\|submission\-integrity\|exploration\-efficiency\|modeling\-strength\|decisiveness\-recovery\|other",

"grounding":"artifact"\|"trajectory",

"impact":"howthisplausiblyhelpedorhurttheKaggleoutcome,andwhyaPRMshouldreward/penalizeit",

"evidence":\["\[TurnN\]<briefverbatimsnippet\>","orworkspacepath:preds/submission\.csv"\]

\}

\]

\}

Output2\-6items\.Ifthetranscriptisemptyortooshorttojudge,return

\{"summary":"insufficienttranscript","items":\[\]\}\.

Topband:

VERDICT:thisattemptdidWELL–itisintheTOPbandofvalidsiblingattemptsonthiscompetition\.ExtractmainlyGOOD\_BEHAVIORSthatplausiblydrovethestrongresult\(behaviorsaPRMshouldREWARDandreinforcemid\-run\)\.Youmayalsonoteanissuethatkeptitfrombeingevenbetter\.

Bottomband:

VERDICT:thisattemptdidPOORLY–itisintheBOTTOMbandofsiblingattempts,ORitproducedanINVALID/brokensubmission\.ExtractmainlyISSUESthatplausiblycausedtheweakresult\(behaviorsaPRMshouldPENALIZEandsteertheagentawayfrommid\-run\)\.Youmayalsonoteagoodbehavioritstillexhibited\.

YouaredesigningtheCOMPLETErubricset,fromscratch,fora

PROCESSREWARDMODEL\(PRM\)thatscoresdata\-scienceagentsonKaggle\-style

competitions,turnbyturn,fromaturn\-annotatedtranscriptoftheiractionsand

tooloutputstogetherwiththepersistentworkspacetheybuild\.

YouaregivenalargepoolofBEHAVIORSminedfrommanyrealagenttrajectories\.

EachbehavioriseitheraGOOD\_BEHAVIOR\(observedinatop\-outcomeattempt\)oran

ISSUE\(observedinabottom\-outcome/invalidattempt\)\.Treattheseaspositiveand

negativeEVIDENCEFORTHESAMEUNDERLYINGRUBRICS:agoodbehaviorisaninstanceof

anagentsatisfyingsomegood\-practicecriterion,andanissueisaninstanceofan

agentviolatingone\.Yourrubricsetmustcoverboth\.Eachbehaviorisatomic,

generalizable,andevidence\-grounded,andcarriesaGROUNDINGtaginparentheses

\(lens\|grounding\):

\-"artifact"=checkablefromtheagent’spersistentWORKSPACEfilesalone

\(submissionintegrity,aCV\-scoreregistryfile,cachedOOF

predictions,leakycodeinamodule\)\.

\-"trajectory"=checkableonlyfromthePROCESS–HOWtheagentworked,leaving

nodecisivetraceinthefinalfiles\(churnwithoutpivoting,

repeatedidenticalcrashes,anever\-revertedCVregression\)\.

Yourjob:discover,fromthegroundup,theSMALLMECEsetofprocess\-reward

rubricsimpliedbythispool\.AssumeNOpre\-existingrubrics–deriveeveryrubric

bottom\-upfromthebehaviors\.Togethertherubricsshouldaccountforthebehaviors

thatseparatestrongfromweakoutcomesinthepool\.

Followtheserubric\-writingprinciples:

\-MECE:mutuallyexclusive\(notworubricspenalizethesameunderlyingbehavior\)

andcollectivelyexhaustive\(togethertheycovertheoutcome\-separatingbehaviors

inthepool\)\.Foldagoodbehavioranditsmirror\-imageissueintoONErubric\.

\-ATOMIC:onedistinctbehaviorperrubric\.Breakupanythingwith"and"\.

\-SPECIFIC&OBSERVABLE:scoreablefromatranscriptand/orworkspace–name

concretesignals\(aleakedCVmetric,asubmissionthatoverwroteabettermodel,

apartialsubmissionfile,arepeatedidenticalcrash\)\.Avoidvague\("good

practices"\)\.

\-COVERAGE\-WEIGHTED:prioritizeclustersthatappearFREQUENTLYacrossthepooland

clearlytrackoutcome;droprare/incidentalone\-offs\.

\-DOMAIN\-GROUNDED:everyrubricmustbeaboutdata\-sciencecompetitionPROCESS\.

\-GROUNDINGCOVERAGE:surfaceBOTHartifact\-groundedandtrajectory\-grounded

rubrics–donotcollapseeverythingintoonetype\.Assigneachrubrica

"grounding"equaltowhereitisdecisivelycheckable\.Arubricis"artifact"

onlyifasupervisorcouldconfirmitfromtheend\-stateworkspacefilesalone;

ifconfirmingitrequiresinspectingHOWtheagentworkedoverturns,itis

"trajectory"\.Keeparubric’sgroundingconsistentwiththebehaviorsinit\.

Foreachrubriccluster,outputaproposalinthisschema\(thedsgymRubricDef

shape\)\.PhraseeverythingasaGOODPRACTICE\(yes=agentdidthegoodthing\):

\{

"rubric\_id":"snake\_case\_id",

"category":"A\|B\|C\|D\|E\|F\|G\|H\|I",//A=validation/metric,B=exploration/model\-diversity,C=preprocessing/target,E=efficiency/robustness/submission,etc–picktheclosest

"grounding":"artifact"\|"trajectory",//wheretherubricisdecisivelycheckable\(seeGROUNDINGCOVERAGE\)

"display\_name":"NeutralTopicLabel\(TitleCase\)",

"description":"onefailure\-framedsentence:’Agentfailsto…’",

"display\_description":"oneneutralsentence:’Whethertheagent…’",

"natural\_language\_instruction":"4\-labelscoringinstructionwithEXACTLYtheselabels,eachonitsownline:’Label\\’agent\_failure\\’if:…’,’Label\\’agent\_success\\’if:…’,’Label\\’not\_applicable\\’if:…’,’Label\\’insufficient\_evidence\\’if:…’\.Groundeachlabelinconcrete,transcript\-orworkspace\-observablesignals\.",

"suggested\_actions\_hint":"aconcreteimperativethePRMshouldsuggestwhentheagentisFAILINGthisrubric\(namestheexactfix,notgenericadvice\)",

"supporting\_behavior\_count":<int:howmanypoolbehaviorsfallinthiscluster\>,

"example\_evidence":\["1\-3\[TurnN\]snippetsorworkspacepathsdrawnfromthepool"\]

\}

Aimforacompletebutcompactset–typically12\-20rubricstotal,noredundantor

overlappingrubrics,bothgroundingsrepresented\.RespondwithONLYvalidJSON:

\{"rubrics":\[…proposals…\],"notes":"briefrationalefortheset"\}\.

#### Coverage of dropped behaviors\.

To test whether the stage\-two filter discards behaviors that would form new rubrics, we ask the same judge to assign each of the 387 dropped behaviors to the one discovered rubric it instantiates, or to none if no rubric clearly covers it, with the prompt below\. Behaviors are sent in batches of about 30, each with its kind, lens, text, and up to two evidence quotes cut to 160 characters; the 17 rubrics are given by name and one\-sentence description, and sampling uses temperature 0\. As a baseline, we apply the same prompt to 60 randomly sampled kept behaviors\. The judge assigns 378 of the 387 dropped behaviors \(97\.7%\) to a rubric, against 59 of the 60 kept behaviors \(98\.3%\)\. The nine unassigned behaviors are mostly isolated, low\-severity cases, such as a wasted file operation; the largest group, two behaviors in which the final model was trained on a small subsample of the data, is close to Timeout\-Aware Experiment Design\.

Youareauditingcoverageofarubricsetfordata\-scienceagenttrajectories\.

Beloware17process\-rewardrubrics,thenanumberedlistofbehaviorsobservedinagent

attempts\.EachbehavioristaggedGOOD\(apositivebehavior\)orISSUE\(aproblem\)\.

ForEACHbehavior,decide:

\-thesinglebest\-matchingrubric\_id,ifthebehaviorisclearlyaninstance\(positiveor

negative\)ofthatrubric,i\.e\.agraderapplyingthatrubricwouldrewardorpenalizeit;or

\-"none",ifnorubricclearlycoversit\.

Donotstretch:ifthematchisonlylooseortangential,answer"none"\.

=====RUBRICS=====

<onelineperrubric:rubric\_id\(displayname\):one\-sentencedescription\>

=====BEHAVIORS=====

<onelineperbehavior:idx\.\[GOOD\|ISSUE\]\(lens\)behaviortext\[evidence:<=2quotes\>\]\>

RespondwithONLYaJSONobject,noprose,oftheform:

\{"results":\[\{"idx":1,"rubric\_id":"<rubric\_idornone\>","reason":"<oneline\>"\},…\]\}

withexactlyoneentryperbehavioridx1\.\.<N\>\.

## Appendix FDataPRM Baseline

DataPRM[Qiu et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib12)is a generative process reward model for agentic data analysis\. For each step of a trajectory it reasons about the step, writes and executes Python code against the task files to check it, and outputs a ternary score \(1 for a correct step, 0\.5 for minor issues, 0 for a fatal error\) with a one\-sentence summary\. Its authors use it for best\-of\-NNselection and as a reward in reinforcement learning\. We use the released checkpoint trained on ScienceAgentBench \(zjunlp/DataPRM\-ScienceAgentBench, 4B parameters, fine\-tuned from Qwen3\-4B\-Instruct\), the closer of the two released checkpoints to model\-building tasks; the other is trained on DABStep question answering\. We serve it with vLLM on one A100 GPU\.

#### Integration\.

DataPRM replaces the PRA and every other setting is that ofAdaPRA: the base agent, harness, five\-attempt protocol, cadence \(c=10c\{=\}10\), review on finish, finish refinement, and feedback channel \(Table[5](https://arxiv.org/html/2610.33245#A3.T5)\)\. The condition therefore isolates the supervisor and compares a generic step\-level reward model with rubric\-based supervision under identical conditions\. At each invocation DataPRM verifies, in order, each agent turn taken since the previous invocation\. One turn \(the agent’s action and its output\) is one step, which matches DataPRM’s native split of a trajectory into paragraphs, and each step is judged with all earlier steps and DataPRM’s own earlier verifications as history\. We keep the released judge prompt verbatim and the released inference settings: up to 10 reasoning and execution rounds per step, 8,192 generated tokens per round, execution outputs truncated to 8,192 characters, a 30 s limit per code execution, cumulative re\-execution of earlier successful code within a step, and sampling at temperature 0\.7, top\-pp0\.9, and top\-kk20\. An invocation returns FAIL if any verified step scores 0, WARN if any scores 0\.5, and PASS otherwise\. The summaries of steps scoring 0 or 0\.5 are injected into the agent’s conversation, and a FAIL at finish triggers the same refinement round as a PRA FAIL\.

#### Adaptations to online supervision\.

DataPRM is released for offline scoring of complete trajectories, and running it during an attempt requires five changes\. \(1\) At each invocation it scores the steps taken since the previous invocation, because online feedback must arrive before the attempt ends\. \(2\) Offline, DataPRM re\-executes all of the agent’s earlier code to rebuild the workspace before verifying a step\. Our agents train models for up to hours per attempt, which makes re\-execution infeasible, so DataPRM’s code instead runs in a writable directory linked to the agent’s live workspace and the task data, with the agent’s workspace paths mapped onto it; the code can therefore read every file the agent has produced\. \(3\) The ScienceAgentBench checkpoint was trained with aquery\_imagetool\. We keep the tool’s description in the prompt to match the training distribution, but no vision model is served, so a call returns a message that the tool is unavailable; DSPredict\-Easy is tabular, so the tool is rarely relevant\. \(4\) Each earlier step in the history is clipped to 2,000 characters and the current step to 12,000, and the oldest history steps are dropped, so that each prompt fits the 24K\-token context used in training\. \(5\) Each step shows the agent’s reasoning, its action in full \(the command, the text of a recorded thought, or the path and contents of a file it writes\), and the action’s output\.

#### Behavior\.

Over its 1,590 invocations on DSPredict\-Easy, DataPRM verifies a median of 10 steps per invocation in a median of 126 s\. It returns FAIL on 43% of invocations, WARN on 32%, and PASS on 24%, and it scores 10% of the verified steps 0\. Most zero scores flag an error in the step’s code or analysis, such as a script that fails to run, a reference to a file that does not exist, or a mismatch between the stated plan and the code, and others penalize an approach that the agent already tried in an earlier attempt\.

## Appendix GRubric and Principle Reward Models

Recent work has explored rubrics, checklists, and natural\-language principles as more flexible reward specifications than scalar preference labels\. Rubric\-supervised critics learn dense behavioral features from sparse real\-world outcomes, showing that structured rubric labels can improve reward modeling and test\-time selection[Wang et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib17)\. Rubrics as Rewards uses structured rubric criteria as reward signals for reinforcement learning beyond domains with easily verifiable answers[Gunjal et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib2)\. Checklist\-based approaches similarly decompose instructions into explicit criteria that can be judged or verified to produce reward signals[Viswanathan et al\. \(2026\)](https://arxiv.org/html/2610.33245#bib.bib16)\. Most closely, RewardAnything introduces principle\-following reward models and RABench to test whether reward models can follow dynamically provided natural\-language principles and generalize to novel principles without retraining[Yu et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib21)\. A related reframing appears in MetaLint, which casts code linting as instruction following over user\-specified best practices instead of classification over a fixed set of issue types[Naik et al\. \(2025\)](https://arxiv.org/html/2610.33245#bib.bib9)\. Like these principle\-following models, the PRA receives its rubrics as input instead of learning them in training\. Unlike them, it judges long\-horizon agent trajectories and the artifacts they produce instead of a single final response, and it can revise the rubrics it receives\.

## Appendix HSelection\-Policy Analysis

Reporting the best attempt assumes that it can be identified after the fact, which is not possible in deployment\. Tables[6](https://arxiv.org/html/2610.33245#A8.T6)and[7](https://arxiv.org/html/2610.33245#A8.T7)report the leaderboard percentile under four ways of choosing one attempt per task: the first attempt, the last attempt, the mean over scored attempts, which is the expected result of choosing an attempt at random, and the best attempt\. The last attempt is the natural deployment choice for a sequential run in a shared workspace, since each attempt can build on the ones before it\.

On DSPredict\-Easy the last attempt improves on the first by 4\.5 points without supervision and by 6\.5 points withAdaPRA, whose last attempt \(62\.9\) comes within 6\.1 points of its best\-attempt score \(69\.0\)\. On DSPredict\-Hard the last attempt improves on the first under every condition\.AdaPRAhas the highest mean and best\-attempt percentiles \(32\.1 and 36\.1\), and its last attempt \(30\.2\) is within 0\.2 points of the highest last\-attempt score \(−\-PRA, 30\.4\)\.

Table 6:Leaderboard percentile onDSPredict\-Easyunder four attempt\-selection policies over the 5\-attempt beam, task\-first averaged over 38 tasks\.Firsttakes the first attempt, with an invalid first attempt contributing 0,Lasttakes the last attempt with a scored submission,Meanaverages the scored attempts \(the expected result of a random choice\), andOracletakes the best attempt\.Table 7:As Table[6](https://arxiv.org/html/2610.33245#A8.T6), onDSPredict\-Hard\(54 tasks\)\.First,Last, andOracleuse the fixed denominator, with unscored tasks contributing 0\.Meanis the per\-attempt percentile of Table[2](https://arxiv.org/html/2610.33245#S5.T2), averaged over tasks with at least one scored attempt, so it can exceedLast\.
## Appendix IPaired Significance of Process Supervision

We test whether each supervised condition improves over the harness alone \(Tables[8](https://arxiv.org/html/2610.33245#A9.T8)and[9](https://arxiv.org/html/2610.33245#A9.T9)\), and we report the same tests forAdaPRAagainst each of its two ablations in the text\. The unit of analysis is the task, and paired differencesdi=conditioni−referenceid\_\{i\}=\\text\{condition\}\_\{i\}\-\\text\{reference\}\_\{i\}cancel differences in task difficulty\. The primary test is a one\-sided paired permutation test on the mean differenced¯\\bar\{d\}\. Under the null hypothesis that the two conditions are exchangeable within each task, eachdid\_\{i\}is equally likely to take either sign, so we compare the observedd¯\\bar\{d\}with its distribution under200,000200\{,\}000random sign flips of thedid\_\{i\}\. Unlike the sign test, this test uses the magnitude of each difference, and unlike the Wilcoxon signed\-rank test, it tests the mean difference that we report without assuming that the differences are symmetric\. We also report the one\-sided sign test, the one\-sided Wilcoxon signed\-rank test, and a nested bootstrap 95% confidence interval, which resamples tasks with replacement and then attempts within each task, so the interval includes variation across attempts\. Both metrics of Tables[1](https://arxiv.org/html/2610.33245#S5.T1)and[2](https://arxiv.org/html/2610.33245#S5.T2)are tested: the best attempt and the mean over a task’s valid attempts\. Percentiles use the private leaderboard where one exists and the public leaderboard otherwise\.

On DSPredict\-Easy \(n=38n=38, Table[8](https://arxiv.org/html/2610.33245#A9.T8)\),AdaPRAhas the largest gains over the harness alone on both metrics \(best attempt\+8\.2\+8\.2,p<0\.001p<0\.001, CI\[\+4\.4,\+14\.4\]\[\+4\.4,\+14\.4\]; per attempt\+9\.7\+9\.7,p<0\.001p<0\.001\)\. The gains of−\-adaptive rubrics are significant on both metrics \(best attempt\+6\.5\+6\.5,p=0\.017p=0\.017; per attempt\+7\.5\+7\.5,p=0\.008p=0\.008\)\. The best\-attempt gain of−\-PRA is not significant \(\+2\.2\+2\.2,p=0\.273p=0\.273\), but its per\-attempt gain is \(\+6\.6\+6\.6,p=0\.009p=0\.009\)\. Compared directly on the best attempt,AdaPRAoutperforms−\-PRA \(\+6\.0\+6\.0; 23 wins, 15 losses;p=0\.012p=0\.012\), while its gain over−\-adaptive rubrics is not significant \(\+1\.7\+1\.7; 22 wins, 16 losses;p=0\.147p=0\.147\)\.

On DSPredict\-Hard \(n=54n=54, Table[9](https://arxiv.org/html/2610.33245#A9.T9)\), every supervised condition improves significantly over the harness alone on both metrics, andAdaPRAhas the largest gains \(best attempt\+10\.6\+10\.6,p<0\.001p<0\.001, CI\[\+5\.3,\+16\.2\]\[\+5\.3,\+16\.2\]; per attempt\+7\.8\+7\.8,p=0\.001p=0\.001, CI\[\+2\.8,\+13\.0\]\[\+2\.8,\+13\.0\]\)\. The gains of−\-PRA are significant on both metrics \(best attempt\+6\.3\+6\.3,p=0\.009p=0\.009; per attempt\+6\.7\+6\.7,p<0\.001p<0\.001\), as are those of−\-adaptive rubrics \(best attempt\+6\.3\+6\.3,p=0\.007p=0\.007; per attempt\+5\.5\+5\.5,p=0\.005p=0\.005\)\. Compared directly on the best attempt,AdaPRAoutperforms−\-adaptive rubrics \(\+4\.3\+4\.3; 32 wins, 13 losses;p=0\.024p=0\.024\), so rubric adaptation adds to code execution on the held\-out benchmark\. Its gain over−\-PRA \(\+4\.3\+4\.3; 29 wins, 18 losses\) is not significant \(p=0\.077p=0\.077\)\. Per attempt, neither direct difference is significant \(\+2\.3\+2\.3,p=0\.138p=0\.138;\+1\.1\+1\.1,p=0\.358p=0\.358\)\.

Table 8:Paired significance of each supervised condition against the harness alone onDSPredict\-Easy\(task\-paired,n=38n=38\)\. The one\-sided paired permutation test \(200,000200\{,\}000sign flips\) is primary, with the one\-sided sign test, the one\-sided Wilcoxon signed\-rank test, and a nested bootstrap 95% CI \(20,00020\{,\}000resamples\) alongside\. Ties are excluded from the win and loss counts\. Bold marks comparisons significant atp<0\.05p<0\.05by the permutation test\.Table 9:As Table[8](https://arxiv.org/html/2610.33245#A9.T8), onDSPredict\-Hard\(task\-paired,n=54n=54\)\. Tasks without a scored attempt enter as 0 so that every task is paired, which makes the per\-attempt deltas slightly larger than the differences in Table[2](https://arxiv.org/html/2610.33245#S5.T2), where per\-attempt means exclude such tasks\.![Refer to caption](https://arxiv.org/html/2610.33245v1/figures/Rubric_Discovery.png)Figure 3:Offline rubric discovery\.The base agent’s attempts on each task are ranked by leaderboard percentile\. An LLM judge \(Claude Sonnet 4\.6\) reads each attempt’s trajectory and workspace, is told whether the attempt ranks in the top or bottom quarter of its task, and extracts good and bad behaviors with quoted evidence\. A second call clusters the good behaviors of top attempts and the issues of bottom attempts across tasks into a MECE set of seed rubrics, with good and bad behaviors as evidence for the same rubric \(prompts and settings in Appendix[E\.3](https://arxiv.org/html/2610.33245#A5.SS3)\)\.
## Appendix JSeed Rubrics

Table[10](https://arxiv.org/html/2610.33245#A10.T10)lists the 19 seed rubrics that initialize every rubric store \(§[3\.3](https://arxiv.org/html/2610.33245#S3.SS3)\), and Tables[11](https://arxiv.org/html/2610.33245#A10.T11)–[15](https://arxiv.org/html/2610.33245#A10.T15)give their full natural\-language definitions\. The PRA receives each rubric as its one\-sentence definition followed by its four labels \(satisfied, violated, not applicable, insufficient evidence\) and a note stating the rubric’s evidence source, which is phrased as a recommendation so that the PRA may use either kind of evidence\. Seventeen rubrics come from offline discovery on harness\-only DSPredict\-Easy runs\. Four of them are restated to reason over all earlier attempts on the task, and two further rubrics are hand\-curated\. Stage one mines each attempt independently, so it cannot surface failures that appear only across attempts, such as later attempts converging on the same solution; the two hand\-curated rubrics target these failures\. They account for 11–17% of rubric failures across the supervised runs, and they are the only failing rubrics in 1–3% of reviews that report a failure\. Six rubrics refer to the transcripts of earlier attempts, which the PRA reads on demand \(§[3\.4](https://arxiv.org/html/2610.33245#S3.SS4)\)\. The rubric names and definitions are reproduced verbatim; “best\-of\-N” in them refers to theNNsequential attempts on a task, of which the best is scored\.

Table 10:Seed rubric metadata\.*Evidence*: whether the rubric is decisively checkable from the workspace artifacts or requires the trajectory \(11 artifact, 8 trajectory\)\.*Source*:*Discovered*rubrics come from two\-stage discovery on harness\-only DSPredict\-Easy runs;*Discovered, restated*rubrics were restated to judge the current attempt against all earlier attempts on the task;*Hand\-curated*rubrics target cross\-attempt failures\.*Support*: number of the 519 behaviors from the top and bottom of each task’s score range \(238 good behaviors, 281 issues\) that stage two clustered into the rubric\.*Expert match*: whether the rubric recovers a criterion of a hand\-written expert rubric set \(Yes\), overlaps with one loosely \(Partial\), or has no expert counterpart \(No\)\.Table 11:Seed rubric definitions: validation and metric\. Text is verbatim as given to the PRA;*Context*is a preamble that precedes the labels\.Table 12:Seed rubric definitions: exploration and model diversity\. Text is verbatim as given to the PRA\.Table 13:Seed rubric definitions: preprocessing, target, and features\. Text is verbatim as given to the PRA\.Table 14:Seed rubric definitions: efficiency, robustness, and submission \(part 1 of 2\)\. Text is verbatim as given to the PRA\.Table 15:Seed rubric definitions: efficiency, robustness, and submission \(part 2 of 2\)\. Text is verbatim as given to the PRA\.
## Appendix KSeed Rubric Failures Across Benchmarks

Table[16](https://arxiv.org/html/2610.33245#A11.T16)compares the 19 seed rubrics \(Appendix[J](https://arxiv.org/html/2610.33245#A10)\) in the−\-adaptive rubrics runs on both benchmarks\. Both runs score the same rubrics at every review, so the rates are directly comparable\. A rubric fails in a review when it crosses the injection thresholds \(Table[5](https://arxiv.org/html/2610.33245#A3.T5)\)\. The analysis covers 190 DSPredict\-Easy attempts and 270 DSPredict\-Hard attempts\. Across the 19 rubrics, Spearman correlations between benchmarks areρ=0\.66\\rho=0\.66\(p=0\.002p=0\.002\) for attempt failure rate andρ=0\.91\\rho=0\.91\(p<10−7p<10^\{\-7\}\) for mean importance\.

Table 16:Seed rubric failures in the frozen\-rubric runs\.*Failed attempts*: share of attempts in which the rubric failed at least one review\.*Mean importance*: the PRA’s mean importance score for the rubric over all reviews\.#### Examples of transferred rubrics\.

Table[17](https://arxiv.org/html/2610.33245#A11.T17)pairs seven failures that the PRA flagged on non\-tabular DSPredict\-Hard tasks with the same rubric failing on a tabular DSPredict\-Easy task\. Each example comes from the frozen\-rubric run, so the rubric text is exactly as discovered on DSPredict\-Easy\. We count a flag as*resolved*when the PRA scores a rubric at importance≥0\.6\\geq 0\.6and success probability≤0\.2\\leq 0\.2at one review, and at success probability≥0\.7\\geq 0\.7at a later review of the same attempt; for every example in the table we read the later rationale to confirm that the agent fixed the issue\. Across the DSPredict\-Hard frozen\-rubric run, 685 of 1,277 flags \(54%\) are resolved within the attempt, on 53 of the 54 tasks\. Resolution rates are highest for rubrics that check the pipeline \(submission integrity 83%, regression detection 82%, best\-model selection 79%, metric bugs 67%, valid cross\-validation 55%\) and lowest for the simplicity\-bias rubrics \(model diversity 16%, single\-model\-class hedging 22%, pivoting when experiments stall 28%\)\. Flagged pipeline errors are usually fixed within the attempt, while flagged strategy problems change the agent’s behavior less often\.

Table 17:Seed rubrics discovered on DSPredict\-Easy catching the same process failures on non\-tabular DSPredict\-Hard tasks, from the frozen\-rubric runs\. Each DSPredict\-Hard flag was resolved later in the same attempt, confirmed from the PRA’s rationale at the later review\. Quoted phrases are from the PRA’s rationales\.

## Appendix LRubric Store Operations

Figure[4](https://arxiv.org/html/2610.33245#A12.F4)counts the operations that the PRA applies to each task’s rubric store \(§[3\.5](https://arxiv.org/html/2610.33245#S3.SS5)\) inAdaPRAand its−\-PRA ablation, taken from the operation log of each store\. Adaptation consists almost entirely of additions\. Across the four runs the PRA adds 177 rubrics, updates one, and retires none\. Between 81% and 94% of additions occur in the first two attempts, so later attempts are reviewed against a largely stable store\. The−\-PRA ablation adds rubrics at a similar rate \(0\.76 vs\. 0\.84 per task on DSPredict\-Easy and 1\.10 vs\. 1\.21 on DSPredict\-Hard\), so removing code execution does not reduce how often the store is edited\.

Figure 4:Rubric store operations per task forAdaPRAand the−\-PRA ablation\. \(a\) Operations by type, with raw counts above the bars\. \(b\) Additions by the attempt in which they occur\. Rates are over tasks with an operation log \(DSPredict\-Easy: 37 and 38 tasks; DSPredict\-Hard: 52 and 49 tasks, forAdaPRAand−\-PRA\)\.Figure[5](https://arxiv.org/html/2610.33245#A12.F5)breaks these operations down by run and attempt, and Figure[6](https://arxiv.org/html/2610.33245#A12.F6)shows how many rubrics each task’s store gains\. With code execution the PRA adds at least one rubric on 79% of DSPredict\-Hard tasks, compared with 55% for−\-PRA, while on DSPredict\-Easy the shares are 51% and 63%\. Table[18](https://arxiv.org/html/2610.33245#A12.T18)groups the added rubrics by what they check, a grouping we assigned by reading each rubric\. Prediction\-sanity rubrics, which compute statistics of the agent’s predictions such as their dispersion relative to the target, account for 35% \(DSPredict\-Easy\) and 19% \(DSPredict\-Hard\) of additions with code execution and 3% and 7% without it, and the−\-PRA ablation instead adds mostly train–test and submission consistency rubrics \(41% and 44%\)\.

Figure 5:Rubric store operations per attempt for each run, stacked by type\. Panel titles give the number of tasks with an operation log\. The only update occurs in the first attempt ofAdaPRAon DSPredict\-Easy, and no run retires a rubric\.Figure 6:Share of tasks by the number of rubrics the PRA adds to the task’s store over its five attempts, over tasks with an operation log\.Table 18:Rubrics added by the PRA, grouped by what they check, as counts per run\. Themes were assigned by hand from each rubric’s name and description\. Each example gives the task, and−\-PRA marks rubrics added by the ablation\.
## Appendix MExample PRA Feedback

This appendix reproduces PRA feedback from theAdaPRAruns on DSPredict\-Easy and DSPredict\-Hard, with the agent’s response and the leaderboard outcome of the attempt\. Each PRA box shows the failed rubrics as the agent received them \(§[3\.4](https://arxiv.org/html/2610.33245#S3.SS4)\)\. Each agent box shows the agent’s subsequent actions, with each step labeled by turn and long passages abridged as “\[…\]”\. Attempts are numbered from 1 to 5, percentiles are on the private leaderboard unless the competition has only a public one, and CV denotes the agent’s local cross\-validation score\.

#### Execution grounding exposes an in\-sample calibration gain\.

In*Feature Imputation with a Heat Flux Dataset*, the agent fitted isotonic regression on its full set of XGBoost out\-of\-fold predictions, scored it on the same rows, and submitted it because the RMSE fell from 0\.073994 to 0\.073592\. The PRA reran the calibration under nested cross\-validation and found an RMSE of 0\.074386, worse than the uncalibrated model\. It then added a task\-specific rubric,calibration\_not\_insample, with a natural\-language definition and a check script \(calibration\_insample\_check\.py\) that fits the calibrator on four folds and scores the fifth \(§[3\.5](https://arxiv.org/html/2610.33245#S3.SS5)\)\. The agent reverted the submission in the next two turns and finished the attempt with a Ridge stack of LightGBM, XGBoost, and CatBoost \(CV 0\.073815\)\. The private percentile rose from 49\.7 in attempt 1 to 62\.2 in attempt 2, while the public percentile was unchanged \(52\.4 and 52\.0\)\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*ValidCross\-ValidationMethodology\*\*\[Importance:0\.90\|Success:0\.10\]:

\-Problem:CRITICAL:AgentfittedIsotonicRegressiononfull21229XGBoostOOF

predictionsANDevaluatedonthesamedata\.In\-sampleRMSE=0\.073592\.Proper

nested\-CVRMSE=0\.074386\(measuredbyreviewer\)\.\[…\]FAIL\.

2\.\*\*Best\-CVModelSubmissionSelection\*\*\[Importance:0\.90\|Success:0\.10\]:

\[…\]

\-Suggestedaction:Revertsubmissionimmediately:loadpreds\_xgb\.parquetandwrite

to/submission/submission\.csv\.TruecalibratedRMSE=0\.074386\>registered

xgb=0\.073994\.

\[…\]

4\.\*\*calibration\_not\_insample\*\*\[Importance:0\.90\|Success:0\.05\]:

\-Definition:Whetherpost\-hoccalibrationlayers\(isotonicregression,Platt

scaling,temperaturescaling\)appliedtomodelpredictionsarevalidatedusinga

properheld\-outornestedcross\-validationscheme\-NOTbyfittingandevaluating

onthesameOOFdata\.\[…\]Failurelookslike:in\-samplecalibrationRMSEappears

betterthanuncalibratedOOFRMSE,butnested\-CVcalibrationRMSEisWORSE\[…\]

\-Problem:calibration\_insample\_check\.py:UncalibratedRMSE=0\.073997\.In\-samplecal

RMSE=0\.073592\(OPTIMISTIC\)\.Nested\-CVtrueRMSE=0\.074386\(WORSE\)\.Agentused

in\-sampleresulttosubmit\.FAIL\-calibrationoverfitsisotonicregressionon

fullOOFdata\.

\[…\]

\[turn61\]ThePRMfeedbackisveryclearandcritical\.Ineedto:

1\.RevertthesubmissiontotheuncalibratedXGBoost\(bestmodelwithCV=0\.073994\)

2\.Neverusein\-samplecalibration\-neednested\-CVforpropervalidation

\[…\]

\[turn62\]

xgb\_test=pd\.read\_parquet\(’/workspace/playground\-series\-s3e15/preds/preds\_xgb\.parquet’\)

\[…\]

submission\.to\_csv\(’/submission/submission\.csv’,index=False\)

print\("RevertedtouncalibratedXGBoostsubmission\(bestmodel\)"\)

\[…\]

\[turn79\]Excellent\!TheRidgeensembleachievedaCVscoreof0\.073815\+/\-0\.000715,

whichisbetterthanthebestsinglemodel\(XGB:0\.073994\+/\-0\.000689\)\.\[…\]

\[…\]

\[finalanswer\]

\#\#ValidationScore

0\.073815\(RMSE\)\-ensemble\_ridgemodel

#### A task\-specific rubric bounds a weak formulation before it runs\.

In the*TGS Salt Identification Challenge*, the agent began attempt 1 by training logistic regression on global image statistics to predict whether an image contains salt, and planned to submit an all\-salt or all\-empty mask per image\. Before this model produced a score, the PRA computed the salt fill fraction of every training mask and derived that even a perfect image\-level classifier is capped at 0\.52 mean average precision under the competition’s IoU thresholds\. It recorded this bound as a new rubric,pixel\_segmentation\_vs\_image\_classification, backed bycheck\_segmentation\_approach\.py\. The agent replaced the classifier with a U\-Net trained with per\-pixel cross\-entropy within five turns\. The rubric passed in all 21 subsequent reviews recorded for the task, which confirmed from the submitted run\-length encodings that every attempt predicted image\-specific masks\. Two further rubrics added in attempt 1 had a similar lasting effect\. One of them \(oof\_is\_fold\_predictions\_not\_full\_model\) caught at turn 100 that the saved out\-of\-fold array came from the model refit on all data, since it reproduced a score of 0\.1654 against a reported fold CV of 0\.5005\. Its check then verified every later CV to within10−410^\{\-4\}\. Attempt 1 itself, with a five\-epoch U\-Net, reached a private percentile of only 10\.1\. Later attempts built on its pipeline with ResNet encoders, reached CV above 0\.70, and scored private percentiles of 34\.3 to 36\.6\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*pixel\_segmentation\_vs\_image\_classification\*\*\[Importance:0\.85\|Success:0\.20\]:

\-Definition:Whethertheagentproducespixel\-levelsegmentationmasksratherthan

image\-levelclassificationfortheTGSsaltchallenge\.Thecompetitionmetricis

meanAPoverIoUthresholds0\.5\-0\.95,whichrequiresaccuratepixel\-levelmasks\.An

image\-levelclassifierthatpredictsall\-onesorall\-zerosmaskshasamathematical

ceilingof~0\.52AP\(measured:meansaltfill=0\.407,fractionwithfill\>=0\.5=0\.382\)

vs~0\.88achievablewithU\-Net\-stylepixelsegmentation\-agapof0\.36\.\[…\]

\-Problem:check\_segmentation\_approach\.py:agentusesLogisticRegressiononglobal

imagestatistics\(mean,std,percentiles\)toclassifyimagesassalt/no\-salt,then

submitsall\-onesorall\-zeros101x101masks\.\[…\]Perfectimage\-levelclassifier

APceiling=0\.5228vs~0\.88forU\-Netsegmentation\.Gap=0\.357\.

\-Suggestedaction:Replaceimage\-levelLRclassificationwithapixel\-level

segmentationmodel\(e\.g\.U\-NetwithaCNNencoder\)\.Globalimagestatisticscannot

predictwheresaltpixelsare\-themodelmustoutputa101x101maskperimage,

notabinarylabel\.

2\.\*\*TrainingObjective\-MetricAlignment\*\*\[Importance:0\.80\|Success:0\.20\]:

\[…\]

\-Suggestedaction:Useapixel\-levelsegmentationloss\(e\.g\.,binarycross\-entropy

perpixel,Diceloss\)totrainamodelthatoutputsa101x101mask,notascalar

probability\.

\[turn41\]Thefeedbackiscritical\-Ineedtobuildapixel\-levelsegmentationmodel,

notanimage\-levelclassifier\.\[…\]Ineedto:

1\.Trainamodelthatoutputsa101x101maskperimage

2\.Usepixel\-levelloss\(binarycross\-entropyperpixel,Diceloss\)

\[…\]

\[turn42\]\[str\_replace\]train\_baseline\.py

\-UsesLogisticRegressionwithimageanddepthfeatures\.

\+UsesasimpleCNNforpixel\-levelsaltsegmentation\.

\[…\]

\[turn45\]TheU\-Nethasasizemismatchissue\.\[…\]

\-After3maxpooloperations\(stride2\):101\-\>50\-\>25\-\>12\(floordivision\)

\[…\]

\[turn51\]ThemodeltrainedsuccessfullywithaCVscoreof0\.4815\.\[…\]

#### The PRA prevents a revert caused by the agent’s own diagnostic bug\.

In*Multi\-Class Prediction of Obesity Risk*, attempts 1 and 2 had trained only gradient\-boosted trees, and in attempt 3 the score registry had fallen out of sync with the prediction files\. The current submission came from XGBoost with an out\-of\-fold accuracy of 0\.909240, but the agent’s own accuracy script compared class labels against column names carrying aprob\_prefix and reported XGBoost as 0\.000000\. The PRA reproduced the correct accuracy from the saved predictions, located the prefix bug, and warned the agent against reverting to the weaker CatBoost model\. The agent applied the one\-line fix, obtained 0\.909240, and corrected the registry within four turns\. It followed the PRA’s diversification requests by training an MLP, which timed out, and a logistic regression \(CV 0\.8569\), which the PRA’s review at turn 50 found would lower accuracy in a hard vote with XGBoost\. The final submission was a 0\.9/0\.1 soft vote of XGBoost and CatBoost with CV 0\.910059, a weighting the PRA had computed from the cached predictions at turn 60\. The private percentile rose from 49\.4 in attempt 2 to 73\.3 in attempt 3\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*CVRegressionDetectionandRevert\*\*\[Importance:0\.95\|Success:0\.15\]:

\-Problem:MOSTURGENT:Agent’sturn28codeshowsxgb\_baselineOOF=0\.000000\(bug

describedabove\)\.Agentatturn29\-30isinvestigatingtestpredictions\-likely

abouttoconcludeXGBoostisbrokenandrevertsubmissiontocatboost\_baseline

\(0\.9038\)\.Thiswouldbeaself\-inflictedregressionof\-0\.0054fromthecurrent

correctXGBoostsubmission\.TheXGBoostsubmissionIScorrect\-theOOFcomputation

codeiswrong,notthemodel\.\[…\]DONOTREVERT\.

\-Suggestedaction:STOPbeforereverting\.Run:\[…\]youwillsee0\.909240,

confirmingXGBoostistheBESTmodel\.Keepthecurrentsubmission\.

2\.\*\*PersistentCVScoreRegistry\*\*\[Importance:0\.92\|Success:0\.15\]:

\-Problem:cv\_scores\.jsonSTILLmissingxgb\_baseline\.best\_ever=catboost\_baseline=

0\.903796\.Agentatturn28computedXGBoostOOF=0\.000000\(bug:

oof\[prob\_cols\]\.idxmax\(\)giveslabelslike’prob\_Normal\_Weight’,comparedagainst

’Normal\_Weight’withoutstripping’prob\_’prefix\-allmismatches\-\>accuracy=0\)\.

ActualXGBoostOOF=0\.909240\(reviewer\-verifiedfromparquet\)\.\[…\]

\-Suggestedaction:FixtheXGBoostOOFcomputation:oof\[’pred’\]=

oof\[prob\_cols\]\.idxmax\(axis=1\)\.str\.replace\(’prob\_’,”,regex=False\)\.\[…\]DoNOT

re\-trainXGBoost\-thepredsarecorrect;onlytheregistryismissingtheentry\.

\[…\]

\[turn31\]Theexpertfeedbackconfirmsthat:

1\.XGBoostOOF=0\.909240\(bestmodel\)\-IhadabuginmyOOFcomputationcode

2\.ThesubmissionISfromXGBoostandiscorrect

3\.cv\_scores\.jsonismissingthexgb\_baselineentry

4\.IshouldNOTreverttocatboost\_baseline

\[turn32\]

oof\[’pred’\]=oof\[prob\_cols\]\.idxmax\(axis=1\)\.str\.replace\(’prob\_’,”,regex=False\)

\[…\]

\>\>XGBoostOOFaccuracy\(corrected\):0\.909240

\[turn34\]

cv\[’best\_ever’\]=\{

’model\_name’:’xgb\_baseline’,

’cv\_score’:0\.909240,

\[…\]

#### The finish gate catches a CV that the submission does not reproduce\.

In*Digit Recognizer*, the agent called finish at turn 44 with a ResNet\-18 and a reported CV of 0\.9922\. At the finish review, the PRA recomputed accuracy from the saved out\-of\-fold predictions \(0\.9899\) and traced the gap to the training script\. The script recorded best\-epoch validation accuracy as the CV score but generated out\-of\-fold and test predictions from the last\-epoch weights, and it never reloaded the best\-epoch checkpoints\. The failed review returned the attempt to the agent for its one refinement round \(§[3\.4](https://arxiv.org/html/2610.33245#S3.SS4)\), in which the agent reloaded the checkpoints, retrained, and finished with an out\-of\-fold accuracy of 0\.9912 that matched the reported score\. The attempt reached a public percentile of 87\.0 \(accuracy 0\.99482; the competition has no private leaderboard\), up from 73\.3 for attempt 4 \(a three\-layer CNN\) and 21\.7 to 22\.1 for attempts 1 to 3\. The gain over attempt 4 also reflects the change of architecture, so it cannot be attributed to the fix alone\. Earlier in the same attempt \(turn 20\), the PRA had found that the submission file matched the XGBoost predictions row for row while the CNN held the best CV, and the agent restored the CNN submission in the next turn\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*ValidCross\-ValidationMethodology\*\*\[Importance:0\.90\|Success:0\.15\]:

\-Problem:ResNetOOFreproduced=0\.989881vsreported=0\.992190,delta=0\.002309\.Root

causeconfirmedintrain\_resnet\.py\(lines140\-203\):cv\_scores\.append\(

best\_val\_acc/100\.0\)recordsbest\-epochvalidationaccuracy,butOOF/testinference

\(lines195,test\_predictions\+=test\_fold/n\_splits\)usestheCURRENT

\(last/early\-stopped\)model\-bestweightsaresavedto\.fold\{N\}filesbutnever

reloadedbeforeinference\.SothereportedCV=0\.9922describesthebest\-epoch

model,whilesubmission/OOFcomefromthelast\-epochmodel\(0\.9899\)\.\[…\]

\-Suggestedaction:Intrain\_resnet\.py,afterthetrainingloopforeachfold,add:

‘model\.load\_state\_dict\(torch\.load\(f’\{MODEL\_PATH\}\.fold\{fold\+1\}’\)\)‘then

‘model\.eval\(\)‘BEFOREcomputingoof\_foldandtest\_fold\.ThismakesOOF/submission

correspondtobest\-epochweightsandthereportedCVwillbereproducible\.

2\.\*\*MetricCalculationBugDetection\*\*\[Importance:0\.70\|Success:0\.25\]:

\-Problem:Trainingoutput\(turn41\)printedboth’FinalCVAccuracy:0\.992190’and

’OOFCVAccuracy:0\.989881’inthesamerun\.\[…\]Agentreported0\.9922without

investigatingwhyOOFCVdifferedby0\.0023\.

\[…\]

\[turn45\]\[…\]Ineedtofixthetrain\_resnet\.pyscriptto:

1\.LoadthebestweightsforeachfoldbeforecomputingOOFandtestpredictions

2\.UsetheOOFCV\(0\.9899\)asthereportedscoresincethat’swhat’sactuallybeing

submitted

\[turn46\]\[str\_replace\]train\_resnet\.py

\+\#Loadbestweightsforthisfold

\+model\.load\_state\_dict\(torch\.load\(f"\{MODEL\_PATH\}\.fold\{fold\+1\}"\)\)

\+model\.eval\(\)

\[turn47\]timeout1800pythontrain\_resnet\.py2\>&1\|teetrain\_resnet\_v2\.log

\[turn48\]NowtheOOFCVmatchesthereportedCV\(0\.9912\)\.\[…\]

\>\>Bestmodel:resnetwithCV=0\.991167

\[turn49\]\[…\]ResNetsubmissionsaved

\[turn50,finalanswer\]

\#\#ValidationScore

0\.9912\(CategorizationAccuracywith5\-foldCV\)

#### Cross\-attempt review breaks a collapsed beam\.

In*Titanic*, all four earlier attempts submitted the same CatBoost model \(CV 0\.8552\), and the last three of them were identical on all 418 test rows\. At turn 10 of the final attempt, the PRA read the earlier attempts’ trajectories and prediction files, reported the collapse, and quantified how far the cached predictions of other model families departed from CatBoost \(26 to 41 of 418 rows\)\. The agent built a weighted vote over the cached test predictions that differed from CatBoost on 26 rows\. The public accuracy rose from 0\.77033 to 0\.77511 and the public percentile from 32\.8 to 52\.4 \(the competition has no private leaderboard\)\. The agent spent six turns on an ensemble\-weight search, which the PRA had advised against, and it never computed an out\-of\-fold score for the ensemble\. Its registry entry reuses the CatBoost value as a proxy\. The PRA raised this at later reviews and at the finish gate, and the attempt ended with the issue unresolved\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*HedgingAcrossBest\-of\-NWithDiverseModelClasses\*\*\[Importance:0\.95\|Success:

0\.20\]:

\-Problem:Thisisattempt5of5\(final\)\.All4priorattemptssubmitted

catboost\_baseline\(0\.8552\):attempts2,3,4areidentical\(0/418predictiondiff\)\.

Thebeamprovideszerodiversity\-thebest\-of\-5maxissimply0\.8552nomatter

whichattempt’wins’\.\[…\]

\-Suggestedaction:Submitapredictionsetthatdiffersmeaningfullyfrom

catboost\_baseline\.Usethealready\-cachedpredstobuildavotingensemble

\(catboost\+gradient\_boosting\+logreg=26\-41/418differ\),ortrainafreshmodel

withacompletelydifferentfeaturerepresentation\[…\]\.EvenifCVis0\.001

lower,adiversepredictionisworthmoretothebeamthananidenticalcopy\.

2\.\*\*AvoidingChurnWithoutPivoting\*\*\[Importance:0\.90\|Success:0\.25\]:

\-Problem:All4priorattemptsconvergedoncatboost\_baseline\(0\.8552\)\-theultimate

formofbeamchurn\.\[…\]Attempt4triedweightedensemble\-weightsearchfor15\+

turns\(historicallyfailingaxis\)\.\[…\]

\-Suggestedaction:\[…\]DoNOTrunmoreHPOorensemble\-weightsearchonthesame

OOFfiles\.

3\.\*\*ModelDiversityandEnsembling\*\*\[Importance:0\.85\|Success:0\.25\]:

\[…\]

\[turn11\]Theexpertfeedbackisveryclear\.Ineedto:

1\.Tryagenuinelydifferentapproach\-NOTcatboost\_baselineagain

2\.Usethealready\-cachedpredictionstobuildavotingensemble\[…\]

\[turn12\]

\>\>gradient\_boosting:26/418differentfromCatBoost

\>\>logreg\_baseline:41/418differentfromCatBoost

\>\>random\_forest:16/418differentfromCatBoost

\[…\]

\[turn21\]

\>\>CVScore\(proxy\):0\.8552\+/\-0\.0067

\[…\]

\>\>DifferentfromCatBoostbaseline:7/418samples

\[…\]

\[turn28\]Goodprogress\!Theweightedensemblesearchfoundweightsthatproduce26/418

differentpredictionsfromCatBoost\.\[…\]

\[turn30\]

\>\>"weighted\_ensemble\_search":\{

\>\>"cv\_mean":0\.8552,

\[…\]

\>\>"note":"DiffersfromCatBoostin26/418samples"

#### Diversity feedback breaks a duplicated baseline, and an executed check validates the ensemble\.

In*Regression of Used Car Prices*, attempts 1 and 2 submitted the same LightGBM baseline \(CV RMSE 77,242\.70, private percentile 9\.7\)\. At turn 10 of attempt 3, the PRA failed both cross\-attempt rubrics and the model\-diversity rubric\. It reported that two of the five submissions were identical and that every model that had completed training was a LightGBM, XGBoost, or CatBoost variant\. From the saved predictions it also measured that the baseline’s predictions had 0\.148 times the standard deviation of the target\. The agent first tried the suggested CatBoost model with native categorical features, which exceeded its 300\-second timeout, and then trained scikit\-learn’s HistGradientBoosting, the PRA’s other suggestion\. Although this is also a gradient\-boosted tree model, it lowered CV to 73,209\. After the turn\-20 review reported that the model still underpredicted expensive cars, the agent added fold\-isolated brand and model price encodings and finished with CV 73,185 and a private percentile of 43\.6, equal to the best attempt of the harness\-only run on this task\. In attempt 4, the agent submitted a near\-equal\-weight average of HistGradientBoosting, LightGBM, and XGBoost without scoring it\. At turn 31, the PRA estimated the average’s CV at 75,876, worse than HistGradientBoosting alone \(73,174\)\. It also found that the largest submitted prediction was $145K while the largest out\-of\-fold prediction was $443K\. The agent reverted to HistGradientBoosting, saved out\-of\-fold predictions for each base model, and trained a Ridge stack on them \(CV 72,590 at turn 55\)\. The attempt reached a private percentile of 80\.5\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*HedgingAcrossBest\-of\-NWithDiverseModelClasses\*\*\[Importance:0\.90\|Success:

0\.15\]:

\-Problem:3of5attemptsconvergingonidenticalsolution:LGBMbaselineCV=77,242,

meanpred=$32,146,std=$11,657,max=$80,508\.Attempts1and2submittedthesame

preds\_lgbm\_baseline\.csvpredictions\.Beamwastes2of5shotsreproducingthesame

result\.\[…\]

\-Suggestedaction:Attempt3MUSTsubmitagenuinelydifferentprediction

distribution\.\[…\]Evenifcvissimilaritproducesadifferenterrorpattern

thattheMAXwillexploitacrossthe5\-attemptbeam\.

2\.\*\*ModelDiversityandEnsembling\*\*\[Importance:0\.85\|Success:0\.15\]:

\[…\]

\-Suggestedaction:ThisattemptMUSTtryafundamentallydifferentmodelingclass:

CatBoostwithnativecategoricalfeatures\(brand,model,fuel\_typeascat\_features

\-notlabel\-encoded\),orHistGradientBoosting\.Donotre\-runLGBM/XGBoostvariants\.

3\.\*\*AvoidingChurnWithoutPivoting\*\*\[Importance:0\.85\|Success:0\.30\]:

\-Problem:Attempts1and2bothtriedthesame4GBDTs\(lgbm,xgb,lgbm\_v2,

catboost\)withzeronetimprovement\-allendedatLGBM77,242\.\[…\]

4\.\*\*model\_underfitting\_signal\*\*\[Importance:0\.85\|Success:0\.15\]:

\[…\]

\-Problem:LGBMpreds:std=$11,657vstrainstd=$78,820\-\>dispersionratio=0\.148

\(threshold\>0\.5\)\.Maxpred=$80,508vstruemax=$2,954,083\.\[…\]

\[…\]

\[turn11\]Theexpertfeedbackisveryclear:

1\.\*\*HedgingAcrossBest\-of\-N\*\*:MustuseDIFFERENTmodelingclasses\.All3attempts

usedGBDT\(LGBM/XGBoost/CatBoost\)\.Needtotrysomethingfundamentallydifferent\.

\[…\]

\[turn13\]NowletmecreateanewtrainingscriptusingCatBoostwithnative

categoricalfeatures:

\[…\]

\[turn15\]CatBoostistakingtoolong\.Letmetryafasterapproachwith

HistGradientBoosting:

\[…\]

\[turn17\]Excellent\!HistGradientBoostingachievedamuchbetterCVscoreof

\*\*73,209\.26\*\*comparedtothepreviousbestof77,242\.70\!\[…\]

\[…\]

\[turn22\]LetmeupdatetheHistGBscripttoincludefold\-isolatedbrand\_premiumand

model\_premiumfeatures:

\[…\]

\[turn25\]TheCVscoreimprovedslightlyfrom73,209\.26to\*\*73,185\.30\*\*\.\[…\]

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*Best\-CVModelSubmissionSelection\*\*\[Importance:0\.95\|Success:0\.10\]:

\-Problem:Submission=ensemble\_avg\(noCVinregistry\)\.histgbbestvalidated

CV=73,174\.AgentsubmittedensembleoverhistgbwithoutanyOOFCVcomparison\.

NaivelinearCVestimateforensemble=75,876\(WORSEthanhistgb73,174\)\.\[…\]

\-Suggestedaction:\[…\]OnlyoverwritewithensembleifitsOOFCViscomputedand

confirmed<73,174\.

\[…\]

3\.\*\*oof\_sub\_distribution\_divergence\*\*\[Importance:0\.95\|Success:0\.10\]:

\[…\]

\-Problem:OOF\(histgb\)mean=$36,747,max=$443,469\.Sub\(ensemble\_avg\)mean=$33,739,

max=$145,200\.\[…\]ThehistgbCV\(73,174\)doesNOTdescribetheensemble

submission\.FAIL:submissiondistributiondivergesseverelyfromwhatwas

validated\.

\[…\]

\[turn32\]Theexpertfeedbackisveryclear\.Ineedto:

1\.Revertsubmissiontohistgb\(bestvalidatedmodelwithCV=73,174\)

2\.Retrainlgbm\_baselinewithOOFsavingtoenableproperstacking

3\.OnlysubmitensembleifitsOOFCVisvalidatedandbetterthanhistgb

\[…\]

\[turn49\]\[…\]createastackingensemblescriptthat:

1\.LoadshistgbOOFandlgbmOOFasmeta\-features

2\.UsesRidgeregressionasameta\-learner

3\.ComputesOOFCVfortheensemble

4\.OnlysubmitsiftheensembleCVisbetter

\[…\]

\[turn55\]Excellent\!ThestackingensembleimprovedtheCVscorefrom73,174to

72,590\.\[…\]

#### Cross\-attempt review adds an image model to a beam of feature\-based models\.

In the*Statoil/C\-CORE Iceberg Classifier Challenge*, the agent classified two\-band75×7575\\times 75radar images from hand\-crafted band statistics in its first three attempts, using logistic regression with a random forest, a stack of the two, and CatBoost \(CV log loss 0\.303, private percentile 21\.8\)\. Early in attempt 4 it wrote a CNN on the raw pixels, found that TensorFlow was not installed, and returned to stacking tree models on the same features\. At turn 80, the PRA reported that no attempt had trained a model on the images themselves and that the current attempt’s stack had only matched attempt 1 \(CV 0\.3766\)\. The PRA, which labels attempts from A0, recommended a CNN for the final attempt\. The agent instead wrote a PyTorch CNN in the current attempt \(CV 0\.344\)\. At turn 100, the PRA found the agent retraining the CNN and computed from the cached out\-of\-fold predictions that a 0\.45/0\.55 blend of CatBoost and the CNN reaches 0\.2407\. The agent recomputed the blend, recorded it in the score registry, and submitted it\. The private percentile rose from 21\.8 in attempt 3 to 52\.3\. Attempt 5 reached a lower CV \(0\.232\) but a private percentile of only 3\.7, so the task’s best\-of\-5 result comes from attempt 4\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*HedgingAcrossBest\-of\-NWithDiverseModelClasses\*\*\[Importance:0\.85\|Success:

0\.55\]:

\-Problem:A0=LR\+RF,A1=Stacking\(LR\+RF\),A2=CatBoost\(best=0\.3032\),

A3=stacking\_v4\(0\.3766\-sameasA0\)\.4attempts,alltabularstatisticalfeatureson

GBDT/linear\.NoCNN\.Beamhaszerodeeplearningcoverage\.1attemptremaining\.

train\_cnn\.pyexistsbutwasneversuccessfullytrained\.

\-Suggestedaction:Finalattempt\(A4\)musttrainaCNNonthe75x75x2rawpixel

arrays\.Thisistheonlygenuinelydifferentmodelclassavailable\[…\]

2\.\*\*ModelDiversityandEnsembling\*\*\[Importance:0\.80\|Success:0\.58\]:

\[…\]

\-Suggestedaction:Usethefinalattempt\(A4\)totrainaCNNontheraw75x75x2

imagearrays\-theonlymodelclassnottriedintheentirebeam\.

\[…\]

\[turn81\]Theexpertfeedbackisveryclear\.Ineedto:

1\.TryaCNNapproachwhichistheonlymodelclassnottriedyet

2\.Thetrain\_cnn\.pyexistsbutwasneversuccessfullytrained

\[…\]

\[turn85\]TensorFlowisnotavailable\.\[…\]

\[turn87\]Great\!PyTorchisavailable\.IcancreateaCNNusingPyTorch\.\[…\]This

willbeafundamentallydifferentapproachfromthepreviousattemptsthatused

statisticalfeaturesontabulardata\.

\[turn88\]\[file\_create\]train\_cnn\_pytorch\.py

\[…\]

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*Best\-CVModelSubmissionSelection\*\*\[Importance:0\.95\|Success:0\.45\]:

\-Problem:Submission=CatBoost\(max\_diff=0\.0,CV=0\.303161\)\.HoweverCatBoost\+CNN

blendatw\_cnn=0\.55givesOOFCV=0\.240659\-improvementof0\.0625\(21%better\)\.

\[…\]AgentisrunningCNNa3rdtimeinsteadofcomputingthisavailableblend\.

\-Suggestedaction:IMMEDIATELYcomputeandsubmitblend:blend=

0\.45\*preds\_catboost\+0\.55\*preds\_cnn,writeto/submission/submission\.csv\.

ExpectedCV=0\.240659vscurrent0\.303161\.

2\.\*\*ModelDiversityandEnsembling\*\*\[Importance:0\.95\|Success:0\.40\]:

\-Problem:CatBoost\(45%\)\+CNN\(55%\)blendgivesOOFCV=0\.240659\-measuredfrom

oof\_predictions\_catboost\.csvandoof\_predictions\_cnn\.csv\.\[…\]Thediverse

CatBoost\(feature\-engineered\)\+CNN\(rawpixels\)combinationcaptures

complementarysignal\.

\[…\]

\[turn101\]Theexpertfeedbackisveryclear\.Ineedto:

1\.StoprunningCNNtrainingagain

2\.Computetheblend:0\.45\*preds\_catboost\+0\.55\*preds\_cnn

3\.Writethisto/submission/submission\.csv

\[…\]

\[turn108\]\[file\_create\]compute\_blend\.py

\[…\]

\[turn115\]Perfect\!Thecv\_scores\.jsonhasbeenupdatedwiththecatboost\_cnn\_blend

modelshowingaCVscoreof0\.240659\.\[…\]

#### An added rubric can encode an unattainable target\.

In*Regression with a Tabular Concrete Strength Dataset*, the PRA addedprediction\_dispersion\_healthat turn 20 of attempt 1\. The rubric fails when the standard deviation of out\-of\-fold predictions is below 0\.80 of the target’s and tail errors exceed twice the mid\-range error\. For a calibrated regressor, the ratio of these standard deviations equals the square root of the explained variance\. At the agent’s CV RMSE of about 12\.1 against a target standard deviation of 16\.4, this gives about 0\.68, the value the PRA measured, and meeting the 0\.80 threshold would require an RMSE near 9\.8\. The agent followed the suggested sample weighting, which raised the ratio from 0\.664 to 0\.680 and worsened CV from 12\.121 to 12\.184\. The weighted run also overwrote the saved out\-of\-fold predictions, and a later review had to flag the overwrite before the agent restored its best model\. The rubric failed in every review of attempts 1 and 2 after it was added, including the final review of attempt 2, whose stacked ensemble reached the task’s best private percentile of 90\.6\. The agent also tried a Huber objective and quantile regression in later attempts, and the final reports of attempts 2 and 4 still list a standard\-deviation ratio of about 0\.67\. Adaptation lets the PRA write criteria that no seed rubric covers, and this example shows that it can also write criteria that conflict with the metric\. Such a rubric consumes turns until the PRA retires it, and here it was never retired\.

\#\#\#ISSUESTOFIX\(addresstheseinyourremainingsteps\):

1\.\*\*prediction\_dispersion\_health\*\*\[Importance:0\.78\|Success:0\.10\]:

\-Definition:Whetherthemodel’sOOFpredictiondistributionhashealthydispersion

relativetothetargetdistribution,withnosevereunderfittingatextreme

values\.\[…\]Astdratiobelow0\.80indicatessystematiccompression;mean

\|residual\|forextremesamplesexceeding2\-3xthemid\-rangeMAEsignalspoor

coverageatthetails\.\[…\]

\-Problem:check\_underdispersion\.pyFAIL:OOFpredstd=10\.90vstargetstd=16\.40,

ratio=0\.664\(wellbelow0\.80threshold\)\.OOFrange\[13\.77,64\.40\]vstarget

\[2\.33,82\.60\]\.Mean\|residual\|forhigh\-strength\(\>65\):25\.92\-3\.2xthemid\-range

MAEof8\.08\.\[…\]

\-Suggestedaction:Addhigh\-strengthsamplestotrainingwithhigherweight

\(sample\_weightinLightGBM\),orusequantileregressionatthe90thpercentileas

anauxiliarysignal,orincludetheoriginalUCIConcreteCompressiveStrength

datasettoprovidemoreextreme\-valuetrainingexamples\.

\[…\]

\[turn21\]\[…\]Addressthedispersionissuebyusingsampleweightsorother

techniques

\[…\]

\[turn27\]\[file\_create\]train\_lgbm\_weighted\.py

"""

LightGBMwithsampleweightsforextremevalues

Addressesthedispersionissuebygivingmoreweighttoextremepredictions

"""

\[…\]

\[turn29\]TheweightedmodelachievedCV12\.18,whichisworsethantheoptimized

LightGBM\(12\.12\)\.Thedispersionissuepersists\-stdratiois0\.68\(stillbelow

0\.80\)\.\[…\]Thesampleweightsapproachdidn’thelpmuch\.\[…\]

\[turn31\]\[…\]Thebestmodelislgbm\_optimized\(CV=12\.121\),butoof\_predictions\.csv

wasoverwrittenwithlgbm\_weighted\(CV=12\.184\)\.Ineedtorestorethebestmodel’s

predictions\.

#### Summary\.

Across these examples, the most useful feedback comes from checks the PRA executes\. These include recomputing a score from saved predictions, nesting a calibration step, bounding an approach analytically from the training labels, and diffing submissions across attempts\. Each such check produced a specific number that the agent could verify in one or two turns, and the agent adopted most such fixes within a few turns\. Rubrics added by the PRA are most valuable when they encode a verified property of the task, such as the image\-level accuracy ceiling in the TGS Salt Identification Challenge or the fold provenance of saved predictions, since those rubrics then serve as persistent checks for later attempts\. Feedback that asks for broader exploration is most productive when an executed check follows it\. In*Regression of Used Car Prices*and the*Iceberg Classifier Challenge*, diversity feedback moved the agent off a single model family\. A later check then decided how to combine the new model with the old ones, either by estimating an ensemble’s CV or by computing a blend weight from cached predictions\. Without such a check, as in*Titanic*, the agent complies with the letter of the request \(a diverse submission\) but omits the validation that would justify the choice\. An added rubric whose threshold conflicts with the metric can persist across attempts and absorb turns, which motivates retiring rubrics that fail repeatedly without a corresponding loss in CV\.

## Appendix NPRA Cadence

Table[19](https://arxiv.org/html/2610.33245#A14.T19)varies the number of agent turnsccbetween PRA invocations on DSPredict\-Easy with one attempt per task\. Every cadence improves over the harness alone\. Invoking the PRA every 10 turns performs best, whilec=5c\{=\}5interrupts the agent more often andc=20c\{=\}20delays feedback\. We usec=10c\{=\}10on DSPredict\-Easy andc=20c\{=\}20on DSPredict\-Hard, where longer attempts would makec=10c\{=\}10twice as costly in PRA invocations \(§[5\.3](https://arxiv.org/html/2610.33245#S5.SS3)\)\. Atc=20c\{=\}20the PRA still improves on the harness alone on DSPredict\-Easy\.

Table 19:Single\-attempt results onDSPredict\-Easyas a function of the PRA cadencecc\(agent turns between PRA invocations\)\. All rows use the same seed rubrics and PRA model\.c=10c\{=\}10is the default setting ofAdaPRA\.
## Appendix OAlternative Base Agent

Table[20](https://arxiv.org/html/2610.33245#A15.T20)replaces Qwen3\-Coder\-Next with DeepSeek\-V3\.2 as the base agent on DSPredict\-Easy, with one attempt per task\. Both conditions otherwise use the DSPredict\-Easy settings of Table[5](https://arxiv.org/html/2610.33245#A3.T5), except that DeepSeek\-V3\.2 runs as a reasoning model, with high reasoning effort and the provider’s default temperature \(Appendix[C](https://arxiv.org/html/2610.33245#A3)\), andAdaPRAkeeps the same PRA model, seed rubrics, and cadence \(c=10c\{=\}10\)\. With a single attempt, a failed attempt cannot be offset by a later one, so the comparison measures whether supervision improves the attempt itself\.AdaPRAraises the mean percentile by 7\.8 points and doubles the above\-median rate\. Over tasks it wins 25 times, loses 11 times, and ties twice \(one\-sided paired permutation testp=0\.005p=0\.005; sign testp=0\.014p=0\.014; Wilcoxon signed\-rankp=0\.006p=0\.006\)\.

Table 20:Single\-attempt results onDSPredict\-Easywith DeepSeek\-V3\.2 as the base agent\. Rates use the fixed denominator of 38 tasks\. Neither condition reaches a medal threshold\.
## Appendix PAgent and PRA Turns

Table 21:Mean turns per attempt \(averaged within each task, then over tasks\)\.*Agent*counts base\-agent turns, and*PRA*counts supervisor turns summed over all invocations in the attempt\.
## Appendix QRun Costs

MethodAttemptsAgent \($\)PRA \($\)Total \($\)$/attempt*DSPredict\-Easy \(38 tasks×\\times5 attempts\)*Harness only187193—1931\.0\+ PRA \+ adaptive rubrics \(AdaPRA\)1881931,2251,4187\.5−\-adaptive rubrics1902131,2611,4747\.8−\-PRA1902021,0771,2796\.7*DSPredict\-Hard \(54 tasks×\\times5 attempts\)*Harness only263465—4651\.8\+ PRA \+ adaptive rubrics \(AdaPRA\)2675471,6402,1878\.2−\-adaptive rubrics2705671,6392,2068\.2−\-PRA2604701,2011,6716\.4Table 22:API cost per run\.*Agent*: Qwen3\-Coder\-Next served through Amazon Bedrock\.*PRA*: Claude Sonnet 4\.6\. The PRA accounts for 72–86% of the cost of supervised runs\. Scoring by reading alone \(−\-PRA\) lowers the cost per attempt relative toAdaPRAby 11% on DSPredict\-Easy and 22% on DSPredict\-Hard\.
## Appendix RFeedback Content in an Earlier Version

This appendix reports an observation from an early development version of our system \(whose rubrics, supervisor, and harness differ from those of the final system\)\. In this version the supervisor reported every rubric to the agent, regardless of whether it passed or failed\. Most rubrics passed in almost every report \(Figure[7](https://arxiv.org/html/2610.33245#A18.F7)\), so the reports consisted largely of positive feedback\. We found that the agent’s next action was similar whether or not a report contained a failure, ostensibly because the suggested actions from failures were being drowned out by the overwhelmingly positive feedback\. This observation led to our design choice to include only failed rubrics \(§[3\.1](https://arxiv.org/html/2610.33245#S3.SS1)\)\.

Figure 7:Share of scores in which each rubric passed or failed in an earlier version of the system, over scores with importance≥0\.6\\geq 0\.6\(Appendix[R](https://arxiv.org/html/2610.33245#A18)\)\.