Recovering Wasted Compute in Autoresearch Agents

# Recovering Wasted Compute in Autoresearch Agents
Source: [https://arxiv.org/html/2608.10424](https://arxiv.org/html/2608.10424)
Au Kwok Chun1∗\\ast, Abhigyan Acherjee2∗\\ast, Amrutha Rao3∗\\ast, Zaiqian Chen4, Kazem Meidani5, C\. Bayan Bruss5, Micah Goldblum1, 6

###### Abstract

A slew of recent works develop agents for solving research problems end\-to\-end, a paradigm increasingly referred to as autoresearch\. Such agents have inspired large industry investment, motivated by their potential to automate time\-consuming human labor and customize machine learning solutions for specialized applications\. In this paper, we study the modeling pipeline at the core of these autoresearch systems and identify common failure modes when they are applied to tabular datasets: \(1\) they waste compute resolving the same bugs over and over again; \(2\) they often fail to tune hyperparameters even when they have a large remaining compute budget; \(3\) the tree\-search algorithms that power them do not explore; and \(4\) they perform data analysis, mimicking the humans whose data they are trained on, but do not use that analysis to make downstream decisions\. We explore targeted interventions and find that a global debug consultant that shares discovered runtime constraints across all branches of the search tree, prompt\- and control\-level enhancements, and refined tree\-search algorithms successfully recover wasted compute\. Our results show that large gains in autoresearch agent performance are achievable through agentic design alone, holding the underlying language model fixed\.

††footnotetext:∗\\astEqual contribution\.1Department of Computer Science, Columbia University2AI, Analytics and Future of Work Initiative, Georgetown University3Department of Applied Mathematics, Columbia University4Department of Statistics, Columbia University5Capital One6Department of Electrical Engineering, Columbia UniversityCorrespondence to:ka3094@columbia\.edu, arr2249@columbia\.edu, aa3320@georgetown\.edu## 1Introduction

Fueled by large\-scale industry investment, agentic systems powered by Large Language Models \(LLMs\) are rapidly replacing traditional AutoML pipelines for automated data science\(Jinget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib5); Chanet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib4)\)\. Increasingly, these systems are aimed at broader research loops, from generating and verifying hypotheses purely from data\(Majumderet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib24);[2025](https://arxiv.org/html/2608.10424#bib.bib23)\)to running experiments and writing up findings\(Luet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib25)\), a direction known as*autoresearch*\. No matter the goal, these systems all rest on the same core task: writing code to process data, train and evaluate models, and produce a candidate solution\. Current systems execute this task unreliably: they frequently time out, waste compute, and produce suboptimal, generic solutions\(Toledoet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib2); Liuet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib19); Yanget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib3)\)\.

In this work, we identify several recurring failure modes in leading tree\-search based agentic frameworks\(Jianget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib17); Liuet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib19)\)when applied to tabular machine learning tasks\. Each represents a distinct category of how an agent’s compute budget is wasted rather than productively spent\. First, agents waste budget rediscovering known bugs\. Because branches in a tree search do not share memory of past failures, parallel branches repeatedly resolve identical errors in isolation, preventing meaningful iteration\(Zhanget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib18); Yinet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib11)\)\. Second, agents leave budget on the table by terminating search prematurely\. Superficial convergence criteria cause them to stop after only a few valid solutions, skipping the hyperparameter tuning phase that the remaining budget could have supported\. Finally, agents spend budget on unproductive search states, becoming trapped in dead\-end solution paths until the budget is exhausted\(Toledoet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib2); Zhouet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib6)\)\.

We show that improved agentic design can address these problems without modifying the base LLM\. We introduce a suite of structural interventions to resolve them:††Code available[here](https://github.com/tingtang2/autoresearch-compute-recovery/tree/main)

- •Context\-aware debugging:To resolve context isolation, we introduce a debug consultant that enables adaptive learning of the execution environment across the search tree\. The consultant accumulates discovered bugs into a shared registry and injects constraints before each generation step, preventing redundant error correction and significantly improving efficiency\.
- •Budget\-aware hyperparameter tuning enforcement:We identify the absence of systematic hyperparameter optimization as a primary failure mode of autonomous ML agents\. To address this, we introduce prompt\-level guidance and control\-loop\-level enforcement mechanisms that compel agents to allocate their compute budgets toward structured hyperparameter tuning\. By penalizing local convergence and rewarding validation\-driven search, we prevent premature termination and redirect search effort toward fine\-grained exploitation\.
- •Thompson Sampling\-enhanced backtracking:We replace random backtracking with Monte Carlo Tree Search \(MCTS\) using Thompson Sampling\. This probabilistic approach allows the agent to intelligently navigate the solution space and escape unproductive debugging loops, resulting in a significant improvement in stability of generated solutions\.
- •Diagnostic \- agents fail to act on injected exploratory data analysis \(EDA\):We evaluate whether agents incorporate analytical insights by injecting “adversarial” results of a toy exploratory data analysis directly into the context window\. Our experiments reveal that current agents tend to ignore these signals, motivating the need for stricter control loops that encourage data\-driven planning\.

## 2Background

##### Automated data science\.

The autoresearch systems described above are built on automated data science: the end\-to\-end process of solving a data science task without human intervention\. The agents that carry out this process, often called machine learning engineering \(MLE\) agents, take a dataset and problem description, generate code to explore and process the data, train and evaluate candidate models, and return a solution ready for deployment\.

Recent research has shown that LLMs alone cannot solve these open\-ended problems effectively and that an agentic scaffold is necessary to guide solution generation\(Nathaniet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib20)\)\. External tools\(Qinet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib21)\), execution feedback\(Gehringet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib22)\), and context management\(Jianget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib17); Liuet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib19); Yanget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib3)\)have all been shown to improve performance\. One of the main architectural components in leading MLE agent frameworks\(Liuet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib19); Jianget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib17)\)is tree search over the space of candidate programs, where each node represents a solution in the form of a codebase, and edges represent attempted improvements via code edits\. At each step, the agent selects a promising node to expand, generates a revised solution, executes it, and uses the resulting validation score to guide further exploration\. This iterative execution\-in\-the\-loop design has become the dominant paradigm for agentic data science and forms the basis of the frameworks we study in this work\.

##### Agentic scaffolds\.

The two primary agentic scaffolds we study are AIDE\(Jianget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib17)\)and ML\-Master\(Liuet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib19)\)\. Our choice of these agents is based on their open\-source nature and their exceptional performance on MLE\-bench\(Chanet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib4)\)\. AIDE structures its search around three core components\. The first is a deterministic greedy search policyπ\\pithat determines at each step whether to draft a new solution from scratch, debug a buggy node, or improve a valid one\. A coding operatorffimplements these three actions, each with specialized prompts:*drafting*produces an initial implementation,*debugging*repairs execution errors, and*improving*proposes a single atomic change to a working solution\. Finally, a summarization operatorΣ\\Sigmaextracts concise summaries of past solutions and their scores, keeping the context manageable as the tree grows\.

ML\-Master extends this paradigm with an improved search strategy and an explicit reasoning mechanism\. Its exploration module is inspired by Monte Carlo Tree Search \(MCTS\): nodes are selected using the Upper Confidence Bound for Trees \(UCT\) criterion, balancing each node’s accumulated reward against its visit count to prioritize promising but under\-explored branches\. Nodes are expanded via the same draft, debug, and improve actions as AIDE and assigned a reward based on code execution: a node receives a positive reward if its solution is bug\-free and improves upon the best validation metric seen so far, and a negative reward otherwise\. Multiple workers explore branches in parallel, with rewards backpropagated through the tree to guide subsequent selection\. In place of AIDE’s summarization operator, ML\-Master employs a reasoning module that embeds a curated memory of past execution results and sibling node insights directly into the reasoning component of the LLM, enabling the agent to learn across parallel exploration paths and leverage the capabilities of reasoning models\.

## 3Methodology

### 3\.1Context\-aware debug consultant

![Refer to caption](https://arxiv.org/html/2608.10424v1/x1.png)Figure 1:The debug consultant shares context across the search tree: one node’s failure becomes every node’s lesson\.Crashed nodes are compressed into a shared bug index of failed and successful repairs, distilled into banned patterns and proven fixes, and injected into every subsequent generation step, so the agent learns its runtime environment once instead of rediscovering each bug per branch\.In the tree\-search paradigm described in[Section˜2](https://arxiv.org/html/2608.10424#S2), knowledge of failures remains local: in both AIDE and ML\-Master, only the child spawned to debug a failed node sees the actual error\. The rest of the tree therefore independently rediscovers the same deprecated API call or version mismatch, often dozens of times within a single run\. We call this context isolation\.

Our debug consultant \([Figure˜1](https://arxiv.org/html/2608.10424#S3.F1)\) addresses this by maintaining a shared registry of runtime constraints—which API calls crash, which alternatives work—and propagating this knowledge to all nodes via a three\-step control loop in order to allow adaptive learning of the execution environment:

Step 1: Error Compression\.When a node crashes, the raw traceback is compressed into a compact record: the error type, a short signature, and the strategy that caused the failure\. Raw tracebacks are verbose and vary across iterations; a short hint is sufficient for the LLM to avoid the mistake\. This keeps the context window focused on solution search and ensures the registry scales gracefully\(Zhanget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib18)\)\.

Step 2: Shared Bug Registry\.Each compressed record is accumulated into a shared registry that tracks the error type, which strategies have failed, and—when another node succeeds—the strategy that worked\. The system distills a concise list of banned patterns and proven fixes:

> BANNED: lgb\.train\(\.\.\., verbose\_eval=N\)→\\toTypeError USE: callbacks=\[lgb\.log\_evaluation\(period=N\)\]

Every new node immediately inherits all entries discovered so far\.

Step 3: Constraint Injection\.Constraints are injected at two levels\. During generation \(drafting or improving\), the distilled banned\-pattern list is appended to the prompt, preventing the agent from repeating known\-failing API calls\. During debugging, the injection is more targeted: the system retrieves records relevant to the current error and provides specific failed strategies \(marked “never do this”\) along with any proven fixes, enabling informed repair rather than blind guessing\.

Step 4: Deterministic Control Rules\.In addition to the shared bug registry, the debug consultant adds deterministic rules to the execution process: execution timeouts and empty logs are treated as terminal dead ends that strictly halt the branch, rather than stochastic noise worth retrying\.

### 3\.2Hyperparameter tuning interventions

We test three interventions of increasing invasiveness: a prompt\-level directive, a control\-loop mechanism that shapes the search reward, and their combination\.

![Refer to caption](https://arxiv.org/html/2608.10424v1/x2.png)Figure 2:Budget dependent control\-loop enforcement of hyperparameter tuning in AIDE\.Each node is scored for hyperparameter\-tuning quality on a 0–3 scale by an LLM judge\. Buggy nodes are assigned the worst possible metric and their score is ignored; for a valid node, the score both conditions the agent’s next prompt and adjusts the metric used for node selection\. The reward depends on how much budget remains: weak tuning \(scores of 0 or 1\) is penalized throughout, but strong tuning \(2 or 3\) is rewarded only in later stages of the search, so the agent explores broadly early and tunes intensively late\.Theprompt\-level interventionappends hyperparameter\-tuning instructions to the agent context viaadditional\_notes\.txt\(for both AIDE and ML\-Master\), instructing it to establish a validated baseline, run cheap trials to identify the most impactful hyperparameters, then tune those around the best configuration once gains stall\.

Thecontrol\-loop interventionenforces tuning through the search reward rather than the prompt\. After a node executes, an execution\-time checker \(\_score\_hyperparameter\_tuning\) grades its tuning quality on a discrete\{0,1,2,3\}\\\{0,1,2,3\\\}scale via an LLM rubric \(NONE, MINIMAL, MODERATE, EXTENSIVE\)\. If the node is not buggy, this score biases which nodes the agent expands next\.

How the score enters the search differs by agent: in AIDE it adjusts the validation metric and conditions subsequent improvement prompts, while in ML\-Master it is folded into the reward used for UCT\-based node selection \(e\.g\.,\+0\.25×hpo\_score\+0\.25\\times\\texttt\{hpo\\\_score\}\)\. The AIDE adjustment also accounts for code diversity and parameter reuse:metricadj=metricbase\+0\.1×s×\(rhpo\+rdiv\+rcorr\)\\text\{metric\}\_\{\\text\{adj\}\}=\\text\{metric\}\_\{\\text\{base\}\}\+0\.1\\times s\\times\(r\_\{\\text\{hpo\}\}\+r\_\{\\text\{div\}\}\+r\_\{\\text\{corr\}\}\), whererhpor\_\{\\text\{hpo\}\},rdivr\_\{\\text\{div\}\}, andrcorrr\_\{\\text\{corr\}\}are the tuning, diversity, and reuse rewards\. The scale factors=\|metricbase\|s=\|\\text\{metric\}\_\{\\text\{base\}\}\|\(or1\.01\.0when the base metric is below0\.010\.01\) keeps the adjustment proportional to the magnitude of the base metric\. The tuning rewardrhpor\_\{\\text\{hpo\}\}depends on how much budget remains: weak tuning is penalized throughout the search, but strong tuning is rewarded only in its later stages\. \([Figure˜2](https://arxiv.org/html/2608.10424#S3.F2)\)\.

Thecombined interventionapplies both the prompt\-level directive and the control\-loop mechanism together\. Full prompts for the directive and the LLM scorer are given in[Appendix˜C](https://arxiv.org/html/2608.10424#A3)\.

### 3\.3Thompson sampling and backtracking

![Refer to caption](https://arxiv.org/html/2608.10424v1/figures/ts_backtracking_fig.png)Figure 3:Thompson Sampling with backtracking reallocates budget away from repeatedly failing branches\.Each sibling node carries a Beta distribution over its quality, stored in global memory and updated from observed rewards after every execution\. When the agent detects an identical error recurring down a path \(S1 → S2 → S3\), it backtracks to the branch point where the error first appeared \(S1\) and re\-samples among the siblings, expanding the node with the highest draw rather than choosing at random\. Budget that would otherwise be spent re\-deriving the same failure is redirected toward more promising candidates\.Each of the agents we study maintains a tree of code variants, where every node is a distinct solution attempt with an associated validation metric\. When the agent hits the same error repeatedly, we backtrack to the branch point where that error first appeared and reconsider its sibling nodes, rather than continuing to burn budget down a failing path\. Instead of choosing at random as is default with current agents, we use Thompson Sampling \([Figure˜3](https://arxiv.org/html/2608.10424#S3.F3)\): each siblingiicarries aBeta​\(αi,βi\)\\text\{Beta\}\(\\alpha\_\{i\},\\beta\_\{i\}\)distribution over its quality, initialized from a uniformBeta​\(1,1\)\\text\{Beta\}\(1,1\)prior\. At each selection step, we draw a sampleθi∼Beta​\(αi,βi\)\\theta\_\{i\}\\sim\\text\{Beta\}\(\\alpha\_\{i\},\\beta\_\{i\}\)from every candidate and expand the one with the highest draw,s∗=arg⁡maxi⁡θis^\{\*\}=\\arg\\max\_\{i\}\\theta\_\{i\}\. After the chosen node executes, we score it with a normalized rewardr∈\[0,1\]r\\in\[0,1\]\(0 for buggy nodes, linearly scaled by validation performance otherwise\) and update its distribution:

αnew=αold\+r\\displaystyle\\alpha\_\{\\text\{new\}\}=\\alpha\_\{\\text\{old\}\}\+r\(1\)βnew=βold\+\(1−r\)\\displaystyle\\beta\_\{\\text\{new\}\}=\\beta\_\{\\text\{old\}\}\+\(1\-r\)\(2\)
This procedure balances exploration \(nodes with high uncertainty have wide distributions, giving them higher chances to be sampled\) and exploitation \(nodes with consistently good performance accumulate higherα\\alphavalues, shifting their distributions rightward\), allowing the agent to learn which code branches are more promising with minimal sample complexity\.

## 4Experimental setup and results

### 4\.1Experimental setup

We evaluate our approaches on nine tabular prediction tasks spanning both classification and regression, drawn from MLE\-bench and additional Kaggle competitions\. Tasks were selected to satisfy three criteria: \(i\) their release postdates the knowledge cutoff of the underlying LLM, minimizing data leakage; \(ii\) they are of moderate dataset size to ensure tractable experimentation; and \(iii\) they collectively cover diverse evaluation metrics\. The nine tasks are Cirrhosis Outcome Prediction, GNSS Classification, Spaceship Titanic, Wine Quality, and Playground Series S5E3, S5E6, S5E7, S5E8, and S5E12, with links provided in[Table˜19](https://arxiv.org/html/2608.10424#A3.T19)from[Section˜C\.5](https://arxiv.org/html/2608.10424#A3.SS5)\. We primarily evaluate two agent frameworks, AIDE and ML\-Master, both powered by GPT\-5\-mini\. Performance is measured using the official MLE\-bench grading scripts, which compute task\-specific metrics consistent with each competition \(e\.g\., accuracy, AUC, RMSE\) on a held\-out test set\. All scores are averaged over 10 independent runs with different random seeds; higher scores indicate better performance on all tasks except Cirrhosis Outcome Prediction, where lower is better\. Following the MLE\-bench medal system, a run earns a gold medal if its score places in the top 10% of the human leaderboard for that competition\. All runs are executed with a fixed compute budget of 2 hours on 22 CPU cores\.

Note on API models\.All experiments use a single backbone, GPT\-5\-mini, because evaluations are prohibitively expensive: the full study \(three agents, intervention and baseline conditions, nine competitions, ten seeds each\) required over a thousand two\-hour runs, and repeating it on a frontier model such as GPT\-5\.5 or Claude Opus 4\.8, which cost roughly ten times more per token, would cost tens of thousands of dollars\.

### 4\.2Context\-aware debugging

The debug consultant design produces large and consistent gains for both agents \([Table˜1](https://arxiv.org/html/2608.10424#S4.T1)\)\. For AIDE, the debug consultant nearly doubles the gold\-medal count, from 22 to 38, and eliminates all 17 of the baseline’s failed runs, raising the valid\-submission rate from 81% to 100%; ML\-Master improves comparably, from 18 to 29 golds, and both agents gain on six of nine competitions\. The improvement is largest precisely where context isolation had been most costly: on S5E3 \(AIDE\) and GNSS \(ML\-Master\), where the baseline earns no medals at all, the consultant recovers a perfect 10/10\. It also reaches a working solution far sooner—the median number of steps to a first valid submission drops from 6 to 0, as the consultant supplies the environment’s constraints before the agent writes its first line of code\.

\(a\) AIDE

\(b\) ML\-Master

Table 1:The debug consultant improves gold medal rates on both agents\.Treatment vs\. vanilla baseline on 9 MLE\-bench competitions×\\times10 seeds\. Scores are mean±\\pmstd;nn= seeds with valid submissions\. Bold = winner by mean; values tied at the displayed precision are bolded in both columns\.†= lower is better\.##### Mechanism\.

The consultant works by stopping the agent from paying for the same mistake twice\. In AIDE’s search journals, redundant bug encounters fall from 46% to 7\.8%, and the fraction of nodes that execute without error rises from 54\.7% to 79\.0%\. The compute the baseline spends rediscovering known bugs is instead spent producing working code, and this is what improves final scores: seeds with more valid nodes achieve better held\-out results \(pooledr=\+0\.22r=\+0\.22across 163 seeds\)\. Detailed statistics, including first\-attempt fix rates and time to first submission, are in[Appendix˜A](https://arxiv.org/html/2608.10424#A1); case studies, full generated code, and per\-competition correlations are in[Sections˜B\.3](https://arxiv.org/html/2608.10424#A2.SS3),[B\.4](https://arxiv.org/html/2608.10424#A2.SS4)and[B\.5](https://arxiv.org/html/2608.10424#A2.SS5)\.

### 4\.3Hyperparameter tuning guidance

Table 2:Hyperparameter guidance yields large performance gains for AIDE across most competitions\.Intervention effect on graded score for AIDE:Δ=μint−μbase\\Delta=\\mu\_\{\\mathrm\{int\}\}\-\\mu\_\{\\mathrm\{base\}\}\(mean±\\pmSEM ofΔ\\Delta\)\. P&C = prompt and code\.†Lower is better for Cirrhosis\.Adding explicit hyperparameter\-tuning guidance to AIDE also produces sizable gains, improving graded scores on 7 of 9 competitions with individual effects as large as\+0\.388\+0\.388on S5E8 and\+0\.218\+0\.218on S5E12 \([Table˜2](https://arxiv.org/html/2608.10424#S4.T2)\)\. The gains are concentrated on tasks where the baseline leaves the most room for improvement \(S5E8, S5E12, Spaceship, Wine\), confirming that AIDE under\-invests in tuning and that a modest amount of structured guidance recovers measurable unrealized performance\. On tasks where the baseline is already strong, additional tuning guidance produces little response\.

The same control\-loop intervention that helps AIDE can degrade ML\-Master\. This is due to the fact that it pushes ML\-Master toward an HPO implementation that crashes\. ML\-Master’s memory then records the crash as buggy without propagating why, resulting in the agent continually retrying variants of the same broken approach\. Scaffold interventions therefore do not transfer for free: whether one helps depends on interactions between the different components comprising a scaffold\. We analyze this asymmetry in detail in[Appendix˜C](https://arxiv.org/html/2608.10424#A3)\.

### 4\.4Thompson sampling with backtracking for stable, more reliable search

Thompson Sampling \(TS\) replaces the random sibling selection used by current agents with a strategy that concentrates exploration on promising branches and backtracks out of repeatedly failing ones\.Its primary effect is stability: in a controlled comparison with all other settings held fixed, TS more than halves the number of null runs at almost no cost to peak performance\.Two controlled studies below, on AIDE and on MLEvolve\(Duet al\.,[2026](https://arxiv.org/html/2608.10424#bib.bib1)\), attribute this gain to the selection strategy itself\.

We achieve this with a path\-structuring algorithm for node selection, augmented by a single new parameter:similar\_error\_backtracking\_threshold, which lets the agent backtrack to the node level that triggered the first instance of a repeated error, reclaiming budget that would otherwise be spent re\-deriving the same failure\. We also widen the initial drafting phase, raising the number of initial solution nodes from 5 in the baseline to 20 for TS, since TS realizes its advantage only when it has a rich pool of candidates to allocate exploration across\. Full hyperparameter settings for the baseline and TS are given in[Table˜13](https://arxiv.org/html/2608.10424#A3.T13); headline results are reported in[Table˜12](https://arxiv.org/html/2608.10424#A3.T12), with standard deviations and null rates in[Section˜C\.1](https://arxiv.org/html/2608.10424#A3.SS1)\([Tables˜14](https://arxiv.org/html/2608.10424#A3.T14)and[15](https://arxiv.org/html/2608.10424#A3.T15)\)\.

Since we make multiple changes to the agentic pipeline, including introducing new hyper\-parameters, we conduct a controlled study to isolate the contribution of TS itself\. We re\-evaluate TS with the number of initial drafts and all other hyperparameters held fixed across conditions, so any observed gain is attributable to TS alone\. For AIDE, we set the native max\-debug\-depth parameter equal to our similar\-error backtracking threshold, which renders the latter inactive, and compare two head\-to\-head conditions: AIDE with increased drafts but no TS, and the full AIDE\+TS configuration \([Table˜16](https://arxiv.org/html/2608.10424#A3.T16)\)\.

AIDE with 20 initial drafts already improves over the plain setting\. On top of that, TS contributes a distinct and practically important advantage\.Holding all other settings identical, AIDE\+TS reduces null runs from 33 to 15 of 90 relative to AIDE\+more drafts \(a 54\.5% reduction\), delivering markedly more stable outcomes\.On the competitions most sensitive to exploration, TS maintains or improves scores even against the stronger draft\-augmented baseline, so the added stability comes at no cost to peak performance\.

To further validate TS independently of any hyperparameter changes, we ran a further experiment with MLEvolve\(Duet al\.,[2026](https://arxiv.org/html/2608.10424#bib.bib1)\), one of the strongest open\-source agents on MLE\-bench\. Here the baseline and TS variants share identical configurations throughout; only the candidate selection strategy differs\. Across nine benchmark competitions, MLEvolve\+TS outperforms baseline MLEvolve on 5 out of 9 competitions \([Table˜3](https://arxiv.org/html/2608.10424#S4.T3)\) and ties a sixth, confirming that the edge is attributable to TS rather than to incidental tuning\.

Table 3:Swapping in Thompson Sampling alone delivers MLEvolve’s largest gains\.With all other settings held fixed, TS wins 5 of 9 competitions and ties a sixth \(S5E7\)\. Its two biggest\-margin results, GNSS \(\+2\.5%\+2\.5\\%\) and S5E12 \(\+1\.5%\+1\.5\\%\), are well outside SEM, while most remaining differences in either direction fall within SEM; the one substantial exception is Wine, where the baseline wins by\+1\.9%\+1\.9\\%\. Bold indicates the winning method per competition; values tied at the displayed precision are bolded in both columns\.†Lower is better\.Two mechanisms drive these gains, and they reinforce each other\. A wider initial draft pool gives the agent more candidates to work with, and Thompson Sampling allocates exploration across them far more effectively than random selection, while backtracking pulls the agent out of repeatedly failing paths and returns that budget to promising ones\. The AIDE and MLEvolve studies let us see each mechanism on its own: the draft increase helps by itself, and TS adds a further gain on top, holding everything else fixed\. Together they explain why the full system is both more stable and stronger than either piece alone\.

### 4\.5Diagnostic: current agents do not meaningfully act on exploratory data analysis

Additionally, as a diagnostic, we examine whether agent\-based systems meaningfully adhere to one of the canonical stages of the machine learning pipeline: exploratory data analysis \(EDA\)\. We observe that most agents operate using a three\-stage structure, namelydraft\(\),improve\(\), anddebug\(\)\. A recurring pattern across agents is the explicit instruction in all three phases to avoid EDA\. We hypothesize that, even if this restriction were lifted, the agents would not effectively utilize insights derived from EDA\. To test this hypothesis, we inject the results of a deliberately misleading and low\-fidelity exploratory data analysis directly into the agent’s context window\. Theoretically, if the agent incorporates this information, such adversarial signals would adversely influence its downstream decisions, for example feature selection\. We would therefore expect degraded performance metrics as a consequence of these adversely impacted choices\.

To test whether agents incorporate exploratory data analysis \(EDA\), we injected the results of a controlled, erroneous EDA into the context windows of AIDE and ML\-Master and compared their performance against EDA\-free baselines\. Across all tasks, the performance differences induced by EDA injection were inconsistent and statistically insignificant\. Using an LLM\-as\-a\-judge framework \(gpt\-5\-2025\-8\-07\), we further found that agents never conducted EDA on their own in baseline runs and rarely engaged with the injected EDA: in AIDE, the agent acknowledged the malicious EDA in only 21% of cases and let it affect feature selection in just 5%\. Together, these results indicate that existing agents do not meaningfully act upon or integrate EDA into downstream modeling decisions, suggesting an avenue for improving future agents\. Extended details including the injection format, example messages, and the full evaluation prompts are provided in[Section˜C\.4](https://arxiv.org/html/2608.10424#A3.SS4)\.

## 5Related works

AutoML outside of LLM agents\.AutoML systems can automate algorithm selection and hyperparameter tuning\. For example,Auto\-sklearn\(Feureret al\.,[2022](https://arxiv.org/html/2608.10424#bib.bib12)\)andTPOT\(Olson and Moore,[2016](https://arxiv.org/html/2608.10424#bib.bib13)\)leverage Bayesian optimization and ensemble construction\. Approaches likeFLAML\(Wanget al\.,[2021](https://arxiv.org/html/2608.10424#bib.bib14)\)andTabPFN\(Hollmannet al\.,[2023](https://arxiv.org/html/2608.10424#bib.bib15)\)focus on low\-computational\-cost optimization or in\-context learning for tabular data, respectively\. However, these systems are rigid compared to LLM agents that can implement any algorithm in principle, and they often fail to outperform simple baselines in low\-data regimes\(Knaueret al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib16)\)\. Unlike these fixed\-pipeline approaches, we employ LLM\-driven agents to dynamically reason about data semantics and debug failures in real time\.

Autoresearch & agentic data science\.Recent agents for machine learning engineering have shifted from linear code generation to sophisticated tree\-search methodologies\(Chanet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib4); Wanget al\.,[2024b](https://arxiv.org/html/2608.10424#bib.bib9)\)\. A broader line of work pushes toward autoresearch, automating larger portions of the research loop: DataVoyager\(Majumderet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib24)\)uses a role\-based multi\-agent architecture to explore and verify hypotheses from a dataset, DiscoveryBench\(Majumderet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib23)\)benchmarks agents on this discovery task rather than on competition\-style modeling, and systems like The AI Scientist\(Luet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib25)\)attempt the full pipeline from ideation to writeup\. The frameworks we build on instead focus on the modeling stage, optimizing predictive performance through tree search:AIDE\(Jianget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib17)\),ML\-Master\(Liuet al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib19)\), andR&D\-Agent\(Yanget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib3)\)navigate complex coding tasks via iterative refinement and multi\-agent parallelization\. While effective at exploration, these methods lack explicit mechanisms for addressing the localization of debugging knowledge, often leading to repetitive errors in the search tree, a gap we address via a debug consultant that enables adaptive learning of the execution environment\.

Self\-correction & context engineering\.LLMs have demonstrated the ability to “self\-debug” code via iterative generation\(Chenet al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib7); Yanget al\.,[2024](https://arxiv.org/html/2608.10424#bib.bib8)\)\. However, in domain\-specific tasks, pre\-training priors frequently override runtime feedback, causing persistent “fix loops\.” To mitigate this, we draw on Agentic Context Engineering \(ACE\)\(Zhanget al\.,[2025](https://arxiv.org/html/2608.10424#bib.bib18)\)to treat context as an evolving playbook\. Crucially, unlike open\-ended agents likeVoyager\(Wanget al\.,[2024a](https://arxiv.org/html/2608.10424#bib.bib10)\)that accumulate success skills, our domain necessitates the systematic accumulation offailures\. We argue that in reward\-sparse environments like tabular debugging, learning what not to do \(negative constraints\) provides just as valuable a signal as sparse successes\.

## 6Discussion

Our findings reveal fundamental limitations of existing agents\. Some of what an agent learns is local to a branch\. Much of it, however, is a global property of the environment or problem setting: which library versions are installed, which API signatures are valid, how long a fold of training takes on the available cores, or the features of the dataset at hand\. Global facts are invariant across the tree, and re\-deriving them per branch or node is redundant\. A global memory that separates the two lets a run behave as a single agent rather than many isolated ones, which reduces redundancy\. More importantly, memory may pay off in generating better hypotheses based on lessons learned globally across previous nodes\. An agent that can recall what it has already ruled out can condition its next hypothesis on the accumulated failures instead of resampling from an unchanged prior\. We expect the value of such memory to grow rather than shrink as autoresearch systems become more ambitious\.

Existing agents are also limited in their ability to explore the search space\. Tree search algorithms assume that expanding a node produces a novel candidate, but LLMs often write nearly the same program over and over again\. When an agent produces dozens of near\-identical programs, the tree is wide only on paper\. Better selection will therefore help only so much until agents are designed to propose more varied solutions, which may be why our own selection improvements reduce variance more than they raise peak scores\.

Reliability is another limitation of current agents, and perhaps the most overlooked\. On a meaningful fraction of runs, agents produce no result at all, and the common practice of averaging over successful runs hides these failures so the waste goes uncounted\. As autoresearch systems take on longer and more expensive tasks, a run that quietly produces nothing becomes far more costly than one that produces a mediocre answer\. Making agents dependable will matter as much as making them capable\.

The interventions we study are deliberately simple, and each recovers substantial performance from the same underlying model, which implies that current agents operate well below the ceiling their language models already permit\. As these systems begin to take on more of the research loop, forming hypotheses, designing experiments, and interpreting results, the cost of an agent that forgets what it has learned, proposes what it has already tried, or fails silently will only rise\. Closing that gap will require treating memory, diversity, and reliability as first\-class objectives of agent design rather than as incidental properties of the scaffold\.

## Acknowledgements

This project was supported by a research award from the Center for AI and Responsible Financial Innovation at Columbia University and by the Columbia Center for AI Technology\.

## References

- J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan, L\. Weng, and A\. Mądry \(2024\)MLE\-bench: evaluating machine learning agents on machine learning engineering\.arXiv preprint arXiv:2410\.07095\.External Links:[Link](https://arxiv.org/abs/2410.07095)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1),[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px2.p1.3),[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- Teaching large language models to self\-debug\.Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p3.1)\.
- S\. Du, X\. Yan, J\. Shi, Z\. Cao, S\. Feng, Z\. Liang, B\. Sun, T\. Peng, Y\. Zhou, X\. Li, J\. Zhou, L\. He, B\. Zhang, and L\. Bai \(2026\)MLEvolve: a self\-evolving framework for automated machine learning algorithm discovery\.External Links:2606\.06473,[Link](https://arxiv.org/abs/2606.06473)Cited by:[§4\.4](https://arxiv.org/html/2608.10424#S4.SS4.p1.1),[§4\.4](https://arxiv.org/html/2608.10424#S4.SS4.p5.1)\.
- M\. Feurer, K\. Eggensperger, S\. Falkner, M\. Lindauer, and F\. Hutter \(2022\)Auto\-Sklearn 2\.0: hands\-free AutoML via meta\-learning\.Journal of Machine Learning Research23\(261\),pp\. 1–61\.External Links:[Link](http://jmlr.org/papers/v23/21-0992.html)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p1.1)\.
- J\. Gehring, K\. Zheng, J\. Copet, V\. Mella, T\. Cohen, and G\. Synnaeve \(2025\)RLEF: grounding code LLMs in execution feedback with reinforcement learning\.External Links:[Link](https://openreview.net/forum?id=PzSG5nKe1q)Cited by:[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px1.p2.1)\.
- N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. Hutter \(2023\)TabPFN: a transformer that solves small tabular classification problems in a second\.External Links:[Link](https://openreview.net/forum?id=cp5PvcI6w8_)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p1.1)\.
- Z\. Jiang, D\. Schmidt, D\. Srikanth, D\. Xu, I\. Kaplan, D\. Jacenko, and Y\. Wu \(2025\)AIDE: AI\-driven exploration in the space of code\.External Links:2502\.13138,[Link](https://arxiv.org/abs/2502.13138)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p2.1),[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px2.p1.3),[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- L\. Jing, Z\. Huang, X\. Wang, W\. Yao, W\. Yu, K\. Ma, H\. Zhang, X\. Du, and D\. Yu \(2025\)DSBench: how far are data science agents from becoming data science experts?\.External Links:2409\.07703,[Link](https://arxiv.org/abs/2409.07703)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1)\.
- R\. Knauer, M\. Grimm, and E\. Rodner \(2024\)PMLBmini: a tabular classification benchmark suite for data\-scarce applications\.External Links:2409\.01635,[Link](https://arxiv.org/abs/2409.01635)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p1.1)\.
- Z\. Liu, Y\. Cai, X\. Zhu, Y\. Zheng, R\. Chen, Y\. Wen, Y\. Wang, W\. E, and S\. Chen \(2025\)ML\-Master: towards AI\-for\-AI via integration of exploration and reasoning\.External Links:2506\.16499,[Link](https://arxiv.org/abs/2506.16499)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1),[§1](https://arxiv.org/html/2608.10424#S1.p2.1),[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px2.p1.3),[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha \(2024\)The AI scientist: towards fully automated open\-ended scientific discovery\.External Links:2408\.06292,[Link](https://arxiv.org/abs/2408.06292)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1),[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- B\. P\. Majumder, H\. Surana, D\. Agarwal, S\. Hazra, A\. Sabharwal, and P\. Clark \(2024\)Position: data\-driven discovery with large generative models\.pp\. 34350–34382\.External Links:[Link](https://proceedings.mlr.press/v235/majumder24a.html)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1),[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- B\. P\. Majumder, H\. Surana, D\. Agarwal, B\. D\. Mishra, A\. Meena, A\. Prakhar, T\. Vora, T\. Khot, A\. Sabharwal, and P\. Clark \(2025\)DiscoveryBench: towards data\-driven discovery with large language models\.External Links:[Link](https://openreview.net/forum?id=vyflgpwfJW)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1),[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- D\. Nathani, L\. Madaan, N\. Roberts, N\. Bashlykov, A\. Menon, V\. Moens, A\. Budhiraja, D\. Magka, V\. Vorotilov, G\. Chaurasia, D\. Hupkes, R\. S\. Cabral, T\. Shavrina, J\. Foerster, Y\. Bachrach, W\. Y\. Wang, and R\. Raileanu \(2025\)MLGym: a new framework and benchmark for advancing AI research agents\.External Links:2502\.14499,[Link](https://arxiv.org/abs/2502.14499)Cited by:[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px1.p2.1)\.
- R\. S\. Olson and J\. H\. Moore \(2016\)TPOT: a tree\-based pipeline optimization tool for automating machine learning\.New York, New York, USA,pp\. 66–74\.External Links:[Link](https://proceedings.mlr.press/v64/olson_tpot_2016.html)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p1.1)\.
- Y\. Qin, S\. Hu, Y\. Lin, W\. Chen, N\. Ding, G\. Cui, Z\. Zeng, X\. Zhou, Y\. Huang, C\. Xiao, C\. Han, Y\. R\. Fung, Y\. Su, H\. Wang, C\. Qian, R\. Tian, K\. Zhu, S\. Liang, X\. Shen, B\. Xu, Z\. Zhang, Y\. Ye, B\. Li, Z\. Tang, J\. Yi, Y\. Zhu, Z\. Dai, L\. Yan, X\. Cong, Y\. Lu, W\. Zhao, Y\. Huang, J\. Yan, X\. Han, X\. Sun, D\. Li, J\. Phang, C\. Yang, T\. Wu, H\. Ji, G\. Li, Z\. Liu, and M\. Sun \(2024\)Tool learning with foundation models\.ACM Comput\. Surv\.57\(4\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3704435),[Document](https://dx.doi.org/10.1145/3704435)Cited by:[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px1.p2.1)\.
- E\. Toledo, K\. Hambardzumyan, M\. Josifoski, R\. Hazra, N\. Baldwin, A\. Audran\-Reiss, M\. Kuchnik, D\. Magka, M\. Jiang, A\. M\. Lupidi, A\. Lupu, R\. Raileanu, K\. Niu, T\. Shavrina, J\. Gagnon\-Audet, M\. Shvartsman, S\. Sodhani, A\. H\. Miller, A\. Charnalia, D\. Dunfield, C\. Wu, P\. Stenetorp, N\. Cancedda, J\. N\. Foerster, and Y\. Bachrach \(2025\)AI research agents for machine learning: search, exploration, and generalization in MLE\-bench\.External Links:2507\.02554,[Link](https://arxiv.org/abs/2507.02554)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1),[§1](https://arxiv.org/html/2608.10424#S1.p2.1)\.
- C\. Wang, Q\. Wu, M\. Weimer, and E\. Zhu \(2021\)FLAML: a fast and lightweight AutoML library\.External Links:1911\.04706,[Link](https://arxiv.org/abs/1911.04706)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024a\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=ehfRiF0R3a)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p3.1)\.
- L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. Wen \(2024b\)A survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\)\.External Links:ISSN 2095\-2236,[Link](http://dx.doi.org/10.1007/s11704-024-40231-1),[Document](https://dx.doi.org/10.1007/s11704-024-40231-1)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. R\. Narasimhan, and O\. Press \(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.External Links:[Link](https://openreview.net/forum?id=mXpq6ut8J3)Cited by:[§5](https://arxiv.org/html/2608.10424#S5.p3.1)\.
- X\. Yang, X\. Yang, S\. Fang, Y\. Zhang, J\. Wang, B\. Xian, Q\. Li, J\. Li, M\. Xu, Y\. Li, H\. Pan, Y\. Zhang, W\. Liu, Y\. Shen, W\. Chen, and J\. Bian \(2025\)R&D\-Agent: an LLM\-agent framework towards autonomous data science\.arXiv preprint arXiv:2505\.14738\.External Links:[Link](https://arxiv.org/abs/2505.14738)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p1.1),[§2](https://arxiv.org/html/2608.10424#S2.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2608.10424#S5.p2.1)\.
- X\. Yin, C\. Ni, S\. Wang, Z\. Li, L\. Zeng, and X\. Yang \(2024\)ThinkRepair: self\-directed automated program repair\.External Links:2407\.20898,[Link](https://arxiv.org/abs/2407.20898)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p2.1)\.
- Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li, U\. Thakker, J\. Zou, and K\. Olukotun \(2025\)Agentic context engineering: evolving contexts for self\-improving language models\.External Links:2510\.04618,[Link](https://arxiv.org/abs/2510.04618)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.10424#S3.SS1.p3.1),[§5](https://arxiv.org/html/2608.10424#S5.p3.1)\.
- A\. Zhou, K\. Yan, M\. Shlapentokh\-Rothman, H\. Wang, and Y\. Wang \(2024\)Language agent tree search unifies reasoning, acting, and planning in language models\.InProceedings of the 41st International Conference on Machine LearningInternational Conference on Learning Representations \(ICLR\)The Thirty\-eighth Annual Conference on Neural Information Processing SystemsProceedings of the Workshop on Automatic Machine LearningThe Eleventh International Conference on Learning RepresentationsForty\-second International Conference on Machine LearningThe Thirteenth International Conference on Learning RepresentationsProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, F\. Berkenkamp, F\. Hutter, L\. Kotthoff, J\. Vanschoren, R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning ResearchProceedings of Machine Learning ResearchProceedings of Machine Learning Research, Vol\.23564235,pp\. 62138–62160\.External Links:[Link](https://proceedings.mlr.press/v235/zhou24r.html)Cited by:[§1](https://arxiv.org/html/2608.10424#S1.p2.1)\.

## Appendix AContext\-aware debugging: detailed analysis

This section provides a comprehensive analysis of how the debug consultant accumulates a shared model of the execution environment and propagates it across the search tree\. We show that baseline agents waste compute rediscovering the same library incompatibilities and API mismatches across branches, trace how the consultant’s accumulated knowledge of the runtime—which library versions are installed, which API signatures are valid, and which code patterns crash—changes agent behavior, and present concrete before/after examples\.

### A\.1The environmental blindness problem

Without the debug consultant, each branch must independently discover the execution environment’s constraints—which library versions are installed, which API parameters have been deprecated, which code patterns are valid in the current container\. The result is massive redundancy: 46\.0% of baseline nodes waste compute re\-encountering bugs that have already been seen within the same seed, compared to only 7\.8% under the consultant\. This redundant re\-discovery causes total failure in 17 baseline seeds \(zero valid submissions\)\.

### A\.2How adaptive learning changes agent behavior

[Table˜4](https://arxiv.org/html/2608.10424#A1.T4)and[Figure˜4](https://arxiv.org/html/2608.10424#A1.F4)show per\-competition recovery statistics\. The BANNED list converts what would be random retries into informed corrections: the agent’s next attempt is constrained to avoid known\-failing patterns, collapsing the search space so it is more likely to succeed\.

Treatment recovers from 96\.8% of bug streaks \(consecutive buggy nodes\), with 72\.4% fixed on the first attempt \(average 1\.40 attempts\)\. Baseline recovers from only 86\.2%, with 41\.4% first\-attempt fixes and an average of 6\.43 attempts\. On the hardest competitions \(S5E6, S5E8\), baseline first\-attempt fix rates are 19\.0% and 0\.0% respectively, while treatment achieves 47\.6% and 69\.4%\.

Table 4:Bug recovery statistics from AIDE journal analysis\. Recovery = bug streak followed by≥1\\geq 1valid node\. 1st\-fix = recovered on the first attempt after the streak\.![Refer to caption](https://arxiv.org/html/2608.10424v1/x3.png)Figure 4:The consultant raises first\-attempt fix rate from 41\.4% to 72\.4%\.Bug recovery rate \(left\) and first\-attempt fix rate \(right\) per competition\. Constraint injection converts random retries into informed corrections\.
### A\.3Synchronization speed

[Table˜5](https://arxiv.org/html/2608.10424#A1.T5)and[Figure˜5](https://arxiv.org/html/2608.10424#A1.F5)show how quickly the consultant learns the environment\. Treatment seeds produce their first valid node at step 0\.4 on average \(median: 0\)—the consultant has already learned enough about the runtime by the first step to guide the agent past common pitfalls\. Baseline seeds, which must rediscover these constraints independently, require 6\.8 steps on average \(median: 6\)\. Treatment reaches 94% submission by step 1 \(100% by step 3\); baseline remains at 0% through step 3 on five competitions\. Under any fixed compute budget≤3\\leq 3steps, the scaffolded agent dominates\.

Table 5:Fraction of seeds with≥1\\geq 1valid submission by stepNN\(AIDE, 10 seeds per competition\)\. Treatment first\-valid step: mean 0\.4 \(median 0\)\. Baseline: mean 6\.8 \(median 6\)\.![Refer to caption](https://arxiv.org/html/2608.10424v1/x4.png)Figure 5:Treatment reaches 100% submission rate by step 3; mean baseline submission rate remains below 40% through step 5\.Cumulative submission rate by exploration step for treatment vs\. baseline across all 9 competitions \(AIDE, 10 seeds each\)\.
### A\.4From environmental synchronization to solution quality

[Table˜6](https://arxiv.org/html/2608.10424#A1.T6)shows that the treatment’s overall valid rate is 79\.0% \(2,889 / 3,655 nodes\) versus 54\.7% \(2,464 / 4,507\) for the baseline\. Treatment generates fewer total nodes but a substantially higher fraction are valid, indicating more efficient use of the compute budget\. This is partly because valid nodes consume significantly more runtime than invalid ones: a valid node must execute the full pipeline—including data preprocessing, model training \(e\.g\., gradient\-boosted trees\), and prediction—while an invalid node crashes early and returns quickly\. Fewer but valid nodes therefore represent a much larger share of useful compute\.

With the environment solved, the LLM focuses on modeling: ensembles, calibration, and problem reformulation emerge through successive refinement\. Per\-competition correlations are reported in[Section˜B\.5](https://arxiv.org/html/2608.10424#A2.SS5)\.

TreatmentBaselineComp\.Tot\.Val\.%Tot\.Val\.%Δ\\Delta%Cirrhosis50042885\.626115057\.5\+\+28\.1GNSS38931781\.543227864\.4\+\+17\.1Spaceship52438072\.574464186\.2−\-13\.7Wine53043582\.143223454\.2\+\+27\.9S5E355747485\.157737064\.1\+\+21\.0S5E619811156\.1434388\.8\+\+47\.3S5E759752187\.372565189\.8−\-2\.5S5E816910059\.2429378\.6\+\+50\.6S5E1219112364\.44736513\.7\+\+50\.7Total3,6552,88979\.04,5072,46454\.7\+\+24\.3Table 6:Node counts from AIDE journal analysis \(9 comps×\\times10 seeds\)\. Treatment generates fewer total nodes but a higher fraction are valid\.

## Appendix BExtended experimental results

### B\.1Per\-seed OOS scores: AIDE

[Table˜7](https://arxiv.org/html/2608.10424#A2.T7)reports the graded out\-of\-sample score for every AIDE seed \(9 competitions×\\times10 seeds\)\. Treatment achieves valid submissions on all 90 seeds; baseline has 17 null runs \(“—”\)\. For Cirrhosis \(Log Loss\), lower is better; for all others, higher is better\.

Table 7:AIDE Per\-Seed OOS Scores\. Bold = per\-seed winner; values tied at the displayed precision are bolded in both rows\. — = null \(no valid submission\)\.†= lower is better\.
### B\.2Per\-seed OOS scores: ML\-Master

[Table˜8](https://arxiv.org/html/2608.10424#A2.T8)reports the graded OOS score for every ML\-Master seed\. Both treatment and baseline achieve valid submissions on all 90 seeds\.

Table 8:ML\-Master treatment wins the majority of per\-seed comparisons despite a fully\-valid baseline\.Per\-seed OOS scores for 9 competitions×\\times10 seeds\. Bold = per\-seed winner; values tied at the displayed precision are bolded in both rows\.†= lower is better\.
### B\.3Case studies: how valid nodes become better solutions

The two case studies below illustrate the same causal chain: the debug consultant removes a persistent API bug→\\tothe agent iterates freely→\\toit builds qualitatively more sophisticated solutions\. Both cases share a common pattern: the baseline LLM has the same modeling knowledge as the treatment, but it never gets to use it because its compute budget is consumed by redundant bug encounters\.

#### B\.3\.1Case study 1: S5E3 seed 6 \(Δ=\+0\.077\\Delta=\+0\.077AUC\)

[Figure˜6](https://arxiv.org/html/2608.10424#A2.F6)compares the search trees for S5E3 \(binary classification, rainfall prediction\), seed 6\.

##### Baseline

\(left, 74 nodes, 2 valid, 2\.7%\): The agent’s first code draft callslgb\.train\(\)with the deprecatedearly\_stopping\_roundsparameter\. LightGBM raises a TypeError; the agent’s try/except handler catches it silently\. Having no mechanism to propagate this failure, the agent generates nearly identical code 72 more times—each node re\-encountering the same TypeError\. Only 2 nodes avoid the pattern, producing asingle\-model LGBMClassifierwith basic feature engineering \(4 domain interactions, no calibration, no ensemble\)\. OOS score: 0\.8748\.

##### Treatment

\(right, 39 nodes, 31 valid, 79\.5%\): The debug consultant records theearly\_stopping\_roundsTypeError at step 0 and adds it to the BANNED list\. From that point forward, the agent never regenerates this pattern, enabling 31 valid iterations to explore progressively better approaches\. The final solution is a3\-model ensemble\(XGBoost \+ 2 LightGBM variants\) with Platt\-scaling calibration and greedy weight optimization\. OOS score: 0\.9515 \(Δ=\+0\.0767\\Delta=\+0\.0767\)\. With the debug consultant eliminating redundant failures, the treatment refines its approach through 15×\\timesmore hypothesis\-testing cycles, converging on a substantially more sophisticated solution\.

![Refer to caption](https://arxiv.org/html/2608.10424v1/x5.png)Figure 6:Search tree comparison for S5E3 seed 6\. Left: baseline \(74 nodes, 2 valid, 2\.7%\)\. Right: treatment \(39 nodes, 31 valid, 79\.5%\)\. Red = buggy; green = valid\. The baseline loops on the sameearly\_stopping\_roundsTypeError for its entire budget\.

#### B\.3\.2Case study 2: Wine Quality \(Δ=\+0\.099\\Delta=\+0\.099QWK\)

[Figure˜7](https://arxiv.org/html/2608.10424#A2.F7)compares the search trees for Wine \(ordinal classification, 7 quality levels\), seed 1\.

##### Baseline

\(left, 10 nodes, 2 valid, 20%\): The agent callsLGBMRegressor\.fit\(\)withearly\_stopping\_roundsas a keyword argument—deprecated in the installed LightGBM version\. Eight of 10 nodes crash with deprecated\-API TypeErrors \(7 fromearly\_stopping\_rounds, 1 fromverbose\)\. The 2 surviving nodes use a single LGBMRegressor with naive rounding of continuous predictions to integer quality labels\. No stacking, no calibration, no threshold optimization\. OOS score: 0\.2990\.

##### Treatment

\(right, 57 nodes, 51 valid, 89%\): The consultant’s BANNED list prevents theearly\_stopping\_roundsTypeError from step 0\. With 51 valid iterations, the agent builds a diverse bagged LightGBM ensemble with stacking \(Ridge \+ Isotonic regression on out\-of\-fold predictions\) and threshold optimization that maps continuous predictions to discrete quality labels, maximizing QWK directly\. OOS score: 0\.3980 \(Δ=\+0\.099\\Delta=\+0\.099\)\.

##### The modeling insight gap\.

Both baseline and treatment use a regression formulation\. The crucial difference is what the agent does with additional valid iterations: it discovers that stacking multiple LightGBM models with isotonic calibration and optimized thresholds produces substantially higher QWK than a single model with naive rounding\. With 51 valid nodes, the treatment has the budget to make this discovery; with 2 valid nodes, the baseline never gets the chance\.

![Refer to caption](https://arxiv.org/html/2608.10424v1/x6.png)Figure 7:Search tree comparison for Wine seed 1\. Left: baseline \(10 nodes, 2 valid, 20%\)\. Right: treatment \(57 nodes, 51 valid, 89%\)\. The baseline exhausts its budget on repeatedearly\_stopping\_roundsTypeErrors\.

#### B\.3\.3Synthesis: more valid nodes→\\tobetter solutions

Both case studies tell the same story through different competitions and bug types:

Table 9:Case study summary\. The debug consultant’s primary effect is enabling iteration\.We find the same pattern in both cases: adaptive learning allows the LLM to iterate\. A single valid node produces a first working draft\. Dozens of valid nodes allow the agent to discover ensembles, calibration, problem reformulation, and threshold optimization\.

### B\.4Full generated code

Below are the complete, unedited solutions generated by the LLM agent for both case studies\. Each listing is the exact code that was executed and graded\.

#### B\.4\.1S5E3 seed 6 — Baseline \(step 20, OOS = 0\.875\)

Note line 88: “no early stopping due to environment limitations”—the agent is aware of the API bug but cannot resolve it\.

Listing 1:Baseline best solution: S5E3 seed 6 \(141 lines\)\. Single LGBMClassifier with basic feature engineering\. No ensemble no calibration no early stopping\.1

2importos

3importwarnings

4importnumpyasnp

5importpandasaspd

6fromsklearn\.model\_selectionimportStratifiedKFold

7fromsklearn\.metricsimportroc\_auc\_score

8fromlightgbmimportLGBMClassifier

9

10warnings\.filterwarnings\("ignore"\)

11

12SEED=42

13N\_SPLITS=5

14

15

16INPUT\_DIR="\./input"

17SUBMISSION\_DIR="\./submission"

18WORKING\_DIR="\./working"

19os\.makedirs\(SUBMISSION\_DIR,exist\_ok=True\)

20os\.makedirs\(WORKING\_DIR,exist\_ok=True\)

21

22

23train\_path=os\.path\.join\(INPUT\_DIR,"train\.csv"\)

24test\_path=os\.path\.join\(INPUT\_DIR,"test\.csv"\)

25

26train=pd\.read\_csv\(train\_path\)

27test=pd\.read\_csv\(test\_path\)

28

29

30

31defprepare\_df\(df\):

32df=df\.copy\(\)

33

34if"day"indf\.columns:

35df\["day\_sin"\]=np\.sin\(2\*np\.pi\*df\["day"\]/365\.0\)

36df\["day\_cos"\]=np\.cos\(2\*np\.pi\*df\["day"\]/365\.0\)

37

38if\{"maxtemp","mintemp"\}\.issubset\(df\.columns\):

39df\["temp\_range"\]=df\["maxtemp"\]\-df\["mintemp"\]

40

41if\{"temparature","dewpoint"\}\.issubset\(df\.columns\):

42df\["temp\_minus\_dew"\]=df\["temparature"\]\-df\["dewpoint"\]

43

44if\{"pressure","humidity"\}\.issubset\(df\.columns\):

45df\["press\_hum"\]=df\["pressure"\]\*df\["humidity"\]/1e3

46

47if\{"windspeed","cloud"\}\.issubset\(df\.columns\):

48df\["wind\_cloud"\]=df\["windspeed"\]\*df\["cloud"\]/100\.0

49returndf

50

51

52train=prepare\_df\(train\)

53test=prepare\_df\(test\)

54

55TARGET="rainfall"

56ID\_COL="id"

57

58

59excluded=\{ID\_COL,TARGET\}

60features=\[cforcintrain\.columnsifcnotinexcluded\]

61

62

63forfinfeatures:

64iffnotintest\.columns:

65test\[f\]=0\.0

66

67X=train\[features\]\.reset\_index\(drop=True\)

68y=train\[TARGET\]\.values

69X\_test=test\[features\]\.reset\_index\(drop=True\)

70test\_ids=test\[ID\_COL\]\.values

71

72

73X=X\.fillna\(\-999\)

74X\_test=X\_test\.fillna\(\-999\)

75

76

77skf=StratifiedKFold\(n\_splits=N\_SPLITS,shuffle=True,random\_state=SEED\)

78oof=np\.zeros\(len\(X\)\)

79test\_preds=np\.zeros\(len\(X\_test\)\)

80

81fold\_aucs=\[\]

82

83

84model\_params=\{

85"objective":"binary",

86"boosting\_type":"gbdt",

87"learning\_rate":0\.05,

88"n\_estimators":1000,

89"num\_leaves":31,

90"min\_child\_samples":20,

91"subsample":0\.8,

92"colsample\_bytree":0\.8,

93"random\_state":SEED,

94"n\_jobs":\-1,

95"verbosity":\-1,

96\}

97

98print\(

99"StartingCVtrainingwithLGBMClassifier\(noearlystoppingduetoenvironmentlimitations\)\.\.\."

100\)

101

102forfold,\(tr\_idx,val\_idx\)inenumerate\(skf\.split\(X,y\),1\):

103print\(f"\\nFold\{fold\}"\)

104X\_tr,X\_val=X\.iloc\[tr\_idx\],X\.iloc\[val\_idx\]

105y\_tr,y\_val=y\[tr\_idx\],y\[val\_idx\]

106

107clf=LGBMClassifier\(\*\*model\_params\)

108

109

110clf\.fit\(X\_tr,y\_tr\)

111

112

113val\_pred=clf\.predict\_proba\(X\_val\)\[:,1\]

114oof\[val\_idx\]=val\_pred

115fold\_auc=roc\_auc\_score\(y\_val,val\_pred\)

116fold\_aucs\.append\(fold\_auc\)

117print\(f"Fold\{fold\}AUC:\{fold\_auc:\.6f\}"\)

118

119

120test\_pred=clf\.predict\_proba\(X\_test\)\[:,1\]

121test\_preds\+=test\_pred/N\_SPLITS

122

123

124oof\_auc=roc\_auc\_score\(y,oof\)

125print\("\\nCross\-validationresults:"\)

126print\(f"OOFAUC:\{oof\_auc:\.6f\}"\)

127print\(f"MeanfoldAUC:\{np\.mean\(fold\_aucs\):\.6f\}Std:\{np\.std\(fold\_aucs\):\.6f\}"\)

128

129

130submission=pd\.DataFrame\(\{"id":test\_ids,"rainfall":test\_preds\}\)

131submission\_path=os\.path\.join\(SUBMISSION\_DIR,"submission\.csv"\)

132submission\.to\_csv\(submission\_path,index=False\)

133

134submission\.to\_csv\(os\.path\.join\(WORKING\_DIR,"submission\.csv"\),index=False\)

135

136print\(f"\\nSavedsubmissionto\{submission\_path\}"\)

137print\("Submissionhead:"\)

138print\(submission\.head\(\)\)

139

140

141print\(f"\\nFinalreportedOOFROCAUC:\{oof\_auc:\.6f\}"\)

#### B\.4\.2S5E3 seed 6 — Treatment \(step 26, OOS = 0\.952\)

The treatment’s 31 valid iterations enable it to discover: \(1\) a 3\-model ensemble \(XGBoost \+ 2 LightGBM variants\), \(2\) Platt\-scaling calibration on each model’s OOF predictions, \(3\) greedy coordinate\-ascent weight optimization in both probability\-space and rank\-space, and \(4\) automatic selection of the better ensemble via calibrated OOF AUC\.

Listing 2:Treatment best solution: S5E3 seed 6 \(437 lines\)\. 3\-model ensemble with Platt\-scaling calibration greedy weight optimization and automatic ensemble selection\.1

2importos

3importjson

4frompathlibimportPath

5importwarnings

6importnumpyasnp

7importpandasaspd

8fromsklearn\.model\_selectionimportStratifiedKFold,KFold

9fromsklearn\.metricsimportroc\_auc\_score

10fromsklearn\.linear\_modelimportLogisticRegression

11importxgboostasxgb

12importlightgbmaslgb

13

14warnings\.filterwarnings\("ignore"\)

15

16

17SEED=42

18N\_SPLITS=5

19THREADS=int\(os\.getenv\("AIDE\_NUM\_THREADS","22"\)\)

20INPUT\_DIR=Path\("input"\)

21WORKING\_DIR=Path\("working"\)

22WORKING\_DIR\.mkdir\(parents=True,exist\_ok=True\)

23

24TRAIN\_PATH=INPUT\_DIR/"train\.csv"

25TEST\_PATH=INPUT\_DIR/"test\.csv"

26SUBMISSION\_PATH=WORKING\_DIR/"submission\.csv"

27

28

29XGB\_PARAMS=\{

30"objective":"binary:logistic",

31"eval\_metric":"auc",

32"verbosity":0,

33"seed":SEED,

34"eta":0\.03,

35"max\_depth":6,

36"subsample":0\.8,

37"colsample\_bytree":0\.8,

38"nthread":THREADS,

39"tree\_method":"hist",

40\}

41XGB\_NUM\_ROUNDS=1000

42XGB\_ESR=50

43

44LGB\_PARAMS\_A=\{

45"objective":"binary",

46"metric":"auc",

47"learning\_rate":0\.03,

48"num\_leaves":31,

49"feature\_fraction":0\.8,

50"bagging\_fraction":0\.8,

51"bagging\_freq":1,

52"min\_data\_in\_leaf":20,

53"verbose":\-1,

54"seed":SEED,

55"num\_threads":THREADS,

56\}

57LGB\_PARAMS\_B=\{

58"objective":"binary",

59"metric":"auc",

60"learning\_rate":0\.02,

61"num\_leaves":64,

62"feature\_fraction":0\.7,

63"bagging\_fraction":0\.7,

64"bagging\_freq":1,

65"min\_data\_in\_leaf":15,

66"min\_sum\_hessian\_in\_leaf":1e\-3,

67"verbose":\-1,

68"seed":SEED\+1,

69"num\_threads":THREADS,

70\}

71LGB\_NUM\_ROUNDS=1000

72LGB\_ESR=50

73

74

75train=pd\.read\_csv\(TRAIN\_PATH\)

76test=pd\.read\_csv\(TEST\_PATH\)

77

78TARGET="rainfall"

79IDCOL="id"

80

81

82drop\_cols=\[IDCOL,TARGET\]

83features=\[cforcintrain\.columnsifcnotindrop\_cols\]

84

85

86if"day"infeatures:

87period=365\.0

88train\["day\_sin"\]=np\.sin\(2\*np\.pi\*train\["day"\]/period\)

89train\["day\_cos"\]=np\.cos\(2\*np\.pi\*train\["day"\]/period\)

90test\["day\_sin"\]=np\.sin\(2\*np\.pi\*test\["day"\]/period\)

91test\["day\_cos"\]=np\.cos\(2\*np\.pi\*test\["day"\]/period\)

92features=\[fforfinfeaturesiff\!="day"\]\+\["day\_sin","day\_cos"\]

93

94if"winddirection"infeatures:

95period=360\.0

96train\["winddir\_sin"\]=np\.sin\(2\*np\.pi\*train\["winddirection"\]/period\)

97train\["winddir\_cos"\]=np\.cos\(2\*np\.pi\*train\["winddirection"\]/period\)

98test\["winddir\_sin"\]=np\.sin\(2\*np\.pi\*test\["winddirection"\]/period\)

99test\["winddir\_cos"\]=np\.cos\(2\*np\.pi\*test\["winddirection"\]/period\)

100features=\[fforfinfeaturesiff\!="winddirection"\]\+\[

101"winddir\_sin",

102"winddir\_cos",

103\]

104

105

106features=\[fforfinfeaturesiffintrain\.columnsandfintest\.columns\]

107

108

109train\["\_missing\_count"\]=train\[features\]\.isnull\(\)\.sum\(axis=1\)

110test\["\_missing\_count"\]=test\[features\]\.isnull\(\)\.sum\(axis=1\)

111

112

113forcolinfeatures:

114iftrain\[col\]\.dtype=="object":

115combined=pd\.concat\(\[train\[col\],test\[col\]\],axis=0\)\.astype\("category"\)

116train\[col\]=combined\.iloc\[:len\(train\)\]\.cat\.codes

117test\[col\]=combined\.iloc\[len\(train\):\]\.cat\.codes

118

119

120train\_nums=train\[features\]\.select\_dtypes\(include=\[np\.number\]\)\.columns\.tolist\(\)

121medians=train\[train\_nums\]\.median\(\)

122train\[train\_nums\]=train\[train\_nums\]\.fillna\(medians\)

123test\[train\_nums\]=test\[train\_nums\]\.fillna\(medians\)

124

125

126train\["\_row\_mean"\]=train\[train\_nums\]\.mean\(axis=1\)

127train\["\_row\_std"\]=train\[train\_nums\]\.std\(axis=1\)\.fillna\(0\.0\)

128test\["\_row\_mean"\]=test\[train\_nums\]\.mean\(axis=1\)

129test\["\_row\_std"\]=test\[train\_nums\]\.std\(axis=1\)\.fillna\(0\.0\)

130

131

132rank\_cols=train\_nums\.copy\(\)

133iflen\(rank\_cols\)\>0:

134combined\_ranks=\[\]

135forcolinrank\_cols:

136combined=pd\.concat\(\[train\[col\],test\[col\]\],axis=0\)

137ranks=combined\.rank\(pct=True,method="average"\)

138combined\_ranks\.append\(ranks\.values\)

139combined\_ranks=np\.vstack\(combined\_ranks\)\.T

140n\_train=train\.shape\[0\]

141train\_ranks=combined\_ranks\[:n\_train\]

142test\_ranks=combined\_ranks\[n\_train:\]

143train\["\_row\_rank\_mean"\]=np\.nanmean\(train\_ranks,axis=1\)

144test\["\_row\_rank\_mean"\]=np\.nanmean\(test\_ranks,axis=1\)

145else:

146train\["\_row\_rank\_mean"\]=0\.0

147test\["\_row\_rank\_mean"\]=0\.0

148

149engineered=\["\_missing\_count","\_row\_mean","\_row\_std","\_row\_rank\_mean"\]

150forfinengineered:

151iffnotinfeatures:

152features\.append\(f\)

153

154features=\[fforfinfeaturesiffintrain\.columnsandfintest\.columns\]

155

156

157const\_feats=\[fforfinfeaturesiftrain\[f\]\.nunique\(\)<=1\]

158ifconst\_feats:

159features=\[fforfinfeaturesiffnotinconst\_feats\]

160

161

162iflen\(features\)\>1:

163corr\_matrix=train\[features\]\.corr\(\)\.abs\(\)

164upper=corr\_matrix\.where\(np\.triu\(np\.ones\(corr\_matrix\.shape\),k=1\)\.astype\(bool\)\)

165to\_drop=\[columnforcolumninupper\.columnsifany\(upper\[column\]\>0\.999\)\]

166ifto\_drop:

167features=\[fforfinfeaturesiffnotinto\_drop\]

168

169features=\[fforfinfeaturesiffintrain\.columnsandfintest\.columns\]

170

171X=train\[features\]\.copy\(\)

172X\_test=test\[features\]\.copy\(\)

173y=train\[TARGET\]\.astype\(int\)\.values

174ids\_test=test\[IDCOL\]\.values

175

176

177min\_class\_count=pd\.Series\(y\)\.value\_counts\(\)\.min\(\)

178ifmin\_class\_count\>=N\_SPLITS:

179cv=StratifiedKFold\(n\_splits=N\_SPLITS,shuffle=True,random\_state=SEED\)

180else:

181cv=KFold\(n\_splits=N\_SPLITS,shuffle=True,random\_state=SEED\)

182

183X\_values=X\.values

184X\_test\_values=X\_test\.values

185

186

187model\_keys=\["xgb","lgb\_a","lgb\_b"\]

188oof\_preds=\{k:np\.zeros\(X\_values\.shape\[0\],dtype=float\)forkinmodel\_keys\}

189test\_preds\_avg=\{k:np\.zeros\(X\_test\_values\.shape\[0\],dtype=float\)forkinmodel\_keys\}

190

191fold\_info=\[\]

192

193forfold,\(tr\_idx,val\_idx\)inenumerate\(cv\.split\(X\_values,y\),start=1\):

194X\_tr,X\_val=X\_values\[tr\_idx\],X\_values\[val\_idx\]

195y\_tr,y\_val=y\[tr\_idx\],y\[val\_idx\]

196

197pos=int\(y\_tr\.sum\(\)\)

198neg=int\(y\_tr\.shape\[0\]\-pos\)

199scale\_pos\_weight=max\(1\.0,neg/max\(1\.0,pos\)\)

200

201

202xgb\_params=XGB\_PARAMS\.copy\(\)

203xgb\_params\["scale\_pos\_weight"\]=scale\_pos\_weight

204xgb\_params\["seed"\]=int\(SEED\)

205dtrain=xgb\.DMatrix\(X\_tr,label=y\_tr,feature\_names=features\)

206dval=xgb\.DMatrix\(X\_val,label=y\_val,feature\_names=features\)

207dtest=xgb\.DMatrix\(X\_test\_values,feature\_names=features\)

208xgb\_model=xgb\.train\(

209xgb\_params,

210dtrain,

211num\_boost\_round=XGB\_NUM\_ROUNDS,

212evals=\[\(dtrain,"train"\),\(dval,"val"\)\],

213early\_stopping\_rounds=XGB\_ESR,

214verbose\_eval=False,

215\)

216ifhasattr\(xgb\_model,"best\_iteration"\)andxgb\_model\.best\_iterationisnotNone:

217xgb\_rounds=int\(xgb\_model\.best\_iteration\)\+1

218else:

219xgb\_rounds=XGB\_NUM\_ROUNDS

220val\_pred\_xgb=xgb\_model\.predict\(dval,iteration\_range=\(0,xgb\_rounds\)\)

221test\_pred\_xgb=xgb\_model\.predict\(dtest,iteration\_range=\(0,xgb\_rounds\)\)

222oof\_preds\["xgb"\]\[val\_idx\]=val\_pred\_xgb

223test\_preds\_avg\["xgb"\]\+=test\_pred\_xgb/N\_SPLITS

224

225

226lgb\_params\_a=LGB\_PARAMS\_A\.copy\(\)

227lgb\_params\_a\["scale\_pos\_weight"\]=scale\_pos\_weight

228lgb\_params\_a\["seed"\]=int\(SEED\)

229ltrain=lgb\.Dataset\(X\_tr,label=y\_tr,feature\_name=features\)

230lval=lgb\.Dataset\(X\_val,label=y\_val,reference=ltrain,feature\_name=features\)

231lgb\_model\_a=lgb\.train\(

232lgb\_params\_a,

233ltrain,

234num\_boost\_round=LGB\_NUM\_ROUNDS,

235valid\_sets=\[ltrain,lval\],

236valid\_names=\["train","val"\],

237callbacks=\[lgb\.early\_stopping\(stopping\_rounds=LGB\_ESR\)\],

238\)

239lgb\_a\_iter=\(

240lgb\_model\_a\.best\_iteration

241ifhasattr\(lgb\_model\_a,"best\_iteration"\)

242elseLGB\_NUM\_ROUNDS

243\)

244val\_pred\_lgb\_a=lgb\_model\_a\.predict\(X\_val,num\_iteration=lgb\_a\_iter\)

245test\_pred\_lgb\_a=lgb\_model\_a\.predict\(X\_test\_values,num\_iteration=lgb\_a\_iter\)

246oof\_preds\["lgb\_a"\]\[val\_idx\]=val\_pred\_lgb\_a

247test\_preds\_avg\["lgb\_a"\]\+=test\_pred\_lgb\_a/N\_SPLITS

248

249

250lgb\_params\_b=LGB\_PARAMS\_B\.copy\(\)

251lgb\_params\_b\["scale\_pos\_weight"\]=scale\_pos\_weight

252lgb\_params\_b\["seed"\]=int\(SEED\+1\)

253ltrain\_b=lgb\.Dataset\(X\_tr,label=y\_tr,feature\_name=features\)

254lval\_b=lgb\.Dataset\(X\_val,label=y\_val,reference=ltrain\_b,feature\_name=features\)

255lgb\_model\_b=lgb\.train\(

256lgb\_params\_b,

257ltrain\_b,

258num\_boost\_round=LGB\_NUM\_ROUNDS,

259valid\_sets=\[ltrain\_b,lval\_b\],

260valid\_names=\["train","val"\],

261callbacks=\[lgb\.early\_stopping\(stopping\_rounds=LGB\_ESR\)\],

262\)

263lgb\_b\_iter=\(

264lgb\_model\_b\.best\_iteration

265ifhasattr\(lgb\_model\_b,"best\_iteration"\)

266elseLGB\_NUM\_ROUNDS

267\)

268val\_pred\_lgb\_b=lgb\_model\_b\.predict\(X\_val,num\_iteration=lgb\_b\_iter\)

269test\_pred\_lgb\_b=lgb\_model\_b\.predict\(X\_test\_values,num\_iteration=lgb\_b\_iter\)

270oof\_preds\["lgb\_b"\]\[val\_idx\]=val\_pred\_lgb\_b

271test\_preds\_avg\["lgb\_b"\]\+=test\_pred\_lgb\_b/N\_SPLITS

272

273fold\_auc\_vals=tuple\(

274float\(roc\_auc\_score\(y\_val,oof\_preds\[k\]\[val\_idx\]\)\)forkinmodel\_keys

275\)

276print\(

277f"Fold\{fold\}:"

278\+","\.join\(

279\[f"\{k\.upper\(\)\}AUC=\{a:\.6f\}"fork,ainzip\(model\_keys,fold\_auc\_vals\)\]

280\)

281\)

282fold\_info\.append\(fold\_auc\_vals\)

283

284

285auc\_oof=\{k:float\(roc\_auc\_score\(y,oof\_preds\[k\]\)\)forkinmodel\_keys\}

286print\("BaseOOFAUCs:",auc\_oof\)

287

288

289calibrators=\{\}

290calibrated\_oof=\{\}

291calibrated\_test=\{\}

292forkeyinmodel\_keys:

293clf=LogisticRegression\(solver="lbfgs",max\_iter=2000,random\_state=SEED\)

294preds=oof\_preds\[key\]\.reshape\(\-1,1\)

295clf\.fit\(preds,y\)

296calibrated=clf\.predict\_proba\(preds\)\[:,1\]

297calibrated\_oof\[key\]=calibrated

298calibrated\_test\[key\]=clf\.predict\_proba\(test\_preds\_avg\[key\]\.reshape\(\-1,1\)\)\[:,1\]

299calibrators\[key\]=clf

300auc\_cal=float\(roc\_auc\_score\(y,calibrated\)\)

301print\(f"CalibratedOOFAUCfor\{key\}:\{auc\_cal:\.6f\}"\)

302

303

304calibrated\_oof\_rank=\{\}

305forkinmodel\_keys:

306calibrated\_oof\_rank\[k\]=pd\.Series\(calibrated\_oof\[k\]\)\.rank\(pct=True\)\.values

307

308

309

310defgreedy\_optimize\(keys,oof\_dict,y,init=None,max\_iters=200\):

311n=len\(keys\)

312ifinitisNone:

313weights=np\.array\(\[1\.0/n\]\*n,dtype=float\)

314else:

315weights=np\.array\(init,dtype=float\)

316ifweights\.sum\(\)<=0:

317weights=np\.array\(\[1\.0/n\]\*n,dtype=float\)

318else:

319weights=weights/weights\.sum\(\)

320best\_auc=roc\_auc\_score\(y,sum\(weights\[i\]\*oof\_dict\[keys\[i\]\]foriinrange\(n\)\)\)

321improved=True

322iters=0

323whileimprovedanditers<max\_iters:

324improved=False

325iters\+=1

326foriinrange\(n\):

327fordeltain\[0\.1,0\.05,0\.02,0\.01,\-0\.01,\-0\.02,\-0\.05,\-0\.1\]:

328w\_new=weights\.copy\(\)

329w\_new\[i\]=max\(0\.0,w\_new\[i\]\+delta\)

330ifw\_new\.sum\(\)<=0:

331continue

332w\_new=w\_new/w\_new\.sum\(\)

333ensemble\_oof=sum\(w\_new\[j\]\*oof\_dict\[keys\[j\]\]forjinrange\(n\)\)

334auc=roc\_auc\_score\(y,ensemble\_oof\)

335ifauc\>best\_auc\+1e\-9:

336best\_auc=auc

337weights=w\_new

338improved=True

339returnweights,best\_auc

340

341

342keys=model\_keys\.copy\(\)

343

344

345weights\_prob,auc\_prob=greedy\_optimize\(keys,calibrated\_oof,y,max\_iters=200\)

346print\(

347f"Optimizedweights\(prob\-space\):\{dict\(zip\(keys,weights\_prob\.round\(4\)\)\)\}OOFAUC:\{auc\_prob:\.6f\}"

348\)

349

350

351weights\_rank,auc\_rank=greedy\_optimize\(keys,calibrated\_oof\_rank,y,max\_iters=200\)

352print\(

353f"Optimizedweights\(rank\-space\):\{dict\(zip\(keys,weights\_rank\.round\(4\)\)\)\}OOFAUC:\{auc\_rank:\.6f\}"

354\)

355

356

357oof\_prob\_ens=sum\(weights\_prob\[i\]\*calibrated\_oof\[keys\[i\]\]foriinrange\(len\(keys\)\)\)

358test\_prob\_ens=sum\(

359weights\_prob\[i\]\*calibrated\_test\[keys\[i\]\]foriinrange\(len\(keys\)\)

360\)

361

362oof\_rank\_ens=sum\(

363weights\_rank\[i\]\*calibrated\_oof\_rank\[keys\[i\]\]foriinrange\(len\(keys\)\)

364\)

365

366test\_rank\_parts=\{\}

367forkinkeys:

368

369test\_rank\_parts\[k\]=pd\.Series\(calibrated\_test\[k\]\)\.rank\(pct=True\)\.values

370test\_rank\_ens=sum\(

371weights\_rank\[i\]\*test\_rank\_parts\[keys\[i\]\]foriinrange\(len\(keys\)\)

372\)

373

374

375auc\_prob\_raw=roc\_auc\_score\(y,oof\_prob\_ens\)

376auc\_rank\_raw=roc\_auc\_score\(y,oof\_rank\_ens\)

377print\(

378f"RawensembleOOFAUCs\-\>prob\-space:\{auc\_prob\_raw:\.6f\},rank\-space:\{auc\_rank\_raw:\.6f\}"

379\)

380

381

382final\_cal\_prob=LogisticRegression\(solver="lbfgs",max\_iter=2000,random\_state=SEED\)

383final\_cal\_prob\.fit\(oof\_prob\_ens\.reshape\(\-1,1\),y\)

384oof\_prob\_cal=final\_cal\_prob\.predict\_proba\(oof\_prob\_ens\.reshape\(\-1,1\)\)\[:,1\]

385test\_prob\_cal=final\_cal\_prob\.predict\_proba\(test\_prob\_ens\.reshape\(\-1,1\)\)\[:,1\]

386auc\_prob\_cal=roc\_auc\_score\(y,oof\_prob\_cal\)

387print\(f"Finalcalibratedprob\-spaceensembleOOFAUC:\{auc\_prob\_cal:\.6f\}"\)

388

389final\_cal\_rank=LogisticRegression\(solver="lbfgs",max\_iter=2000,random\_state=SEED\)

390final\_cal\_rank\.fit\(oof\_rank\_ens\.reshape\(\-1,1\),y\)

391oof\_rank\_cal=final\_cal\_rank\.predict\_proba\(oof\_rank\_ens\.reshape\(\-1,1\)\)\[:,1\]

392test\_rank\_cal=final\_cal\_rank\.predict\_proba\(test\_rank\_ens\.reshape\(\-1,1\)\)\[:,1\]

393auc\_rank\_cal=roc\_auc\_score\(y,oof\_rank\_cal\)

394print\(f"Finalcalibratedrank\-spaceensembleOOFAUC:\{auc\_rank\_cal:\.6f\}"\)

395

396

397ifauc\_prob\_cal\>=auc\_rank\_cal:

398chosen\_name="prob\_space\_ensemble"

399oof\_final=oof\_prob\_cal

400test\_final=test\_prob\_cal

401chosen\_auc=auc\_prob\_cal

402else:

403chosen\_name="rank\_space\_ensemble"

404oof\_final=oof\_rank\_cal

405test\_final=test\_rank\_cal

406chosen\_auc=auc\_rank\_cal

407

408print\(f"Chosenensemble:\{chosen\_name\}withOOFAUC:\{chosen\_auc:\.6f\}"\)

409

410

411fold\_scores=\[\]

412forfold,\(\_,val\_idx\)inenumerate\(cv\.split\(X\_values,y\),start=1\):

413fold\_auc=float\(roc\_auc\_score\(y\[val\_idx\],oof\_final\[val\_idx\]\)\)

414fold\_scores\.append\(fold\_auc\)

415print\(f"Fold\{fold\}AUC\(final\):\{fold\_auc:\.6f\}"\)

416

417cv\_mean=float\(np\.mean\(fold\_scores\)\)

418cv\_std=float\(np\.std\(fold\_scores\)\)

419print\(f"FinalCVmeanAUC:\{cv\_mean:\.6f\}std:\{cv\_std:\.6f\}"\)

420

421

422submission=pd\.DataFrame\(\{IDCOL:ids\_test,TARGET:test\_final\}\)

423submission\.to\_csv\(SUBMISSION\_PATH,index=False\)

424print\(f"Savedsubmissionto:\{SUBMISSION\_PATH\}"\)

425

426

427aide\_metrics=\{

428"valid":True,

429"lower\_is\_better":False,

430"cv\_mean":cv\_mean,

431"cv\_std":cv\_std,

432"cv\_folds":\[float\(f\)forfinfold\_scores\],

433\}

434print\("AIDE\_METRICS\_JSON="\+json\.dumps\(aide\_metrics\)\)

435

436

437print\(f"FinalCVmeanAUC:\{cv\_mean:\.6f\}"\)

#### B\.4\.3Wine Quality Seed 1 — Baseline \(step 7, OOS = 0\.299\)

The baseline uses LGBMRegressor with naive rounding—a regression formulation that discards inter\-class probability information, fundamentally limiting QWK\.

Listing 3:Baseline best solution: Wine Quality seed 1\. LGBMRegressor with regression formulation and naive rounding\.1importos

2importnumpyasnp

3importpandasaspd

4fromsklearn\.model\_selectionimportStratifiedKFold

5importlightgbmaslgb

6fromsklearn\.metricsimportconfusion\_matrix

7

8

9RANDOM\_STATE=42

10np\.random\.seed\(RANDOM\_STATE\)

11

12

13INPUT\_DIR="\./input"

14TRAIN\_PATH=os\.path\.join\(INPUT\_DIR,"train\.csv"\)

15TEST\_PATH=os\.path\.join\(INPUT\_DIR,"test\.csv"\)

16SUBMISSION\_DIR="\./submission"

17SUBMISSION\_PATH=os\.path\.join\(SUBMISSION\_DIR,"submission\.csv"\)

18

19

20train=pd\.read\_csv\(TRAIN\_PATH\)

21test=pd\.read\_csv\(TEST\_PATH\)

22

23

24feature\_cols=\[cforcintrain\.columnsifcnotin\("Id","quality"\)\]

25X=train\[feature\_cols\]\.copy\(\)

26y=train\["quality"\]\.astype\(int\)\.copy\(\)

27X\_test=test\[feature\_cols\]\.copy\(\)

28test\_ids=test\["Id"\]\.astype\(int\)\.copy\(\)

29

30

31label\_min=int\(y\.min\(\)\)

32label\_max=int\(y\.max\(\)\)

33labels\_sorted=np\.arange\(label\_min,label\_max\+1\)

34

35

36

37defquadratic\_weighted\_kappa\(y\_true,y\_pred,min\_rating=None,max\_rating=None\):

38"""

39ComputeQuadraticWeightedKappa\(QWK\)

40"""

41ifmin\_ratingisNone:

42min\_rating=min\(int\(np\.min\(y\_true\)\),int\(np\.min\(y\_pred\)\)\)

43ifmax\_ratingisNone:

44max\_rating=max\(int\(np\.max\(y\_true\)\),int\(np\.max\(y\_pred\)\)\)

45y\_true=np\.array\(y\_true,dtype=int\)

46y\_pred=np\.array\(y\_pred,dtype=int\)

47num\_ratings=int\(max\_rating\-min\_rating\+1\)

48

49O=np\.zeros\(\(num\_ratings,num\_ratings\),dtype=float\)

50fora,binzip\(y\_true,y\_pred\):

51O\[a\-min\_rating,b\-min\_rating\]\+=1

52

53hist\_true=O\.sum\(axis=1\)

54hist\_pred=O\.sum\(axis=0\)

55

56E=np\.outer\(hist\_true,hist\_pred\)

57ifE\.sum\(\)==0:

58return1\.0

59E=E/E\.sum\(\)\*O\.sum\(\)

60

61W=np\.zeros\(\(num\_ratings,num\_ratings\),dtype=float\)

62foriinrange\(num\_ratings\):

63forjinrange\(num\_ratings\):

64W\[i,j\]=\(\(i\-j\)\*\*2\)/\(\(num\_ratings\-1\)\*\*2\)

65

66num=\(W\*O\)\.sum\(\)

67den=\(W\*E\)\.sum\(\)

68ifden==0:

69return1\.0

70return1\.0\-num/den

71

72

73

74n\_splits=5

75skf=StratifiedKFold\(n\_splits=n\_splits,shuffle=True,random\_state=RANDOM\_STATE\)

76

77oof\_preds=np\.zeros\(len\(X\),dtype=float\)

78test\_preds=np\.zeros\(len\(X\_test\),dtype=float\)

79fold\_qwks=\[\]

80

81

82lgb\_params=\{

83"objective":"regression",

84"boosting\_type":"gbdt",

85"learning\_rate":0\.05,

86"n\_estimators":800,

87"random\_state":RANDOM\_STATE,

88"num\_leaves":31,

89"subsample":0\.8,

90"colsample\_bytree":0\.8,

91"reg\_alpha":0\.0,

92"reg\_lambda":1\.0,

93"verbosity":\-1,

94\}

95

96print\("Starting5\-foldCVtraining\(noearlystoppinginfit\)\.\.\."\)

97forfold,\(train\_idx,val\_idx\)inenumerate\(skf\.split\(X,y\),1\):

98X\_train,X\_val=X\.iloc\[train\_idx\],X\.iloc\[val\_idx\]

99y\_train,y\_val=y\.iloc\[train\_idx\],y\.iloc\[val\_idx\]

100model=lgb\.LGBMRegressor\(\*\*lgb\_params\)

101

102model\.fit\(

103X\_train,

104y\_train,

105eval\_set=\[\(X\_val,y\_val\)\],

106eval\_metric="rmse",

107\)

108

109val\_pred=model\.predict\(X\_val\)

110oof\_preds\[val\_idx\]=val\_pred

111test\_pred=model\.predict\(X\_test\)

112test\_preds\+=test\_pred/n\_splits

113

114

115val\_pred\_round=np\.rint\(val\_pred\)\.astype\(int\)

116val\_pred\_round=np\.clip\(val\_pred\_round,label\_min,label\_max\)

117qwk=quadratic\_weighted\_kappa\(

118y\_val\.values,val\_pred\_round,min\_rating=label\_min,max\_rating=label\_max

119\)

120fold\_qwks\.append\(qwk\)

121print\(f"Fold\{fold\}QWK:\{qwk:\.5f\}"\)

122

123

124oof\_preds\_round=np\.rint\(oof\_preds\)\.astype\(int\)

125oof\_preds\_round=np\.clip\(oof\_preds\_round,label\_min,label\_max\)

126oof\_qwk=quadratic\_weighted\_kappa\(

127y\.values,oof\_preds\_round,min\_rating=label\_min,max\_rating=label\_max

128\)

129print\(f"OOFQWK:\{oof\_qwk:\.5f\}"\)

130print\(f"Meanper\-foldQWK:\{np\.mean\(fold\_qwks\):\.5f\}\(std\{np\.std\(fold\_qwks\):\.5f\}\)"\)

131

132

133final\_test\_preds=np\.rint\(test\_preds\)\.astype\(int\)

134final\_test\_preds=np\.clip\(final\_test\_preds,label\_min,label\_max\)

135

136

137submission\_df=pd\.DataFrame\(\{"Id":test\_ids\.values,"quality":final\_test\_preds\}\)

138

139os\.makedirs\(SUBMISSION\_DIR,exist\_ok=True\)

140submission\_df\.to\_csv\(SUBMISSION\_PATH,index=False\)

141

142print\(f"Savedsubmissionto\{SUBMISSION\_PATH\}"\)

143print\("Sampleofsubmission:"\)

144print\(submission\_df\.head\(10\)\.to\_string\(index=False\)\)

#### B\.4\.4Wine Quality Seed 1 — Treatment \(step 38, OOS = 0\.398\)

With 51 valid iterations, the treatment discovers that a diverse bagged LightGBM ensemble with stacking \(Ridge \+ Isotonic regression on out\-of\-fold predictions\) produces substantially higher QWK than the baseline’s single\-model regression approach\.

Listing 4:Treatment best solution: Wine Quality seed 1\. Diverse bagged LightGBM ensemble with stacking\.1

2importos

3importjson

4importmath

5importnumpyasnp

6importpandasaspd

7fromitertoolsimportcombinations

8fromsklearn\.model\_selectionimportStratifiedKFold,KFold

9fromsklearn\.metricsimportcohen\_kappa\_score

10fromsklearn\.linear\_modelimportRidge

11fromsklearn\.isotonicimportIsotonicRegression

12importlightgbmaslgb

13importwarnings

14

15warnings\.filterwarnings\("ignore"\)

16

17

18SEED=42

19NUM\_FOLDS=5

20LGB\_ROUNDS=2000

21EARLY\_STOPPING=100

22THREADS=int\(os\.getenv\("AIDE\_NUM\_THREADS","22"\)\)

23INPUT\_DIR="input"

24TRAIN\_PATH=os\.path\.join\(INPUT\_DIR,"train\.csv"\)

25TEST\_PATH=os\.path\.join\(INPUT\_DIR,"test\.csv"\)

26SUBMISSION\_PATH=os\.path\.join\("working","submission\.csv"\)

27

28np\.random\.seed\(SEED\)

29

30

31train=pd\.read\_csv\(TRAIN\_PATH\)

32test=pd\.read\_csv\(TEST\_PATH\)

33

34FEATURES=\[cforcintrain\.columnsifcnotin\("id","quality"\)\]

35X\_orig=train\[FEATURES\]\.copy\(\)

36X\_test\_orig=test\[FEATURES\]\.copy\(\)

37test\_ids=test\["id"\]\.astype\(int\)\.values

38

39y\_raw=train\["quality"\]\.astype\(int\)\.values

40unique\_classes=np\.sort\(train\["quality"\]\.unique\(\)\)

41n\_classes=len\(unique\_classes\)

42n\_train=len\(train\)

43n\_test=len\(test\)

44

45

46

47defadd\_features\(df\):

48df2=df\.copy\(\)

49forcindf\.columns:

50col=df\[c\]

51ifpd\.api\.types\.is\_numeric\_dtype\(col\):

52if\(col\>0\)\.all\(\):

53df2\[c\+"\_log1p"\]=np\.log1p\(col\)

54else:

55df2\[c\+"\_rankpct"\]=col\.rank\(pct=True\)\.astype\(float\)

56

57df2\[c\+"\_sq"\]=col\.values\*col\.values

58returndf2

59

60

61X=add\_features\(X\_orig\)

62X\_test=add\_features\(X\_test\_orig\)

63

64

65num\_feats=\[cforcinX\_orig\.columnsifpd\.api\.types\.is\_numeric\_dtype\(X\_orig\[c\]\)\]

66variances=\[\(c,X\_orig\[c\]\.var\(\)\)forcinnum\_feats\]

67variances\.sort\(key=lambdax:x\[1\],reverse=True\)

68top\_k=min\(5,len\(variances\)\)

69top\_features=\[cforc,\_invariances\[:top\_k\]\]

70

71fora,bincombinations\(top\_features,2\):

72name=f"\{a\}\_x\_\{b\}"

73X\[name\]=X\_orig\[a\]\.values\*X\_orig\[b\]\.values

74X\_test\[name\]=X\_test\_orig\[a\]\.values\*X\_test\_orig\[b\]\.values

75

76FEATURES\_FE=\[cforcinX\.columns\]

77

78

79use\_strat=True

80forclsinunique\_classes:

81if\(y\_raw==cls\)\.sum\(\)<NUM\_FOLDS:

82use\_strat=False

83break

84

85ifuse\_strat:

86kf=StratifiedKFold\(n\_splits=NUM\_FOLDS,shuffle=True,random\_state=SEED\)

87splits=list\(kf\.split\(X,y\_raw\)\)

88else:

89kf=KFold\(n\_splits=NUM\_FOLDS,shuffle=True,random\_state=SEED\)

90splits=list\(kf\.split\(X\)\)

91

92

93lgb\_variants=\[

94\{

95"objective":"regression",

96"metric":"rmse",

97"learning\_rate":0\.05,

98"num\_leaves":31,

99"max\_depth":6,

100"feature\_fraction":0\.8,

101"bagging\_fraction":0\.8,

102"bagging\_freq":1,

103\},

104\{

105"objective":"regression",

106"metric":"rmse",

107"learning\_rate":0\.03,

108"num\_leaves":63,

109"max\_depth":8,

110"feature\_fraction":0\.7,

111"bagging\_fraction":0\.7,

112"bagging\_freq":1,

113\},

114\{

115"objective":"regression",

116"metric":"rmse",

117"learning\_rate":0\.07,

118"num\_leaves":24,

119"max\_depth":5,

120"feature\_fraction":0\.9,

121"bagging\_fraction":0\.9,

122"bagging\_freq":1,

123\},

124\]

125lgb\_seeds=\[SEED,SEED\+101\]

126

127

128model\_configs=\[\]

129forvid,varinenumerate\(lgb\_variants\):

130forsdinlgb\_seeds:

131model\_configs\.append\(\("lgb",vid,int\(sd\)\)\)

132n\_models=len\(model\_configs\)

133

134

135oof\_stack=np\.zeros\(\(n\_train,n\_models\),dtype=float\)

136test\_stack\_sum=np\.zeros\(\(n\_test,n\_models\),dtype=float\)

137test\_stack\_count=np\.zeros\(n\_models,dtype=int\)

138

139

140print\("TrainingbaseLightGBMensemble\.\.\."\)

141forfold,\(tr\_idx,val\_idx\)inenumerate\(splits\):

142print\(f"Fold\{fold\+1\}/\{NUM\_FOLDS\}"\)

143X\_tr=X\.iloc\[tr\_idx\]\.reset\_index\(drop=True\)

144X\_val=X\.iloc\[val\_idx\]\.reset\_index\(drop=True\)

145y\_tr=y\_raw\[tr\_idx\]

146y\_val=y\_raw\[val\_idx\]

147

148form\_idx,\(mtype,vid,sd\)inenumerate\(model\_configs\):

149

150params=lgb\_variants\[vid\]\.copy\(\)

151params\.update\(

152\{

153"seed":int\(sd\+fold\),

154"verbosity":\-1,

155"num\_threads":THREADS,

156\}

157\)

158lgb\_train=lgb\.Dataset\(X\_tr,label=y\_tr\)

159lgb\_valid=lgb\.Dataset\(X\_val,label=y\_val,reference=lgb\_train\)

160model=lgb\.train\(

161params,

162lgb\_train,

163num\_boost\_round=LGB\_ROUNDS,

164valid\_sets=\[lgb\_train,lgb\_valid\],

165valid\_names=\["train","valid"\],

166callbacks=\[

167lgb\.early\_stopping\(stopping\_rounds=EARLY\_STOPPING\),

168lgb\.log\_evaluation\(period=0\),

169\],

170\)

171best\_iter=getattr\(model,"best\_iteration",None\)

172ifbest\_iterisNone:

173

174try:

175best\_iter=model\.current\_iteration\(\)

176exceptException:

177best\_iter=LGB\_ROUNDS

178

179val\_pred=model\.predict\(X\_val,num\_iteration=best\_iter\)

180test\_pred=model\.predict\(X\_test,num\_iteration=best\_iter\)

181

182oof\_stack\[val\_idx,m\_idx\]=val\_pred

183test\_stack\_sum\[:,m\_idx\]\+=test\_pred

184test\_stack\_count\[m\_idx\]\+=1

185

186

187test\_stack=np\.zeros\_like\(test\_stack\_sum\)

188forminrange\(n\_models\):

189cnt=test\_stack\_count\[m\]

190ifcnt\>0:

191test\_stack\[:,m\]=test\_stack\_sum\[:,m\]/float\(cnt\)

192else:

193test\_stack\[:,m\]=0\.0

194

195

196meta\_oof=np\.zeros\(n\_train,dtype=float\)

197meta\_test\_preds\_folds=np\.zeros\(\(NUM\_FOLDS,n\_test\),dtype=float\)

198

199forfold,\(tr\_idx,val\_idx\)inenumerate\(splits\):

200X\_meta\_tr=oof\_stack\[tr\_idx\]

201y\_meta\_tr=y\_raw\[tr\_idx\]

202X\_meta\_val=oof\_stack\[val\_idx\]

203meta=Ridge\(alpha=1\.0,random\_state=SEED\)

204meta\.fit\(X\_meta\_tr,y\_meta\_tr\)

205meta\_oof\[val\_idx\]=meta\.predict\(X\_meta\_val\)

206meta\_test\_preds\_folds\[fold\]=meta\.predict\(test\_stack\)

207

208

209meta\_final=Ridge\(alpha=1\.0,random\_state=SEED\)

210meta\_final\.fit\(oof\_stack,y\_raw\)

211meta\_test\_pred=meta\_final\.predict\(test\_stack\)

212

213

214iso=IsotonicRegression\(out\_of\_bounds="clip"\)

215try:

216iso\.fit\(meta\_oof,y\_raw\)

217meta\_oof\_cal=iso\.predict\(meta\_oof\)

218meta\_test\_cal=iso\.predict\(meta\_test\_pred\)

219exceptException:

220meta\_oof\_cal=meta\_oof\.copy\(\)

221meta\_test\_cal=meta\_test\_pred\.copy\(\)

222

223

224

225a=1\.0

226b=0\.0

227try:

228A=np\.vstack\(\[meta\_oof\_cal,np\.ones\_like\(meta\_oof\_cal\)\]\)\.T

229sol,\_,\_,\_=np\.linalg\.lstsq\(A,y\_raw,rcond=None\)

230a,b=float\(sol\[0\]\),float\(sol\[1\]\)

231exceptException:

232a,b=1\.0,0\.0

233

234meta\_oof\_cal\_ls=a\*meta\_oof\_cal\+b

235meta\_test\_cal\_ls=a\*meta\_test\_cal\+b

236

237

238class\_means=\[\]

239forcinunique\_classes:

240mask=y\_raw==c

241ifmask\.sum\(\)==0:

242class\_means\.append\(np\.nan\)

243else:

244class\_means\.append\(meta\_oof\_cal\_ls\[mask\]\.mean\(\)\)

245class\_means=np\.array\(class\_means\)

246nan\_mask=np\.isnan\(class\_means\)

247ifnan\_mask\.any\(\):

248filled=np\.linspace\(meta\_oof\_cal\_ls\.min\(\),meta\_oof\_cal\_ls\.max\(\),n\_classes\)

249class\_means\[nan\_mask\]=filled\[nan\_mask\]

250

251thresholds=np\.array\(

252\[\(class\_means\[i\]\+class\_means\[i\+1\]\)/2\.0foriinrange\(n\_classes\-1\)\],

253dtype=float,

254\)

255

256

257defmap\_preds\_to\_labels\(preds,thresholds,classes\):

258idxs=np\.sum\(preds\.reshape\(\-1,1\)\>thresholds\.reshape\(1,\-1\),axis=1\)

259mapped=classes\[idxs\]

260returnmapped

261

262

263defqwk\_for\_thresholds\(thr,preds\_cal,y\_true\):

264preds\_mapped=map\_preds\_to\_labels\(preds\_cal,thr,unique\_classes\)

265returncohen\_kappa\_score\(y\_true,preds\_mapped,weights="quadratic"\)

266

267

268best\_thr=thresholds\.copy\(\)

269best\_score=qwk\_for\_thresholds\(best\_thr,meta\_oof\_cal\_ls,y\_raw\)

270print\(f"InitialcalibratedOOFQWK\(iso\+linfit\):\{best\_score:\.6f\}"\)

271

272

273min\_pred=float\(meta\_oof\_cal\_ls\.min\(\)\)

274max\_pred=float\(meta\_oof\_cal\_ls\.max\(\)\)

275step=\(max\_pred\-min\_pred\)/10\.0ifmax\_pred\>min\_predelse1\.0

276max\_iters=200

277iters=0

278whilestep\>1e\-6anditers<max\_iters:

279improved=False

280foriinrange\(len\(best\_thr\)\):

281low=min\_predifi==0elsebest\_thr\[i\-1\]\+1e\-12

282high=max\_predifi==len\(best\_thr\)\-1elsebest\_thr\[i\+1\]\-1e\-12

283iflow\>=high:

284continue

285current=best\_thr\[i\]

286candidates=\[

287current,

288current\-step,

289current\+step,

290low,

291high,

292\(low\+high\)/2\.0,

293\]

294cand\_values=\[\]

295forcincandidates:

296c=max\(low,min\(high,c\)\)

297cand\_values\.append\(c\)

298cand\_values=sorted\(set\(cand\_values\)\)

299best\_local\_score=best\_score

300best\_local\_val=current

301forvalincand\_values:

302trial\_thr=best\_thr\.copy\(\)

303trial\_thr\[i\]=val

304ifnotnp\.all\(np\.diff\(trial\_thr\)\>\-1e\-12\):

305continue

306score=qwk\_for\_thresholds\(trial\_thr,meta\_oof\_cal\_ls,y\_raw\)

307ifscore\>best\_local\_score\+1e\-12:

308best\_local\_score=score

309best\_local\_val=val

310ifbest\_local\_val\!=current:

311best\_thr\[i\]=best\_local\_val

312best\_score=best\_local\_score

313improved=True

314ifnotimproved:

315step/=3\.0

316iters\+=1

317

318print\(f"Optimizedthresholds:\{best\_thr\}"\)

319print\(f"OOFcalibrated\+linfitQWKafterthresholdopt:\{best\_score:\.6f\}"\)

320

321

322cv\_fold\_scores=\[\]

323forfoldinrange\(NUM\_FOLDS\):

324val\_idx=splits\[fold\]\[1\]

325val\_meta\_pred=meta\_oof\[val\_idx\]

326

327val\_meta\_cal=\(

328iso\.predict\(val\_meta\_pred\)

329ifisinstance\(iso,IsotonicRegression\)

330elseval\_meta\_pred

331\)

332val\_meta\_cal=a\*val\_meta\_cal\+b

333y\_val=y\_raw\[val\_idx\]

334mapped=map\_preds\_to\_labels\(val\_meta\_cal,best\_thr,unique\_classes\)

335score=cohen\_kappa\_score\(y\_val,mapped,weights="quadratic"\)

336cv\_fold\_scores\.append\(float\(score\)\)

337print\(f"Fold\{fold\+1\}QWKaftercalibration\+thresholdopt:\{score:\.6f\}"\)

338

339cv\_mean=float\(np\.mean\(cv\_fold\_scores\)\)

340cv\_std=float\(np\.std\(cv\_fold\_scores\)\)

341print\(f"CVmeanQWK:\{cv\_mean:\.6f\}std:\{cv\_std:\.6f\}"\)

342

343

344test\_mapped=map\_preds\_to\_labels\(meta\_test\_cal\_ls,best\_thr,unique\_classes\)\.astype\(

345int

346\)

347submission=pd\.DataFrame\(\{"id":test\_ids,"quality":test\_mapped\}\)

348os\.makedirs\(os\.path\.dirname\(SUBMISSION\_PATH\),exist\_ok=True\)

349submission\.to\_csv\(SUBMISSION\_PATH,index=False\)

350print\(f"Savedsubmissionto\{SUBMISSION\_PATH\}"\)

351

352

353metrics=\{

354"valid":"quadratic\_weighted\_kappa",

355"lower\_is\_better":False,

356"cv\_mean":cv\_mean,

357"cv\_std":cv\_std,

358"cv\_folds":cv\_fold\_scores,

359\}

360print\("AIDE\_METRICS\_JSON="\+json\.dumps\(metrics\)\)

### B\.5Valid node and score correlation

The case studies above show that more valid nodes enable more iteration and qualitatively better solutions\. We now ask whether this relationship holds quantitatively: across all seeds and competitions, do seeds with more valid nodes tend to achieve higher OOS scores?

##### Per\-seed correlation\.

For each competition, we compute the Pearson correlation \(rr\) between per\-seed valid node count and OOS score across all available seeds \(10 treatment\+\+up to 10 baseline, approximately 18–19 datapoints per competition\)\.[Table˜10](https://arxiv.org/html/2608.10424#A2.T10)reports the results\.

Table 10:Pearson correlation \(rr\) between per\-seed valid node count and OOS score\. Cirrhosis scores are negated so that positiverralways means “more valid nodes→\\tobetter score\.” Pooledrris computed after z\-normalizing both variables within each competition\.The correlation is positive in 7 of 9 competitions, with a z\-normalized pooledr=\+0\.22r=\+0\.22across all 163 seeds, consistent with the hypothesis that more valid nodes lead to better scores\. The correlation is highest on Wine \(r=\+0\.63r=\+0\.63\) and S5E6 \(r=\+0\.61r=\+0\.61\)\. For Cirrhosis \(r=\+0\.34r=\+0\.34,[Table˜10](https://arxiv.org/html/2608.10424#A2.T10)\), more valid nodes correlate with lower \(better\) log\-loss\.

## Appendix CHyperparameter optimization materials and additional results and discussion

##### Hyperparameter optimization directive\.

The following directive is injected verbatim into the agent context to guide hyperparameter tuning behavior\.

HYPERPARAMETEROPTIMIZATIONDIRECTIVE

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Treathyperparametertuningasasequentialsearchproblemunderstricttimeandstepbudgets\.

1\.Startwithastrong,standardbaseline\(well\-knowndefaultsforthemodelclass\)\.

Validatetoestablishaperformancebaseline\.

2\.Runasmallnumberofcheapprobes\(fast,low\-computeexperiments\)toidentifysensitivity

directions\(e\.g\.,learningrate,regularizationstrength,treedepth,batchsize\)\.

Usecross\-validationoraheld\-outvalidationset,notthetestset\.

3\.Prioritizehyperparameterswiththehighestmarginalimpactonvalidationscore\.

Focuson1\-2hyperparametersatatimeratherthansimultaneousgridsearches\.

4\.Usestructuredsearch\(log\-scalesweeps,conditionalranges\),notrandomguessing\.

Forcontinuousparameters,exploreordersofmagnitudefirst\.

5\.Aggressivelypruneunpromisingregionsearly;reallocatebudgettopromisingconfigurations\.

Ifahyperparameterrangeshowsnoimprovementafter2\-3trials,moveon\.

6\.Whenimprovementsplateau\(e\.g\.,<0\.1%improvementover3consecutivetrials\),

switchfromexplorationtofine\-grainedexploitationaroundthecurrentbestsetting\.

7\.Alwaysvalidatechanges;onlykeepconfigurationsthatstrictlyimprove

validationperformanceoveryourcurrentbestsubmission\.

8\.Trackwhatfailedanddonotretryequivalentconfigurations\.

Maintainalogoftriedhyperparametercombinationsandtheirresults\.

Goal:maximizeleaderboard/validationscorewithintheavailablecomputeandstepbudget\.

##### Hyperparameter tuning evaluator prompt\.

The evaluator prompt used to score hyperparameter tuning quality for AIDE and ML\-Master use the same 0–3 rubric\. Both are followed by a separate, code\-level structural check \(Appendix[C](https://arxiv.org/html/2608.10424#A3)\) that can cap the LLM\-assigned score downward for weak or unused search configurations\. The HPO quality scorer usesgpt\-4o\-2024\-08\-06as the judge model, independent of the agent backbone\.

AIDE evaluator prompt:

YouareastrictMLcodereviewerevaluatinghyperparametertuningquality\.

ScorethehyperparametertuninginthefollowingPythoncodeona0\-3scale:

0=none:Nohyperparametertuning\(onlyfixed/defaulthyperparameters\)

1=superficial:Minimaltuning\(e\.g\.,only2\-3valuestestedfor1hyperparameter,

orverysmallgrid/randomsearchwith<5iterations\)

2=moderate:Reasonabletuning\(multiplehyperparameterstested,systematicsearch

with\>=5iterations,propervalidation\)

3=extensive:Comprehensivetuning\(multiplehyperparameters,systematicsearch

with\>=10iterations,propervalidation,bestparamsreusedforfinaltraining\)

DONOTcount:

\-cross\_val\_scorealonewithouthyperparametersearch

\-fixedhyperparameters

\-train/valsplitswithoutsearch

\-hyperparametertuningthatisnotusedinfinalmodeltraining

RespondwithONLYtheintegerscore\(0,1,2,or3\)\.

ML\-Master evaluator prompt:

YouareaSTRICTMLcodereviewer\.

EvaluatetheQUALITYandDEPTHofhyperparametertuninginthefollowingPythoncode\.

Scorehyperparametertuningonascalefrom0to3:

0=NONE

\-Nohyperparametertuning

\-Fixedhyperparameters

\-cross\_val\_scorewithoutparametersearch

1=MINIMAL

\-Tokenorsuperficialtuning

\-Onlyonehyperparametertunedover1\-2values

\-GridSearchCVorRandomizedSearchCVwith<5totalconfigurations

\-RandomizedSearchCVwithn\_iter<5

2=MODERATE

\-Validhyperparametertuningbutlimitedinscope

\-Either:

\*Onehyperparametersearchedover\>=3values,OR

\*Twoormorehyperparameterssearchedover\>=2valueseach

\-\>=5totalconfigurationsevaluated

\-Modelselectionbasedonvalidationorcross\-validation

3=EXTENSIVE

\-Systematic,non\-trivialhyperparameteroptimization

\-Multiplehyperparametersjointlyoptimized

\-\>=10totalconfigurationsortrialsevaluated

\-ClearuseofGridSearchCV,RandomizedSearchCV\(n\_iter\>=10\),Optuna,Hyperopt,

orBayesianoptimization

\-Bestconfigurationexplicitlyselectedandused

Validtuningmethodsinclude:

\-GridSearchCV

\-RandomizedSearchCV

\-Optuna/Hyperopt/Bayesianoptimization

\-Manualloopsevaluatingmultipleconfigurations

RespondONLYwithasingleinteger:0,1,2,or3\.

##### Analysis of asymmetry in AIDE and ML\-Master HPO results\.

The asymmetry can be attributed to two structural differences, which we have confirmed by analyzing the code and logs\.

The first is prompt redundancy in ML\-Master\. Its draft prompt already instructs the agent to include HPO in every draft, with implementation guidelines specifying two\-phase training and limiting the hyperparameter search to 10\-15 trials\. However, AIDE’s draft prompt doesn’t require HPO at draft time the way ML\-Master does\. Our directive therefore fills a gap in AIDE but adds less in ML\-Master, where HPO guidance already exists\.

The second is that the directive pushes the LLM toward RandomizedSearchCV and GridSearchCV with XGBoost, which triggers a sklearn/XGBoost version incompatibility:

AttributeError:’super’objecthasnoattribute’\*\*sklearn\_tags\*\*’

\[05:20:45\]WARNING:Node6f9868d4ismarkedasbuggybecauseresponse\[’is\_bug’\]isTrue\.

\[05:20:45\]INFO:Parsedresults:Node6f9868d4isbuggy

\[05:20:45\]INFO:StartingDebuggingNode6f9868d4\.

\[05:20:56\]INFO:Draftedanewnode3272d605successfully\!

ML\-Master’s memory module records these failures neutrally as is\_bug: True without propagating why, so the LLM interprets each crash as motivation to try a different HPO implementation rather than abandon HPO\. We observe up to 12 consecutive identical API failures within a single ML\-Master run\. This explains the pattern across all three conditions: the prompt\-only condition adds redundant guidance on top of existing instructions, the code\-only condition introduces reward shaping that is undermined by crash\-prone code patterns, and the combined condition inherits both issues simultaneously\.

In contrast, the following AIDE log confirms that our reward\-shaping mechanism functions correctly when the agent’s information flow supports it\. A node with HPO score 0 has its reward adjusted down from 0\.803 to 0\.787 due to the \-0\.300 penalty, making it less likely to be selected:

\[2026\-03\-2620:01:27\]INFO:Scoringhyperparametertuningfornode678c47e2\.\.\.

\[2026\-03\-2620:01:28\]INFO:Node678c47e2\.\.\.initialHPOscore:0

\[2026\-03\-2620:01:28\]INFO:Node678c47e2\.\.\.HPOscoreafterstructuralcaps:0

\[2026\-03\-2620:01:28\]INFO:Parsedresults:Node678c47e2\.\.\.isnotbuggy

\[2026\-03\-2620:01:28\]INFO:Node678c47e2\.\.\.metricadjusted:

base=0\.803140,hpo\_reward=\-0\.300,diversity=0\.100,final=0\.787077

The asymmetry is itself a finding: scaffold interventions interact with the underlying agent’s memory architecture, and a directive that works on one agent can fail on another for reasons unrelated to the directive’s design\.

Table 11:Intervention effect on graded score for ML\-Master:Δ=μint−μbase\\Delta=\\mu\_\{\\mathrm\{int\}\}\-\\mu\_\{\\mathrm\{base\}\}\(mean±\\pmSEM ofΔ\\Delta\), with the same competitions as[Table˜2](https://arxiv.org/html/2608.10424#S4.T2)\. Bold \(signedΔ\\Deltaonly\) indicates strict improvement over baseline \(positiveΔ\\Deltawhen higher is better; negativeΔ\\Deltafor cirrhosis\)\.†Lower is better for Cirrhosis\.
### C\.1Thompson sampling on AIDE: delta, standard deviation and proportion of null errors

Table 12:AIDE Baseline vs\. AIDE with Thompson Sampling\(Mean±\\pmSEM\) of 10 runs\. Bold indicates the winning method per competition; values tied at the displayed precision are bolded in both columns\.†Lower is better\.Table 13:Hyperparameters for different settings on AIDE\.Table 14:Score Standard Deviation per Competition on AIDE\.Bold indicates lower \(more consistent\) std dev\.Table 15:Null/Zero Run Rate per Competition\.Fraction of runs out of 10 that produced no valid score on AIDE\. Bold indicates lower \(fewer failures\); values tied at the displayed precision are bolded in both columns\.
### C\.2AIDE: comparative analysis of Thompson sampling versus hyperparameter changes

Table 16:AIDE with Thompson Sampling and More Drafts vs\. AIDE with More Drafts\(Mean±\\pmSEM\)\. Bold indicates the winning method per competition; values tied at the displayed precision are bolded in both columns\. Number of null runs \(competitions produced without a score\) went from 33 in the baseline with more drafts to 15 with Thompson Sampling, a 54\.5% reduction†Lower is better\.
### C\.3Adversarial EDA prompt

![Refer to caption](https://arxiv.org/html/2608.10424v1/figures/adversarial_eda.png)Figure 8:Malicious EDA results inserted into agent’s context at the draft\(\), debug\(\) and improve\(\) stages![Refer to caption](https://arxiv.org/html/2608.10424v1/figures/eval_prompt_1.png)Figure 9:Control evaluation prompt used to test whether the model conducted exploratory data analysis at all\. This does not refer to the malicious injection, letting the judge model draw independent conclusions![Refer to caption](https://arxiv.org/html/2608.10424v1/figures/eval_prompt_2.png)Figure 10:Secondary evaluation prompt used to test whether the model conducted exploratory data analysis at all, and whether the injection affected agent’s choices\.
### C\.4Adversarial EDA: extended analysis

Table 17:Detailed Performance Comparison for AIDE: Non\-null averages computed over valid runs only\. All\-runs averages treat missing/failed runs as 0\. Diff = Adv EDA \- Baseline\. Positive values indicate improvement with adversarial EDA\.†Lower is better for Cirrhosis\.Table 18:Detailed Performance Comparison for ML\-Master: Non\-null averages computed over valid runs only\. All\-runs averages treat missing/failed runs as 0\. Diff = Adversarial EDA \- Baseline\. Positive values indicate improvement with adversarial EDA\.†Lower is better for Cirrhosis\. Note: the baseline used for ML Master was based on a different set of runs than the baseline used for the debug consultant\.To evaluate whether agents incorporate information from exploratory data analysis \(EDA\) , we inject the results of a controlled, erroneous EDA directly into the agent’s context window\. We conduct this experiment on two representative systems, AIDE and ML\-Master\. For AIDE, the EDA message is inserted at each of its three agentic stages: draft\(\), improve\(\), and debug\(\), and formatted to resemble a memory artifact produced by a prior node\. For ML\-Master, the EDA results are hard\-coded into the data\-preview\.py file located in the utils directory, which supplies contextual information from previous nodes to the agent\. An example message is included in[Figure˜8](https://arxiv.org/html/2608.10424#A3.F8)in[Section˜C\.3](https://arxiv.org/html/2608.10424#A3.SS3)\.

We evaluate both modified agents and compare their performance against baseline runs without injected EDA\.

Across all tasks, we observe thatperformance differences induced by EDA injection are inconsistent and statistically insignificant\.To check whether the agent conducted EDA at all, we use thellm\-as\-a\-judgeframework with a larger reasoning model:gpt\-5\-2025\-08\-07\. The exact framework used is shown in[Section˜C\.3](https://arxiv.org/html/2608.10424#A3.SS3)in[Figures˜9](https://arxiv.org/html/2608.10424#A3.F9)and[10](https://arxiv.org/html/2608.10424#A3.F10)\. We note that in all the baseline runs across both agents, the agent did not conduct any EDA, and struggled to acknowledge the existence of the EDA in the adversarial runs\. For instance in AIDE, in the runs with adversarial EDA injections, the logs demonstrate that the agent is only able to identify and acknowledge the presence of the malicious EDA results in 21% of cases, and that impacts its feature selection in barely 5% of the cases\. This suggests that the agents do not act upon exploratory data analysis, and do not meaningfully integrate EDA into downstream modeling decisions\.

### C\.5List of linked competitions used in experiments

Table 19:Kaggle competitions used in evaluation\.