MLE 智能体:大规模推荐系统中的自主机器学习工程
摘要
Meta 研究人员推出了 MLE Agents,这是一类由人类在环监督的 LLM 智能体,能够自主提出、实施、训练和评估针对日均处理数十亿次请求的搜索排序模型的改进方案。它们共运行了 1,097 项实验,采纳率达 29.6%,使线上搜索量提升了 0.23%,同时表明,智能体运行框架的设计与底层模型同样关键。
查看缓存全文
缓存时间: 2026/11/20 09:11
# MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems
Source: [https://arxiv.org/html/2611.23910](https://arxiv.org/html/2611.23910)
\\workshoptitle
Workshop for Autonomous ML Research
Albert Jiongqian Liang11footnotemark:1Affiliation:Meta PlatformsEmail:[albertlg@meta\.com](mailto:)Dilip RajAffiliation:Meta PlatformsEmail:[dilipraj@meta\.com](mailto:)
###### Abstract
The typical workflow for improving large\-scale ranking and recommendation models is slow and tedious: an engineer forms a hypothesis, implements it as a code change, trains a model, and evaluates it both offline and online, often relying on their own intuition and experience along the way\. Autonomous LLM agents promise to speed this up by running the loop on their own, but it is still unclear whether they can do this in industry\-scale systems rather than on benchmarks or small proxy tasks\. We build human\-on\-the\-loop machine learning engineer \(MLE\) agents that propose, implement, train, and evaluate model changes on large\-scale search rankers\. Engineers retain control over objective design, search direction, and final\-model review\. Each agent writes a model change as code, launches a training job, reads the offline evaluation, and decides whether to keep it\. We identify three obstacles that make this industry setting challenging: feedback takes hours per experiment, offline metrics do not always predict online impact, and improvements saturate after the early wins\. We address these with parallel training runs, an objective calibrated to match online impact, and careful context engineering \(a "what didn’t work" blocklist, strict output formats, and an anti\-staleness rule\)\. In one campaign, the system ran 1,097 completed candidate experiments and accepted 325, a 29\.6% acceptance rate compared with 15\.6% in the earlier campaign under a different objective and search constraint\. Its best model increases search volume by 0\.23% in an A/B test of large scale\. Several of these improvements were architectural changes the team had never tried – mechanisms drawn from the broader industry and literature that had not previously been considered for this stack – suggesting that MLE agents can explore regions of the design space that human intuition overlooks\. Notably, we find that the design of the harness around the model matters: intuition suggests that a stronger underlying model is enough, but in our experience, the design of the loop around it is just as critical\. More broadly, our work shows that autonomous agents can do real machine\-learning engineering in industry\-scale systems\.
Keywords:MLE agents, Autonomous ML, Recommendation Systems, Ranking models, LLM agents
Figure 1:MLE agents 2\.0 progress on Nullstate \(weighted NE, lower is better\): 1,097 distinct configurations, 325 kept\. Best\-so\-far staircase annotated with the discoveries that moved it; the curve flattens after the early wins\.## 1Autonomous research result
### 1\.1Hypothesis
Can an autonomous MLE agent discover a sequence of model improvements for a large\-scale search ranking system with over a billion requests per day that \(a\) compound into an offline quality improvement, \(b\) transfer reliably to online user impact, and \(c\) include genuinely new architectural contributions beyond what the human team had explored? A positive result requires an agent\-discovered model that improves a calibrated offline objective and produces a statistically significant online improvement – while surfacing at least some architectural changes the team had not previously attempted\.
### 1\.2Method
The experiment was deliberately kept simple: let the agent run one long campaign against an objective baseline and incrementally keep what survives\. We judge it on whether the final model with accumulated changes makes an online improvement\. The target model is a multi\-task ranking system that scores search query suggestions across multiple engagement objectives \(click and various downstream actions\)\. It serves a major search platform, using a transformer\-based architecture with sparse and dense feature processing, multi\-task prediction towers, and a distillation pathway\.
#### Baseline\.
An large\-scale ranker trained on multiple partitions of user\-engagement data on the scale of hundreds\-of\-millions of examples and evaluated on held\-out traffic\. Baseline metrics: click NE = 0\.7056 and content\_downstream NE = 0\.6421 on the keyword \+ high\-confidence evaluation slice\.
#### Calibrated objective\.
A weighted composite of normalized entropy \(NE; lower is better\):
ℒ=0\.3×click\_NE\+0\.7×content\_downstream\_NE\.\\mathcal\{L\}=0\.3\\times\\text\{click\\\_NE\}\+0\.7\\times\\text\{content\\\_downstream\\\_NE\}\.\(1\)We chose the offline evaluation objective based on the alignment we observed between offline model metrics and online user engagement in earlier experiments\. Because improvements in predicting downstream engagement were more closely aligned with online improvements than improvements in click prediction alone, we gave downstream engagement greater weight while retaining click prediction in the objective\. A candidate "wins" only if it improves the weighted objective beyond a variance\-probe threshold of 0\.0005 NE, ensuring improvements exceed training noise\. We set this threshold using the average difference in weighted NE observed across historical variance\-probe runs\.
#### Search protocol\.
Both generations operate in an iterative loop: the agent proposes one coherent model modification per candidate, trains it on large\-scale infrastructure, and records the evaluation and acceptance decision in persistent model\-state logs\. MLE agents 1\.0 evaluated one candidate at a time using Click AUPRC; MLE agents 2\.0 uses the calibrated objective of Equation[1](https://arxiv.org/html/2611.23910#S1.E1)and evaluates nominally seven candidates plus a variance probe in parallel per iteration\. Because each candidate takes hours to train and evaluate, parallel execution increases experiment throughput without making an individual training run faster\.
### 1\.3Evidence
#### Offline metrics\.
We present findings from MLE agents 2\.0 first since it was the most successful\. Across 1,097 completed candidate runs over 131 iterations – 1,252 completed runs in total, the remaining 155 being variance probes – the agent accepted 325 candidates, a win rate of 29\.6% of candidate runs, converging to a cumulative improvement of \-0\.0224weighted NE\(weighted baseline 0\.66110\.6388\)\. The improvement is driven more by content\_downstream \(\-0\.0273 NE\) than click \(\-0\.0110 NE\), consistent with the 0\.7 content weight in Equation[1](https://arxiv.org/html/2611.23910#S1.E1)\. Both task metrics improved in the final model, so the end\-to\-end result does not show a sacrifice in click prediction; it does not establish that click prediction had saturated\. The primary offline result is a stack of distinct architectural and objective edits that compound; the largest contributors are shown in Table[1](https://arxiv.org/html/2611.23910#S1.T1)\. Some other interesting structural wins that the agent tried include a backbone swap to a fused Self\-Attention projection MLP with a 4\-layer SE\-DHEN \(Squeeze\-and\-Excitation Deep and Hierarchical Ensemble Network\) block, and bidirectional dense/sparse cross\-attention with a dedicated content\-embedding transformer\.
Table 1:Largest contributors among the changes retained in the final model\. Each contribution is marginal: the change in the weighted objective when that change was carried forward on top of those already retained\. Contributions do not sum to the end\-to\-end result, since some retained steps moved the objective the wrong way\.
#### Online metrics\.
To measure real\-world impact, we ran a standard online A/B test \- a control arm served by the existing ranker, and a test arm served by the new ranker carrying the MLE\-agent\-discovered optimizations\. Differences between the two groups’ behaviors across a wide range of engagement metrics are measured, from how often users search to whether they follow through to downstream actions\. On both primary metrics the agent\-discovered model performed statistically significantly better than the existing baseline, including against a refreshed model, with downstream engagement metrics directionally positive; the main metrics and their 95% confidence intervals are listed in Table[2](https://arxiv.org/html/2611.23910#S1.T2)\.
Table 2:Online A/B Results\. Measured over more than a week of exposure\.
#### Provenance of the online numbers\.
The online results reported here are from the final experiment review, after the A/B test reached its full data read\. The treatment arm served the consolidated MLE agents 2\.0 checkpoint – the same stack whose offline result is reported above – so the offline and online results refer to one campaign and one model\.
#### Online experiment details\.
The test ran over one week\. Reported movements are relative changes with 95% confidence intervals computed by Fieller’s method\.
#### Guardrail regression\.
The treatment also showed a latency regression on our surface time\-to\-render guardrail\. We report it alongside the engagement improvements: the launch traded a measurable latency cost against the engagement improvement, and an agent optimizing a pure quality objective does not surface such trade\-offs on its own\.
### 1\.4Primary Result
At the end of the day, the offline improvements are good\. More importantly, we observed the online improvement \(\+0\.23% search volume\)\. Not only was that an improvement on online metrics, not a benchmark score, but several of the changes the agent found were architectural ideas the team had never tried, drawn from the broader industry and literature rather than from this team’s own design history\. The offline improvements still matter and are impressive, but mainly as a fast signal for the agent to iterate against\. The success from these MLE agent runs encouraged us to invest further in this direction\.
## 2Related Work
Recent work has studied LLM agents for automated scientific discovery and machine\-learning engineering\. The AI Scientist\([Lu et al\., 2024](https://arxiv.org/html/2611.23910#bib.bib8)\)and AIDE\([Jiang et al\., 2025](https://arxiv.org/html/2611.23910#bib.bib9)\)use language models to formulate hypotheses, implement experiments, and iteratively improve empirical results\. MLE\-bench\([Chan et al\., 2024](https://arxiv.org/html/2611.23910#bib.bib10)\)and RE\-Bench\([Wijk et al\., 2024](https://arxiv.org/html/2611.23910#bib.bib11)\)evaluate related capabilities on curated ML\-engineering and research tasks\. These studies establish that LLM agents can perform substantial experimental work, but most of their evidence comes from benchmarks or offline task suites\.
The closest industry\-scale comparison is the Self\-Evolving Recommendation System of[Wang et al\. \(2026\)](https://arxiv.org/html/2611.23910#bib.bib12)\. Their system combines a high\-frequency offline optimization loop with a lower\-frequency online deployment loop at YouTube\. It uses specialized optimizer, architecture, and reward personas, a shared Experiment Journal, and a state machine for validating, training, launching, and monitoring candidates\. They report multiple statistically significant launches, increased experiment throughput, and controlled ablations showing the importance of the controller model, persona prompting, and experiment\-history organization\.
Our work is complementary rather than a claim of the first large\-scale MLE\-agent system\. We provide independent evidence on a different stack: a multi\-task search\-ranking model with sparse and dense features, hours\-long distributed training jobs, and an offline\-to\-online evaluation gap\. Our study emphasizes three aspects\. First, we report a longitudinal campaign with more than 1,000 candidate evaluations, exposing saturation and repeated failure modes that are difficult to observe in shorter studies\. Second, we focus on reliability mechanisms for the offline loop, including parallel variance probes, structured memory of failed hypotheses, strict output contracts, and an anti\-staleness policy\. Third, we retain explicit human control over objective design, search redirection, final\-model review, and online experiment, and document the operational challenges and lessons from deploying MLE agents in an industry\-scale environment\.
## 3System Design and Meta\-Analysis
We now provide a meta\-analysis of the research process\.
### 3\.1Agent and Harness
We develop two generations of MLE agents:MLE agents 1\.0, an exploratory version run on two industry\-scale ranking models \(Nullstate and Typeahead\)111*Nullstate*ranks the query suggestions shown when a user taps the search bar, before typing anything\.*Typeahead*ranks autocomplete suggestions as the user types\., andMLE agents 2\.0, a redesign informed by 1\.0’s limitations and run on Nullstate\. Each agent receives structured context about the current model state, past experiment results, and strategic guidance, then generates code modifications to the model configuration, which are submitted as distributed training jobs on industry\-scale infrastructure\. MLE agents 1\.0 evaluates one candidate at a time\. In 2\.0, each iteration begins with a Claude Opus 4\.8 research call to gather architectural ideas, followed by a brainstorming call to select seven candidate changes\. Seven implementation calls then run in parallel, one per candidate\. The resulting model variants and a variance probe are trained in parallel, and completed runs are evaluated against the calibrated objective\. Intuition suggests the lever is a stronger underlying model, and in our limited experience the design of the loop around it also makes a big difference\. We never tested other models, so we cannot say how the two compare\. What we can say is that an important driver of quality we observed was context engineering and objective calibration: the controller LLM was held fixed throughout, so every improvement reported in this paper came from harness and objective design rather than a stronger base model; the agent itself changed the ranker\. The headline campaign in Section[1](https://arxiv.org/html/2611.23910#S1)is a 2\.0 run on Nullstate; Section[3\.3](https://arxiv.org/html/2611.23910#S3.SS3)traces the path from 1\.0 to 2\.0\.
### 3\.2Tools
The agent operates through an industry\-scale ML platform\. It edits the model’s configuration and code directly under a strict output contract, then launches each candidate as a distributed training job on Grand Teton class trainers with 64 GPUs \(8x8, batch size 256\)\. Early 1\.0 experiments typically required approximately 6–10 hours for training and evaluation\. Following training\-throughput optimizations, including less frequent metrics reporting, completed candidate and variance\-probe runs in the reported 2\.0 campaign had a median recorded elapsed time of 3\.54 hours and a mean of 4\.02 hours\. These timings include submission, scheduling, training, evaluation, and metric\-retrieval delays, rather than active training time alone\. The campaigns were not a matched runtime benchmark, so the reduction should not be attributed solely to throughput tuning\. When a job finishes, the agent reads the calibrated offline metrics – click and content normalized entropy on both the full and the keyword\+high\-confidence slices – and compares them against the current best\. Campaign state \(every hypothesis, commit hash, job ID, metric, and accept/reject decision\) persists in a json state file, backed by a seeds file and an automatic path that resubmits jobs after transient infrastructure failures; a spreadsheet mirrors it for monitoring\. Finally, in a dedicated research phase the agent searches the model codebase and the external literature \(e\.g\., FRNet, BilinearSE, MaskNet, FiLM, differential attention\) for architectural ideas worth trying\.
#### Compute\.
Across the 2\.0 campaign, 1,252 runs completed \(1,097 candidate runs and 155 variance probes\), with a median recorded elapsed time of 3\.54 hours \(mean 4\.02; 10th–90th percentile 2\.75–5\.12 hours\), and a further 2,798 attempts failed or were retried, overwhelmingly on transient infrastructure errors\. Summing job time gives roughly 5,000 job\-hours of completed work plus about 600 for failed attempts, or approximately 360,000 GPU\-hours in total\. The campaign spanned 31 days\. Each 2\.0 iteration issued one research call and one brainstorming call, followed by nominally seven parallel implementation calls, one per selected candidate\.
### 3\.3Research Loop
Each iteration proceeds as: \(1\)context assembly– current config and accumulated changes, recent wins/losses with metric deltas, the "What Didn’t Work" blocklist, human strategic direction, and \(in the research phase\) related architecture code and paper summaries; \(2\)hypothesis generation– a single coherent modification under strict output contracts; \(3\)implementation– one coherent change per candidate to avoid confounded edits, with seven implementation calls running in parallel in MLE agents 2\.0; \(4\)training– a distributed job \(candidates run in parallel in MLE agents 2\.0\); \(5\)evaluation– reading calibrated metrics after job completion \(median recorded elapsed time 3\.54 hours in the 2\.0 campaign\); \(6\)decision– accept/reject on the variance\-aware threshold, with rejects logged to the blocklist\.
MLE agents 1\.0\.We first ran the loop as an exploratory system on two large\-scale ranking models, optimizing Click AUPRC \(higher is better\)\. On Nullstate it ran 64 experiments and kept 10; on Typeahead, 43 experiments and kept 5 \(Figure[2](https://arxiv.org/html/2611.23910#S3.F2)\)\. That one loop produced compounding improvements on two distinct surfaces was our first evidence that the approach generalizes rather than exploiting a single model’s quirks\.
Figure 2:MLE agents 1\.0 progress on two surfaces \(Click AUPRC, higher is better\): Nullstate, 64 experiments with 10 kept; Typeahead, 43 experiments with 5 kept\. The same loop compounded improvements on both surfaces\.#### What 1\.0 taught us\.
Three limitations shaped the redesign\. First, the objective was miscalibrated: Click AUPRC over\-indexed on trivially\-predictable direct\-navigation results and did not track online impact, so offline "wins" did not reliably transfer\. Second, feeding the agent the full unstructured experiment history caused "prompt bloat," with roughly a third of generated edits malformed\. Third, serial execution – one candidate per iteration – made the hours\-long feedback loop the binding constraint on throughput\.
#### MLE agents 2\.0\.
Each learning became a design change: we replaced Click AUPRC with a calibrated weighted\-NE objective aligned to online impact; we replaced the full history with structured context – the "What Didn’t Work" blocklist, strict output contracts, a one\-coherent\-change\-per\-candidate rule, and an anti\-staleness policy \(only recent, un\-carried winners are fed back\); and we ran candidates in parallel \(nominally seven candidates plus a variance probe per iteration; actual counts vary with retries and reseeding\)\. Across the 2\.0 campaign \(Nullstate, 131 iterations\), these changes raised the win rate from 15\.6% \(MLE agents 1\.0 on Nullstate: 10 accepted of 64 experiments\) to 29\.6% \(MLE agents 2\.0 on Nullstate: 325 accepted of 1,097 candidate runs\), and cut malformed output from 33\.8% to 0\.8%\. Because the objective and search budget also differed, this comparison is not a controlled ablation\.
Figure 3:Completed experiments vs\. accepted, by change category\. Orange ticks mark the count expected by chance\.Figure 4:Largest logged regressions on the “What Didn’t Work” blocklist \(weighted NE degradation vs\. baseline\)\.
### 3\.4Human Interventions
MLE agents 2\.0 runshuman\-on\-the\-loop: it launches and evaluates candidates continuously without blocking on a person, while an engineer periodically steers it through the same context channels the agent already reads\. Examples of human intervention across development of the two generations were: \(i\) seeding specific ideas into the seeds file when progress stalled \(for example, pointing the agent at content\-embedding transformers\); \(ii\) writing failed directions into the "What Didn’t Work" blocklist after the agent looped on them – most notably a 0\-for\-8 streak on optimizer tuning in 1\.0, after which we redirected it toward architecture search; \(iii\) recalibrating the objective once we diagnosed that click AUPRC was rewarding trivially\-predictable direct\-navigation results; and \(iv\) tightening training constraints \(e\.g\., cutting LR warmup from 300K to 200K steps\) to keep candidates deployable\. None of these paused the loop – each took effect on the next iteration while the agent kept running\.
### 3\.5Verification
We use verification at three points\.During the run, a candidate is accepted only if it beats its same\-iteration variance probe by at least 0\.0005 NE, screening out improvements smaller than the operational training\-variation threshold\. Each job’s completion is confirmed through two independent status signals, and jobs that fail for transient reasons are automatically resubmitted \(up to two more times\), so a crashed or mis\-logged run is never mistaken for a result\.Offline, every candidate is scored on fixed evaluation sets \(131,072 click and 55,234 content\_kw\_hc samples\) so deltas are comparable, and an engineer reviews the final accumulated stack for coherence offline\.Online, the model is evaluated in an independent experiment \(\+0\.23% search\-volume results\)\. The 325 offline acceptances should therefore be interpreted as search decisions\.
### 3\.6Significance of result
The significance is that an autonomous agent’s work cleared—and survived—an industry\-level bar far higher than a benchmark’s\. Prior work evaluates ML\-engineering agents on curated benchmarks and offline task suites\([Lu et al\., 2024](https://arxiv.org/html/2611.23910#bib.bib8);[Jiang et al\., 2025](https://arxiv.org/html/2611.23910#bib.bib9);[Chan et al\., 2024](https://arxiv.org/html/2611.23910#bib.bib10);[Wijk et al\., 2024](https://arxiv.org/html/2611.23910#bib.bib11)\); here the model serves a large\-scale ranker and moved top\-line metrics in a controlled experiment \(\+0\.23% search volume\), not a held\-out score\. Several of its accepted changes \(RFF dense gating\([Rahimi and Recht, 2007](https://arxiv.org/html/2611.23910#bib.bib1)\), learned\-sigmoid attention gating, a zero\-init gated distillation bypass\) were mechanisms the team had not previously tried, drawn from the broader industry and literature\. The agent’s contribution was sourcing and composing them for a large\-scale ranker where they had not been applied\. The transferable lesson for autonomous ML research is that the leverage sits in the loop, not the model: a calibrated objective, structured context, and parallel evaluation turned a general LLM into a productive ML engineer on a live industrial system\.
### 3\.7Interpretability
Every model candidate is tracked as a row in a persistent state file: its hypothesis text, the config commit hash, the training\-job ID, its four evaluation metrics \(click and content NE on the full and keyword\+high\-confidence slices\), the resulting weighted\-NE delta, and the accept/reject decision\. We also store the running "What Didn’t Work" blocklist – each rejected direction with its measured regression \(e\.g\., ADAM\-epsilon change \+6\.2% NE, homoscedastic\-uncertainty weighting \+4\.5%, DB\-MTL balancing \+1\.8%\) – and mirror the state file to a spreadsheet for monitoring\. Because every accepted change is an atomic commit, the model replays step by step from the baseline commit, and any number we report traces back to the specific training run that produced it\.
### 3\.8Limitations
The approach has clear limits\.Exploration collapse:the agent can spiral within a failing category – in 1\.0 it went 0\-for\-8 on optimizer tuning before a human redirected it – and the blocklist mitigates but does not prevent this\.Unreliable self\-correction:it sometimes prunes its own bad decisions \(it reverted a quadratic\-gating change after 18 later experiments showed it was net\-negative\) but not consistently\.Diminishing returns:within a single campaign the best model was found well before the end and later iterations added nothing, so more compute on the same search space does not help\.Offline\-online gap:not every offline winner moves online, and the agent has no mechanism beyond the calibrated objective to predict which will – nor does it run the online experiments itself\.
## 4Broader Impact
MLE agents do not replace ML engineers – they change the role fromhypothesis executortoresearch director\. The human contribution shifts from "form a hypothesis, implement it, check the result" to "set the objective, calibrate the reward, diagnose failure modes, and decide what to deploy" – a higher\-leverage role that demands deeper domain expertise, not less\. On credit and authorship: when an agent discovers RFF gating, learned attention gating, sparse pooling, and a dozen other changes, we argue the intellectual credit belongs to the system designers who built the loop, calibrated the objective, and structured the context; the agent is a productive tool\.
Our experience suggests a flywheel \- better agents produce more wins, more wins produce better offline\-to\-online calibration data, and better calibration produces smarter agents \- though returns diminish within a single campaign, so sustaining progress requires expanding the hypothesis space \(research phase, new model surfaces, multi\-agent teams\) rather than merely running more iterations\. We believe the community should prepare for a world where autonomous agents routinely contribute to ML research, and that the field should focus on: \(1\) verifying agent\-generated results with the same rigor applied to human research, \(2\) crediting the human judgment that makes agent research productive, and \(3\) ensuring autonomous research strengthens rather than dilutes scientific standards\. Concretely, we advocate mandatory online/holdout verification of agent claims, durable logging of every experiment and decision, and explicit human accountability for artifacts\.
## Appendix AReproducibility Statement
The primary result is a model\-improvement campaign on an industry\-scale ranker\. To support reproducibility of the*method*, we specify: the calibrated objective of Equation[1](https://arxiv.org/html/2611.23910#S1.E1)\(on the keyword \+ high\-confidence slice\), the 0\.0005 NE variance\-probe win threshold, the six\-step research loop of Section[3\.3](https://arxiv.org/html/2611.23910#S3.SS3), the context\-engineering components \(blocklist, one\-change rule, strict output contracts, anti\-staleness\), and the parallel\-training regime \(seven candidates plus a variance probe per iteration, across 131 iterations\)\. Compute consisted of industry\-scale distributed\-training jobs with 64 GPUs per job, with a median recorded elapsed time of 3\.54 hours \(mean 4\.02; 10th–90th percentile 2\.75–5\.12 hours\) in the 2\.0 campaign\. The architecture discoveries in Table[1](https://arxiv.org/html/2611.23910#S1.T1)are described in sufficient detail to be re\-implemented on comparable multi\-task ranking stacks, and online verification followed the controlled\-experiment protocol in Section[3\.5](https://arxiv.org/html/2611.23910#S3.SS5)\.
## Appendix BDisclosure Statement
#### Agent\(s\) / model\(s\) used
An LLM\-driven MLE agent \(Claude Opus 4\.8, within the harness of Section[3\.3](https://arxiv.org/html/2611.23910#S3.SS3)\) generated the model changes; the human authors designed the harness, curated the campaign, and verified the work\.
- ⊠\\boxtimesA human author has curated this work and verified every claim, figure, methodology detail, and reference\.
## Appendix CReferences
## References
- Rahimi and Recht \(2007\)Rahimi, A\. and Recht, B\. Random Features for Large\-Scale Kernel Machines\.*NeurIPS*, 2007\.
- Tang et al\. \(2020\)Tang, H\. et al\. Progressive Layered Extraction \(PLE\): A Novel Multi\-Task Learning Model\.*RecSys*, 2020\.
- Ma et al\. \(2018\)Ma, J\. et al\. Modeling Task Relationships in Multi\-task Learning with Multi\-gate Mixture\-of\-Experts \(MMoE\)\.*KDD*, 2018\.
- Wang et al\. \(2021a\)Wang, R\. et al\. DCN V2: Improved Deep & Cross Network for Feature Interactions\.*WWW*, 2021\.
- Mao et al\. \(2023\)Mao, K\. et al\. FinalMLP: An Enhanced Two\-Stream MLP Model for CTR Prediction\.*AAAI*, 2023\.
- Wang et al\. \(2023\)Wang, F\. et al\. Gated Deep Cross Network \(GDCN\)\. 2023\.
- Wang et al\. \(2021b\)Wang, Z\. et al\. MaskNet: Introducing Feature\-Wise Multiplication to CTR Ranking\. 2021\.
- Lu et al\. \(2024\)Lu, C\. et al\. The AI Scientist: Towards Fully Automated Open\-Ended Scientific Discovery\. 2024\.
- Jiang et al\. \(2025\)Jiang, Z\. et al\. AIDE: AI\-Driven Exploration for ML Engineering\. 2025\.
- Chan et al\. \(2024\)Chan, J\. et al\. MLE\-bench: Evaluating Machine Learning Agents on ML Engineering\. 2024\.
- Wijk et al\. \(2024\)Wijk, H\. et al\. RE\-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents\. 2024\.
- Wang et al\. \(2026\)Wang, H\., Wu, Y\., Chang, D\., Wei, L\., and Heldt, L\. Self\-Evolving Recommendation System: End\-To\-End Autonomous Model Optimization With LLM Agents\.*arXiv preprint arXiv:2602\.10226*, 2026\.
## NeurIPS Paper Checklist
1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The abstract and Section 1 state the large\-scale system setting, the agent’s role in the experimental loop, and the main offline and online results; Sections 3\.4 and 3\.8 delimit human control and the system’s current limitations\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: Section 3\.8 discusses exploration collapse, unreliable self\-correction, diminishing returns, and the offline–online gap, while Section 3\.4 makes the remaining human interventions explicit\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[N/A\]
14. Justification: The paper reports an empirical large\-scale system study and does not claim new theoretical results or provide theorem\-style analyses\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[No\]
19. Justification: Sections 1\.2 and 3\.1–3\.5 and Appendix A describe the objective, acceptance threshold, research loop, context\-management mechanisms, and evaluation protocol\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[No\]
24. Justification: Appendix A documents the reproducible method components that can be described without releasing proprietary assets\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[No\]
29. Justification: Sections 1\.2, 3\.2, and 3\.5 report the data scale, fixed evaluation\-set sizes, objective, acceptance rule, training duration, and distributed\-training setup\. Some implementation details, including the complete data\-split construction, optimizer configuration, and full model hyperparameters, are not disclosed\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[No\]
34. Justification: Table 2 reports 95% confidence intervals computed using Fieller’s method for the primary online A/B metrics, and Section 3\.5 describes the variance\-probe threshold used offline\. Offline uncertainty is not fully quantified\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: Section 3\.2 and Appendix A report distributed jobs using 64 GPUs per job, with a median recorded elapsed time of 3\.54 hours and a mean of 4\.02 hours in the 2\.0 campaign\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: The work followed the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[No\]
49. Justification: Section 4 discusses potential benefits for ML\-engineering productivity, scientific verification, and human accountability\. It does not yet give a comparably concrete discussion of risks such as reward misspecification, adaptive overfitting, unsafe model changes, increased compute use, or disparate effects across users\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: The work does not release a dataset, pretrained model, or other high\-risk asset\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: External methods and models are credited, and no third\-party code or data is redistributed\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[N/A\]
64. Justification: The submission does not release a new public dataset, codebase, model checkpoint, or benchmark artifact\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: The work involved no crowdsourcing, participant recruitment, researcher\-provided instructions, or compensation\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: The work was not classified as human\-subjects research and therefore did not require IRB approval or an institutional equivalent\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[Yes\]
79. Justification: Sections 3\.1 and 3\.3 describe the controller LLM’s role in architecture research, hypothesis generation, and implementation, and Appendix B identifies the model and separates agent\-generated changes from human design, and steering\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.相似文章
Meta 的机器学习工程智能体对一个搜索排序模型进行了 1,097 次实验
Meta 研究人员推出了一款智能体,能够自主为大规模搜索排序模型提出代码修改、启动训练任务并评估结果。该智能体完成了 1,097 次实验,并报告称,在内部 A/B 测试中,搜索量提升了 0.23%。
MLE-bench:评估机器学习代理在机器学习工程中的表现
# MLE-bench:评估机器学习代理在机器学习工程中的表现 来源:[https://openai.com/index/mle-bench/](https://openai.com/index/mle-bench/) OpenAI 评估机器学习代理在机器学习工程中的表现 我们推出了 MLE-bench,这是一个用于衡量 AI 代理在机器学习工程中表现如何的基准。为此,我们从 Kaggle 精选了 75 个与 ML 工程相关的竞赛,创建了一个多样化的具有挑战性的任务集合,用于测试真实的 ML 工程
@rohanpaul_ai:Meta 发布了 A-MLE,这是一个他们实际构建并部署的AI代理框架,用于自动化跨其广告排名模型的机器学习实验…
Meta 开发并部署了 A-MLE,这是一款AI代理框架,用于自动化其广告排名模型的机器学习实验,展现出相较于通用LLMs的显著性能优势。
@sherryyangML:机器学习工程(MLE)正成为新的智能体前沿。我将分享我们在扩展面向MLE智能体的强化学习方面的成果……
ICLR 2026 的两篇论文展示了小型 RL 训练智能体如何在机器学习工程任务上击败前沿模型,以及 MLE-Smith 如何自动扩展 MLE 工作负载。
MLExpert:迈向面向自主机器学习工程的通用智能体模型
提出了 MLExpert,一个 35B 混合专家(MoE)智能体模型,通过分层监督微调和窗口化交互强化学习(WIRL)自主优化长周期机器学习工程工作流,在 MLE-Bench Lite 上将基模型奖牌率从 37.9% 提升至 72.7%,媲美更大的前沿模型。