Must-read paper from Meta on reliable auto-research agents.

If you let coding agents run ML experiments, one silent bug like leaked eval data or a disconnected gradient can invalidate hours of training and every iteration built on it.

RankEvolve enforces each research phase and gate through a compiled protocol. It runs Claude Code and Codex as separate nodes that review and repair each other’s changes.

At a matched budget, combining the two products raises execution accuracy from 45.8% for the best single product to 62.5%.

Over twelve iterations on the open-source HSTU recommender, it improved NDCG@10 on MovieLens-20M by 4.48% over the published result.

Paper: https://academy.dair.ai/papers/rankevolve-a-reliable-multi-agent-auto-research-harness-for-evolving-ranking-mod-2609.39551…

RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

Source: https://academy.dair.ai/papers/rankevolve-a-reliable-multi-agent-auto-research-harness-for-evolving-ranking-mod-2609.39551 AgentsChat with Paper

First page

The curator’s take

Zheng Chen, Linfeng Liu, Hong Li and Hong Yan at Meta present RankEvolve, an auto-research framework for generative ranking models that treats execution accuracy (whether each code change is implemented correctly) as the main constraint on long research runs.

Ask this paper

Question about this paper Key points01

Failure model. A single silent defect, such as leaked held-out data, a missing layer norm, a disconnected gradient or an unwired train/eval flag, wastes hours of accelerator time and produces an invalid metric that later iterations build on.

02

Executable Operating Protocol. Phases, gates, branches and loops are declared in an EOP and compiled to a state machine that the runtime enforces, instead of being left to the agent’s prompt.

03

Composing coding-agent products. A meta-meta-harness uses complete products such as Claude Code and Codex as graph nodes that review and repair one another; in a budget-matched test this raises all-oracle execution accuracy from 45.8% (best single product) to 62.5% (+16.7 points, 95% CI 6.6 to 26.7) with a 10.4% silent critical-defect rate.

04

Deployment. Twelve iterations on the open-source HSTU recommender reached NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48% over the published anchor); a LitGPT split reproduces the composition effect (+12.5 points).

AbstractAuto-research agents, LLM systems that propose, implement, train, and evaluate model changes across iterations, promise to automate applied ML’s experimental loop. Over long horizons, execution accuracy is a binding constraint: a change can silently leak held-out data, omit normalization, disconnect a gradient, or leave a train/eval flag unwired, invalidating expensive runs and compounding error across iterations. We present RankEvolve, an auto-research framework for evolving generative ranking models. An Executable Operating Protocol (EOP) declares phases, gates, branches, and loops, and the runtime enforces the compiled state machine. A meta-meta-harness composes complete black-box coding-agent products, including Claude Code and Codex, as execution-graph nodes that review and repair one another’s work. In a budget-matched evaluation, heterogeneous composition raises all-oracle execution accuracy from the best single-product baseline of 45.8 percent to 62.5 percent (paired +16.7 points, 95 percent CI [6.6, 26.7]) while achieving a 10.4 percent silent critical-defect rate. An implemented knowledge layer carries findings, including negative results, across iterations. In a twelve-iteration deployment on the open-source HSTU recommender, RankEvolve reported NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48 percent over the published anchor) and 0.1948 on BASE (+2.80 percent). ExecML-HSTU, seeded by incidents from that deployment, provides the oracle benchmark for the execution-accuracy evaluation. A pre-specified LitGPT transfer split replicates the heterogeneous-composition effect beyond recommendation (+12.5 points, 95 percent CI [3.0, 22.0]), and a paired ablation isolates per-step from full-protocol instruction injection. These results characterize when runtime-controlled composition of coding-agent products improves execution accuracy.