How much of MLE-Bench's gains are the algorithm vs. better models + more search? [R]

Reddit r/MachineLearning 06/01/26, 02:34 PM Papers

ml-bench automated-ml benchmark algorithmic-efficiency research-critique agent-evaluation

Summary

A new benchmark, FML-Bench, reveals that recent improvements in MLE-Bench scores are largely due to better base models and increased search budget rather than algorithmic advances.

MLE-Bench scores have jumped from 30% to 80% over the last two years. But how much of that is real algorithmic progress vs. better base models + problem definition shifts + overfitting? Turns out: not much. Once you control for the same step budget and models, and then test on a different set of tasks, the two-year-old AIDE algorithm matches modern agent/evolutionary search systems. Figure from FML-Bench, a new automated ML research benchmark, which unifies the code editing agent, step definition, and val/test split, and tries to benchmark the algorithmic efficiency (search/memory) of the agents. paper link: [https://arxiv.org/pdf/2605.17373](https://t.co/8QllTan4cX) [test improvement and pairwise win-rate](https://preview.redd.it/j9ev4x8kmo4h1.png?width=894&format=png&auto=webp&s=d38392fdc4d14f371db23592d8ef45b34bc1e7a8)

Original Article

Similar Articles

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

OpenAI Blog

OpenAI introduces MLE-bench, a benchmark of 75 Kaggle ML competitions to evaluate AI agents on real-world ML engineering tasks. The best setup, o1-preview with AIDE scaffolding, achieves at least a Kaggle bronze medal in 16.9% of competitions.

@lu__jasper: Some early results from playing around with search on a subsampled version of OBLIQ-bench. Mixedbread's reranker is a b…

X AI KOLs Following

Early results from testing search on a subsampled OBLIQ-bench show that Mixedbread's reranker achieves strong MRR, sometimes outperforming GPT 5.5 on certain metrics with faster speed, though the benchmark remains challenging.

MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

Hugging Face Daily Papers

This paper introduces MLS-Bench, a benchmark designed to assess whether AI systems can invent generalizable and scalable machine learning methods rather than just performing engineering tuning.

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

arXiv cs.AI

InferenceBench is a benchmark that evaluates AI agents on optimizing LLM inference speed using an H100 GPU across multiple bottleneck scenarios. Results show agents improve over naive baselines but frequently converge on single frameworks and underperform simple hyperparameter searches, indicating a need for better exploration strategies.

Compute Allocation in Evolutionary Search: From Depth-Breadth to Multi-Armed Bandits

arXiv cs.CL

This paper studies compute allocation in LLM-guided evolutionary search, identifies empirical regularities, and proposes BaSE, a multi-armed bandit algorithm that improves mean fitness and reliability across multiple models and tasks.