How much of MLE-Bench's gains are the algorithm vs. better models + more search? [R]
Summary
A new benchmark, FML-Bench, reveals that recent improvements in MLE-Bench scores are largely due to better base models and increased search budget rather than algorithmic advances.
Similar Articles
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
OpenAI introduces MLE-bench, a benchmark of 75 Kaggle ML competitions to evaluate AI agents on real-world ML engineering tasks. The best setup, o1-preview with AIDE scaffolding, achieves at least a Kaggle bronze medal in 16.9% of competitions.
@lu__jasper: Some early results from playing around with search on a subsampled version of OBLIQ-bench. Mixedbread's reranker is a b…
Early results from testing search on a subsampled OBLIQ-bench show that Mixedbread's reranker achieves strong MRR, sometimes outperforming GPT 5.5 on certain metrics with faster speed, though the benchmark remains challenging.
The Two MMLU Scores: What a Benchmark Name Does Not Fix (23 minute read)
The article examines the issues with MMLU benchmark scores, showing that different model builds can yield incomparable accuracies due to open evaluation variables, and proposes a content-addressed framework for better traceability.
Why do benchmark results go up every release, always?
A discussion post questions why AI benchmark scores consistently improve each release even when users perceive little real-world gains, asking whether providers game proprietary, closed-source benchmarks rather than genuinely improving models.
DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories
Introduces DeltaML-Bench, a benchmark for evaluating machine learning agents on real-world research repositories, showing that ARG scaffolding significantly improves success rates over standard methods.