Paper page - MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

Source: https://huggingface.co/papers/2605.08678 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Current AI agents struggle to invent generalizable and scalable machine learning methods, relying more on engineering tuning than true method discovery, with performance bottlenecks stemming from scientific insight rather than computational resources.

Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes. Aslarge language modelsdemonstrate advanced capabilities in reasoning, coding, and engineering tasks, it is increasingly important to understand whether they can discover such methods rather than only apply existing ones. We introduceMLS-Bench, a benchmark for evaluating whether AI systems can invent generalizable andscalable ML methods.MLS-Benchcontains 140 tasks across 12 domains, each requiring an agent to improve one targeted component of an ML system or algorithm and demonstrate that the improvement generalizes across controlled settings and scales. We find that current agents remain far from reliably surpassing human-designed methods, and thatengineering-style tuningis easier for them than genuinemethod invention. We further study the effects oftest-time scaling,adaptive compute allocation, andcontext provisionon agents’ discovery performance, together with case studies of their behavior. Our analyses suggest that the bottleneck is not only in proposing new methods, but also in thescientific insightneeded to plan, validate, and scale claims about them. More search, compute, or context alone does not remove this bottleneck. We build and maintain a community platform for cumulative andcomparable iteration, and release the data and code at https://mls-bench.com.

View arXiv pageView PDFProject pageGitHub10Add to collection

Get this paper in your agent:

hf papers read 2605\.08678

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.08678 in a model README.md to link it from this page.

Datasets citing this paper1

#### Bohan22/MLS-Bench-Tasks Viewer• Updatedabout 10 hours ago • 140 • 42

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.08678 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.