Paper page - BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

Source: https://huggingface.co/papers/2606.01286 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

BenchEvolver is an evolutionary framework that automatically generates harder coding problems from existing ones, creating challenging benchmarks that maintain validity and diversity while enabling model self-improvement and enhanced training performance.

The rapid progress of frontier large language models has led to widespreadbenchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide usefultraining signal. For instance, onLiveCodeBench,frontier modelsachieve over 99%Pass@1on easy splits and exceed 90%Pass@1on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centricevolutionary frameworkthat automatically transforms existingcoding problemsinto harder variants. Rather than generating problems from scratch, BenchEvolver evolvesreference solutionsthroughstructured transformationsand derives corresponding statements and tests from the evolved solutions. This design grounds generation inexecutable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver toLiveCodeBenchandSciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curateLiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult originalLCB-v6tasks, where frontier-modelPass@1ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enablingself-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3Pass@1gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusabletraining signal.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2606\.01286

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.01286 in a model README.md to link it from this page.

Datasets citing this paper1

#### BenchEvolver/livecodebench-plus Viewer• Updated1 day ago • 91 • 19

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.01286 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.