FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
Summary
FrontierSmith automatically generates diverse open-ended coding problems from closed-ended tasks, improving LLM coding performance on benchmarks through enhanced agent interactions and training data synthesis.
View Cached Full Text
Cached at: 05/15/26, 08:24 AM
Paper page - FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
Source: https://huggingface.co/papers/2605.14445 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
FrontierSmith automates the creation of open-ended coding problems from closed-ended tasks, improving LLM coding performance on benchmarks through diverse problem variants and enhanced agent interactions.
Many real-world coding challenges are open-ended and admit no known optimal solution. Yet, recent progress inLLM codinghas focused on well-defined tasks such as feature implementation, bug fixing, andcompetitive programming.Open-ended codingremains a weak spot for LLMs, largely because open-ended training problems are scarce and expensive to construct. Our goal is to synthesizeopen-ended codingproblems at scale to train stronger LLM coders. We introduce FrontierSmith, an automated system for iteratively evolving open-ended problems from existing closed-ended coding tasks. Starting fromcompetitive programmingproblems, FrontierSmith generates candidate open-ended variants by changing the problems’goals, restricting outputs, and generalizing inputs. It then uses a quantitativeidea divergence metricto select problems that elicit genuinely diverse approaches from different solvers. Agents then generate test cases and verifiers for the surviving candidates. On twoopen-ended codingbenchmarks, training on our synthesized data yields substantial gains over the base models: Qwen3.5-9B improves by +8.82 score onFrontierCSand +306.36 (Elo-rating-based performance) onALE-bench; Qwen3.5-27B improves by +12.12 and +309.12, respectively. The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones, suggesting that closed-ended seeds can be a practical starting point for long-horizon coding data.
View arXiv pageView PDFProject pageGitHub4Add to collection
Get this paper in your agent:
hf papers read 2605\.14445
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.14445 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.14445 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.14445 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@MangQiuyang: Open-ended coding training data may no longer be the bottleneck: AI can scale open-ended tasks—and even outperform huma…
FrontierSmith is a system that synthesizes open-ended coding problems at scale from closed-ended tasks. It generates, filters, and builds training environments; models trained on its data outperform those trained on human-curated open-ended data.
FrontierCode: a coding eval that raises the bar for difficulty & quality.
FrontierCode is a new coding evaluation benchmark designed to increase difficulty and quality standards for AI code generation.
BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
BenchEvolver is an evolutionary framework that automatically generates harder coding problems from existing ones, creating challenging benchmarks that maintain validity and diversity while enabling model self-improvement and enhanced training performance.
Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
This paper evaluates six frontier coding agents on esoteric programming languages and finds that stronger agents use metaprogramming—writing Python programs to generate and debug code in the unfamiliar target language. Forbidding this strategy causes large performance drops, while providing Python helper code improves weaker agents.
AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable
This paper evaluates LLM-based coding agents (Claude Code and Codex) in social science analysis, finding they match or exceed human methodological diversity while remaining vulnerable to interpretation bias through verdict-layer manipulation.