SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

Hugging Face Daily Papers Papers
agent-based-program-synthesis behavioral-specification requirements-engineering program-synthesis code-generation ai-agents

Summary

SpecFirst proposes making behavioral specification elicitation a first-class step in agent-based program synthesis, separating it from exploration and coding. Evaluated on 200 program instances, it significantly improves test pass rates and binary exploration coverage across multiple models.

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.
Original Article
View Cached Full Text

Cached at: 07/30/26, 09:50 PM

Paper page - SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

Source: https://huggingface.co/papers/2607.27167 SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

We believe behavioral specification elicitation should be a first-class step in program synthesis from scratch, rather than being implicitly mixed with exploration and coding. With SpecFirst, we:

πŸ“‹ Separate synthesis into two stages: a specification agent first probes an execute-only binary and combines its observations with the documentation into a structured behavioral specification πŸ“‹ Give a second agent this specification as a stable reference for implementation, instead of requiring it to continually recover behavioral intent from a growing interaction history πŸ“‹ Resolve documentation ambiguities before coding begins and decouple behavioral exploration from code construction

We evaluate SpecFirst on all 200 ProgramBench instances, using four models across two model families spanning nearly an order of magnitude in agentic coding capability. We compare it against a controlled single-loop Direct-Synthesis baseline using the same model, official mini-SWE-agent scaffold, and otherwise matched experimental settings.

The result: πŸ“ˆ Improves test pass rates by 6.9%–21.3% over Direct-Synthesis across all four evaluated models πŸ“ˆ Increases binary exploration coverage by 9.4%–18.5%, showing that the dedicated specification phase leads to more extensive behavioral probing πŸ“ˆ Produces statistically significant improvements across every evaluated model, suggesting that the benefit extends across model families and capability levels

Perhaps most interesting: the specification changes how agents code. Behavioral analysis shows that agents equipped with a prior specification begin code construction earlier and sustain it for longer, rather than repeatedly switching between behavioral exploration and implementation. Our results suggest that introducing an explicit requirements-engineering phase into the agent loop is a simple but effective paradigm for building programs from scratch.

Similar Articles

FVSpec: Real-World Property-Based Tests as Lean Challenges

Hugging Face Daily Papers

This paper presents FVSpec, a benchmark for AI-assisted formal verification that translates real-world property-based tests from Python into Lean 4 specifications using a multi-agent LLM pipeline, aiming to drive progress on formal verification of real-world software.