CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
Summary
CalibForge is an autonomous terminal-task synthesis system that uses adversarial solver calibration to create learnable tasks for training terminal agents. It constructs 5,431 calibrated tasks and improves agent performance on Terminal-Bench2.0, SWE-bench Pro, and Doc2Repo.
View Cached Full Text
Cached at: 08/07/26, 05:55 AM
Paper page - CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
Source: https://huggingface.co/papers/2608.06352
Abstract
Trainingterminalagentsrequiresexecutableandverifiabletasksthatarenotmerelysolvable,butappropriatelychallengingforlearning.Executablevalidationestablishesfeasibility,yetdoesnotrevealhowataskbehavesrelativetoagivensolversetting.Inthispaper,wepresentCalibForge,anautonomousterminal-tasksynthesissystemthatusesverifiedsolverbehaviortorevisecandidatetasksthroughadversarialsolvercalibration.Multi-solvercalibrationtargetsdisagreementwithinaheterogeneoussolverpool,whereascontrastivesolvercalibrationtargetsadesignatedstrong-pass/weak-failrelation;bothoperationalizeasolver-relativelearnablezoneanchoredindemonstratedsolvability.UsingCalibForge,weconstruct5,431calibratedterminaltasks.Ourablationsshowthatbothstrategiesyieldmoreeffectivesupervisionthanauthoringandvalidationaloneorordinarysingle-solverfeedback.Modelstrainedonthefullcollectionachieve32.58%and47.57%onTerminal-Bench2.0.Thelargestimprovementsoverthecorrespondingbasemodelreach24.71percentagepointsonTerminal-Bench2.0,27.68pointsonSWE-benchPro,and30.04pointsonDoc2Repo.Together,theseresultssupportsolver-relativelearnabilityasapracticaltargetforconstructingeffectiveandtransferableagenttrainingdata.
View arXiv pageView PDFGitHub4Add to collection
Get this paper in your agent:
hf papers read 2608\.06352
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.06352 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.06352 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.06352 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
CLI-Universe is a synthesis engine that generates verifiable terminal-agent tasks via multi-dimensional capability taxonomy and evidence-guided research, producing a distilled dataset of 6,000 trajectories. Fine-tuning Qwen3-32B on this dataset achieves 33.4% on Terminal-Bench 2.0, setting a new state-of-the-art for open-source models at or below 32B parameters.
CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks
CalVerT introduces calibrated verifier telemetry for LLM agents, providing confidence and grounding scores to reduce over-retrieval and unsupported answers, improving performance on knowledge-intensive QA tasks.
LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents
LiteCoder-Terminal-Gen introduces a zero-dependency synthetic pipeline that generates executable terminal training environments, producing SFT and RL datasets that enable language agents to achieve significant performance gains on Terminal Bench benchmarks.
Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
Terminal-World introduces a fully automated pipeline that uses agent skills to synthesize high-quality training data for terminal agents, enabling models to outperform baselines with only 1.2% of the training data. The method co-derives task instructions, environments, and teacher trajectories from skill primitives.
NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs
NexForge is a requirement-driven framework that synthesizes diverse, executable agent tasks and expert trajectories for LLM post-training, outperforming prior methods and achieving state-of-the-art open-source agent performance on Terminal-Bench.