CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Summary
CLI-Universe is a synthesis engine that generates verifiable terminal-agent tasks via multi-dimensional capability taxonomy and evidence-guided research, producing a distilled dataset of 6,000 trajectories. Fine-tuning Qwen3-32B on this dataset achieves 33.4% on Terminal-Bench 2.0, setting a new state-of-the-art for open-source models at or below 32B parameters.
View Cached Full Text
Cached at: 06/23/26, 05:40 AM
Paper page - CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Source: https://huggingface.co/papers/2606.22883 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
A principled synthesis engine generates high-quality terminal-agent tasks through multi-dimensional capability taxonomy and evidence-guided research, creating a distilled dataset that enables significant performance gains in LLM training.
While recentLLM-based terminal agentshave demonstrated promising capabilities, the scarcity of high-quality,executable training dataremains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principledsynthesis enginethat constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensionalcapability taxonomy(domain, skill type, capability, and engineering pillar), then grounds each candidate throughevidence-guided deep researchover real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated intoDockerized environmentsand subjected to a multi-stageexecutable verification pipelinefeaturingrubric-gated test construction, hint-conditional filtering, and strictfail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories calledCLI-Universe-6K. Remarkably, fine-tuningQwen3-32BonCLI-Universe-6Kachieves 33.4% onTerminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.22883
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.22883 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.22883 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.22883 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments
Terminal-Universe is a framework that reconstructs executable workspaces from agent trajectories to synthesize diverse training tasks, improving AI agent post-training performance through supervised fine-tuning.
Scaling Agentic Data for Long-Horizon Terminal Intelligence
The paper introduces TerminalHorizon, an automated data-synthesis engine that generates long-horizon terminal agent trajectories by reconstructing environments around real-world professional tasks, then fine-tunes Qwen3.5-27B to obtain TerminalHorizon-27B, achieving gains across 12 coding, agentic, and reasoning benchmarks.
Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
Terminal-World introduces a fully automated pipeline that uses agent skills to synthesize high-quality training data for terminal agents, enabling models to outperform baselines with only 1.2% of the training data. The method co-derives task instructions, environments, and teacher trajectories from skill primitives.
RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
The paper introduces REALCLI, a paired benchmark that jointly evaluates CLI agent task execution and human-agent collaboration quality, revealing that frontier agents score only 49.2% on tasks and 37.6% on joint collaboration, with major gaps in steerability, trust calibration, and safety boundary management.
GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis
GraphForge introduces an evidence-graph framework that grounds both agent task generation and verification in real files, enabling synthesis of verifiable training trajectories for working agents. Fine-tuning Qwen3.6-27B on 2,169 GraphForge trajectories improves GDPVal, Workspace-Bench-Lite, and SpreadsheetBench II, with rejection fine-tuning yielding further gains.