Paper page - ProCUA-SFT Technical Report

Source: https://huggingface.co/papers/2606.17321 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Training computer-use agents using a large-scale synthetic dataset with automated task generation and verification achieves significantly improved performance on desktop interaction benchmarks.

Trainingcomputer-use agents(CUAs) -- models that interact with graphical desktops through screenshots and keyboard/mouse actions -- requires large-scale, diverse trajectory data collected in full desktop environments. The largest public resource, AgentNet (22.5K human trajectories), leads to negative transfer when used forsupervised fine-tuning(SFT): continuing trainingUI-TARS7B on AgentNet causesOSWorldsuccess rate to fall from 26.3% to 8-10%. We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93Ksynthetic trajectoriesacross 2,484 application combinations. The dataset is produced by a fully automated pipeline that (i) synthesizes grounded tasks on live desktops seeded with real-world content -- 912 spreadsheets from SpreadsheetBench, approximately 10K permissively-licensed presentations from Zenodo10K, and multi-applicationOSWorldconfigs -- and (ii) verifies each task’s feasibility through binaryprecondition checkingbefore rollout. A singleVLM(Kimi-K2.5) serves as goal generator, precondition judge, and trajectory executor, eliminating planner-actor capability gaps. Each trajectory is expanded intostep-prefix samplesthat exactly reproduce the context layout seen at inference time. Fine-tuningUI-TARS7B on ProCUA-SFT for one epoch yields 45.0% onOSWorld-- an 18.7 percentage-point improvement over the base model and over 35% above AgentNet-trained counterparts. A subset of ProCUA was incorporated into the training data for the Nemotron 3 Nano Omni model, contributing to its computer-use capabilities.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2606\.17321

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.17321 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.17321 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.17321 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.