DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

Hugging Face Daily Papers Papers
computer-use-agents gui-grounding dataset vision-language-models synthetic-supervision desktop-automation fine-tuning

Summary

DeskForge introduces a controllable desktop environment that composes real applications to generate 1.2M densely annotated desktop observations (DeskForge-1M) for training computer-use agents. Fine-tuning four VLMs on this data yields significant gains on GUI grounding benchmarks (e.g., Qwen3.5-4B improves 11.51 points on ScreenSpot-Pro) and long-horizon task completion.

Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for attention. Existing training data rarely pair such scenes with dense annotations or vary them in a controlled way. We introduce DeskForge, a controllable desktop environment that composes and explores real applications to generate large-scale supervision for computer-use agents. It varies application states, content, window layout, appearance, and resolution, and fuses screenshots, accessibility trees, and window geometry into dense element annotations while recording the outcome of each executed action. Using this environment, we construct DeskForge-1M, a corpus of 1.2M annotated desktop observations containing 159.7M element instances. We fine-tune four vision-language models on 200K grounding examples drawn from DeskForge-1M. All four improve across held-out desktop conditions and on all five external GUI grounding benchmarks; for Qwen3.5-4B, accuracy increases by 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. The gains also translate to long-horizon task completion: under a fixed planner, the fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, with Qwen3.5-4B increasing from 31 to 50 of 119 tasks and from 3 to 15 of 100 tasks, respectively. These results show that controllable composition of real desktop environments provides a scalable source of supervision for improving both GUI grounding and long-horizon computer use. The framework code, the dataset, and the fine-tuned model are available from the project page: https://saidgurbuz.github.io/deskforge/
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:56 PM

Paper page - DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

Source: https://huggingface.co/papers/2610.02320

Abstract

Computer-useagentsneedtoreliablygroundactiontargetsincomplexdesktopscenes,wheremultipleapplications,overlappingwindows,andvisuallysimilarcontrolscompeteforattention.Existingtrainingdatararelypairsuchsceneswithdenseannotationsorvarytheminacontrolledway.WeintroduceDeskForge,acontrollabledesktopenvironmentthatcomposesandexploresrealapplicationstogeneratelarge-scalesupervisionforcomputer-useagents.Itvariesapplicationstates,content,windowlayout,appearance,andresolution,andfusesscreenshots,accessibilitytrees,andwindowgeometryintodenseelementannotationswhilerecordingtheoutcomeofeachexecutedaction.Usingthisenvironment,weconstructDeskForge-1M,acorpusof1.2Mannotateddesktopobservationscontaining159.7Melementinstances.Wefine-tunefourvision-languagemodelson200KgroundingexamplesdrawnfromDeskForge-1M.Allfourimproveacrossheld-outdesktopconditionsandonallfiveexternalGUIgroundingbenchmarks;forQwen3.5-4B,accuracyincreasesby11.51percentagepointsonScreenSpot-Proand10.11pointsonOSWorld-G.Thegainsalsotranslatetolong-horizontaskcompletion:underafixedplanner,thefine-tunedactionmodelssolvemoreWebArena-InfinityandOpenAppstasks,withQwen3.5-4Bincreasingfrom31to50of119tasksandfrom3to15of100tasks,respectively.TheseresultsshowthatcontrollablecompositionofrealdesktopenvironmentsprovidesascalablesourceofsupervisionforimprovingbothGUIgroundingandlong-horizoncomputeruse.Theframeworkcode,thedataset,andthefine-tunedmodelareavailablefromtheprojectpage:https://saidgurbuz.github.io/deskforge/

View arXiv pageView PDFProject pageGitHub4Add to collection

Get this paper in your agent:

hf papers read 2610\.02320

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper2

#### docling-project/DeskForge-Qwen3.5-4B Image-Text-to-Text• 5B• Updated1 day ago • 32 • 2 #### docling-project/DeskForge-Gemma4-E4B Image-Text-to-Text• 8B• Updated1 day ago • 30 • 1

Datasets citing this paper1

#### docling-project/DeskForge-1M Preview• Updated1 day ago • 229 • 14

Spaces citing this paper1

Collections including this paper1

Similar Articles

GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis

arXiv cs.CL

GraphForge introduces an evidence-graph framework that grounds both agent task generation and verification in real files, enabling synthesis of verifiable training trajectories for working agents. Fine-tuning Qwen3.6-27B on 2,169 GraphForge trajectories improves GDPVal, Workspace-Bench-Lite, and SpreadsheetBench II, with rejection fine-tuning yielding further gains.