Paper page - UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Source: https://huggingface.co/papers/2608.15930 Published on Aug 16

·

Submitted byhttps://huggingface.co/lkeab

Lei Keon Aug 18

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

UI-Mate is a foundation GUI agent that uses environment-grounded training and in-context demonstration learning to improve reliability on long-horizon office tasks, achieving state-of-the-art results on computer-use benchmarks.

Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workflows rely on user-specific tools and tacit conventions, so unstated instructions can produce arbitrary variations across runs. We present UI-Mate, afoundation GUI agentthat integrates anenvironment-grounded trainingstack within-context demonstration learning. UI-Mate makes three contributions: A ScalableEnvironment-Grounded TrainingStack: Aclosed-loop data engineautomates task generation, environment construction, rollout, filtering, capability balancing,SFT, andonline RLacross massively parallel environments via unifiedtask-verifier bundles.In-Context Demonstration Learning: A mechanism that transformsmultimodal demonstrationsinto flexiblesubtask-level workflows, follows relevant demonstrated steps, and re-plans from the live interface.OSWorkerBenchBenchmark and Insights: A benchmark of 100 long-horizon office tasks across 41 applications that supports instruction-only and demonstration-guided evaluation. Its demonstration resources separate a 33-taskself-demosetting, built from successful strong-agent rollouts of the same targets, from a 45-taskvariant-demosetting, built from human recordings of related but non-identical tasks. Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% onOSWorld-Verifiedand 66.2% onWindowsAgentArena. OnOSWorkerBench, it reaches 41.0% strict success and 76.9% progress, outperforming its Qwen3.6-27B base by 17.7 and 24.5 points. On the 33-taskself-demosubset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, substantially improving long-horizon reliability. Project page: https://ui-mate.github.io.

View arXiv pageView PDFProject pageGitHubAdd to collection

Get this paper in your agent:

hf papers read 2608\.15930

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.15930 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.15930 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.15930 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.