tencent/UI-Mate-27B · Hugging Face

Source: https://huggingface.co/tencent/UI-Mate-27B

https://huggingface.co/tencent/UI-Mate-27B#ui-mate-advancing-open-weight-foundation-gui-agents-with-in-context-demonstrationsUI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Show the workflow once. Let the agent adapt it to the task at hand.

Tencent HY Frontier · Multimodal Agent Team

Project Page·GitHub

https://huggingface.co/tencent/UI-Mate-27B#overviewOverview

UI-Mate-27Bis an open-weight foundation GUI agent for long-horizon work across applications and operating systems. It observes live screenshots, reasons over the visible state, and produces structured keyboard and mouse actions for native desktop interaction.

UI-Mate supports two complementary modes:

• **General computer use:**execute tasks from natural-language instructions and live screenshots.
• **Demonstration-guided computer use:**adapt a reusable workflow extracted from one successful demonstration to a new task.

A demonstration is treated as guidance rather than a fixed action script. The model continues to re-plan from the live interface when the content, layout, or application state differs.

https://huggingface.co/tencent/UI-Mate-27B#model-detailsModel Details

• **Parameters:**27B
• Base model:Qwen3.6-27B
• **Input:**task instruction, screenshots, interaction history, and optional demonstration context
• **Output:**reasoning, a concise action description, and structured computer-use tool calls
• **Action space:**mouse, keyboard, scrolling, waiting, user interaction, and task completion
• **Training:**supervised fine-tuning followed by online reinforcement learning in executable GUI environments
• **License:**Apache-2.0

UI-Mate is an agent checkpoint rather than a standalone visual-chat model. We recommend using the official prompt, response parser, and interaction harness from theUI-Mate repository.

https://huggingface.co/tencent/UI-Mate-27B#highlightsHighlights

• Strong general computer-use performance across Ubuntu and Windows benchmarks.
• Long-horizon execution across multiple applications.
• One-shot procedural learning from demonstrations.
• Live-screen grounding instead of coordinate replay.
• Structured actions compatible withpyautogui.
• OpenAI-compatible serving and client interface.

https://huggingface.co/tencent/UI-Mate-27B#evaluationEvaluation

https://huggingface.co/tencent/UI-Mate-27B#instruction-only-executionInstruction-only execution

BenchmarkUI-Mate-27BOSWorld-Verified · average score77.0WindowsAgentArena · average score66.2OSWorkerBench · strict success41.00OSWorkerBench · progress76.86

https://huggingface.co/tencent/UI-Mate-27B#demonstration-guided-executionDemonstration-guided execution

Evaluation set · metricInstruction only+ one demonstrationChangeOSWorkerBench-Subset (33) · strict success17.1735.35+18.18 ppOSWorkerBench-Subset (33) · progress67.8581.14+13.29 ppOSWorld-Subset (30) · progress40.2765.75+25.48 ppGameDev (10) · average score76.7681.15+4.39 pp On GameDev, demonstration guidance also reduces the average trajectory length from 303.6 to 253.1 steps.

Results are averaged over three runs per target on OSWorkerBench-Subset and five runs elsewhere. See theproject pagefor evaluation details and updates.

https://huggingface.co/tencent/UI-Mate-27B#quick-startQuick Start

https://huggingface.co/tencent/UI-Mate-27B#1-serve-the-model-with-vllm1. Serve the model with vLLM

vLLM automatically downloads the checkpoint from Hugging Face Hub on first launch.

pip install -U vllm openai pillow

vllm serve tencent/UI-Mate-27B \
    --trust-remote-code \
    --served-model-name UI_Mate \
    --port 8000 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.85 \
    --mm-encoder-tp-mode data \
    --chat-template-content-format openai \
    --limit-mm-per-prompt '{"image":6,"video":0}'

The default agent retains five screenshots in context. The server must therefore admit at least six images because the newest screenshot arrives before the oldest one is collapsed.

Confirm that the endpoint exposes the expected model name:

curl -s http://127.0.0.1:8000/v1/models

https://huggingface.co/tencent/UI-Mate-27B#2-run-the-reference-examples2. Run the reference examples

git clone https://github.com/Tencent/UI-Mate.git
cd UI-Mate

python examples/run_agent.py \
    --base-url http://127.0.0.1:8000/v1

Run a complete recorded trajectory:

python examples/run_agent.py \
    --replay \
    --base-url http://127.0.0.1:8000/v1

Or provide your own screenshot:

python examples/run_agent.py \
    --image /path/to/screen.png \
    --instruction "Export this spreadsheet as HTML and open it in Chrome"

https://huggingface.co/tencent/UI-Mate-27B#3-use-the-python-agent3. Use the Python agent

from agents.ui_mate_agent import UIMateAgent

agent = UIMateAgent(
    base_url="http://127.0.0.1:8000/v1",
    model="UI_Mate",
)

with open("screen.png", "rb") as f:
    response, actions = agent.predict(
        "Install the autoDocstring extension in VS Code.",
        {"screenshot": f.read()},
    )

print(response)
print(actions)

# Reset the interaction history before starting a new task.
agent.reset()

The model reasons in a normalized1000 × 1000coordinate space. The reference agent automatically rescales predicted coordinates to the original screenshot resolution.

https://huggingface.co/tencent/UI-Mate-27B#demonstration-guided-execution-1Demonstration-Guided Execution

A UI-Mate demonstration records screenshots immediately before and after each keyboard or pointer action. The trace is then:

1. normalized into a consistent action-and-frame representation;
2. annotated with observations, intent, actions, and verification evidence;
3. segmented into named subtasks with completion criteria; and
4. injected at inference time as a compact workflow for the active subtask.

The live screenshot remains authoritative throughout execution. For demonstration recording and management, see theUI-Mate project page.

https://huggingface.co/tencent/UI-Mate-27B#intended-use-and-limitationsIntended Use and Limitations

UI-Mate-27B is intended for research and development of screenshot-based GUI agents in controlled desktop environments.

Its behavior can be affected by application versions, screen layouts, display scaling, latency, and unexpected UI state. Benchmark performance does not guarantee reliable execution in arbitrary environments, and the model requires an external runtime to execute its predicted actions.

https://huggingface.co/tencent/UI-Mate-27B#safetySafety

Computer-use agents can make mistakes, encounter prompt injection, or trigger consequential actions.

• Prefer isolated or disposable environments.
• Avoid unattended, high-stakes, or destructive workflows.
• Require human confirmation before sensitive operations.
• Monitor the interaction trajectory and verify the resulting application state.
• Do not treat a model-reported success as proof that the intended outcome was achieved.

https://huggingface.co/tencent/UI-Mate-27B#licenseLicense

UI-Mate is released under theApache License 2.0. Third-party components remain subject to their respective licenses. See the repositoryLICENSEfor details.

https://huggingface.co/tencent/UI-Mate-27B#citationCitation

Citation details will be updated when the technical report is publicly released.

@article{uimate2026,
  title   = {UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations},
  author  = {{Tencent HY Frontier Multimodal Agent Team}},
  journal = {arXiv preprint},
  year    = {2026}
}