Tag
介绍 Simular 的 Agent S4 论文：通过元智能体从任务轨迹中自动构建并进化计算机操作接口，使开源 Qwen3.8-27B 模型在 OSWorld V2 上达到 21.3%，超越 GPT-5.5 和 Claude Opus 4.8，且成本仅为后者的极小部分。
论文提出 MAGE 框架，利用用户手册作为可扩展的程序性知识来源，同时改进计算机使用代理（CUA）的推理与训练，在六个应用上提升了任务成功率并减少了对人工标注数据的依赖。
作者分享了将桌面智能体的规划与执行分离的架构：由 Claude Code 等前沿模型批量规划步骤，小型快速模型 Jev 负责执行点击与操作，普通步骤跳过模型，从而显著提升速度与降低成本，测试中 12 个随机按钮达成 47/48、约 1.2 秒每次点击。代码已开源。
OS-Helix is an end-to-end trained multimodal agent that unifies GUI actions and code execution in a single action space, using hybrid SFT plus online RL with a Step-balanced GSPO objective to reach 66.44% success on OS-World-Verified while showing cross-platform transfer to WindowsAgentArena.
OpenGCUA introduces an open generalist computer-use agent stack combining visual and programmatic interfaces, code execution, and harness-to-model decision transfer, improving task scores across eight models on OSWorld-v2 while reducing steps per task by 53%.
OpenMobile-2 introduces MobileGym++, a scalable environment with 110+ controllable commercial-style apps, and a hybrid GUI/CLI setting for mobile agents, achieving nearly 2× performance on real-device tasks with all environments, data, benchmarks, and models fully open-sourced.
iphone-use 开源工具让 AI 智能体操控真实 iPhone 屏幕（tap/type/scroll via WebDriverAgent），并提供 MCP server、HTTP API 与浏览器远程控制，即使是无 API 的应用也能被自动化。
UI-Venus-2 is an open-source multimodal GUI agent designed for digital automation across mobile, web, and desktop environments, using unified reasoning-action loops and robust verification to enable reliable real-world applications.
This article argues that current AI computer-use agents often bypass the actual user interface by directly accessing APIs or scripting, leading to inflated benchmark results and unreliable real-world performance. It suggests that ignoring the interface is a flawed approach that does not generalize to messy GUI environments.
MobileForge presents an annotation-free adaptation system for mobile GUI agents that uses real app interaction and hierarchical feedback-guided policy optimization to improve performance, achieving near state-of-the-art results on AndroidWorld with open data.
MIRAGE is a framework for mobile GUI agents that replaces verbose chain-of-thought reasoning with compact continuous latent representations, incorporating a generative world model perspective to predict future screen states before acting. On AndroidWorld and AndroidControl benchmarks, it achieves competitive or superior performance while reducing generated tokens by over 75%.
This paper introduces PRO-CUA, a process-reward optimization framework for training Computer Use Agents (CUAs) using iterative step-level reinforcement learning. The method decouples on-policy environment interaction from policy optimization, enabling dense credit assignment without relying on expert trajectories, and demonstrates effectiveness on live web benchmarks.
AutoRPA is a framework that automatically distills the decision logic of ReAct-style LLM agents into robust, token-efficient RPA functions for repetitive GUI tasks, reducing token usage by 82-96%.
OpenGUI is a tool that allows AI agents to directly operate real Android apps by reading the screen and interacting naturally, rather than relying on APIs or scripts.
The author discusses building a small VLM for desktop GUI automation to move data between apps without APIs, expressing interest in non-coding autonomous use cases for local models.
ToolCUA is a new agent framework that optimizes GUI-tool path selection for computer use agents through staged training and reinforcement learning. It achieves state-of-the-art performance on OSWorld-MCP by effectively interleaving GUI actions and high-level tool calls.
ByteDance open-sourced UI-TARS-desktop, a native desktop GUI agent with 31.4k GitHub stars that uses vision models to control local or remote applications via natural language. The tool runs locally for privacy, supports Windows and macOS, and includes a CLI with streaming output for developers.
Agent S2 is a new compositional framework for computer use agents that achieves state-of-the-art performance on multiple benchmarks by utilizing Mixture-of-Grounding and Proactive Hierarchical Planning.
OpenAI introduced the Computer-Using Agent (CUA), a model combining GPT-4o's vision with reinforcement learning to interact with GUIs like a human, powering the new Operator agent. CUA sets new state-of-the-art benchmarks including 38.1% on OSWorld and 58.1% on WebArena, and is available as a research preview for ChatGPT Pro users in the US.