Tag
Vantage is an open-source Windows 11 app that turns a typed prompt into autonomous desktop actions, using AI models to read actual window text via UI Automation, click, type, and run PowerShell rather than relying on screenshots. It brings your own API key for any OpenAI-compatible provider, stores everything locally, and is an early-stage project still seeking failure-case feedback.
Cursor now lets AI agents control desktop applications like the iOS Simulator, enabling them to test the features they build directly on the machine.
Someone has extracted Codex's Computer Use runtime from the ChatGPT desktop app into an open-source tool called LCU, letting agents like Claude Code, Pi, and Hermes see the screen, click buttons, and operate desktop apps — no Codex login required.
Hermes v0.22.0 (The Threshold Release) introduces its flagship feature Bot Screen, giving AI agents their own dedicated Linux gateway desktop with live-streaming monitoring, on-demand human takeover for 2FA and CAPTCHA verification, and continued background operation even after the local client is closed — delivering a new level of human-AI collaborative automation.
Betrox AI is an autonomous multi-worker laptop agent that controls the Windows OS via screen vision and native UI Automation, runs on BYOK API keys with local encryption, and self-heals from UI errors. The developer is seeking community feedback ahead of a planned 2027 alpha and launch.
该论文提出 DeskIF-Bench 基准（跨6款桌面应用的255个任务）研究软件界面演化对 LLM 代理性能的影响，并开发 ANIS 框架从已安装应用能力合成代理原生接口，在368个任务上显著提升成功率并降低约73%的 token 用量。
Semwright 是一款开源 Rust 运行时，让 AI 智能体通过结构化驱动器访问桌面及专业软件的应用对象与项目数据，并统一处理权限、执行、依赖、产物交接与验证，支持 MCP 但不局限于单一协议或模型。
DeskForge introduces a controllable desktop environment that composes real applications to generate 1.2M densely annotated desktop observations (DeskForge-1M) for training computer-use agents. Fine-tuning four VLMs on this data yields significant gains on GUI grounding benchmarks (e.g., Qwen3.5-4B improves 11.51 points on ScreenSpot-Pro) and long-horizon task completion.
cua released a new version of its open-source computer-use framework, letting AI click and type on desktops like a human so developers can build agents without reinventing the wheel — directly challenging proprietary offerings like Manus.
ZeroSphere is a product that provides AI with a virtual display on Windows, allowing it to interact with desktop applications while users maintain control over their own desktop.
Claude AI introduces a background computer use feature in Claude Cowork and Claude Code, enabling autonomous desktop task execution while the user focuses on other work.
Sai defeats Opus 5 and GPT-5.6 in the OSWorld 2.0 benchmark at roughly two-thirds the cost, scoring 73%, and demonstrates efficient desktop task execution capabilities.
This paper proposes Evidence-First Reflection (EFR) to improve reflection in desktop GUI agents by decoupling action-induced visual difference extraction from outcome verification, yielding accuracy gains of 7.11% on benchmarks.
The article introduces Windows Harness, an open-source Python tool designed for automating a wide range of Windows tasks, featuring UI automation, screenshots, raw input, and integration with browsers, files, and PowerShell.
pi-computer-use v0.4.3 is released, bringing parity with Codex, adding a ghost cursor, parallel tasks for subagents, and improved batching, resulting in 50% latency improvement and better handling of complex use cases like Microsoft apps.
A record-and-compile tool lets users demonstrate desktop tasks, then an LLM converts the session into structured skill files for agents to replay, aiming to reduce brittleness and lower the barrier to creating reusable agent capabilities.
Vantage is an open-source Windows app that uses an LLM to automate desktop tasks via natural language, enabling users to drive any desktop app with plain English commands.
China released an open-source desktop automation agent that runs 100% locally, capable of controlling desktop apps, files, and browsing without internet.
ByteDance open-sources UI-TARS Desktop, a 100% local desktop automation tool that operates purely on pixels with no API calls, resolving the two major pain points of data privacy and API costs, providing an efficient open-source solution for building private automation workflows.
Microsoft launches Fara-7B, an efficient Computer Use Agent with only 7B parameters, surpassing larger models on web tasks, supporting pure local deployment, and achieving low-cost desktop automation.