@quanruzhuoxiu: When using Midscene's Computer Agent, desktop automation runs headless in Linux CI. Everyone assumes desktop UI automation must use a real machine or VM, so Mac/Windows desktop E2E can only run locally and cannot enter CI. Result...

X AI KOLs Timeline Tools
midscene ui-automation headless-testing linux-ci desktop-automation xvfb e2e-testing

Summary

Midscene's Computer Agent enables desktop UI automation to run headless in Linux CI, automated via xvfb-run, without needing a real machine or VM, and supports Electron, Qt, and GTK applications.

Using Midscene's Computer Agent, desktop automation runs headless in Linux CI. Everyone assumes that desktop UI automation requires a real machine or a VM, so Mac/Windows desktop E2E usually can only run locally and cannot enter CI. The result is that regression testing for desktop applications always relies on manual testing. The solution in the documentation is as simple as one line of apt-get install + one API flag—Midscene will automatically wrap your command with xvfb-run, so you don't have to start Xvfb manually. GitHub Actions / GitLab Runner can run it directly, and Electron, Qt, and native GTK applications can all be recognized because we use pure vision without relying on any accessibility API. Midscene Studio's own complete workflow running in CI is publicly available in the repository; you can copy it into your project and modify a few lines to use it: http://github.com/web-infra-dev/midscene/blob/main/.github/workflows/studio-headless-linux.yml… The attached image shows the latest successful run of this workflow, run #25790418497.
Original Article
View Cached Full Text

Cached at: 05/25/26, 02:41 AM

Midscene.js

AI-powered, vision-driven UI automation for every platform.

Similar Articles

@quanruzhuoxiu: Often asked: What's the difference between Midscene and Browser-Use? Both are open-source, both use vision, both solve their respective problems. Here's an honest comparison, not to bash Browser-Use. Browser-Use is a web agent, positioned as "open the browser, get this done…"

X AI KOLs Timeline

A comparison of Midscene and Browser-Use, two open-source tools with different focuses: Browser-Use is a web agent for one-time tasks, while Midscene is a vision SDK designed for reliable multi-platform repeated execution.

@canghe: https://x.com/canghe/status/2061431572306518501

X AI KOLs Timeline

WeSight is now open source. It is a desktop AI agent console that unifies management and scheduling of multiple agent engines like Claude Code and Codex, offering a visual workspace, team collaboration, and Feishu integration.

@quanruzhuoxiu: Over the two years of developing Midscene.js, we made a belated but critical decision: UI automation will sooner or later shift from 'understanding the DOM' to 'looking at the screen'. So in the December 1.0 release, we directly cut the DOM compatibility path. In the early days, like everyone else, we followed a DOM + visual hybrid approach...

X AI KOLs Timeline

The Midscene.js team decided to completely shift from a DOM + visual hybrid approach to pure visual UI automation, believing that future UI automation must be based on screenshots rather than the DOM. This change reduced token consumption and simplified cross-platform adaptation.

@VincentLogic: Found an incredible open-source desktop AI tool from ByteDance! UI-TARS Desktop, with 31k stars, truly lives up to the hype. It can actually understand your screen and automate computer operations for you. Just tell it "Enable auto-save in VS Code and set the delay to 500ms", and it will automatically: -…

X AI KOLs Timeline

ByteDance's open-source desktop AI automation tool, UI-TARS Desktop, supports local execution and screen visual understanding. It can autonomously control your computer to handle daily tasks through natural language commands.