A self-improving GUI agent that learns from its mistakes, using a perception stack with YOLO and OCR, calling LLM only when needed, and running locally. It's open source under the MIT license.
So I've been working on this project for the past couple months and wanted to share. The idea is pretty simple: an agent that watches your screen, figures out what's on it, and clicks/keys things to get stuff done. Like a digital intern that never sleeps but also doesn't have a clue until you teach it. What makes it interesting: instead of throwing a $200/month vision model at every UI interaction, I built a perception stack that runs YOLO for element detection + OCR for text + only calls the LLM when it actually needs to make a decision. Most clicks never touch the expensive model. Saved me a ton of API costs. But the part I'm most excited about is the self-improvement loop. After each task, the agent reviews what happened and extracts reusable skills into persistent memory. So next time it needs to do something similar, it doesn't start from scratch — it already knows the pattern. Gets genuinely better the more you use it. Right now it controls any Windows desktop app — games, music players, productivity tools. You register an app, describe what you want in plain language, and it figures out the rest. There's a web dashboard where you can watch every step in real-time: screenshots, detected elements, what it's thinking, what action it takes. Zero black box. I'll be honest — it's not perfect. Sometimes it misreads UI elements and clicks the wrong thing. The YOLO model needs fine-tuning for specific apps. And the self-improvement feature is still early — it's smart about extracting patterns but occasionally learns the wrong lesson (one time it decided that closing a dialog meant clicking the title bar × close button, which worked but was hilariously inefficient). But the v0.4.0 I just shipped has DirectML GPU acceleration, so YOLO detection went from "wait for it..." to basically instant. That was a game changer for the overall feel. Tech stack: FastAPI backend, Vue 3 dashboard, Hermes Agent (NousResearch) for the agent loop, OmniParser YOLO weights, RapidOCR. MIT license. Runs entirely locally — your screenshots never leave your machine except for the AI API call. Curious if anyone else is working on the local-first GUI agent approach. What's your perception stack look like? Also happy to take brutal feedback — the only way this gets better is if people tell me what's broken.
GUICrafter introduces a weakly-supervised GUI agent that leverages massive unannotated screenshots and a two-stage curriculum learning framework to reduce reliance on expensive human annotations, achieving competitive performance with advanced systems like UI-TARS using only 0.1% of its data.
A practical guide explaining three levels of building self-improving AI agents, from manual loops to automated design, with recommended tools and frameworks.
A self-optimizing agentic pipeline that improves benchmark performance from ~30% to ~90% on TerminalBench, and can be extended to everyday chats by logging interactions, reflecting with a local model, and injecting lessons into future system prompts.
ClawGUI is an open-source framework for training, evaluating, and deploying GUI agents using reinforcement learning, featuring standardized benchmarks and cross-platform deployment to Android, iOS, and HarmonyOS.
A developer built a local web GUI for the TradingAgents multi-agent LLM stock analysis framework, supporting various LLM providers and adding features like live pipeline visualization, a report reader, and multi-session chat.