The author questions traditional mobile agent approaches relying on API access and proposes using screen understanding via hardware like aiden-firmware to enable more general-purpose agents that interact like humans.
Okay, this might be a dumb take, but the more I look into mobile agents, the less convinced I am that giving AI deeper and deeper access to the operating system is the end goal. Most automation today works because we explicitly tell the system what it can do: Call this API. Run this function. Send this ADB command. Find this accessibility element. And yeah, that works great. Until the app gets redesigned, the API changes, or the phone OS gets updated. Then suddenly a bunch of your automation stops working, because the whole thing was built around one assumption: the software's UI and underlying logic won't change. But that's not how humans use phones. When I'm using my phone, I have no idea what APIs Instagram exposes, and honestly, I don't care. I just look at the screen, recognize the buttons, understand what's going on, and tap where I need to tap. So why couldn't a general-purpose agent work the same way? I recently came across something on GitHub called aiden-firmware, and I thought the approach was pretty interesting. Instead of trying to give AI deeper and deeper software-level access, it uses hardware to capture what's actually being shown on the screen, lets the model understand what it's seeing, and then sends actions back to the device through USB HID. Basically: See the screen → understand what's happening → decide what to do → interact with it like a human would. No need to build a separate integration for every app. No ADB. No root access. Obviously, this approach isn't perfect. It's probably slower than directly calling an API, and visual reasoning can still make mistakes. And if you're just doing repetitive tasks and there's already a stable API available, then yeah, this could be massive overengineering. But what if the goal is a truly general-purpose agent? Something that doesn't need someone to build an integration for every single app beforehand. Something you could put in front of a device it's never seen before, and it could observe the screen and figure out how to use it. I'm starting to think that understanding the screen might matter more than having deeper system-level access. Maybe I'm missing something obvious here, but I'd genuinely like to know what people think.
The author argues that hardware-level interaction (HDMI capture + USB HID touch simulation) is superior to ADB-based automation for building OS-agnostic general-purpose AI agents, and introduces their project Aiden using a cheap RV1106 dev board.
The author discusses how AI agents may serve as a new abstraction layer over existing software, shifting user interaction from navigating UIs to describing outcomes, reducing friction in converting intent into executable tasks.
The author observes that AI agents are becoming the primary users of business software, shifting focus from human UIs to agent-accessible APIs and tool schemas.
An opinion piece arguing that AI agents are not killing UI but shifting products toward hybrid interfaces, requiring agent-friendly onboarding alongside human-facing controls for approval, review, and orchestration.
OpenGUI is a tool that allows AI agents to directly operate real Android apps by reading the screen and interacting naturally, rather than relying on APIs or scripts.