A tool that enables coding agents to interact with real browsers by fixing click registration issues through hit-testing and shadow root traversal, while adding hover and drag capabilities and ensuring safety by blocking consequential actions.
Disclosure: I built this. It is free and MIT. Links in a comment below, per rule 3. Most agent setups that touch a browser drive a headless clone that is logged into nothing, so they hit a sign-in wall on step one. The ones that do drive your real browser had a subtler failure that cost me a lot of time: the agent reports "clicked Save", nothing actually happens, and you burn five turns debugging a button that was never pressed. Nine times out of ten a cookie banner or a modal backdrop was painted over the click point and swallowed it, and the tool still returned success. The fix was to hit-test before clicking instead of resolving a bounding box and firing at its centre. It descends through open and closed shadow roots to find what is genuinely painted at that pixel. If the target is covered it says so, scrolls clear of pinned bars, or gives the covering layer pointer-events none for exactly one click and restores the inline styles after. Same real CDP click, so isTrusted stays true. Two things mattered more than I expected. Hover and drag as first-class operations. A click-only tool surface cannot reach hover-only menus, sliders, canvas apps or drag-to-reorder lists at all, and no amount of retrying a click will get you there. It stops before consequential actions. Posts, payments, passwords, 2FA. It fills in everything around them, highlights the button and hands control back to you. An agent holding your session cookies should not be able to publish in your name. Works with Claude Code, Gemini CLI and Codex. No account, no telemetry, the server only talks to your own browser. Honest caveat that someone already caught: the overlay piercing classifies layers by geometry and style rather than semantics, so it does not yet tell a consent dialog apart from a sticky nav. Fix is queued. Happy to go deep on the CDP side if anyone here is building something similar.
An article sharing hard-won lessons from building a web agent that can interact with real web applications, offering insights into challenges and best practices.
This article explains the key difference between computer-use agents that operate on full desktop interfaces using pixel screenshots and browser-use agents that can leverage the DOM's hidden structure, making the former a harder technical problem.