Ran Online-Mind2Web with a local 35B browser agent. Same model, three harness fixes, and a wrong answer became the right one

Reddit r/AI_Agents News

Summary

作者用单张 3090 上运行的本地 35B 浏览器 agent(Ornith 1.5)在 Online-Mind2Web 上跑了部分任务,通过三轮 harness/清单提示改进将一个错误任务修好,并报告了 30 个新任务的结果:研究类任务表现尚可,UI 操作(下拉框、cookie 遮罩、位置自动补全)是主要失分点。

I'm building a browser agent that runs on a local model (Ornith 1.5 35B, one 3090). I wanted a test someone else wrote instead of tasks we picked ourselves, so I ran a few from Online-Mind2Web (300 tasks on 136 live sites). First 3 tasks: 2 right, 1 wrong. The miss was the TED one: "most viewed TED talk on robots that lasts between 12 and 18 minutes." I ran that task three times and never changed the model. Run 1. It typed "robots" into the search box and opened results one by one. The right answer is a quadcopter talk with no "robot" in the title, so search never showed it. Run 2. I added a checklist item: "did you use the site's own sort/filter?" It got to the topic page, but sorted by newest and skipped the duration filter. Our grading still passed it, because any filter counted. Run 3. I tightened the item to "applied everything the request needs", and moved the hint from the end of the run to the first search result. This time it set most-viewed sort plus the 12-18 minute filter and answered #1. It even added "strictly speaking this is a drone talk, not a robot one." What I took from it: when the easy path looks like it's working, the model doesn't leave it. If search results look plausible, it never opens the filter page. A "use filters first" line in the prompt didn't fix it. A checklist item plus a nudge in the tool result did. Then I ran 30 fresh tasks. - First 15: by mistake I didn't give the start site. Excluding 5 that hit bot checks, 6 of 10. - Next 15: start site given, like the official setup. Excluding 1 bot check, 5 of 14. Of the 9 misses, 6 were UI manipulation: location autocomplete, select boxes, a cookie overlay, a search box. So research-style tasks mostly work, and manipulation is where we lose. Notes: this is our own grading, not WebJudge. Bot checks go to a human by our rule, so I left them out of the denominator. Small n, so no big claims. Questions: Location autocomplete, select boxes, cookie overlays. How does your agent handle these? That's where we lose the most right now. For "cheapest" or "most viewed" tasks, how do you get your agent to use the site's filters instead of search? Did prompting work for you? How do you count sites that throw bot checks in a benchmark? Excluded, or failures?
Original Article

Similar Articles

Same model, same prompt, two agent harnesses: 45/50 vs 43/50

Reddit r/AI_Agents

The article benchmarks two open-source coding agents on the same deepseek-v4-flash model, finding similar task success rates but significant differences in performance metrics and a critical bug in one agent's error handling.

@SunNeverSetsX: Sharing my curated 4 infrastructures that make Agent Loop run faster and better—you must let your agents use them! 1. http://agent-browser.dev Let agents click and verify UI changes in a real browser for true self-closing test loops...

X AI KOLs Timeline

agent-browser is a CLI tool for browser automation designed for AI agents, using compact text output and ref-based element selection to minimize token usage. The post also highlights three other tools—portless, emulate, and ai-cli—for improving agent loop efficiency.

What actually breaks agents on a 4B model? Notes from 50 runs on Qwen3.5-4B

Reddit r/LocalLLaMA

A practitioner ran 10 agent harnesses on Qwen3.5-4B across 5 tasks and found that failures came from exact-match edits, literal instruction-following, miscounting, and API mismatches — not malformed tool calls. Six of ten agents passed all tasks, with an 11x spread in run time on identical hardware.