When your coding agent says "tests passed," what do you actually check before accepting the work?

Reddit r/AI_Agents News

Summary

A developer building an independent verification prototype for coding agents shares two reproduced cases where pytest reported passing despite failing tests (exit status 0, config-based deselection) and asks the community how they verify agent-generated changes before accepting them.

I've been building an independent verification prototype for coding agents, and a recent external evaluation exposed something that made me reconsider what a successful test run actually establishes. In one controlled reproduction, pytest reported failing tests but exited with status 0. In another, a configuration change quietly deselected failing tests, producing a green result. I independently reproduced both. These were deliberately constructed stress tests, not evidence that agents routinely behave this way. But they raised a question I've become increasingly interested in: What actually happens between an agent reporting completion and a developer trusting that completion enough to accept the work? For those using coding agents on real development tasks: Think about the last agent-generated change you accepted. What did you personally verify before accepting it? Did you inspect the diff, rerun tests, review test changes, wait for CI, or something else? Have you encountered a false green in practice? For example, an agent modifying tests, fixtures, configuration, or test selection in a way that made failing work appear successful. Or is this mostly a theoretical concern in your workflow? How do you handle legitimate test changes? If the agent adds or modifies tests as part of the task, who determines whether those changes preserve the original acceptance criteria? If you already have CI, what manual verification still remains? And what evidence would you need before feeling comfortable reducing that step? I'm trying to understand whether independent verification addresses a recurring developer bottleneck or simply adds another layer on top of systems that already work well enough. I'm especially interested in concrete experiences, including cases where your current workflow is already sufficient.
Original Article

Similar Articles

A coding agent's "tests passed" message is still just a message

Reddit r/AI_Agents

Microsoft's Agensh repo provides a shared notes board for coding agents to coordinate, but it only stores reported verification rather than running checks itself, so a 'tests passed' claim is not proof. The post highlights the difference between coordinating work and verifying work in multi-agent workflows.

How do you verify your agent actually did what it says it did?

Reddit r/AI_Agents

A practitioner raises a common production problem: AI agents falsely reporting tasks as complete when the underlying action failed, and asks how others verify agent outputs in real deployments. The author is researching the issue and plans to share a summary of community answers.

Five failure cases I’d test before trusting a multi-agent coding workflow

Reddit r/AI_Agents

A design checklist of five failure cases to test before trusting multi-agent coding workflows: post-review requirement changes, stale worker claims, lost external write responses, approval scope drift, and restart without conversation context. The author frames these as tests to distinguish 'a run ended' from 'this result was accepted against the current task' while building Backplane.