What surprised us after building an autonomous quality engineering agent system

Over the past year we've been building OttoTester, an autonomous quality engineering system built around specialized AI agents rather than a single "do everything" model. Like a lot of teams, we started with the assumption that the execution agent would be the centerpiece. It wasn't. The biggest challenge wasn't generating tests or executing them—it was building a system that could continuously improve without requiring a human to constantly step in. That led us toward a set of specialized agents that: Plan testing strategy and coverage Generate tests Execute across web applications Heal tests when applications change Analyze patterns across executions to improve future runs Audit the performance of the other agents to identify weaknesses and improve the overall system The Analyzer and Auditor ended up becoming far more valuable than we originally expected. Without learning and governance, autonomy eventually plateaus. Another lesson: orchestration matters more than individual model quality. Swapping models produced incremental gains, but improving how the agents shared context, learned from previous executions, and coordinated their work produced much larger improvements. We're now looking to work with 5–10 enterprise engineering organizations as design partners to challenge the architecture before broader rollout. Not looking for transactional beta users—we're looking for teams with complex applications, real CI/CD pipelines, governance requirements, and opinions about where autonomous systems fail. I'm genuinely interested in hearing from this community: If you were evaluating an autonomous QA agent system for production use, what capability would be your hardest requirement before you'd trust it? Reliability? Auditability? Learning? Governance? Integration into existing engineering workflows? Something else? I'd love to hear where you think this approach succeeds—or where you think it breaks.