Tag
A practitioner discusses how to properly test AI agents that edit records when names collide, proposing state-based checks over tool-call success and asking whether to evaluate at the tool-selection level, the database level, or both, plus how to score correct clarification requests without encouraging unnecessary questions.
An analysis reveals that 28.9% of GPT 5.5's failures on SWEBench Pro are due to broken or incorrect test cases, and similar issues affect other major AI benchmarks, raising concerns about the accuracy of current evaluation methods.