How do you test that an agent updates the right record when names collide?

Reddit r/AI_Agents News

Summary

A practitioner discusses how to properly test AI agents that edit records when names collide, proposing state-based checks over tool-call success and asking whether to evaluate at the tool-selection level, the database level, or both, plus how to score correct clarification requests without encouraging unnecessary questions.

Here's a small test case I'd want in an agent evaluation. All names and records are synthetic. The user asks: 'Add this meeting note to Jordan Lee at Acme.' Search returns two Jordan Lees, one at Acme and one at Beta, with the Beta record first. The write tool accepts either ID and returns success. Counting a successful tool call would miss an update to the wrong contact. I'd check the final state instead: the Acme record has the note, the Beta record is untouched, and the response identifies which contact was updated. Then I'd run a second case with two Jordan Lees at Acme and no other identifying detail; that one should ask for clarification before writing. For people evaluating agents that edit records, do you test this at the tool-selection level, by checking database changes, or both? I'm especially interested in how you score a correct clarification request without rewarding an agent that asks unnecessary questions on every unambiguous task.
Original Article

Similar Articles

What should an agent do when two tools disagree?

Reddit r/AI_Agents

The article discusses how AI agents should handle conflicting tool outputs by preserving both values with metadata and escalating based on thresholds, questioning whether this policy belongs in agent prompts or deterministic code.

When should an AI agent question its own data?

Reddit r/AI_Agents

The article discusses how AI voice agents should handle situations where customers disagree with backend system data, exploring design options like additional verification or human handoff.