caught a vendor contradicting themselves across two sections of the same proposal, only because our agent doesn't evaluate section by section

Reddit r/AI_Agents Tools

Summary

An AI agent for supplier evaluation in nvelop identifies internal contradictions in proposals through full document cross-referencing, enhancing accuracy compared to isolated section scoring.

related to the flagging post above, a different case: our ai agent for supplier evaluation in nvelop doesn't just score each section of a proposal in isolation, it cross-references the whole document. real example: a vendor stated a 90-day payment schedule in one section, then a different part of the same proposal said 45 days. scoring each section separately, both would've looked fine individually. cross-referencing the full proposal is what caught the contradiction, and that vendor's score got dinged for it. it's a small thing but it's turned out to be one of the more useful checks, most scoring tools we've seen (including earlier versions of ours) score section by section and miss this kind of internal inconsistency completely. anyone else building evaluation tooling running into the same silo problem?
Original Article

Similar Articles

A right answer from your agent doesn't mean it did the right thing

Reddit r/AI_Agents

The article discusses the pitfalls of evaluating AI agents solely based on their final answers, emphasizing the importance of inspecting intermediate steps, tool calls, and reasoning to catch confidently wrong outputs. It suggests using automated scoring and trace replays to measure and improve agent behavior.