Tag
A tweet thread discussing the iterative process of building evaluations and environments for AI agents, emphasizing human-agent collaboration and the importance of data and verifier design.
Discusses that production agent evaluations should include failure replay and resume capabilities, not just happy-path task success, emphasizing the need for observability that enables recovery.
A guide on running agent evaluations using Harbor framework and LangSmith sandboxes with full trace support.
A Twitter thread discussing two distinct evaluation suites needed for general AI agents: a lightweight benchmark eval for quick iteration and a comprehensive test coverage eval for thorough validation across diverse user paths.
The article discusses the challenges of debugging and evaluating LLM judges using Arize Phoenix, which traces evaluator runs via OpenTelemetry to inspect decision logic, costs, and potential biases.