Tag
This paper presents efficient methods for benchmarking production LLM agents, demonstrating that adaptive testing based on item response theory can reduce evaluation costs while maintaining accuracy, with practical deployment insights for recurring assessments.
The paper presents CART, a closed-loop adaptive red teaming framework for large language models that dynamically adjusts test cases based on failures, outperforming static methods in discovering safety weaknesses.
IQ Tester is an AI agent that estimates cognitive profiles with 30 adaptive questions, delivering detailed assessment reports based on psychometric frameworks across multiple reasoning domains.
This paper explores efficient recurring evaluation methods for production LLM agents, comparing techniques like adaptive testing and fixed subsets, and provides practical recommendations for deployment.
This paper introduces a statistical framework for adaptively auditing AI systems using Safe Anytime-Valid Inference (SAVI) to draw rigorous conclusions with limited data. It proposes a 'testing by betting' approach to validate model robustness while controlling type-I errors during adaptive sampling.