Tag
The paper introduces MedBenchAgent, a multi-agent framework that automates the full specification and construction of medical vision-language model benchmarks via a 'constrained compilation' approach, achieving a Task-Space F1 of 90.9% and high human audit pass rates across twelve evaluated VLMs.
ToolGate is an executable pipeline for constructing scientific benchmarks that validates generated tasks through executable scripts, random no-tool screening, and tool-using agents, reducing manual labor and enhancing audibility.
This paper proposes Embodied-BenchClaw, an autonomous multi-agent system that automatically constructs embodied spatial intelligence benchmarks from user intent through a five-stage pipeline with process quality control and an extensible Skill Library.
Introduces Benchmark Agent, a fully autonomous system for creating diverse benchmarks with minimal human intervention, enabling continuous model assessment across domains.
This paper presents QuestBench, a benchmark built by students to evaluate deep research systems across humanities and social science domains. Results show that even advanced systems like GPT-5.5 pass only 57.58% of questions, highlighting failures in trustworthiness.