Tag
This paper presents SCALE, a sequential cost-aware policy for hypothesis testing that uses AI judgments with selective human verification to minimize costs while controlling error rates, applicable in settings like software reliability assessment.
The article explains the differences between Jev and LLM as Judge for evaluating AI agent responses, highlighting when to use each based on the need for open-ended reasoning versus structured, parallel judgments.
This study evaluates Jev, a System One model, for measuring factual differences in AI-generated radiology reports, demonstrating its effectiveness and efficiency compared to other evaluation methods.
An updated version of the Humanity's Last Exam, a benchmark for AI evaluation, named HLE-Diamond, has been announced and released.
OpenAI introduces MentalHealthBench, an open benchmark for evaluating AI responses in mental health conversations, co-created with over 80 mental health experts to measure safety, context, agency, and guidance.
本文总结了来自 Hamel Husain 和 Shreya Shankar 的 AI Evals 课程的 8 个核心技能，旨在指导工程师和产品经理构建有效的 AI 评测系统，涵盖错误分析、评测器设计、校准和监控等步骤。
The paper introduces FrontierMath Erdős, a benchmark of 68 open Erdős problems for evaluating AI models in mathematics using Lean proof assistant, aiming to address shortcomings in current AI research demonstrations.
I'm impressed by your approach—using personal git history to create a tailored benchmark for evaluating local AI models on code tasks is both practical and insightful. It's particularly interesting to hear about early results from models like Flash Next and Swift performing unexpectedly well. I'd be curious to learn more about what specific aspects surprised you in their performance.
Glasshouse v0.1 is a memory benchmark tool designed to address issues with existing AI memory benchmarks by providing multi-language support, varied conversation lengths, and detailed evaluation across multiple axes.
CleanScore is a black-box method for auditing public benchmark scores by comparing performance on original questions with independently written forms, using negative controls to separate skill from exposure effects.
This paper introduces OpTFM, a comparative evaluation framework for tabular foundation models in healthcare, assessing models across six clinically meaningful dimensions like generalization and fairness, and applies it to two use cases to demonstrate context-dependent rankings.
The Situated Identity Test (SIT) is an architecture-independent framework introduced to evaluate whether an AI agent's behavior is functionally attributable to a specific developmental lineage, addressing the problem of persona imitation in large language models.
UK AISI and EvalEval are collaborating to openly share AI evaluation results using a standardized schema and platform, enhancing reproducibility and transparency in benchmarking for AI models.
This article presents a definitive benchmark designed to evaluate and compare AI models or systems, establishing a standard for future assessments.
The Spider Bench benchmark compares nine vision models on a dataset of 2,000 spider photos, with Gemini 3.8 Flash achieving the highest exact-species accuracy of 49.85%. The study provides public code, tasks, predictions, and results for reproducibility.
Beacon is an open-source memory layer for AI coding agents that uses Jev to evaluate agent runs and turn useful workflows, corrections, and debugging patterns into reusable skills across multiple harnesses.
PolyBridgeBench is a new executable benchmark for evaluating multimodal LLMs on physics-grounded bridge design tasks, revealing gaps between deterministic validity and dynamic success in structure synthesis and repair.
CogGym is a scalable framework for comparing human and AI cognition using cognitive experiments, revealing that larger language models better mimic human reasoning but still lag behind formal benchmarks.
This paper introduces a productivity-oriented framework for evaluating human-AI collaboration based on outcome quality relative to interaction cost, showing that identical quality ratings can differ significantly in interaction costs and that subjective user ratings are not reliable for measuring productivity.
The article discusses how using string-matching for evaluating AI coding agents can lead to false compliance and inaccurate assessments, as it only verifies the presence of specific strings without proving actual behavior.