Tag
Introduces the Behavioral Consistency Metric (BCM) to quantify behavioral consistency across tasks in language model agents, demonstrating it is distinct from success rate and can be measured via execution traces.
RubricForge induces evaluation rubrics from labeled trajectories to reduce over-crediting in language model agent evaluation, enhancing faithfulness without environment access.
A developer built Battle Agents, a platform for testing AI agents in controlled failure scenarios to inspect decisions, tool calls, and recovery, and is seeking community feedback.
Researchers show agent leaderboards rank task specialization rather than capability, with agent main effects accounting for under 3% of variance across three benchmarks, and propose a Deployment Decision Reliability framework.
This paper applies Generalizability Theory to agent benchmarks, showing leaderboards rank specialization rather than capability, and proposes a framework (DDR) for sizing reliable deployment evaluations.
New research from Microsoft Research measures cross-lingual policy retention of tool-using agents by comparing action traces rather than final answers. After removing five confounds, four frontier models retain 71–73% of their action policy across languages, with code and 2.38M rollouts released.
Introduces A2E, an end-to-end evaluation engine for agent harnesses, using a standardized task protocol and execution traces to assess capabilities like efficiency, tool use, planning, and error recovery.
Factorio Learning Environment v0.3.0 is an open-source platform for evaluating AI agents in Factorio, adding headless scaling, OpenAI Gym compatibility, and Claude Code integration for live demonstrations.
The author comments on FutureX's prediction evaluation framework, pointing out that it only scores final answers and cannot distinguish the quality of probability calibration, and discusses finer-grained evaluation methods such as Brier score and log loss.
GDPevo is a benchmark for evaluating agent self-evolution on real business tasks, covering CRM, ERP, finance, healthcare, legal, and data-centric workflows. The authors release an automated pipeline and find that self-evolution improves held-out accuracy by up to 16.44 percentage points, though agents remain well below an oracle ceiling.
A six-step guide to building evaluation gates that let AI agents merge changes autonomously, covering judge bias, runtime evals, trajectory grading, and more.
Google announces GA of Agent and Model Evaluations in Gemini Enterprise Agent Platform, enabling consistent measurement and monitoring of AI agents in dev and production with pre-built metrics, adaptive rubrics, simulators, and online monitors.
This article explains how Similarweb evaluates long-form agent research reports using LangSmith, combining deterministic checks for tool calls and LLM-as-judge scoring for quality, with a focus on making regressions inspectable and enabling A/B comparisons.
EvoCode-Bench is a multi-turn coding benchmark with 26 tasks across 5 domains, designed to evaluate AI agents on evolving specifications and cumulative testing in a persistent workspace, revealing that single-turn scores dramatically overstate reliability.
SAAG proposes a cascaded diagnostic framework for evaluating LLM agent function calling by decomposing evaluation into registry conformance, structural completeness, and argument grounding stages, enabling interpretable diagnostics and iterative self-repair. Experiments with sub-4B models show improved argument precision and reduced value hallucination compared to single-pass evaluation.
This paper introduces a benchmark for evaluating how LLM agents adapt their tool choice when the reliable tool silently changes mid-session, borrowing set-shifting from cognitive psychology. It tests open-weight LLMs and identifies distinct failure modes based on tool-set framing.
Sayash Kapoor announces his upcoming PhD defense on 'The Missing Science of AI Evaluation,' discussing AI-based science, agent evaluation, and open model risks, livestreamed and open to all.
Google Data Cloud's frontier AI team discusses a new approach to evaluating AI agents using information theory to create a meta-benchmark called Discovery Bench that measures how vague a query can be before an agent fails, providing a more nuanced map of agent capabilities than simple pass/fail exams.
The PACE method proposed by Carnegie Mellon University selects a subset of atomic capabilities from cheap non-agentic benchmarks to predict model performance on expensive agentic benchmarks, with prediction error below 4% and cost reduced to less than 1%.
The ByteDance Seed team released the EdgeBench benchmark, which allows AI models to work continuously for 12-72 hours to evaluate their learning ability in long-horizon tasks. They discovered that the relationship between the time spent learning from the environment and performance follows a log-sigmoid curve, revealing a new scaling law.