evaluation

A Google and University of Hong Kong paper proposes VeriCUA, which verifies computer-use agent tasks by checking saved files and app state instead of trusting the agent's own completion report, cutting false positives to 5.4% versus 24.8% for a VLM judge.

HyperHat's founder shares a design sketch for benchmarking whether AI agents can identify contradictory task requirements and ask clarifying questions rather than blindly executing, pairing broken tasks with satisfiable controls to prevent trivial gaming of the eval.

OpenAI announced a new internal benchmark evaluating how well its AI handles everyday human tasks like shopping and meal planning, reporting that GPT-6 Astra scored 93.1% on finding a kindergarten after a move and 90.4% on creating a diabetes-friendly meal plan.

OpenAI announced LifeVal, a new benchmark built from feedback by thousands of ChatGPT users on everyday tasks they most want help with, and has used it over the past year to measure and improve model performance on practical real-world workflows.

A 216-run benchmark compares Claude Code (Opus 5.5) and Codex (GPT 6 Astra) across agent-browser, Playwright, and Chrome DevTools via CLI, skill, and MCP on a browser task; all runs passed, tool choice depended on the agent, and a generated Python script vastly outperformed every agent run in speed and cost.

OpenAI announced LifeVal, an evaluation built to measure how well its models perform on practical everyday tasks—such as childcare, budgeting, shopping, and government services—derived from time-use studies and a new 3,995-respondent survey, and used over the past year to drive model improvements.

A new frontier-level, long-horizon benchmark called GDPx, the successor to GDPval, has been released; even the best models only score around 12%, highlighting a large gap in agentic task performance.

The paper introduces the Collaborative Intelligence Benchmark, which evaluates AI agents as teammates across 240 tasks in 8 teamwork dimensions, finding that even top models score at most 82.2/100 and that general capability does not guarantee strong teamwork.

Introduces a 400-question benchmark evaluating when commercial LLMs choose to search the web versus answering from internal memory, finding that models like GPT-5 mini and Grok 4 Fast search appropriately on outside-knowledge questions while Mistral Small often abstains instead, with search behavior heavily influenced by explicit temporal cues.

The paper proposes PACE (Proxy Alignment and Curvature Expansion), a pre-optimization diagnostic that uses repeated human ratings to predict the human-gain curve induced by a fixed proxy, modeling proxy optimization as a KL-regularized exponential tilt with first-order alignment and second-order correction terms, and shows reduced held-out gain-prediction error across multiple benchmarks.

The paper introduces ProClaim, a scientific agent that persists a structured evidence state outside the model context, and ProClaim-eval, a 718-claim consensus verification benchmark, showing that both the evidence state and a trained stopping judge improve consensus claim verification over baselines and general-purpose agent harnesses.

The paper introduces an incongruence-aware evaluation framework measuring disagreement between textual and acoustic emotional cues in audio LLMs, finding that cross-modal incongruence consistently increases emotion recognition errors across three speech corpora while supervised classifiers show the opposite trend.

This paper introduces Aligned RAG Evaluation, a matched-query protocol showing that attack success rate (ASR) hides defense-induced failures in RAG knowledge-poisoning defenses, distinguishing prevented attacks from newly introduced target-positive answers on HotpotQA and Natural Questions.

The paper introduces PersonaChatBench, a framework that evaluates 20 language models across six domains by pairing fixed tasks with diverse simulated users guided by private persona cards, measuring task outcomes, persona need fulfillment, and simulator fidelity. Results show that strong task performance does not consistently translate into satisfying individual user needs, with model strengths varying across personas and domains.

该论文提出 Grounded Generative Speech Recognition (GSR) 新范式，将语音识别重新定义为生成语义有据的文本而非单一转录，并发布 GSR-Bench 基准与多维评估框架，揭示了转录保真度与生成重构能力之间的权衡及常见失败模式。

The paper introduces Bilal-Bench, a 240-question expert-verified Bengali benchmark for open-ended Islamic-source fabrication, evaluating seven open-weight models (1B–120B) and finding that six of seven confidently fabricate on 79.3–95.3% of source-production questions with no monotonic improvement from scale.

StrictPRMBench introduces a replayable benchmark for auditing how fixed-score PRM aggregation rules (product vs. geometric mean) affect best-of-N selection, showing product yields higher deletion-induced selection instability across 14 cells with reproducible reanalysis artifacts.

SemiAnalysis is releasing GDPx, a frontier-level benchmark for measuring agentic capabilities across the most economically valuable domains, where even the best current model scores only 12% (1/7).

Moralitybench.ai has been updated with results from 20 AI models answering 56 standardized moral psychology questions five times each, including new Trolley Problem variants, plus two DIT-2 tests pending.

The Continual Learning Exam (CLE) introduces 120 expert-written professional tasks organized into 5-level ladders to test whether LLM agents can reuse knowledge gained earlier; even the best setup only clears 47.7% of levels.