evaluation

Researchers introduce "Training Witnesses," a system for generating "honesty" certificates of ML training data and evaluations that conference reviewers can cheaply verify and trust, aiming to fix the broken trust model in ML conferences.

KlinikeBench introduces a clinician-verified benchmark of 333 interactive clinical tasks where LMs must gather patient history, use clinical tools, and reach a diagnosis within a turn budget. Across 31 models, the best (e.g., GPT-6-astra, Claude Opus 5) succeed on under 30% of tasks despite 90.7% diagnostic accuracy, revealing a large gap between diagnostic knowledge and real-world clinical interaction.

The paper introduces ContextAdapt, an evaluation framework testing how well 12 LLMs adapt values like honesty, autonomy, and confidentiality across professional domains (medicine, law, finance, national security). Models achieve high appropriateness scores but fail to apply domain-specific justifications consistently, revealing localized failures when stakes vary.

This NeurIPS 2026 workshop paper introduces Long-Transduction, a controlled diagnostic benchmark measuring how well models sustain stateful, context-dependent operations over long generations. Evaluating seven open-weight models, it finds severe degradation when scaling context length (up to 128K), varying input format, and increasing local task complexity.

This arXiv paper studies how language models decide whether two scientific findings are comparable before resolving contradictions. In a controlled task with unsatisfiable XOR constraints translated into lab reports, GPT-5.6 Sol and Claude Opo 5 recover the best-supported assignment well when constraints are explicit, but models often defer to biological expectations when findings are presented in scientific prose.

Veris AI introduces VAmoS Energy, a harder voice-agent benchmark combining multi-request utility billing calls, stateful Stripe/Fineract twins, caller verification gates, and background speech — finding completion rates of 17.3-44.7% across 14 voice stacks, with Grok Voice leading and background TV slashing pooled completion from 38.7% to 8.6%.

The paper introduces CHOIR (Collective Hierarchically-Ordered Inquiry Responses), a framework adapting free-list elicitation from cognitive anthropology to probe and measure hidden diversity beneath surface agreement in LLM ensembles, finding that base-model identity is the strongest recoverable signature of distinct model voices.

SimuVerity introduces a benchmark of 101 text-to-executable Simulink model generation tasks across ten engineering domains, using a hierarchical evaluator to assess engineering qualification beyond structural similarity. The best evaluated agent system scores only 42.86, revealing major capability bottlenecks in engineering-grade model generation.

The paper introduces Ego2Act, a goal-directed benchmark of 2,640 egocentric videos across 110 real-world tasks for evaluating whether video generation models can simulate multi-step physical manipulation, along with Ego2Act Judge, a reference-free evaluation pipeline that aligns better with human judgments of task completion and physics plausibility.

VTR-Bench introduces a systematic benchmark for evaluating visual text rendering in video generation models, covering 300 prompts across five scenario categories, an automated evaluation pipeline with human alignment, and a Keyframe-Guided Agentic Framework. Experiments on 11 state-of-the-art models reveal widespread difficulties in rendering scene text, with the best model achieving a word error rate of 0.250.

Argo-Bench introduces an evaluation framework of 210 enterprise-scale data science tasks built on a simulated food-delivery platform with an ERP warehouse of 235 tables and 7.5 billion rows, testing agents that must navigate and act rather than just generate SQL. The strongest of 14 frontier and open-weight models averages only 59.5 points, highlighting a significant gap in real-world data agent capabilities.

PROWBench introduces 170 programmatically constructed episodes and 600 proxy videos to evaluate whether generative video models faithfully render program-specified world dynamics, using VLM-based metrics for logic-render alignment and interaction success.

This paper introduces 'Sharpening Tax,' a metric quantifying how RL post-training improves pass@1 but reduces solution coverage (pass@K) in LLMs, and proposes posterior-tempered group sampling (PTGS), a plug-and-play Bayesian sampler that adapts temperature per prompt to pay a smaller tax while improving single-shot accuracy.

该论文提出"模式跳变"(mode-hopping)概念，指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变（如OLMo3-32B在算术提示上81%→0%→81.7%的反复），并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。

A practical walkthrough on using real Reddit threads to build a small test set for checking whether an LLM preserves the hidden constraints (privacy, offline-only, installation limits) in real-world questions. The author proposes an answer-key workflow that counts constraint violations as failures and separates persuasive-sounding prose from correctness.

An O'Reilly article outlining 'The Agentic Data Science Playbook' — how AI-native data scientists delegate analysis to AI agents while designing workflows, verification, and feedback loops. Using a modified Elliptic Bitcoin fraud dataset, Claude Opus 5.0 initially reported inflated results (F1 0.87) due to a temporal-split error and a planted leaking feature; with proper guidance, corrected performance dropped to F1 0.70 and 0.21 recall on high-degree nodes, showing why expert oversight still matters.

GPT-6.1 Sol has reached third place on the Humanity's Last Exam (Diamond) leaderboard, marking a significant milestone in frontier AI benchmark performance.

LangChain introduces an Engine that learns how your agent works by reading traces and repositories, then tests for agent-specific weaknesses such as hallucinations and system prompt violations, returning a list of verified issues to fix before they impact users.

A reporter stress-tested GSA's new America.gov AI assistant with 15 research-grade questions from prior reporting, finding 9 correct answers, 6 incomplete, and no hallucinations, though the system missed available public data and lacked transparency about its sources. The finding suggests the tool refuses to fabricate when it lacks information but cannot establish what the government has not published.

PADMÉ reformulates meta-evaluation of LM agent evaluators as a preference alignment problem and introduces a data synthesis method using only small language models, boosting agreement with human judgment from 73% to 85% over a naive baseline while enabling evaluation of 25 common models across agentic domains.