benchmark

A new frontier-level, long-horizon benchmark called GDPx, the successor to GDPval, has been released; even the best models only score around 12%, highlighting a large gap in agentic task performance.

The paper introduces the Collaborative Intelligence Benchmark, which evaluates AI agents as teammates across 240 tasks in 8 teamwork dimensions, finding that even top models score at most 82.2/100 and that general capability does not guarantee strong teamwork.

The paper introduces SERIAL-7, a Korean–English benchmark with an executable verification framework that evaluates LLMs on deriving, realizing, and preserving machine-verifiable outputs across seven representations and 43 atomic constraints, revealing significant gaps in structured generation reliability across ten models.

FrontierMat introduces a benchmark of 330 validated frontier materials R&D problems for language models, finding the strongest model scores only 53.7% and that a domain-adaptation method lifts an open-weight model to 55.4%, matching proprietary leaders.

Introduces a 400-question benchmark evaluating when commercial LLMs choose to search the web versus answering from internal memory, finding that models like GPT-5 mini and Grok 4 Fast search appropriately on outside-knowledge questions while Mistral Small often abstains instead, with search behavior heavily influenced by explicit temporal cues.

The paper introduces BABA-Bench, a benchmark of 47 interactive patient-advocacy worlds that checks each agent action in real time for permissions, evidence, and prohibited steps. Evaluating 14 frontier models across 1,974 runs, it finds agents often reach correct outcomes via invalid routes—e.g., Claude Opus 5 hits all outcomes in 81.6% of runs but passes all procedural checks only 52.5%—and that supplying a verified route boosts pass rates to over 91%.

ReplayRCA is a benchmark that scores post-hoc diagnosis and live oversight of language-model agent failures by counterfactual replay, measuring causal repair rather than annotation agreement. On 62 systematic failures, Grok 4.7 leads at +51.3 points, and results show post-hoc diagnostic skill is a poor proxy for live oversight performance.

The paper introduces P-ContextBench, a scalable benchmark framework that builds synthetic, population-grounded personas embedded in fragmented mobile-app traces to evaluate LLMs on personal-context capabilities including temporal tracking, pattern inference, and constraint-aware action boundaries, finding SOTA models score below 60% on key tasks.

The paper introduces FABench, a scalable backtesting environment for forecasting agents with an automated pipeline that discovers events, generates questions, and resolves outcomes to avoid training-cutoff leakage. Evaluating agents on eight recent models, it finds retrieval consistently improves Brier scores and lifts leading agents to Polymarket market consensus, while revealing recurring evidence-weighting failures.

The paper introduces CLR-Bench, a benchmark curriculum of hidden rules for evaluating whether self-evolving LLM agents can learn latent rules from feedback, finding that most current self-evolution methods barely learn unless given a fixed lesson or a stronger base model.

The paper introduces PersonaChatBench, a framework that evaluates 20 language models across six domains by pairing fixed tasks with diverse simulated users guided by private persona cards, measuring task outcomes, persona need fulfillment, and simulator fidelity. Results show that strong task performance does not consistently translate into satisfying individual user needs, with model strengths varying across personas and domains.

论文提出 GraphNIAH，一个基于真实引文关系自动构建的长上下文多跳推理基准，用于评估 GPT-5.6、Gemini-3、Qwen3.5 等八个大模型，发现随着推理路径加深，模型准确率显著下降甚至接近随机猜测。

The paper introduces BlindToChange, a benchmark measuring whether AI search agents proactively discover recent real-world changes that affect their responses, finding accuracy gaps of up to 27.5 points when changes aren't explicitly mentioned in the request.

The paper compares five LLM-based review paradigms across three input representations and three LLMs on 232 ICLR 2025 submissions, finding that direct rating matches comparative ranking after recalibration with far fewer API calls, abstract-only input beats full-paper summaries, and human scores still predict citations better than any LLM paradigm except Group Rating.

该论文提出 Grounded Generative Speech Recognition (GSR) 新范式，将语音识别重新定义为生成语义有据的文本而非单一转录，并发布 GSR-Bench 基准与多维评估框架，揭示了转录保真度与生成重构能力之间的权衡及常见失败模式。

EDU-Jev introduces a family of single-pass 'System One' decision models (0.8B/2B/4B) that return probability distributions over fixed options for educational assessment tasks, achieving 67.6% average accuracy across twelve benchmarks and outperforming leading commercial decision APIs by 9.4 points.

The paper introduces Bilal-Bench, a 240-question expert-verified Bengali benchmark for open-ended Islamic-source fabrication, evaluating seven open-weight models (1B–120B) and finding that six of seven confidently fabricate on 79.3–95.3% of source-production questions with no monotonic improvement from scale.

Meta Superintelligence Labs researchers formalize 'adaptive personalization under epistemic uncertainty' and introduce a multilingual benchmark revealing a recognition–generation gap in LLMs, where models default to exploit-first behavior despite appropriate strategy identification; targeted training on internal strategy representations narrows the generalization gap.

Microsoft Mage Team presents Mage-VL-Cowork, a full training recipe (data synthesis from evolved harnesses, multi-stage SFT, and online hybrid RL) for learning hybrid computer use, with the 4B model achieving strong results on OSWorld-Verified relative to its size.

StrictPRMBench introduces a replayable benchmark for auditing how fixed-score PRM aggregation rules (product vs. geometric mean) affect best-of-N selection, showing product yields higher deletion-induced selection instability across 14 cells with reproducible reanalysis artifacts.