Tag
This paper introduces LLM-NRM, an option-level psychometric framework for multiple-choice benchmarks that models the full distribution over answer choices rather than binary correctness, showing that incorrect responses carry useful measurement information and improving ability estimation and benchmarking efficiency.
This paper proposes automatically auditing answer correctness and legal authority grounding jointly in legal benchmarks, showing that LLMs often produce correct answers while citing wrong governing provisions under ordinary prompts.
This paper evaluates whether clinician pairwise preferences reliably indicate clinical safety in LLMs, using 26,804 judgments from 736+ clinicians across 13 models. It finds that preference rankings poorly track safety-critical failures and proposes a clinically adjusted ranking that better incorporates rubric-based safety signals.
Introduces BBOWP-Bench, a benchmark suite for evaluating LLMs on black-box optimization word problems, where systems must infer both a search space and an optimization algorithm from natural-language descriptions. Initial results show LLMs can select suitable algorithms but struggle with search space design.
This paper presents a modular multi-agent platform for adversarially stress-testing role-playing language agents, using a strategy-driven Interrogator Agent and automated Judging Agent to reveal cumulative behavioral failures across multi-turn dialogues. Experiments across three personas and LLM families show multi-strategy adversarial evaluation reduces robustness scores by 0.17-0.20 and identifies common failure patterns, with strong human alignment.
Introduces MedPIC-Bench, a benchmark with counterfactual questions to evaluate whether LLMs correctly apply medication-safety rules when patient-specific conditions change; across 28 LLMs, accuracy drops significantly on counterfactual questions, revealing a common failure to revise judgments.
This paper introduces CurveShift, an analysis method that separates overall ability gains from difficulty-specific improvements in LLM agents. Using METR time-horizon data and LiveCodeBench, it finds that most apparent shifts toward harder tasks are ceiling effects, though a genuine hard-task effect exists for reasoning models in competitive programming.
This arXiv preprint studies the semantic dispersion of sixteen language models forming ensembles, showing that ensemble diversity is small on average and that model identity only partially explains which model is most divergent. The authors propose a per-model dissent contribution metric and find that dispersion is organized by clinical content rather than interpretive openness.
Introduces XL-DocBench, a human-verified benchmark for extra-long document understanding with 1,519 questions across six professional domains, requiring multi-page evidence and structured reasoning, showing current LLMs still struggle with long-context professional documents.
CalibratedRubric is a task-adaptive framework for building compact, measurable rubric banks for open-ended LLM evaluation, using Bayesian measurability filtering and IRT-based selection to improve human-gold agreement and rank fidelity across financial, healthcare, general, and legal benchmarks.
Introduces Hy-MultiTurn, a Chinese benchmark for deep multi-turn dialogue understanding, evaluating 22 frontier models across six controlled modes (constraint memory, precise execution, constraint synthesis, object localization, action suppression, reference resolution) with 209 tasks spanning 12-76 turns. Even the strongest model, GPT-5.5, satisfies only 41.1% of responses.
This paper proposes a dataset-centric meta-evaluation framework that audits LLM benchmarks at the sample level across five latent dimensions, exposing internal heterogeneity and enabling criterion-driven composition of benchmark subsets for targeted model evaluation.
DungeonBench is a new benchmark for evaluating tactical reasoning in Dungeons & Dragons combat, testing AI policies on rules-rich decision-making across single encounters and linked adventuring days. Frontier language models often win direct fights but struggle with resource budgeting and rest timing over longer horizons.
A research paper introducing Chain-of-Models (CoM), an automated pipeline where a second LLM audits a first model's reasoning trace to correct cognitive biases. It finds that auditor effectiveness depends on model family and bias type, and proposes a bias-specific auditor selection rule that improves judgment accuracy.
ModelEquivBench is a certifying multi-relational evaluation system for LLM-generated optimization models, reporting per-pair semantic profiles across seven equivalence relations instead of a single accuracy score. It evaluates GPT-5.4, Claude Sonnet 4.6, and Qwen3.5-397B-A17B on a fixed benchmark, revealing stage-wise failures that coarse baselines miss.
EarlyDx is a new large-scale benchmark for evaluating LLMs on open-ended, evidence-supported diagnosis generation at emergency department admission, built from 154,834 MIMIC-IV encounters. It reveals that even frontier and medical-specialized models struggle to synthesize admission-time evidence, with post-training only partially improving inference-dependent recall.
This paper proposes a benchmarking protocol using automated multi-model LLM review to evaluate AI Scientist systems, comparing frameworks like Sakana AI, CycleResearcher, and Data-to-Paper, and finds that FARS benchmark papers significantly outperform other systems.
A new paper introduces SaliTrap, a benchmark of 1,145 prompts revealing that LLMs know impossible tasks but still optimize for explicit details, a failure called salience bias. Even the best models avoid traps only ~55% of the time, and awareness often doesn't prevent compliance.
A tweet from Sentdex urging AI companies to stop over-optimizing benchmark harnesses in AI releases, criticizing the practice of gaming evaluation metrics.
A developer built an open-source benchmark called The Slop Index to measure how much 18 AI models produce 'AI slop', using human baselines and five dimensions including conciseness, templating, and human preference.