Tag
This paper proposes matched-record evaluation for text classifiers in industrial settings, demonstrating that record selection significantly impacts performance metrics across maintenance, safety, and recall systems.
GAVEL is an LLM-judge protocol for evaluating clinical timelines extracted from case reports, allowing comparison and revision without treating either timeline as ground truth. It reduces discrepancies and improves accuracy in temporal information extraction.
This paper proposes TEAM-Design, a budgeted rule for allocating replay tasks to evaluate human-AI workflow effectiveness compared to human-only or agent-only alternatives, with applications in clinical and coding settings.
The article critiques AI leaderboards for oversimplifying agent evaluations by hiding the impact of harnesses, using Questflow's financial-intelligence benchmark as an example, and emphasizes the need for comprehensive reporting in agent benchmarks.
The paper introduces diversity profiles as a curve-valued method to evaluate diversity in AI-generated content, addressing the limitations of ambiguous scalar metrics by providing a more transparent and resolution-aware framework.
This paper introduces the 'crystallization problem' for evaluating reusable memory in text-to-SQL systems, showing that storing verified corrected queries in a per-database bank improves held-out first-attempt accuracy by 4.34 points on BIRD, capturing 44.4% of the headroom provided by on-demand repair. Controlled interventions identify database-specific content as the main driver.
Introduces Contrastive Error Span Annotation (cESA), a protocol for human evaluation of multiple translations simultaneously, reducing annotation time and noise compared to standard methods.
This article explains how Similarweb evaluates long-form agent research reports using LangSmith, combining deterministic checks for tool calls and LLM-as-judge scoring for quality, with a focus on making regressions inspectable and enabling A/B comparisons.
This paper demonstrates that naive train/test splitting on sliding-window sequences can severely inflate or deflate performance metrics in multi-task learning for predictive maintenance, and proposes a leakage-robust evaluation protocol.
This paper identifies a confound in best-of-N TTS evaluation where the apparent quality of ASR verifiers depends strongly on which ASR family is used as evaluator. The authors propose cross-family rank ensembles that achieve lower word error rates across multiple evaluators.
This paper investigates how discourse-role labels (e.g., 'Reference:', 'Instruction:', 'Example:') used to wrap context in RAG systems significantly affect how much language models adopt misleading information, with shifts of 56–84 percentage points observed across GPT-5.5, DeepSeek V4 Pro, Llama-3-8B-Instruct, and Qwen2.5-7B-Instruct. The authors argue that wrapper labels should be treated as presentation-time variables and reported/controlled in context-utilization benchmarks.
This paper adapts paired binary sample-size calculations to 4-bit quantization benchmarks, providing a conservative minimum detectable effect (MDE) bound that helps benchmark designers determine reliability before running experiments. A pilot audit shows that much of the observed variance across small subsamples is binomial sampling noise, not true model unreliability.
OpenAI shares lessons and recommended approaches for designing trustworthy third-party evaluations of frontier models, emphasizing the critical role of evaluation harnesses and validity checks.
This paper proposes a two-stage sampling design where LLM evaluations are used to augment, rather than replace, human ratings, and provides guidance on determining sample sizes for human and LLM reviews using a doubly robust estimator from missing data literature.
This paper argues that simple averaging in AI benchmarks fails under data sparsity and difficulty heterogeneity, proposing Item Response Theory (IRT) as a robust alternative to recover ground truth rankings.
A new paper from PwC challenges the intuition that 'earlier is better' for agent clarification, showing via a forced-injection framework that goal clarification loses value quickly while input clarification remains useful longer. The study provides quantitative demand curves for when agents should ask questions, revealing that current frontier models often mistime their clarifications.
Anthropic reveals that infrastructure configuration and resource enforcement significantly impact scores in agentic coding benchmarks like Terminal-Bench 2.0, often exceeding the margins between top models.