evaluation-metrics

The paper presents Multi-Label Agreement (MLA), a flexible framework for evaluating inter-annotator reliability in multi-label annotation tasks using pairwise Hamming agreement with multiple normalization options, along with a Python implementation.

This paper extends the Platonic Representation Hypothesis to robotics, introducing Sensorimotor Alignment (S²) as an offline proxy that measures overlap between a model's sensory representations and end-effector trajectories, showing it correlates with downstream robot task success across VLAs, WAMs, and real-robot policies.

该论文提出一个多粒度评估框架，结合字符错误率、音节错误率和分词错误率，评估藏语语音识别在多方言与不同适配策略下的表现，发现低字符错误率并不总带来更优的高层级指标，且 LoRA 适配在测试范围内仍逊于全量微调。

This paper formalizes benchmark evaluation as an empirical measure over question difficulty, identifying failure modes like measure mismatch and coverage gaps, and proposes diagnostic metrics plus BenchFill, a difficulty-targeted item rewriting algorithm, with analysis across 25 popular benchmarks in 7 domains.

OpenAI 探讨了 g-mean² 之外用于衡量模型可监控性的补充指标，针对结果、过程与干预三类评估，分析任务混合差异与部署时控制使用场景对单一监控性数值的局限。

The article questions whether embodied AI is advancing faster or slower than language models, noting that robots are judged by demos rather than benchmarks, and asks what fair metrics could compare physical AI progress with text and reasoning progress.

该论文提出了一种基于回归的元评估框架，用于评估多语言风格个性化文本生成中的自动指标，发现经过校准的简单字符串和嵌入指标在40种评估设置下可媲美LLM-judge。

论文提出 semCER，一种利用 LLM 评审按严重程度对字符编辑加权的 OCR 评估指标，相比标准 CER 更能贴合人类评分，并在英、越、中等 16 种语言上验证有效，同时发布了 6100 万条已评分编辑数据集与蒸馏打分模型。

MFVSR 提出一个大规模、多风格的视频字幕移除基准，通过空间解耦的细粒度协议评估修复保真度、区域保持性、时间一致性与可靠性，涵盖3000对视频和16种语言。

微软发布 ThinkingBox-Bench，包含 507 个状态化企业工作流任务，每个任务独立运行 20 次并以终端数据库状态评分，揭示 pass@20 与 all-20 排名几乎完全相反，且 67% 的失败轨迹在完成式代理下会被误判为成功。研究显示发现能力与可重复性对模型的排名差异巨大，Kimi-K3 覆盖最广但重复率低，Claude Opus 5 重复稳定性更高。

A Lancaster University study benchmarks summarisation methods on pre-trained transformers for long UK financial annual reports (FNS 2020 dataset), compares evaluation metrics, proposes the novel BRUGEscore (harmonic mean of ROUGE-2 and BERTscore), and validates results with statistical significance testing and adversarial corruption analysis.

This paper identifies blind spots in evaluating deep imbalanced regression, proposing balanced metrics and showing high tail-region instability across random seeds.

This blog post discusses how to benchmark LLMs for specific production use cases, proposing a comprehensive evaluation process that includes task-specific metrics, trade-offs between quality and cost, and analysis of failure cases.

This paper introduces WildfireSpreadBench to benchmark wildfire spread prediction models, revealing that evaluation metrics like AP versus F1 can lead to different model rankings and affect operational suitability.

This study benchmarks gender bias in machine translation evaluation metrics across occupations, revealing that masculine translations tend to score higher and that biases vary by language and evaluator.

This paper introduces methods to quantify how effectively a panel of language models approximates human judgments, using spectral diversity and distribution recovery metrics to measure effective representation.

This paper diagnoses the gap between next-turn evaluation metrics and autonomous workflow execution success for AI agents, showing that supervised fine-tuning improves turn-level performance but fails to enhance end-to-end workflow success.

This paper critiques conventional depth truncation methods for evaluating recursive language models and introduces the Depth Control Protocol (DCP) to disentangle and isolate factors affecting depth utilization, improving evaluation accuracy.

The article argues that three independent findings on LLM reliability problems converge on the need for calibrated abstention, where models can appropriately decline to answer when uncertain, and proposes evaluation reforms such as triple-scoring and calibration metrics to address this gap.

This paper describes a method for building pseudo-references in machine translation evaluation without human references, using multiple translation models, quality estimation, and GPT-5.5 post-editing to improve selection accuracy.