Tag
Cartesia launched Managed Agents, an end-to-end enterprise voice agent platform that topped Artificial Analysis' Speech Agent Arena and ServiceNow's EVA-Bench in blind human preference rankings by co-designing its cascaded voice models and agent harness as one system.
该论文批判了传统机器翻译评估方法（MQM、ESA）的局限，提出了RATE（Refined Assessment for Translation Evaluation）协议和由专业语言学家标注的高质量英俄翻译评估数据集，发现SOTA机器翻译系统在准确度上可能已超越人类翻译，但在流畅度上仍存在差距。
Claude Opus 5.5 reportedly composed an oboe concerto that 60% of listeners judged as human-written, surpassing the misattribution rates for actual Handel (58%) and Vivaldi (43%) pieces in the same test.
This paper evaluates how well LLMs can replace human annotators for machine translation quality assessment across MQM and ESA schemes, 70 language pairs, and WMT23/WMT25 data, finding that both humans and LLMs remain unreliable in many settings and proposing human-LLM collaborative annotation pipelines.
该论文从心理测量学视角评估 LLM-as-a-judge，使用 Many-Facet Rasch 模型将评分分解为潜在质量、评分者严苛度与残差难度，发现人类与 LLM 在汇总对齐之外仍存在明显的残差难度结构错配。
The article discusses the challenge of verifying AI-written code when AI also creates verification tools, highlighting human evaluation as a bottleneck and the strategic importance of training data between US and Chinese labs.
Introduces Human Bench 3.0, a benchmark for evaluating AI systems in comparison to human performance, with Humanity-2 AI-0 likely indicating a specific score or version.
The paper proposes information satisfaction as a reader-centered axis for summarization evaluation, showing that current metrics fail to capture user-specific informational needs and agree poorly with human judgment.
This paper introduces EuroExec, a human-expert benchmark for evaluating frontier LLMs on open-ended European executive decision tasks. It finds that the strongest model solves only 56.9% of tasks, falling well short of expert-written reference answers, highlighting gaps in real-world open-ended problem-solving.
This paper formalizes multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup, adaptively allocating annotation effort to focus on competitive models and improve ranking discrimination.
FriendBench is a new benchmark for evaluating whether humans and multimodal LLMs can infer if two people are familiar or strangers from a 20-second video clip of an ice-breaker conversation. Results show the best models match human accuracy but differ in bias, and only humans benefit from richer visual behavior.
This paper presents a controlled comparison of span-guided and unguided text detoxification via human evaluation, finding a trade-off: span-guided rewriting is preferred when preserving original stance is important, while unguided rewriting is favored for more complete mitigation, especially on milder toxicity. The study also assesses automatic evaluators as diagnostics rather than substitutes for human judgment.
Introduces Contrastive Error Span Annotation (cESA), a protocol for human evaluation of multiple translations simultaneously, reducing annotation time and noise compared to standard methods.
A large-scale human evaluation of 10 frontier LLMs across 62 questions finds that all models exhibit response drift, with most converging to a 78-81% deviation ceiling, while two achieve lower deviation. Drift varies by domain and question, and automated metrics explain little of human judgments, highlighting the need for human evaluation.
Real World VoiceEQ is a new benchmark for evaluating the human quality of voice AI, based on over a million human ratings, assessing models across speech recognition, synthesis, and understanding in real-world conditions.
A study comparing human and AI translations of literary works shows that while machine translations are deemed 'fine', readers still prefer human translations for their immersiveness and clarity. Automatic metrics fail to capture reader preferences.
The author praises GLM-5.2, an MIT open-weights model, for its exceptional real-world performance in human evaluation benchmarks, claiming it rivals the best closed-source models like those from Claude.
This exploratory study evaluates whether augmenting AI agents with a medical research skill package improves the quality of transcriptomic research analysis outputs compared to native AI, using a multi-model human evaluation in an NSCLC biomarker task. Results show a directional but statistically non-significant improvement, highlighting the need for larger, more robust evaluations.
This paper introduces RQ-Bench, a benchmark to evaluate LLMs' ability to assess the novelty of scientific research questions. It finds that LLM judges consistently rate generated questions as more novel than human experts do, raising concerns about the reliability of using LLMs for scientific novelty evaluation.
This paper investigates the effectiveness of LLM personalization by putting real humans back into the evaluation loop, revealing systematic gaps between human judgments and LLM outputs at every stage of the personalization pipeline, and highlighting the limitations of synthetic data and LLM judges.