Tag
该论文提出了一种基于全概率定律的统计自一致性评测框架,通过二叉树分区、提示与聚合来检验LLM条件概率估计的一致性,发现前沿模型普遍存在违反一致性的问题,并揭示了细粒度子群体估计反而更贴近人类参考数据的'宏观谬误'现象。
CrisisBench introduces a privacy-preserving evaluation suite for crisis counseling AI, featuring 1,085 single-turn tasks, 1,185 simulated patient LLMs, and a calibrated 23-item LLM judge. GPT-based systems still diverge from expert counselors on response-level failure modes, with retrieval-augmented generation scoring highest in multi-turn evaluation.
The next breakthrough in voice AI may come from rethinking how we evaluate models rather than crowning a new top performer, with Voice Arena's Turing TTS benchmark measuring how often human listeners mistake AI voices for real ones.
该论文提出 Adversarial Surface-Form Robustness Dataset (ASRD),用四态评估框架对五个开源大模型在 2,100 个非规范形式提示(表情符号、Leetspeak、编码封装、隐形 Unicode 等)下的安全表现进行评测,揭示了表情符号几乎不影响理解而编码类输入导致高理解失败率等规律。
The paper introduces TestPrism, a benchmark of 300 test tasks and 3000 candidate implementations, showing that single-reference evaluation overstates test quality (Joint Success Function only reaches 28% vs 59.67%). It also proposes TestHelix, improving Joint Success by 8.67-9.00 points via heterogeneous synthesis, peer cross-validation, and recursive self-improvement.
ArrivalBench introduces an evaluation method that re-executes agent-generated data pipelines under adversarial replay schedules (late, duplicated, out-of-order records) and finds that 7-79% of pipelines certified by single-execution benchmarks are silently wrong when re-run.
XiangqiBench is an executable benchmark for closed-loop evaluation of LLM agents in Xiangqi endgames, measuring whether models can convert forced-mate positions into verified wins against an engine defender, and revealing large gaps between static move-finding competence and actual game-winning performance across 12 frontier LLMs.
SlideLab introduces a training-free multi-agent framework for generating audience-centered scientific presentations from research papers, which outperforms existing systems in human preference studies, and presents ConfArena for slide-by-slide evaluation.
This paper introduces Corpus Task Complexity (CTC) to characterize how task difficulty scales with corpus size, presents high-CTC tasks, and releases CTC-Bench, showing that high-CTC tasks are more challenging for long-context language models.
This paper presents a bilingual evaluation suite for cross-lingual legal question answering in Vietnamese labour law, assessing retrieval, translation, and verifier-guided correction to enhance answer faithfulness and address unsupported claims.
This paper introduces a comprehensive benchmark for evaluating LLMs in key-value extraction from documents under OCR noise, revealing substantial performance degradation and emphasizing the need for joint optimization of OCR quality and LLM reasoning.
This paper extends the FLORES+ benchmark with evaluation sets for Mozambican Xichangana, Nyanja, and Sena, demonstrating that reference variations significantly impact machine translation evaluation scores.
Sci-MMR is a benchmark for evaluating multi-step evidence-grounded scientific reasoning in multimodal agents, revealing gaps where answer accuracy exceeds evidence recovery by over 20%.
The paper introduces DuplexSpeechBench-IFEval, a benchmark for evaluating implicit instruction following in full-duplex voice agents, with 1,038 test cases across eight roles and five protocols to assess real-time speech systems' adherence to explicit vs. persona-implied behaviors.
TabScope is a question-adaptive framework for table question answering that dynamically chooses between localized and full-table reasoning based on question type, improving performance on long tables through operation-aware decomposition and a new benchmark.
Garry Tan has released new evaluations for gbrain, an open-source retrieval layer for AI agents, showcasing state-of-the-art performance in memory reading and writing without LLM-in-loop retrieval.
The paper introduces PACEShop, a benchmark dataset, and PACEJudge, a judging protocol for evaluating personalized, actionable, compositional, and evidence-grounded shopping assistants, addressing gaps in current evaluation methods.
This paper presents a unified benchmark comparing outline-driven frameworks for long-form generation with LLMs, evaluating outlines independently and finding that performance varies with framework-granularity match, supporting the decoupling of outline and writing stages.
FrontierChallenge is a cross-domain benchmark for evaluating end-to-end scientific workflows, revealing that frontier AI models achieve only 20% completion despite high partial scores, emphasizing the need for better evaluation of complete task delivery.
This paper introduces a hierarchical taxonomy and dense supervision training strategy for diffusion-based image editing, leveraging a large dataset of 12 million editing pairs and a granular evaluation benchmark to enhance model performance.