evaluation-benchmark

Tag

Cards List
#evaluation-benchmark

Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

arXiv cs.CL ↗ · 2d ago Cached

该论文提出了一种基于全概率定律的统计自一致性评测框架,通过二叉树分区、提示与聚合来检验LLM条件概率估计的一致性,发现前沿模型普遍存在违反一致性的问题,并揭示了细粒度子群体估计反而更贴近人类参考数据的'宏观谬误'现象。

0 favorites 0 likes
#evaluation-benchmark

CrisisBench: an evaluation suite for crisis counseling AI

arXiv cs.CL ↗ · 4d ago Cached

CrisisBench introduces a privacy-preserving evaluation suite for crisis counseling AI, featuring 1,085 single-turn tasks, 1,185 simulated patient LLMs, and a calibrated 23-item LLM judge. GPT-based systems still diverge from expert counselors on response-level failure modes, with retrieval-augmented generation scoring highest in multi-turn evaluation.

0 favorites 0 likes
#evaluation-benchmark

@omarsar0: The next breakthrough in voice AI might not be another model taking the #1 spot. It might be a better understanding of …

X AI KOLs Timeline ↗ · 2026-10-30 Cached

The next breakthrough in voice AI may come from rethinking how we evaluate models rather than crowning a new top performer, with Voice Arena's Turing TTS benchmark measuring how often human listeners mistake AI voices for real ones.

0 favorites 0 likes
#evaluation-benchmark

Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs

arXiv cs.CL ↗ · 2026-10-08 Cached

该论文提出 Adversarial Surface-Form Robustness Dataset (ASRD),用四态评估框架对五个开源大模型在 2,100 个非规范形式提示(表情符号、Leetspeak、编码封装、隐形 Unicode 等)下的安全表现进行评测,揭示了表情符号几乎不影响理解而编码类输入导致高理解失败率等规律。

0 favorites 0 likes
#evaluation-benchmark

TestPrism: Rethinking Test Evaluation Beyond a Single Reference

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

The paper introduces TestPrism, a benchmark of 300 test tasks and 3000 candidate implementations, showing that single-reference evaluation overstates test quality (Joint Success Function only reaches 28% vs 59.67%). It also proposes TestHelix, improving Joint Success by 8.67-9.00 points via heterogeneous synthesis, peer cross-validation, and recursive self-improvement.

0 favorites 0 likes
#evaluation-benchmark

ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

arXiv cs.LG ↗ · 2026-10-05 Cached

ArrivalBench introduces an evaluation method that re-executes agent-generated data pipelines under adversarial replay schedules (late, duplicated, out-of-order records) and finds that 7-79% of pipelines certified by single-execution benchmarks are silently wrong when re-run.

0 favorites 0 likes
#evaluation-benchmark

Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents

arXiv cs.CL ↗ · 2026-10-05 Cached

XiangqiBench is an executable benchmark for closed-loop evaluation of LLM agents in Xiangqi endgames, measuring whether models can convert forced-mate positions into verified wins against an engine defender, and revealing large gaps between static move-finding competence and actual game-winning performance across 12 frontier LLMs.

0 favorites 0 likes
#evaluation-benchmark

SlideLab: Audience-Centered Scientific Slide Generation and Evaluation

arXiv cs.CL ↗ · 2026-09-28 Cached

SlideLab introduces a training-free multi-agent framework for generating audience-centered scientific presentations from research papers, which outperforms existing systems in human preference studies, and presents ConfArena for slide-by-slide evaluation.

0 favorites 0 likes
#evaluation-benchmark

No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow

arXiv cs.CL ↗ · 2026-09-25 Cached

This paper introduces Corpus Task Complexity (CTC) to characterize how task difficulty scales with corpus size, presents high-CTC tasks, and releases CTC-Bench, showing that high-CTC tasks are more challenging for long-context language models.

0 favorites 0 likes
#evaluation-benchmark

Cross-Lingual Legal QA for Vietnamese Labour Law: Retrieval, Translation, and Verifier-Guided Correction

arXiv cs.CL ↗ · 2026-09-24 Cached

This paper presents a bilingual evaluation suite for cross-lingual legal question answering in Vietnamese labour law, assessing retrieval, translation, and verifier-guided correction to enhance answer faithfulness and address unsupported claims.

0 favorites 0 likes
#evaluation-benchmark

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings

arXiv cs.CL ↗ · 2026-09-17 Cached

This paper introduces a comprehensive benchmark for evaluating LLMs in key-value extraction from documents under OCR noise, revealing substantial performance degradation and emphasizing the need for joint optimization of OCR quality and LLM reasoning.

0 favorites 0 likes
#evaluation-benchmark

Measuring the Cost of Variety Conflation in Multilingual MT Evaluation: Adding Mozambican Xichangana, Nyanja and Sena to FLORES+

arXiv cs.CL ↗ · 2026-09-15 Cached

This paper extends the FLORES+ benchmark with evaluation sets for Mozambican Xichangana, Nyanja, and Sena, demonstrating that reference variations significantly impact machine translation evaluation scores.

0 favorites 0 likes
#evaluation-benchmark

Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents

arXiv cs.AI ↗ · 2026-09-12 Cached

Sci-MMR is a benchmark for evaluating multi-step evidence-grounded scientific reasoning in multimodal agents, revealing gaps where answer accuracy exceeds evidence recovery by over 20%.

0 favorites 0 likes
#evaluation-benchmark

DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents

arXiv cs.AI ↗ · 2026-09-04 Cached

The paper introduces DuplexSpeechBench-IFEval, a benchmark for evaluating implicit instruction following in full-duplex voice agents, with 1,038 test cases across eight roles and five protocols to assess real-time speech systems' adherence to explicit vs. persona-implied behaviors.

0 favorites 0 likes
#evaluation-benchmark

TabScope: Question-Adaptive Scope Selection for Table Question Answering

arXiv cs.CL ↗ · 2026-09-04 Cached

TabScope is a question-adaptive framework for table question answering that dynamically chooses between localized and full-table reasoning based on question type, improving performance on long tables through operation-aware decomposition and a new benchmark.

0 favorites 0 likes
#evaluation-benchmark

@garrytan: I just made some new GBrain evals that help prove that my retrieval-for-AI-agent open source layer is SOTA for reading …

X AI KOLs Timeline ↗ · 2026-08-31 Cached

Garry Tan has released new evaluations for gbrain, an open-source retrieval layer for AI agents, showcasing state-of-the-art performance in memory reading and writing without LLM-in-loop retrieval.

0 favorites 0 likes
#evaluation-benchmark

PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants

arXiv cs.CL ↗ · 2026-08-28 Cached

The paper introduces PACEShop, a benchmark dataset, and PACEJudge, a judging protocol for evaluating personalized, actionable, compositional, and evidence-grounded shopping assistants, addressing gaps in current evaluation methods.

0 favorites 0 likes
#evaluation-benchmark

A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs

arXiv cs.CL ↗ · 2026-08-28 Cached

This paper presents a unified benchmark comparing outline-driven frameworks for long-form generation with LLMs, evaluating outlines independently and finding that performance varies with framework-granularity match, supporting the decoupling of outline and writing stages.

0 favorites 0 likes
#evaluation-benchmark

FrontierChallenge: Evaluating Scientific Workflow Completion

Hugging Face Daily Papers ↗ · 2026-08-25 Cached

FrontierChallenge is a cross-domain benchmark for evaluating end-to-end scientific workflows, revealing that frontier AI models achieve only 20% completion despite high partial scores, emphasizing the need for better evaluation of complete task delivery.

0 favorites 0 likes
#evaluation-benchmark

Unlocking the Potential of Image Editing via Concept Scaling and Dense Supervision

Hugging Face Daily Papers ↗ · 2026-08-17 Cached

This paper introduces a hierarchical taxonomy and dense supervision training strategy for diffusion-based image editing, leveraging a large dataset of 12 million editing pairs and a granular evaluation benchmark to enhance model performance.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback