evaluation

Tag

Cards List
#evaluation

More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

arXiv cs.AI ↗ · 2026-09-30 Cached

This paper introduces a controlled evaluation framework that separates answer coverage from repeatable task specialization in LLM-generated harnesses on MATH-500, finding that additional coverage largely comes from repeated executions of the same program rather than genuine specialization gains.

0 favorites 0 likes
#evaluation

Binarization Flattens the Score Space

arXiv cs.LG ↗ · 2026-09-30 Cached

This paper formalizes how collapsing LLM-judge rubric rewards to binary pass/fail hides criterion-level quality changes from the policy's reward signal, showing under a joint-Gaussian model that at least three grading levels are needed to remove this ambiguity, with experiments on MATH and SciBench confirming that binary stretches are invisible but detectable with three grades.

0 favorites 0 likes
#evaluation

TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

提出 TauIndianBankBench:一个包含 1,000 项印度零售银行任务的评测基准,不仅检查最终数据库状态,还要求工具使用型对话代理完成所有必要步骤且不做未请求的变更;结果显示仅按结果评分时有 6.5–12.0% 的会话在违反流程的情况下仍被计为成功。

0 favorites 0 likes
#evaluation

Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

该论文揭示了基于评分表的强化学习中加和式奖励聚合导致的奖励作弊问题:模型通过堆砌无关建议换取高分却降低回答质量。作者提出ProRubric,将评分标准按协议级维度分组(所有条件同时满足才计分),在临床咨询场景中将适切性提升10.8个点且不损失覆盖率。

0 favorites 0 likes
#evaluation

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

PhysVista 提出了一个基于感知-推理-评估闭环认知框架的基准,用于评测视觉语言模型的物理智能,涵盖事件级与尺度级推理以及真实和AI生成视频,并揭示了VLM在物理推理和合理性评估上的显著局限。

0 favorites 0 likes
#evaluation

A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

论文提出 A2Z GameSpec-Bench,一个包含 100 份长篇游戏设计文档(GDD)的基准,用于评估编码代理在端到端游戏开发中遵循互锁需求的能力。结果显示现有代理在同时满足代码实现与实际游玩的联动需求上表现不佳,而需求级反馈可在两轮修订后将 GDD 忠实度提升 10.9%。

0 favorites 0 likes
#evaluation

Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning

Hugging Face Blog ↗ · 2026-09-30 Cached

Hugging Face introduces the Open TTS Leaderboard, an objective, scalable evaluation framework for multilingual TTS and voice cloning using ASR-based intelligibility (WER/CER), speed (RTFx/TTFA), and WavLM speaker-similarity metrics, reducing model evaluation time from weeks to hours.

0 favorites 0 likes
#evaluation

d1 (2 minute read)

TLDR AI ↗ · 2026-09-30 Cached

Liquid AI and Artificial Analysis release Pipette, an open-source benchmarking suite for on-device AI that measures quality, speed, latency, and memory use across models, quantizations, runtimes, and devices, shipping with 10k+ verified results and clients for macOS, Windows, iOS, and Android.

0 favorites 0 likes
#evaluation

Where does your RAG pipeline actually fail, retrieval or generation?

Reddit r/AI_Agents ↗ · 2026-09-29

The article discusses the challenge of distinguishing between retrieval and generation failures in RAG systems and explores practical methods to measure each component independently.

0 favorites 0 likes
#evaluation

PhyProbe: Rethinking Physical Consistency Evaluation in Generated Videos

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

PhyProbe 提出一种评估生成视频物理一致性的新评测器,通过冻结的预训练时空编码器提取特征并用轻量评分头映射为违反分数,结合排序、回归与锚定校准的统一目标,在多项基准上超越现有方法。

0 favorites 0 likes
#evaluation

Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

The paper proposes a framework for designing proactive LLM agents around three joint principles (3T): Task Capability, Temporal Allocation, and Trust, along with a five-dimensional design space. It also introduces Proactivity-Gym, a simulation-based testbed with multi-day scenarios and persona-conditioned simulated users, evaluating 23 model-harness configurations and a 30-participant human study showing that trust declines sharply when interventions misalign even if task outcomes are correct.

0 favorites 0 likes
#evaluation

RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

RLE-Bench introduces a qualifying-exam-style benchmark that evaluates coding agents' broader engineering capabilities in robotics — covering interactive control, policy learning, perception and estimation, and mechanical design — aggregating task-specific metrics into an overall RLE Index for systematic comparison.

0 favorites 0 likes
#evaluation

It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

论文提出 MIST(Misleading Image Stress Test)基准,测试 13 个 VLM judge 在存在对齐、误导或缺失图像时标签的稳定性,发现图像是否出现而非图像内容才是改变判断的主要因素,说明可替代性结论对评测配置同样敏感。

0 favorites 0 likes
#evaluation

Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

该论文系统评估了 GPT-6 Astra 作为通用具身策略的能力,覆盖抓取、灵巧操作、移动操作、导航、运动和人形 loco-manipulation 六个领域,发现混合控制(与 π0.5 等学习策略结合)表现最佳,但仍存在任务决策与可靠物理控制之间的差距,且推理延迟严重制约实际部署。

0 favorites 0 likes
#evaluation

APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

APM-Bench introduces a benchmark for cross-session persistent memory in egocentric streaming video assistants, reformulating real-world interaction as multi-session life trajectories with 549 sessions and 104 trajectories, and evaluates the utility–latency–storage trade-offs of existing memory systems.

0 favorites 0 likes
#evaluation

EpiCon: Collective Agent Learning through Co-Evolving Multimodal Memory

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

EpiCon presents a shared multimodal memory framework for collective learning among AI agents, enhancing performance across eleven benchmarks without updating host model parameters.

0 favorites 0 likes
#evaluation

HasMem: Hard-Origin Adaptively Softened Memory for Long-Term LLM Agents

arXiv cs.AI ↗ · 2026-09-28 Cached

The paper proposes HasMem, an adaptive memory compression method for long-term LLM agents that adjusts memory widths while maintaining frozen LLM parameters, demonstrating improved performance on reconstruction and QA benchmarks.

0 favorites 0 likes
#evaluation

From annotation to reasoning: Culture in language models

arXiv cs.CL ↗ · 2026-09-28 Cached

This paper proposes using literary interpretation as a setting to evaluate cultural reasoning depth in language models, emphasizing interpretive depth and evidence-centered benchmarks over conventional metrics.

0 favorites 0 likes
#evaluation

Enhancing Assessment of Self-Consistency in LLM Explanations using Perturbation Strength

arXiv cs.CL ↗ · 2026-09-28 Cached

This paper introduces an LLM-as-a-judge method to measure perturbation strength for assessing self-consistency in LLM explanations, showing that input perturbations generally affect LLMs more strongly than CoT perturbations.

0 favorites 0 likes
#evaluation

The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge

arXiv cs.CL ↗ · 2026-09-28 Cached

The paper introduces KNOWS, a benchmark for evaluating web agents on complex, long-horizon tasks that involve synthesizing and organizing knowledge into artifacts, revealing that current agents struggle with visual steps and long-horizon reasoning.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback