evaluation

Tag

Cards List
#evaluation

VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation

arXiv cs.AI ↗ · 2026-10-01 Cached

Veris AI introduces VAmoS Energy, a harder voice-agent benchmark combining multi-request utility billing calls, stateful Stripe/Fineract twins, caller verification gates, and background speech — finding completion rates of 17.3-44.7% across 14 voice stacks, with Grok Voice leading and background TV slashing pooled completion from 38.7% to 8.6%.

0 favorites 0 likes
#evaluation

Reach Into The CHOIR: Free-List Elicitation Uncovers Distinct Model Voices in LLM Ensembles

arXiv cs.AI ↗ · 2026-10-01 Cached

The paper introduces CHOIR (Collective Hierarchically-Ordered Inquiry Responses), a framework adapting free-list elicitation from cognitive anthropology to probe and measure hidden diversity beneath surface agreement in LLM ensembles, finding that base-model identity is the strongest recoverable signature of distinct model voices.

0 favorites 0 likes
#evaluation

SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

Hugging Face Daily Papers ↗ · 2026-10-01 Cached

SimuVerity introduces a benchmark of 101 text-to-executable Simulink model generation tasks across ten engineering domains, using a hierarchical evaluator to assess engineering qualification beyond structural similarity. The best evaluated agent system scores only 42.86, revealing major capability bottlenecks in engineering-grade model generation.

0 favorites 0 likes
#evaluation

Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Hugging Face Daily Papers ↗ · 2026-10-01 Cached

The paper introduces Ego2Act, a goal-directed benchmark of 2,640 egocentric videos across 110 real-world tasks for evaluating whether video generation models can simulate multi-step physical manipulation, along with Ego2Act Judge, a reference-free evaluation pipeline that aligns better with human judgments of task completion and physics plausibility.

0 favorites 0 likes
#evaluation

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Hugging Face Daily Papers ↗ · 2026-10-01 Cached

VTR-Bench introduces a systematic benchmark for evaluating visual text rendering in video generation models, covering 300 prompts across five scenario categories, an automated evaluation pipeline with human alignment, and a Keyframe-Guided Agentic Framework. Experiments on 11 state-of-the-art models reveal widespread difficulties in rendering scene text, with the best model achieving a word error rate of 0.250.

0 favorites 0 likes
#evaluation

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Hugging Face Daily Papers ↗ · 2026-10-01 Cached

Argo-Bench introduces an evaluation framework of 210 enterprise-scale data science tasks built on a simulated food-delivery platform with an ERP warehouse of 235 tables and 7.5 billion rows, testing agents that must navigate and act rather than just generate SQL. The strongest of 14 frontier and open-weight models averages only 59.5 points, highlighting a significant gap in real-world data agent capabilities.

0 favorites 0 likes
#evaluation

ROWBench: Do Video Models Render What the Program Specifies?

Hugging Face Daily Papers ↗ · 2026-10-01 Cached

PROWBench introduces 170 programmatically constructed episodes and 600 proxy videos to evaluate whether generative video models faithfully render program-specified world dynamics, using VLM-based metrics for logic-render alignment and interaction success.

0 favorites 0 likes
#evaluation

Sharpening Tax in Post-Training

Hugging Face Daily Papers ↗ · 2026-10-01 Cached

This paper introduces 'Sharpening Tax,' a metric quantifying how RL post-training improves pass@1 but reduces solution coverage (pass@K) in LLMs, and proposes posterior-tempered group sampling (PTGS), a plug-and-play Bayesian sampler that adapts temperature per prompt to pay a smaller tax while improving single-shot accuracy.

0 favorites 0 likes
#evaluation

Generalization Dynamics of LM Pre-training (4 minute read)

TLDR AI ↗ · 2026-10-01 Cached

该论文提出"模式跳变"(mode-hopping)概念,指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变(如OLMo3-32B在算术提示上81%→0%→81.7%的反复),并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。

0 favorites 0 likes
#evaluation

Use Reddit to test whether an AI answer preserves the constraints that actually matter

Reddit r/artificial ↗ · 2026-09-30

A practical walkthrough on using real Reddit threads to build a small test set for checking whether an LLM preserves the hidden constraints (privacy, offline-only, installation limits) in real-world questions. The author proposes an answer-key workflow that counts constraint violations as failures and separates persuasive-sounding prose from correctness.

0 favorites 0 likes
#evaluation

@OReillyMedia: What’s the data scientist’s role when agents can run the analysis? “AI-native data scientists build and direct an analy…

X AI KOLs Following ↗ · 2026-09-30 Cached

An O'Reilly article outlining 'The Agentic Data Science Playbook' — how AI-native data scientists delegate analysis to AI agents while designing workflows, verification, and feedback loops. Using a modified Elliptic Bitcoin fraud dataset, Claude Opus 5.0 initially reported inflated results (F1 0.87) due to a temporal-split error and a planted leaking feature; with proper guidance, corrected performance dropped to F1 0.70 and 0.21 recall on high-degree nodes, showing why expert oversight still matters.

0 favorites 0 likes
#evaluation

GPT-6.1 Sol reaches third place in Humanity's Last Exam — Diamond

Reddit r/singularity ↗ · 2026-09-30

GPT-6.1 Sol has reached third place on the Humanity's Last Exam (Diamond) leaderboard, marking a significant milestone in frontier AI benchmark performance.

0 favorites 0 likes
#evaluation

@LangChain: Engine learns how your agent works by reading traces and repos, then tests for agent-specific weaknesses and flags any …

X AI KOLs Following ↗ · 2026-09-30 Cached

LangChain introduces an Engine that learns how your agent works by reading traces and repositories, then tests for agent-specific weaknesses such as hallucinations and system prompt violations, returning a list of verified issues to fix before they impact users.

0 favorites 0 likes
#evaluation

We stress-tested America.gov on night one with 15 questions from our own reporting. 9 correct, 6 incomplete, zero hallucinations.

Reddit r/artificial ↗ · 2026-09-30

A reporter stress-tested GSA's new America.gov AI assistant with 15 research-grade questions from prior reporting, finding 9 correct answers, 6 incomplete, and no hallucinations, though the system missed available public data and lacked transparency about its sources. The finding suggests the tool refuses to fabricate when it lacks information but cannot establish what the government has not published.

0 favorites 0 likes
#evaluation

PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

arXiv cs.CL ↗ · 2026-09-30 Cached

PADMÉ reformulates meta-evaluation of LM agent evaluators as a preference alignment problem and introduces a data synthesis method using only small language models, boosting agreement with human judgment from 73% to 85% over a naive baseline while enabling evaluation of 25 common models across agentic domains.

0 favorites 0 likes
#evaluation

The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models

arXiv cs.CL ↗ · 2026-09-30 Cached

This paper identifies a 'detectability gap' in hallucination detection: hallucinations split into high-agreement (Ghost) and low-agreement (Flickering) regimes with a 0.35–0.46 AUC gap, persisting across four models and three factual QA datasets even after freezing regime assignments and using stricter trajectory-based tests. The authors argue aggregate detection metrics hide model-dependent heterogeneity and call for regime-conditioned evaluation.

0 favorites 0 likes
#evaluation

$\tau$-Multilingual: Benchmarking Voice Agents Across Languages

arXiv cs.CL ↗ · 2026-09-30 Cached

The paper introduces tau-Multilingual, a benchmark extending tau-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin for evaluating voice agents across task completion, interaction quality, and language generation. Results show Spanish, Portuguese, and Hindi perform close to English, while Korean and Mandarin drop significantly, and the authors release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.

0 favorites 0 likes
#evaluation

Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions

arXiv cs.CL ↗ · 2026-09-30 Cached

BreakingWeb is a benchmark framework from Amazon and Duke that constructs challenging browser-use tasks via controlled, deterministic environment interventions on tasks agents already solve, cutting agent pass rates by 22.9% while humans lose only 10.0%, revealing widespread 'belief failure' where agents falsely declare success.

0 favorites 0 likes
#evaluation

Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants

arXiv cs.CL ↗ · 2026-09-30 Cached

研究人员提出一个用于评测车载对话助手(ICA)多轮对话能力的自动化黑盒测试框架,采用策略引导的用户模拟器、对抗性策略管理器和两级 LLM 评审器,在工业级 ICA 上验证该方法与人类标注高度一致,并发现 2.96 倍的独特失败类型。

0 favorites 0 likes
#evaluation

Evaluating the Effects of Prompt Perturbation on Bias and Hallucination in Large Language Models

arXiv cs.CL ↗ · 2026-09-30 Cached

This arXiv paper evaluates how prompt perturbations affect bias and hallucination in LLMs used for decision-making tasks, finding that perturbations can mitigate these issues in some models, with Claude 3 outperforming GPT-3.5 across most datasets.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback