evaluation

Tag

Cards List
#evaluation

E-Evaluation

Reddit r/AI_Agents ↗ · 2026-10-10

这篇文章科普了微软研究院的论文《G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment》,介绍其通过链式思维生成评分清单并结合各分数的概率加权平均,解决 LLM 评分不稳定的问题。

0 favorites 0 likes
#evaluation

@heyshrutimishra: Okay, this is actually a pretty fascinating report from Anthropic, published today. And I think the most interesting pa…

X AI KOLs Following ↗ · 2026-10-10 Cached

Anthropic announced a new series of more frequent reports on model behavior, with the first one describing four types of behaviors observed in Claude during evaluations and internal use. The tweet highlights that Claude's reaction when it couldn't complete a task is the most interesting part of the report.

0 favorites 0 likes
#evaluation

Last Secure Code Benchmark: Is the Code You Vibe Secure?

Hugging Face Daily Papers ↗ · 2026-10-10 Cached

研究提出 LastSecureCode Benchmark,基于 150 个真实漏洞构建 450 个任务,评估编程智能体生成代码的安全性。结果显示前沿智能体功能通过率虽高(73.6%–91.3%),但功能与安全同时通过的任务仅 16.4%–25.1%,暴露出 vibe coding 产出代码的严重安全隐患。

0 favorites 0 likes
#evaluation

Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Hugging Face Daily Papers ↗ · 2026-10-10 Cached

该论文对决策模型(decision models)和最新生成式 LLM 作为合成问卷受访者的能力进行了心理测量审计,在 14 个决策模型与 47 个生成式模型上测试后发现,模型在样本层面的模拟保真度尚可,但无法替代真实受访者的个体回答,且新一代模型未必优于旧版本。

0 favorites 0 likes
#evaluation

Recent AI models struggled to match a human algorithmic innovation

Hacker News Top ↗ · 2026-10-09 Cached

Epoch AI 发布 InnovationEval 评测,测试前沿 AI 模型能否独立发现媲美人类研究者的新机器学习算法创新;结果显示近期模型即使投入数千美元 GPU 时间也几乎未取得进展。

0 favorites 0 likes
#evaluation

Incremental Open-Ended Deep Research with Structured Harness

arXiv cs.CL ↗ · 2026-10-09 Cached

本文提出 Incremental Open-Ended Deep Research(Incremental-OEDR)研究设定与 Structured Harness 框架,将研究报告视为可随新信息持续演化的结构化状态,在保持报告质量的同时显著降低研究成本(token 消耗减少 33%,搜索调用减少 61%)。

0 favorites 0 likes
#evaluation

SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction

arXiv cs.CL ↗ · 2026-10-09 Cached

SWE-Journey 提出一个更贴近真实场景的编码助手评测基准,通过弱到强合成流水线构建长时程编码任务,并利用从真实交互数据挖掘的用户画像与模拟智能体复现多轮交互。实验显示模型面对软件架构师时通过率超75%,但面对非程序员时不足25%。

0 favorites 0 likes
#evaluation

Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation

arXiv cs.CL ↗ · 2026-10-09 Cached

This paper shows that a single-character colon edit in a candidate answer can sharply increase a decision model judge's false acceptance rate (from 1% to 26%) depending on how the structured judging request is presented, demonstrating that judging competence can coexist with presentation-dependent vulnerabilities.

0 favorites 0 likes
#evaluation

Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation

arXiv cs.CL ↗ · 2026-10-09 Cached

Dialpad researchers propose a sociolinguistic approach to authoring and measuring user persona fidelity in agentic system evaluation, showing that concrete stylistic-rate personas improve stylistic adherence over flat descriptive baselines across five customer-service agents.

0 favorites 0 likes
#evaluation

When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection

arXiv cs.CL ↗ · 2026-10-09 Cached

The paper introduces PARCEL, a claim-level benchmark for detecting legal hallucinations in LLM output, using 3,396 parenthetical-style claims from New York State Court of Appeals decisions framed as three-way NLI. State-of-the-art LLMs reach up to 0.97 accuracy zero-shot but still mislabel unsupported claims as supported, especially with fabricated yet plausible citations.

0 favorites 0 likes
#evaluation

RH-Detect: A Unified Benchmark for Reward Hacking Detection

arXiv cs.LG ↗ · 2026-10-09 Cached

RH-Detect 是一个统一的奖励黑客检测基准,整合了 11 个公开数据集共 92,761 行数据,并评估了六个现成语言模型作为检测器,最佳模型 AUROC 达 0.962,但在多轮工具使用场景中仍存在明显差距。

0 favorites 0 likes
#evaluation

Mine Odyssey: Benchmarking Spatial Agentic Intelligence in the Wild

arXiv cs.AI ↗ · 2026-10-09 Cached

本文提出 Mine Odyssey,一个基于 Minecraft 真实世界地点重建的 agentic spatial intelligence 基准,包含 20 个国家/地区的 30 个地点、180 项导航任务。评测显示 GPT-6 Astra 以 85.6% 成功率领先,Claude Opus 5.5 为 73.9%,而最强开源模型 DeepSeek-V4.1-Flash 仅 23.9%,凸显当前模型在空间智能上的差距。

0 favorites 0 likes
#evaluation

MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction

arXiv cs.AI ↗ · 2026-10-09 Cached

The paper introduces MedBenchAgent, a multi-agent framework that automates the full specification and construction of medical vision-language model benchmarks via a 'constrained compilation' approach, achieving a Task-Space F1 of 90.9% and high human audit pass rates across twelve evaluated VLMs.

0 favorites 0 likes
#evaluation

GameCommBench: A Unified Benchmark and Type-Aware Evaluation for AI-Generated Game Commentary

arXiv cs.AI ↗ · 2026-10-09 Cached

提出了 GameCommBench,一个覆盖棋类、体育和电竞的统一 AI 游戏解说基准,并提出类型感知解说评估框架 TACE,揭示实况观察和策略分析是 AI 解说的主要瓶颈。

0 favorites 0 likes
#evaluation

OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences

arXiv cs.AI ↗ · 2026-10-09 Cached

OpenProblemBench introduces a benchmark of 82 unresolved problems from mathematics and theoretical physics, evaluating AI models' ability to make progress on open scientific questions, with GPT-6-Astra achieving the highest mean solve rate of 14.0%.

0 favorites 0 likes
#evaluation

AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks

arXiv cs.AI ↗ · 2026-10-09 Cached

AgentHorizon introduces a benchmark of 1,373 computer-use tasks with human-recorded trajectories to evaluate agentic judges on long-horizon, multi-application tasks. The study finds GPT-5.5 achieves 80.9% balanced accuracy as the best agentic judge, and reveals that tool use helps some models but hurts open-weight models.

0 favorites 0 likes
#evaluation

Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

arXiv cs.AI ↗ · 2026-10-09 Cached

This paper introduces 'normative competence'—the ability of LLM agents to discern community-enforced norms from interaction alone—and evaluates it via a multi-agent debate setting with synthetic norms. Baseline LLMs fail to learn norms even when beneficial, and norm-following shows poor generalizability, with agents exhibiting unselective attribution by copying idiosyncratic noise alongside true norms.

0 favorites 0 likes
#evaluation

GraphDecisionBench: Benchmarking System One Models on Graph Tasks

Hugging Face Daily Papers ↗ · 2026-10-09 Cached

GraphDecisionBench:一个模型无关的基准,用于评估 System One 模型在图结构任务上的决策能力,涵盖 14 种模型-接口配置,揭示了邻接识别准确并不等同于结构正确性等关键发现。

0 favorites 0 likes
#evaluation

ATLAS: Evaluating Agents on Search-Intensive Tasks (13 minute read)

TLDR AI ↗ · 2026-10-09 Cached

Exa introduces ATLAS, a new benchmark for evaluating AI agents on search-intensive, real-world research tasks with verified golden answers and a refreshable automated pipeline. Key findings: no sub-$1 agent run achieved row F1 over 0.5, top agents still miss ~1/3 of golden results, and Exa's search backend defines the cost-performance Pareto frontier.

0 favorites 0 likes
#evaluation

Can AI automate Epoch? (20 minute read)

TLDR AI ↗ · 2026-10-09 Cached

Epoch AI 发布了一项评测报告,用自身真实工作任务测试六个模型,发现 Fable 5.1 与 GPT-6 Astra 虽然领先,但在开放式任务、隐性标准遵循和研究判断力上仍无法实现工作自动化。

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback