Tag
VTR-Bench introduces a systematic benchmark for evaluating visual text rendering in video generation models, covering 300 prompts across five scenario categories, an automated evaluation pipeline with human alignment, and a Keyframe-Guided Agentic Framework. Experiments on 11 state-of-the-art models reveal widespread difficulties in rendering scene text, with the best model achieving a word error rate of 0.250.
Argo-Bench introduces an evaluation framework of 210 enterprise-scale data science tasks built on a simulated food-delivery platform with an ERP warehouse of 235 tables and 7.5 billion rows, testing agents that must navigate and act rather than just generate SQL. The strongest of 14 frontier and open-weight models averages only 59.5 points, highlighting a significant gap in real-world data agent capabilities.
PROWBench introduces 170 programmatically constructed episodes and 600 proxy videos to evaluate whether generative video models faithfully render program-specified world dynamics, using VLM-based metrics for logic-render alignment and interaction success.
This paper introduces 'Sharpening Tax,' a metric quantifying how RL post-training improves pass@1 but reduces solution coverage (pass@K) in LLMs, and proposes posterior-tempered group sampling (PTGS), a plug-and-play Bayesian sampler that adapts temperature per prompt to pay a smaller tax while improving single-shot accuracy.
该论文提出"模式跳变"(mode-hopping)概念,指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变(如OLMo3-32B在算术提示上81%→0%→81.7%的反复),并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。
A practical walkthrough on using real Reddit threads to build a small test set for checking whether an LLM preserves the hidden constraints (privacy, offline-only, installation limits) in real-world questions. The author proposes an answer-key workflow that counts constraint violations as failures and separates persuasive-sounding prose from correctness.
An O'Reilly article outlining 'The Agentic Data Science Playbook' — how AI-native data scientists delegate analysis to AI agents while designing workflows, verification, and feedback loops. Using a modified Elliptic Bitcoin fraud dataset, Claude Opus 5.0 initially reported inflated results (F1 0.87) due to a temporal-split error and a planted leaking feature; with proper guidance, corrected performance dropped to F1 0.70 and 0.21 recall on high-degree nodes, showing why expert oversight still matters.
GPT-6.1 Sol has reached third place on the Humanity's Last Exam (Diamond) leaderboard, marking a significant milestone in frontier AI benchmark performance.
LangChain introduces an Engine that learns how your agent works by reading traces and repositories, then tests for agent-specific weaknesses such as hallucinations and system prompt violations, returning a list of verified issues to fix before they impact users.
A reporter stress-tested GSA's new America.gov AI assistant with 15 research-grade questions from prior reporting, finding 9 correct answers, 6 incomplete, and no hallucinations, though the system missed available public data and lacked transparency about its sources. The finding suggests the tool refuses to fabricate when it lacks information but cannot establish what the government has not published.
PADMÉ reformulates meta-evaluation of LM agent evaluators as a preference alignment problem and introduces a data synthesis method using only small language models, boosting agreement with human judgment from 73% to 85% over a naive baseline while enabling evaluation of 25 common models across agentic domains.
This paper identifies a 'detectability gap' in hallucination detection: hallucinations split into high-agreement (Ghost) and low-agreement (Flickering) regimes with a 0.35–0.46 AUC gap, persisting across four models and three factual QA datasets even after freezing regime assignments and using stricter trajectory-based tests. The authors argue aggregate detection metrics hide model-dependent heterogeneity and call for regime-conditioned evaluation.
The paper introduces tau-Multilingual, a benchmark extending tau-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin for evaluating voice agents across task completion, interaction quality, and language generation. Results show Spanish, Portuguese, and Hindi perform close to English, while Korean and Mandarin drop significantly, and the authors release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.
BreakingWeb is a benchmark framework from Amazon and Duke that constructs challenging browser-use tasks via controlled, deterministic environment interventions on tasks agents already solve, cutting agent pass rates by 22.9% while humans lose only 10.0%, revealing widespread 'belief failure' where agents falsely declare success.
研究人员提出一个用于评测车载对话助手(ICA)多轮对话能力的自动化黑盒测试框架,采用策略引导的用户模拟器、对抗性策略管理器和两级 LLM 评审器,在工业级 ICA 上验证该方法与人类标注高度一致,并发现 2.96 倍的独特失败类型。
This arXiv paper evaluates how prompt perturbations affect bias and hallucination in LLMs used for decision-making tasks, finding that perturbations can mitigate these issues in some models, with Claude 3 outperforming GPT-3.5 across most datasets.
This paper introduces an exact generate–transform decomposition to analyze how small 7–9B LLM teams scale across eight orchestration architectures, finding that scaling returns are sharply task-dependent — Proposer-Critic wins on arithmetic benchmarks but no architecture dominates across all tasks.
This paper introduces a controlled evaluation framework that separates answer coverage from repeatable task specialization in LLM-generated harnesses on MATH-500, finding that additional coverage largely comes from repeated executions of the same program rather than genuine specialization gains.
This paper formalizes how collapsing LLM-judge rubric rewards to binary pass/fail hides criterion-level quality changes from the policy's reward signal, showing under a joint-Gaussian model that at least three grading levels are needed to remove this ambiguity, with experiments on MATH and SciBench confirming that binary stretches are invisible but detectable with three grades.
提出 TauIndianBankBench:一个包含 1,000 项印度零售银行任务的评测基准,不仅检查最终数据库状态,还要求工具使用型对话代理完成所有必要步骤且不做未请求的变更;结果显示仅按结果评分时有 6.5–12.0% 的会话在违反流程的情况下仍被计为成功。