evaluation

Microsoft's ThinkingBox, a sandbox for evaluating AI agents on stateful business workflows by grading terminal database state rather than agent messages, is now available as a Hugging Face OpenEnv environment alongside the 507-task ThinkingBox-Bench benchmark.

Safe AI lab director Ding Zhao's startup Safeworld emerges from stealth with over $12 million in seed funding to build third-party safety evaluation for GenAI-driven robots, using simulation environments populated with realistic human models to test robots against edge cases before deployment.

Introduces the Multilingual Distractor Interference (MDI) protocol showing that foreign-language distractor sentences cause LLMs to switch output scripts (e.g., Devanagari) or trigger abstention, inflating exact-match hallucination metrics — Llama-3.1-8B's apparent 0.710 hallucination rate drops to 0.470 once script-switched but semantically correct answers are accounted for.

This arXiv paper studies how false training data can influence a language model's factual answers and downstream decisions without any activation trigger, identifying an audit gap between direct probing of factual recall and the model's actual behavior, including a real-world case of bushfire arson misinformation on Facebook.

This paper evaluates how Vision Language Models such as ChatGPT, Claude, Gemini, and LLaVA handle VQA questions that violate Grice's maxims through non-essential, ambiguous, or false modifiers, showing diminished performance and differences in pragmatic reasoning compared to humans.

该论文从心理测量学视角评估 LLM-as-a-judge，使用 Many-Facet Rasch 模型将评分分解为潜在质量、评分者严苛度与残差难度，发现人类与 LLM 在汇总对齐之外仍存在明显的残差难度结构错配。

TPBench introduces a benchmark that isolates 'turning-point eviction' in dialogue compression by probing separately for the user's initial goal (P1), the current value of a revised slot (P2), and both together (P3), using existing MultiWOZ and SGD dialogue-state annotations. The paper finds that probe-specific evaluations rank compression methods differently, and that deleting update-carrying turns sharply degrades current-value accuracy.

The paper analyzes six commercial dynamic LLM routers across 14 settings and finds none outperforms a router that randomly selects between two well-chosen models at matched cost, attributing failures to difficulty blindness, length reversal, semantic matching, and roster suboptimality — patterns the standard evaluation objective actually rewards.

This paper argues that time series forecasting benchmarks need scenario-grounded stress testing, where test cases include semantic scenarios, explicit failure operators, and measurable difficulty levels to capture structured anomalies like regime shifts and sensor failures that i.i.d. perturbation-based robustness tests miss.

The paper introduces Open-Endedness Bench (OEB), a benchmark-agnostic methodology that evaluates AI agents' epistemic process by compiling execution logs into epistemic event graphs and scoring how well claims are supported by executed experiments, finding that only 16-29% of claimed agent improvements are real.

MIT 的 Neil Thompson 提出 O*NET-BENCH 审计套件，评测 33 种 LLM judge 配置，发现排序一致性虽高但接受率估计偏差极大（3.0%–97.9% vs 工人实际的 61.1%），表明仅凭排序一致性选择 judge 不足以支撑职业层面的 AI 测量。

This paper presents a paired, self-audited evaluation of two System-1 decision models (open-weight Laya and hosted Jev) across 11 agent-harness decision points, finding Jev significantly more accurate on 9 of 11 tasks while auditing several analysis errors that had distorted prior deployment claims.

The author theorizes that new AI model baselines may be so high that humans can't distinguish between them without benchmarks, raising questions about whether benchmark-maxing matters most for private AI lab valuation.

A practitioner experiment found that when a single model writes both the code and the tests, the generated test suites falsely rejected a known-correct reference solution 77% of the time, even with high mutation scores, and the full agent pipeline was outperformed by simple pass@k sampling at similar token cost.

A practitioner reflects on AI agent evaluation beyond binary success/failure rates, arguing that a '70% success rate' metric hides root causes like wrong tool choices or bad early workflow decisions. They ask the community what debugging information they find most useful — final outputs, tool calls, execution traces, or something else.

A self-run experiment compares a six-model 1-2B local swarm (modeled on a 1930s Soviet design institute's discipline-based org chart) against a single model on a document extraction task, finding the real win came from splitting inputs into blocks rather than agent collaboration (+0.05 from the swarm, with strict emergence at zero).

A GP trainee built a medical consultation game (doctorfoo.ai) and benchmarked 13 AI models across 195 consultations: all models correctly diagnosed every case, but safety behavior (catching red flags like contraindicated prescriptions) varied widely, with GPT-6, GPT-6.1 Sol, and Claude Opus 5.5 performing best and price barely predicting quality.

本文研究 agentic RAG 评测中评估预算在问题数、搜索轨迹和重复读取之间的分配策略，通过 HotpotQA 和 MuSiQue 上的检索-反馈对比，发现更广的问题覆盖在固定 token 预算下能显著降低标准误差，并提出了预算分配的预测模型。

UndoBench introduces a benchmark that decouples task competence from fault-recovery capability in tool-using AI agents across 36 workflows and 36 fault scenarios in 8 enterprise domains, finding nominal task completion of 83.54% masks conditional recovery success of just 46.72%, with naive retry producing duplicate side effects in 53.33% of trials.

Microsoft and Hugging Face introduce ThinkingBox, a benchmark that grades AI agents on terminal database state and side effects across 507 stateful business workflows run 20 times each, now available through Hugging Face and OpenEnv.