Tag
论文提出 Investor's Last Exam (ILE)，一个由买方投资者主导的股票研究基准，评估AI智能体在冻结信息集下的研究质量、预测准确度与模拟投资表现，结果显示最强模型（GPT-6 Astra、Claude Opus 5.5）研究能力强但预测表现仍显著不足。
A paper introducing MIOE, a causal-ablation benchmark that uses interchange interventions to evaluate whether LLMs (Qwen3-8B/4B) mechanistically solve ontology tasks like entity typing and subsumption prediction, finding distributed subspace methods work best but don't yet fully disentangle ontological knowledge from auxiliary class information.
论文提出 Bottle-SPU（Bottle-SKU）大规模多SKU洗发水瓶表面缺陷检测基准，包含13,119张图像与20,419个实例级标注框，并提出原型引导的动态域调制方法 ProDMM，通过SKU视觉原型生成低秩Query/Value调制以提升跨产品变体的缺陷检测泛化能力。
论文提出 DecompGym 基准，包含来自九类问题族的 81 个实例，用于评估 12 个 GPT、Claude、Gemini 编码智能体设计与实现 Benders 分解和列生成求解器的能力，并比较了独立重生成与反思两种失败修复策略的效果。
SkillCT 提出一个多维度智能体技能评估基准，从静态规范、动态行为稳健性与人工辅助三个层面覆盖九个评估维度，并提出无需执行的质量评估基线，可降低评测成本并提升下游任务成功率。
The paper critically examines CLadder, a popular benchmark for evaluating causal reasoning in LLMs, revealing that it embeds superficial, highly predictive patterns. Simple models like random forests perform well on its generated datasets, and commercial LLMs' performance is brittle, suggesting their strong results come from exploiting shortcuts rather than genuine causal reasoning.
Researchers from Los Alamos National Laboratory and NVIDIA present an expert-curated benchmark of 250 questions on inertial confinement fusion to evaluate LLMs' scientific fluency, physical reasoning, and quantitative analysis. They compare frontier open/closed models with Prospero, a domain-specialized model, finding that specialization improves scores but critical physics errors persist, and recommend assessing existing capabilities and retrieval benefits before investing in further specialization.
The paper studies LLMs as numerical optimizers, finding they can capture problem structure and produce near-optimal solutions, but reliability degrades with problem dimension—improved via iterative refinement—motivating hybrid LLM plus deterministic solver approaches.
SciGame is a controlled benchmark that converts scientific literature into structured cards with Original and Counterfactual environments to evaluate LLM agents on scientific law discovery. Experiments on five frontier models show that prior knowledge can hurt performance, adaptive experimentation helps only stronger models, and good explanations can mask quantitative failures.
FilingLoop introduces an interactive benchmark and self-evolving LLM agent framework for corporate disclosure review and revision, grounded in 380 SEC IPO review tasks with requirement-level evaluation that improves fulfillment from 24.28% to 27.96% via procedural learning without weight updates.
The paper introduces LibStructBench, a document-grounded benchmark evaluating AI agents' ability to reconstruct sequencing library structures and molecular workflows, finding that frontier agents remain unreliable at exact reconstruction and that model choice matters more than agent harness choice.
Astra reportedly outperforms all other models on robot intelligence benchmarks, marking a notable advance in embodied AI capabilities.
Article comparing Alibaba's Qwen 2 7B model against OpenAI's GPT-6 in terms of capabilities and performance.
Voice Arena launched Turing TTS, a benchmark that measures how often listeners mistake AI speech for human speech via a 'Fool Rate'; even the best model, Gemini 3.8 Flash TTS, fools listeners only about 10% of the time.
A new benchmark called Turing TTS measures how often human listeners mistake AI-generated voices for real human speech, with Gemini 3.8 Flash TTS currently the best at fooling people in A/B voice identification tests.
Introducing Turing TTS, an evaluation measuring how often human listeners mistake AI-generated speech for real human speech; even the best TTS models fooled listeners only ~10% of the time, prompting discussion on better metrics for voice AI progress.
The author built 5dive, an open-source server-side browser runtime for AI agents, and benchmarked it against OpenClaw, Hermes, and Claude in Chrome on 100 live website tasks from Online-Mind2Web plus 4 controlled tasks — finding similar pass rates among the top setups but major differences in speed and cost, with 5dive (pi) being the cheapest at $0.65 total.
VictoriaMetrics published an open-source benchmark comparing structured log ingestion performance and internals of VictoriaLogs, Loki, Elasticsearch, OpenSearch, SigNoz and ClickStack on identical single-node hardware, highlighting trade-offs in ingestion speed, query latency and disk usage.
SwarmChat team benchmarked 1–16 coding agents on ProgramBench, finding that parallel agent teams trade ~36% more tokens for ~5 hours of speedup, but don't consistently beat solo agents given enough time.
Artificial Analysis 更新了 HLE Diamond 基准评测，显示开源权重模型在该高难度基准上落后闭源模型超过 30 分。