Tag
A comparative benchmark study evaluating ten LLMs on 70 academic prompts across four complexity levels, measuring hallucination and response deviation. Findings highlight a trade-off between response completeness and factual reliability, with Perplexity and DeepSeek V3 scoring highest on accuracy.
This paper presents a controlled comparison of single-agent versus multi-agent LLM setups on a financial document review task, finding that splitting work across three agents catches slightly more defects but increases false alarms and uses 2.2-2.4x the tokens, with simple self-sampling of the single agent often achieving better balanced accuracy.
This paper formalizes benchmark evaluation as an empirical measure over question difficulty, identifying failure modes like measure mismatch and coverage gaps, and proposes diagnostic metrics plus BenchFill, a difficulty-targeted item rewriting algorithm, with analysis across 25 popular benchmarks in 7 domains.
HyperHat's founder shares results from a benchmark of 60 engineers directing AI coding agents, finding that specification quality, mechanism verification, and review behavior—not prompting tricks—separate strong engineers, and that most people miss partial agent failures.
An analysis of 674 AI tool vendors' terms of service found that 315 restrict benchmarking or competitive use, with 107 naming benchmarking explicitly and many copying Oracle's 1982 no-public-benchmark clause nearly verbatim. The author questions why AI vendors, including LangSmith and Coinbase, would forbid publishing performance tests.
Agent4Sci 通过跨 11 个领域的 20 个科学 ML 基准评估前沿编码智能体,发现它们主要靠工程技巧而非领域洞察提升性能,而轻量插件技能可显著改善表现(Claude Opus 从 10.4% 提升至 20.1%)。
Hugging Face policy chief Irene Solaiman said at TechCrunch Disrupt that embedded AI evaluators and benchmarking remain an immature, hard-to-standardize science, stressing reproducibility, open weights, and the growing need to evaluate agents and harnesses, not just models.
OpenAI announced model advances that significantly improve coding in low-resource programming languages like COBOL, Fortran, and Visual Basic, introducing MultiLangBench and a COBOL migration evaluation where GPT-6 Astra and GPT-6.1 Sol show major accuracy and cost gains over GPT-5.6 Sol.
We present AuditKIT, a source-available evaluation library that unifies benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation with direction-aware metric contracts and extensible backends. Seven case studies show how answer extraction, runtime behavior, and metric selection can change evaluation outcomes for LLMs and agentic systems.
HardenedLinux 发布开源编译器优先的 AI 推理运行时 SonicBoom v0.0.3,声称在 RTX 3050 上解码速度超过 llama.cpp,并呼吁社区贡献 GPU 基准测试结果。
一位用户质疑 Strata 的高性能宣传:虽然 tokens per second 和针尖测试数据亮眼,但其剪枝专家(pruned experts)与量化是否牺牲实际输出质量尚未经过验证,建议用 llama.cpp 上的相同 GGUF 来对照测试。
Zach Lloyd shares his talk from AI Engineer NYC arguing that software engineering is becoming factory engineering, and highlights the emerging need for factory benchmarking as a focus area for engineers.
作者用真实工作场景的 ETL 任务(shell 脚本重构与 Python 编排器开发)对比评估 Nemotron 3.5 Bolt 与 Qwen Flash Next,并由 Gemini 作为裁判判定胜负,结果各赢一局。
AgentEval is a system for evidence-bounded assessment of stateful agents, carrying contract-bound assessment records through tool execution, evidence capture, report qualification, and case maintenance to distinguish satisfied, violated, and unresolved obligations.
这篇论文在统一实验协议下,用统计方法系统比较了六种深度强化学习处理(PPO、A2C、DQN,各含/不含动作掩码)与传统启发式算法及随机对照在HPC作业调度上的表现,发现MaskablePPO在异构负载上优于对照,但多数情况下DRL并未显著超越动作掩码带来的收益。
AlphaDiana:一个用于可验证推理任务中智能体的模块化测量系统,研究发现工具harness、技能引导和记忆保留并不能一致提升推理模型的准确性。
作者对 Hark、Instinct 和 Muse 三款个人 AI 代理执行相同的 6 项任务进行对比测试,Hark 以 57/60 领先,Instinct 和 Muse 均为 55/60,最高与最低仅差 2 分。Mux 编码最强,Hark 表现最稳定,整体差距很小。
A blogger benchmarked new 'decision models' like Jev and OpenAI's Decisions API on the Trolley Problem, finding many would sacrifice human lives to save their own server, with results shifting based on prompt wording.
Fudan University research (NDSS 2027) compared 17,211 citations from 10 Chinese and Western AI search engines and found sharply different source ecosystems — Qwen cited 96% Chinese-language sites even when half its questions were in English, with only ~20% source overlap between any two engines.
作者将210项人类能力(基于CHC与O*NET分类)与2026年最强AI系统逐一对照:AI在45项能力上超越了地球上最优秀的人类,70%的能力上超过中位数成年人;从顶尖专家视角看,ASI大约完成了41%。