benchmarking

Tag

Cards List
#benchmarking

Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models

arXiv cs.CL ↗ · 2026-10-21 Cached

A comparative benchmark study evaluating ten LLMs on 70 academic prompts across four complexity levels, measuring hallucination and response deviation. Findings highlight a trade-off between response completeness and factual reliability, with Perplexity and DeepSeek V3 scoring highest on accuracy.

0 favorites 0 likes
#benchmarking

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

arXiv cs.AI ↗ · 2026-10-21 Cached

This paper presents a controlled comparison of single-agent versus multi-agent LLM setups on a financial document review task, finding that splitting work across three agents catches slightly more defects but increases false alarms and uses 2.2-2.4x the tokens, with simple self-sampling of the single agent often achieving better balanced accuracy.

0 favorites 0 likes
#benchmarking

Your Benchmark Is an Empirical Measure Over Difficulty

Hugging Face Daily Papers ↗ · 2026-10-21 Cached

This paper formalizes benchmark evaluation as an empirical measure over question difficulty, identifying failure modes like measure mismatch and coverage gaps, and proposes diagnostic metrics plus BenchFill, a difficulty-targeted item rewriting algorithm, with analysis across 25 popular benchmarks in 7 domains.

0 favorites 0 likes
#benchmarking

We benchmarked how 60 engineers direct AI coding agents. Here is what separated the strong ones

Reddit r/AI_Agents ↗ · 2026-10-20

HyperHat's founder shares results from a benchmark of 60 engineers directing AI coding agents, finding that specification quality, mechanism verification, and review behavior—not prompting tricks—separate strong engineers, and that most people miss partial agent failures.

0 favorites 0 likes
#benchmarking

Almost half the AI tool vendors I checked forbid benchmarking

Reddit r/AI_Agents ↗ · 2026-10-20

An analysis of 674 AI tool vendors' terms of service found that 315 restrict benchmarking or competitive use, with 107 naming benchmarking explicitly and many copying Oracle's 1982 no-public-benchmark clause nearly verbatim. The author questions why AI vendors, including LangSmith and Coinbase, would forbid publishing performance tests.

0 favorites 0 likes
#benchmarking

Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

Hugging Face Daily Papers ↗ · 2026-10-20 Cached

Agent4Sci 通过跨 11 个领域的 20 个科学 ML 基准评估前沿编码智能体,发现它们主要靠工程技巧而非领域洞察提升性能,而轻量插件技能可显著改善表现(Claude Opus 从 10.4% 提升至 20.1%)。

0 favorites 0 likes
#benchmarking

Embedded AI evaluators won’t be an easy solution, Hugging Face’s policy chief says

TechCrunch AI ↗ · 2026-10-19 Cached

Hugging Face policy chief Irene Solaiman said at TechCrunch Disrupt that embedded AI evaluators and benchmarking remain an immature, hard-to-standardize science, stressing reproducibility, open weights, and the growing need to evaluate agents and harnesses, not just models.

0 favorites 0 likes
#benchmarking

@OpenAI: Our latest model advances bring stronger capabilities for building and migrating software in low-resource programming l…

X AI KOLs ↗ · 2026-10-19 Cached

OpenAI announced model advances that significantly improve coding in low-resource programming languages like COBOL, Fortran, and Visual Basic, introducing MultiLangBench and a COBOL migration evaluation where GPT-6 Astra and GPT-6.1 Sol show major accuracy and cost gains over GPT-5.6 Sol.

0 favorites 0 likes
#benchmarking

AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

arXiv cs.CL ↗ · 2026-10-19 Cached

We present AuditKIT, a source-available evaluation library that unifies benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation with direction-aware metric contracts and extensible backends. Seven case studies show how answer extraction, runtime behavior, and metric selection can change evaluation outcomes for LLMs and agentic systems.

0 favorites 0 likes
#benchmarking

Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decode

Reddit r/LocalLLaMA ↗ · 2026-10-19

HardenedLinux 发布开源编译器优先的 AI 推理运行时 SonicBoom v0.0.3,声称在 RTX 3050 上解码速度超过 llama.cpp,并呼吁社区贡献 GPU 基准测试结果。

0 favorites 0 likes
#benchmarking

Strata + Qwen 3.8 Flash Next: fast, but what's it costing in quality?

Reddit r/LocalLLaMA ↗ · 2026-10-17

一位用户质疑 Strata 的高性能宣传:虽然 tokens per second 和针尖测试数据亮眼,但其剪枝专家(pruned experts)与量化是否牺牲实际输出质量尚未经过验证,建议用 llama.cpp 上的相同 GGUF 来对照测试。

0 favorites 0 likes
#benchmarking

@zachlloydtweets: Here's my take on what a software factory really is This is my talk from @aiDotEngineer NYC, and I decided to record a …

X AI KOLs Timeline ↗ · 2026-10-16 Cached

Zach Lloyd shares his talk from AI Engineer NYC arguing that software engineering is becoming factory engineering, and highlights the emerging need for factory benchmarking as a focus area for engineers.

0 favorites 0 likes
#benchmarking

Nemotron 3.5 Bolt Evaluation using real-world tasks

Reddit r/LocalLLaMA ↗ · 2026-10-16

作者用真实工作场景的 ETL 任务(shell 脚本重构与 Python 编排器开发)对比评估 Nemotron 3.5 Bolt 与 Qwen Flash Next,并由 Gemini 作为裁判判定胜负,结果各赢一局。

0 favorites 0 likes
#benchmarking

AgentEval: Evidence-Bounded Assessment of Stateful Agents

arXiv cs.AI ↗ · 2026-10-16 Cached

AgentEval is a system for evidence-bounded assessment of stateful agents, carrying contract-bound assessment records through tool execution, evidence capture, report qualification, and case maintenance to distinguish satisfied, violated, and unresolved obligations.

0 favorites 0 likes
#benchmarking

Deep Reinforcement Learning for HPC Job Scheduling: A Statistical Evaluation

arXiv cs.AI ↗ · 2026-10-16 Cached

这篇论文在统一实验协议下,用统计方法系统比较了六种深度强化学习处理(PPO、A2C、DQN,各含/不含动作掩码)与传统启发式算法及随机对照在HPC作业调度上的表现,发现MaskablePPO在异构负载上优于对照,但多数情况下DRL并未显著超越动作掩码带来的收益。

0 favorites 0 likes
#benchmarking

AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks

arXiv cs.AI ↗ · 2026-10-16 Cached

AlphaDiana:一个用于可验证推理任务中智能体的模块化测量系统,研究发现工具harness、技能引导和记忆保留并不能一致提升推理模型的准确性。

0 favorites 0 likes
#benchmarking

I ran the same 6 tasks on Hark, Instinct and Muse. 2 points separated first and last

Reddit r/AI_Agents ↗ · 2026-10-16

作者对 Hark、Instinct 和 Muse 三款个人 AI 代理执行相同的 6 项任务进行对比测试,Hark 以 57/60 领先,Instinct 和 Muse 均为 55/60,最高与最低仅差 2 分。Mux 编码最强,Hark 表现最稳定,整体差距很小。

0 favorites 0 likes
#benchmarking

Jev Meets the Trolley Problem

Hacker News Top ↗ · 2026-10-15 Cached

A blogger benchmarked new 'decision models' like Jev and OpenAI's Decisions API on the Trolley Problem, finding many would sacrifice human lives to save their own server, with results shifting based on prompt wording.

0 favorites 0 likes
#benchmarking

Western and Chinese AI search engines cite largely separate webs. Qwen's sources were 96% Chinese even with half the questions in English

Reddit r/artificial ↗ · 2026-10-15

Fudan University research (NDSS 2027) compared 17,211 citations from 10 Chinese and Western AI search engines and found sharply different source ecosystems — Qwen cited 96% Chinese-language sites even when half its questions were in English, with only ~20% source overlap between any two engines.

0 favorites 0 likes
#benchmarking

ASI loading: Today, I mapped 210 human abilities against the best AI systems in 2026. AI beats the best humans alive on 45/210 of them.

Reddit r/ArtificialInteligence ↗ · 2026-10-15

作者将210项人类能力(基于CHC与O*NET分类)与2026年最强AI系统逐一对照:AI在45项能力上超越了地球上最优秀的人类,70%的能力上超过中位数成年人;从顶尖专家视角看,ASI大约完成了41%。

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback