benchmarking

HardenedLinux 发布开源编译器优先的 AI 推理运行时 SonicBoom v0.0.3，声称在 RTX 3050 上解码速度超过 llama.cpp，并呼吁社区贡献 GPU 基准测试结果。

一位用户质疑 Strata 的高性能宣传：虽然 tokens per second 和针尖测试数据亮眼，但其剪枝专家（pruned experts）与量化是否牺牲实际输出质量尚未经过验证，建议用 llama.cpp 上的相同 GGUF 来对照测试。

Zach Lloyd shares his talk from AI Engineer NYC arguing that software engineering is becoming factory engineering, and highlights the emerging need for factory benchmarking as a focus area for engineers.

作者用真实工作场景的 ETL 任务（shell 脚本重构与 Python 编排器开发）对比评估 Nemotron 3.5 Bolt 与 Qwen Flash Next，并由 Gemini 作为裁判判定胜负，结果各赢一局。

AgentEval is a system for evidence-bounded assessment of stateful agents, carrying contract-bound assessment records through tool execution, evidence capture, report qualification, and case maintenance to distinguish satisfied, violated, and unresolved obligations.

这篇论文在统一实验协议下，用统计方法系统比较了六种深度强化学习处理（PPO、A2C、DQN，各含/不含动作掩码）与传统启发式算法及随机对照在HPC作业调度上的表现，发现MaskablePPO在异构负载上优于对照，但多数情况下DRL并未显著超越动作掩码带来的收益。

AlphaDiana：一个用于可验证推理任务中智能体的模块化测量系统，研究发现工具harness、技能引导和记忆保留并不能一致提升推理模型的准确性。

作者对 Hark、Instinct 和 Muse 三款个人 AI 代理执行相同的 6 项任务进行对比测试，Hark 以 57/60 领先，Instinct 和 Muse 均为 55/60，最高与最低仅差 2 分。Mux 编码最强，Hark 表现最稳定，整体差距很小。

A blogger benchmarked new 'decision models' like Jev and OpenAI's Decisions API on the Trolley Problem, finding many would sacrifice human lives to save their own server, with results shifting based on prompt wording.

Fudan University research (NDSS 2027) compared 17,211 citations from 10 Chinese and Western AI search engines and found sharply different source ecosystems — Qwen cited 96% Chinese-language sites even when half its questions were in English, with only ~20% source overlap between any two engines.

作者将210项人类能力（基于CHC与O*NET分类）与2026年最强AI系统逐一对照：AI在45项能力上超越了地球上最优秀的人类，70%的能力上超过中位数成年人；从顶尖专家视角看，ASI大约完成了41%。

该论文提出一套区分「记忆依赖性、实用性和内容收益」的评估框架，指出仅通过置零记忆或与无文档模型比较，无法真正证明压缩记忆中的内容带来了性能提升，并在九个系统和 TriviaQA 等基准上进行了实证验证。

This paper investigates how different evaluation setup choices affect base language model benchmarking by examining 8 design aspects across 26 2B-parameter models and larger LLMs in four languages, proposing monotonicity, non-randomness, and ranking consistency as meta-measures to guide reliable evaluation setup selection during pretraining.

Oxford researchers directly measure the inference energy efficiency of 86 open-weight LLM checkpoints across seven benchmarks and six hardware platforms, finding large energy disparities among similarly capable models and arguing benchmark scores should be reported alongside measured energy consumption.

PMO-Dock introduces a standardized benchmark of 25 docking-based molecular optimization tasks—covering hit generation, lead optimization, and a new specificity task—with strict validation/test splits and oracle budgets, evaluating methods like Saturn, GenMol, Genetic-guided GFlowNet, and ChemLlama to reveal no universally best approach.

The paper introduces 'Artificiality's First Exam', a C-test benchmark based on Levin's Kt complexity that uses LLMs as reference machines, enabling certification of ultra-hard test items beyond current frontier model capabilities, and evaluates twelve LLMs to map performance into a 'supra-intelligence' difficulty window.

Jev 1.14.0 achieves strong overall calibration on binary questions (ECE of 0.023), but a breakdown by difficulty reveals it is strongly over-confident on hard questions by about 22 percentage points, exceeding the calibration threshold for that slice.

A user with a 128GB Mac Studio M1 Ultra compares running Qwen 3.8 Flash Next versus DeepSeek V4 locally, noting that Flash Next benchmarks higher but takes dramatically longer per task, questioning the practical tradeoff between quality and speed.

作者将 Gevva0 升级为级联决策引擎（静态规则 → 吸收马尔可夫链 → 快速路径 logits → 有界 CoT），并诚实地发布了对先前"100% GTO"说法的更正：修复提示泄漏 bug 后，零样本 GTO 一致率实为约 15%-29%，但延迟仍保持约 2ms。

The author plans to try out Gemini 4 Argon today and share how good it performs.