benchmarking

A practitioner shares their deterministic evaluation harness for QA/coding agents — fixed corpora, injected bugs, hidden tests, and pass^k/token metrics — and asks the community how others balance smoke tests vs full benchmarks, whether LLM-as-a-judge adds value, and whether tools like Promptfoo or DeepEval reduce maintenance burden.

Nicholas Carlini demonstrates that a modern Stockfish port running on a 1996-era 200 MHz Pentium Pro achieves ~3300 Elo, far surpassing Kasparov's ~2800, showing that most of the +1335 Elo gain since 1996 comes from better programming (+720) rather than faster hardware — meaning Deep Blue's custom supercomputer was never necessary.

该论文对开源 5G RAN 平台 OpenAirInterface (OAI) 和 srsRAN 进行了对比评估，涵盖特性、部署灵活性以及基于视频点播、直播和云游戏等真实应用负载的性能分析。

HyperHat's founder shares a design sketch for benchmarking whether AI agents can identify contradictory task requirements and ask clarifying questions rather than blindly executing, pairing broken tasks with satisfiable controls to prevent trivial gaming of the eval.

The author benchmarks several System 1 decision endpoints (tev1, TypeSafe Jev, pplx-decider-v1.2, liquid/d1.1) against a local 9B LLM for multi-agent routing and tool gating, showing 40-question parallel 'Blind Semantic Surmise' cuts latency to 159-319 ms with 100% accuracy and zero LLM fallbacks.

The tweet introduces an evaluation paper that probes 10 vision foundation models across 4 levels, shifting the evaluation focus from single-frame image understanding to spatial states maintained across observations. It offers useful insights for researchers in embodied AI and VLA.

This paper shows that using incomplete relevance judgments as a shortcut for scoring abstention in conversational RAG benchmarks systematically distorts evaluation: stronger retrievers yield more 'false miss' contexts that still contain answers, badly deflating answerability AUROC. The authors release code, runs, and labels and recommend scoring refusals against content-based labels instead of gold-document presence.

论文提出 TSFLab，一个自我改进、与具体智能体解耦的时间序列 forecasting 研究工作台，为编码智能体提供共享资源与工具，以低成本完成复现、基准测试与新方法开发，并显著提升任务完成率、降低 token 与开发时间成本。

The paper explores using large language models (Claude Opus 5) to automatically construct simple baselines for 36 deep learning tasks from Nature/Science/Cell-family studies, finding these baselines recover a median 95.5% of the performance gap and match or exceed the published deep learning model in 12 studies, suggesting automated baseline checking as a scalable safeguard for scientific peer review.

Kapitsa 0.2.26, an open-source Apple Silicon inference engine, publishes M5 Max benchmarks for Qwen, Nemotron, Flint and Clover models, featuring its KFlash best-first tree verification and SSD offloading that runs a 180B MoE at 61 tok/s on a 48GB MacBook Pro.

The author benchmarks Speedy against Avalon and Strata for running a Qwen3.8 model locally, finding Speedy dramatically faster: 4x faster reading, generation tok/s around 105-110, and 26-31% shorter total wall-clock time than Avalon.

A user benchmarked LLM inference on rented vast.ai servers with dual AMD EPYC 7642 (ROME) boards, large DRAM, and RTX 3090s, finding that RAM-heavy server builds deliver near-linear scaling up to 8 concurrent streams — hitting 94 tok/s at c=1 and 456 tok/s at c=8 on Qwen 3.8 Flash Next Unsloth Q8.

Building multi-agent workflows is now trivial, so the real competitive advantage has shifted from execution to programmatic evaluation with systematic benchmarks against ground truth. Without a structured eval pipeline, teams are effectively shipping randomized output generators with no domain intuition to catch hallucinations or logic drift.

The authors announce Gear, an open-source, meta-learning-inspired framework described as "a scikit-learn for RSI," which separates search decisions from execution and evaluation so methods like GEPA, RRSI, and Harness² are easier to implement and compare. A working GEPA-style variant with shared failure diagnosis and staged evaluation showed pass-rate gains on AutomationBench and Terminal-Bench, plus a ~95% lower per-task API cost for GPT-5.6 Luna versus Astra using the same optimized harness.

The paper introduces Seq@K, an evaluation protocol measuring whether a language model can solve a problem within K sequential attempts given feedback after each failure, showing it reveals a materially different model-selection landscape than Pass@K across 7 benchmarks and 16 frontier models.

WebAnything presents a fully automated framework that reconstructs source-grounded, reusable web agent environments from real URLs, building 244 website artifacts supporting 12,984 tasks and showing that fine-tuning Qwen3.5-9B on its interaction trajectories improves web agent success rates across multiple benchmarks.

A detailed benchmark shows QEngine 1.3.1 (compiled for RTX 5090s) is the fastest Qwen3.8-Flash-Next engine for single-stream decode, ~1.4-1.9x faster than Strata and ~3x faster than tuned vLLM, with a KV-in-RAM mode that dramatically improves warm returns to idle conversations.

ISTA disputes Unsloth's published quantization comparisons for DeepSeek-V4.1-Flash-1017, arguing the performance gap comes from a 32K vs 128K reasoning-token budget rather than any hidden hyperparameter tuning, and invites independent reproduction of their benchmarks.

The article benchmarks Go 1.28's new HTTP/2 'parking' mechanism, which lets idle connection-serving goroutines exit and be recreated on demand, showing 84.7–92.6 MiB (about 39–41%) RSS savings at 5,000 idle connections compared to the same binary with parking disabled.

Unsloth 发布 DeepSeek-V4.1-Flash-1017 Dynamic V3.1 (preview) GGUF 量化模型，宣称在 7 项基准上精度保持优于 ISTA 的 GSQ-RCO GGUFs，并公开了无法复现对方 AIME25 100% 精度恢复声明的对比结果。