benchmarking

该论文通过六语言析因设计，独立考察问题语言、回答语言与验证指令语言对多语言 P(True) 自评估的影响，发现三者分别影响准确率、区分度与校准，且未经验证的评判者会严重扭曲模型表现差距。

研究通过预注册的受控实验比较了声明式动画规格与自由HTML/JavaScript代码两种LLM生成解释的方案，发现规格虽更短，但由于推理开销增加并未更便宜、更快或更可靠，小模型的规格可靠性甚至显著更低。

Independent safety evaluation of DeepSeek V4.1 Flash-1017 across 34 benchmarks finds substantial safety improvements over prior versions and near-parity with leading closed-source models, recommending process-level observability, incremental safety testing, and private benchmarks to address loss-of-control and harmful manipulation risks.

A meta-study analyzing hallucination detection as a benchmark-conditioned measurement problem across provenance, typology, and granularity axes, proposing a Span Coverage F1 metric to address variability in span-level annotation boundaries.

Puget Systems outlines the current state of ARM support for its Puget Bench benchmarking tools on Snapdragon and NVIDIA RTX Spark devices, noting the benchmark suite is fully ARM-native but host apps like Lightroom Classic still lack native ARM versions.

RSIHub 是一个开源框架，为智能体进化（agent-evolution）研究提供可组合的共享基础设施，标准化方法的构建、比较与扩展，并在 Terminal Bench 2 和 τ3 Banking 上评测了 AHE、Hyperagents、A-Evolve 和 GEPA 等方法。

A hobbyist shares test results showing ~30% faster prefill and TTFT in vLLM Flash-Next by increasing the chunk size to 8192 on a 4x Intel B65 build, with detailed Qwen3.8-Flash-Next benchmark tables and a LibreChat demo.

This paper shows that asking vision-language models which blocks a director can see before acting significantly changes their choices in the Director Task, raising hidden-block selections by 12-88 percentage points across GPT models and 17 points on Qwen3.8-27B. The authors argue perspective-taking benchmarks need paired baselines that differ only by the report question to avoid misreading failures of perspective use.

The paper introduces SkillEval, a framework that combines LLM-discovered interpretable skills with cognitive diagnosis models to produce fine-grained ability profiles of LLMs across benchmarks, improving on average-score evaluation and enabling LLM routing and adaptive testing.

This paper studies how small an encoder can be for closed-set decisions like tool selection, finding that per-option scoring beats joint scoring and that 68M/150M parameter joint encoders outperform a 435M DeBERTa-v3 zero-shot classifier while answering requests in milliseconds.

Nace.AI's Kapitsa 0.2 ships KFlash, a best-first tree-based speculative decoding method for Apple Silicon, benchmarked against Splash 1.4.0's DFlash2, mlx-serve, Ollama, llama.cpp and others on Qwen models across M4 Pro and M1 Pro MacBooks, showing top decode throughput (~65-201 tok/s).

A comparative benchmark study evaluating ten LLMs on 70 academic prompts across four complexity levels, measuring hallucination and response deviation. Findings highlight a trade-off between response completeness and factual reliability, with Perplexity and DeepSeek V3 scoring highest on accuracy.

This paper presents a controlled comparison of single-agent versus multi-agent LLM setups on a financial document review task, finding that splitting work across three agents catches slightly more defects but increases false alarms and uses 2.2-2.4x the tokens, with simple self-sampling of the single agent often achieving better balanced accuracy.

This paper formalizes benchmark evaluation as an empirical measure over question difficulty, identifying failure modes like measure mismatch and coverage gaps, and proposes diagnostic metrics plus BenchFill, a difficulty-targeted item rewriting algorithm, with analysis across 25 popular benchmarks in 7 domains.

HyperHat's founder shares results from a benchmark of 60 engineers directing AI coding agents, finding that specification quality, mechanism verification, and review behavior—not prompting tricks—separate strong engineers, and that most people miss partial agent failures.

An analysis of 674 AI tool vendors' terms of service found that 315 restrict benchmarking or competitive use, with 107 naming benchmarking explicitly and many copying Oracle's 1982 no-public-benchmark clause nearly verbatim. The author questions why AI vendors, including LangSmith and Coinbase, would forbid publishing performance tests.

Agent4Sci 通过跨 11 个领域的 20 个科学 ML 基准评估前沿编码智能体，发现它们主要靠工程技巧而非领域洞察提升性能，而轻量插件技能可显著改善表现（Claude Opus 从 10.4% 提升至 20.1%）。

Hugging Face policy chief Irene Solaiman said at TechCrunch Disrupt that embedded AI evaluators and benchmarking remain an immature, hard-to-standardize science, stressing reproducibility, open weights, and the growing need to evaluate agents and harnesses, not just models.

OpenAI announced model advances that significantly improve coding in low-resource programming languages like COBOL, Fortran, and Visual Basic, introducing MultiLangBench and a COBOL migration evaluation where GPT-6 Astra and GPT-6.1 Sol show major accuracy and cost gains over GPT-5.6 Sol.

We present AuditKIT, a source-available evaluation library that unifies benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation with direction-aware metric contracts and extensible backends. Seven case studies show how answer extraction, runtime behavior, and metric selection can change evaluation outcomes for LLMs and agentic systems.