benchmarking

Tag

Cards List
#benchmarking

RSI Arena: Towards Recursive Self-Improvement with Human Feedback

arXiv cs.CL ↗ · 2026-10-27 Cached

RSI Arena introduces a two-stage study where ten frontier AI agents autonomously post-train language models from a shared base under fixed GPU and API budgets, guided by human preference feedback. The strongest agent (GPT-6 Astra) closed 76% of the base-to-Instruct gap, and most models improved in human preference from stage 1 to stage 2, demonstrating a step toward recursive self-improvement with human feedback (RSIHF).

0 favorites 0 likes
#benchmarking

Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis

arXiv cs.AI ↗ · 2026-10-27 Cached

This paper audits evaluation artifacts and data leakage in augmented LLM systems for rare disease diagnosis, showing that candidate-space contamination and temporal misalignment artificially inflate benchmark performance, and proposes a rigorous data-isolation evaluation framework to ensure genuine clinical robustness.

0 favorites 0 likes
#benchmarking

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

arXiv cs.CL ↗ · 2026-10-27 Cached

This paper audits a cheap frozen 'System One' classification model against 14 frontier LLMs on content moderation, finding it accuracy-competitive at far lower cost, and shows that 'confidence' conflates two distinct uncertainties—human disagreement and decision risk—that require opposite escalation signals.

0 favorites 0 likes
#benchmarking

Benchmarking System One Models for Time-Series Decision Making: A Case Study of Jev

arXiv cs.AI ↗ · 2026-10-27 Cached

该论文提出一个统一评估框架,使用 Jev 1.13.0 作为案例研究 Structured System 1 模型在时间序列分类、事件类型预测、油温方向及股价走势预测等任务上的表现,结果显示其在显式时间模式和事件预测上较强,但在许多数值序列分类和股票走势预测上接近随机水平。

0 favorites 0 likes
#benchmarking

The Social World Model Evaluation Gap

arXiv cs.CL ↗ · 2026-10-27 Cached

该论文审计了 1999 至 2026 年间的 113 项研究,发现只有 9 项主要评测真正测量了真实人类对系统所选行动的响应,其中仅 2 项还将显式的响应预测与实测响应进行比较;在 25 项语言模型相关工作的全文检查中没有发现符合这些条件的实验。作者提出了一套直接预测评测标准,并设计了一个用于谈判助手的单步实验来区分任务成功、当前状态推断与真实响应预测。

0 favorites 0 likes
#benchmarking

Controlled Evaluation of EEG Foundation Models: Random Initialisation, Band-Power and Validation-Selected Probes

arXiv cs.LG ↗ · 2026-10-27 Cached

A controlled evaluation of five public EEG foundation models plus an in-house JEPA-style model, showing that random-initialisation, band-power baselines, and validation-selected probe layers substantially alter reported pretraining gains on sleep, emotion, and motor imagery benchmarks.

0 favorites 0 likes
#benchmarking

Beyond Local Accuracy: Benchmarking the Rollout Reliability of PDE Foundation Models

arXiv cs.LG ↗ · 2026-10-27 Cached

This paper benchmarks the rollout reliability of seven PDE foundation models across seven 2D time-dependent PDE datasets, showing that while fine-tuned foundation models often beat scratch-trained baselines at short horizons, full-horizon rollouts suffer substantial error amplification (median 2.04x), arguing for horizon-resolved reporting in future benchmarks.

0 favorites 0 likes
#benchmarking

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

arXiv cs.AI ↗ · 2026-10-26

论文提出 TraceTrial-Bench,从七个维度评估 AI Agent 的 harness 可靠性,并通过轨迹追踪与失败归因模块,在 4 个 harness、9 个模型和 95 个任务上完成 3,420 次运行的可审计评测。

0 favorites 0 likes
#benchmarking

RSIHub: A Composable Framework for Agent-Evolution Methods

arXiv cs.AI ↗ · 2026-10-26

提出了 RSIHub,一个用于构建、比较和扩展 agent-evolution 方法的开源框架,通过可复用的算子与配方简化实验基础设施。论文在 Terminal Bench 2 和 τ³ Banking 上评测了 AHE、Hyperagents、A-Evolve 与 GEPA,最大 pass@1 提升达 13.5 个百分点,但方法排名随设置变化且优化收益不一定迁移到留出任务。

0 favorites 0 likes
#benchmarking

Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

arXiv cs.CL ↗ · 2026-10-26

该论文通过六语言析因设计,独立考察问题语言、回答语言与验证指令语言对多语言 P(True) 自评估的影响,发现三者分别影响准确率、区分度与校准,且未经验证的评判者会严重扭曲模型表现差距。

0 favorites 0 likes
#benchmarking

Shorter Is Not Cheaper: A Controlled Comparison of Declarative Animation Specifications and Free-Form Code for LLM-Generated Explanations

arXiv cs.CL ↗ · 2026-10-26

研究通过预注册的受控实验比较了声明式动画规格与自由HTML/JavaScript代码两种LLM生成解释的方案,发现规格虽更短,但由于推理开销增加并未更便宜、更快或更可靠,小模型的规格可靠性甚至显著更低。

0 favorites 0 likes
#benchmarking

Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks

Hugging Face Daily Papers ↗ · 2026-10-26 Cached

Independent safety evaluation of DeepSeek V4.1 Flash-1017 across 34 benchmarks finds substantial safety improvements over prior versions and near-parity with leading closed-source models, recommending process-level observability, incremental safety testing, and private benchmarks to address loss-of-control and harmful manipulation risks.

0 favorites 0 likes
#benchmarking

Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities

Hugging Face Daily Papers ↗ · 2026-10-24 Cached

A meta-study analyzing hallucination detection as a benchmark-conditioned measurement problem across provenance, typology, and granularity axes, proposing a Span Coverage F1 metric to address variability in span-level annotation boundaries.

0 favorites 0 likes
#benchmarking

State of ARM Support for Puget Bench

Hacker News Top ↗ · 2026-10-23 Cached

Puget Systems outlines the current state of ARM support for its Puget Bench benchmarking tools on Snapdragon and NVIDIA RTX Spark devices, noting the benchmark suite is fully ARM-native but host apps like Lightroom Classic still lack native ARM versions.

0 favorites 0 likes
#benchmarking

RSIHub: A Composable Framework for Agent-Evolution Methods

Papers with Code Trending ↗ · 2026-10-23 Cached

RSIHub 是一个开源框架,为智能体进化(agent-evolution)研究提供可组合的共享基础设施,标准化方法的构建、比较与扩展,并在 Terminal Bench 2 和 τ3 Banking 上评测了 AHE、Hyperagents、A-Evolve 和 GEPA 等方法。

0 favorites 0 likes
#benchmarking

More 4x Intel B65 test results - Performance improvements - ~30% faster prefill and TTFT in flash-next vllm by increasing chunk size. & MPT benchmarks - and a video showing me actually using it in LibreChat.

Reddit r/LocalLLaMA ↗ · 2026-10-22

A hobbyist shares test results showing ~30% faster prefill and TTFT in vLLM Flash-Next by increasing the chunk size to 8192 on a 4x Intel B65 build, with detailed Qwen3.8-Flash-Next benchmark tables and a LibreChat demo.

0 favorites 0 likes
#benchmarking

Asking What a Director Sees Can Change What Vision-Language Models Choose

arXiv cs.CL ↗ · 2026-10-22 Cached

This paper shows that asking vision-language models which blocks a director can see before acting significantly changes their choices in the Director Task, raising hidden-block selections by 12-88 percentage points across GPT models and 17 points on Qwen3.8-27B. The authors argue perspective-taking benchmarks need paired baselines that differ only by the report question to avoid misreading failures of perspective use.

0 favorites 0 likes
#benchmarking

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

arXiv cs.CL ↗ · 2026-10-22 Cached

The paper introduces SkillEval, a framework that combines LLM-discovered interpretable skills with cognitive diagnosis models to produce fine-grained ability profiles of LLMs across benchmarks, improving on average-score evaluation and enabling LLM routing and adaptive testing.

0 favorites 0 likes
#benchmarking

Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters

Hugging Face Daily Papers ↗ · 2026-10-22 Cached

This paper studies how small an encoder can be for closed-set decisions like tool selection, finding that per-option scoring beats joint scoring and that 68M/150M parameter joint encoders outperform a 435M DeBERTa-v3 zero-shot classifier while answering requests in milliseconds.

0 favorites 0 likes
#benchmarking

KFlash: Benchmarks of newly released versions of Apple Silicon optimized engines: 1.4.0 Splash (DFlash2 ~ 50 tok/s) vs 0.2 Kapitsa (KFlash ~65 tok/s), MLX(MTP) & more on Qwen 3.8 27b/4b/35b 4Q on Macbook Pro M4 Pro 48 GB 2024, 48GB & Macbook Pro M1 Pro 16GB 2021

Reddit r/LocalLLaMA ↗ · 2026-10-21

Nace.AI's Kapitsa 0.2 ships KFlash, a best-first tree-based speculative decoding method for Apple Silicon, benchmarked against Splash 1.4.0's DFlash2, mlx-serve, Ollama, llama.cpp and others on Qwen models across M4 Pro and M1 Pro MacBooks, showing top decode throughput (~65-201 tok/s).

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback