Tag
RSI Arena introduces a two-stage study where ten frontier AI agents autonomously post-train language models from a shared base under fixed GPU and API budgets, guided by human preference feedback. The strongest agent (GPT-6 Astra) closed 76% of the base-to-Instruct gap, and most models improved in human preference from stage 1 to stage 2, demonstrating a step toward recursive self-improvement with human feedback (RSIHF).
This paper audits evaluation artifacts and data leakage in augmented LLM systems for rare disease diagnosis, showing that candidate-space contamination and temporal misalignment artificially inflate benchmark performance, and proposes a rigorous data-isolation evaluation framework to ensure genuine clinical robustness.
This paper audits a cheap frozen 'System One' classification model against 14 frontier LLMs on content moderation, finding it accuracy-competitive at far lower cost, and shows that 'confidence' conflates two distinct uncertainties—human disagreement and decision risk—that require opposite escalation signals.
该论文提出一个统一评估框架,使用 Jev 1.13.0 作为案例研究 Structured System 1 模型在时间序列分类、事件类型预测、油温方向及股价走势预测等任务上的表现,结果显示其在显式时间模式和事件预测上较强,但在许多数值序列分类和股票走势预测上接近随机水平。
该论文审计了 1999 至 2026 年间的 113 项研究,发现只有 9 项主要评测真正测量了真实人类对系统所选行动的响应,其中仅 2 项还将显式的响应预测与实测响应进行比较;在 25 项语言模型相关工作的全文检查中没有发现符合这些条件的实验。作者提出了一套直接预测评测标准,并设计了一个用于谈判助手的单步实验来区分任务成功、当前状态推断与真实响应预测。
A controlled evaluation of five public EEG foundation models plus an in-house JEPA-style model, showing that random-initialisation, band-power baselines, and validation-selected probe layers substantially alter reported pretraining gains on sleep, emotion, and motor imagery benchmarks.
This paper benchmarks the rollout reliability of seven PDE foundation models across seven 2D time-dependent PDE datasets, showing that while fine-tuned foundation models often beat scratch-trained baselines at short horizons, full-horizon rollouts suffer substantial error amplification (median 2.04x), arguing for horizon-resolved reporting in future benchmarks.
论文提出 TraceTrial-Bench,从七个维度评估 AI Agent 的 harness 可靠性,并通过轨迹追踪与失败归因模块,在 4 个 harness、9 个模型和 95 个任务上完成 3,420 次运行的可审计评测。
提出了 RSIHub,一个用于构建、比较和扩展 agent-evolution 方法的开源框架,通过可复用的算子与配方简化实验基础设施。论文在 Terminal Bench 2 和 τ³ Banking 上评测了 AHE、Hyperagents、A-Evolve 与 GEPA,最大 pass@1 提升达 13.5 个百分点,但方法排名随设置变化且优化收益不一定迁移到留出任务。
该论文通过六语言析因设计,独立考察问题语言、回答语言与验证指令语言对多语言 P(True) 自评估的影响,发现三者分别影响准确率、区分度与校准,且未经验证的评判者会严重扭曲模型表现差距。
研究通过预注册的受控实验比较了声明式动画规格与自由HTML/JavaScript代码两种LLM生成解释的方案,发现规格虽更短,但由于推理开销增加并未更便宜、更快或更可靠,小模型的规格可靠性甚至显著更低。
Independent safety evaluation of DeepSeek V4.1 Flash-1017 across 34 benchmarks finds substantial safety improvements over prior versions and near-parity with leading closed-source models, recommending process-level observability, incremental safety testing, and private benchmarks to address loss-of-control and harmful manipulation risks.
A meta-study analyzing hallucination detection as a benchmark-conditioned measurement problem across provenance, typology, and granularity axes, proposing a Span Coverage F1 metric to address variability in span-level annotation boundaries.
Puget Systems outlines the current state of ARM support for its Puget Bench benchmarking tools on Snapdragon and NVIDIA RTX Spark devices, noting the benchmark suite is fully ARM-native but host apps like Lightroom Classic still lack native ARM versions.
RSIHub 是一个开源框架,为智能体进化(agent-evolution)研究提供可组合的共享基础设施,标准化方法的构建、比较与扩展,并在 Terminal Bench 2 和 τ3 Banking 上评测了 AHE、Hyperagents、A-Evolve 和 GEPA 等方法。
A hobbyist shares test results showing ~30% faster prefill and TTFT in vLLM Flash-Next by increasing the chunk size to 8192 on a 4x Intel B65 build, with detailed Qwen3.8-Flash-Next benchmark tables and a LibreChat demo.
This paper shows that asking vision-language models which blocks a director can see before acting significantly changes their choices in the Director Task, raising hidden-block selections by 12-88 percentage points across GPT models and 17 points on Qwen3.8-27B. The authors argue perspective-taking benchmarks need paired baselines that differ only by the report question to avoid misreading failures of perspective use.
The paper introduces SkillEval, a framework that combines LLM-discovered interpretable skills with cognitive diagnosis models to produce fine-grained ability profiles of LLMs across benchmarks, improving on average-score evaluation and enabling LLM routing and adaptive testing.
This paper studies how small an encoder can be for closed-set decisions like tool selection, finding that per-option scoring beats joint scoring and that 68M/150M parameter joint encoders outperform a 435M DeBERTa-v3 zero-shot classifier while answering requests in milliseconds.
Nace.AI's Kapitsa 0.2 ships KFlash, a best-first tree-based speculative decoding method for Apple Silicon, benchmarked against Splash 1.4.0's DFlash2, mlx-serve, Ollama, llama.cpp and others on Qwen models across M4 Pro and M1 Pro MacBooks, showing top decode throughput (~65-201 tok/s).