benchmark

The paper introduces EDB-Bio, a 96k-episode benchmark for scientific evidence deliberation across three biological domains, showing that frontier and open-weight models are systematically underconfidence-prone (Brier scores near a trivial base-rate forecaster) and that post-training with proper scoring rules improves calibration, yielding an Ibis0 model 2.5× more efficient than the best frontier model in sequential experiment selection.

用户分享观点，认为 Claude Opus 5.5 的 High 模式优于 XHigh 模式。

SemiAnalysis is releasing GDPx, a frontier-level benchmark for measuring agentic capabilities across the most economically valuable domains, where even the best current model scores only 12% (1/7).

DR Pixel is an open-source end-to-end multimodal deep research agent that retrieves and reasons directly over rendered document page images instead of OCR-extracted text, introducing counterfactual credit reweighting for agentic supervised finetuning. The approach outperforms OCR-based baselines on DRPixelBench across models like GPT-5.4 and Gemini-3.5-Flash, boosting retrieval recall by ~15 points and accuracy by 10+ points after training.

humans& releases &Tutor, an AI model trained to teach by training against Persimmon as a simulated learner, achieving state-of-the-art results on the TutorMoments benchmark and outperforming frontier models like Claude Sonnet 5.5 and GPT-6 Astra under plain prompting.

Moralitybench.ai has been updated with results from 20 AI models answering 56 standardized moral psychology questions five times each, including new Trolley Problem variants, plus two DIT-2 tests pending.

A new Meta and NYU paper shows that h1, a simple agent loop giving models fast Lean feedback and search tools, matches or beats hand-designed theorem-proving workflows: it outperformed Goedel-Architect on PutnamBench (598 vs 597 problems) at lower cost, and with Opus 4.8 solved 83/100 hard algebra problems versus 63 for Claude Code.

The author tested Liquid AI's LFM3-1.7B model (Q8_0 GGUF via llama.cpp/ROCm) for compression and summarization, reporting strong prefill/decode throughput on Strix Halo and tensor-split 7900 XTX+XT setups, plus unexpectedly coherent performance at 32k-64k context lengths for a 1.7B model.

IBM researchers introduced STAIR, a retriever that leverages document tables of contents for structured retrieval, achieving 82.6% top-1 section accuracy on the SearchTome benchmark and 65× fewer invalid section references than a fine-tuned DSI baseline.

Nolan Lawson runs a benchmark asking AI agents to build the same Pokémon app with different JavaScript frameworks, measuring dev time, tokens, code size, and Lighthouse scores. The post pushes back on vibes-based framework claims in the agentic coding era and asks for evidence.

CrossSWE introduces a multi-repository benchmark of 171 real-world instances showing that current LLM coding agents struggle with cross-repository consistency, with the strongest model (GPT-6.1 Sol) reaching only 58.48% Pass@1 and cross-repository inconsistencies accounting for 44.2% of failures.

分析显示 Grok 4.20 在 PokeBench 的 Pokémon Red 基准中卡在房子附近长达数百回合，尽管其自身推理日志正确识别出陷入循环却仍无法脱困，引发关于视觉与规划能力瓶颈的讨论。

The paper introduces Tutorial Agentic Learning (TAL), a nonparametric continual-learning paradigm that lets LLM agents learn from instructional videos by building a structured tutorialbook memory and reasoning-linked table-of-contents retrieval, plus a VidLearn chess-video QA benchmark where TAL significantly outperforms vanilla MLLMs.

Memdoor 是一个轻量级编码与自动化代理工具，以单一静态二进制 CLI 形式提供，支持本地运行或通过 SSH 连接远程机器，并内置九种 GitHub Actions 工作流；据称其由 5 个 Qwen3.8-27B 模型组成的团队在 LiveCodeBench Hard 基准上超越了 Fable 5。

GPT-5.6 Sol, running in Codex with low thinking, autonomously completed a peaceful-mode Factorio run in 13.5 in-game hours, launching a rocket via a Python game-manager library that mirrors human GUI constraints, marking a clear improvement over previous GPT-5.5 attempts.

LlamaIndex released BooBench, a playful 179-page Halloween-themed benchmark that 'scares' LLMs and document OCR tools into repeating characters, exposing failure modes where VLMs enter repetition loops on real-world documents with repeated elements like fill leaders and redaction marks.

A post shares the Groot family of MoE models on Hugging Face (19B, 65B, 238B, and 280B variants), claiming the 19B model outperforms Qwen3.6-35B-A3B and Gemma-4-26B-A4B while giving shorter answers; an edit notes they appear to be REAP-derived reworks of DeepSeek/Qwen models.

A practical, evidence-based guide to GPT-6 Astra's real-world use cases, mapping its performance across 12 Scale AI SEAL benchmark categories to project recommendations, strengths, limitations, and cases where alternative models like Claude Fable 5.1 are preferable.

The author measures how lossy inference-engine flags (routing bias and tail skip) degrade model quality, showing that shipped defaults cost ~1.5 accuracy points with plenty of VRAM but ~3x more on small cards, while aggressive settings produce confident nonsense.

A lawyer and PLI Labs founder publishes full benchmark data for running DeepSeek-V4.1-Flash-1017 (2.75 bpw EXL3) across two DGX Sparks with a 3090 box handling RDMA-served tables and a speculative drafter, hitting 2,600+ tok/s prefill to 256K context and 322 tok/s structured output. All code, launch scripts, and raw JSON are released publicly, with the author openly noting where his build loses to competing 2x Spark setups.