reproducibility

A cookbook-style guide demonstrating how to build a NixOS system using Nixtamal, an input pinner that replaces stateful nix-channel workflows, and highlighting the new system.nix entry point introduced in NixOS 26.05.

This arXiv paper examines what it truly means for AI assistance to save time, arguing for a 'ledger view' that weighs time saved against errors that slip through rather than a simple stopwatch metric. It replicates METR's developer study showing estimand-dependent slowdowns, analyzes GDPval's review-time constraints on GPT-5 speedups, and runs a pre-specified experiment showing that over-trusting an AI drafter leads reviewers to let more errors through while appearing faster.

该论文提出一个可复现的离线评估框架，用于在生产环境的可变企业数据上评测对话智能体，并将其作为 GEPA 提示优化的适应度函数，在正确性约束下实现提示压缩 3.5-7.6% 而不损失任务准确率。

该论文重新评估了基于LLM的文本属性图节点分类方法，发现其相对简单基线（如Correct & Smooth、MLP）的报告优势往往很小，并记录了测试集调参、标签泄漏等评估风险。

A benchmark study on Terminal-Bench Lite finds that a bare IPython harness can match heavier production agent harnesses for Qwen3.5-9B, and that much of the run-to-run variance in small-model agents stems from noisy tool outputs rather than the model itself.

本文介绍并审计了一个针对 Bengali 幻觉检测挑战赛（Olikbochon）的推理系统，该系统结合响应条件化查询重写、混合检索与任务特化的语言模型裁决，实现了 0.956 的私有宏 F1，同时指出语料重叠与运行不可复现性限制了其泛化性声明。

This paper evaluates whether an LLM can serve as a calibration agent for expensive engineering simulators, finding that physics-labeled prompts cut early calibration error 22-fold but lose their advantage after about ten runs, so a hybrid warm-start plus numerical optimizer workflow reaches target accuracy in 9 runs versus 17 for Nelder-Mead.

该论文提出一种作者参与的智能体协作复现性审计流程，将作者声明作为AI agent调查与评估的显式输入，并在某机器学习会议的36篇论文、133个实验表格上部署，人工复核后提出20个问题并全部闭环解决。

该论文研究了在医疗设备合规文档评估中使用大语言模型时的可复现性问题，发现14个商业LLM部署在确定性采样设置下仍产生不一致的评估结果，并构建了一个从OCR到推理全链路确定性的流水线，实现跨机器、跨天的逐字节一致输出。

一个 Apache-2.0 开源桌面应用 Open-Science 试图让 AI 辅助科研像带校验和与版本控制的代码一样可追溯：AI 代理阅读论文、调用科学数据库、运行 Python/R 代码，并将输入数据、代码与运行历史一并保存为可分享的 .science 文件，实现本地优先、模型无关的可复现研究。

Open-Science 是 AIPOCH 推出的开源、本地优先 AI 科研工作台，结合智能体工作流、Python 与 R 执行以及可追溯的研究产物，所有结果都保留输入、代码与执行证据，便于复现与对比工作流。

jinx 是一个将 Nix 编译为 TypeScript 的转译器，能够在浏览器中评估完整的 nixpkgs（231,777 个属性）并生成与 Nix 完全一致的 drvPath，从而让前端开发者在不安装 Nix 的情况下探索存储模型。

A practitioner reports that Nvidia's ICML spotlight paper DreamDojo, built on Cosmos 2.5, contains multiple severe code bugs affecting pre-training, post-training and evaluation, raising doubts about its reported marginal gains despite massive compute and 44k hours of human data.

该论文系统研究了文本分类器在不同服务上下文（批量大小、数值精度、硬件等）下的非不变性，发现标签稳定并不意味着分数稳定，且在 bf16 下仅改变批量形状就能导致高达 56.7 个百分点的预测概率质量偏移，并为每种机制推导了标签稳定性的充分条件和缓解方法。

Centific Research introduces a Trust Layer for Agent Evaluation, a post-hoc framework that verifies benchmark scores via four deterministic checks—grading support, traceable computation, honest completion claims, and run-to-run stability. Applied to five agent configurations on 108 Agents' Last Exam tasks, only 22.6% of recorded passes clear all checks, revealing widespread fabricated computation, false success claims, and score instability.

SPLATIFY 是一个多智能体框架，可将 3D Gaussian Splatting 研究论文转化为可训练的 gsplat 代码，支持复现、发现与创新，将实现时间从数周缩短至数分钟，并提出覆盖 30 篇论文的 SPLATIFY-Bench 评测基准。

A discussion on how unreliable LLM memory benchmarks can be, noting that scores swing wildly depending on the evaluator model (e.g., 51.4 under token-overlap F1 vs. 75.8 under an LLM judge), the reader model, and run counts — while many results are vendor-reported with unclear noise levels.

ArrivalBench introduces an evaluation method that re-executes agent-generated data pipelines under adversarial replay schedules (late, duplicated, out-of-order records) and finds that 7-79% of pipelines certified by single-execution benchmarks are silently wrong when re-run.

Sat researcher Mate Soos argues that academic incentives have drifted away from scientific integrity, citing cases where authors knowingly kept papers with incorrect evaluations rather than retracting them, and warning about a growing indifference toward correctness among PhD students.

A commentary on an Anthropic guest essay describing BootLoops, tools in which Claude-generated calculations were technically correct but only became scientifically meaningful after domain experts redirected the questions. The piece argues that arithmetic verification and relevance/novelty judgment require separate expert reviews before AI output is called a discovery, while noting the essay's author is an Anthropic visiting researcher rather than an independent benchmark.