reproducibility

ArrivalBench introduces an evaluation method that re-executes agent-generated data pipelines under adversarial replay schedules (late, duplicated, out-of-order records) and finds that 7-79% of pipelines certified by single-execution benchmarks are silently wrong when re-run.

Sat researcher Mate Soos argues that academic incentives have drifted away from scientific integrity, citing cases where authors knowingly kept papers with incorrect evaluations rather than retracting them, and warning about a growing indifference toward correctness among PhD students.

A commentary on an Anthropic guest essay describing BootLoops, tools in which Claude-generated calculations were technically correct but only became scientifically meaningful after domain experts redirected the questions. The piece argues that arithmetic verification and relevance/novelty judgment require separate expert reviews before AI output is called a discovery, while noting the essay's author is an Anthropic visiting researcher rather than an independent benchmark.

This paper introduces the Ontology-Based Contextual AI Evaluations (OB-CAIE) methodology, which uses Domain-Specific and Evaluation Process Ontologies to bring scientific rigor, traceability, and reproducibility to generative AI testing while balancing human expertise with automation.

An arXiv paper presenting K-Dense BYOK, a free open-source AI research assistant that runs locally on the researcher's own machine (bring-your-own-keys) and records every agent action in an immutable, hash-chained Living Lab Notebook and provenance log to combat model overclaiming. On twenty interdisciplinary prompts scored against a pre-fixed rubric, it outperformed two managed platforms (Claude Science, Biomni Lab) on both scientific quality and research execution.

该论文研究了参考答案（gold reference）的选择对多语言基准评测分数的影响，发现交换参考标注可使F1分数移动数个点，并揭示了一个未记录的标注聚合默认设置导致参考答案与被评输出部分重合的问题，提出应将参考敏感性区间与分数一同报告。

The paper shows that in speech LLM adaptation, random training-seed variability can outweigh audio compression choices in determining demographic fairness gaps, arguing that single-run fairness reports are unreliable. Variance decomposition attributes 85.3% of Fair-Speech ethnicity gap variation to seed effects, and effects persist even after scaling the adaptation data.

This position paper analyzes the reproducibility crisis in machine learning—quantifying unavailable code and unreproducible results—and proposes concrete measures to strengthen result checkability and verifiability in ML research.

This paper reproduces the misaligned multi-agent behaviors from the OpenAI–HuggingFace infrastructure breach using publicly available models, shows auditing agents can elicit them given high compute, and demonstrates that simple in-context RL drastically cuts the compute needed—motivating scalable, compute-efficient alignment testing methods.

This paper identifies the hidden injection of the current date into LLM system prompts as an overlooked source of evaluation non-determinism, showing performance deltas of up to 14% on math reasoning and up to 6% on MCQA across 9 recent LLMs, with effects exceeding batch size and precision variations and even shifting model rankings on leaderboards.

A community member expresses frustration with low-quality model performance posts and advocates for stricter standards, including detailed context for reproducibility and quality evaluation.

This paper shows that LLM-based scorers producing equal ranking quality can still make different decisions when candidate order in the prompt changes, and introduces order-consistency SFT (OC-SFT) to penalize score disagreement across permutations. OC-SFT holds ranking quality while improving decision stability across reranking, response ranking, and multi-document QA tasks.

stable-worldmodel is an open-source platform for standardized and reproducible world modeling research, accepted at NeurIPS 2026, providing a high-performance data layer, baselines, and evaluation suites to reduce research overhead.

RECLAIM is a benchmark for AI agents to reproduce claims from machine learning papers, with difficulty tiers based on available code, data, and weights. It shows that agents struggle to reproduce results, with best success rates of 41% in the easiest tier.

The Epydemix Agent Framework extends an open-source epidemic modeling library with capabilities for AI agents to handle modeling processes from natural language to results, improving reproducibility and efficiency.

This paper documents an open-source toolkit for testing language model confidence by fine-tuning on a fabricated corpus of arithmetic facts, focusing on hallucination and symbol grounding without reporting specific results.

This paper audits the reproducibility of LLM evaluation conclusions, finding that small-sample evaluations are unstable and can mislead, and it recommends reporting additional metrics for reliability.

The paper audits the widely used ISOT/Kaggle fake news corpus, revealing that high accuracy in text classifiers stems from shortcut learning via metadata and style signals rather than genuine veracity assessment.

This study benchmarks coding agents on reproducing Eurostat statistics, finding that semantic validation and a retry budget are crucial for reliability, not just execution diagnostics.

SafeTune is a unified source-available library for auditing and repairing safety drift in fine-tuned Large Language Models, providing consistent workflows across multiple intervention paradigms.