Tag
Ethan Mollick suggests that AI-generated analyses should be accompanied by multiverse-style reporting and full disclosure of prompts to enhance reproducibility in science.
Prior Labs releases open-source tools including RelArena-α, TabPFN-Rel, and RPI to advance reproducible relational learning, providing a unified benchmark framework and a TabPFN-based model.
This paper presents a reconstructed benchmark analyzing federated aggregation methods under model poisoning and backdoor attacks, finding Trimmed Mean most accurate in clean settings and Krum most robust under attacks, while auditing metric implementations and noting reproducibility caveats.
Hugging Face shares findings from a community hackathon where 1,200 participants used coding agents to reproduce 2,226 ICML papers, highlighting issues in review rigor and the potential for AI agents to scale verification.
A reviewer for AAAI 2027 expresses surprise at the low number of paper submissions with code, despite the conference's emphasis on reproducibility, and asks for opinions on whether lack of code should affect review scores.
A reproducibility study of Khatri et al.'s latent-space safety probes, testing generalization across model families and sensitivity to non-determinism. Results show the probes extend to other models with similar F1 scores, and final token latent vectors remain consistent across seeds.
The author conducts an independent spin audit of IBM's SQD/QSCI iron-sulfur cluster benchmarks, finding that the quantum results converge to high-spin states rather than the target singlet. Using IBM's own archived hardware data, they show the shipped mitigation fails to fix the spin problem, and a Claude-assisted review is provided.
JudgeArena is an open-source framework that unifies major LLM-judge benchmarks under a single interface, enabling systematic study of judge choices and offering open-model judges that match or outperform closed models, with the ability to simulate LMArena Elo scores.
A technical deep-dive into how Nix's sandbox paths act as a hidden input to derivations, breaking reproducibility by allowing the same derivation to produce different outputs depending on sandbox configuration.
The author, after reviewing for three major conferences, argues that papers without code to reproduce results should be desk rejected, citing that only 1 of 12 papers reviewed provided full code and 7 provided none.
This paper examines how evaluation protocols affect reported accuracy in EEG emotion recognition, using a DGCNN on SEED and SEED-IV datasets. It demonstrates that subject-dependent, subject-disjoint, and cross-session evaluations answer different questions, and that checkpoint selection and test-set reuse can inflate accuracy.
ECG-InterpBench is a new benchmark that systematically evaluates the interpretability of ECG foundation model representations using matched-scale sparse autoencoders, covering reconstruction fidelity, clinical concept accessibility, and reproducibility across 450 cells.
A position paper arguing that autonomous AI agents in science widen the verification gap and that scientific verification infrastructure must evolve with observable-by-default workflows, scalable verification, and clear attribution to sustain trustworthy science.
This paper performs a forensic reproducibility audit of a radiology vision-language model benchmark, finding divergences between the intended protocol and released artifacts that invalidate the original claims. The authors propose a benchmark contract to expose such failure classes.
A blog post discussing how Nix helps solve dependency and reproducibility issues in building systems software, particularly for fast-evolving subsystems like BPF and io_uring.
This paper presents a distributed benchmark study on scaling laws for classical machine learning models on tabular data, showing that power-law fits hold for most model families and quantifying replicator-implementation variance across 127 student runs.
STeMP is a standardized protocol for reporting and guiding spatio-temporal predictive machine-learning modelling in environmental research, aimed at improving transparency and reproducibility. It is accompanied by an R package and web application, and hosted on GitHub.
A developer shares lessons learned when making an AI that analyzes trading charts give consistent answers. The fix involves setting temperature to 0 and using a fixed seed across all pipeline stages, and emphasizes that determinism is a trust feature that must be paired with grounding against ground truth.
This paper investigates how format, instruction count, and context length affect instruction adherence and hallucination in LLMs through controlled experiments on a synthetic corpus, finding that instruction-following collapses beyond 80 rules regardless of format, and recall degrades sharply after 64-128k tokens with format-dependent effects. It releases the VeyraBench harness for reproduction.
Introduces PsiLogic, a chaos-aware optimizer that augments Adam with a dynamic damping term based on gradient instability, and proposes FairBench for reproducible evaluation. Shows competitive or superior results on NLP, ViT, and ResNet tasks with full transparency on limitations.