Failure to Reproduce Modern Paper Claims [D]
Summary
A researcher reports failure to reproduce claims from modern papers, with 4 out of 7 checked claims being irreproducible and 2 having unresolved GitHub issues, raising concerns about research quality standards.
Similar Articles
It's time to desk reject papers that don't include code that can reproduce the results [D]
The author, after reviewing for three major conferences, argues that papers without code to reproduce results should be desk rejected, citing that only 1 of 12 papers reviewed provided full code and 7 provided none.
Reproducibility seems to be headed towards irrelevance in ML research. Is it too late? [D]
An opinion piece arguing that reproducibility in machine learning research is becoming a lost cause due to the rise of physical AI requiring expensive hardware, unverifiable performance claims from big tech companies, and competitive incentives that discourage authors from sharing code.
Do Methods Support the Claims? Intra-Paper Verification for Peer Review
This paper introduces intra-paper claim verification, a framework that uses LLMs to evaluate whether novelty claims in a paper are supported by its methodological evidence, addressing a gap in existing automated peer review systems. Human evaluation shows significant alignment with human reviewer concerns, especially for novelty-related issues.
6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts, a Six-Stage Audit Framework and First Instantiation
The paper presents a six-stage audit framework for assessing reproducibility in neuro-symbolic AI literature, finding only 6.5% of studies with published artifacts can be reproduced, highlighting a crisis in research reproducibility.
AAAI 2027 Review: No code submission? [D]
A reviewer for AAAI 2027 expresses surprise at the low number of paper submissions with code, despite the conference's emphasis on reproducibility, and asks for opinions on whether lack of code should affect review scores.