How much published AI research is wrong because of data leakage?

Reddit r/artificial News
data-leakage ai-research reproducibility machine-learning research-integrity methodology

Summary

A Princeton study found data leakage in nearly 300 AI papers across 17 fields, causing overoptimistic results. The author highlights how easy it is to accidentally leak data and cautions against trusting impressive AI claims without checking for leakage.

There is a Princeton paper by Kapoor and Narayanan. They found data leakage in close to 300 papers across 17 fields, including medicine and economics. Leakage means the model was trained on information it would never have when it makes a real prediction. So it looks great on the test set and then fails in the real world. My favorite example is civil war prediction. Complex models were reported to crush old logistic regression. Once the leakage was fixed, the fancy models were no better than the decades old stats. I have built enough models to know how easy this is to do by accident. You scale the data before you split it, or you use one feature that is really a stand in for the answer, and your numbers look amazing. So now when I read another "AI cracked X" headline, my first thought is whether anyone checked it for leakage.
Original Article

Similar Articles

The real AI risk is inside the labs (5 minute read)

TLDR AI

The author argues that the primary AI risk comes from leaks inside frontier labs, not from open-weight models, and calls for international safety oversight and balanced consideration of progress.

AI is confidently wrong way more than people give it credit for, change my mind

Reddit r/ArtificialInteligence

A user shares concerns about AI models presenting thin or ambiguous data with the same confidence as well-supported findings, citing a case where a complaint appearing only twice in 200 comments was ranked as a top concern. The piece questions whether this is a fixable prompting issue or a fundamental limitation requiring manual verification.

The most important AI failure may be false confidence, not wrong answers

Reddit r/ArtificialInteligence

This article argues that the most dangerous AI failures stem not from wrong answers but from systems acting with false confidence based on incomplete data, outdated context, or bad assumptions, suggesting that AI evaluation should prioritize handling uncertainty over raw intelligence.

AI research tools are still too eager to turn public signals into certainty

Reddit r/artificial

The author critiques AI research tools for overconfidence in weak signals, praising Komo AI's rapid discovery and source-attached summaries but highlighting the need for better uncertainty and contradiction handling. They describe a workflow that splits discovery, verification, and structured checking across multiple AI tools.