@FinanceYF5: Much of the “common knowledge” about LLMs may have a shelf life of just 15 months. Stanford researchers used 20 new models to retest 50 widely cited findings: 44% no longer held, 34% had significantly weakened, and only 16% still held true. Fabricated citations, hallucinations, and stereotypes are declining, but sycophancy toward users and…
Summary
Stanford researchers retested 50 widely cited findings about LLMs using 20 new models and found that 44% no longer held, 34% had significantly weakened, and only 16% still held, suggesting that LLM research findings may have a “shelf life” of only about 15 months.
View Cached Full Text
Cached at: 11/03/26, 10:23 AM
Much of the “common knowledge” about LLMs may have a shelf life of just 15 months.
Stanford researchers used 20 new models to retest 50 widely cited findings:
44% no longer held, 34% had weakened significantly, and only 16% still held true.
Fabricated citations, hallucinations, and stereotypes are declining, but sycophancy toward users and left-leaning political views persist.
When studying AI, model versions and testing dates may be just as important as the findings themselves https://t.co/HnWPTxhdf8
Similar Articles
@FinanceYF5: 1/ The cost of intelligence is collapsing. The price of LLM intelligence has dropped 100-fold in 18 months; this is a fact. But David says pessimists stop there—they miss the second half: as inputs get cheaper, demand expands outward.
The article notes that the price of LLM intelligence has dropped 100-fold in 18 months, and argues that this cost reduction will drive demand to expand outward, countering purely pessimistic views.
The Problem with “Mathematically Proven” Claims About LLMs (15 minute read)
This article critiques the sensationalized media coverage of mathematical proofs regarding LLM limitations, specifically highlighting how conditional results about self-improvement are often misrepresented as universal impossibilities.
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
Introduces FinIndices, a large-scale benchmark evaluating LLM data-processing fidelity on uncropped financial statements, revealing knowledge and structural bottlenecks in financial reasoning.
Can LLMs predict the outcomes of machine learning experiments?
该论文构建了包含2882个LLM提出的ML实验想法的数据集，评估五个前沿大模型预测实验结果的准确性，发现最佳模型Claude Opus 5.5仅解释约一半方差（R²=0.49），且所有模型都系统性低估了变化幅度，表明当前LLM对ML想法的直觉有用但有限。
@FinanceYF5: Google new paper: Let LLM solve math competition problems, accuracy jumps from 10% to 70%. [LEAP framework] Instead of having the model write a complete proof at once, it breaks down the problem into a goal tree, learns step by step from Lean verifier feedback, and reuses proven lemmas. Result: All 12 problems of Putnam 2025 solved, IMO style…
Google new paper proposes the LEAP framework, which decomposes math problems into goal trees, learns from Lean verifier feedback, and improves LLM accuracy on math competition problems from 10% to 70%. It solves all 12 problems of Putnam 2025 and surpasses dedicated gold-medal-level systems on IMO-style benchmarks.