@GaryMarcus: Am old enough to remember when @GeoffreyHinton told me I was stupid for saying that LLMs regurgitate training data. He …
Summary
Gary Marcus highlights recent DeepMind research confirming that LLMs frequently memorize and regurgitate training data, countering past criticism from Geoffrey Hinton. The post underscores ongoing debates about LLM limitations and their real-world capabilities.
Similar Articles
@geoffreyhinton: I believe you said that they JUST (my caps) regurgitate training data. That IS stupid. Here is a quote from you: "It gl…
Geoffrey Hinton counters Gary Marcus's claim that language models merely regurgitate training data, citing Marcus's own words.
Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation
The study reveals substantial gaps in machine unlearning for LLMs, showing that adversarial evaluation uncovers recoverability of forgotten information despite strong standard metrics, highlighting the need for adversarial stress-testing.
LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs
PropMe is a propensity-aware framework for evaluating LLM memorization, distinguishing between forced reproduction capabilities and natural propensity using SimpleTrace for deterministic attribution across open models and datasets.
@oneill_c: https://x.com/oneill_c/status/2077453217609453784
A researcher discusses the challenge of continual learning in LLMs, comparing them to amnesiac interns, and explores approaches like extending context windows, building stateful memory, and compressing context into latent representations, citing their work on Still.
@rohanpaul_ai: New Illinois+ Tsinghua University and other labs study finds that LLM agents still have unreliable memory and that it c…
A study by University of Illinois and Tsinghua University finds that LLM agents' memory becomes unreliable when continuously rewritten, with performance dropping from 100% to 54% on ARC-AGI tasks. The paper proposes preserving raw episodes instead of always summarizing them.