@BetaTomorrow: Title: A Bitter Lesson for Data Filtering Authors : Christopher Mohri , John Duchi, Tatsunori Hashimoto (@tatsu_hashimo…

X AI KOLs Following Papers
data-filtering large-language-models generalization scaling-laws curation manifold-learning

Summary

This paper argues that for large enough models, unfiltered data can improve generalization by providing weak perturbations, contrary to the common assumption that only high-quality filtered data is beneficial. The authors caution that harmful conditional shifts can still damage models, but over-curation may remove useful perturbations.

Title: A Bitter Lesson for Data Filtering Authors : Christopher Mohri , John Duchi, Tatsunori Hashimoto (@tatsu_hashimoto) Filtering helps when the model lacks enough capacity to separate manifold regions. But when the model is large enough, unfiltered data supplies weak stochastic perturbations across a broader manifold. These perturbations can activate more intrinsic pathways, stabilize more fixed-point basins, and improve generalization. The “bitter lesson” here is not only scale beats curation; it is that over-curation may remove the very perturbations needed for fixed-point construction in high-order nonlinear data. One caution: this should not be overstated as “all data is good.” The paper itself says harmful conditional shifts can still damage the model, for example systematically false statements that look like normal high-quality text. Deep Manifold would say the same: useful perturbation nudges the manifold; adversarial or wrong conditional structure can anchor the wrong fixed point. ** Dataualism ** https://x.com/BetaTomorrow/status/2048580677290070016… #DeepManifoldInterpretation
Original Article
View Cached Full Text

Cached at: 06/13/26, 02:17 PM

Title: A Bitter Lesson for Data Filtering Authors : Christopher Mohri , John Duchi, Tatsunori Hashimoto (@tatsu_hashimoto) Filtering helps when the model lacks enough capacity to separate manifold regions. But when the model is large enough, unfiltered data supplies weak stochastic perturbations across a broader manifold. These perturbations can activate more intrinsic pathways, stabilize more fixed-point basins, and improve generalization. The “bitter lesson” here is not only scale beats curation; it is that over-curation may remove the very perturbations needed for fixed-point construction in high-order nonlinear data. One caution: this should not be overstated as “all data is good.” The paper itself says harmful conditional shifts can still damage the model, for example systematically false statements that look like normal high-quality text. Deep Manifold would say the same: useful perturbation nudges the manifold; adversarial or wrong conditional structure can anchor the wrong fixed point. ** Dataualism ** https://x.com/BetaTomorrow/status/2048580677290070016… #DeepManifoldInterpretation

Turing Post (@TheTuringPost): Wow, this is interesting..

@Stanford researchers put a common assumption to the test: large models need only “high-quality” filtered training data.

What if the best filter is no filter at all?

They compared full Common Crawl data with heavily filtered versions of it and got

Similar Articles

Five out of six AI chatbots stay in the political middle ground

Reddit r/artificial

Chalmers University researchers analyzed over 148,000 AI-generated texts across six major LLMs and found most chatbots adopt a centrist stance on Swedish political issues, with Grok being the notable outlier on migration, crime, and gender.

Language Impairment as a Lens onto Large Language Models

arXiv cs.CL

The paper introduces an aphasia-inspired mechanistic interpretability framework using classifier-guided lesioning and activation steering to induce language breakdowns in LLMs, revealing densely entangled but mappable functional circuits.

Where Do Legal Factors Live in LLMs?

arXiv cs.CL

A University of Pittsburgh paper investigates whether LLMs internally represent the 26 CATO legal factors, showing all factors are linearly decodable in Qwen3-1.7B, Qwen3-8B, and Gemma-2 2B via sparse autoencoder features with test AUCs of 0.86-1.00.