Bitter Lesson, Part 2: If You Have Enough Compute, the Best Data Filter Is No Filter.
Reading this paper, the biggest takeaway is: Rich Sutton’s bitter lesson is now coming to the data side? Stanford’s Hashimoto published “A Bitter Lesson for Data Filtering”, and the core conclusion is simple: if you have enough compute, the best data filter is no filter. Their point is that the data cleaning pipelines that the industry has spent years perfecting may lose their advantage when facing large-scale scaling. At least in the setting of this paper, many filtering strategies that seem reasonable at small compute levels end up losing to the crudest approach when scaled up: just use the full pool directly.

The experimental design is not that complicated. They scaled Common Crawl and its various filtered versions (light filter, heavy filter) proportionally down, and then observed which pool ultimately produced the best model as model size increased and training steps grew. Result: In an experiment with a 670M token CC subset, the unfiltered full pool outperformed all the filtered versions they tested. When they then scaled the pool size up by two orders of magnitude, at least in the CC vs. RefinedWeb comparison, this trend remained stable. However, they only ran experiments up to 10B tokens, which is still a very small scale.

They also performed more extreme tests: injecting low-quality data into the training pool.

1. First, construct a vocabulary of 10,000 random words, then randomly sample from it to form documents.
2. Completely shuffle the word order of CC documents.

The injection amount for word-shuffled documents went up to 8 times the original pool size. The result: sufficiently large models showed remarkable robustness to such low-quality data.

The most counterintuitive finding: Adding word-shuffled documents to a 330M model not only did not hurt performance, but actually helped the model surpass the performance of the pure CC pool (except for the +800% injection group, which hadn’t been trained enough yet).

They also built a scaling law to predict: for the full 240T token CC pool from DCLM-Pool, the no-filter approach would become optimal as early as 1e30 FLOPs. And 1e30 is not that unimaginable. Currently, frontier model pretraining uses about 5e26 FLOPs; by 2030, some predictions suggest single training runs could reach 1e29 FLOPs. In other words, we may be closer than we think to the tipping point where “no filter is actually better.”

This echoes the core observation from Sutton’s original essay: trying to encode your domain knowledge into an algorithm often gets beaten in the long run by simpler methods that scale gracefully with compute.

But there is a crucial caveat: when compute is still the bottleneck, filtering remains important. And more importantly, as models grow larger, the demand for compute also increases, so we might never reach the point where compute is not a bottleneck, haha.

The authors also list the boundaries of applicability: They discuss standard pretraining of dense models, without data curricula, data weighting, or post-training; MoE, synthetic data, and data strategies in later training stages might be a different story.

And from another perspective, if the filtering itself is perfect, then of course we should filter.

https://arxiv.org/html/2605.19407v1#S6…

A Bitter Lesson for Data Filtering

Source: https://arxiv.org/html/2605.19407v1

Christopher Mohri
Department of Computer Science
Stanford University
[email protected]

&John Duchi
Departments of Statistics and Electrical Engineering
Stanford University
[email protected]

&Tatsunori Hashimoto
Department of Computer Science
Stanford University
[email protected]

Abstract

We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally “poor” data.

1 Introduction

The standard approach to select pretraining data for language models is to filter text from sources like Common Crawl (CC) (Common Crawl,2024 (https://arxiv.org/html/2605.19407v1#bib.bib9)). It is widely documented that in compute-constrained regimes, where one must train on a subset of CC, different data selection strategies can have a large impact on performance. This is intuitive: all else equal, it seems natural to train on “higher-quality” data. As a result, a large body of research has emerged to tackle the data selection problem, with the goal of finding the best subset for pretraining language models (Albalak et al.,2024 (https://arxiv.org/html/2605.19407v1#bib.bib7); Li et al.,2025a (https://arxiv.org/html/2605.19407v1#bib.bib8)).

However, not only is large-scale filter ablation heuristic and expensive, but filtering removes data, which is at odds with scaling trends that prescribe ever-increasing amounts of data to improve model performance. For example, the heavily-filtered DCLM-Baseline dataset keeps ∼1% of the original CC, leading to about 3.8 trillion tokens (Li et al.,2025a (https://arxiv.org/html/2605.19407v1#bib.bib8)). While this is still enormous, it falls short of the Chinchilla-optimal token budget for a 1 trillion parameter model, even after accounting for diminishing returns when epoching (Muennighoff et al.,2025 (https://arxiv.org/html/2605.19407v1#bib.bib5)). The current trend is also to over-train relative to Chinchilla-optimal, which prescribes even more tokens to allow for (relatively) smaller models that are financially feasible to serve (Sardana et al.,2025 (https://arxiv.org/html/2605.19407v1#bib.bib10)).

We begin by testing the hypothesis that data filtering is necessary at all in the large compute limit. While large-scale machine learning has moved toward task-agnostic pretraining (Raffel et al.,2023 (https://arxiv.org/html/2605.19407v1#bib.bib6)), and there is anecdotal evidence that larger computational budgets benefit from looser data filters (Goyal et al.,2024 (https://arxiv.org/html/2605.19407v1#bib.bib1); Muennighoff et al.,2025 (https://arxiv.org/html/2605.19407v1#bib.bib5)), removing all data filtering would be an extreme intervention that uses data considered to be actively harmful (Raffel et al.,2023 (https://arxiv.org/html/2605.19407v1#bib.bib6)). Our goal in this work is to take this extreme seriously and study the limits of (low-quality) data for transformer pretraining. We find evidence that rejects the hypothesis that data filtering is necessary, and that eventually, no existing data filter is likely to improve upon training directly on Common Crawl.

In our experiments, we scale down both CC and its filtered versions to keep their relative sizes intact, and then scale computational resources for pretraining on these different datasets. Our two main levers to do so are scaling model size (which requires more compute per training step) and training steps (which eventually leads to epoching). When comparing the best achieved performance, regardless of computational cost, our main finding is that the full pool outperforms our selected filters. Our findings are robust as we scale our experiments by 2 orders of magnitude, and we find that we can continue to see the effects from our small pool experiments as long as the models are sufficiently large. Furthermore, we find a predictable relationship between pool size, training steps, and model size which enables us to build scaling laws that predict how much compute is needed for no filter to be optimal for a particular pool size. Using this, we find that the 240 trillion token Common Crawl pool from DCLM-Pool may become optimal as soon as 1e+30 FLOPs.

These initial findings lead us to study the robustness of pretraining to “junk” data. Surprisingly, sufficiently large models are highly robust to irrelevant or junk data and can extract useful information even from highly noisy data. We test this using randomly generated strings and documents with shuffled word orders. While performance degrades at low compute budgets, sufficiently trained large models close the gap. Remarkably, these models even benefit from shuffled-word documents, despite only the unigram distribution of the documents remaining intact.

Overall, our experiments suggest that sufficiently large models that are trained for sufficiently long can benefit from the full CC dataset. While it is possible to construct harmful data, which could for example be non-factual content that looks identical to high-quality data, we do not find large amounts of this in CC. As a result, data filtering may suffer from the bitter lesson (Sutton,2019 (https://arxiv.org/html/2605.19407v1#bib.bib39)) in which human-designed filters that perform well at the small scale are eventually replaced by simple, no-filter approaches that scale more gracefully with compute.

We structure the paper as follows. In Section 2 (https://arxiv.org/html/2605.19407v1#S2), we provide the basic experimental setup, followed by experiments on filtering in Section 3 (https://arxiv.org/html/2605.19407v1#S3). We then move to adding data to our CC pool in Section 4 (https://arxiv.org/html/2605.19407v1#S4), and scaling the pool size in Section 5 (https://arxiv.org/html/2605.19407v1#S5). We finish with edge cases in Section 6 (https://arxiv.org/html/2605.19407v1#S6) and a theoretical model in Section 7 (https://arxiv.org/html/2605.19407v1#S7) to provide a post-hoc explanation of the observed phenomena.

1.1 Related Work

Data-constrained pretraining. Several prior works consider the data-constrained pretraining regime. Muennighoff et al. (2025 (https://arxiv.org/html/2605.19407v1#bib.bib5)) derive scaling laws that factor data repetition into the original Chinchilla scaling laws, finding diminishing returns after around 4 epochs on the data and that adding code data and using looser perplexity-based filters mitigates data scarcity. However, the authors recommend filtering “noisy datasets” and train on subsets of C4 (Raffel et al.,2023 (https://arxiv.org/html/2605.19407v1#bib.bib6)), while the current work directly trains on (parsed) Common Crawl and finds evidence in support of no filtering. Kim et al. (2025 (https://arxiv.org/html/2605.19407v1#bib.bib16)) study the question of algorithmic improvements in a data-constrained but compute-unlimited setting. We share a similar experimental setup (where we take subsets of a dataset, scale compute on this subset, and then scale the subset size) but differ in the object of analysis (dataset filtering).

Loose data filters. The closest work to ours is Goyal et al. (2024 (https://arxiv.org/html/2605.19407v1#bib.bib1)), who argue that filter thresholds should depend on the compute budget, showing evidence for vision-language models. They derive a scaling law to predict the filtering threshold as a function of compute budget, and conclude that “less aggressive filtering is best” with “large compute” but do not identify the parameter scaling interactions that are critical to our work, and do not show our main findings that for language models, no filter can be the best filter. Fang et al. (2025 (https://arxiv.org/html/2605.19407v1#bib.bib27)) tackle a related question by artificially repeating “high-quality” data to match the scale of loosely filtered data. They find that the former can outperform the latter in low-compute regimes, but the high compute regime studied in this work remains fully speculative in their work. Finally, Gao (2021 (https://arxiv.org/html/2605.19407v1#bib.bib28)) finds that filtering aggressively can hurt performance, speculating that this follows from Goodhart’s law [1984 (https://arxiv.org/html/2605.19407v1#bib.bib29)], and Saada et al. (2025 (https://arxiv.org/html/2605.19407v1#bib.bib30)) find that filtering with a quality classifier may improve downstream benchmarks but not validation losses on “high-quality” data. On the theoretical side, Cheng et al. (2024 (https://arxiv.org/html/2605.19407v1#bib.bib37)) develop theoretical models of the data cleaning process, arguing that given models that have enough fidelity to model noisy data generation schemes, it is better to not clean data, while cleaning data can yield more robust learning when models are not perfect. This prediction dovetails with our subsequent findings.

Low quality data. Recent works’ exploration of the impact of low-quality or intentionally degraded data on model performance motivates our experiments in Section 4 (https://arxiv.org/html/2605.19407v1#S4). Allen-Zhu and Li (2024 (https://arxiv.org/html/2605.19407v1#bib.bib36)) find that “junk data” significantly reduces knowledge capacity in a synthetic data setting, which aligns with our findings on sufficient model sizes. Counterintuitively, Li et al. (2025b (https://arxiv.org/html/2605.19407v1#bib.bib33)) argue that pretraining on toxic data leads to better representations, which makes it easier to remove toxic behavior during the post-training phase. Investigating the limits of data structure, Sinha et al. (2021 (https://arxiv.org/html/2605.19407v1#bib.bib31)) train on shuffled-word data similar to our shuffled-word experiments, arguing that the success of masked language models is primarily due to modeling “higher-order word co-occurrence statistics”. Finally, Ru et al. (2025 (https://arxiv.org/html/2605.19407v1#bib.bib32)) train models on randomly generated integers similar to our randomly generated text in Section 4 (https://arxiv.org/html/2605.19407v1#S4), and notice only a small performance drop.

2 Preliminaries

We begin with our problem setup. Our goal is to measure the value of a dataset in terms of best possible performance, regardless of computational cost, on metrics of interest such as perplexity and downstream benchmarks. More formally, for a training algorithm (\mathcal{A}) which accepts as arguments a dataset (D) of any size, parameter count (M), and training steps (N), and outputs a model (\theta \in \Theta) to be evaluated at a loss (\ell: \Theta \to \mathbb{R}), our goal is to find the best achievable performance

[ \mathcal{L}^\star(D) := \min_{M,N} \ell(\mathcal{A}(D, M, N)), \tag{1} ]

as a function of the pretraining data. Our formulation has an unconstrained minimum over parameter count (M) and training steps (N) in an attempt to extract all the “juice” out of a dataset, no matter its size. Empirically, we compute this minimum by varying (M) and (N) over several orders of magnitude until either performance improvements start to plateau or we run out of compute.

Since we do not have the compute budget to train on all of Common Crawl (let alone perform multiple epochs), our experiments are structured around randomly sampled subsets. Let (D_{cc}) be the entire CC, (D_{cc,m} \subseteq D_{cc}) be a randomly sampled subset of (m) tokens, and (f(D_{cc,m}) \subseteq D_{cc,m}) be a filtered variant of the subset. In Section 3 (https://arxiv.org/html/2605.19407v1#S3), we compare (\mathcal{L}^\star(D_{cc,m})) and (\mathcal{L}^\star(f(D_{cc,m}))) for standard filtering functions (f) such as DCLM-Baseline and RefinedWeb and our smallest subset size (m), to test if the commonly removed documents (D_{cc,m} \setminus f(D_{cc,m})) are indeed helpful for improving performance. In Section 4 (https://arxiv.org/html/2605.19407v1#S4), we test model robustness by injecting various “junk data” (J) to form (D_{cc,m} \cup J), challenging the hypothesis that (\mathcal{L}^\star(D_{cc,m}) < \mathcal{L}^\star(D_{cc,m} \cup J)) holds.

Our smaller scale experiments implicitly assume that the better of (\mathcal{L}^\star(f(D_{cc,m}))) and (\mathcal{L}^\star(D_{cc,m})) does not change (or at least changes predictably) with (m), which allows us to scale down and study the function (\mathcal{L}^\star) at reasonable compute budgets. To investigate whether this is indeed the case, and understand how performance changes as a function of (m), (M), and (N), we additionally scale over the pool size (m) in Section 5 (https://arxiv.org/html/2605.19407v1#S5).

2.1 Experiment details

We use the version of Common Crawl provided by Li et al. (2025a (https://arxiv.org/html/2605.19407v1#bib.bib8)) in their DCLM-Pool dataset, which is all of CC before 2023 with text extracted from HTML via resiliparse (Bevendorff et al.,2018 (https://arxiv.org/html/2605.19407v1#bib.bib43)). This dataset is 240 trillion GPT-NeoX (Black et al.,2022 (https://arxiv.org/html/2605.19407v1#bib.bib12)) tokens and our randomly sampled subsets range from about 670 million to 10 billion tokens. When filtering, we use the code provided by Li et al. (2025a (https://arxiv.org/html/2605.19407v1#bib.bib8)). We do not use any specialized data curricula or data weights.

Our models are Llama-style dense transformers ranging from 15 million to 7 billion parameters, trained with the Meta Lingua code repository (Videau et al.,2024 (https://arxiv.org/html/2605.19407v1#bib.bib11)). For each of the models, we tune the training step count and weight decay, following prior studies to increase repeatability of the data (Fang et al.,2025 (https://arxiv.org/html/2605.19407v1#bib.bib27); Kim et al.,2025 (https://arxiv.org/html/2605.19407v1#bib.bib16)). As is standard, we set the learning rate to decay with model size (Brown et al.,2020 (https://arxiv.org/html/2605.19407v1#bib.bib35); Kaplan et al.,2020 (https://arxiv.org/html/2605.19407v1#bib.bib26)), with an initial tuning stage to determine the decay. We release our configurat