Host-Aware Web Crawling for LLM Pretraining Data

# Host-Aware Web Crawling for LLM Pretraining Data
Source: [https://arxiv.org/html/2610.13321](https://arxiv.org/html/2610.13321)
###### Abstract

Web crawling is a major source of large language model \(LLM\) pretraining data, yet data\-selection pipelines discard a large share of crawled pages\. We investigate how a crawler should allocate its budget across and within hosts to collect more high\-quality pretraining data\. Combining document\-level quality classifier scores, web\-graph centrality metrics, and Common Crawl’s per\-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts\. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages\. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation\. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers\. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher\-scoring samples collects several times more high\-quality pages per fetch than uniform depth or centrality\-based allocation\. Finally, Common Crawl’s observed per\-host coverage has little correlation with the evaluated quality signals\. We estimate that tens of millions of high\-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts\.

## 1Introduction

Large language model \(LLM\) pretraining relies heavily on Web data such as the Common Crawl \(CC\) corpus, which is collected by crawling\. A Web crawler with a finite budget, like Common Crawl’s crawler, must decide continuously*which*URLs to fetch next\([Cho et al\., 1998](https://arxiv.org/html/2610.13321#bib.bib28)\)\. CC decides this primarily based on a Web graph metric, namely harmonic centrality\([Boldi and Vigna, 2014](https://arxiv.org/html/2610.13321#bib.bib14), HC;\)\.

Because most crawled pages are later discarded as low quality by pretraining pipelines\([Li et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib5);[Albalak et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib27)\), the crawl policy decision can be seen as a pretraining data selection or quality decision\. However, the crawling aspect of pretraining data has been mostly neglected in the literature\. Only recently,[Yu et al\. \(2025\)](https://arxiv.org/html/2610.13321#bib.bib6)proposed withCraw4LLMa policy that prioritizes the crawl frontier by the pretraining\-classifier score of each fetched page, reaching parity with a full crawl from a fraction of the URLs\.

Such page\-level schedulers are host\-blind: they maintain a global per\-page priority queue and must fetch, extract, and classify*every*candidate page to obtain the scheduling signal\. We ask a complementary question at the host level, one that requires far less compute: given the Web graph and data\-selection classifiers, how should a crawler allocate its budget across and within hosts \- and is Common Crawl’s deployed allocation already aligned with what pre\-training quality selection would choose? To answer it, we study ten data\-selection classifiers across several crawls and how they correlate with Web graph metrics and Common Crawl’s own per\-host crawl budget\. The paper is organized around four research questions that lead to the host\-level crawl policy\.

#### \(RQ1\) Is document quality a host\-level property?

A host\-level variance decomposition shows quality is moderately concentrated by host \(ICC1 0\.43–0\.62\), motivating our host\-aware policy and providing the per\-host sample size it needs \(§[4\.1](https://arxiv.org/html/2610.13321#S4.SS1)\)\.

#### \(RQ2\) Do graph\-centrality metrics work as a crawl\-yield signal?

A crawl\-yield evaluation confirms and*explains*the negative graph\-prior result of[Yu et al\. \(2025\)](https://arxiv.org/html/2610.13321#bib.bib6)through a language confound and heterogeneous hub composition, across ten classifiers \(§[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\)\.

#### \(RQ3\) How deep should a crawler go per host, and can a small sample decide?

A host\-aware crawl\-*depth*analysis shows a∼\\sim3\-page sample predicts a host’s remaining quality, and a sample\-then\-deepen policy approaches the oracle policy while centrality\-biased depth does not \(§[4\.3](https://arxiv.org/html/2610.13321#S4.SS3)\)\.

#### \(RQ4\) Does Common Crawl’s actual budget align with quality?

An analysis of historical crawl data finds that CC’s per\-host budget is uncorrelated with quality under every classifier we test, leaving tens of millions of known high\-quality URLs uncrawled on identifiable hosts \(§[4\.4](https://arxiv.org/html/2610.13321#S4.SS4)\)\.

## 2Related Work

#### LLM Pretraining Data\.

Pretraining datasets are typically filtered using methods that can process large volumes of text efficiently\. These include hand\-written rules based on document length, symbol frequency, stopword frequency, or language, as well as learned classifiers that estimate each document’s quality from its text\. The prevailing choice is the efficient fastText text classifier\([Joulin et al\., 2017](https://arxiv.org/html/2610.13321#bib.bib10)\), trained on high\-quality reference corpora in English: DCLM\([Li et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib5)\)uses a fastText OH\-2\.5\+ELI5 classifier, GneissWeb\([Gohari et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib3)\)an ensemble of five topic\-specialized fastText classifiers, and Nemotron\-CC\([Su et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib4)\)a three\-way fastText ensemble bucketed into five quality tiers; FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib1)\)uses a more compute\-intensive Transformer model\. The same recipe extends across languages: JQL\-Edu\([Ali et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib19)\)for over 35 languages, and the FineWeb2\-HQ classifier\([Messmer et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib8)\)for the multilingual FineWeb2 corpus\([Penedo et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib7)\)\. Large open pretraining corpora such as Dolma\([Soldaini et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib2)\)combine such signals into full, reproducible curation pipelines\. All of these signals are content\-internal, since they look only at the text and not at Web graph or host\-level features\. The only graph\-adjacent signals in production pipelines are categorical domain blocklists and the social\-graph seeds\([Gokaslan and Cohen, 2019](https://arxiv.org/html/2610.13321#bib.bib11);[Brack et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib23)\)\.

#### Graph Signals in Web IR\.

Classical link\-analysis methods include PageRank\([Page et al\., 1999](https://arxiv.org/html/2610.13321#bib.bib12)\), HITS\([Kleinberg, 1999](https://arxiv.org/html/2610.13321#bib.bib13)\), harmonic centrality\([Boldi and Vigna, 2014](https://arxiv.org/html/2610.13321#bib.bib14)\), TrustRank\([Gyöngyi et al\., 2004](https://arxiv.org/html/2610.13321#bib.bib15)\), and anti\-TrustRank\.[Castillo et al\. \(2007\)](https://arxiv.org/html/2610.13321#bib.bib16)showed that link\-topology features classify Web spam better than content alone\. CrediBench\([Kondrup et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib17)\)trains a graph neural network on a 45M\-node CC subgraph for source credibility prediction\. LinkBERT\([Yasunaga et al\., 2022](https://arxiv.org/html/2610.13321#bib.bib18)\)mines QA pairs from Wikipedia hyperlink graphs for synthetic pretraining data\. Both use graph structure for purposes other than document filtering\.

#### URL\-level Crawl Policy\.

Craw4LLM\([Yu et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib6)\)is the closest prior work\. On ClueWeb22\([Overwijk et al\., 2022](https://arxiv.org/html/2610.13321#bib.bib29)\), the authors measure a weak Spearmanρ≈−0\.11\\rho\\approx\-0\.11between a page’s indegree and its DCLM score \(on the ClueWeb22\-B sample\), and simulate crawling on the 900M\-page ClueWeb22\-A subset\. Indegree\-prioritised crawling reaches only 69\.5% of oracle downstream LM performance, vs\. 78\.1% for random crawling and 95\.3% for fasttext\-driven crawling\. They conclude that graph centrality is a poor standalone LLM data selector\. Our results reproduce the negative overall correlation \(ρ=−0\.119\\rho=\-0\.119forHCon CC\-MAIN\-2024\-10\) and provide a plausible mechanism: English\-trained classifier scores covary with host language composition, which itself covaries withHC\. We do not analyse not\-public ClueWeb22\-A directly\. Furthermore, we also find thatPRis positively correlated with quality when comparing hosts within the same top\-level domain \(TLD\)\.

#### Graph\-based Data Selection\.

WebGraphMix\([Badoni et al\., 2026](https://arxiv.org/html/2610.13321#bib.bib20)\)evaluates mixtures of already\-collected documents for LLM pretraining\. We evaluate host ordering and depth allocation during crawling, measuring high\-quality pages collected under a fixed fetch budget\. We focus on harmonic centrality \(HC\) and PageRank \(PR\), the metrics Common Crawl publishes\. WebGraphMix’s finding thatPR\-based selection does not improve downstream performance is consistent with the limited crawl\-yield gains from centrality\-based ordering in our experiments \(§[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\), although the two studies evaluate different setups\. DCLM\([Li et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib5)\)grouped documents into five bins byPRand found that no bin outperformed random sampling\. Our results suggest a possible explanation: pooling hosts across TLDs can obscure the positive correlation betweenPRand quality within individual TLDs\.

## 3Data and Methods

Our experiments draw on*two distinct data regimes*, and keeping them separate is important for interpreting the results\. In theclassified\-sampleregime we do our own extraction and classification: we read WARC records, extract main text, and run our data\-selection classifiers on documents extracted from 224 WARC files, approximately 0\.25% of the WARC files in CC\-MAIN\-2024\-10 \(RQ1\-\-RQ4\)\.111Our own classification is needed since pretraining datasets are typically published without low quality samples and without the corresponding full range of classifier scores\.In thefull\-crawlregime, we instead use precomputed artifacts \(Common Crawl’s host index and the GneissWeb quality annotations covering*every*fetched page\) without re\-classification \(RQ3 replication, RQ4\)\. Appendix[D](https://arxiv.org/html/2610.13321#A4)lists every dataset and model and which research question uses it\.

### 3\.1Classifying Web Documents

Our primary classified sample is from CC\-MAIN\-2024\-10 \(February/March 2024\)\. Specifically, we extract the text from HTML pages using Trafilatura\([Barbaresi, 2021](https://arxiv.org/html/2610.13321#bib.bib22)\)from 224 randomly selected WARC files\. Keeping hosts with at least three documents yields 3\.01M documents on 532,628 hosts for our primary host\-level analyses\. These document counts refer to our sample of WARC files, not to all pages that Common Crawl has fetched from each host\. The depth analysis uses stricter inclusion thresholds, described in §[4\.3](https://arxiv.org/html/2610.13321#S4.SS3)\.

We score each extracted document with ten fasttext classifiers and take host\-median scores: DCLM\([Li et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib5)\), FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib1)\), three Nemotron\-CC classifiers \(quality, educational, information\)\([Su et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib4)\), and five GneissWeb classifiers \(a quality annotator plus science, education, technology, and medical topic scorers\)\([Gohari et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib3)\)\. Six of the ten classifiers target pretraining*quality*and four score*topic*\. Throughout, we use*quality*to mean a classifier’s predicted usefulness for pretraining, which is itself only a proxy for downstream benchmark performance and not intrinsic document quality\. For FineWeb\-Edu, we run a compute\-efficient fastText re\-implementation\([Tsui and Nguyen, 2024](https://arxiv.org/html/2610.13321#bib.bib9)\)of the original Transformer\-based classifier, at comparable accuracy\.

For the language analysis \(§[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\), we additionally run two multilingual Transformer classifiers, JQL\-Edu\([Ali et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib19)\)and the FineWeb2\-HQ classifier\([Messmer et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib8);[Penedo et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib7)\), on a smaller core set of 462,494 documents derived from 16 WARC files\. Exact model identifiers are in Table[10](https://arxiv.org/html/2610.13321#A4.T10)\. These scores drive the variance decomposition \(RQ1\), the crawl\-yield simulation \(RQ2\), and the primary depth simulation \(RQ3\)\. We run the same extraction\-and\-scoring pipeline on a second snapshot, CC\-MAIN\-2023\-40 \(32 WARC files\), which we use for the language and cross\-crawl robustness checks \(Appendix[D](https://arxiv.org/html/2610.13321#A4)\)\.

### 3\.2Web Graph

Graph features come from Common Crawl’s public host\-level hyperlinkgraph222[https://commoncrawl\.org/web\-graphs](https://commoncrawl.org/web-graphs), using the alignedcc\-main\-2023\-24release \(built from crawls spanning September 2023 to February 2024\) so that a single graph covers our document snapshots\. For each host we take harmonic centrality \(HC\) and PageRank \(PR\), the two centrality measures Common Crawl publishes, inlog10⁡\(1\+x\)\\log\_\{10\}\(1\{\+\}x\)form\. Fewer than 0\.3% of our hosts are missing from the graph and are excluded\.

### 3\.3Crawl Budget and Quality Density

The full\-crawl regime uses two Common Crawl artifacts we do*not*recompute ourselves\. Thehost indexreports, per host, the number of known URLs \(nutch\_numRecords\), how many were fetched \(nutch\_fetched\), and the resulting*crawl budget or coverage*\(nutch\_fetched\_pct\)\. TheGneissWeb annotation set\([Gohari et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib3)\), released by Common Crawl, labels the pages a strong quality filter kept; every annotated page passed the filter, and the annotations cover*every*fetched page rather than our WARC sample\. Joining the two on the same crawl gives a per\-host*quality density*,

quality density=\#​GW\-quality pages\#​fetched pages,\\text\{quality density \}=\\frac\{\\\#\\text\{GW\-quality pages\}\}\{\\\#\\text\{fetched pages\}\},\(1\)the fraction of a host’s crawled pages that are high quality\. Because it is full\-crawl and independent of our own extraction and classifiers, it both replicates the depth analysis at full scale \(RQ3\) and grounds the crawl\-budget analysis \(RQ4\)\. We validate it against our own classifiers in §[4\.4](https://arxiv.org/html/2610.13321#S4.SS4)\. Concretely, the crawl\-budget analysis joins the host index with GneissWeb quality annotations on CC\-MAIN\-2024\-10 and also across six crawls for the temporal panel \(CC\-MAIN\-\{2021\-49, 2022\-49, 2023\-23, 2023\-40, 2024\-10, 2024\-18\}\)\. The full\-crawl depth replication uses the GneissWeb depth annotations for CC\-MAIN\-2017\-34\.

### 3\.4Metrics

We use different metrics for different research questions\. RQ1 examines variation in individual page scores\. The graph\-quality correlations use the median classifier score of each host\. The crawl\-yield and depth simulations instead count pages whose classifier score exceeds a threshold and measure the proportion of high\-quality pages\. For RQ4, quality density is the proportion of fetched pages retained by the GneissWeb filter\. A host’s median score and its proportion of high\-quality pages are different quantities\.

#### Spearmanρ\\rho\.

Our primary statistic is Spearman rank correlation, which measures whether two quantities tend to rise together \(\+1\+1\) or fall as the other rises \(−1\-1\), with00meaning no monotone relationship\.

#### Host\-level variance \(RQ1\)\.

The intraclass correlation \(Shrout–Fleiss one\-way estimator\)

ICC1=σbetween2σbetween2\+σwithin2\\text\{ICC1\}=\\frac\{\\sigma^\{2\}\_\{\\text\{between\}\}\}\{\\sigma^\{2\}\_\{\\text\{between\}\}\+\\sigma^\{2\}\_\{\\text\{within\}\}\}\(2\)is the share of per\-document score variance that lies*between*hosts\. ICC1=1=1would mean a host’s pages are uniform in that score and00that a host tells us nothing about a page\. Withm¯\\bar\{m\}documents per host, the Spearman–Brown reliability

m¯​ICC11\+\(m¯−1\)​ICC1\\frac\{\\bar\{m\}\\,\\text\{ICC1\}\}\{1\+\(\\bar\{m\}\-1\)\\,\\text\{ICC1\}\}\(3\)bounds the host\-level correlation any signal can achieve, so ICC1 also sets the per\-host sample size needed to rank hosts\.

#### Crawl\-yield simulation \(RQ2\)\.

We fix a crawl budget as a fraction of corpus documents and fetch hosts in the order a policy defines until the budget is spent\. The policies are random, Common Crawl’s current allocation,HC,PR, or an oracle that knows each host’s true quality density\. We report yield as the share of fetched pages that are high quality \(Figure[2](https://arxiv.org/html/2610.13321#S4.F2)\) and, for cross\-classifier comparison, as a multiple of the random baseline \(Appendix[B](https://arxiv.org/html/2610.13321#A2)\)\.

#### Depth simulation \(RQ3\)\.

We count a page as*high quality*when its quality score exceeds0\.50\.5, and a host’s quality density is its fraction of such pages, using both our FineWeb\-Edu scores on the classified sample and GneissWeb quality scores on the full crawl\. RQ1 already implies that a small per\-host sample characterizes a host, but we also confirm this directly by correlating akk\-page sample’s quality density with that of the host’s remaining pages \(Figure[4](https://arxiv.org/html/2610.13321#S4.F4)\), and then use the sample to decide crawl depth\.

Each policy gets the same budget: a fixed number of pages to fetch, set as a fraction of all available pages\. We score a policy by its*yield*, i\.e\., the high\-quality pages it collects per page fetched\. A crawler cannot tell a host’s high quality pages from its low quality ones until it fetches them\. So the expected high\-quality count is the host’s quality density times the number of pages fetched\. A policy therefore cannot cherry\-pick high quality pages*within*a host\. Instead, it can only choose*which*hosts to crawl deeply\. We compare four policies\.

*Uniform breadth*crawls every host to the same shallow depth\.*HCdepth*crawls hosts fully in order of harmonic centrality, highest first, until the budget is spent\.*Sample\-then\-deepen*first probes three pages per host, then crawls fully the hosts whose sampled quality density is the highest\. The*oracle*crawls fully the hosts with the highest*true*quality density first, giving the upper bound\.

#### Crawl\-budget analysis \(RQ4\)\.

We correlate each host’s crawl coverage with its quality density, and define*hidden gems*as hosts below the median coverage whose quality density is in the top decile\. We estimate the number of high\-quality pages among a host’s known but uncrawled URLs by multiplying their count by its observed quality density\. This assumes that uncrawled pages are as likely to be high quality as the fetched pages from the same host\.

## 4Results

### 4\.1RQ1: Quality Is a Host\-Level Property

Allocating more crawl budget to some hosts can improve yield if their pages tend to be higher quality than those on other hosts\. We therefore measure how much of the variation in classifier scores occurs between hosts\. Across all 10 classifiers, the between\-host share of per\-document score variance \(ICC1\) lies in 0\.43–0\.62, indicating moderate host\-level consistency \(Figure[1](https://arxiv.org/html/2610.13321#S4.F1)\)\. In other words, roughly half of the observed variation in classifier scores is between hosts\.

Figure 1:The intraclass correlation ICC1 is the share of a classifier’s per\-document score variance that lies*between*hosts rather than within them: Across all ten classifiers ICC1 is 0\.43–0\.62 \(N=532,628N=532\{,\}628\{\}hosts\), so roughly half of the variation in page quality is between hosts\. This suggests that a few sampled pages already characterize a host, enabling a host\-level crawl policy\.Moderate ICC1 has two consequences we use later\. Withm¯≈5\\bar\{m\}\\approx 5observed documents per host in our WARC sample and ICC1≈0\.5\\approx 0\.5, Spearman–Brown reliability is≈0\.83\\approx 0\.83, so the largest host\-level correlation any signal can reach is≈0\.9\\approx 0\.9, not11; near\-zero correlations must be read against this ceiling\. The observed host\-level consistency also motivates testing whether a*small*sample predicts the quality of remaining pages, which we evaluate directly in RQ3\.

### 4\.2RQ2: Graph Centrality Is a Weak Crawl\-Yield Signal

Figure 2:Graph centrality is a weak signal for*which*hosts to crawl\. When only 10% of pages are crawled, the share of fetched pages that are high quality: centrality ordering \(current/HC/PR\) captures only a fraction of the*oracle*\(crawler knows host’s quality\), and even trails random for DCLM\.Common Crawl uses harmonic centrality \(HC\) to guide which hosts to crawl and how deeply to crawl them\. We therefore test whether prioritizing hosts by centrality increases the number of high\-quality pages collected under a fixed crawl budget\.

At a 10% crawl budget, ordering hosts byHCcollects 0\.7–1\.8 times as many high\-quality pages as random ordering across classifiers, compared with 4\.0–8\.0 times for an oracle that knows each host’s quality density \(Figure[2](https://arxiv.org/html/2610.13321#S4.F2)\)\. PageRank \(PR\) and Common Crawl’s current allocation also yield substantially fewer high\-quality pages than the oracle\. For DCLM, centrality\-based ordering performs worse than random\. Ordering byHCalso roughly halves the number of top\-level domains represented compared with random ordering, reducing TLD diversity\. Appendix[B](https://arxiv.org/html/2610.13321#A2)reports results by classifier and crawl budget\.

Figure 3:Median host quality scores by language\. English\-trained classifiers differ in how they score English and non\-English hosts: DCLM assigns higher scores to non\-English hosts, while Nemotron\-Q assigns higher scores to English hosts\. Because centrality also varies by language, its correlation with classifier scores partly reflects these language differences\.#### Language partly explains the weak correlation\.

Prior work reports the similar findings\. Craw4LLM measures a negative host\-indegree/quality correlation on ClueWeb22\-A, and WebGraphMix rejects centrality for post\-hoc selection\. We reproduce the finding across ten classifiers on CC data and find that language differences partly explain the observed patterns\.

A host’s primary language covaries with its centrality, and the English\-trained quality classifiers score English and non\-English hosts very differently\([Lucy et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib26)\), but not in a single direction \(Figure[3](https://arxiv.org/html/2610.13321#S4.F3)\)\. DCLM is the extreme case since it rates non\-English hosts far higher than English ones \(median 0\.29 vs\. 0\.0005\), so pooling the two languages makes its overallHCcorrelation*negative*\(ρ=−0\.083\\rho=\-0\.083\) even though it is positive within English hosts \(ρ=\+0\.064\\rho=\+0\.064\)\. Nemotron\-Q leans the opposite way, scoring English hosts higher \(0\.49 vs\. 0\.37\)\. Multilingual classifiers, which do not share the bias, already show positive signal on non\-English hosts \(JQL\-Edu\+0\.137\+0\.137, FineWeb2\-HQ\+0\.186\+0\.186; full language\-stratified table in Appendix[C](https://arxiv.org/html/2610.13321#A3)\)\. Centrality is thus entangled with language and not a clean quality signal\.

### 4\.3RQ3: A Small Per\-Host Sample Decides Crawl Depth

Figure 4:The correlation between the quality ofkksampled pages and the quality of the host’s remaining pages, where1\.01\.0is perfect prediction and00is none\. The curve climbs withkkand plateaus byk≈15k\\approx 15–2020, but a∼\\sim3–5 page sample already captures most of the signal, so a few pages suffice to decide crawl depth cheaply\. Shown for our FineWeb\-Edu sample \(hosts with≥10\\geq 10pages, sokkreaches99\) and full\-crawl GneissWeb \(≥30\\geq 30pages\)\.Figure 5:Crawl efficiency based on FineWeb\-Edu, measured as high\-quality pages obtained per page actually fetched, and for four ways of spending a fixed budget across hosts\.*Sample\-then\-deepen*\(our host\-level crawl policy\) nearly matches the*oracle*and collects several times more high\-quality pages per fetch than crawling every host equally \(*uniform breadth*\) or biasing depth toward high\-centrality hosts \(*HCdepth*\)\.We now ask*how deep*to crawl each host\. Because quality is a host property \(RQ1\), a few pages sampled from a host predict the rest: Figure[4](https://arxiv.org/html/2610.13321#S4.F4)shows the rank correlation between akk\-page sample and a host’s remaining pages rises withkkand plateaus byk≈15k\\approx 15–2020\. The gain per added page shrinks steadily, so a small33–55page sample \(ρ=0\.53\\rho=0\.53\{\}atk=3k\{=\}3,0\.570\.57\{\}atk=5k\{=\}5\) already carries most of the signal\. Ranking hosts on a three\-page sample and deepening the best hosts already recovers most of the oracle’s depth yield\. A larger sample would rank hosts better at more computational costs\.

This motivates a*sample\-then\-deepen*crawl policy, which samples and classifies a few pages per host, then spends depth on the hosts whose sample are higher quality\. Such a policy obtains up to 7\.8×\\timesmore high\-quality pages per fetched page than the uniform breadth policy at small budgets \(down to 1\.7×\\timesas the budget grows\) and approaches the oracle policy\.

In contrast, depth biased by harmonic centrality \(similar to what CC does today\) barely differs from uniform breadth \(Figure[5](https://arxiv.org/html/2610.13321#S4.F5)\)\. The*sample\-then\-deepen*policy has also the pratical advantage that the classifier is not in the crawler’s hot path: unlike a page\-level scorer \(e\.g\., Craw4LLM\) it runs on a∼\\sim3\-page sample per host, not on every page\.

### 4\.4RQ4: Common Crawl’s Budget Is Blind to LLM Data Quality

Finally, we turn from simulation to the deployed crawl\. Using historical per\-host budget from CC\-MAIN\-2024\-10 and GneissWeb quality scores, we find its crawl coverage carries no positive quality signal: across 5,654,186 hosts, the correlation between how fully a host is crawled and its quality density is \-0\.08, and the quality distribution does not rise with coverage \(Figure[6](https://arxiv.org/html/2610.13321#S4.F6)\)\. If anything the least\-crawled hosts are the highest quality\. Assuming that uncrawled pages have the same quality density as fetched pages from the same host, we estimate 58M high\-quality pages among the known URLs left uncrawled in this one crawl\. This missed budget concentrates on barely\-crawled hosts: 50% of it lies on hosts crawled below 20%, and 62% falls on a compact set of 369,534 “hidden\-gem” hosts \(below\-median coverage but top\-decile quality\) that represent only 6\.5% of all hosts \(Figure[7](https://arxiv.org/html/2610.13321#S4.F7)\)\. The URLs are known, but the quality of their unfetched pages has not been directly measured\.

Figure 6:Host\-level quality density does not rise with crawl coverage \(Spearman \-0\.08\)\. The median quality density stays flat and low across coverage \(from 20% at the least\-crawled hosts to 4% at the near\-fully\-crawled majority\)\. Grey bars show the share of hosts in each coverage bin: most are already almost fully crawled, which is why the overall rank correlation is near zero\.![Refer to caption](https://arxiv.org/html/2610.13321v1/fig_rq5_coverage_quality_b.png)Figure 7:The missed high\-quality budget concentrates on barely\-crawled hosts\. Each cell gives the share of the estimated≈\\approx58M uncrawled high\-quality pages that falls in one coverage×\\timesquality bin \(columns: crawl coverage; rows: quality density; axes as in Figure[6](https://arxiv.org/html/2610.13321#S4.F6)\)\. Half \(50%\) of the missed pages lie on hosts crawled below 20%, and the single densest “hidden\-gem” bin holds 26% of missed high\-quality pages alone\.Expressed as pretraining tokens, this single\-crawl shortfall is on the order of tens of billions of high\-quality tokens \(roughly 58 billion at an assumed∼\\sim1,000 tokens per page\)\. We find shortfalls of a similar scale in each crawl examined \(Table[1](https://arxiv.org/html/2610.13321#S4.T1)\)\. Extrapolating this estimate to the \+100 existing CC crawls would give approximately 6 trillion tokens before accounting for repeated pages or duplicate content\. This is an illustrative extrapolation\. We have not measured overlap across crawls or established that the same shortfall applies throughout the whole crawl archive\.

Table 1:The coverage–quality disconnect is not a one\-off\. For each of 6 crawls \(2021–2024\) we report the Spearman correlation between crawl coverage and quality density \(all near00and slightly negative, i\.e\. no positive alignment\), the number of under\-crawled high\-quality “gem” hosts, and the estimated high\-quality URLs those hosts leave uncrawled\. Every crawl shows the same pattern and tens of millions of missed URLs\.Figure 8:Coverage\-blindness does not depend on the GneissWeb filter\. Per classifier, on one Spearman axis: its agreement with the GneissWeb quality density \(green\) versus the correlation of Common Crawl coverage with it \(orange\)\. Quality classifiers \(Nemotron, FineWeb\-Edu\) agree substantially with GneissWeb \(DCLM and the topic scorers less so\), yet every coverage bar stays inside the negligible band \(\|ρ\|<0\.1\|\\rho\|<0\.1\): whichever quality signal we use, how fully a host is crawled does not track how good it is\.#### Stable and classifier\-robust\.

The pattern is not a one\-crawl or one\-classifier artifact\. Across six crawls spanning 2021–2024 the coverage–quality correlation stays near zero and negative \(\-0\.057 to \-0\.151; Table[1](https://arxiv.org/html/2610.13321#S4.T1)\)\. Also, the GneissWeb quality signal is confirmed by independent classifiers: Nemotron \(0\.53–0\.54\) and FineWeb\-Edu \(\+0\.29\+0\.29\) agree with it, while Common Crawl coverage is uncorrelated with*every*classifier we test \(\|ρ\|≤0\.07\|\\rho\|\\leq 0\.07\{\}; Figure[8](https://arxiv.org/html/2610.13321#S4.F8)\)\. DCLM is the lone non\-agreeing classifier \(−0\.06\-0\.06\), consistent with its known orthogonality to other quality signals \(§[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\)\. The missed budget skews toward national and regional news publishers and scholarly repositories: the\.ukTLD alone accounts for 23% of it from 6\.1% of hosts, at just 34% median coverage \(Appendix, Figure[11](https://arxiv.org/html/2610.13321#A5.F11)\)\.

## 5Discussion

#### A host\-level companion to Craw4LLM\.

Craw4LLM\([Yu et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib6)\)prioritizes the crawl frontier by the quality score of each fetched page\. Thus, a crawler must fetch, extract, and score every candidate to obtain that signal\. Our results address a different, complementary lever\. Because quality is a host property \(RQ1\) and a∼\\sim3\-page sample predicts a host’s rest \(RQ3\), the scheduling signal for*how deep*to crawl a host can be obtained from a small sample rather than from every page, and the classifier need not live in the crawler’s hot path\. We answer “how much to crawl each host, and which under\-crawled hosts are worth more,” which the page\-level, host\-blind formulation does not address\. Our analysis complements page\-level scheduling and informs how much crawl budget should be allocate to each host\.

#### Explaining earlier findings on graph centrality\.

Craw4LLM and WebGraphMix\([Badoni et al\., 2026](https://arxiv.org/html/2610.13321#bib.bib20)\)report limited benefits from the centrality measures they evaluate\. Our crawl\-yield results show a similar pattern \(RQ2\): centrality\-based ordering performs much worse than the oracle and, for DCLM, worse than random ordering\.

Language differences help explain some of the observed centrality–quality correlations\. English\-trained classifiers score English and non\-English hosts differently\. For instance, DCLM assigns higher scores to non\-English hosts\. Centrality also varies by language\. The correlation betweenHCand DCLM scores is negative when all hosts are pooled, but positive when only English\-language hosts are considered\. High\-centrality hosts also contain a mix of high\- and low\-quality pages, limiting centrality’s usefulness as a quality signal\.

However, the language analysis does not establish that accounting for language improves crawl yield\. A positive correlation within a language group need not translate into an effective ranking under a limited crawl budget\. Likewise, positive correlations with multilingual classifiers do not by themselves demonstrate a better crawl policy\. Lastly, centrality could still be used to group hosts for sampling, although we do not evaluate whether this improves crawl yield\.

#### What the deployed crawl leaves on the table\.

RQ4 turns the diagnostic on CC’s deployed crawl policy itself\. Its per\-host budget is uncorrelated with quality under every classifier we tested, so the crawl is not preferentially deepening the hosts worth deepening\. This misalignment is expected rather than a defect: Common Crawl was built as a broad, general\-purpose sample of the Web serving diverse use cases, not as an LLM\-oriented crawler\([Baack, 2024](https://arxiv.org/html/2610.13321#bib.bib21)\)\.

Assuming that uncrawled pages have the same quality density as fetched pages from the same host, we estimate that tens of millions of high\-quality pages remain among known but uncrawled URLs in each crawl\. The presented*sample\-then\-deepen*approach could help recover these pages by allocating more budget to hosts with high\-scoring samples\. It is worth noting that this gap is aboutknownURLs, for an estimation of whole URL population we refer to[Paris et al\. \(2026\)](https://arxiv.org/html/2610.13321#bib.bib24)\.

#### Quality and discovery are different objectives\.

Our depth simulations allocate a crawl budget over a fixed set of already\-fetched pages\. They do not measure which additional URLs or hosts a crawler would discover by following links\. A host with low classifier scores could nevertheless provide links to useful content elsewhere\. Weak performance as a quality\-ranking signal therefore does not rule out a role for centrality in discovery\. A crawler could combine centrality\-based exploration with sample\-guided depth allocation, but our experiments do not evaluate this combination\.

## 6Conclusions

In this paper, we studied Web crawl policies for LLM pretraining data at the level of hosts and reached the following conclusions\.

Document quality is largely a host property, i\.e, between\-host variance is about half of the total\. This finding motivates the host\-aware policy and lets a small sample stand in for a host\. Ordering hosts by graph centrality \(HCorPR\) collects at most 1\.8 times as many high\-quality pages as random ordering, and fewer for DCLM\. An oracle policy that knows each host’s quality density collects 4\.0–8\.0 times as many\. Centrality is a weak guide partly because it varies with host language, which also affects classifier scores, and because highly central hosts contain a mix of high\- and low\-quality pages\.

In our crawl simulations, a three\-page sample provides a useful signal for allocating crawl depth\. The sample\-then\-deepen policy approaches the oracle high\-quality yield, reaching up to 7\.8 times the yield of the uniform breadth policy, whereas centrality\-biased depth provides little to none improvement\. Finally, Common Crawl’s deployed crawl budget is itself blind to LLM data quality\. The crawl coverage is uncorrelated with quality \(Spearman \-0\.08\) under every classifier, leaving an estimated 58M known high\-quality URLs uncrawled in a single crawl, 62% of them on a compact set of 369,534 under\-crawled hosts\. This finding can be attributed to Common Crawl aiming for a broad, general\-purpose sample of Web, and not LLM pretraining data\.

In summary, our results motivate a crawl policy that first samples a few pages from each host, scores their quality, and then fetches more pages from hosts with higher\-scoring samples\. This could recover high\-quality pages missed by the current crawl policy\. Graph centrality could help group hosts for sampling, but should not be used on its own to judge their content quality\.

### 6\.1Future Work

A live sample\-then\-deepen crawl\.All four research questions rely on existing data from Common Crawl and did not involve any live crawling\. RQ1 measures host\-level variation in classifier scores, while RQ4 examines observed crawl coverage and quality density\. RQ2 and RQ3 additionally simulate alternative crawl allocations over already\-fetched pages\. None of these analyses tests a newly deployed crawler under production settings, future work needs to address this\. In particular, a live crawl experiment must account for fetch latency, content freshness, politeness limits, and the discovery of new URLs and hosts\.

Recovering the hidden gems\.RQ4 identifies tens of millions of known high\-quality URLs on under\-crawled hosts\. A targeted re\-crawl of the top gem hosts, followed by a downstream ablation, would quantify how much the missed budget is worth in trained\-model terms\.

Downstream model ablation\.We measure crawl yield and coverage, not trained\-model performance\. Given that proxy metrics and downstream accuracy can diverge\([Li et al\., 2024](https://arxiv.org/html/2610.13321#bib.bib5);[Yu et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib6)\), one would need to train small models \(∼\\sim1B parameters\) on data selected by uniform, centrality, and sample\-then\-deepen policies test whether the yield gains translate into downstream quality\.

Beyond a single quality filter\.The quality\-density signal uses primarily a single filter \(GneissWeb\) due to the availability of precomputed annotations\. Combining several classifiers, and modeling their per\-host agreement, may sharpen both the depth decision and the hidden\-gem ranking\.

## Limitations

The following discussion complements the limitations described in §[6\.1](https://arxiv.org/html/2610.13321#S6.SS1)\.

Quality density is a proxy\.Our per\-host quality signal is the fraction of a host’s crawled pages retained by a classifier model \(GneissWeb\)\. It inherits that models’s biases\. We mitigate this by showing the RQ4 result holds under independent models \(Nemotron, FineWeb\-Edu\), but no model is ground truth\([Gururangan et al\., 2022](https://arxiv.org/html/2610.13321#bib.bib25)\)\.

Missed\-budget estimate\.The uncrawled\-high\-quality\-URL count multiplies a host’s uncrawled known URLs by its observed quality density, assuming the uncrawled pages resemble the crawled ones\. It is an estimate of scale, not an exact count\. Furthermore, the estimate does not account for overlap between crawls and other factors like crawl politeness or content freshness\.

Selection of pages and hosts\.Both the host\-consistency estimates and the sample\-size analysis concern pages already selected by Common Crawl\. Our WARC sample introduces a further sampling step, and minimum\-document thresholds exclude sparsely represented hosts\. The full\-crawl GneissWeb analysis removes the WARC subsampling step, but still reflects Common Crawl’s page selection\. These choices may affect measured within\-host variation and how well a small sample predicts other pages\. Our results therefore do not establish that three pages suffice for arbitrary hosts or for pages outside the observed crawls\.

Representation and diversity\.Allocating more crawl budget to hosts with high classifier scores may reduce the representation of languages, regions, or content types that those classifiers score poorly\. At crawl time, this affects which pages are collected at all, including potentially useful pages on hosts whose initial samples receive low scores\. Higher classifier\-based yield therefore does not necessarily imply a more representative or diverse corpus\. A deployment could reserve budget for broader exploration and monitor coverage across languages and content types, but the effectiveness of these measures remains to be evaluated\.

## References

- Albalaket al\.\(2024\)A\. Albalak, Y\. Elazar, S\. M\. Xie, S\. Longpre, N\. Lambert, X\. Wang, N\. Muennighoff, B\. Hou, L\. Pan, H\. Jeong, C\. Raffel, S\. Chang, T\. Hashimoto, and W\. Y\. WangA survey on data selection for language models\.Transactions on Machine Learning Research \(TMLR\)\.Note:arXiv:2402\.16827External Links:[Link](https://arxiv.org/abs/2402.16827)Cited by:[§1](https://arxiv.org/html/2610.13321#S1.p2.1)\.
- Aliet al\.\(2025\)M\. Ali, M\. Brack, M\. Lübbering, E\. Wendt, A\. G\. Khan, R\. Rutmann, A\. Jude, M\. Kraus, A\. A\. Weber, D\. Kaczér, F\. Mai, L\. Flek, R\. Sifa, N\. Flores\-Herr, J\. Köhler, P\. Schramowski, M\. Fromm, and K\. KerstingJudging quality across languages: a multilingual approach to pretraining data filtering with language models\.Note:arXiv:2505\.22232External Links:[Link](https://arxiv.org/abs/2505.22232)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p3.1)\.
- Baack \(2024\)S\. BaackA critical analysis of the largest source for generative AI training data: Common Crawl\.InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency \(FAccT\),External Links:[Document](https://dx.doi.org/10.1145/3630106.3659033)Cited by:[§5](https://arxiv.org/html/2610.13321#S5.SS0.SSS0.Px3.p1.1)\.
- Badoniet al\.\(2026\)V\. Badoni, D\. Chen, and X\. WangHubs or fringes: pretraining data selection via web graph centrality\.Note:arXiv:2606\.11499External Links:[Link](https://arxiv.org/abs/2606.11499)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2610.13321#S5.SS0.SSS0.Px2.p1.1)\.
- Barbaresi \(2021\)A\. BarbaresiTrafilatura: A Web Scraping Library and Command\-Line Tool for Text Discovery and Extraction\.InProceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations,pp\. 122–131\.External Links:[Link](https://aclanthology.org/2021.acl-demo.15)Cited by:[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p1.1)\.
- Boldi and Vigna \(2014\)P\. Boldi and S\. VignaAxioms for centrality\.Internet Mathematics10\(3–4\),pp\. 222–262\.External Links:[Document](https://dx.doi.org/10.1080/15427951.2013.865686)Cited by:[§1](https://arxiv.org/html/2610.13321#S1.p1.1),[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px2.p1.1)\.
- Bracket al\.\(2024\)M\. Brack, M\. Ostendorff, P\. Ortiz Suarez, J\. J\. Saiz, I\. L\. Castilla, J\. Palomar\-Giner, A\. Shvets, P\. Schramowski, G\. Rehm, M\. Villegas, and K\. KerstingCommunity OSCAR: a community effort for multilingual web data\.InProceedings of the Fourth Workshop on Multilingual Representation Learning \(MRL 2024\),J\. Sälevä and A\. Owodunni \(Eds\.\),Miami, Florida, USA,pp\. 232–235\.External Links:[Link](https://aclanthology.org/2024.mrl-1.19/),[Document](https://dx.doi.org/10.18653/v1/2024.mrl-1.19)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1)\.
- Castilloet al\.\(2007\)C\. Castillo, D\. Donato, L\. Becchetti, P\. Boldi, S\. Leonardi, M\. Santini, and S\. VignaKnow your neighbors: web spam detection using the web topology\.InProceedings of the 30th Annual International ACM SIGIR Conference,pp\. 423–430\.External Links:[Document](https://dx.doi.org/10.1145/1277741.1277814)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px2.p1.1)\.
- Choet al\.\(1998\)J\. Cho, H\. Garcia\-Molina, and L\. PageEfficient crawling through URL ordering\.Computer Networks and ISDN Systems30\(1–7\),pp\. 161–172\.External Links:[Document](https://dx.doi.org/10.1016/S0169-7552%2898%2900108-1)Cited by:[§1](https://arxiv.org/html/2610.13321#S1.p1.1)\.
- Gohariet al\.\(2025\)H\. E\. Gohari, S\. R\. Kadhe, S\. Y\. Shah, C\. Adam, A\. Adebayo, P\. Adusumilli, F\. Ahmed, N\. B\. Angel, S\. S\. Borse, Y\. Chang, X\. Dang, N\. Desai, R\. Eres, R\. Iwamoto, A\. Karve, Y\. Koyfman, W\. Lee, C\. Liu, B\. Lublinsky, T\. Ohko, P\. Pesce, M\. Touma, S\. Wang, S\. Witherspoon, H\. Woisetschläger, D\. Wood, K\. Wu, I\. Yoshida, S\. Zawad, P\. Zerfos, Y\. Zhou, and B\. BhattacharjeeGneissWeb: preparing high quality data for LLMs at scale\.Technical reportIBM Research\.Note:arXiv:2502\.14907External Links:[Link](https://arxiv.org/abs/2502.14907)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p2.1),[§3\.3](https://arxiv.org/html/2610.13321#S3.SS3.p1.1)\.
- Gokaslan and Cohen \(2019\)A\. Gokaslan and V\. CohenOpenWebText corpus\.External Links:[Link](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1)\.
- Gururanganet al\.\(2022\)S\. Gururangan, D\. Card, S\. K\. Dreier, E\. K\. Gade, L\. Z\. Wang, Z\. Wang, L\. Zettlemoyer, and N\. A\. SmithWhose language counts as high quality? measuring language ideologies in text data selection\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2201\.10474External Links:[Link](https://arxiv.org/abs/2201.10474)Cited by:[Limitations](https://arxiv.org/html/2610.13321#Sx1.p2.1)\.
- Gyöngyiet al\.\(2004\)Z\. Gyöngyi, H\. Garcia\-Molina, and J\. PedersenCombating web spam with TrustRank\.InProceedings of the 30th International Conference on Very Large Data Bases \(VLDB\),pp\. 576–587\.External Links:[Document](https://dx.doi.org/10.1016/B978-012088469-8.50052-8)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px2.p1.1)\.
- Joulinet al\.\(2017\)A\. Joulin, E\. Grave, P\. Bojanowski, and T\. MikolovBag of tricks for efficient text classification\.InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\),pp\. 427–431\.Note:arXiv:1607\.01759External Links:[Link](https://arxiv.org/abs/1607.01759)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1)\.
- Kleinberg \(1999\)J\. M\. KleinbergAuthoritative sources in a hyperlinked environment\.Journal of the ACM46\(5\),pp\. 604–632\.External Links:[Document](https://dx.doi.org/10.1145/324133.324140)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px2.p1.1)\.
- Kondrupet al\.\(2025\)E\. Kondrup, S\. Sabry, H\. Abdallah, Z\. Yang, J\. Xiong, K\. Pelrine, J\. Zhou, Z\. Guo, M\. M\. Bronstein, J\. Godbout, R\. Rabbany, and S\. HuangCrediBench: building web\-scale network datasets for information integrity\.Note:arXiv:2509\.23340External Links:[Link](https://arxiv.org/abs/2509.23340)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha, S\. Keh, K\. Arora, S\. Garg, R\. Xin, N\. Muennighoff, R\. Heckel, J\. Mercat, M\. Chen, S\. Gururangan, M\. Wortsman, A\. Albalak, Y\. Bitton, M\. Nezhurina, A\. Abbas, C\. Hsieh, D\. Ghosh, J\. Gardner, M\. Kilian, H\. Zhang, R\. Shao, S\. Pratt, S\. Sanyal, G\. Ilharco, G\. Daras, K\. Marathe, A\. Gokaslan, J\. Zhang, K\. Chandu, T\. Nguyen, I\. Vasiljevic, S\. Kakade, S\. Song, S\. Sanghavi, F\. Faghri, S\. Oh, L\. Zettlemoyer, K\. Lo, A\. El\-Nouby, H\. Pouransari, A\. Toshev, S\. Wang, D\. Groeneveld, L\. Soldaini, P\. W\. Koh, J\. Jitsev, T\. Kollar, A\. G\. Dimakis, Y\. Carmon, A\. Dave, L\. Schmidt, and V\. ShankarDataComp\-LM: in search of the next generation of training sets for language models\.InAdvances in Neural Information Processing Systems,Note:arXiv:2406\.11794External Links:[Link](https://arxiv.org/abs/2406.11794)Cited by:[§1](https://arxiv.org/html/2610.13321#S1.p2.1),[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.13321#S6.SS1.p3.1)\.
- Lucyet al\.\(2024\)L\. Lucy, S\. Gururangan, L\. Soldaini, E\. Strubell, D\. Bamman, L\. F\. Klein, and J\. DodgeAboutMe: using self\-descriptions in webpages to document the effects of English pretraining data filters\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2401\.06408External Links:[Link](https://arxiv.org/abs/2401.06408)Cited by:[§4\.2](https://arxiv.org/html/2610.13321#S4.SS2.SSS0.Px1.p2.1)\.
- Messmeret al\.\(2025\)B\. Messmer, V\. Sabolčec, and M\. JaggiEnhancing multilingual LLM pretraining with model\-based data selection\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks\),Note:arXiv:2502\.10361External Links:[Link](https://arxiv.org/abs/2502.10361)Cited by:[Table 10](https://arxiv.org/html/2610.13321#A4.T10),[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p3.1)\.
- Overwijket al\.\(2022\)A\. Overwijk, C\. Xiong, X\. Liu, C\. VandenBerg, and J\. CallanClueWeb22: 10 billion web documents with visual and semantic information\.External Links:2211\.15848,[Link](https://arxiv.org/abs/2211.15848)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px3.p1.1)\.
- Pageet al\.\(1999\)L\. Page, S\. Brin, R\. Motwani, and T\. WinogradThe PageRank citation ranking: bringing order to the web\.Technical reportTechnical Report1999\-66,Stanford InfoLab\.Note:Stanford Technical Report, ILPUBS\-422External Links:[Link](http://ilpubs.stanford.edu:8090/422/)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px2.p1.1)\.
- Pariset al\.\(2026\)M\. Paris, H\. Celikkanat, and L\. FoppianoMeasuring what the crawler sees: discovery curves, core persistence, and shell dynamics in longitudinal web crawls\.External Links:2607\.13636,[Link](https://arxiv.org/abs/2607.13636)Cited by:[§5](https://arxiv.org/html/2610.13321#S5.SS0.SSS0.Px3.p2.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. WolfThe FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks\),Note:arXiv:2406\.17557External Links:[Link](https://arxiv.org/abs/2406.17557)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p2.1)\.
- Penedoet al\.\(2025\)G\. Penedo, H\. Kydlíček, V\. Sabolčec, B\. Messmer, N\. Foroutan, A\. H\. Kargaran, C\. Raffel, M\. Jaggi, L\. Von Werra, and T\. WolfFineWeb2: one pipeline to scale them all—adapting pre\-training data processing to every language\.Technical reportHugging Face\.Note:arXiv:2506\.20920External Links:[Link](https://arxiv.org/abs/2506.20920)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p3.1)\.
- Soldainiet al\.\(2024\)L\. Soldaini, R\. Kinney, A\. Bhagia, D\. Schwenk, D\. Atkinson, R\. Authur, B\. Bogin, K\. Chandu, J\. Dumas, Y\. Elazar, V\. Hofmann, A\. H\. Jha, S\. Kumar, L\. Lucy, X\. Lyu, N\. Lambert, I\. Magnusson, J\. Morrison, N\. Muennighoff, A\. Naik, C\. Nam, M\. E\. Peters, A\. Ravichander, K\. Richardson, Z\. Shen, E\. Strubell, N\. Subramani, O\. Tafjord, P\. Walsh, L\. Zettlemoyer, N\. A\. Smith, H\. Hajishirzi, I\. Beltagy, D\. Groeneveld, J\. Dodge, and K\. LoDolma: an open corpus of three trillion tokens for language model pretraining research\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2402\.00159External Links:[Link](https://arxiv.org/abs/2402.00159)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1)\.
- Suet al\.\(2024\)D\. Su, K\. Kong, Y\. Lin, J\. Jennings, B\. Norick, M\. Kliegl, M\. Patwary, M\. Shoeybi, and B\. CatanzaroNemotron\-CC: transforming Common Crawl into a refined long\-horizon pretraining dataset\.Technical reportNVIDIA\.Note:arXiv:2412\.02595External Links:[Link](https://arxiv.org/abs/2412.02595)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p2.1)\.
- Tsui and Nguyen \(2024\)K\. Tsui and H\. NguyenLow latency CPU based educational value classifier with generic educational value\.Note:Hugging Face modelExternal Links:[Link](https://huggingface.co/kenhktsui/fineweb-edu-fasttext-classifier)Cited by:[Table 10](https://arxiv.org/html/2610.13321#A4.T10),[§3\.1](https://arxiv.org/html/2610.13321#S3.SS1.p2.1)\.
- Yasunagaet al\.\(2022\)M\. Yasunaga, J\. Leskovec, and P\. LiangLinkBERT: pretraining language models with document links\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 8003–8016\.Note:arXiv:2203\.15827External Links:[Link](https://arxiv.org/abs/2203.15827)Cited by:[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2025\)S\. Yu, Z\. Liu, and C\. XiongCraw4LLM: efficient web crawling for LLM pretraining\.InFindings of the Association for Computational Linguistics: ACL 2025,Note:arXiv:2502\.13347External Links:[Link](https://arxiv.org/abs/2502.13347)Cited by:[§1](https://arxiv.org/html/2610.13321#S1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.13321#S1.p2.1),[§2](https://arxiv.org/html/2610.13321#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.13321#S5.SS0.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2610.13321#S6.SS1.p3.1)\.

## Appendix ATemporal Stability of Graph Ranks

Table 2:Spearman rank correlation between a host’s centrality rank in our reference graph and its rank in later Common Crawl graph releases, where1\.01\.0would mean an identical ordering of all hosts\. Both harmonic centrality \(HC\) and PageRank \(PR\) stay above0\.850\.85even∼\\sim15 months apart, so host rankings barely move over a year\. This stability is what lets us pair one aligned graph with document snapshots from different crawls instead of rebuilding the graph each time\.We assess how host centrality rankings change over time by comparing the reference graph with releases up to approximately 15 months later\. Host\-levelHCandPRranks have Spearman correlations of 0\.851–0\.923 across these comparisons \(Table[2](https://arxiv.org/html/2610.13321#A1.T2)\)\. This stability supports using a single reference graph for the document snapshots in our study\.

## Appendix BCrawl\-Yield Detail \(RQ2\)

Table 3:High\-quality documents collected at a 10% crawl budget, as a multiple of a random crawl \(1\.0=1\.0=random,2×=2\\times=twice as many\)\. Centrality ordering \(HC,PR\) gains at most 1\.8×\\times—and drops*below*11for DCLM—while the oracle reaches 4\.0–8\.0×\\times, so most of the achievable gain is left untaken\. The last column shows centrality ordering also roughly halves the number of top\-level domains represented \(HC vs\. random\), narrowing corpus diversity\. This is the per\-classifier, fixed\-budget companion to Figure[2](https://arxiv.org/html/2610.13321#S4.F2); the full budget sweep is Figure[9](https://arxiv.org/html/2610.13321#A5.F9)\.This section gives the per\-classifier crawl\-yield numbers behind §[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\. Table[3](https://arxiv.org/html/2610.13321#A2.T3)fixes the budget at 10% of pages and reports, for each classifier, the high\-quality yield of each ordering policy as a multiple of a random crawl, together with the TLD diversity the policy retains; Figure[9](https://arxiv.org/html/2610.13321#A5.F9)plots the same comparison across all budgets\. It makes the two headline points of RQ2 concrete at a single budget: centrality ordering barely lifts the yield above random \(and drops below it for DCLM\) while the oracle is several times higher, and centrality ordering simultaneously narrows the range of top\-level domains crawled\.

## Appendix CFull Graph\-Classifier Correlation Tables

These tables expand the correlation analysis behind RQ1–RQ2 using alternative classifiers, host\-inclusion thresholds, and aggregation methods\.

### C\.1HC and PageRank Correlations Across Classifiers

Table[4](https://arxiv.org/html/2610.13321#A3.T4)reports correlations of host\-levelHCandPRwith all twelve classifiers, using Spearmanρ\\rho, Kendallτ\\tau, and weighted Spearmanρ\\rho\. Spearman correlations range from−0\.086\-0\.086to\+0\.207\+0\.207\. For each classifier,HCandPRhave the same correlation sign and similar magnitudes\.

Table 4:Host\-level correlations of harmonic centrality and PageRank with twelve classifier scores\. Spearmanρ\\rho, Kendallτ\\tau, and weighted Spearmanρ\\rhoagree in sign for every metric–classifier pair\.
### C\.2Language\-Stratified Correlations \(log⁡HC\\log\\textsc\{HC\},n≥3n\\geq 3\)

Table[5](https://arxiv.org/html/2610.13321#A3.T5)is the full version of the language split behind Figure[3](https://arxiv.org/html/2610.13321#S4.F3): theHC–quality correlation for every classifier, computed on all hosts and separately on English\- and non\-English\-primary hosts\. It shows the sign flip is general \(classifiers tend to correlate more positively within a single language than pooled across languages\) and that the multilingual classifiers \(JQL\-Edu, FineWeb2\-HQ\) are the ones that retain a positive signal on non\-English hosts\.

Table 5:Full language\-stratifiedHCcorrelations\. For JQL\-Edu and FineWeb2\-HQ, “English” rows reflect host\-primary language classification; inference excluded documents mapped as English\.
### C\.3TLD Macro\-Category Breakdown

Table[6](https://arxiv.org/html/2610.13321#A3.T6)breaks theHC–DCLM correlation down by TLD macro\-category\. It is negative precisely within the academic and commercial categories, which concentrate the non\-English hosts that DCLM scores highly, i\.e\., localising the language confound of §[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)to the domains where language and centrality covary most\.

Table 6:HC–DCLM Spearmanρ\\rhoby TLD macro\-category\. Negative within academic and commercial TLDs reflects DCLM’s language inflation of non\-English domains\.
### C\.4Score Distributions by Language Stratum

Table[7](https://arxiv.org/html/2610.13321#A3.T7)gives the full score distributions \(mean, median, and percentiles\) behind Figure[3](https://arxiv.org/html/2610.13321#S4.F3), split by host language\. DCLM’s median score is substantially higher on non\-English hosts than on English hosts, whereas FineWeb\-Edu assigns similar scores to the two groups\. These differences are consistent with language composition contributing to the pooled centrality\-quality correlation\.

Table 7:Score distributions by language stratum \(selected classifiers\)\. JQL\-Edu scores are log\-probabilities\.
### C\.5Full ICC1 Table

Table[8](https://arxiv.org/html/2610.13321#A3.T8)lists the full RQ1 variance decomposition: ICC1 for every classifier atn≥3n\\geq 3documents per host\. The between\-host share is stable across then≥3n\\geq 3,55, and1010thresholds and near one\-half throughout, the property that lets a small per\-host sample stand in for a host \(§[4\.1](https://arxiv.org/html/2610.13321#S4.SS1)\)\.

Table 8:Full ICC1 table \(n≥3n\\geq 3docs per host\)\. ICC1 stable acrossn≥3/5/10n\\geq 3/5/10thresholds\.

## Appendix DDatasets and Models

Our study combines several public Common Crawl artifacts and Hugging Face models across the two data regimes of §[3](https://arxiv.org/html/2610.13321#S3)\. Table[9](https://arxiv.org/html/2610.13321#A4.T9)lists every dataset and which research question uses it\. Table[10](https://arxiv.org/html/2610.13321#A4.T10)lists the exact classifier models we run in the classified\-sample regime\.

Table 9:Datasets used, by data regime; the research question each supports is noted in parentheses\. WARC crawls are extracted and scored by us \(classified\-sample regime\); the host index and GneissWeb annotations are Common Crawl’s precomputed full\-crawl artifacts\.Table 10:Exact Hugging Face models used to classify pages in the classified\-sample regime\. FineWeb\-Edu is a compute\-efficient fastText re\-implementation\([Tsui and Nguyen, 2024](https://arxiv.org/html/2610.13321#bib.bib9)\)of the original Transformer classifier, at comparable accuracy\. The Transformer classifiers \(JQL\-Edu, FineWeb2\-HQ; the latter is the classifier of[Messmer et al\., 2025](https://arxiv.org/html/2610.13321#bib.bib8)\) are frozen\-embedding models with lightweight per\-language heads and are run on non\-English documents only\.
## Appendix ESupplementary Figures

These figures are the visual companions to results summarised in the body: the full crawl\-yield budget sweep behind Figure[2](https://arxiv.org/html/2610.13321#S4.F2)\(§[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\), the temporal robustness of the coverage–quality disconnect \(Table[1](https://arxiv.org/html/2610.13321#S4.T1)\), the composition of the missed high\-quality budget \(§[4\.4](https://arxiv.org/html/2610.13321#S4.SS4)\), and the agreement structure among the twelve classifiers \(§[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\)\.

#### Crawl yield across budgets\.

Figure[9](https://arxiv.org/html/2610.13321#A5.F9)is the full\-budget companion to the fixed\-budget bars of Figure[2](https://arxiv.org/html/2610.13321#S4.F2)\. Sweeping the crawl budget from a small fraction of pages up to the full crawl shows that centrality ordering trails the oracle at*every*budget: the gap is widest at the small budgets where crawl order matters most and closes only as the crawl approaches completion, so the fixed\-10% snapshot in the body is representative rather than a cherry\-picked operating point\.

Figure 9:Full budget sweep behind Figure[2](https://arxiv.org/html/2610.13321#S4.F2): high\-quality documents collected as the crawl budget grows, divided by what a random crawl collects at the same budget \(log\-scaledyy\-axis; the dashed line at11means “no better than random”\)\. Ordering hosts by harmonic centrality \(HC\), PageRank \(PR\), or Common Crawl’s current allocation hugs the random line and stays far below the*oracle*; the shaded band shows the gap between centrality\-based yield and the oracle upper bound, e\.g, widest at small budgets, where crawl order matters most, and closing as the budget approaches a full crawl\. For DCLM, centrality is even worse than random\.
#### Persistence across crawls\.

Figure[10](https://arxiv.org/html/2610.13321#A5.F10)is the plotted form of Table[1](https://arxiv.org/html/2610.13321#S4.T1): the coverage–quality disconnect of RQ4 holds in each of 6 crawls spanning 2021–2024, both as a near\-zero, slightly negative correlation \(panel a\) and as tens of millions of uncrawled high\-quality URLs per crawl \(panel b\)\. The misallocation is thus a standing property of the deployed crawl, not a single\-crawl artifact\.

Figure 10:The coverage–quality disconnect across 6 Common Crawl snapshots \(2021–2024\)\. \(a\) The Spearman correlation between crawl coverage and host quality density is near zero and negative in every crawl \(\-0\.057 to \-0\.151\)\. \(b\) Each crawl leaves tens of millions of high\-quality URLs uncrawled on hidden\-gem hosts\. The misallocation is a persistent, not a single\-crawl, artifact—the plotted form of Table[1](https://arxiv.org/html/2610.13321#S4.T1)\.
#### Where the missed budget sits\.

Figure[11](https://arxiv.org/html/2610.13321#A5.F11)characterises the hidden\-gem hosts in aggregate \(no individual host is named\)\. Country\-code publisher TLDs—led by\.uk—are heavily over\-represented relative to their host count, and among the hosts a keyword classifier can label, news/media outlets and scholarly repositories dominate\. This is what the recoverable pool of §[4\.4](https://arxiv.org/html/2610.13321#S4.SS4)concretely consists of, and it motivates the targeted re\-crawl proposed in the future work\.

Figure 11:Where the missed high\-quality budget sits, by \(a\) top\-level domain and \(b\) content category \(aggregates only; no host is named\)\. Country\-code publisher TLDs are over\-represented relative to their host count—\.ukalone holds 23% of the missed budget from just 6\.1% of hosts at 34% median coverage\. Among the hosts a keyword classifier can label, news/media and scholarly hosts dominate; the remaining∼\\sim80% is uncategorised \(a keyword lower bound\), so panel \(b\) shows only the labelled share and omits the “Other” bucket\. “Reference” denotes encyclopedic/reference sites \(e\.g\. Wikipedia\-family, dictionaries, encyclopedias\); it is near zero because the major reference sites are already well crawled, so only a few dozen such hosts remain under\-crawled gems\.
#### How the classifiers relate\.

Figure[12](https://arxiv.org/html/2610.13321#A5.F12)gives the pairwise Spearman correlation between the twelve classifiers’ scores across the 462,494 core documents \(every document scored by all classifiers\)\. It contextualises two threads in the body\. DCLM is largely orthogonal to the other quality signals—weakly correlated with most and*negative*against the Nemotron heads \(−0\.459\-0\.459with Nemotron\-Q\)—consistent with its outlier behaviour throughout \(§[4\.2](https://arxiv.org/html/2610.13321#S4.SS2)\) and its lone non\-agreement in the coverage\-robustness check \(Figure[8](https://arxiv.org/html/2610.13321#S4.F8)\)\. The GneissWeb topic scorers and the Nemotron heads each form positively correlated blocks, and FineWeb\-Edu aligns most closely with GneissWeb\-Q, so the general\-purpose “quality” classifiers largely share a common signal that DCLM does not; FineWeb2\-HQ, the multilingual model, correlates weakly or negatively with the English\-trained ones\.

![Refer to caption](https://arxiv.org/html/2610.13321v1/fig_classifier_corr.png)Figure 12:Pairwise Spearman rank correlation between the twelve classifiers’ document scores \(462,494 core documents; the diagonal is11by construction\)\. The scale runs from−1\-1\(blue\) through00\(white\) to\+1\+1\(red\)\. DCLM is largely orthogonal to the other quality signals and*negative*against the Nemotron heads, whereas the GneissWeb topic scorers and the Nemotron heads each form positively correlated blocks\. This is the full pairwise view behind the single\-classifier robustness of the coverage result \(Figure[8](https://arxiv.org/html/2610.13321#S4.F8)\)\.