Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining

# Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
Source: [https://arxiv.org/html/2608.03089](https://arxiv.org/html/2608.03089)
Hai Wang∗†Chenhao Wang∗Qifeng Cai∗Yixiu LiuMiao Peng Nuo ChenYuanlin TuChengcheng XuFeng Zhang Hunyuan Team, Tencent

###### Abstract

Large\-scale pretraining corpora contain substantial duplicate content\. Although document\-level deduplication is widely used, removing subdocument\-level redundancy remains challenging\. At corpus scale, suffix\-array\-based methods are commonly applied independently within shards, leaving cross\-shard duplicates undetected and making the resulting retention behavior sensitive to the sharding configuration\. Hash\-based methods enable global exact duplicate counting, but often rely on fixed copy\-retention policies that cannot accommodate heterogeneous repetition patterns\. We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention\. It identifies duplicate groups through natural\-boundary segmentation, normalized exact hashing, and distributed aggregation, and then applies an explicit frequency\- and length\-aware retention policy that allocates an adaptive copy budget to each group, retaining more copies of low\-frequency or short repetitions while more aggressively deleting high\-frequency or long ones\. Experiments on FineWeb\-Edu and a code\-containing web corpus show that models trained on data processed by our method achieve the best overall performance among the evaluated settings\. These results underscore the importance of explicit copy\-retention control\.

††footnotetext:∗Equal contribution\.†Project leader\.## 1Introduction

Pretraining is the core stage during which large language models acquire general knowledge and foundational capabilities, and its effectiveness depends not only on the scale of training data but also on data quality\(Hoffmannet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib38); Li and others,[2024](https://arxiv.org/html/2608.03089#bib.bib8); Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11)\)\. However, large\-scale pretraining data often contain substantial duplicate content, such as web templates, announcements, quoted text, copied code, and repeated segments arising from different versions\(Dodgeet al\.,[2021](https://arxiv.org/html/2608.03089#bib.bib35); Leeet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib4); Kohlschütteret al\.,[2010](https://arxiv.org/html/2608.03089#bib.bib62)\)\. Such duplication wastes limited training compute and can cause a disproportionate degradation in model performance that may not be fully offset by training on additional tokens\(Hernandezet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib19)\)\. Consequently, deduplication has become a critical step in the data preprocessing pipeline for large language model pretraining\.

Most existing pretraining pipelines include document\-level deduplication, typically using exact hashing or MinHash\-based near\-duplicate detection\(Broder,[1997](https://arxiv.org/html/2608.03089#bib.bib1); Gaoet al\.,[2020](https://arxiv.org/html/2608.03089#bib.bib6); Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11); Grattafiori and others,[2024](https://arxiv.org/html/2608.03089#bib.bib7)\)\. These methods are efficient and scalable, but treat the entire document as the unit of deduplication\. In real\-world corpora, however, repeated content often covers only part of a document\(Leeet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib4)\)\. For example, documents with otherwise distinct content may share web templates, navigation bars, copyright notices, quoted passages, or code snippets\. Lowering the document\-similarity threshold may detect more such local overlap, but also risks removing documents that contain only limited shared text alongside substantial unique content\. Document\-level deduplication therefore struggles to remove local redundancy without discarding valuable document\-specific content\.

To address this limitation, several data processing pipelines adopt subdocument deduplication, which identifies and removes repeated regions within documents at finer granularities, such as paragraphs, sentences, lines, or substrings\(Wenzeket al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib5); Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11); Gohariet al\.,[2026](https://arxiv.org/html/2608.03089#bib.bib63)\)\. Prior work has shown that removing repeated substrings can reduce verbatim memorization and improve training efficiency\(Leeet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib4)\)\. Subdocument deduplication raises three central questions: \(1\)*how duplicate content should be defined and detected*, \(2\)*how detection can be scaled to large\-scale corpus*, and \(3\)*how many copies of each detected duplicate should be retained*\.

The definition of duplication depends on the detection unit and matching criterion\. Suffix arrays can identify variable\-length exact repetitions without relying on predefined segmentation boundaries, whereas segmentation\-based methods first divide documents into natural units and then match these units using exact or approximate signals\. Although suffix arrays provide flexible substring\-level detection, constructing and querying a single global suffix array is costly at corpus scale\. Practical implementations therefore commonly partition the corpus into shards and perform matching independently within each shard\(Leeet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib4); Penedoet al\.,[2023](https://arxiv.org/html/2608.03089#bib.bib10); Gohariet al\.,[2026](https://arxiv.org/html/2608.03089#bib.bib63)\)\. Because copies assigned to different shards are never compared, shard\-local matching observes only the within\-shard portion of each global duplicate group and does not directly reveal its global frequency\. Consequently, the deduplication result is sensitive to both the number and composition of shards, which determine which duplicate copies are placed together and can therefore be detected\. In contrast, segmentation\-based hashing naturally maps to distributed key–value aggregation, allowing identical units to be counted globally regardless of their physical partitioning\. Its main limitation is that repetitions remain undetected when they either cross predefined segmentation boundaries or occur as substrings within a segment\.

Even after duplicate groups have been identified, deciding how many copies to retain remains a critical issue\. In sharded suffix\-array pipelines, copy retention is implicitly induced by shard\-local co\-occurrence rather than explicitly controlled according to the global frequency of each duplicate group\. Although hash\-based aggregation can directly compute global duplicate frequencies, existing pipelines often apply frequency\-independent retention policies, such as keep\-one or keep\-kk\(Wenzeket al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib5); Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11)\)\. Such policies cannot adapt to the heterogeneous repetition patterns found in real\-world corpora\. High\-frequency repeated segments are more likely to correspond to webpage templates, automatically generated content, or other low\-information\-density structures and may therefore warrant stronger compression\(Kohlschütteret al\.,[2010](https://arxiv.org/html/2608.03089#bib.bib62); Wenzeket al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib5)\)\. In contrast, low\-frequency repetitions may arise from reasonable quotations, version evolution, or localized content reuse and may still contain useful information\(Muennighoffet al\.,[2023](https://arxiv.org/html/2608.03089#bib.bib20)\)\. A suitable retention policy should therefore adapt the retained copy budget to the frequency and length of each duplicate group rather than assigning the same budget uniformly\.

To address these limitations, we propose a scalable subdocument deduplication framework that explicitly decouples duplicate detection from copy retention\. For duplicate detection, we segment documents at natural boundaries and use normalized exact hashing with distributed aggregation, allowing all occurrences of the same duplicate unit to be counted globally regardless of shard placement\. For copy retention, we introduce an explicit retention function that adaptively allocates the copy budget according to the global frequency and text length of each duplicate group\. The resulting policy treats low\-frequency repetitions conservatively, applies stronger deletion to high\-frequency repetitions, and further reduces the retention budget for long repeated spans that are more likely to reflect template reuse or direct copying\. Coherence\-preserving deletion is then applied to reduce document fragmentation\. Experiments on FineWeb\-Edu and a code\-containing web corpus show that our method achieves the best overall performance on FineWeb\-Edu and consistent gains across all evaluated benchmarks on the code\-containing web corpus\.

Our contributions are as follows:

- •We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention\. Natural\-boundary segmentation, normalized exact hashing, and distributed aggregation enable global duplicate counting without making duplicate visibility dependent on shard placement\.
- •We introduce an explicit frequency\- and length\-aware copy\-retention function that adaptively assigns a retention budget to each duplicate group, applying stronger compression to high\-frequency or long repetitions while treating low\-frequency repetitions more conservatively\.
- •We evaluate the framework on FineWeb\-Edu and a code\-containing web corpus\. Models trained on the resulting data achieve the best overall performance on the FineWeb\-Edu evaluation suite and consistent gains across all evaluated benchmarks in the code\-containing web setting\.

## 2Related Work

### 2\.1Subdocument Duplicate Detection

Existing methods for subdocument duplicate detection mainly differ in their detection units and matching criteria\. Suffix\-array\-based methods identify variable\-length exact repeated substrings without requiring predefined segmentation boundaries\(Manber and Myers,[1993](https://arxiv.org/html/2608.03089#bib.bib3)\)\. Lee et al\.\(Leeet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib4)\)develop a scalable exact\-substring deduplication method based on suffix arrays to detect and remove repeated spans above a predefined length threshold\. RefinedWeb\(Penedoet al\.,[2023](https://arxiv.org/html/2608.03089#bib.bib10)\)applies exact\-substring deduplication in its web\-scale processing pipeline, while GneissWeb\(Gohariet al\.,[2026](https://arxiv.org/html/2608.03089#bib.bib63)\)extends this line of work with sharded exact\-substring deduplication\. Segmentation\-based methods instead divide documents into predefined natural units and compare their exact or normalized representations\. CCNet\(Wenzeket al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib5)\), for example, performs paragraph\-level exact deduplication\. DCLM\(Li and others,[2024](https://arxiv.org/html/2608.03089#bib.bib8)\)uses Bloom\-filter\-based n\-gram matching for document\- and paragraph\-level deduplication\. Suffix\-array methods offer flexible matching of variable\-length spans, whereas predefined\-unit and n\-gram\-based methods are easier to distribute but depend on the selected units or overlap criterion\.

### 2\.2Copy\-Retention Strategies

Once duplicate content is detected, existing pipelines generally apply fixed deletion or retention rules\. In web\-scale exact\-substring pipelines, matching may be performed independently within shards, such that only occurrences assigned to the same shard can be compared and removed\(Gohariet al\.,[2026](https://arxiv.org/html/2608.03089#bib.bib63)\)\. The resulting number of retained copies therefore depends implicitly on the corpus partitioning\. Segmentation\-based pipelines more commonly use explicit but fixed policies, such as retaining one occurrence per duplicate group or filtering units whose frequencies exceed a predefined threshold\(Wenzeket al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib5); Grattafiori and others,[2024](https://arxiv.org/html/2608.03089#bib.bib7)\)\. FineWeb\(Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11)\)evaluates several global line\-level keep\-one variants and reports that they underperform its per\-snapshot MinHash\-deduplicated baseline\. Other work controls the contribution of repetitive content without directly deleting all detected copies\. SoftDedup\(Heet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib12)\)downweights documents according to their estimated commonness, while FineWeb2\(Penedoet al\.,[2025](https://arxiv.org/html/2608.03089#bib.bib13)\)uses duplicate\-cluster information to construct deduplication\-aware resampling strategies\.

## 3Preliminaries

In this section, we first analyze the frequency, length, and content characteristics of duplicate chunks to clarify the basic requirements that a copy retention strategy should satisfy\. We then examine the implicit copy retention behavior induced by sharded suffix array deduplication\.

### 3\.1Characteristics of Duplicate Chunks

#### Frequency and content\.

After applying global document\-level MinHash deduplication, we segment the resulting corpus into line\- and sentence\-level chunks\. We then normalize the chunks and compute the global occurrence frequency of each normalized chunk across the corpus\. For a normalized chunkzz, letC​\(z\)C\(z\)denote its global frequency\. The left panel of Figure[1](https://arxiv.org/html/2608.03089#S3.F1)reports the share of analyzed text length contributed by chunks in each frequency bucket\. Frequencies from 1 to 20 are shown individually, while the final bucket aggregates all chunks withC​\(z\)\>20C\(z\)\>20\.

The distributions exhibit pronounced long tails at both segmentation granularities\. Unique chunks withC​\(z\)=1C\(z\)=1account for 54\.2% and 42\.9% of the analyzed text length at the line and sentence levels, respectively\. Low\-frequency duplicate chunks withC​\(z\)∈\{2,3,4\}C\(z\)\\in\\\{2,3,4\\\}contribute a further 21\.8% and 23\.9%\. At the other extreme, chunks occurring more than 20 times still account for 11\.1% and 16\.7% of the analyzed text length\. These results show that the corpus contains both substantial low\-frequency repetition and a non\-negligible high\-frequency tail, motivating a retention policy that treats different frequency regimes differently\.

To examine the relationship between frequency and content type, we stratify duplicate chunks by frequency and manually inspect representative duplicate groups from each stratum\. High\-frequency duplicates predominantly consist of webpage templates, navigation elements, copyright and license statements, advertisements, automatically generated messages, and other site\-level boilerplate\. In contrast, low\-frequency duplicates exhibit more diverse content, including ordinary natural\-language paragraphs, reasonable quotations, text shared across different versions, and localized reposted content\. High\-frequency duplicates are therefore more likely to represent templated and highly redundant content, for which retaining additional copies provides little marginal information\. Low\-frequency duplicates, by contrast, are more likely to contain information worth retaining\.

#### Length and content\.

We further perform stratified sampling and content inspection on duplicate chunks of varying lengths within the same or similar frequency ranges\. Results show that longer duplicate chunks more often correspond to complete webpage templates, fixed disclaimers, or large copied passages; shorter duplicate chunks are more heterogeneous in content, including titles, common phrases, and other short expressions\.

This distinction suggests that, after controlling for frequency, chunk length still provides additional information regarding the degree of content redundancy\. When long chunks are repeated identically across multiple positions, they are more likely to arise from full\-template reuse or direct copying, and their different copies typically have low marginal information\. In contrast, short text segments may appear independently in different contexts; identical surface forms do not necessarily imply actual copying relationships, and overly aggressive deletion may inadvertently remove legitimate linguistic reuse\. Therefore, for duplicate chunks with the same frequency, the retention strategy should apply stronger compression to longer chunks\.

These observations suggest that retention should depend on both frequency and length\. Frequency captures how widely a chunk is repeated, while length helps distinguish likely copied or templated content from short expressions that may recur naturally\.

![Refer to caption](https://arxiv.org/html/2608.03089v1/x1.png)Figure 1:The left panel shows the length\-weighted distributions of global duplicate frequency for line\- and sentence\-level chunks after document\-level MinHash deduplication\. Frequencies from 1 to 20 are shown individually, while the final bucket aggregates chunks withC​\(z\)\>20C\(z\)\>20\. The right panel shows the expected retained fractionrN​\(C\)r\_\{N\}\(C\)under singleton\-only shard\-local retention for different effective shard countsNN\.

### 3\.2Implicit Retention Behavior of Shard\-Local Deduplication

The foregoing analysis suggests that a copy\-retention strategy should treat low\-frequency duplicates conservatively while applying stronger compression to high\-frequency duplicates\. However, directly specifying a function with this behavior would be largely ad hoc\. We observe that random shard\-local deduplication naturally induces a similar frequency\-dependent retention pattern: low\-frequency duplicate groups are more likely to appear as isolated copies within shards, whereas high\-frequency groups are more likely to contain multiple co\-located copies and undergo stronger deletion\. We therefore analyze this implicit retention behavior and use it to derive an explicit retention function\.

At corpus scale, suffix\-array deduplication is typically performed independently across multiple shards because constructing and querying a single global suffix array is computationally expensive\(Leeet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib4); Penedoet al\.,[2023](https://arxiv.org/html/2608.03089#bib.bib10); Gohariet al\.,[2026](https://arxiv.org/html/2608.03089#bib.bib63)\)\. Because only copies assigned to the same shard are compared, whether a copy is retained depends on the number of matching copies within its shard rather than directly on the global occurrence count\. Under the shard\-local deletion abstraction considered here, a copy is retained only when it is the sole occurrence of its duplicate group within a shard; if two or more copies occur in the same shard, the corresponding repeated regions are removed\. We refer to this behavior as*singleton\-only retention*\.

To isolate the effect of shard count, we analyze an idealized random\-sharding model\. Suppose that a duplicate group containsCCcopies in the global corpus, whereC∈ℕ\+C\\in\\mathbb\{N\}\_\{\+\}, and that each copy is assigned independently and uniformly at random to one ofNNshards, whereN∈ℕN\\in\\mathbb\{N\}andN≥2N\\geq 2\. LetXiX\_\{i\}denote the number of copies assigned to theii\-th shard\. The shard\-level copy counts jointly follow a multinomial distribution,\(X1,…,XN\)∼x⁡\(C;1N,…,1N\),\(X\_\{1\},\\ldots,X\_\{N\}\)\\sim\\operatorname\{x\}\\left\(C;\\frac\{1\}\{N\},\\ldots,\\frac\{1\}\{N\}\\right\),and the marginal distribution of eachXiX\_\{i\}isXi∼Binomial⁡\(C,1N\)\.X\_\{i\}\\sim\\operatorname\{Binomial\}\\left\(C,\\frac\{1\}\{N\}\\right\)\.Under singleton\-only retention, theii\-th shard contributes one retained copy if and only ifXi=1X\_\{i\}=1\. LetR=∑i=1N𝟏​\[Xi=1\]R=\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\[X\_\{i\}=1\]denote the total number of retained copies\. Its expectation is

gN​\(C\)\\displaystyle g\_\{N\}\(C\)=𝔼​\[R\]=∑i=1NPr⁡\(Xi=1\)\\displaystyle=\\mathbb\{E\}\[R\]=\\sum\_\{i=1\}^\{N\}\\Pr\(X\_\{i\}=1\)\(1\)=N​Pr⁡\(X1=1\)=N​\(C1\)​1N​\(1−1N\)C−1\\displaystyle=N\\Pr\(X\_\{1\}=1\)=N\\binom\{C\}\{1\}\\frac\{1\}\{N\}\\left\(1\-\\frac\{1\}\{N\}\\right\)^\{C\-1\}=C​\(1−1N\)C−1\.\\displaystyle=C\\left\(1\-\\frac\{1\}\{N\}\\right\)^\{C\-1\}\.Therefore, the expected retention ratio is

rN​\(C\)=gN​\(C\)C=\(1−1N\)C−1\.r\_\{N\}\(C\)=\\frac\{g\_\{N\}\(C\)\}\{C\}=\\left\(1\-\\frac\{1\}\{N\}\\right\)^\{C\-1\}\.\(2\)
This expression characterizes the expected behavior under uniform random assignment\. Source\-aware or otherwise non\-random shard construction may produce different retention behavior, further illustrating the dependence of shard\-local deduplication on the physical partitioning scheme\.

Since0<1−1N<10<1\-\\frac\{1\}\{N\}<1, the retention ratiorN​\(C\)r\_\{N\}\(C\)decreases monotonically with global frequencyCC\. As shown in the right panel of Figure[1](https://arxiv.org/html/2608.03089#S3.F1), low\-frequency duplicate groups rarely contain multiple copies within the same shard and therefore retain most of their copies\. AsCCincreases, intra\-shard co\-occurrence becomes more likely, and the retained fraction decreases\. A largerNNproduces a slower decay, whereas a smallerNNresults in stronger compression\.

The induced curve exhibits the desired frequency dependence\. However, in shard\-local suffix\-array pipelines, its shape is coupled to the physical sharding configuration and cannot be controlled independently of data partitioning\. We therefore abstract the curve from the sharding mechanism and use it as the basis of an explicit frequency\-aware retention function\. After global hash\-based counting,NNis reinterpreted as an effective hyperparameter controlling compression strength rather than as the number of physical shards\. The length\-aware adjustment is introduced in Section[4\.2](https://arxiv.org/html/2608.03089#S4.SS2)\.

## 4Method

### 4\.1Overall Pipeline

Our framework consists of five stages: segmentation, text normalization, global duplicate counting, document reconstruction, and retention\-guided deletion\.

#### Segmentation\.

Given a document collection𝒟\\mathcal\{D\}, we segment each document into a sequence of contiguous text units, referred to as*chunks*\. Initial chunks are obtained at natural boundaries, such as paragraph separators, line breaks, and sentence\-ending punctuation\(Wenzeket al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib5); Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11)\)\. We then scan the chunks from left to right\. If a chunk is shorter than the minimum segmentation thresholdτseg\\tau\_\{\\mathrm\{seg\}\}, it is successively merged with the following chunks until its accumulated length reachesτseg\\tau\_\{\\mathrm\{seg\}\}\. A residual chunk at the end of a document is retained even if its length remains belowτseg\\tau\_\{\\mathrm\{seg\}\}\. All chunk lengths in this stage are measured in characters\. This procedure prevents short titles, list markers, and isolated phrases from becoming independent detection units while preserving the complete document content\.

For documents containing code, we additionally preserve code structure\. Markdown code chunks and brace\-delimited structures with reliably matched boundaries are treated as indivisible units and are not segmented internally\. This rule reduces the risk of breaking syntactic or structural integrity during segmentation, a concern that is specific to source\-code corpus construction and deduplication\(Kocetkovet al\.,[2022](https://arxiv.org/html/2608.03089#bib.bib9)\)\.

#### Text normalization\.

Letnorm⁡\(x\)\\operatorname\{norm\}\(x\)denote the normalized text of a chunk occurrencexx\. For natural\-language chunks, numeric expressions such as dates, page numbers, prices, and statistical values are replaced with unified placeholders, allowing structurally identical template variants to be grouped together\(Wenzeket al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib5); Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11)\)\. For code chunks, no content normalization is applied, and we setnorm⁡\(x\)=x\\operatorname\{norm\}\(x\)=xto avoid grouping code that differs in constants or other potentially meaningful tokens\. Normalization is deterministic and is applied before hashing\.

Chunk occurrences with identical normalized text form a*duplicate group*\. Normalization is used only for matching and frequency counting; the original text and positional information of each occurrence are retained for document reconstruction and output\.

#### Counting\.

For each chunk occurrencexx, we compute a content hash from its normalized text ash​\(x\)=H​\(norm⁡\(x\)\)h\(x\)=H\(\\operatorname\{norm\}\(x\)\), whereH​\(⋅\)H\(\\cdot\)denotes a content hash function\. The resulting hash is used as the key for distributed aggregation\. Let𝒳​\(𝒟\)\\mathcal\{X\}\(\\mathcal\{D\}\)denote the multiset of all chunk occurrences extracted from𝒟\\mathcal\{D\}\. For a normalized text representationzz, its global frequency is defined as

C​\(z\)=∑x∈𝒳​\(𝒟\)𝟏​\[norm⁡\(x\)=z\],C\(z\)=\\sum\_\{x\\in\\mathcal\{X\}\(\\mathcal\{D\}\)\}\\mathbf\{1\}\\\!\\left\[\\operatorname\{norm\}\(x\)=z\\right\],\(3\)where𝟏​\[⋅\]\\mathbf\{1\}\[\\cdot\]denotes the indicator function\. This stage is a standard key–value aggregation operation and can be implemented using the shuffle\-and\-aggregate pattern in MapReduce or Spark\(Dean and Ghemawat,[2004](https://arxiv.org/html/2608.03089#bib.bib36); Zahariaet al\.,[2012](https://arxiv.org/html/2608.03089#bib.bib37)\)\.

#### Reconstruction\.

After frequency counting, we join the duplicate\-group metadata, including its global frequency and normalized\-text length, back to the corresponding chunk records\. We then reconstruct each document according to its document identifier and original chunk positions\. This stage restores document order and context for subsequent deletion\.

#### Deletion\.

The retention functionT​\(C,L\)T\(C,L\)introduced in Section[4\.2](https://arxiv.org/html/2608.03089#S4.SS2)assigns an initial retention budget to each duplicate group\. For a duplicate group with frequencyCCand chunk lengthLL, we order itsCCoccurrences by document ID and retain the firstT​\(C,L\)T\(C,L\)occurrences\. Letrank⁡\(o\)∈\{1,…,C\}\\operatorname\{rank\}\(o\)\\in\\\{1,\\ldots,C\\\}denote the position of occurrenceooin the resulting order\. Its initial retention decision is

keep0⁡\(o\)=𝟏​\[rank⁡\(o\)≤T​\(C,L\)\]\.\\operatorname\{keep\}\_\{0\}\(o\)=\\mathbf\{1\}\\\!\\left\[\\operatorname\{rank\}\(o\)\\leq T\(C,L\)\\right\]\.\(4\)When the retention boundary falls within a group of occurrences sharing the same document ID, we retain the entire group, including those ranked immediately beyondT​\(C,L\)T\(C,L\)\. All remaining occurrences beyond the retention budget are marked as candidate deletion chunks\. However, we do not delete these candidates independently\. Instead, we group consecutive candidate deletion chunks within each document to reduce fragmentation caused by isolated local deletions\. Let\(xs,…,xt\)\(x\_\{s\},\\ldots,x\_\{t\}\)denote a maximal contiguous run of candidate deletion chunks, and letℓ​\(xj\)\\ell\(x\_\{j\}\)denote the original character length of chunkxjx\_\{j\}\. The entire run is deleted only if

∑j=stℓ​\(xj\)≥τdel\.\\sum\_\{j=s\}^\{t\}\\ell\(x\_\{j\}\)\\geq\\tau\_\{\\mathrm\{del\}\}\.\(5\)Otherwise, all chunks in the run are retained\. Because the rule is applied to maximal contiguous runs, the accumulated length is reset whenever a non\-candidate chunk is encountered\.

The chunk lengthLLat the duplicate\-group level and the run length at the document level serve different purposes: the former adjusts the initial retention budget of a duplicate group, whereas the latter determines whether a candidate region is sufficiently long to be removed without excessive fragmentation\. Consequently,T​\(C,L\)T\(C,L\)specifies only the initial retention budget, and the final number of retained occurrences may be larger after coherence\-aware deletion is applied\.

### 4\.2Frequency\- and Length\-Aware Retention Function

Based on the analysis in Section[3](https://arxiv.org/html/2608.03089#S3), we combine a frequency\-derived base budget with a length\-aware adjustment to determine the number of initially retained copies\.

#### Frequency\-Aware Retention Budget\.

Under the shard\-based singleton\-only retention behavior analyzed in Section[3\.2](https://arxiv.org/html/2608.03089#S3.SS2), the expected number of retained copies for a duplicate group with global frequencyCCisgN​\(C\)=C​\(1−1N\)C−1g\_\{N\}\(C\)=C\\left\(1\-\\frac\{1\}\{N\}\\right\)^\{C\-1\}, and the corresponding retained fraction isrN​\(C\)=\(1−1N\)C−1r\_\{N\}\(C\)=\\left\(1\-\\frac\{1\}\{N\}\\right\)^\{C\-1\}\. The retained fractionrN​\(C\)r\_\{N\}\(C\)decreases monotonically withCC, thereby applying progressively stronger relative compression to higher\-frequency duplicate groups\. We use the corresponding expected retained countgN​\(C\)g\_\{N\}\(C\)as the base retention budget\.

The parameterNNoriginally denotes the number of physical shards\. After adopting distributed exact hash counting, we reinterpret it as an effective hyperparameter controlling the shape of the frequency\-dependent retention curve\. A largerNNcauses the retained fraction to decay more slowly with frequency and therefore preserves more low\- and medium\-frequency repetitions, whereas a smallerNNapplies stronger compression to medium\- and high\-frequency duplicate groups\.

#### Length\-Aware Adjustment\.

Section[3\.1](https://arxiv.org/html/2608.03089#S3.SS1)shows that chunk length provides additional information about the likely redundancy of duplicate content\. The base budgetgN​\(C\)g\_\{N\}\(C\)depends only on global frequency and therefore assigns the same budget to duplicate groups with identical frequencies but different lengths\. However, longer repeated chunks are more likely to correspond to templates, fixed disclaimers, or large copied passages and should therefore receive stronger compression\.

LetLLdenote the character length of the normalized chunk text\. Because chunk lengths are unbounded and their distributions can vary substantially across corpora, we introduce a reference lengthL0\>0L\_\{0\}\>0that controls the decay rate and saturation point of the length adjustment\.

The length adjustment functionα​\(L\)\\alpha\(L\)should take values in\[0,1\]\[0,1\], decrease monotonically withLL, and remain constant onceL≥L0L\\geq L\_\{0\}\. We use the following truncated linear function:

α​\(L\)=max⁡\{0,1−LL0\}\.\\alpha\(L\)=\\max\\\!\\left\\\{0,\\,1\-\\frac\{L\}\{L\_\{0\}\}\\right\\\}\.\(6\)When0≤L<L00\\leq L<L\_\{0\},α​\(L\)\\alpha\(L\)decreases linearly from11to0, assigning higher weights to shorter chunks and lower weights to longer chunks\. ForL≥L0L\\geq L\_\{0\}, the adjustment saturates atα​\(L\)=0\\alpha\(L\)=0\. This form introduces only one parameter, is monotonic and interpretable, and avoids imposing an unbounded length penalty on extremely long chunks\.

#### Joint Frequency–Length Retention\.

Combining the frequency\-derived budget with the length adjustment, we define the initial number of retained copies for each duplicate group as

T​\(C,L\)=⌈1\+\(gN​\(C\)−1\)​α​\(L\)⌉\.T\(C,L\)=\\left\\lceil 1\+\\bigl\(g\_\{N\}\(C\)\-1\\bigr\)\\alpha\(L\)\\right\\rceil\.\(7\)The resulting rule has intuitive boundary behavior\. WhenLLis small relative toL0L\_\{0\},α​\(L\)\\alpha\(L\)remains close to11, and the retention budget is primarily determined bygN​\(C\)g\_\{N\}\(C\)\. For duplicate groups withgN​\(C\)\>1g\_\{N\}\(C\)\>1, increasingLLmonotonically decreases the underlying real\-valued budget fromgN​\(C\)g\_\{N\}\(C\)toward11, while the integer\-valuedT​\(C,L\)T\(C,L\)changes in discrete steps because of the ceiling operation\. WhengN​\(C\)≤1g\_\{N\}\(C\)\\leq 1, the construction yieldsT​\(C,L\)=1T\(C,L\)=1for allLL, so the length adjustment has no further effect\. OnceL≥L0L\\geq L\_\{0\},α​\(L\)=0\\alpha\(L\)=0and the initial retention budget becomesT​\(C,L\)=1T\(C,L\)=1\.

The additive anchor at11guarantees that at least one occurrence is initially retained, while the ceiling operation converts the real\-valued budget into a conservative integer budget and avoids reducing the retained count through downward rounding\. The resulting function is a deterministic extension of the shard\-based singleton\-only retention curve rather than a stochastic simulation\.

For anyC≥1C\\geq 1andN\>1N\>1, the retention budget satisfies1≤T​\(C,L\)≤C1\\leq T\(C,L\)\\leq C\. The parameterL0L\_\{0\}controls the decay rate and saturation point of the length adjustment\. For chunks satisfyingL≥L0L\\geq L\_\{0\}, the adjustment reaches its maximum strength, and the duplicate group retains one initial occurrence\.

Table 1:Downstream performance after 36K training steps on FineWeb\-Edu processed using different deduplication methods\. Average scores are computed over all eight benchmarks\.

## 5Experiments

### 5\.1Experimental Setup

#### Implementation Details\.

We apply the deduplication methods to two source corpora and train Hy\-MT2\-30B\-A3B models\(Zhenget al\.,[2026](https://arxiv.org/html/2608.03089#bib.bib59)\)on the resulting corpora\. For the general\-web setting, the source corpus is FineWeb\-Edu\(Penedoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib11)\), containing 6\.28T tokens before additional deduplication\. For the code\-containing web setting, we construct a 561\.53B\-token source corpus by applying a code classifier to webpages from Common Crawl\. All corpus sizes are measured using the Hy\-MT2\-30B\-A3B tokenizer\. Each optimization step processes 2,000 sequences of length 4,096, corresponding to 8\.192M tokens\. All models are trained for 36,000 steps, consuming 294\.9B tokens in total\. The learning rate is linearly warmed up to a peak value of3×10−43\\times 10^\{\-4\}over the first 2,000 steps\. We evaluate the models every 2,000 steps from step 2,000 through step 36,000\. For deduplication, we setτseg=32\\tau\_\{\\mathrm\{seg\}\}=32characters,τdel=100\\tau\_\{\\mathrm\{del\}\}=100characters, andL0=512L\_\{0\}=512characters\. We considerN∈\[20,100\]N\\in\[20,100\]and useN=100/3N=100/3in the final configuration\. For the document\-level MinHash baseline, we use 5\-gram shingles, 128 MinHash permutations, and a target Jaccard similarity of0\.70\.7\.

#### Benchmarks and metrics\.

For the FineWeb\-Edu experiment, we evaluate Natural Questions \(NQ\), TriviaQA, HellaSwag, PIQA, GSM8K, MATH, MMLU, and CMMLU\(Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib44); Joshiet al\.,[2017](https://arxiv.org/html/2608.03089#bib.bib45); Zellerset al\.,[2019](https://arxiv.org/html/2608.03089#bib.bib24); Bisket al\.,[2020](https://arxiv.org/html/2608.03089#bib.bib60); Cobbeet al\.,[2021](https://arxiv.org/html/2608.03089#bib.bib26); Hendryckset al\.,[2021b](https://arxiv.org/html/2608.03089#bib.bib25);[a](https://arxiv.org/html/2608.03089#bib.bib21); Liet al\.,[2023](https://arxiv.org/html/2608.03089#bib.bib46)\)\. We report accuracy for individual generative and multiple\-choice tasks and macro\-averaged accuracy for multi\-category suites\. Each domain score is the unweighted mean of its two benchmarks, and the overall score is the unweighted mean across all eight benchmarks\. For the code\-containing web experiment, we evaluate BigCodeBench\-Full, HumanEval\+, LiveCodeBench, FullStackBench\-en, and ARC\-Challenge\(Zhuoet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib40); Liuet al\.,[2023](https://arxiv.org/html/2608.03089#bib.bib34); Jainet al\.,[2024](https://arxiv.org/html/2608.03089#bib.bib41); ByteDance Seed Foundation Code Team,[2024](https://arxiv.org/html/2608.03089#bib.bib42); Clarket al\.,[2018](https://arxiv.org/html/2608.03089#bib.bib28)\)\. The overall score is the unweighted mean across the five benchmarks\.

#### Baselines\.

We evaluate the following deduplication settings\.FineWeb\-Eduuses the original FineWeb\-Edu corpus without deduplication\.Doc\-MinHashapplies only document\-level MinHash deduplication using resemblance sketches\(Broder,[1997](https://arxiv.org/html/2608.03089#bib.bib1)\)\. Starting from theDoc\-MinHashcorpus,Suffix\-Arrayperforms shard\-local exact substring deduplication followingLeeet al\.\([2022](https://arxiv.org/html/2608.03089#bib.bib4)\), whileKeep\-Oneuses the same segmentation, normalization, and duplicate grouping as our method but retains exactly one occurrence from each duplicate group\.Ours\(Subdoc\-only\) applies our frequency\- and length\-aware subdocument deduplication directly to FineWeb\-Edu without document\-level deduplication\. The completeOurspipeline first appliesDoc\-MinHash, followed by our subdocument deduplication procedure\. For the code\-containing web experiment,No\-Dedupuses the source corpus without additional document\- or subdocument\-level deduplication\.

### 5\.2Main Results on FineWeb\-Edu

As shown in Table[1](https://arxiv.org/html/2608.03089#S4.T1), our method achieves top overall performance both with and without preceding document\-level deduplication\. With document\-level MinHash,Oursobtains the best results on four benchmarks\(TriviaQA, HellaSwag, PIQA, and MATHand\) and ranks second overall with an average score of52\.9052\.90\. Without document\-level deduplication,Ours\(Subdoc\-only\) leads on GSM8K, MMLU, and CMMLU and achieves the highest overall average of52\.9252\.92\. The two configurations improve over FineWeb\-Edu by1\.011\.01and1\.031\.03points, respectively, and are numerically0\.110\.11and0\.130\.13points above Suffix\-Array\. Together, they achieve the best result on seven of the eight benchmarks, indicating that the proposed subdocument deduplication procedure is effective under both pipeline configurations\.

The baseline comparison separates the effects of detection granularity and copy retention\. Doc\-MinHash reduces the average from51\.8951\.89to51\.5151\.51, whereas all four settings with subdocument\-level deduplication outperform both FineWeb\-Edu and Doc\-MinHash\. This comparison supports the benefit of removing redundancy at a finer granularity in the evaluated setting\. More importantly, Keep\-One uses the same segmentation, normalization, and duplicate grouping as our method but achieves a lower average of52\.1452\.14\. This controlled comparison provides more direct evidence that adaptive copy retention is more effective than uniformly retaining one occurrence per duplicate group\. Suffix\-Array remains competitive, and the margins of our two variants over it are modest; nevertheless, both variants achieve stronger aggregate performance\.

![Refer to caption](https://arxiv.org/html/2608.03089v1/x2.png)Figure 2:Training dynamics of different deduplication methods, evaluated every 2k steps up to 36k\. Each capability\-domain panel reports the unweighted average of its two benchmarks, while Overall averages all eight benchmarks\.Figure[2](https://arxiv.org/html/2608.03089#S5.F2)shows that the effects of deduplication are limited during early training, when the trajectories largely overlap\. The differences become more consistent in the later stages\. Both variants of our method remain near the top of the overall curve and finish above all baselines at36​k36\\mathrm\{k\}steps\. Their domain\-level trajectories differ:Oursends with the highest performance in knowledge and reasoning, whereasOurs\(Subdoc\-only\) performs best in mathematics and comprehensive evaluation\. Suffix\-Array remains competitive throughout training, consistent with its small gap in the final average\. Overall, the curves suggest that the final gains reflect a sustained late\-stage trend rather than an isolated fluctuation at the last checkpoint\.

### 5\.3Results on Code\-Containing Webpages

Table 2:Performance after 36k training steps on code\-containing webpages\. Average denotes the unweighted mean across the five benchmarks\.Table[2](https://arxiv.org/html/2608.03089#S5.T2)shows that our method improves performance on all five evaluated benchmarks, increasing the average score from37\.9837\.98to41\.4041\.40\. The largest gains are observed on HumanEval\+ and ARC\-Challenge, with improvements of5\.535\.53and5\.355\.35points, respectively\. BigCodeBench, LiveCodeBench, and FullStackBench\-en also improve by2\.372\.37,1\.731\.73, and2\.082\.08points\. The gains across four code benchmarks indicate that structure\-aware subdocument deduplication benefits several forms of code capability rather than only one specific evaluation setting\. The improvement on ARC\-Challenge further suggests that removing local redundancy from code\-containing webpages does not trade off broader non\-code capability in this experiment\.

![Refer to caption](https://arxiv.org/html/2608.03089v1/x3.png)Figure 3:Training dynamics on code\-containing webpages\. Panels \(a\)–\(e\) report the five individual benchmarks, while panel \(f\) shows their unweighted average\. Each panel uses its own vertical scale\.Figure[3](https://arxiv.org/html/2608.03089#S5.F3)shows that the endpoint improvements are also reflected in the training dynamics\. The two settings perform similarly during early training, after which our method develops clearer advantages on HumanEval\+, LiveCodeBench, and ARC\-Challenge\. The overall curve follows the same pattern and remains higher toward the end of training\. Therefore, the final improvement is supported by a broader late\-stage trend rather than an isolated advantage at the36​k36\\mathrm\{k\}\-step checkpoint\.

Table 3:Corpus sizes after document\- and subdocument\-level deduplication\. Retention ratios are computed relative to the corresponding raw corpus\.### 5\.4Corpus Reduction Analysis

Table[5\.3](https://arxiv.org/html/2608.03089#S5.SS3)reports the corpus sizes after each deduplication stage\. On FineWeb\-Edu, document\-level MinHash reduces the corpus from6\.286\.28T to1\.011\.01T tokens, retaining16\.08%16\.08\\%of the raw data\. Subdocument\-level deduplication further reduces the corpus to937\.41937\.41B tokens for Suffix\-Array,799\.55799\.55B for Keep\-One, and905\.36905\.36B for our method, corresponding to raw\-corpus retention ratios of14\.93%14\.93\\%,12\.73%12\.73\\%, and14\.42%14\.42\\%, respectively\. Keep\-One therefore applies the strongest additional compression, while our method retains more data than Keep\-One but less than Suffix\-Array\.

This comparison also shows that downstream performance is not determined solely by the amount of data removed\. Although Keep\-One produces the smallest FineWeb\-Edu corpus, it performs below both Suffix\-Array and our method, whereas our method achieves the best overall benchmark performance with an intermediate retention ratio\. On code\-containing webpages, our method retains502\.23502\.23B of the original561\.53561\.53B tokens, corresponding to a retention ratio of89\.44%89\.44\\%\.

![Refer to caption](https://arxiv.org/html/2608.03089v1/x4.png)Figure 4:Retention behavior under the experimental configurationN=100/3N=100/3andL0=512L\_\{0\}=512\. The left shows the continuous pre\-rounding retention budget, and the right panel shows the corresponding retained fraction for different chunk lengths\. Both panels apply the one\-copy lower bound\.### 5\.5Retention Behavior Across Frequency and Length

As shown in Figure[4](https://arxiv.org/html/2608.03089#S5.F4), forL<L0L<L\_\{0\}, the absolute retention budget first increases with duplicate frequency, reaches its maximum at approximatelyC=33C=33, and then decreases toward one copy\. IncreasingLLshifts the retention curve downward, so longer repeated chunks receive smaller budgets at the same frequency\. TheL=0L=0curve corresponds to frequency\-only retention, whereasL=L0=512L=L\_\{0\}=512yields a one\-copy budget across all frequencies\. Although the absolute number of retained copies may initially increase, the retained fraction decreases monotonically with duplicate frequency\. The policy therefore preserves more copies of short, low\-frequency repetitions while applying stronger compression to long or highly frequent duplicate content\.

## 6Conclusion

In this paper, we presented a scalable subdocument deduplication framework that decouples duplicate detection from copy retention\. It identifies duplicate groups through natural\-boundary segmentation, normalized exact hashing, and distributed aggregation, and allocates retention budgets according to global duplicate frequency and span length\. Coherence\-preserving deletion further reduces fragmentation in both natural\-language and code documents\. Under a fixed training budget, models trained on data processed by our method achieve competitive performance\. These results show that effective subdocument deduplication requires not only identifying repeated content, but also explicitly controlling how many copies to retain\.

## References

- PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6239),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/6239)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- A\. Z\. Broder \(1997\)On the resemblance and containment of documents\.InProceedings of the Compression and Complexity of Sequences,pp\. 21–29\.External Links:[Document](https://dx.doi.org/10.1109/SEQUEN.1997.666900)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px3.p1.1)\.
- ByteDance Seed Foundation Code Team \(2024\)FullStack Bench: evaluating LLMs as full stack coders\.External Links:2412\.00535,[Link](https://arxiv.org/abs/2412.00535)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.InWorkshop on Machine Reading for Question Answering,External Links:[Link](https://arxiv.org/abs/1803.05457)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- J\. Dean and S\. Ghemawat \(2004\)MapReduce: simplified data processing on large clusters\.In6th Symposium on Operating Systems Design and Implementation \(OSDI 2004\),pp\. 137–150\.External Links:[Link](https://www.usenix.org/conference/osdi-04/mapreduce-simplified-data-processing-large-clusters)Cited by:[§4\.1](https://arxiv.org/html/2608.03089#S4.SS1.SSS0.Px3.p1.7)\.
- J\. Dodge, M\. Sap, A\. Marasović, W\. Agnew, G\. Ilharco, D\. Groeneveld, M\. Mitchell, and M\. Gardner \(2021\)Documenting large webtext corpora: a case study on the colossal clean crawled corpus\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 1286–1305\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.98),[Link](https://aclanthology.org/2021.emnlp-main.98/)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p1.1)\.
- L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. Leahy \(2020\)The Pile: an 800GB dataset of diverse text for language modeling\.External Links:2101\.00027,[Link](https://arxiv.org/abs/2101.00027)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p2.1)\.
- H\. E\. Gohari, S\. R\. Kadhe, Y\. Shah, C\. M\. Adam, A\. Adebayo, P\. Adusumilli, F\. Ahmed, N\. Baracaldo, S\. S\. Borse, Y\. Chang, X\. Dang, N\. Desai, R\. Eres, R\. Iwamoto, A\. A\. Karve, Y\. Koyfman, W\. Lee, C\. Liu, B\. Lublinsky, T\. Ohko,et al\.\(2026\)GneissWeb: preparing high quality data for LLMs at scale\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NRWUAo075J)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p3.1),[§1](https://arxiv.org/html/2608.03089#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.03089#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.03089#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.03089#S3.SS2.p2.1)\.
- A\. Grattafioriet al\.\(2024\)The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.03089#S2.SS2.p1.1)\.
- N\. He, W\. Xiong, H\. Liu, Y\. Liao, L\. Ding, K\. Zhang, G\. Tang, X\. Han, and W\. Yang \(2024\)SoftDedup: an efficient data reweighting method for speeding up language model pre\-training\.External Links:2407\.06654,[Link](https://arxiv.org/abs/2407.06654)Cited by:[§2\.2](https://arxiv.org/html/2608.03089#S2.SS2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021a\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2009.03300)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021b\)Measuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2103.03874)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- D\. Hernandez, T\. Brown, T\. Conerly, N\. DasSarma, D\. Drain, S\. El\-Showk, S\. Fort, D\. Ganguli, Z\. Hatfield\-Dodds, T\. Henighan, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, J\. Clark, J\. Kaplan, B\. Mann, and S\. McCandlish \(2022\)Scaling laws and interpretability of learning from repeated data\.External Links:2205\.10487,[Link](https://arxiv.org/abs/2205.10487)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, O\. Vinyals, J\. W\. Rae, and L\. Sifre \(2022\)Training compute\-optimal large language models\.InAdvances in Neural Information Processing Systems 35,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p1.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2024\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.External Links:2403\.07974,[Link](https://arxiv.org/abs/2403.07974)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- M\. Joshi, E\. Choi, D\. Weld, and L\. Zettlemoyer \(2017\)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1601–1611\.External Links:[Document](https://dx.doi.org/10.18653/v1/P17-1147),[Link](https://aclanthology.org/P17-1147/)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- D\. Kocetkov, R\. Li, L\. Ben Allal, J\. Li, C\. Mou, C\. M\. Ferrandis, N\. Muennighoff,et al\.\(2022\)The Stack: 3 TB of permissively licensed source code\.External Links:2211\.15533,[Link](https://arxiv.org/abs/2211.15533)Cited by:[§4\.1](https://arxiv.org/html/2608.03089#S4.SS1.SSS0.Px1.p2.1)\.
- C\. Kohlschütter, P\. Fankhauser, and W\. Nejdl \(2010\)Boilerplate detection using shallow text features\.InProceedings of the Third ACM International Conference on Web Search and Data Mining,pp\. 441–450\.External Links:[Document](https://dx.doi.org/10.1145/1718487.1718542),[Link](https://doi.org/10.1145/1718487.1718542)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p1.1),[§1](https://arxiv.org/html/2608.03089#S1.p5.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.\(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276),[Link](https://aclanthology.org/Q19-1026/)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- K\. Lee, D\. Ippolito, A\. Nystrom, C\. Zhang, D\. Eck, C\. Callison\-Burch, and N\. Carlini \(2022\)Deduplicating training data makes language models better\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8424–8445\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.577),[Link](https://aclanthology.org/2022.acl-long.577/)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p1.1),[§1](https://arxiv.org/html/2608.03089#S1.p2.1),[§1](https://arxiv.org/html/2608.03089#S1.p3.1),[§1](https://arxiv.org/html/2608.03089#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.03089#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.03089#S3.SS2.p2.1),[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px3.p1.1)\.
- H\. Li, Y\. Zhang, F\. Koto, Y\. Yang, H\. Zhao, Y\. Gong, N\. Duan, and T\. Baldwin \(2023\)CMMLU: measuring massive multitask language understanding in chinese\.External Links:2306\.09212,[Link](https://arxiv.org/abs/2306.09212)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- J\. Liet al\.\(2024\)DataComp\-LM: in search of the next generation of training sets for language models\.External Links:2406\.11794,[Link](https://arxiv.org/abs/2406.11794)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.03089#S2.SS1.p1.1)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation\.External Links:2305\.01210,[Link](https://arxiv.org/abs/2305.01210)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- U\. Manber and G\. Myers \(1993\)Suffix arrays: a new method for on\-line string searches\.SIAM Journal on Computing22\(5\),pp\. 935–948\.External Links:[Document](https://dx.doi.org/10.1137/0222058)Cited by:[§2\.1](https://arxiv.org/html/2608.03089#S2.SS1.p1.1)\.
- N\. Muennighoff, A\. M\. Rush, B\. Barak, T\. Le Scao, N\. Tazi, A\. Piktus, S\. Pyysalo, T\. Wolf, and C\. Raffel \(2023\)Scaling data\-constrained language models\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2305.16264)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p5.1)\.
- G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. Wolf \(2024\)The FineWeb datasets: decanting the web for the finest text data at scale\.External Links:2406\.17557,[Link](https://arxiv.org/abs/2406.17557)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p1.1),[§1](https://arxiv.org/html/2608.03089#S1.p2.1),[§1](https://arxiv.org/html/2608.03089#S1.p3.1),[§1](https://arxiv.org/html/2608.03089#S1.p5.1),[§2\.2](https://arxiv.org/html/2608.03089#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.03089#S4.SS1.SSS0.Px1.p1.4),[§4\.1](https://arxiv.org/html/2608.03089#S4.SS1.SSS0.Px2.p1.3),[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px1.p1.7)\.
- G\. Penedo, H\. Kydlíček, V\. Sabolčec, B\. Messmer, N\. Foroutan, A\. H\. Kargaran, C\. Raffel, M\. Jaggi, L\. Von Werra, and T\. Wolf \(2025\)FineWeb2: one pipeline to scale them all—adapting pre\-training data processing to every language\.External Links:2506\.20920,[Link](https://arxiv.org/abs/2506.20920)Cited by:[§2\.2](https://arxiv.org/html/2608.03089#S2.SS2.p1.1)\.
- G\. Penedo, Q\. Malartic, D\. Hesslow, R\. Cojocaru, A\. Cappelli, H\. Alobeidli, B\. Pannier, E\. Almazrouei, and J\. Launay \(2023\)The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only\.External Links:2306\.01116,[Link](https://arxiv.org/abs/2306.01116)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.03089#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.03089#S3.SS2.p2.1)\.
- G\. Wenzek, M\. Lachaux, A\. Conneau, V\. Chaudhary, F\. Guzmán, A\. Joulin, and E\. Grave \(2019\)CCNet: extracting high quality monolingual datasets from web crawl data\.External Links:1911\.00359,[Link](https://arxiv.org/abs/1911.00359)Cited by:[§1](https://arxiv.org/html/2608.03089#S1.p3.1),[§1](https://arxiv.org/html/2608.03089#S1.p5.1),[§2\.1](https://arxiv.org/html/2608.03089#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.03089#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.03089#S4.SS1.SSS0.Px1.p1.4),[§4\.1](https://arxiv.org/html/2608.03089#S4.SS1.SSS0.Px2.p1.3)\.
- M\. Zaharia, M\. Chowdhury, T\. Das, A\. Dave, J\. Ma, M\. McCauley, M\. J\. Franklin, S\. Shenker, and I\. Stoica \(2012\)Resilient distributed datasets: a fault\-tolerant abstraction for in\-memory cluster computing\.In9th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 12\),pp\. 15–28\.External Links:[Link](https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/zaharia)Cited by:[§4\.1](https://arxiv.org/html/2608.03089#S4.SS1.SSS0.Px3.p1.7)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://arxiv.org/abs/1905.07830)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.
- M\. Zheng, Z\. Li, T\. Chen, B\. Lv, M\. Sun, M\. Song, J\. Song, H\. Huang, D\. Wu, H\. Wang, Y\. Song, Y\. Chen, and G\. Zhang \(2026\)Hy\-mt2: a family of fast, efficient and powerful multilingual translation models in the wild\.External Links:2605\.22064,[Link](https://arxiv.org/abs/2605.22064)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px1.p1.7)\.
- T\. Y\. Zhuo, M\. C\. Vu, J\. Chim, H\. Hu, W\. Yu, R\. Widyasari,et al\.\(2024\)BigCodeBench: benchmarking code generation with diverse function calls and complex instructions\.External Links:2406\.15877,[Link](https://arxiv.org/abs/2406.15877)Cited by:[§5\.1](https://arxiv.org/html/2608.03089#S5.SS1.SSS0.Px2.p1.1)\.