Ultra-FineWeb-L1 Open English web corpus for LLM pre-training from @OpenBMB https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1… - 1T+ tokens / ~1.14B documents - Common Crawl data up to CC-MAIN-2025-51 - Apache 2.0 - Advanced cleaning with Trafilatura 2.0 + customized quality inspection

openbmb/Ultra-FineWeb-L1 · Datasets at Hugging Face

Source: https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1

📜 Ultra-FineWeb Technical Report|📦 UltraData Collection|🌐 UltraData

English |中文

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%F0%9F%93%9A-introduction📚 Introduction

Ultra-FineWeb-L1is a large-scale English web corpus built fromCommon Crawlsnapshots. WithinUltraData’sL0-L4 tiered data management framework, it serves as theL1 filtered layerfor general web data and provides the foundation for subsequent L2 selection and L3 refinement. Building on theFineWebprocessing pipeline, we perform main-text extraction, language filtering, heuristic filtering, sensitive-field replacement, MinHash deduplication, and customized data cleaning. We upgrade main-text extraction totrafilatura 2.0, and use UltraData’s data quality inspection tools and data-cleaning agents to optimize cleaning rules and apply targeted processing for residual HTML, encoding corruption and mojibake, invisible characters, corrupted content, and abnormal document lengths. The first release contains**1T+ tokens (approximately 1.14 billion documents)**from six Common Crawl snapshots in 2025. We also releaseUltra-FineWeb, the L2 selected subset produced by theUltra-FineWeb classifier.

• Ultra-FineWeb-L1:L1 filtered dataafter basic cleaning, heuristic filtering, sensitive-field replacement, and deduplication. (Current dataset)
• Ultra-FineWeb:L2 selected dataproduced by theUltra-FineWeb classifier, containing approximately1T English tokensand120B Chinese tokens.
• Ultra-FineWeb-L3:L3 refined databuilt on Ultra-FineWeb via Q&A pair generation and multi-style rewriting, containing400B+ English tokensand200B+ Chinese tokens.

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%F0%9F%93%A2-whats-new📢 What’s New

• [2026.08.20]TheUltra-FineWeb-L1dataset is released! Built from Common Crawl snapshots, it undergoes main-text extraction, language filtering, heuristic filtering, sensitive-field replacement, customized cleaning, and deduplication, yielding1T+ tokens (approximately 1.14 billion documents). We simultaneously release the L2 selected dataUltra-FineWeb, selected by theUltra-FineWeb classifier. To our best knowledge, this open-source web pre-training dataset covers the most recent Common Crawl snapshots, up toCC\-MAIN\-2025\-51. 🚀🚀🚀
• [2026.05.28]TheUltra-FineWeb-L3dataset is released! TheL3 refined databuilt onUltra-FineWebviaQ&A Pair GenerationandMulti-style Rewriting, with400B+ Englishand200B+ Chinesetokens. To our best knowledge, it is the largest open-source Chinese pre-training synthetic corpus to date. 🚀🚀🚀
• [2026.05.25]***MiniCPM5-1Bis released!***It is the first model in the MiniCPM5 series, a dense 1B Transformer designed for on-device, local deployment, and resource-constrained scenarios, reaching 1B-class open-source SOTA. Ultra-FineWeb serves as its core pre-training web dataset.
• [2026.02.08]TheUltraDataplatform is now live, introducing theL0-L4 tiered data management framework.Ultra-FineWebserves as theL2 selected layerfor general web data in this framework. 🔍🔍🔍
• [2025.06.16]TheUltra-FineWeb-classifieris now available on Hugging Face:openbmb/Ultra-FineWeb-classifier.
• [2025.06.06]****Ultra-FineWeb-enandUltra-FineWeb-zhare now available on Hugging Face, released alongside theMiniCPM4 Seriesmodels.
• [2025.05.15]****Ultra-FineWebtops the Hugging Face Datasets Trending list, reaching the #1 spot! ⭐️⭐️⭐️
• [2025.05.09]TheUltra-FineWebtechnical report is available onarXiv. 🔥🔥🔥

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%F0%9F%8F%97%EF%B8%8F-data-processing-pipeline🏗️ Data Processing Pipeline

Each Common Crawl dump is processed independently:

1. **Page and main-text extraction:**retain valid HTML pages from Common Crawl and use trafilatura 2.0 to extract plain text while excluding comments, navigation bars, and other non-main content.
2. **Language filtering:**use fastText language identification to retain high-confidence English documents.
3. **Heuristic filtering:**build on FineWeb’s heuristic filtering approach to address repetition, low-quality documents, boilerplate text, and abnormal line structures.
4. **Sensitive-field replacement:**replace detected email addresses, IP addresses, phone numbers, identity-card numbers, and credit-card numbers with valid placeholders.
5. **MinHash deduplication:**identify near-duplicate documents within each Common Crawl dump. Following FineWeb, deduplication is performed independently within each dump rather than across the full collection.
6. **Customized cleaning:**use UltraData’s data quality inspection tools and data-cleaning agents to optimize cleaning rules and apply targeted processing for residual HTML, encoding corruption and mojibake, invisible characters, corrupted content, and abnormal document lengths.

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%F0%9F%93%81-dataset-structure📁 Dataset Structure

Ultra-FineWeb-L1 is organized by Common Crawl dump:

data/
└── CC-MAIN-YYYY-WW/
    ├── CC-MAIN-YYYY-WW-part-0001-of-NNNN.parquet
    ├── CC-MAIN-YYYY-WW-part-0002-of-NNNN.parquet
    └── ...

The classifier-selected L2 subset is released inUltra-FineWeb:

data/ultrafineweb_l1_en_hq/
└── CC-MAIN-YYYY-WW/
    ├── ultrafineweb-l1-en-hq-CC-MAIN-YYYY-WW-part-0001-of-NNNN.parquet
    └── ...

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#data-fieldsData fields

FieldTypeDescriptionuid``stringUUID4 document identifiercontent``stringCleaned plain-text documentmeta``stringJSON-encoded source and processing metadata, includingurl,language,language\_score,warc\_record\_id,warc\_date, andsource\_file``dataset\_index``int64Dataset-source identifier

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%F0%9F%93%88-experimental-results📈 Experimental Results

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#evaluation-setupEvaluation Setup

We follow the FinePhrase evaluation setup and adopt theFineWebablation methodology: each data configuration is evaluated on 12 benchmarks across six categories using 3-shot prompting with a single random seed. For most tasks, we use the Cloze Format (CF), framing evaluation as next-token prediction. Compared with standard multiple-choice evaluation, CF reduces interference from instruction following and answer formatting for smaller models and provides a more stable evaluation signal. The benchmarks are:

• General Knowledge: ARC, MMLU Redux
• Reading Comprehension: SQuAD v2, DROP
• Reasoning: OpenBookQA, XCSQA
• Natural Language Understanding: WinoGrande, PIQA, HellaSwag
• Math: GSM8K
• Table Understanding: WikiTableQuestions, TriviaQA

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#training-setupTraining Setup

To evaluate the effectiveness of the cleaning pipeline, we selectCC\-MAIN\-2025\-26, the latest Common Crawl snapshot covered byFineWeb v1.4, and compare data processed by the FineWeb v1.4 and Ultra-FineWeb-L1 pipelines. Both groups are trained for20B tokenswith the**MiniCPM5-1B**model under identical settings: 32 GPUs, a micro batch size of 16, a global batch size of 512, the Muon optimizer, an initial learning rate of 0.000522, and a minimum learning rate of 0.0000522.

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#resultsResults

The figures below show the six-category macro-average and 12-task micro-average scores of four data configurations throughout training. Scores generally improve as the number of training tokens increases. At the end of training,Ultra-FineWeb-L1reaches9.668%macro and9.180%micro, outperformingFineWebby0.635and0.696percentage points, respectively, and validating the effectiveness of the optimized cleaning pipeline. Quality selection yields further gains:Ultra-FineWeb-from-FWreaches9.954% / 9.414%, whileUltra-FineWeb-from-L1achieves the best results at10.379% / 9.798%, indicating that L1 cleaning and subsequent quality selection provide complementary benefits.

DataSix-category Macro Average12-task Micro AverageFineWeb9.033%8.484%Ultra-FineWeb-L19.668%9.180%Ultra-FineWeb-from-FW9.954%9.414%Ultra-FineWeb-from-L110.379%9.798%

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%E2%9D%A4%EF%B8%8F-acknowledgements❤️ Acknowledgements

• Source Web Data:Common Crawl
• Data Processing Pipeline:FineWeb,DataTrove
• Text Extraction and Cleaning Tools:trafilatura,fastText,NLTK,ftfy

We thank these outstanding open-source projects—their contributions make Ultra-FineWeb-L1 possible! 🙌

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%F0%9F%93%96-citation📖 Citation

If you findUltra-FineWeb-L1useful in your research, please consider citing:

@misc{wang2025ultrafineweb,
  title={{Ultra-FineWeb}: Efficient Data Filtering and Verification for High-Quality LLM Training Data},
  author={Yudong Wang and Zixuan Fu and Jie Cai and Peijun Tang and Hongya Lyu and Yewei Fang and Zhi Zheng and Jie Zhou and Guoyang Zeng and Chaojun Xiao and Xu Han and Zhiyuan Liu},
  year={2025},
  eprint={2505.05427},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

@misc{ultra-fineweb-l1,
  title={Ultra-FineWeb-L1},
  author={Junshao Guo and Shuaikang Xue and Xiaofei Yang and Chuyue Zhou and Qiang Yin and Qingyang Yuan and Qiang Ma and Yuzhou Zhang and Jie Zhou and Chaojun Xiao and Yudong Wang and Zhiyuan Liu},
  year={2026},
  url={https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1},
  publisher={Hugging Face}
}

https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1#%F0%9F%93%9C-license📜 License

This project is licensed under theApache 2.0license. Because the corpus is derived from web content, users must also comply with the rights, licenses, and terms associated with the original sources.

**No unauthorized unchanged redistribution:**Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.

Downloads last month1,343## Papers foropenbmb/Ultra-FineWeb-L1