Tag
该论文研究了 LLM 预训练数据采集中的爬虫预算分配问题，发现文档质量在主机层面存在显著聚集性，通过先采样少量页面再按质量深入爬取，可比统一深度或基于中心度的分配多收集数倍高质量页面。
Project Timestamper 宣布将 Common Crawl 过去 18 年收集的 3390 亿个网页快照通过哈希链提交到比特币区块链，实现可秒级验证的数据完整性证明，以应对 AI 生成内容带来的真伪辨别挑战。
A Pew Research Center study found that around one in ten .com domains show signs of AI authorship, indicating a growing prevalence of AI-generated content on the internet.
OpenBMB releases Ultra-FineWeb-L1, an open English web corpus with over 1 trillion tokens for LLM pre-training, derived from Common Crawl and featuring advanced cleaning with Trafilatura 2.0.
This paper introduces WebGraphMix, a lightweight framework that uses web graph centrality scores from Common Crawl to select pretraining data, showing that mixing central and peripheral documents improves language model performance.
Kyutai Labs trains 6B-parameter models on Common Crawl data ordered sequentially from 2018 to 2025, showing that performance drop on recent years disappears, and open-sources the checkpoints for continual learning research.
A research paper from Stanford University proposes that with sufficient compute, the best data filtering strategy is no filtering. Experiments show that large-scale models are robust to low-quality data, and unfiltered data pools perform better at larger scales. However, this conclusion applies to standard pre-training of dense models, and filtering remains important when compute is limited.
A Hugging Face Space allows running SQL queries over 2.19 billion web pages from Common Crawl without downloading, using DuckDB to read directly from Hugging Face storage buckets.
Learn how to set up and use Common Crawl data locally for web data processing tasks.
This paper investigates data filtering for large model pretraining and finds that in the high-compute, data-scarce regime, filtering may not be necessary and can even be detrimental; sufficiently trained large models benefit from nominally low-quality data.
A survey analyzing over 300,000 web feeds on the top 500k sites reveals that while feeds remain prevalent, most are abandoned or low quality due to automatic CMS generation. The author used AI agents to process Common Crawl data and calls for better feed management practices.