How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

Hugging Face Daily Papers Papers
scaling-laws pretraining-data ai-generated-text language-models data-quality wild-ai

Summary

This paper studies how unlabeled, AI-generated web text affects language model pretraining by training 800 models and fitting new scaling laws. It finds AI text helps data-starved models initially but quickly turns harmful for human-heavy budgets, proposes a scaling law with separate benefit and harm terms that outperforms existing laws, and releases the WildAI 83B-token corpus and all models.

Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.1% by August. Unlike synthetic data or model-collapse setups, this *wild* AI text comes from many models, is written for human readers, and arrives unlabeled in pretraining corpora. How does AI text in the wild affect language model pretraining? To answer this question, we pretrain 800 language models, varying the ratio of added AI tokens to human tokens, and fit scaling laws to held-out losses on both human and AI-generated text. For data-starved models, adding AI tokens to pretraining data initially lowers loss on human text, but the benefit saturates as more are added and quickly *reverses* into harm. For models trained on high budgets of human text, AI tokens raise loss almost immediately, while the same number of fresh human tokens keeps lowering it. Scaling laws such as Hoffman et al. (2022) fail to predict this behavior. We propose a new scaling law with separate benefit and harm terms that allows the value of an AI token to change sign while also reducing to Chinchilla in the absence of AI text. When fit on smaller models, our scaling law predicts the effect of AI text on held-out human-text loss for models up to 3.6x larger with 41% lower error than the best existing law over all AI ratios. We recommend filtering AI text when the target is human text, repeating human text before expanding the training dataset with AI-generated web text, and reporting validation loss on human and AI text separately AI text remains valuable when the target is AI text. We release WildAI, an 83B-token corpus with AI, topic, and format labels, all 800 models and code at https://github.com/pangramlabs/WildAI.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:23 PM

Paper page - How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

Source: https://huggingface.co/papers/2609.40295

Abstract

WebtextmakesupthemajorityofpretrainingdataandisincreasinglyAI-generated.AfterapplyingFineWebqualityfiltering,wefindthat27.5%oftokensfromJune2026webdataarelabeledasAI-generatedbyPangram,risingto31.1%byAugust.Unlikesyntheticdataormodel-collapsesetups,this*wild*AItextcomesfrommanymodels,iswrittenforhumanreaders,andarrivesunlabeledinpretrainingcorpora.HowdoesAItextinthewildaffectlanguagemodelpretraining?Toanswerthisquestion,wepretrain800languagemodels,varyingtheratioofaddedAItokenstohumantokens,andfitscalinglawstoheld-outlossesonbothhumanandAI-generatedtext.Fordata-starvedmodels,addingAItokenstopretrainingdatainitiallylowerslossonhumantext,butthebenefitsaturatesasmoreareaddedandquickly*reverses*intoharm.Formodelstrainedonhighbudgetsofhumantext,AItokensraiselossalmostimmediately,whilethesamenumberoffreshhumantokenskeepsloweringit.ScalinglawssuchasHoffmanetal.(2022)failtopredictthisbehavior.WeproposeanewscalinglawwithseparatebenefitandharmtermsthatallowsthevalueofanAItokentochangesignwhilealsoreducingtoChinchillaintheabsenceofAItext.Whenfitonsmallermodels,ourscalinglawpredictstheeffectofAItextonheld-outhuman-textlossformodelsupto3.6xlargerwith41%lowererrorthanthebestexistinglawoverallAIratios.WerecommendfilteringAItextwhenthetargetishumantext,repeatinghumantextbeforeexpandingthetrainingdatasetwithAI-generatedwebtext,andreportingvalidationlossonhumanandAItextseparatelyAItextremainsvaluablewhenthetargetisAItext.WereleaseWildAI,an83B-tokencorpuswithAI,topic,andformatlabels,all800modelsandcodeathttps://github.com/pangramlabs/WildAI.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2609\.40295

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### pangram/WildAI-models Text Generation• Updatedabout 2 hours ago

Datasets citing this paper1

#### pangram/WildAI Viewer• Updatedabout 1 hour ago • 192M

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.40295 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

The uneven rise of AI-generated web content

arXiv cs.AI

This arXiv paper analyzes over 5 million webpages from 2019 to 2026, finding that the share of pages with AI-generated text grew from near zero before 2022 to nearly 25% by August 2026. The rise is uneven: English-language and .com domains have higher AI prevalence than other languages or .gov/.edu domains, and AI text concentrates in the least central pages of the web graph, with estimates produced via a Bayesian hierarchical model correcting for detector error rates.

Every AI prompt costs money — and that changes everything

Reddit r/AI_Agents

The article argues that the real challenge in AI isn't just building smarter models but making them cost-efficient at scale, highlighting the importance of reducing token usage, improving speed, and optimizing infrastructure.

Why is tokenisation of AI so high?

Reddit r/AI_Agents

The article discusses why token usage escalates quickly in AI agents due to factors like system prompts and tool definitions, and inquires about effective techniques to manage token consumption.