Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Summary
研究系统评估了合成数据预预训练（PPT）在 500M 到 7B 参数、最高 100B token 预训练预算下的表现，发现 PPT 的 token 效率增益在大规模下依然存在，但这些增益并非来自先前假设的语法先验，而是源于提升长程检索能力的 PPT 任务。
View Cached Full Text
Cached at: 10/01/26, 12:24 PM
Paper page - Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Source: https://huggingface.co/papers/2609.39827
Abstract
Pre-pretraining(PPT)onsyntheticnon-naturallanguagedataimprovestokenefficiencyduringlanguagemodelpre-training(PT).Priorworkattributesthisgaintoagrammaticalprior,i.e.,astructuralinductivebiaslearnedduringPPTthattransferstonaturallanguagegrammar.However,PPThasonlybeentestedonmodelsofatmost1BparametersandPTbudgetsbelow2Btokensonpredominantlywebtext.ItisunknownwhetherPPTiseffectiveatlargerscalesandundermorerealisticPTdatamixturesthatcombinediversesources(e.g.,codeandmath).WethereforepresentacomprehensivestudyonPPTspanningfivePPTtasks,fourPTdatamixtures,fourparameterscales(500Mto7B),andPTbudgetsofupto100Btokens.OurresultsdemonstratethatthedownstreamperformanceandtokenefficiencygainsofPPTpersistatscale,e.g.,savingatleast21BPTtokensatthe3Bscale.However,incontrasttopriorwork,wefindnoconsistentevidencethatthesegainsstemfromagrammaticalprior.Downstreamperformancedoesnotconsistentlyalignwithgrammaticalacceptabilityacrossmodelsizes.Instead,wefindthatdownstreamgainsarisefromPPTtasksthatimprovelong-rangeretrieval.Finally,PPTperformancegainsarerobusttohowPTdatamixturesarecomposedanddiminishonlywhenwebtextisabsent.Overall,PPTisalow-costadditiontoPT,andfuturePPTtaskdesignshouldtargetlong-rangeretrievalratherthannaturallanguagegrammar.
View arXiv pageView PDFProject pageGitHub0Add to collection
Get this paper in your agent:
hf papers read 2609\.39827
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.39827 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.39827 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.39827 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models
论文提出用经过筛选和再平衡的教师监督信号微调 Qwen3.5 小模型（0.8B/4B），在语法概念标注任务上超越提示式 GPT-5.4/GPT-5.6 Sol，同时将服务成本降低约 16 倍，并在在线实验中显著提升学习者参与度与业务指标。
Specialization, Diversity, and Scaling in Child-Scale Language Models
研究训练小规模GPT-2模型于儿童日常语音转录（BabyView）和合成儿童导向对话（TinyDialogues）数据上，发现模型会显著特化于特定儿童的语言环境，多儿童数据能提升对混合留出语言的预测但对语法基准改进有限，表明小数据语言学习不仅取决于数据量，还取决于环境来源与匹配度。
Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
该论文提出了一种经验性加法幂律，将长上下文持续预训练（LCCP）的收益与模型规模及短/长上下文token预算关联起来，并引入Constant-Last4协议实现稳定监控。实验表明LCCP的收益在评测范围内无明显饱和迹象，主张不应过早停止长上下文持续预训练。
@FinanceYF5: Everyone is hyping up the million-token context, but a Prime Intellect engineer spoke the truth: GPT-5.5 has 80% retrieval accuracy at 256k, but when extended to one million, it drops to 36%. The model doesn't fail to hold it—it fails to reason over it—the so-called context rot. Why more...
A Prime Intellect engineer pointed out that large language models like GPT-5.5 see retrieval accuracy drop from 80% at 256k tokens to 36% at one million tokens, indicating the 'context rot' problem—the model can accommodate but cannot effectively reason over long contexts, posing a challenge to agent applications.
How to Use Parallel Data for Multilingual Pretraining
The paper proposes Parallel Pretraining (PPT), which concatenates documents with their translations in the same training sequence, and finds that a two-stage PPT-then-standard-pretraining approach consistently improves general language understanding and cross-lingual performance over standard pretraining at matched token budgets.