synthetic-priors

This paper introduces the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained exclusively on samples from a synthetic structural causal model, enabling it to learn languages in context with frozen weights despite never seeing real text. Results show near-competitivenatural-language modeling, arithmetic, and compression capabilities below gzip/PPMd, suggesting meta-learning of language rather than memorization.