Learning to Learn a Language

Hugging Face Daily Papers Papers
language-model meta-learning synthetic-priors transformers compression byte-level

Summary

This paper introduces the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained exclusively on samples from a synthetic structural causal model, enabling it to learn languages in context with frozen weights despite never seeing real text. Results show near-competitivenatural-language modeling, arithmetic, and compression capabilities below gzip/PPMd, suggesting meta-learning of language rather than memorization.

We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior. Given a prefix of real text, it learns to predict the language in context with frozen weights, having never seen a word of any real language. Every training sequence is generated by a recurrent structural causal model drawn fresh from a distribution over such models. The model never sees the same language twice during training, so the only way to predict the continuation is to infer the language from the prefix. Samples from this prior share the statistical signatures of natural text: Zipfian frequencies, slow entropy-rate convergence, and long-range dependence. On Wikipedia in six languages, bits per byte fall from the uniform eight to between 0.9 and 2.4 at one million bytes of context. Given numerals instead of text, PFLM learns to count, to compare magnitudes, and to add approximately. It predicts deterministic sequences like Rudin-Shapiro or the prime indicator, and it compresses six non-text domains, from source code to speech, below gzip and PPMd. The model has not learned a language. It has learned to learn one.
Original Article
View Cached Full Text

Cached at: 10/06/26, 12:55 PM

Paper page - Learning to Learn a Language

Source: https://huggingface.co/papers/2610.05879

Abstract

WepresentthePrior-FittedLanguageModel(PFLM),a300M-parameterbyte-leveltransformerpretrainedonlyonsamplesfromasyntheticnon-linguisticprior.Givenaprefixofrealtext,itlearnstopredictthelanguageincontextwithfrozenweights,havingneverseenawordofanyreallanguage.Everytrainingsequenceisgeneratedbyarecurrentstructuralcausalmodeldrawnfreshfromadistributionoversuchmodels.Themodelneverseesthesamelanguagetwiceduringtraining,sotheonlywaytopredictthecontinuationistoinferthelanguagefromtheprefix.Samplesfromthispriorsharethestatisticalsignaturesofnaturaltext:Zipfianfrequencies,slowentropy-rateconvergence,andlong-rangedependence.OnWikipediainsixlanguages,bitsperbytefallfromtheuniformeighttobetween0.9and2.4atonemillionbytesofcontext.Givennumeralsinsteadoftext,PFLMlearnstocount,tocomparemagnitudes,andtoaddapproximately.ItpredictsdeterministicsequenceslikeRudin-Shapiroortheprimeindicator,anditcompressessixnon-textdomains,fromsourcecodetospeech,belowgzipandPPMd.Themodelhasnotlearnedalanguage.Ithaslearnedtolearnone.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2610\.05879

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### lennartcb/pflm1 Text Generation• 0.3B• Updatedabout 6 hours ago • 327

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.05879 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.05879 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Learning to Learn a Language: in-context learning of natural language from a synthetic non-linguistic prior [R]

Reddit r/MachineLearning

This paper extends prior-fitted networks (TabPFN) to natural language by training a 300M-parameter byte-level transformer entirely on synthetic sequences drawn from randomly sampled recurrent causal models — a 'prior over languages'. With frozen weights, the model learns to predict real languages in context across six languages, and can also count, compare numbers, add approximately, and predict deterministic sequences like primes and Kolakoski, showing in-context language learning can emerge from a synthetic non-linguistic prior.

Language Acquisition Device in Large Language Models

arXiv cs.CL

This paper proposes LAD-inspired pre-pretraining using a formal language called MP-Struct that encodes natural-language-like structures. It shows that this approach improves token efficiency and imparts human-like resistance to structurally implausible languages, challenging prior hypotheses about effective pre-pretraining languages.

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

Hugging Face Daily Papers

This paper proposes the Implicit Curriculum Hypothesis, demonstrating that language model pretraining follows a structured, compositional curriculum where capabilities emerge consistently across architectures and can be predicted from internal representations. The authors validate this through designed tasks spanning retrieval, morphology, coreference, reasoning, and mathematics, finding highly consistent emergence orderings (ρ=0.81) across four model families.

Structural priors for data-efficient language learning

arXiv cs.CL

This paper investigates structural transfer from non-language data to improve data efficiency in language modeling, finding that while it reduces next-token prediction loss, it does not reliably enhance downstream linguistic performance.

Hierarchical Latent Prediction for Language Models

arXiv cs.CL

This paper introduces HiLP, a hierarchical representation training method that adds multi-scale self-predictive learning to transformer pretraining, aiming to reduce compounding error and improve long-horizon reasoning and speculative decoding efficiency.