synthetic-prior

This paper extends prior-fitted networks (TabPFN) to natural language by training a 300M-parameter byte-level transformer entirely on synthetic sequences drawn from randomly sampled recurrent causal models — a 'prior over languages'. With frozen weights, the model learns to predict real languages in context across six languages, and can also count, compare numbers, add approximately, and predict deterministic sequences like primes and Kolakoski, showing in-context language learning can emerge from a synthetic non-linguistic prior.