Tag
The article introduces Self-Play Pretraining with Zero Data, a research method where two LLMs start from random initialization and learn through a generator proposing programs for a universal Turing machine, demonstrating how inductive biases can enable general learning.
Researchers propose that the harness (training setup) should carry inductive biases for generalization, showing that training RLMs is far superior to vanilla Transformers for scaling and generalization to harder tasks.
The Cognitive Categorical Transformer (CCT) augments GPT-2 Small with category-theoretic components, achieving a 12% relative perplexity reduction on WikiText-103 under matched training conditions, with simplicial message passing responsible for 84% of the improvement.