Tag
Researchers propose that the harness (training setup) should carry inductive biases for generalization, showing that training RLMs is far superior to vanilla Transformers for scaling and generalization to harder tasks.
The Cognitive Categorical Transformer (CCT) augments GPT-2 Small with category-theoretic components, achieving a 12% relative perplexity reduction on WikiText-103 under matched training conditions, with simplicial message passing responsible for 84% of the improvement.