@andrew_n_carr: Turns out you can initialize an MLP with knowledge inside of it, no training required. Hazy research just showed that i…
Summary
Hazy Research showed that an MLP can be initialized with embedded knowledge without training, and a transformer can then query and use that knowledge, hinting at continual learning capabilities.
View Cached Full Text
Cached at: 07/24/26, 05:20 AM
Turns out you can initialize an MLP with knowledge inside of it, no training required.
Hazy research just showed that if you do that properly, a transformer can query that knowledge and use it!
Continual learning??? https://t.co/W5CE8Gt84f
Similar Articles
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
This paper presents a theoretical account of how MLPs in Transformers store facts at an information-theoretically optimal rate, and provides a closed-form construction that achieves optimal storage capacity with far fewer parameters than prior methods, enabling modular fact editing.
@TeachTheMachine: Using a Transformer Model: From Training to Inference
This tutorial covers how to use a transformer model from training to inference, focusing on autoregressive generation, prefill vs. decode phases, and key-value caching for efficient inference.
@rohanpaul_ai: New paper from @NaceAI shows a possible way to add large bodies of knowledge without rewriting the language model’s cor…
A new paper from NaceAI proposes a hypernetwork-based method for injecting knowledge into large language models without modifying their core parameters, potentially enabling efficient continual learning. The approach uses generated low-rank adapters to encode new facts while keeping the base model frozen.
@lateinteraction: The "harness" is starting to blur with the neural architecture, in terms of who carries the inductive biases that unloc…
Researchers propose that the harness (training setup) should carry inductive biases for generalization, showing that training RLMs is far superior to vanilla Transformers for scaling and generalization to harder tasks.
@swyx: co-sign. a very handy mental framework for what kinds of learning transformers do well today, and why it runs into limi…
The article discusses a mental framework for understanding what transformers learn well and their limitations, arguing that scaling current paradigms may be inefficient compared to approaches that hypothesize and seek truth, referencing the need for adversarial world models and reinforcement learning.