Tag
A commentary on Nick Cammarata's Goodfire essay 'Neural Networks Are Already Clean' exploring whether a sparse 'semantic skeleton' of bipolar neurons plus external VSA hypervector memory could enable continual learning with small frozen models, while noting obstacles like tangled fact-routing and coordinate-registry alignment.
Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.