mixture-of-neighbors

This paper introduces MoNIM, a learnable memory module that integrates induction capabilities of attention heads with feed-forward networks to enable scalable continual learning in semiparametric language models, improving efficiency and retention of new knowledge.