fact-storage

This paper presents a theoretical account of how MLPs in Transformers store facts at an information-theoretically optimal rate, and provides a closed-form construction that achieves optimal storage capacity with far fewer parameters than prior methods, enabling modular fact editing.