Geometric Factual Recall in Transformers
Summary
This paper introduces a theoretical framework for geometric factual recall in transformers, demonstrating that embeddings can encode relational structure via linear superpositions while MLPs act as selectors. It provides empirical and theoretical evidence that this mechanism allows for efficient memorization of facts and multi-hop queries.
View Cached Full Text
Cached at: 05/13/26, 04:10 AM
Paper page - Geometric Factual Recall in Transformers
Source: https://huggingface.co/papers/2605.12426
Abstract
Transformer language models use geometric memorization where embeddings encode linear superpositions of attributes and MLPs act as relation-conditioned selectors rather than associative key-value mappings.
How dotransformer language modelsmemorize factual associations? A common view castsinternal weight matricesasassociative memoriesover pairs of embeddings, requiring parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, geometric form of memorization in which learned embeddings encode relational structure directly, and theMLPplays a qualitatively different role. In a controlled setting where a single-layer transformer must memorize random bijections from subjects to a shared attribute set, we prove that a logarithmicembedding dimensionsuffices: subject embeddings encodelinear superpositionsof their associated attribute vectors, and a smallMLPacts as a relation-conditioned selector that extracts the relevant attribute viaReLU gating, and not as an associative key-value mapping. We extend these results to the multi-hop setting -- chains ofrelational queriessuch as ``Who is the mother of the wife of x?‘’ -- providing constructions with and withoutchain-of-thoughtthat exhibit a provable capacity-depth tradeoff, complemented by a matchinginformation-theoretic lower bound. Empirically,gradient descentdiscovers solutions with precisely the predicted structure. Once trained, theMLPtransfers zero-shot to entirely new bijections when subject embeddings are appropriately re-initialized, revealing that it has learned a generic selection mechanism rather than memorized any particular set of facts.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.12426
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.12426 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.12426 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.12426 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
This paper presents a theoretical account of how MLPs in Transformers store facts at an information-theoretically optimal rate, and provides a closed-form construction that achieves optimal storage capacity with far fewer parameters than prior methods, enabling modular fact editing.
Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination
This paper presents a unified geometric framework for understanding transformer memory failures, distinguishing between conflict arbitration and hallucination through hidden-state attractor basins. It demonstrates that geometric margin is a superior diagnostic for detecting these failures compared to output entropy, particularly as model scale increases.
I Found a Hidden Ratio in Transformers That Predicts Geometric Stability [R]
The article presents a discovered spectral ratio between MLP and attention norms that predicts geometric stability in transformer models, with an optimal range of 0.5–2 to prevent rank collapse.
A mathematical theory of balancing relational generalization and memorization
This paper introduces a novel task, transitive inference with exceptions, and analytically characterizes how neural network models (kernel ridge regression) balance relational generalization and memorization. The theory is validated in pretrained language models, showing systematic mistakes predicted by the theory.
Transformers Learn the Mestre-Nagao Heuristic
This paper trains a two-layer transformer encoder to classify rational elliptic curves by rank from Frobenius traces, achieving >99% accuracy. Mechanistic interpretability reveals the model learns the Mestre-Nagao heuristic and concentrates attention on prime positions, demonstrating that transformers can learn number-theoretic algorithms.