Tag
Kernel Ridge Attention (KRA) is a recurrent sequence mixer that uses recursive ridge regression with covariance-guided memory updates and an Ornstein-Uhlenbeck transition for selective forgetting, achieving constant-size inference state and improved language-modeling performance over attention and recurrent baselines.
This paper introduces a novel task, transitive inference with exceptions, and analytically characterizes how neural network models (kernel ridge regression) balance relational generalization and memorization. The theory is validated in pretrained language models, showing systematic mistakes predicted by the theory.