efficient-transformers

This paper theoretically analyzes linear transformers for in-context learning under domain generalization, establishing dimension-independent convergence rates and proposing novel activation and loss designs for linearizing pretrained softmax LLMs.