Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization
Summary
This paper theoretically analyzes linear transformers for in-context learning under domain generalization, establishing dimension-independent convergence rates and proposing novel activation and loss designs for linearizing pretrained softmax LLMs.
View Cached Full Text
Cached at: 07/02/26, 05:38 AM
# Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization Source: [https://arxiv.org/abs/2607.00479](https://arxiv.org/abs/2607.00479) [View PDF](https://arxiv.org/pdf/2607.00479) > Abstract:Transformer\-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context\-aware attention module for in\-context learning\. With richer context, transformers adapt more effectively to the current use case without any parameter updates\. However, the quadratic computational and memory complexity with respect to context length significantly slows data processing in softmax transformers\. Linear transformers were proposed to address this issue by reducing the complexity to linear dependence on context length, but the design and understanding of the feature mapping in linear attention, from a theoretical viewpoint, remain unclear\. In this paper, we investigate the approximation and generalization abilities of linear transformers under a two\-staged sampling process from domain generalization\. We show that linear transformers perform in\-context learning as learning a mapping from context distributions to response functions\. A dimension\-independent convergence rate is obtained for our generalization analysis, which also exhibits the tradeoff between the regularities of data distributions and latent features\. Guided by our theoretical framework, we propose a new perspective on activation and loss design for linearizing pretrained softmax large language models\. ## Submission history From: Peilin Liu \[[view email](https://arxiv.org/show-email/ce7444e3/2607.00479)\] **\[v1\]**Wed, 1 Jul 2026 06:03:14 UTC \(628 KB\)
Similar Articles
At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization
This paper proposes using sparse autoencoders to detect out-of-distribution inputs for transformers, including typos and jailbreak prompts, by analyzing spurious concept activations. The method enables a mechanistically grounded fine-tuning strategy to improve LLM robustness.
In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
This paper constructs a transformer with linear self-attention that performs in-context learning of the closed-form least squares solution for simple linear regression, using layer normalization to approximate the analytical solution rather than gradient descent.
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
Proposes Memory-Efficient Looped Transformer (MELT), a novel recurrent LLM architecture that decouples reasoning depth from memory consumption by sharing a single KV cache across loops and using chunk-wise training with interpolated transition and attention-aligned distillation.
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
This paper introduces Variational Linear Attention (VLA), a method that stabilizes memory states in linear attention mechanisms for long-context transformers. VLA reframes memory updates as an online regularized least-squares problem, proving bounded state norms and demonstrating significant speedups and improved retrieval accuracy over standard linear attention and DeltaNet.
@a1zhang: Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it i…
The article proposes that Transformers can generalize to new tasks through a well-designed harness that induces composition, without needing intrinsic model generalization. It shows RLMs can generalize from short tasks to 8-32x longer tasks and across domains.