Induction Heads Interpolate N-Grams
Summary
This paper studies transformers trained on Markov chains and identifies that induction heads implement soft context-matching and Dirichlet-style smoothing, showing that transformers regularize in-context estimation rather than simply counting n-grams.
View Cached Full Text
Cached at: 07/07/26, 04:39 AM
# Induction Heads Interpolate N-Grams Source: [https://arxiv.org/abs/2607.02800](https://arxiv.org/abs/2607.02800) [View PDF](https://arxiv.org/pdf/2607.02800) > Abstract:Induction heads are attention circuits believed to underlie in\-context learning in transformers, yet a precise characterization of the estimators they implement remains elusive\. We study transformers trained on order\-$k$ Markov chains and identify two complementary smoothing mechanisms\. First, at finite attention\-weight scale, the circuit implements a soft context\-matching estimator: it aggregates contributions from exact and partial context matches, weighted exponentially by their overlap, and induces a data\-dependent interpolation across context orders analogous to Jelinek\-Mercer smoothing\. Second, a beginning\-of\-sequence \(BOS\) token induces additive pseudo\-counts, recovering Dirichlet\-style smoothing\. We construct a disentangled transformer implementing both mechanisms and show that trained transformers recover the predicted attention patterns\. Across settings where pseudo\-count smoothing is optimal or lower\-order contexts provide structured evidence, trained transformers match or outperform classical count\-based baselines\. Our results bridge mechanistic interpretability of induction heads with classical statistical smoothing, revealing that transformers learn to regularize in\-context estimation rather than simply count\. ## Submission history From: Francesco D'Angelo \[[view email](https://arxiv.org/show-email/bdc7f83e/2607.02800)\] **\[v1\]**Thu, 2 Jul 2026 22:19:26 UTC \(1,477 KB\)
Similar Articles
Comparing Transformers and Hybrid Models at the Token Level
This paper analyzes token-level prediction differences between transformers and hybrid attention-recurrent models using Olmo 3 and Olmo Hybrid, finding that hybrids improve on semantic state tracking while transformers excel at n-gram copying and syntactic bracket matching.
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
This paper proposes intervention-based metrics to differentiate retrieval and positional heads in RoPE Transformers, leading to a principled hybrid architecture (HwH) that combines full and linear attention for improved language modeling and long-context extrapolation.
@_albertgu: Transformers are better at copying, while RNNs are better at modeling "meaning-bearing words—the nouns, verbs, & adject…
A thread from Ai2 compares transformer (Olmo 3) and hybrid (Olmo Hybrid) models, finding that transformers excel at copying while RNNs better model meaning-bearing words, highlighting the growing viability of hybrid architectures.
@TeachTheMachine: Using a Transformer Model: From Training to Inference
This tutorial covers how to use a transformer model from training to inference, focusing on autoregressive generation, prefill vs. decode phases, and key-value caching for efficient inference.
When Irregularity Helps: A Subclass Analysis of Inductive Bias in Neural Morphology
This paper investigates how character-level transformer models generalize to irregular verb subtypes in Japanese past-tense inflection. Controlled experiments show that including irregular examples can improve generalization, challenging the assumption that regularity simplifies learning.