Convergent Evolution: How Different Language Models Learn Similar Number Representations
Summary
Study reveals that diverse language-model architectures independently evolve similar periodic Fourier features for representing numbers, with only some achieving geometric separability for modular arithmetic.
View Cached Full Text
Cached at: 04/23/26, 03:35 AM
Paper page - Convergent Evolution: How Different Language Models Learn Similar Number Representations
Source: https://huggingface.co/papers/2604.20817
Abstract
Transformers and other language models exhibit periodic numerical representations in their Fourier domains, with some models developing geometrically separable features for linear classification of numbers modulo T, though Fourier sparsity alone is insufficient for this separability.
Language models trained on natural text learn to represent numbers using periodic features with dominant periods at T=2, 5, 10. In this paper, we identify a two-tiered hierarchy of these features: whileTransformers,Linear RNNs,LSTMs, and classicalword embeddingstrained in different ways all learn features that haveperiod-T spikesin theFourier domain, only some learn geometrically separable features that can be used to linearly classify a numbermod-T. To explain this incongruity, we prove thatFourier domain sparsityis necessary but not sufficient formod-Tgeometric separability. Empirically, we investigate when model training yields geometrically separable features, finding that the data, architecture, optimizer, and tokenizer all play key roles. In particular, we identify two different routes through which models can acquire geometrically separable features: they can learn them from complementaryco-occurrence signalsin general language data, includingtext-number co-occurrenceandcross-number interaction, or from multi-token (but not single-token) addition problems. Overall, our results highlight the phenomenon ofconvergent evolutionin feature learning: A diverse range of models learn similar features from different training signals.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2604\.20817
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.20817 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.20817 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.20817 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
@LiorOnAI: Large language models spontaneously develop the same specialized brain regions humans have for language, math, physics,…
A preprint finds that large language models spontaneously develop specialized modular brain regions for language, math, physics, and social reasoning, similar to the human brain, suggesting convergence in intelligent system design.
Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning
This paper investigates the Platonic Representation Hypothesis by examining 16 language models across 8 families on 800 reasoning problems. It finds that while models converge in internal representations, they diverge in reasoning processes, especially post-decision, and shared representations have minimal causal influence on predictions.
Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models
This paper investigates when large language models develop domain-specific parametric shells (causally necessary neuron populations), finding that modular training data at the token level (e.g., languages, code) produces functional shells, while academic subject domains do not, despite being linearly decodable.
Local and Global Regimes of Geometric Complexity in Language Model Representations
This paper investigates how lexical diversity affects intrinsic dimensionality estimates in language model representations, revealing a scale-dependent transition between two regimes and deriving a formula for the reversal point.
Modular Cognitive Architecture Emerges in Large Language Models
This paper investigates whether modular cognitive architecture emerges in large language models, finding that LLMs develop specialized neural networks mirroring human brain organization across cognitive domains, suggesting modularity is a fundamental property of intelligent systems.