The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers
Summary
This paper investigates how the grammatical role of tokens shapes the geometry of transformer representations across layers, finding distinct evolution patterns in encoder versus decoder models.
View Cached Full Text
Cached at: 08/27/26, 09:16 AM
# The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers Source: [https://arxiv.org/abs/2608.25166](https://arxiv.org/abs/2608.25166) [View PDF](https://arxiv.org/pdf/2608.25166) > Abstract:Transformer representations describe trajectories through high\-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers\. Such data tend to concentrate on lower\-dimensional sub\-manifolds, a form of compression quantified by the Intrinsic Dimensionality \(ID\), the minimum number of independent variables needed to represent them without significant information loss\. In this work, we ask whether the grammatical role of tokens, as marked by their part\-of\-speech \(PoS\) tag, shapes the local geometry of this manifold\. To this end: \(1\) We investigate the layer\-wise evolution of ID, finding that closed\-class items expand earlier and collapse sooner than open\-class ones; \(2\) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; \(3\) We compare encoders \(ModernBERT, bigbird\-roberta\-large\) and decoders \(gemma\-2\-2B, Llama\-3\.2\-3B\), finding that the two families evolve differently across layers, consistently with how each integrates context;\(4\) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task\. ## Submission history From: Federico Ravenda \[[view email](https://arxiv.org/show-email/b90e13cf/2608.25166)\] **\[v1\]**Tue, 25 Aug 2026 21:27:21 UTC \(4,586 KB\)
Similar Articles
The Geometry of Inference in Transformer Residual Streams
This paper studies how transformer intermediate residual states become specialized to their own final output states, finding that directional alignment and endpoint rank improve even when Euclidean distance barely changes across six pretrained language models. It offers a high-dimensional model separating norm, alignment, and endpoint geometry, proving that straight-path convergence cannot introduce new competitors, and linking residual geometry to output token rankings.
Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation
This paper investigates how syntactic information is processed in later transformer layers using Greek language data, revealing a representational shift towards canonical word order through cross-layer generalization analysis.
Geometric and Behavioral Stratification in Transformer Residual Streams
This paper investigates the residual stream geometry of trained transformers, showing that the prediction direction acts as a privileged anchor that stratifies residual-stream variation into narrow, readout-relevant and broad, computational regions across 18 models. The findings have implications for interpretability and evaluation.
The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
Presents a continuous geometric framework modeling Transformer operations as integro-differential equations on a semantic fiber bundle, validated across multiple architectures.
An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars
This paper provides a theoretical analysis of deep transformers' ability to model hierarchical structures using bounded-depth context-free grammars, constructing explicit positional-attention transformers that encode grammatical states in linearly separable subspaces.