The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers
Summary
This paper investigates how the grammatical role of tokens shapes the geometry of transformer representations across layers, finding distinct evolution patterns in encoder versus decoder models.
View Cached Full Text
Cached at: 08/27/26, 09:16 AM
# The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers Source: [https://arxiv.org/abs/2608.25166](https://arxiv.org/abs/2608.25166) [View PDF](https://arxiv.org/pdf/2608.25166) > Abstract:Transformer representations describe trajectories through high\-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers\. Such data tend to concentrate on lower\-dimensional sub\-manifolds, a form of compression quantified by the Intrinsic Dimensionality \(ID\), the minimum number of independent variables needed to represent them without significant information loss\. In this work, we ask whether the grammatical role of tokens, as marked by their part\-of\-speech \(PoS\) tag, shapes the local geometry of this manifold\. To this end: \(1\) We investigate the layer\-wise evolution of ID, finding that closed\-class items expand earlier and collapse sooner than open\-class ones; \(2\) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; \(3\) We compare encoders \(ModernBERT, bigbird\-roberta\-large\) and decoders \(gemma\-2\-2B, Llama\-3\.2\-3B\), finding that the two families evolve differently across layers, consistently with how each integrates context;\(4\) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task\. ## Submission history From: Federico Ravenda \[[view email](https://arxiv.org/show-email/b90e13cf/2608.25166)\] **\[v1\]**Tue, 25 Aug 2026 21:27:21 UTC \(4,586 KB\)
Similar Articles
Geometric and Behavioral Stratification in Transformer Residual Streams
This paper investigates the residual stream geometry of trained transformers, showing that the prediction direction acts as a privileged anchor that stratifies residual-stream variation into narrow, readout-relevant and broad, computational regions across 18 models. The findings have implications for interpretability and evaluation.
The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
Presents a continuous geometric framework modeling Transformer operations as integro-differential equations on a semantic fiber bundle, validated across multiple architectures.
An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars
This paper provides a theoretical analysis of deep transformers' ability to model hierarchical structures using bounded-depth context-free grammars, constructing explicit positional-attention transformers that encode grammatical states in linearly separable subspaces.
An Analysis of Residual-Stream Geometry Across Transformer Depth
This paper proposes a geometric analysis of transformer residual streams across depth, using relative displacement and orthogonal Procrustes analysis to reveal structured regularities in six instruction-tuned models on code generation and translation tasks.
Feature Geometry of LoRA Adapters: A Sparse Autoencoder Analysis of Representational Divergence in Fine-Tuned Language Models
This paper uses Sparse Autoencoders to analyze the geometry of LoRA-induced representations in language models, finding that LoRA updates occupy partially distinct feature structures not fully captured by pretrained interpretability dictionaries.