Local and Global Regimes of Geometric Complexity in Language Model Representations

# Local and Global Regimes of Geometric Complexity in Language Model Representations
Source: [https://arxiv.org/html/2608.14361](https://arxiv.org/html/2608.14361)
Marco BaroniAffiliation:Universitat Pompeu Fabra \(UPF\)Affiliation:ICREACorrespondence:arwa\.osman@upf\.eduIuri MacoccoAffiliation:Universitat Pompeu Fabra \(UPF\)

###### Abstract

Intrinsic dimensionality \(ID\) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed\. In this paper, we focus specifically on how lexical diversity, the number of unique last\-token items present in a dataset, affects ID estimates of that dataset\. We find a scale\-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID\. We derive an exact, parameter\-free formula for the point at which this reversal occurs, which matches the observed transition point at every scale tested\. On the one hand, our results highlight how care must be taken when interpreting the intrinsic dimensionality of a set of representations as a straightforward cue of their complexity\. On the other hand, our discovery of the two ID regimes reveals a general principle of organisation of linguistic data in LLMs that sheds new light on their inner manifold structures\.

## 1Introduction

Intrinsic dimensionality \(ID\) is the effective number of degrees of freedom needed to describe the structure of high\-dimensional data\([1](https://arxiv.org/html/2608.14361#bib.bib5);[16](https://arxiv.org/html/2608.14361#bib.bib11);[15](https://arxiv.org/html/2608.14361#bib.bib7)\), and it has become a common tool in NLP for probing the representational complexity of language models\([18](https://arxiv.org/html/2608.14361#bib.bib15);[19](https://arxiv.org/html/2608.14361#bib.bib17);[5](https://arxiv.org/html/2608.14361#bib.bib8)\): how “spread out” or constrained hidden states are across layers or models\.

However, it remains poorly understood whether ID measurements reflect properties of language itself, or simply artefacts of how the underlying dataset was constructed\. For example, a natural question is whether different parts of speech differ in intrinsic dimensionality\([7](https://arxiv.org/html/2608.14361#bib.bib19)\)\. We find that they do: in a naive comparison, nouns occupy a substantially higher\-ID space than prepositions across nearly every layer of a transformer \(Figure[1](https://arxiv.org/html/2608.14361#S1.F1), left\)\. One could be tempted to conclude that the larger semantic variety of nouns is reflected by richer geometric structure in the transformer representations\.

But this comparison is not as fair as it first appears\. In natural text, the noun class contains thousands of unique word types, while prepositions are drawn from a closed set of only a few dozen\. When we repeat the comparison after matching the two classes in both lexical diversity and token frequency, the pattern does not just weaken; it reverses: prepositions now showhigherID than nouns across the layers \(Figure[1](https://arxiv.org/html/2608.14361#S1.F1), right\)\. What looked like a part\-of\-speech effect turns out to be an artefact of lexical diversity\.

Figure 1:Apparent NOUN vs\. ADP differences in intrinsic dimensionality reverse once lexical diversity is controlled \(GRIDE, Qwen3\-8B,k=128k=128;N=10,000N=10\{,\}000samples\)\.Left:naive comparison, unmatched vocabulary size \(NOUN: 2600\+ types; ADP:∼\\sim43 types\)\.Right:both classes matched to 25 types, with 400 tokens per type \(Section[3](https://arxiv.org/html/2608.14361#S3)\)\.This example leads to the question we address in this paper: independent of syntactic category or semantic content, how does lexical diversity shape the intrinsic dimensionality of contextualised word representations? We isolate lexical diversity as a controlled experimental variable\. Using a custom WikiText\-derived dataset, we construct eleven conditions spanning from 1 to 10,000 unique contextualised nouns, holding total sample size fixed atN=10,000N=10\{,\}000across all conditions, and estimate ID using the generalised ratios intrinsic dimension estimator \(GRIDE\), introduced by[6](https://arxiv.org/html/2608.14361#bib.bib9), across the hidden states of two language models: Qwen3\-8B and Meta\-Llama\-3\-8B\.

Our results reveal a transition in ID that depends on how the neighbourhood scale used to estimate it compares to the number of samples available per word: when the scale is small relative to this sample count, ID reflects a word’s own contextual variability, while once the scale reaches it, ID instead reflects the diversity of the vocabulary as a whole\. We derive the exact point at which this shift occurs and confirm it empirically across models, estimators, and word classes\.

Any comparison of ID across categories, whether word classes, languages, domains, or models, must not rest solely on the assumption that the categories differ only in the property under study\. We show this assumption can fail without any visible sign: two categories can diverge in ID purely because they differ in lexical diversity, and the resulting curves look just as clean and reproducible as a genuine effect would\. More generally, our results show how the numerical values returned by standard ID estimators might, depending on scale, be based on very different views of the same broader manifold\.

## 2Related Work

#### Methods for estimating intrinsic dimensionality\.

Intrinsic dimensionality \(ID\) refers to the minimal number of parameters required to describe a data representation without significant loss of information\([1](https://arxiv.org/html/2608.14361#bib.bib5)\), capturing the effective number of degrees of freedom along the manifold on which the data resides\([15](https://arxiv.org/html/2608.14361#bib.bib7)\)\. Estimating ID in high\-dimensional data is a well\-studied problem, with methods developed to address challenges of noise, scale, and computational cost\([4](https://arxiv.org/html/2608.14361#bib.bib10);[12](https://arxiv.org/html/2608.14361#bib.bib6)\)\. Maximum Likelihood Estimation \(MLE\), proposed by[12](https://arxiv.org/html/2608.14361#bib.bib6), models the points within a small neighbourhood of a query point as a homogeneous Poisson process, and estimates ID from the log\-ratio of the distance to thekk\-th nearest neighbour relative to each closer neighbour; here,kkdirectly sets the size of the local neighbourhood used to estimate the density around each point, with largerkkincorporating more neighbours into the estimate at the cost of a stronger assumption of locally constant density\. This estimator is computationally efficient but can underestimate ID in high\-dimensional or complex\-manifold settings due to negative bias\([6](https://arxiv.org/html/2608.14361#bib.bib9)\)\. TwoNN\([9](https://arxiv.org/html/2608.14361#bib.bib12)\)instead relies on the ratio of distances to the first and second nearest neighbours, offering robustness to local noise but remaining sensitive to short\-scale irregularities; a common mitigation, data decimation, reduces this bias at the cost of statistical power\([6](https://arxiv.org/html/2608.14361#bib.bib9)\)\. GRIDE\([6](https://arxiv.org/html/2608.14361#bib.bib9)\)addresses these limitations by leveraging higher\-order nearest neighbours to estimate ID across multiple neighbourhood scales, mitigating noise by modelling distance ratios at larger scales without the power loss associated with decimation\. This multi\-scale property is central to our approach, as it allows us to directly examine how ID estimates shift with neighbourhood scalekk\.

#### Intrinsic dimensionality in neural network representations\.

ID has been used to study the geometry of representations in deep networks since[1](https://arxiv.org/html/2608.14361#bib.bib5)showed that ID follows a characteristic hump\-shaped profile across layers, first expanding, probably to disentangle features in early layers, before compressing toward the output, with lower ID in the final hidden layer correlating with better generalisation\.[16](https://arxiv.org/html/2608.14361#bib.bib11)further established that this compression, the degree to which a network reduces the dimensionality of its final\-layer representations, is itself a strong predictor of generalisation to unseen data\.[15](https://arxiv.org/html/2608.14361#bib.bib7)extended this line of work by estimating the ID of common image datasets, finding that even high\-resolution ImageNet images have an intrinsic dimensionality several orders of magnitude smaller than their embedding dimension, and showing that dataset ID correlates with learning difficulty\. More recently,[5](https://arxiv.org/html/2608.14361#bib.bib8)identified a distinct high\-dimensional “abstraction phase” in transformer language models, in which ID rises sharply at intermediate layers before collapsing toward lower\-dimensional representations near the output, interpreted as evidence that models integrate abstract linguistic information before refining it in later layers\.

#### ID as a cue of complexity in NLP settings\.

ID has increasingly been used as a cue of linguistic or representational complexity in NLP\.[18](https://arxiv.org/html/2608.14361#bib.bib15)used ID to detect AI\-generated text, exploiting the observation that machine\-generated text occupies a lower\-dimensional subspace than human\-written text\.[21](https://arxiv.org/html/2608.14361#bib.bib14)characterised model truthfulness in question answering using local intrinsic dimension, showing that hallucinated model outputs occupy higher\-dimensional activation manifolds than correct ground\-truth answers\.[19](https://arxiv.org/html/2608.14361#bib.bib17)examined the geometry of hidden representations across transformer layers to probe representational complexity more broadly\. More directly relevant to our concerns,[7](https://arxiv.org/html/2608.14361#bib.bib19)find that open\-class words occupy more isotropic, higher\-dimensional subspaces than closed\-class words across both encoder and decoder architectures, and note that the two groups also differ in lexical and syntactic diversity, without isolating diversity as a controlled variable\. Similarly,[3](https://arxiv.org/html/2608.14361#bib.bib21)trace how ID varies across a range of linguistic phenomena to draw conclusions about their relative complexity\. These studies illustrate exactly the kind of inference our work cautions against: attributing ID differences to a linguistic property of interest without ruling out differences in the underlying richness of the compared categories\.

#### Confounds in ID estimation\.

A separate line of work has questioned whether ID estimates on neural representations reliably track the quantity they are assumed to measure\.[17](https://arxiv.org/html/2608.14361#bib.bib20)show both theoretically and empirically that common ID estimators do not track the true underlying ID of a representation, and investigate which factors actually drive commonly reported ID results in the literature\.

Taken together, the current literature has established ID as a valuable probe of representational complexity in both vision and language models\. Yet, input diversity remains an overlooked source of variation in ID estimates\. To our knowledge, no prior work isolates lexical diversity as an experimental variable in its own right\. We address this gap directly\.

## 3Experimental Setup

### 3\.1Dataset

#### Sample Construction\.

We use WikiText\-103\([13](https://arxiv.org/html/2608.14361#bib.bib3)\), a large\-scale English corpus derived from verified Wikipedia articles, which we part\-of\-speech tag using spaCy\([11](https://arxiv.org/html/2608.14361#bib.bib13)\)\. From the tagged corpus, we construct approximately 84 million samples, each consisting of four consecutive non\-overlapping sentences, with a target word randomly selected from the fourth sentence and its POS tag recorded; placing the target in the fourth sentence ensures that its representation is conditioned on three complete sentences of preceding context\. Each sample is truncated at the target word, making it the final token the model processes\.

We focus on nouns as target words, since their large natural vocabulary allows us to construct conditions spanning a wide range of lexical diversity, from a single repeated noun to thousands of unique types\. To ensure POS purity and avoid tagging noise, we retain only words tagged as a noun in at least 80% of their occurrences and appearing at least 300 times in our corpus, yielding a contextualised noun pool from which our dataset conditions are constructed\. We additionally construct a POS\-diversified variant of this dataset, sampling target words across multiple POS categories rather than nouns only; details are given in Appendix[A](https://arxiv.org/html/2608.14361#A1)\.

#### Dataset Conditions\.

From this contextualised noun pool, we construct eleven datasets at different lexical diversity levels, each fixed atN=10,000N=10\{,\}000samples, by samplingnnunique nouns withm=N/nm=N/nsamples per noun\. Critically, lexical diversity here refers only to the target noun’s identity, not its context: the preceding sentences still differ across samples regardless of diversity level, drawn independently from the corpus, so our conditions isolate the target word’s diversity alone\. To assess robustness, for each lexical diversity level we construct five independent partitions, each with the same set of unique nouns but different randomly drawn samples\. The eleven conditions are summarised in Table[1](https://arxiv.org/html/2608.14361#S3.T1)\.

Table 1:Lexical diversity conditions\.

### 3\.2Models

We use Qwen3\-8B\([20](https://arxiv.org/html/2608.14361#bib.bib2)\)\(36 layers, hidden size 4096\) as our primary model\. We additionally use Meta\-Llama\-3\-8B\([14](https://arxiv.org/html/2608.14361#bib.bib1)\)for cross\-architecture validation \(Section[7\.2](https://arxiv.org/html/2608.14361#S7.SS2)\)\.

### 3\.3ID Estimation

For each sample, we extract the hidden state at the final token position from every layer of the model\. We then estimate the intrinsic dimensionality \(ID\) of these representations using GRIDE\. Formally, for a pointiiwithn1n\_\{1\}\-th andn2n\_\{2\}\-th nearest\-neighbour distancesri,n1r\_\{i,n\_\{1\}\}andri,n2r\_\{i,n\_\{2\}\}, GRIDE estimates the intrinsic dimensionddby maximum likelihood from the distribution of the ratioμi=ri,n2/ri,n1\\mu\_\{i\}=r\_\{i,n\_\{2\}\}/r\_\{i,n\_\{1\}\}, whose density depends ondd\([6](https://arxiv.org/html/2608.14361#bib.bib9)\)\. We follow then2=2​n1n\_\{2\}=2n\_\{1\}convention recommended by[6](https://arxiv.org/html/2608.14361#bib.bib9)as a robust trade\-off between scale coverage and computational cost, and refer tokk\(vian2n\_\{2\}\) as the neighbourhood scale used by GRIDE at each estimate\. Intuitively,kksets how far into the representation space the estimator looks when comparing distances: smallkkcaptures the local geometry immediately surrounding a point, while largerkkcaptures how that point’s neighbourhood relates to increasingly distant regions of representation space\. We run GRIDE at six scalesk∈\{16,32,64,128,256,512\}k\\in\\\{16,32,64,128,256,512\\\}\.

Estimation is performed using the DADApy library\([10](https://arxiv.org/html/2608.14361#bib.bib4)\)\. ID is estimated independently for each of the five partitions per condition, and we report the mean and standard deviation of these estimates across partitions\. This variance is typically extremely low, so the standard deviation is often not visible in the plots\.

## 4A Scale\-Dependent Transition in ID

Figure[2](https://arxiv.org/html/2608.14361#S4.F2)shows ID across layers for Qwen3\-8B at six scales\. At every scale, conditions separate into two regimes that hold across all layers: low\-diversity conditions remain in a low\-ID regime throughout the network, while high\-diversity conditions remain in a high\-ID regime throughout\. In the low\-ID regime, the relationship between lexical variety and ID is inverse: conditions withfewerunique nouns have higher ID\. In the high\-ID regime, the relationship becomes direct:moreunique nouns lead to higher ID, up to a level at which the ID estimate tends to stabilise across datasets\.

Between these two regimes lies a single condition, a “transition point”, whose curve \(displayed with a thicker line in the figure\) does not stay in either regime: it starts near the low\-ID cluster in early layers and rises across layers to join the high\-ID cluster by the final layers\. This transition point shifts systematically with scale: askkincreases, it occurs at progressively lower lexical diversity\.

A complementary view of the same phenomenon, organised by lexical diversity level rather than scale, is given in Appendix[B](https://arxiv.org/html/2608.14361#A2)\(Figure[6](https://arxiv.org/html/2608.14361#A2.F6)\)\.

Figure 2:ID curves across layers for all considered scales and lexical\-diversity conditions on Qwen3\-8B\. The “transition point” curve is shown with a thicker line\.### 4\.1Local and Global Regimes

We now explain why these two regimes exist, and why the transition between them occurs where it does\. Since GRIDE is a nearest\-neighbour estimator, the transition in ID must ultimately be explained by a change in the nearest\-neighbour structure of the data\. We build on prior evidence that, despite being context\-sensitive, representations of the same word in different contexts remain more similar to one another than representations of different words\([8](https://arxiv.org/html/2608.14361#bib.bib18)\), and assume that this holds strongly enough for different contextualised representations of the same word to cluster closer together than representations of different words\. Given this assumption, a neighbourhood of sizekkfalls into one of two cases: either allkknearest neighbours belong to the same word type, the local regime, or at least one neighbour belongs to a different word type, the global regime \(Figure[3](https://arxiv.org/html/2608.14361#S4.F3)\)\.

In the local regime, the estimator samples entirely within a single word’s representation cloud, measuring how the model varies representations of the same word across different contexts\. In the global regime, the neighbourhood extends beyond the word cloud, and the estimator instead measures the geometry of the broader noun space\.

Figure 3:Schematic illustration of the local and global regimes\.Left:at low lexical diversity, a small neighbourhood scale \(blue\) stays within the same word’s cluster \(local regime\), while a large scale \(orange\) already extends into a different word’s cluster \(global regime\)\.Right:at high lexical variety, clusters are smaller and more numerous, so even a small scale can extend beyond the target word’s own cluster\.
### 4\.2Deriving the Transition Point

Letnnbe the number of unique nouns,m=N/nm=N/nthe number of samples per noun, andkkthe GRIDE scale\. For any query point, there are at mostm−1m\-1other samples ending with the same word\. Therefore, ifk<mk<m, the entire neighbourhood can in principle be filled with same\-word samples, and the local regime is geometrically possible\. Atk=mk=m, at least one neighbour must come from a different word type, marking the boundary at which the global regime is forced\. The transition therefore occurs when:

m=k⟹Nn=k⟹ntransition=Nk\.m=k\\;\\Longrightarrow\\;\\frac\{N\}\{n\}=k\\;\\Longrightarrow\\;n\_\{\\mathrm\{transition\}\}=\\frac\{N\}\{k\}\.\(1\)
The minimum fraction of thekkneighbours that must come from different word types, which we call theglobal ratio, is:

ρmin​\(n,k\)=max⁡\(0,1−m−1k\)\.\\rho\_\{\\min\}\(n,k\)=\\max\\\!\\left\(0,\\,1\-\\frac\{m\-1\}\{k\}\\right\)\.\(2\)

### 4\.3Confirming the Transition Across Scales

Our derivation implies that the transition point should depend only on the ratiok/mk/m, rather than onkkandmmindividually\. If this holds, curves measured at very different scales \(k=16k=16tok=512k=512\) should collapse onto the same trajectory when plotted against this ratio, each crossing through the transition atk/m=1k/m=1\.

Figure[4](https://arxiv.org/html/2608.14361#S4.F4)tests this prediction\. Each panel shows one layer of Qwen3\-8B, with one line per scalekk\. Despite this wide range of rawkkvalues, all six lines collapse onto a single trajectory once plotted againstk/mk/m, and all of them rise through the predicted transition point \(dashed red line\), with no fitted parameters\. This holds consistently across early, middle, and late layers \(5, 21, 30, and 36\), indicating that a single rule, governed entirely by the neighbourhood\-counting argument above, explains the transition regardless of scale or network depth\. We note that layer 36, the final layer, shows a somewhat noisier collapse than earlier layers, likely reflecting a broader reshuffling of neighbourhood structure at the output layer that is not specific to the local\-global transition itself\.

Figure 4:ID as a function ofk/mk/macross four layers of Qwen3\-8B \(layers 5, 21, 30, and 36\)\.

## 5Empirical Verification

The theoretical derivation in Section[4\.2](https://arxiv.org/html/2608.14361#S4.SS2)assumes that representations of the same word type form local clusters in the embedding space\. We test this assumption by measuring thesame\-word fraction: for each token, the proportion of itskknearest neighbours that share the same word type, evaluated atk=m−1k=m\-1for each lexical diversity condition, and averaged across all datapoints\.

Figure[5](https://arxiv.org/html/2608.14361#S5.F5)shows this fraction across layers for each condition \(excludingn=1n=1, where the fraction is 1, andn=10,000n=10\{,\}000, wherem−1=0m\-1=0makes the diagnostic undefined\)\. In early and middle layers, the same\-word fraction remains close to 1 for all conditions, confirming that same\-word tokens almost entirely fill the neighbourhood, exactly as assumed\. In later layers, this fraction declines, most noticeably for high\-diversity conditions, falling to around 0\.2–0\.4 by the final layers\. This behaviour is also reflected in the thick transition ID curves in Figure[2](https://arxiv.org/html/2608.14361#S4.F2), where indeed the last layers display a pronounced peak that is not present otherwise\.

The shape of these curves and their final decline show that the local\-cloud assumption is not strictly layer\-independent: same\-word tokens do not remain perfectly segregated from other word types throughout the network\. However, even at its lowest point in later layers, the same\-word fraction remains far above the no\-clustering baseline shown by the dashed lines in Figure[5](https://arxiv.org/html/2608.14361#S5.F5)\.

If representations were not clustered by word type, a token’s neighbours would be drawn uniformly from thennword types, giving an expected same\-word fraction of approximately1/n1/n—three orders of magnitude below the fractions observed even in late layers for the most diverse conditions\.

Figure 5:Same\-word fraction across layers, evaluated atk=m−1k=m\-1for each lexical diversity condition\.
## 6Geometric Properties of the Two Regimes

Having established that the transition is real and geometrically driven, we characterise how ID behaves within each regime\. Two consistent observations emerge across all scales\.

#### Curve ordering reverses at the transition\.

In the local regime, conditions with fewer unique nouns produce higher ID\. With fewer unique nouns, more samples per noun are available\. We conjecture that, in this setup, the estimator is “exploring” to a greater extent the word’s contextual variation across all geometric directions\. The larger the cluster of same\-token points around each target word \(fewer unique types with more tokens each\), the richer this local geometry will be\. Hence, we observe IDdecreasingwith higher lexical diversity\. In the global regime, on the other hand, the ordering inverts: conditions with more unique nouns produce higher ID\. We hypothesise that, as neighbours are now instances of different nouns that will occupy more independent geometric directions in representational space, the larger the set of noun types in the sample, the more accurately we can estimate the overall size of the global noun manifold by looking at a wider set of independent dimensions\. This reversal is visible across all scales in Figure[2](https://arxiv.org/html/2608.14361#S4.F2)\.

#### ID saturates in the global regime\.

Within the global regime, ID increases with lexical diversity, but with strongly diminishing returns\. Asnngrows large,ρ→1\\rho\\to 1asymptotically: the neighbourhood is already almost entirely filled with different\-word tokens, so adding more unique nouns changes its composition only marginally\. At scale 512, for instance, the difference in global ratio between 2,500 nouns \(99\.4%\) and 5,000 nouns \(99\.8%\) is only 0\.4%, and their ID curves are nearly indistinguishable across all layers, suggesting that, at these sizes, we are reaching a stable estimate of the ID of noun representations\.

#### Do scale effects affect ID profiles?

Arguments about deep net geometry are often cast in terms of theprofileof ID increase or decrease across network layers\. For example, both[19](https://arxiv.org/html/2608.14361#bib.bib17)and[5](https://arxiv.org/html/2608.14361#bib.bib8)reported the presence of an ID “peak” in the central layers of LLMs and other models\. By looking at the profiles in Figure[2](https://arxiv.org/html/2608.14361#S4.F2), we see that the central peak pattern is relatively stable across scales and datasets, suggesting that it is a genuine property of how the geometric organisation of representations evolves across network layers\. From this perspective, it is indeed interesting that the peak emerges in both regimes, suggesting that it is a property that characterises how representations are organised by the network both within tight, clustered neighbourhoods \(local regime\), and when considering a wide sample of unrelated points spanning a large space \(in our case, the nominal domain, as explored in the global regime\)\. However, note that at the transition point \(thicker lines in the same plot\), we also observe a second, sharper peak\. It would be tempting to interpret this second peak as another substantive property of how models distribute representations in their space, but actually our analysis suggests that it is simply an artefact of the transition from the local to the global regime in the later layers\.

## 7Cross\-Validation

To test whether the transition reflects a general geometric property of lexical diversity, rather than an artefact of our specific estimator, model, or word class, we replicate our main analysis along three independent axes: estimator, model architecture, and part\-of\-speech composition\. Full results for each are given in Appendix[C](https://arxiv.org/html/2608.14361#A3)\.

### 7\.1MLE Replication

We repeat the analysis using the Maximum Likelihood Estimator \(MLE\) of[12](https://arxiv.org/html/2608.14361#bib.bib6), as implemented in theskdimlibrary\([2](https://arxiv.org/html/2608.14361#bib.bib16)\), in place of GRIDE\. The same transition pattern emerges at every scalekk: the condition atn=N/kn=N/knouns separates from the cluster, matching the GRIDE results exactly \(Appendix[C](https://arxiv.org/html/2608.14361#A3), Figure[7](https://arxiv.org/html/2608.14361#A3.F7)\)\. This indicates the transition is a property of the underlying representations rather than a characteristic of GRIDE\-based ID estimation specifically\.

### 7\.2Cross\-Architecture Validation

We repeat the analysis on Meta\-Llama\-3\-8B\([14](https://arxiv.org/html/2608.14361#bib.bib1)\)\. At all six scales tested, the transition occurs at the predicted levelntransition=N/kn\_\{\\mathrm\{transition\}\}=N/k\(Appendix[C](https://arxiv.org/html/2608.14361#A3), Figure[8](https://arxiv.org/html/2608.14361#A3.F8)\), indicating that the transition point is governed by dataset construction rather than by architecture\-specific properties of any single model\.

### 7\.3POS\-diversified Dataset

Finally, we test whether the transition depends on nouns specifically, or reflects lexical diversity more generally\. We construct a POS\-diversified dataset, sampling words across nouns, verbs, adjectives, adverbs, and proper nouns, with adverbs excluded at the highest diversity levels due to data availability \(Appendix[A](https://arxiv.org/html/2608.14361#A1)\)\. The transition pattern is preserved across all six scales \(Appendix[C](https://arxiv.org/html/2608.14361#A3), Figure[9](https://arxiv.org/html/2608.14361#A3.F9)\), indicating that the effect is driven by lexical diversity itself rather than any property specific to nouns\.

## 8Discussion and Conclusion

The central finding of this work is that neighbourhood\-based ID estimators applied to LLM contextualised word representations operate in two qualitatively distinct modes depending on the relationship between lexical diversity, sample size, and scale\. In the local regime, they measure the contextual variation of individual lexical items\. In the global regime, they measure how the model distributes different word types relative to one another\. These are meaningful geometric quantities, but they are not the same quantity, and comparing ID values across the two regimes is not straightforward\.

This point is illustrated directly by our own results\. The naive noun\-versus\-preposition comparison in Figure[1](https://arxiv.org/html/2608.14361#S1.F1)involved two word classes that differ substantially in lexical diversity, and, once they were matched on lexical diversity, the result of the comparison reversed\. This suggests that the original difference was driven by how the underlying dataset was constructed rather than by any intrinsic property of the syntactic categories themselves\.

The relationship between lexical diversity and ID likewise differs between the two regimes\. Below the transition, lexical diversity and ID are negatively related: more unique nouns mean fewer samples per noun, sparser cloud sampling, and lower ID\. Above the transition, they are positively related: more unique nouns mean more word clouds, a richer inter\-word geometry, and higher ID\. At the transition point, spurious ID effects might appear, such as a second “peak” in the ID profile that is due to the transition and a more pronounced neighbourhood reorganisation\.

We also observed, within the global regime, a ceiling beyond which increasing lexical diversity yields no additional geometric information\. This ceiling is determined by the global ratio approaching 1 and can be predicted in advance from Equation[2](https://arxiv.org/html/2608.14361#S4.E2)\. On a positive note, the fact that we observe this ceiling suggests that the sample sizes we are considering are sufficient to come up with a stable estimate of global\-regime ID\.

More broadly, our findings suggest that ID differences attributed to a linguistic or task\-relevant property may in some cases reflect uncontrolled differences in dataset construction between the compared categories, rather than the property itself\. Any study using ID to compare word representations across conditions with different lexical diversity must verify which regime each condition falls into\. The formulantransition=N/kn\_\{\\mathrm\{transition\}\}=N/kand the same\-word fraction diagnostic introduced in Section[5](https://arxiv.org/html/2608.14361#S5)provide the tools to do this\. Conditions in the same regime can be compared directly; conditions in different regimes measure different aspects of the geometry and require careful interpretation\. Note that we have focused on lexical diversity because it is an easy\-to\-control property with a predictable effect on neighbourhood structure, but other properties, whose effect might be more difficult to measure, might also affect how data are clustered and, consequently, the ID regime we are in\. For example, when comparing a generic text to a highly specialised one \(e\.g\., in a technical domain\), it is possible that samples from the latter will form a tighter cluster than samples from the former\. Future work should assess the impact of such fuzzier factors on ID\.

It is interesting and relevant to underline that ID can be used as a probe of the structure of both local neighbourhoods, where the space surrounding a point is occupied by very similar representations, and global neighbourhoods, such as the large space spanned by a random selection of unrelated nouns\. It is remarkable that, while other properties change with scale, certain basic features of ID profiles, such as the presence of a mid\-layer peak or bump\([19](https://arxiv.org/html/2608.14361#bib.bib17);[5](https://arxiv.org/html/2608.14361#bib.bib8)\), appear in both regimes, suggesting a “fractal\-like” structure where the same patterns emerge at narrower and wider sections of the same manifold\. Future work should seek to understand why such patterns emerge\.

## Limitations

Our study has several limitations\. First, our dataset is derived entirely from WikiText\-103, an English\-language corpus of Wikipedia articles\. Whether the transition formulantransition=N/kn\_\{\\mathrm\{transition\}\}=N/kgeneralises to other languages, genres, or domains with different distributional properties remains untested\. Second, our models are limited to the 8B parameter scale\. While we validate our findings across two architectures \(Qwen3\-8B and Meta\-Llama\-3\-8B\), it is unclear whether the same relationship holds at substantially smaller or larger model scales, or whether the transition point itself depends on model size\.

Third, our theoretical derivation and the same\-word fraction diagnostic in Section[5](https://arxiv.org/html/2608.14361#S5)rely on the assumption that contextualised representations of the same word type cluster more closely together than representations of different words\. We find that this assumption holds strongly in early and middle layers, but weakens in later layers, where the same\-word fraction declines substantially for high\-diversity conditions\. Although this decline does not prevent the transition point from matching our predictions even in late layers, it indicates that same\-word tokens are not perfectly segregated from other word types at every layer: same\-word neighbours dominate far more often than chance would predict, but not with absolute certainty, and the reasons for this late\-layer weakening are not fully understood\.

## Acknowledgments

MB and IM received funding from the European Research Council \(ERC\) under the European Union’s Horizon 2020 research and innovation program \(grant agreement No\. 101019291\)\. We also thank the COLT: Computational Linguistics and Linguistic Theory group for their valuable feedback and discussions\.

## References

- Ansuiniet al\.\(2019\)A\. Ansuini, A\. Laio, J\. H\. Macke, and D\. ZoccolanIntrinsic dimension of data representations in deep neural networks\.InAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\-14, 2019, Vancouver, BC, Canada,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/hash/cfcce0621b49c983991ead4c3d4d3b6b-Abstract.html)Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px2.p1.1)\.
- Bacet al\.\(2021\)J\. Bac, E\. M\. Mirkes, A\. N\. Gorban, I\. Tyukin, and A\. ZinovyevScikit\-dimension: a Python package for intrinsic dimension estimation\.Entropy23\(10\),pp\. 1368\.External Links:[Document](https://dx.doi.org/10.3390/e23101368)Cited by:[§7\.1](https://arxiv.org/html/2608.14361#S7.SS1.p1.1)\.
- Baroniet al\.\(2026\)M\. Baroni, E\. Cheng, I\. de Dios\-Flores, and F\. FranzonTracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of llm representations\.External Links:2601\.03779,[Link](https://arxiv.org/abs/2601.03779)Cited by:[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px3.p1.1)\.
- Camastra and Vinciarelli \(2003\)F\. Camastra and A\. VinciarelliData dimensionality estimation methods: a survey\.Pattern Recognit\.36\(12\),pp\. 2945–2954\.External Links:[Link](https://doi.org/10.1016/S0031-3203(03)00176-6),[Document](https://dx.doi.org/10.1016/S0031-3203%2803%2900176-6)Cited by:[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px1.p1.1)\.
- Chenget al\.\(2025\)E\. Cheng, D\. Doimo, C\. Kervadec, I\. Macocco, J\. Yu, A\. Laio, and M\. BaroniEmergence of a high\-dimensional abstraction phase in language transformers\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=0fD3iIBhlV)Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2608.14361#S6.SS0.SSS0.Px3.p1.1),[§8](https://arxiv.org/html/2608.14361#S8.p6.1)\.
- Dentiet al\.\(2022\)F\. Denti, D\. Doimo, A\. Laio, and A\. MiraThe generalized ratios intrinsic dimension estimator\.Scientific Reports12\(1\),pp\. 20005\.External Links:[Document](https://dx.doi.org/10.1038/s41598-022-20991-1),[Link](https://www.nature.com/articles/s41598-022-20991-1)Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p4.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2608.14361#S3.SS3.p1.1)\.
- Domenichelliet al\.\(2026\)L\. Domenichelli, D\. Brunato, and F\. Dell’OrlettaLinguistic profiling of transformer embedding geometry\.InProceedings of the 30th Conference on Computational Natural Language Learning,,pp\. 145–164\.Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p2.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px3.p1.1)\.
- Ethayarajh \(2019\)K\. EthayarajhHow contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT\-2 embeddings\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 55–65\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1006)Cited by:[§4\.1](https://arxiv.org/html/2608.14361#S4.SS1.p1.1)\.
- Faccoet al\.\(2017\)E\. Facco, M\. d’Errico, A\. Rodriguez, and A\. LaioEstimating the intrinsic dimension of datasets by a minimal neighborhood information\.Scientific reports7\(1\),pp\. 12140\.Cited by:[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px1.p1.1)\.
- Glielmoet al\.\(2022\)A\. Glielmo, I\. Macocco, D\. Doimo, M\. Carli, C\. Zeni, R\. Wild, M\. d’Errico, A\. Rodriguez, and A\. LaioDADApy: distance\-based analysis of data\-manifolds in python\.Patterns3\(10\),pp\. 100589\.External Links:[Link](https://doi.org/10.1016/j.patter.2022.100589),[Document](https://dx.doi.org/10.1016/J.PATTER.2022.100589)Cited by:[§3\.3](https://arxiv.org/html/2608.14361#S3.SS3.p2.1)\.
- Honnibal and Montani \(2017\)M\. Honnibal and I\. MontanispaCy 2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing\.Cited by:[§3\.1](https://arxiv.org/html/2608.14361#S3.SS1.SSS0.Px1.p1.1)\.
- Levina and Bickel \(2004\)E\. Levina and P\. J\. BickelMaximum likelihood estimation of intrinsic dimension\.InAdvances in Neural Information Processing Systems 17 \[Neural Information Processing Systems, NIPS 2004, December 13\-18, 2004, Vancouver, British Columbia, Canada\],pp\. 777–784\.External Links:[Link](https://proceedings.neurips.cc/paper/2004/hash/74934548253bcab8490ebd74afed7031-Abstract.html)Cited by:[§C\.1](https://arxiv.org/html/2608.14361#A3.SS1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2608.14361#S7.SS1.p1.1)\.
- Merityet al\.\(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24\-26, 2017, Conference Track Proceedings,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[§3\.1](https://arxiv.org/html/2608.14361#S3.SS1.SSS0.Px1.p1.1)\.
- Meta AI \(2024\)Meta AIIntroducing meta llama 3: the most capable openly available llm to date\.External Links:[Link](https://ai.meta.com/blog/meta-llama-3/)Cited by:[§3\.2](https://arxiv.org/html/2608.14361#S3.SS2.p1.1),[§7\.2](https://arxiv.org/html/2608.14361#S7.SS2.p1.1)\.
- Popeet al\.\(2021\)P\. Pope, C\. Zhu, A\. Abdelkader, M\. Goldblum, and T\. GoldsteinThe intrinsic dimension of images and its impact on learning\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=XJk19XzGq2J)Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px2.p1.1)\.
- Recanatesiet al\.\(2019\)S\. Recanatesi, M\. Farrell, M\. S\. Advani, S\. Lahiri, and S\. GanguliDimensionality compression and expansion in deep neural networks\.External Links:1906\.00443,[Link](http://arxiv.org/abs/1906.00443)Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px2.p1.1)\.
- Schulte and Rügamer \(2026\)R\. Schulte and D\. RügamerRethinking intrinsic dimension estimation in neural representations\.InInternational Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px4.p1.1)\.
- Tulchinskiiet al\.\(2023\)E\. Tulchinskii, K\. Kuznetsov, K\. Laida, D\. Cherniavskii, S\. Nikolenko, E\. Burnaev, S\. Barannikov, and I\. PiontkovskayaIntrinsic dimension estimation for robust detection of ai\-generated texts\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=8uOZ0kNji6)Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px3.p1.1)\.
- Valerianiet al\.\(2023\)L\. Valeriani, D\. Doimo, F\. Cuturello, A\. Laio, A\. Ansuini, and A\. CazzanigaThe geometry of hidden representations of large transformer models\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 51234–51252\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a0e66093d7168b40246af1cddc025daa-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.14361#S1.p1.1),[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.14361#S6.SS0.SSS0.Px3.p1.1),[§8](https://arxiv.org/html/2608.14361#S8.p6.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388),[Document](https://dx.doi.org/10.48550/arXiv.2505.09388)Cited by:[§3\.2](https://arxiv.org/html/2608.14361#S3.SS2.p1.1)\.
- Yinet al\.\(2024\)F\. Yin, J\. Srinivasa, and K\. ChangCharacterizing truthfulness in large language model generations with local intrinsic dimension\.InProceedings of ICML,Vienna, Austria,pp\. 57069–57084\.Cited by:[§2](https://arxiv.org/html/2608.14361#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix APOS\-Diversified Dataset

In addition to the noun\-only dataset described in Section[3](https://arxiv.org/html/2608.14361#S3), we construct a POS\-diversified variant in which target words are sampled across multiple part\-of\-speech categories\. The dataset follows the same construction procedure and lexical diversity levels as the main dataset, withN=10,000N=10\{,\}000samples per condition\.

Forn≤1,250n\\leq 1\{,\}250unique words, target words are sampled uniformly across five POS categories:noun,verb,adj,adv, andpropn\. Forn≥2,500n\\geq 2\{,\}500, adverbs are excluded and sampling is restricted to four categories:noun,verb,adj, andpropn\. This adjustment is necessary because adverbs occur with insufficient frequency in the corpus to satisfy the minimum sample requirements at high lexical diversity levels\.

The same POS purity and minimum frequency filters applied to the noun\-only dataset are applied independently to each POS category\.

## Appendix BTransition Point by Lexical Diversity

Figure[6](https://arxiv.org/html/2608.14361#A2.F6)presents the same data as Figure[2](https://arxiv.org/html/2608.14361#S4.F2), reorganised to allow a complementary reading: each panel now corresponds to a fixed lexical diversity levelnn, with one line per scalekk, rather than a fixed scale with one line pernn\. Within each panel, the bolded line marks the scalekkat which that lexical diversity level serves as the transition point between the local and global regimes\.

Figure 6:ID curves across layers for each lexical diversity levelnn, with one line per scalekk\. The bold line marks the transition scale\.
## Appendix CCross\-Validation Results

This appendix presents the full figures supporting the robustness checks described in Section[7](https://arxiv.org/html/2608.14361#S7): replication with an alternative ID estimator \(MLE\), replication across model architectures, and replication on a part\-of\-speech\-diversified dataset\.

### C\.1MLE Replication

Figure[7](https://arxiv.org/html/2608.14361#A3.F7)shows ID curves across all layers of Qwen3\-8B at all six scales, estimated using the Maximum Likelihood Estimator \(MLE\) of[12](https://arxiv.org/html/2608.14361#bib.bib6)in place of GRIDE\. The transition pattern matches the GRIDE results in Figure[2](https://arxiv.org/html/2608.14361#S4.F2)exactly, with the condition atn=N/kn=N/knouns separating from the cluster at every scale\.

Figure 7:MLE ID curves across layers for all scales on Qwen3\-8B\. The transition pattern observed with GRIDE \(Figure[2](https://arxiv.org/html/2608.14361#S4.F2)\) is reproduced at every scalekk\.
### C\.2Cross\-Architecture Validation

Figure[8](https://arxiv.org/html/2608.14361#A3.F8)shows ID curves across all layers and all six scales for Meta\-Llama\-3\-8B\. The transition occurs at the predicted levelntransition=N/kn\_\{\\mathrm\{transition\}\}=N/kat every scale, matching the pattern observed on Qwen3\-8B\.

Figure 8:ID curves across layers for all scales on Meta\-Llama\-3\-8B\. The transition occurs atn=N/kn=N/knouns at every scale, matching the pattern observed on Qwen3\-8B \(Figure[2](https://arxiv.org/html/2608.14361#S4.F2)\)\.
### C\.3POS\-diversified Dataset

Figure[9](https://arxiv.org/html/2608.14361#A3.F9)shows ID curves across layers and all six scales for Qwen3\-8B, computed on the POS\-diversified dataset described in Appendix[A](https://arxiv.org/html/2608.14361#A1)\. The transition pattern is preserved at every scale, indicating that the effect is driven by lexical diversity itself rather than by any property specific to nouns\.

Figure 9:ID curves across layers for all scales on Qwen3\-8B, computed on the POS\-diversified dataset \(Appendix[A](https://arxiv.org/html/2608.14361#A1)\)\. The transition pattern is preserved at each scalekk, matching the noun\-only results in Figure[2](https://arxiv.org/html/2608.14361#S4.F2)\.

## Appendix DLicenses

URLs and licenses of the used assets are provided in the following list: