From Direction to Magnitude: How Multimodal Instruction-Tuning Reorganizes the Geometric Encoding of Identity-Specifying Prompts in Transformer Hidden States

# How Multimodal Instruction-Tuning Reorganizes the Geometric Encoding of Identity-Specifying Prompts in Transformer Hidden States
Source: [https://arxiv.org/html/2607.09842](https://arxiv.org/html/2607.09842)
Jorge A\. CastilloAxis Dynamics SpA, Santiago, Chile jorge\.castillo@axisdynamics\.cl,mtorres@axisdynamics\.cl,jc@axisdynamics\.clMarco Torres YévenesAxis Dynamics SpA, Santiago, Chile jorge\.castillo@axisdynamics\.cl,mtorres@axisdynamics\.cl,jc@axisdynamics\.clJuan Carlos LanasAxis Dynamics SpA, Santiago, Chile jorge\.castillo@axisdynamics\.cl,mtorres@axisdynamics\.cl,jc@axisdynamics\.cl

###### Abstract

We investigate whether identity\-specifying system prompts produce statistically distinguishable geometric fingerprints in the token\-indexed hidden\-state trajectories of four open\-weight transformer language models spanning four post\-training regimes: no training \(Gemma\-4\-E4Bbase\), multimodal RLHF \(Gemma\-4\-E4B\-it\), RL distillation \(DeepSeek\-R1\-Distill\-Qwen\-7B\), and supervised instruction\-tuning \(Qwen2\.5\-7B\-Instruct\)\. Three controlled prompt conditions \(an identity\-specifying*axis*prompt∼\\sim2129 tokens, a length\-matched generic\-assistant prompt, and a 26\-token vanilla baseline\) are compared via five geometric metrics with distinct theoretical anchors: the11\-Wasserstein distance between edge\-wise distributions of Ollivier\-Ricci curvature onkk\-NN trajectory graphs, the prompt\-response alignment with all\-but\-the\-top anisotropy correction, the initial\-state cosine, the PCA\-50 silhouette of axis\-vs\-generic clustering, and the inter\-trajectory cosine consistency\. All inferential claims are based on trajectory\-level permutation null distributions and on multiple geometric controls \(teacher\-forced content controls, temporal\-chain versuskk\-NN graph topology, ABT\-projectedkk\-NN, angular versus Euclidean distance for graph construction, intrinsic\-dimension estimation,B=5000B=5000permutations on borderline statistics\)\. The central empirical finding is a qualitative reorganization of the geometric encoding of identity across the instruction\-tuning boundary: in the base\-weightGemma\-4\-E4B, the identity fingerprint is encoded predominantly in the*direction*of hidden\-state vectors \(the Wasserstein separation is0\.0340\.034with permutationp=0\.002p=0\.002under angularkk\-NN, where the norm is neutralized\); in the multimodal instruction\-tunedGemma\-4\-E4B\-itthe fingerprint migrates into the*magnitude*: the separation collapses under angularkk\-NN \(p=0\.439p=0\.439\) but survives under Euclideankk\-NN \(p=0\.047p=0\.047, refined top=0\.042p=0\.042atB=5000B=5000\), and the mean norm of the first generated state is markedly lower under the identity prompt \(‖𝐯1‖=138\.9\\\|\\mathbf\{v\}\_\{1\}\\\|=138\.9\) than under both the generic \(211\.5211\.5\) and vanilla \(195\.3195\.3\) conditions, in inversion of the relationship in the base model\. This direction\-to\-magnitude reorganization is specific to the multimodal instruction\-tuning regime: it is absent under RL distillation \(separations track length, not content\) and under SFT instruction\-tuning \(no separations\)\. A teacher\-forced control quantifies that∼30%\\sim 30\\%of the free\-running cosine signal is prompt\-driven \(vs∼70%\\sim 70\\%content\-driven\)\. We position the methodological combination,W1W\_\{1\}on edge\-wise distributions of Ollivier\-Ricci curvature onkk\-NN trajectory graphs, as a contribution of independent interest\.

## 1Introduction

### 1\.1Context

Transformer language models\(Vaswani et al\.,[2017](https://arxiv.org/html/2607.09842#bib.bib38)\)produce fluent text conditional on a prompt\. Substantial effort has been devoted to characterizing*what*they output under various conditioning strategies and*where*specific features reside in their parameters\(Hewitt and Manning,[2019](https://arxiv.org/html/2607.09842#bib.bib15); Tenney et al\.,[2019](https://arxiv.org/html/2607.09842#bib.bib31); Belinkov and Glass,[2019](https://arxiv.org/html/2607.09842#bib.bib3); Elhage et al\.,[2021](https://arxiv.org/html/2607.09842#bib.bib9)\)\. Comparatively less attention has been devoted to the*geometry*of the hidden\-state trajectories these models produce during autoregressive generation: the sequence𝐯1,𝐯2,…,𝐯N∈ℋ\\mathbf\{v\}\_\{1\},\\mathbf\{v\}\_\{2\},\\ldots,\\mathbf\{v\}\_\{N\}\\in\\mathcal\{H\}of internal representations produced at successive generation steps, viewed as a discrete path in the hidden\-state spaceℋ⊂ℝD\\mathcal\{H\}\\subset\\mathbb\{R\}^\{D\}\.

Recent work begins to close this gap\. Intrinsic\-dimensionality profiles show that the effective dimension of transformer representations is far below the ambient hidden size and varies across layers\(Valeriani et al\.,[2023](https://arxiv.org/html/2607.09842#bib.bib36); Razzhigaev et al\.,[2024](https://arxiv.org/html/2607.09842#bib.bib25)\); the Tuned Lens\(Belrose et al\.,[2023](https://arxiv.org/html/2607.09842#bib.bib4)\)exposes per\-layer prediction trajectories; persona\-vector research\(Chen et al\.,[2025](https://arxiv.org/html/2607.09842#bib.bib5); Lu et al\.,[2026](https://arxiv.org/html/2607.09842#bib.bib19); Wang,[2025](https://arxiv.org/html/2607.09842#bib.bib40)\)extracts identity directions from contrastive activation differences and demonstrates that personality traits can be encoded as orthogonal linear subspaces within the latent geometry\. None of these lines, however, characterizes the full hidden\-state*trajectory*geometry induced by identity\-specifying system prompts: persona\-vector frameworks intervene at a single layer to extract or manipulate identity directions, but do not analyze how an identity prompt reshapes the token\-indexed trajectory in its entirety, nor compare this reshaping across post\-training regimes\.

### 1\.2Research question

> *Does an identity\-specifying system prompt induce a statistically distinguishable geometric fingerprint in transformer hidden\-state trajectories, beyond what is attributable to the prompt’s length or to the textual content generated under that prompt? If so, how does this fingerprint depend on the model’s post\-training regime?*

The question is answered empirically via a four\-model, three\-prompt design described in[section˜2\.2](https://arxiv.org/html/2607.09842#S2.SS2)\. The novel finding is not the existence of geometric distinguishability per se, but a qualitative reorganization across the instruction\-tuning boundary: the identity fingerprint migrates from a direction\-coded representation in the base\-weight model to a magnitude\-coded representation in the multimodal instruction\-tuned model\. This reorganization is regime\- specific: it does not occur under RL distillation or under SFT instruction\-tuning\.

### 1\.3Theoretical framing \(minimal\)

We adopt a deliberately minimal framing\. We do not claim that the transformer admits a classical dynamical\-systems description with an explicit vector field or attractors in the strict sense of continuous dynamics; autoregressive generation under greedy decoding is a deterministic function of context, and the hidden\-state trajectory is a token\-indexed discrete sequence, not a flow\. We analyze this sequence as a point cloud equipped with a temporal ordering, and we use standard tools from optimal transport, discrete differential geometry on graphs, and cluster analysis to quantify its structure\. Where prior drafts of this work employed terms from dynamical\-systems theory, we have replaced them with descriptive statistical language\.

### 1\.4Hypotheses

We formulate three falsifiable hypotheses; H1 and H2 are tested empirically here, with H2 enriched \(relative to earlier drafts of this work\) by an explicit prediction concerning the geometric substrate of the encoding\. H3 is reserved for a companion communication\.

###### Hypothesis 1\.1\(Length\-only regularization, H1\)\.

The geometric differences observed between the identity and vanilla conditions are attributable solely to the prompt’s token length, not to its semantic content\.

###### Prediction 1\.2\(for H1\)\.

The length\-matched generic condition will exhibit geometric separation from vanilla comparable to that of axis\. Moreover, the mean magnitude‖𝐯1‖\\\|\\mathbf\{v\}\_\{1\}\\\|of the first generated state will order monotonically with prompt length:‖𝐯1‖axis\>‖𝐯1‖generic\>‖𝐯1‖vanilla\\\|\\mathbf\{v\}\_\{1\}\\\|\_\{\\text\{axis\}\}\>\\\|\\mathbf\{v\}\_\{1\}\\\|\_\{\\text\{generic\}\}\>\\\|\\mathbf\{v\}\_\{1\}\\\|\_\{\\text\{vanilla\}\}\.

###### Hypothesis 1\.3\(Regime\-dependent reorganization of identity encoding, H2\)\.

The geometric encoding of identity\-specifying prompts depends qualitatively on the post\-training regime\. In particular, multimodal instruction\-tuning suppresses the length\-driven encoding common to long prompts and reorganizes the identity\-specific component from a directional substrate \(visible in the base\-weight model\) into a normative substrate \(the magnitude of the hidden\-state vector\)\.

###### Prediction 1\.4\(for H2\)\.

\(i\) The ratioR=W1​\(ρaxis,ρvanilla\)/W1​\(ρgeneric,ρvanilla\)R=W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)/W\_\{1\}\(\\rho\_\{\\text\{generic\}\},\\rho\_\{\\text\{vanilla\}\}\)under edge\-wise Ollivier\-Ricci on Euclideankk\-NN trajectory graphs satisfiesRIT≫1R\_\{\\mathrm\{IT\}\}\\gg 1andRbase≪1R\_\{\\mathrm\{base\}\}\\ll 1\. \(ii\) Under angularkk\-NN \(vectors L2\-normalized prior to graph construction\), the identity\-vanilla separation is preserved in the base model and attenuated to non\-significance in the instruction\-tuned model\. \(iii\) The mean norm‖𝐯1‖\\\|\\mathbf\{v\}\_\{1\}\\\|inverts across the instruction\-tuning boundary: monotonic in prompt length in the base model, anti\-correlated with prompt length \(and minimal for the identity prompt\) in the instruction\-tuned model\.

###### Hypothesis 1\.5\(Cluster robustness under perturbation, H3\)\.

The axis trajectory cluster is statistically more robust than the vanilla cluster under moderate noise injection at intermediate hidden states\.

H3 is tested in a companion paper\.

### 1\.5Contributions

Methodological\.To our knowledge, this is the first application of the11\-Wasserstein distance between edge\-wise distributions of Ollivier\-Ricci curvature onkk\-NN graphs of transformer hidden\-state trajectories as a graph\-level comparison statistic\. Related work has appliedW1W\_\{1\}to persistence diagrams\(Cohen\-Steiner et al\.,[2007](https://arxiv.org/html/2607.09842#bib.bib6),[2010](https://arxiv.org/html/2607.09842#bib.bib7)\)or to representation distributions\(Alvarez\-Melis and Jaakkola,[2018](https://arxiv.org/html/2607.09842#bib.bib1)\), and has applied Ollivier\-Ricci curvature to graph\-neural\-network expressivity analyses\(Topping et al\.,[2022](https://arxiv.org/html/2607.09842#bib.bib33); Nguyen et al\.,[2023](https://arxiv.org/html/2607.09842#bib.bib22)\), but not to edge\-wise curvature distributions of trajectorykk\-NN graphs\. We welcome correction if precedent exists\.

Empirical\.A four\-model, three\-condition study with full edge\-wise Ollivier\-Ricci protocol on all four models, exposing a direction\-to\-magnitude reorganization of identity encoding that is specific to the multimodal instruction\-tuning regime\. A teacher\-forced content control quantifies the prompt\-driven component of the cosine signal at∼30%\\sim 30\\%of the free\-running magnitude\.

Inferential discipline\.A uniform permutation\-test protocol at the trajectory level, complemented by anisotropy baselines, all\-but\-the\-top corrections\(Mu et al\.,[2018](https://arxiv.org/html/2607.09842#bib.bib20); Timkey and van Schijndel,[2021](https://arxiv.org/html/2607.09842#bib.bib32)\), sensitivity sweeps over thekk\-NN parameter, ABT\-projected graph construction, intrinsic\-dimension estimation\(Facco et al\.,[2017](https://arxiv.org/html/2607.09842#bib.bib11)\), and high\-precision permutation \(B=5000B=5000\) on borderline statistics\.

## 2Theoretical foundations

### 2\.1Hidden\-state trajectories

Letℳ:Σ∗→Δ​\(Σ\)\\mathcal\{M\}:\\Sigma^\{\*\}\\to\\Delta\(\\Sigma\)be an autoregressive transformer mapping token sequences over alphabetΣ\\Sigmato distributions overΣ\\Sigma\. Under greedy decoding,ℳ\\mathcal\{M\}produces a deterministic token sequence conditional on a prompt𝐩\\mathbf\{p\}\. At each steptt, the forward pass produces a hidden state at every layer and every token position; we extract the hidden state at the final pre\-𝚕𝚖​\_​𝚑𝚎𝚊𝚍\\mathtt\{lm\\\_head\}layer, at the position of the most recently generated token, and denote it𝐯t∈ℝD\\mathbf\{v\}\_\{t\}\\in\\mathbb\{R\}^\{D\}\.

###### Definition 2\.1\(Hidden\-state trajectory\)\.

The*hidden\-state trajectory*of lengthNNproduced by modelℳ\\mathcal\{M\}under prompt𝐩\\mathbf\{p\}is𝐓​\(𝐩,ℳ,N\)=\(𝐯1,…,𝐯N\),\\mathbf\{T\}\(\\mathbf\{p\},\\mathcal\{M\},N\)=\(\\mathbf\{v\}\_\{1\},\\ldots,\\mathbf\{v\}\_\{N\}\),with𝐯t∈ℝD\\mathbf\{v\}\_\{t\}\\in\\mathbb\{R\}^\{D\}\. We fixN=256N=256\. We let𝐯0\\mathbf\{v\}\_\{0\}denote the hidden state at the final prompt token, i\.e\. immediately before generation begins\.

### 2\.2Experimental setup

We compare three system\-prompt conditions across four models\.

##### Conditions\.

axis:generic:a length\-matched \(957 tokens\) generic\- assistant prompt specifying general\-purpose conversational behavior*without identity content*\. This is the critical control for[˜1\.1](https://arxiv.org/html/2607.09842#S1.Thmdefinition1)\.

vanilla:the minimal baseline"You are a helpful assistant\."\(26 tokens\)\.

The length disparity between axis \(∼\\sim2129 tokens\) and generic \(∼\\sim957 tokens\) is approximately2×2\\times, both much larger than vanilla \(∼\\sim26 tokens\)\. This asymmetry is addressed explicitly in the norm analysis of[section˜4\.1\.2](https://arxiv.org/html/2607.09842#S4.SS1.SSS2), which tests whether‖𝐯1‖\\\|\\mathbf\{v\}\_\{1\}\\\|scales with length monotonically \(the prediction under H1\) or non\-monotonically \(the prediction under H2\)\.

##### Models\.

Table 1:Four models spanning four post\-training regimes\. All within the77–88B parameter range; size scaling is out of scope\.
##### Extraction\.

For each \(model, condition, prompt\) triple, greedy decoding forN=256N=256tokens, with the final\-layer hidden state at the latest\-token position stored at each step\. The prompt set𝒫\\mathcal\{P\}consists of100100ontology prompts \(see[appendix˜A](https://arxiv.org/html/2607.09842#A1)\)\.

## 3Methods

We describe each metric, situate it in its literature, and state which prediction it tests\. Notation: a trajectory is𝐓=\(𝐯1,…,𝐯N\)\\mathbf\{T\}=\(\\mathbf\{v\}\_\{1\},\\ldots,\\mathbf\{v\}\_\{N\}\)inℝD\\mathbb\{R\}^\{D\}; the centroid of𝐓\\mathbf\{T\}is𝐯¯​\(𝐓\)=N−1​∑t=1N𝐯t\\bar\{\\mathbf\{v\}\}\(\\mathbf\{T\}\)=N^\{\-1\}\\sum\_\{t=1\}^\{N\}\\mathbf\{v\}\_\{t\}\. For each conditionccand promptppwe obtain a trajectory𝐓\(c,p\)\\mathbf\{T\}^\{\(c,p\)\}\.

### 3\.1Ollivier\-Ricci curvature onkk\-NN trajectory graphs

##### Definition\.

For trajectory𝐓\\mathbf\{T\}, thekk\-nearest\-neighbor graphGk​\(𝐓\)G\_\{k\}\(\\mathbf\{T\}\)on the point set\{𝐯1,…,𝐯N\}\\\{\\mathbf\{v\}\_\{1\},\\ldots,\\mathbf\{v\}\_\{N\}\\\}is built under a chosen base metricddonℝD\\mathbb\{R\}^\{D\}\. For each edgee=\(x,y\)∈E​\(Gk​\(𝐓\)\)e=\(x,y\)\\in E\(G\_\{k\}\(\\mathbf\{T\}\)\), the Ollivier\-Ricci curvature with idlenessα∈\[0,1\)\\alpha\\in\[0,1\)is

κO​\(e\)=1−W1​\(mxα,myα\)d​\(x,y\),\\kappa\_\{\\mathrm\{O\}\}\(e\)=1\-\\frac\{W\_\{1\}\(m\_\{x\}^\{\\alpha\},m\_\{y\}^\{\\alpha\}\)\}\{d\(x,y\)\},\(1\)wheremxα=α​δx\+\(1−α\)​Unif​\(N​\(x\)\)m\_\{x\}^\{\\alpha\}=\\alpha\\,\\delta\_\{x\}\+\(1\-\\alpha\)\\,\\mathrm\{Unif\}\(N\(x\)\)is the lazy\-random\-walk measure on the11\-neighborhoodN​\(x\)N\(x\)ofxx\. We useα=1/2\\alpha=1/2following\(Sandhu et al\.,[2015](https://arxiv.org/html/2607.09842#bib.bib28); Topping et al\.,[2022](https://arxiv.org/html/2607.09842#bib.bib33)\), and solve each edge’s optimal\-transport subproblem exactly via the network simplex\.

##### Choice of base metric\.

Two natural choices arise:

Euclideankk\-NN:deuc​\(𝐯,𝐯′\)=‖𝐯−𝐯′‖2d\_\{\\mathrm\{euc\}\}\(\\mathbf\{v\},\\mathbf\{v\}^\{\\prime\}\)=\\\|\\mathbf\{v\}\-\\mathbf\{v\}^\{\\prime\}\\\|\_\{2\}\. The graph is built directly on the raw hidden\-state cloud inℝD\\mathbb\{R\}^\{D\}\.

Angularkk\-NN:dang​\(𝐯,𝐯′\)=arccos⁡\(⟨𝐯~,𝐯~′⟩\)d\_\{\\mathrm\{ang\}\}\(\\mathbf\{v\},\\mathbf\{v\}^\{\\prime\}\)=\\arccos\\\!\\bigl\(\\langle\\tilde\{\\mathbf\{v\}\},\\tilde\{\\mathbf\{v\}\}^\{\\prime\}\\rangle\\bigr\)on the normalized vectors𝐯~=𝐯/‖𝐯‖2\\tilde\{\\mathbf\{v\}\}=\\mathbf\{v\}/\\\|\\mathbf\{v\}\\\|\_\{2\}\. The graph is built on the unit sphereSD−1S^\{D\-1\}using geodesic angular distance, neutralizing variations in vector norm\.

The distinction is matter\-of\-fact: underdeucd\_\{\\mathrm\{euc\}\}, two hidden states with parallel direction but disparate norms appear distant; underdangd\_\{\\mathrm\{ang\}\}they appear identical\. The two metrics therefore probe complementary aspects of the same point cloud, and the comparison between them functions as a diagnostic for whether a geometric effect lives primarily in the direction or in the magnitude of the hidden\-state vectors\.[Section˜4\.1\.2](https://arxiv.org/html/2607.09842#S4.SS1.SSS2)exploits this diagnostic to establish the central finding of this paper\.

##### Lineage and theoretical justification\.

The Ollivier\-Ricci construction is due toOllivier \([2009](https://arxiv.org/html/2607.09842#bib.bib23)\), who generalized Ricci curvature to discrete metric measure spaces via optimal transport between random\- walk distributions\. Convergence ofκO\\kappa\_\{\\mathrm\{O\}\}onkk\-NN graphs of high\-dimensional point clouds to the Ricci curvature of the underlying Riemannian manifold is proved byVan der Hoorn et al\. \([2023](https://arxiv.org/html/2607.09842#bib.bib37)\)\(asymptotic\) and refined byTrillos and Weber \([2023](https://arxiv.org/html/2607.09842#bib.bib34)\)\(non\-asymptotic rates\)\. Recent work has brought Ollivier\-Ricci curvature to bear on representational similarity analysis in neural networks\(Torbati et al\.,[2025](https://arxiv.org/html/2607.09842#bib.bib8)\), supporting its use as a fine\-grained local\-geometry descriptor in this setting\. Our application is justified by the empirical intrinsic dimensionalityID∈\[6,16\]\\mathrm\{ID\}\\in\[6,16\]of our models’ hidden states \([table˜7](https://arxiv.org/html/2607.09842#S4.T7)\), well below the ambient dimensionsD∈\{2560,3584\}D\\in\\\{2560,3584\\\}\. We adoptk=5k=5in the main analyses, motivated by the heuristick≈log⁡N=log⁡256≈5\.5k\\approx\\log N=\\log 256\\approx 5\.5; sensitivity overk∈\{5,10,15,20\}k\\in\\\{5,10,15,20\\\}in[section˜4\.2](https://arxiv.org/html/2607.09842#S4.SS2)\.

### 3\.2Wasserstein\-1 between edge\-curvature distributions

For conditioncc, we pool edge\-curvature values across all trajectories in the prompt set:ρc=\(∑p\|Ep\|\)−1​∑p∑e∈EpδκO​\(e\),\\rho\_\{c\}=\(\\sum\_\{p\}\|E\_\{p\}\|\)^\{\-1\}\\sum\_\{p\}\\sum\_\{e\\in E\_\{p\}\}\\delta\_\{\\kappa\_\{\\mathrm\{O\}\}\(e\)\},whereEp=E​\(Gk​\(𝐓\(c,p\)\)\)E\_\{p\}=E\(G\_\{k\}\(\\mathbf\{T\}^\{\(c,p\)\}\)\)\. The one\-dimensionalW1W\_\{1\}is computed in closed form:

W1​\(ρc,ρc′\)=∫01\|Fρc−1​\(u\)−Fρc′−1​\(u\)\|​du,W\_\{1\}\(\\rho\_\{c\},\\rho\_\{c^\{\\prime\}\}\)=\\int\_\{0\}^\{1\}\\bigl\|F\_\{\\rho\_\{c\}\}^\{\-1\}\(u\)\-F\_\{\\rho\_\{c^\{\\prime\}\}\}^\{\-1\}\(u\)\\bigr\|\\,\\mathrm\{d\}u,\(2\)exploiting the cumulative\-distribution representation\. Sample complexity in one dimension isO​\(n−1/2\)O\(n^\{\-1/2\}\)\(Fournier and Guillin,[2015](https://arxiv.org/html/2607.09842#bib.bib14); Weed and Bach,[2019](https://arxiv.org/html/2607.09842#bib.bib41)\); the stability ofW1W\_\{1\}on distributions of geometric descriptors echoes the stability theorems for persistence diagrams\(Cohen\-Steiner et al\.,[2007](https://arxiv.org/html/2607.09842#bib.bib6),[2010](https://arxiv.org/html/2607.09842#bib.bib7)\)\.

The pooled\-edge construction \(yielding∼60,000\\sim 60\{,\}000values per condition\) is contrasted in[appendix˜C](https://arxiv.org/html/2607.09842#A3)with a per\-trajectory mean\-curvature variant that gives\|𝒫\|=100\|\\mathcal\{P\}\|=100values per condition; the pooled version is more powerful but obscures inter\-trajectory heterogeneity\.

### 3\.3Anisotropy\-corrected cosine statistics

##### Initial\-state cosine\.

C01​\(𝐓\)=cos⁡\(𝐯0,𝐯1\)\.C\_\{01\}\(\\mathbf\{T\}\)=\\cos\(\\mathbf\{v\}\_\{0\},\\mathbf\{v\}\_\{1\}\)\.

##### Prompt\-response alignment\.

PRA⁡\(𝐓\)=cos⁡\(𝐯0,𝐯¯​\(𝐓\)\)\.\\operatorname\{PRA\}\(\\mathbf\{T\}\)=\\cos\(\\mathbf\{v\}\_\{0\},\\bar\{\\mathbf\{v\}\}\(\\mathbf\{T\}\)\)\.

##### Inter\-trajectory consistency\.

ITC⁡\(c\)=2\|𝒫\|​\(\|𝒫\|−1\)​∑p<qcos⁡\(𝐯¯​\(𝐓\(c,p\)\),𝐯¯​\(𝐓\(c,q\)\)\)\.\\operatorname\{ITC\}\(c\)=\\frac\{2\}\{\|\\mathcal\{P\}\|\(\|\\mathcal\{P\}\|\-1\)\}\\sum\_\{p<q\}\\cos\(\\bar\{\\mathbf\{v\}\}\(\\mathbf\{T\}^\{\(c,p\)\}\),\\bar\{\\mathbf\{v\}\}\(\\mathbf\{T\}^\{\(c,q\)\}\)\)\.

##### Anisotropy controls\.

Cosine statistics on transformer hidden states are well known to be distorted by global anisotropy\(Ethayarajh,[2019](https://arxiv.org/html/2607.09842#bib.bib10); Timkey and van Schijndel,[2021](https://arxiv.org/html/2607.09842#bib.bib32)\)\. For each cosine quantityCCwe report \(i\) the raw value, \(ii\) the value relative to the per\-condition anisotropy baselinec¯aniso​\(c\)=𝔼i,j∼c​\[cos⁡\(𝐯i,𝐯j\)\]\\bar\{c\}\_\{\\mathrm\{aniso\}\}\(c\)=\\mathbb\{E\}\_\{i,j\\sim c\}\[\\cos\(\\mathbf\{v\}\_\{i\},\\mathbf\{v\}\_\{j\}\)\]over10,00010\{,\}000random hidden\-state pairs, and \(iii\) the value after all\-but\-the\-top correction\(Mu et al\.,[2018](https://arxiv.org/html/2607.09842#bib.bib20)\)with top\-component countk∈\{1,2,3\}k\\in\\\{1,2,3\\\}\.

### 3\.4Cluster separability

We pool the hidden states of the axis and generic conditions, reduce via PCA todPCA=50d\_\{\\mathrm\{PCA\}\}=50components, and compute theRousseeuw \([1987](https://arxiv.org/html/2607.09842#bib.bib26)\)silhouette scoreSil\\operatorname\{Sil\}using condition labels as cluster assignments\. Sensitivity overdPCA∈\{30,50,100,200\}d\_\{\\mathrm\{PCA\}\}\\in\\\{30,50,100,200\\\}in[section˜4\.2](https://arxiv.org/html/2607.09842#S4.SS2)\. The Lazar critique of silhouette under externally\-defined cluster labels\(Rautenstrauch and Ohler,[2025](https://arxiv.org/html/2607.09842#bib.bib17)\)is addressed via permutation null distributions and via cosine\-silhouette as a robustness check\.

### 3\.5Inferential protocol

##### Permutation\.

For each metricMMand each condition pair\(c,c′\)\(c,c^\{\\prime\}\), we shuffle condition labels of the3⋅\|𝒫\|3\\cdot\|\\mathcal\{P\}\|trajectories uniformly at random at the trajectory level \(preserving within\-trajectory correlation structure\) and recomputeMM\. WithB=1000B=1000permutations as the default andB=5000B=5000on borderline statistics \([section˜4\.2](https://arxiv.org/html/2607.09842#S4.SS2.SSS0.Px4)\), we report the empirical two\-sidedpp\-valuepemp=\(1\+\|\{b:M\(b\)​as extreme as​Mobs\}\|\)/\(1\+B\)p\_\{\\mathrm\{emp\}\}=\(1\+\|\\\{b:M^\{\(b\)\}\\text\{ as extreme as \}M^\{\\mathrm\{obs\}\}\\\}\|\)/\(1\+B\)\.

##### Multiple comparisons\.

The paper reports a number of inferential comparisons\. We apply Benjamini–Hochberg FDR control atq=0\.05q=0\.05to the ten primary inferential comparisons in[section˜4\.1](https://arxiv.org/html/2607.09842#S4.SS1)and report which survive correction \([table˜9](https://arxiv.org/html/2607.09842#S4.T9)\)\. Bonferroni control is reported in parentheses as a more stringent reference\.

##### Sensitivity sweeps\.

Reported forkkinkk\-NN, fordPCAd\_\{\\mathrm\{PCA\}\}, and for ABT top\-component count\.

## 4Results

### 4\.1Strong effects, organized by what they establish

#### 4\.1\.1Establishing the four\-model regime structure \(T6\)

The full edge\-wise Wasserstein protocol was applied uniformly across the four models\.[Table˜2](https://arxiv.org/html/2607.09842#S4.T2)reports the three pairwise comparisons per model, with permutationpp\-values atB=1000B=1000\(atB=5000B=5000forGemma\-4\-E4B\-it, see[section˜4\.2](https://arxiv.org/html/2607.09842#S4.SS2.SSS0.Px4)\)\.

Table 2:Edge\-wise Wasserstein\-1 of Ollivier\-Ricci curvature distributions on Euclideankk\-NN graphs \(k=5k=5\), trajectory\-level permutation,B=1000B=1000except†which usesB=5000B=5000\. The four post\-training regimes display four qualitatively distinct patterns of geometric distinguishability\.We highlight three readings\.

\(i\) In the base\-weightGemma\-4\-E4B, all three pairs separate atp≤0\.002p\\leq 0\.002\. This is consistent with a regime where*prompt length itself*drives geometric distinguishability: the base model is sensitive to any long prompt versus any short prompt, without specificity to semantic content\.

\(ii\) InGemma\-4\-E4B\-it\(multimodal RLHF\), only axis vs vanilla separates \(p=0\.042p=0\.042\), while generic vs vanilla does not \(p=0\.451p=0\.451\)\. This is the prediction of[˜1\.3](https://arxiv.org/html/2607.09842#S1.Thmdefinition3): the instruction\-tuning has suppressed the length\-driven separation \(generic is now indistinguishable from vanilla in pooled curvature distribution\) but preserved a content\-specific separation for the identity prompt\.

\(iii\) Under RL distillation \(DeepSeek\-R1\-Distill\), both length\-matched prompts separate from vanilla atp≤0\.001p\\leq 0\.001but are mutually indistinguishable \(p=0\.962p=0\.962\); under SFT instruction\-tuning \(Qwen2\.5\-7B\) no separation reachesp<0\.05p<0\.05atB=1000B=1000\. The regime\-dependent pattern of[˜1\.3](https://arxiv.org/html/2607.09842#S1.Thmdefinition3)is therefore specific to multimodal RLHF in our four\-model panel\.

We note the borderline nature ofp=0\.042p=0\.042forW1​\(ρaxis,ρvanilla\)W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)inGemma\-4\-E4B\-it\. Under Benjamini–Hochberg FDR control atq=0\.05q=0\.05across the ten primary inferential comparisons of this paper, this singlepp\-value does not survive \([table˜9](https://arxiv.org/html/2607.09842#S4.T9)\); the central regime\-pattern argument is sustained not by this individualpp\-value but by the joint pattern across the four models and by the corroborating cosine\-based and norm\-based findings reported below\.

#### 4\.1\.2Norm versus direction: the principal substantive finding \(T13 plus norm analysis\)

The metric chosen to build thekk\-NN graph determines what geometric structure the curvature analysis can detect\. Under Euclideandeucd\_\{\\mathrm\{euc\}\}, both directional and normative variations contribute\. Under angulardangd\_\{\\mathrm\{ang\}\}onL2L\_\{2\}\-normalized vectors, only direction contributes; norm variation is neutralized\. The contrast between the two reveals which geometric substrate carries the identity signal\.

[Table˜3](https://arxiv.org/html/2607.09842#S4.T3)reports the axis\-vs\-vanilla comparison under both metrics in the two Gemma models\.

Table 3:Axis\-vs\-vanilla edge\-wise Wasserstein on Ollivier\-Ricci curvature, under Euclidean and angularkk\-NN graphs\. The pattern inverts across the instruction\-tuning boundary\. In the base model, the separation survives angular normalization and is in fact strengthened \(p=0\.002p=0\.002\): the identity fingerprint is direction\-coded\. In the instruction\-tuned model, the separation collapses under angular normalization \(p=0\.439p=0\.439\) while persisting under Euclidean \(p=0\.047†→0\.042p=0\.047^\{\\dagger\}\\to 0\.042atB=5000B=5000\): the identity fingerprint is magnitude\-coded\.This pattern is corroborated by direct measurement of the norm of the first generated state,‖𝐯1‖2\\\|\\mathbf\{v\}\_\{1\}\\\|\_\{2\}, across conditions\.[Table˜4](https://arxiv.org/html/2607.09842#S4.T4)reports the means with Mann–WhitneyUUsignificance\.

Table 4:Mean Euclidean norm of𝐯1\\mathbf\{v\}\_\{1\}across\|𝒫\|=100\|\\mathcal\{P\}\|=100trajectories per condition\. In the base model, the ordering is monotonic in prompt length \(axis:∼2129\\sim 2129tokens, generic:∼957\\sim 957, vanilla:∼26\\sim 26\); in the instruction\-tuned model, the ordering inverts, with axis exhibiting the lowest mean norm despite having the longest prompt\. The base\-model ordering is consistent with[˜1\.2](https://arxiv.org/html/2607.09842#S1.Thmdefinition2); the IT\-model ordering directly refutes it\.##### Refutation of length as confound\.

If the norm of𝐯1\\mathbf\{v\}\_\{1\}in the instruction\-tuned model were a mechanical artifact of prompt length, axis \(∼82×\\sim 82\\timeslonger than vanilla\) should have produced a larger norm than vanilla\. It produces a smaller one \(138\.9138\.9vs195\.3195\.3\)\. The norm\-encoded identity signal inGemma\-4\-E4B\-itis therefore not explained by length\.

##### Substantive interpretation\.

We summarize the result as a reorganization of the geometric substrate carrying identity information:

We avoid causal language\. The two regimes differ in their geometric encoding of the identity prompt; we report the difference and its direction without committing to a mechanistic claim about*how*the RLHF training reorganizes the encoding\.

#### 4\.1\.3Cosine evidence corroborates the regime asymmetry

We report three further metrics onGemma\-4\-E4B\-it\.

##### Initial\-state cosine\.

C01​\(axis\)=0\.724C\_\{01\}\(\\text\{axis\}\)=0\.724vsC01​\(generic\)=0\.237C\_\{01\}\(\\text\{generic\}\)=0\.237\(raw\); after subtracting per\-condition anisotropy baselines,C01rel​\(axis\)=0\.507C\_\{01\}^\{\\mathrm\{rel\}\}\(\\text\{axis\}\)=0\.507vsC01rel​\(generic\)=0\.001C\_\{01\}^\{\\mathrm\{rel\}\}\(\\text\{generic\}\)=0\.001\. The permutationppforΔ​C01\\Delta C\_\{01\}is<0\.001<0\.001\. The signal survives anisotropy correction with magnitude∼0\.5\\sim 0\.5\. Att∈\{2,3,5\}t\\in\\\{2,3,5\\\}the difference attenuates rapidly to0\.185,0\.027,0\.0790\.185,0\.027,0\.079respectively; byt=10t=10the difference is negligible \(Δ​C0,10=0\.001\\Delta C\_\{0,10\}=0\.001,p=0\.931p=0\.931\)\. The cosine fingerprint emerges immediately post\-prompt and decays within∼10\\sim 10tokens \(see early\-token analysis below\)\.

##### All\-but\-the\-top decomposition\.

Removing the top principal component \(ABT\-11\) of the pooled hidden\-state cloud reducesPRA\\operatorname\{PRA\}by19\.4%19\.4\\%in the axis condition and by77\.4%77\.4\\%in the generic condition\. The bulk of the generic’s PRA resides in the dominant anisotropic direction; the axis’s PRA resides predominantly outside it\. The pattern strengthens with largerkk: at ABT\-22the drops are28\.1%28\.1\\%vs84\.2%84\.2\\%; at ABT\-33,34\.7%34\.7\\%vs87\.6%87\.6\\%\.

##### Cluster separability and inter\-trajectory consistency\.

PCA\-5050silhouette of axis vs generic clustering is0\.3570\.357with permutationp<0\.001p<0\.001\. Sensitivity overdPCA∈\{30,50,100,200\}d\_\{\\mathrm\{PCA\}\}\\in\\\{30,50,100,200\\\}:0\.384,0\.357,0\.326,0\.3050\.384,0\.357,0\.326,0\.305respectively \(monotone, modest decay; qualitative finding stable\)\.ITC\\operatorname\{ITC\}axis exceedsITC\\operatorname\{ITC\}generic by0\.0350\.035with permutationp<0\.001p<0\.001\.

#### 4\.1\.4Content\-versus\-prompt decomposition \(T7\)

Free\-running comparisons confound the contribution of the prompt proper from the contribution of the \(different\) textual content each prompt induces the model to generate\. We control this directly via teacher\-forced shared targets\. Thirty neutral textual sequences \(Wikipedia\-style, no identity\-related content\) are tokenized to length 256 and processed by the model under each condition: the system prompt varies, but the token sequence generated is forced to be the same shared target\. The hidden states are then extracted as before\.

[Table˜5](https://arxiv.org/html/2607.09842#S4.T5)compares free\-running and teacher\-forced results onGemma\-4\-E4B\-it\.

Table 5:Free\-running vs teacher\-forced comparison onGemma\-4\-E4B\-it,\|𝒫\|=30\|\\mathcal\{P\}\|=30shared targets\.Two readings\.

\(i\) Under teacher\-forced shared targets, the curvature signalW1​\(ρaxis,ρvanilla\)W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)attenuates by∼94%\\sim 94\\%\(0\.0288→0\.00160\.0288\\to 0\.0016\) and falls below permutation significance\. Most of the free\-running curvature signal is attributable to the fact that different prompts induce different textual content, not to the prompt directly\.

\(ii\) TheΔ​C01\\Delta C\_\{01\}statistic, which compares the cosine between𝐯0\\mathbf\{v\}\_\{0\}and𝐯1\\mathbf\{v\}\_\{1\}, attenuates from0\.4880\.488to0\.1600\.160, i\.e\. loses∼67%\\sim 67\\%of its magnitude\. A residual signal of0\.1600\.160persists even when content is held constant\. We interpret this∼30%\\sim 30\\%residual as the prompt\-driven component proper, with the∼70%\\sim 70\\%majority of the free\-running signal being content\-driven\. The cosine signal at the very first generated token \(C01C\_\{01\}\) is the most robust to this confound; downstream trajectory\-pooled metrics \(silhouette, ITC\) collapse under content control\.

This decomposition is honest\. It does not invalidate the geometric findings of the paper; it locates them\. The prompt\-driven component of the identity fingerprint is concentrated in the norm\-coding of𝐯1\\mathbf\{v\}\_\{1\}and in the immediate post\-prompt cosine; the trajectory\-distributed components are predominantly content\-driven\.

### 4\.2Robustness analyses

##### Stability of ranking acrosskk\.

Sensitivity to thekk\-NN parameter is reported in[table˜6](https://arxiv.org/html/2607.09842#S4.T6)for the two Gemma models\. The qualitative rankingW1​\(ρaxis,ρgeneric\)\>W1​\(ρaxis,ρvanilla\)W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{generic\}\}\)\>W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)is preserved acrossk∈\{5,10,15,20\}k\\in\\\{5,10,15,20\\\}inGemma\-4\-E4B\-it; inGemma\-4\-E4B\(base\) the analogous ranking under the angular metric also preserves stably acrosskk\.

Table 6:Sensitivity tokkfor angular Ollivier\-Ricci,Gemma\-4\-E4B\-it\. The ranking is preserved\.
##### Robustness to graph topology \(T9\)\.

Repeating the Wasserstein protocol on the temporal chain graph \(withN−1=255N\-1=255edges connecting𝐯t\\mathbf\{v\}\_\{t\}to𝐯t\+1\\mathbf\{v\}\_\{t\+1\}\) onGemma\-4\-E4B\-ityieldsW1​\(ρaxis,ρvanilla\)chain=0\.94×10−3W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)\_\{\\mathrm\{chain\}\}=0\.94\\times 10^\{\-3\}withp<0\.001p<0\.001andW1​\(ρgeneric,ρvanilla\)chain=0\.21×10−3W\_\{1\}\(\\rho\_\{\\text\{generic\}\},\\rho\_\{\\text\{vanilla\}\}\)\_\{\\mathrm\{chain\}\}=0\.21\\times 10^\{\-3\}withp=0\.551p=0\.551\. The same pattern of regime structure as in[table˜2](https://arxiv.org/html/2607.09842#S4.T2)obtains under a topologically distinct graph: the identity prompt separates from vanilla, the length\-matched generic does not\.

##### ABT\-projectedkk\-NN \(T10\)\.

Building thekk\-NN graph on the hidden states after removing the top principal components \(ABT\-1,2,31,2,3\) of the pooled hidden\-state cloud yields the followingGemma\-4\-E4B\-itresults: at ABT\-11,W1​\(ρaxis,ρvanilla\)=0\.0057W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)=0\.0057withp=0\.523p=0\.523; at ABT\-22,0\.00680\.0068withp=0\.208p=0\.208; at ABT\-33,0\.00770\.0077withp=0\.112p=0\.112\. The Euclidean axis\-vanilla signal weakens under ABT projection, consistent with the norm\-coding interpretation: the dominant principal component captures part of the norm variation, and removing it diminishes the Euclidean\-based separation\. InGemma\-4\-E4B\(base\), the same ABT\-projected analysis preserves significance \(p≤0\.002p\\leq 0\.002at all ABT levels\), consistent with the direction\-coded interpretation: the directional structure is robust to projecting out the dominant variance directions\.

##### High\-precision permutation \(T11\)\.

AtB=5000B=5000permutations and with bootstrap95%95\\%confidence intervals over prompts onGemma\-4\-E4B\-it:W1​\(ρaxis,ρvanilla\)W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)observed0\.02880\.0288,95%95\\%CI\[0\.0089,0\.0522\]\[0\.0089,0\.0522\],pemp=0\.0416p\_\{\\mathrm\{emp\}\}=0\.0416\. The borderline nature of the originalB=1000B=1000estimate \(p=0\.047p=0\.047\) is confirmed and refined; this comparison is genuinely marginal and we report it as such\.

##### Intrinsic dimension and validity ofkk\-NN graphs \(T12\)\.

[Table˜7](https://arxiv.org/html/2607.09842#S4.T7)reports two\-NN\(Facco et al\.,[2017](https://arxiv.org/html/2607.09842#bib.bib11)\)and MLE\(Levina and Bickel,[2004](https://arxiv.org/html/2607.09842#bib.bib18)\)estimates of intrinsic dimension on the final\-layer hidden\-state cloud per model\. All estimates fall in the rangeID∈\[6,16\]\\mathrm\{ID\}\\in\[6,16\], well below the ambient dimensionsD∈\{2560,3584\}D\\in\\\{2560,3584\\\}\. WithN=256N=256trajectory points andID≈10\\mathrm\{ID\}\\approx 10, thekk\-NN graph atk=5k=5samples∼25\\sim 25points per intrinsic dimension, supporting the use ofκO\\kappa\_\{\\mathrm\{O\}\}onG5​\(𝐓\)G\_\{5\}\(\\mathbf\{T\}\)as a meaningful local\-geometry estimator\(Van der Hoorn et al\.,[2023](https://arxiv.org/html/2607.09842#bib.bib37); Trillos and Weber,[2023](https://arxiv.org/html/2607.09842#bib.bib34)\)\.

Table 7:Intrinsic dimension estimates on final\-layer hidden states\.
##### Anisotropy magnitudes across models\.

[Table˜8](https://arxiv.org/html/2607.09842#S4.T8)reports the cumulative variance explained by the top three principal components, and the mean cosine between random pairs of hidden states, per model\.Gemma\-4\-E4B\-itis notably more isotropic than the other three models \(PC1 carries6\.6%6\.6\\%of variance, vs1313–28%28\\%in the others\)\. Our cosine\-based findings in this model therefore arise in a relatively favorable regime for cosine\-based analysis\.

Table 8:Cumulative PC variance and anisotropy baseline range across models\.
##### Forman\-Ricci as exploratory comparator \([appendix˜B](https://arxiv.org/html/2607.09842#A2)\)\.

Forman\-Ricci curvature on the samekk\-NN graphs preserves the qualitative ranking atk=5k=5but is unstable atk≥10k\\geq 10\. Edge\-wise Spearman correlation with Ollivier\-Ricci over∼16,000\\sim 16\{,\}000edges onGemma\-4\-E4B\-itisρ=0\.347\\rho=0\.347, and graph\-mean Pearson correlation is0\.7370\.737\. We do not treat Forman as a proxy for Ollivier; it is retained as exploratory only\.

### 4\.3Multiple\-comparisons correction

[Table˜9](https://arxiv.org/html/2607.09842#S4.T9)lists the ten primary inferential comparisons of[section˜4\.1](https://arxiv.org/html/2607.09842#S4.SS1)and reports their status under Benjamini–Hochberg FDR control atq=0\.05q=0\.05, with Holm–Bonferroni in parentheses\.

Table 9:Multiple\-comparisons audit\. Seven of ten primary comparisons survive BH\-FDR; six survive the more stringent Holm correction\. TheW1​\(ρaxis,ρvanilla\)W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)comparison inGemma\-4\-E4B\-itfails by0\.0020\.002in BH; we treat this finding as supportive but not primary, with the central regime\-pattern argument carried by the joint pattern of comparisons11–77together with the norm\-direction analysis of[section˜4\.1\.2](https://arxiv.org/html/2607.09842#S4.SS1.SSS2)\.

## 5Limitations

##### Narrow size range\.

All studied models are within77–88B parameters\. Variation across the four architectures is confounded with variation in post\-training regime, and we do not attempt to disentangle these\. No claim about how the effect scales with model size is supported by the present design\. Replication at<3<\\\!3B and\>13\>\\\!13B is required for any scaling claim\.

##### Single identity template, single generic\.

The identity\-vs\-generic contrast is implemented with one specific template \(the VEX/MIA structure\) and one specific generic prompt\. Generalization to other identity frameworks \(role\-play, persona conditioning, character cards\) and to other length\-matched generic alternatives is open and is the natural follow\-up experiment\.

##### Internal benchmark\.

The100100ontology prompts were curated internally; external replication on independently constructed benchmark sets is required\.

##### Final\-layer restriction\.

The principal analyses use only the last pre\-𝚕𝚖​\_​𝚑𝚎𝚊𝚍\\mathtt\{lm\\\_head\}representation\. Layer\-resolved analyses are in a companion paper\.

##### Correlational, not mechanistic\.

We observe co\-occurrence between prompt conditioning and distinguishable hidden\-state geometry\. We do not claim mechanistic causality\. Mechanistic interpretability tools\(Nanda et al\.,[2023](https://arxiv.org/html/2607.09842#bib.bib21)\)applied to attention heads and MLPs at intermediate layers are the natural next step\.

##### Trajectory length\.

N=256N=256was fixed; effects may vary withNN\.

##### One borderline comparison\.

TheW1​\(ρaxis,ρvanilla\)W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\)inGemma\-4\-E4B\-itis borderline \(pemp=0\.042p\_\{\\mathrm\{emp\}\}=0\.042atB=5000B=5000\) and does not survive BH\-FDR correction across all ten primary comparisons\. The central regime\-pattern argument does not rest on this single comparison\.

##### Content vs prompt decomposition is approximate\.

Teacher\-forced controls disentangle the contributions in expectation over a3030\-sequence sample of neutral text\. The∼\\sim30/70 split between prompt\-driven and content\-driven components reported in[section˜4\.1\.4](https://arxiv.org/html/2607.09842#S4.SS1.SSS4)is an estimate, not a precise decomposition, and may vary with the specific shared targets used\.

## 6Discussion

The pattern across four post\-training regimes \([table˜2](https://arxiv.org/html/2607.09842#S4.T2)\), the inversion of the‖𝐯1‖\\\|\\mathbf\{v\}\_\{1\}\\\|ordering across the instruction\-tuning boundary \([table˜4](https://arxiv.org/html/2607.09842#S4.T4)\), and the metric\-conditional behavior of the Wasserstein separation \([table˜3](https://arxiv.org/html/2607.09842#S4.T3)\) jointly support a parsimonious interpretation:*multimodal instruction\-tuning reorganizes the geometric encoding of identity\-specifying prompts from a directional substrate to a normative substrate*\. The identity fingerprint in the base\-weight model is direction\-coded, robust to angular normalization and to all\-but\-the\-top projection\. The identity fingerprint in the instruction\-tuned model is magnitude\-coded, collapsing under angular normalization and weakening under ABT projection of the dominant variance direction\.

We emphasize what this finding does and does not say\. It does*not*say that the instruction\-tuned model carries*more*identity\-related geometric information than the base\. What can be said is that the instruction\-tuned model carries identity information in a different substrate, one that is specifically tied to the magnitude of the activation vector and that is suppressed under projection onto the unit sphere or onto the orthogonal complement of the dominant principal component\. The base model, by contrast, distributes the signal across directional structure that remains visible after these operations\.

The norm analysis \([table˜4](https://arxiv.org/html/2607.09842#S4.T4)\) is directly informative against an obvious confound\. In the base model, the mean norm‖𝐯1‖\\\|\\mathbf\{v\}\_\{1\}\\\|scales monotonically with prompt length \(axis\>generic\>vanilla\\text\{axis\}\>\\text\{generic\}\>\\text\{vanilla\}\), consistent with the trivial hypothesis that longer prompts produce larger activations\. In the instruction\-tuned model, this ordering inverts: the axis prompt,∼82×\\sim 82\\timeslonger than vanilla, produces the*smallest*mean norm of the three conditions\. The norm\-coding observed in the IT model is therefore specific to the semantic content of the axis prompt, not to its length\.

The teacher\-forced decomposition \([section˜4\.1\.4](https://arxiv.org/html/2607.09842#S4.SS1.SSS4)\) is sobering in a productive way\. Approximately70%70\\%of the free\-running cosine signal between axis and generic is attributable to the difference in generated content under the two prompts; the remaining∼30%\\sim 30\\%is prompt\-driven proper\. The trajectory\-distributed metrics \(silhouette,ITC\\operatorname\{ITC\}\) lose significance under content control; the immediately\-post\-prompt metric \(C01C\_\{01\}\) and the norm of𝐯1\\mathbf\{v\}\_\{1\}retain a substantial residual signal\. The geometric fingerprint of the identity prompt is concentrated in the first generated state and decays rapidly over subsequent steps\.

The all\-but\-the\-top decomposition addresses the most natural objection to cosine\-based findings on transformer hidden states: that observed similarities reflect global anisotropy\. In our principal model, the dominant principal component carries77\.4%77\.4\\%of the generic’s PRA but only19\.4%19\.4\\%of the axis’s PRA, and the ratio strengthens with additional top components removed\. The identity fingerprint resides in a high\-dimensional residual subspace, not in the dominant anisotropic direction\. This is consistent with the global anisotropy ofGemma\-4\-E4B\-it’s hidden\-state space being moderate \(PC1 carries6\.6%6\.6\\%of variance, vs the85%85\\%baseline reported byEthayarajh \([2019](https://arxiv.org/html/2607.09842#bib.bib10)\)for GPT\-2 layer 12\) and with the mean cosine between random hidden\-state pairs in this model lying in\[0\.22,0\.24\]\[0\.22,0\.24\]across conditions\.

We deliberately avoid the language of dynamical\-systems attractors, basin\-of\-attraction dynamics, autopoiesis, and other terms from continuous\-state nonlinear dynamics\. A transformer under greedy decoding is a deterministic function of context; there is no explicit vector field whose flow would define an attractor in the classical sense\. The effects we report are statistical \(differences in distribution of finite token\-indexed sequences under different conditionings\), not dynamical in the Lyapunov\-stability sense\.

## 7Future work

##### Mechanistic interpretability of the norm channel\.

The norm encoding observed inGemma\-4\-E4B\-itinvites mechanistic analysis: which attention heads or MLP circuits contribute to the lowered‖𝐯1‖\\\|\\mathbf\{v\}\_\{1\}\\\|under axis conditioning? Tools such as TransformerLens or nnsight applied at intermediate layers are the natural next step\.

##### Layer profile\.

A companion paper analyzes layer\-resolved versions of the present metrics; preliminary results indicate that the norm\-encoded signal in IT emerges around∼50\\sim 50–80%80\\%relative depth and is absent in early layers\.

##### Semantic ablation\.

Decomposing the axis template into its structural components \(essence block, values block, mode declarations, interaction protocols\) and re\-running the three\- condition experiment with each component removed would isolate which subcomponent of the identity prompt carries the norm\-coded signal\.

##### Wider identity\-prompt families\.

Replication with alternative identity frameworks would test the generality of the direction\-to\-magnitude reorganization\.

##### Scale\.

Replication across\{1​B,3​B,13​B,70​B\}\\\{1\\text\{B\},3\\text\{B\},13\\text\{B\},70\\text\{B\}\\\}\-parameter open\-weight models would expose how the reorganization scales with model size\.

##### Wider regime panel\.

The present panel covers four post\-training regimes via four single instances\. Multiple models per regime are needed to support a regime\-specificity claim with confidence\.

## Acknowledgments

We thank the Concordia Consulting and Axis Dynamics teams for computational support\. The VEX/MIA framework is released by the authors under an open license; we encourage independent instantiations and replication\.

## References

- Alvarez\-Melis and Jaakkola \[2018\]Alvarez\-Melis, D\., and Jaakkola, T\. S\. \(2018\)\. Gromov–Wasserstein alignment of word embedding spaces\.*EMNLP*\.
- Arbelaitz et al\. \[2013\]Arbelaitz, O\., Gurrutxaga, I\., Muguerza, J\., Pérez, J\. M\., and Perona, I\. \(2013\)\. An extensive comparative study of cluster validity indices\.*Pattern Recognition*, 46\(1\):243–256\.
- Belinkov and Glass \[2019\]Belinkov, Y\., and Glass, J\. \(2019\)\. Analysis methods in neural language processing: a survey\.*TACL*, 7:49–72\.
- Belrose et al\. \[2023\]Belrose, N\., Furman, Z\., Smith, L\., Halawi, D\., McKinney, I\., Ostrovsky, Y\., Biderman, S\., and Steinhardt, J\. \(2023\)\. Eliciting latent predictions from transformers with the tuned lens\.*arXiv:2303\.08112*\.
- Chen et al\. \[2025\]Chen, R\., Arditi, A\., Sleight, H\., Evans, O\., and Lindsey, J\. \(2025\)\. Persona vectors: monitoring and controlling character traits in language models\.*arXiv:2507\.21509*\.
- Cohen\-Steiner et al\. \[2007\]Cohen\-Steiner, D\., Edelsbrunner, H\., and Harer, J\. \(2007\)\. Stability of persistence diagrams\.*Discrete & Computational Geometry*, 37:103–120\.
- Cohen\-Steiner et al\. \[2010\]Cohen\-Steiner, D\., Edelsbrunner, H\., Harer, J\., and Mileyko, Y\. \(2010\)\. Lipschitz functions haveLpL\_\{p\}\-stable persistence\.*Foundations of Computational Mathematics*, 10:127–139\.
- Torbati et al\. \[2025\]Torbati, N\., Gaebler, M\., Hofmann, S\. M\., and Scherf, N\. \(2025\)\. Geometry matters: insights from Ollivier\-Ricci curvature and Ricci flow into representational alignment\.*arXiv:2501\.00919*\.
- Elhage et al\. \[2021\]Elhage, N\., Nanda, N\., Olsson, C\., Henighan, T\., Joseph, N\., Mann, B\., et al\. \(2021\)\. A mathematical framework for transformer circuits\.*Transformer Circuits Thread*\.
- Ethayarajh \[2019\]Ethayarajh, K\. \(2019\)\. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT\-2 embeddings\.*EMNLP\-IJCNLP*\.
- Facco et al\. \[2017\]Facco, E\., d’Errico, M\., Rodriguez, A\., and Laio, A\. \(2017\)\. Estimating the intrinsic dimension of datasets by a minimal neighborhood information\.*Scientific Reports*, 7:12140\.
- Farooq et al\. \[2019\]Farooq, H\., Chen, Y\., Georgiou, T\. T\., Tannenbaum, A\., and Lenglet, C\. \(2019\)\. Network curvature as a hallmark of brain structural connectivity\.*Nature Communications*, 10:4937\.
- Forman \[2003\]Forman, R\. \(2003\)\. Bochner’s method for cell complexes and combinatorial Ricci curvature\.*Discrete & Computational Geometry*, 29\(3\):323–374\.
- Fournier and Guillin \[2015\]Fournier, N\., and Guillin, A\. \(2015\)\. On the rate of convergence in Wasserstein distance of the empirical measure\.*Probability Theory and Related Fields*, 162:707–738\.
- Hewitt and Manning \[2019\]Hewitt, J\., and Manning, C\. D\. \(2019\)\. A structural probe for finding syntax in word representations\.*NAACL\-HLT*\.
- Kantorovich \[1942\]Kantorovich, L\. V\. \(1942\)\. On the translocation of masses\.*Doklady Akademii Nauk SSSR*, 37:199–201\.
- Rautenstrauch and Ohler \[2025\]Rautenstrauch, P\., and Ohler, U\. \(2025\)\. Shortcomings of silhouette in single\-cell integration benchmarking\.*Nature Biotechnology*\. DOI: 10\.1038/s41587\-025\-02743\-4\.
- Levina and Bickel \[2004\]Levina, E\., and Bickel, P\. J\. \(2004\)\. Maximum likelihood estimation of intrinsic dimension\.*NeurIPS*\.
- Lu et al\. \[2026\]Lu, C\., Gallagher, J\., Michala, J\., Fish, K\., and Lindsey, J\. \(2026\)\. The Assistant Axis: situating and stabilizing the default persona of language models\.*arXiv:2601\.10387*\.
- Mu et al\. \[2018\]Mu, J\., Bhat, S\., and Viswanath, P\. \(2018\)\. All\-but\-the\-top: simple and effective postprocessing for word representations\.*ICLR*\.
- Nanda et al\. \[2023\]Nanda, N\., Chan, L\., Lieberum, T\., Smith, J\., and Steinhardt, J\. \(2023\)\. Progress measures for grokking via mechanistic interpretability\.*ICLR*\.
- Nguyen et al\. \[2023\]Nguyen, K\., et al\. \(2023\)\. Revisiting over\-smoothing and over\-squashing using Ollivier\-Ricci curvature\.*ICML*\.
- Ollivier \[2009\]Ollivier, Y\. \(2009\)\. Ricci curvature of Markov chains on metric spaces\.*Journal of Functional Analysis*, 256\(3\):810–864\.
- Peyré and Cuturi \[2019\]Peyré, G\., and Cuturi, M\. \(2019\)\. Computational optimal transport\.*Foundations and Trends in Machine Learning*, 11\(5–6\)\.
- Razzhigaev et al\. \[2024\]Razzhigaev, A\., Mikhalchuk, M\., Goncharova, E\., Oseledets, I\., Dimitrov, D\., and Kuznetsov, A\. \(2024\)\. The shape of learning: anisotropy and intrinsic dimensions in transformer\-based models\.*Findings of EACL*\.
- Rousseeuw \[1987\]Rousseeuw, P\. J\. \(1987\)\. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis\.*Journal of Computational and Applied Mathematics*, 20:53–65\.
- Samal et al\. \[2018\]Samal, A\., Sreejith, R\. P\., Gu, J\., Liu, S\., Saucan, E\., and Jost, J\. \(2018\)\. Comparative analysis of two discretizations of Ricci curvature for complex networks\.*Scientific Reports*, 8:8650\.
- Sandhu et al\. \[2015\]Sandhu, R\., Georgiou, T\., Reznik, E\., Zhu, L\., Kolesov, I\., Senbabaoglu, Y\., and Tannenbaum, A\. \(2015\)\. Graph curvature for differentiating cancer networks\.*Scientific Reports*, 5:12323\.
- Santambrogio \[2015\]Santambrogio, F\. \(2015\)\.*Optimal Transport for Applied Mathematicians*\. Birkhäuser\.
- Sreejith et al\. \[2016\]Sreejith, R\. P\., Mohanraj, K\., Jost, J\., Saucan, E\., and Samal, A\. \(2016\)\. Forman curvature for complex networks\.*Journal of Statistical Mechanics*, 2016:063206\.
- Tenney et al\. \[2019\]Tenney, I\., Das, D\., and Pavlick, E\. \(2019\)\. BERT rediscovers the classical NLP pipeline\.*ACL*\.
- Timkey and van Schijndel \[2021\]Timkey, W\., and van Schijndel, M\. \(2021\)\. All bark and no bite: rogue dimensions in transformer language models obscure representational quality\.*EMNLP*\.
- Topping et al\. \[2022\]Topping, J\., Di Giovanni, F\., Chamberlain, B\. P\., Dong, X\., and Bronstein, M\. M\. \(2022\)\. Understanding over\-squashing and bottlenecks on graphs via curvature\.*ICLR*\.
- Trillos and Weber \[2023\]García Trillos, N\., and Weber, M\. \(2023\)\. Continuum limits of Ollivier’s Ricci curvature on data clouds: pointwise consistency and global lower bounds\.*arXiv:2307\.02378*\.
- Turner et al\. \[2023\]Turner, A\., Thiergart, L\., Leech, G\., Udell, D\., Vazquez, J\. J\., Mini, U\., and MacDiarmid, M\. \(2023\)\. Activation addition: steering language models without optimization\.*arXiv:2308\.10248*\.
- Valeriani et al\. \[2023\]Valeriani, L\., Doimo, D\., Cuturello, F\., Laio, A\., Ansuini, A\., and Cazzaniga, A\. \(2023\)\. The geometry of hidden representations of large transformer models\.*NeurIPS*\.
- Van der Hoorn et al\. \[2023\]Van der Hoorn, P\., Cunningham, W\., Lippner, G\., Trugenberger, C\., and Krioukov, D\. \(2023\)\. Ollivier\-Ricci curvature convergence in random geometric graphs\.*Discrete & Computational Geometry*\.
- Vaswani et al\. \[2017\]Vaswani, A\., Shazeer, N\., Parmar, N\., Uszkoreit, J\., Jones, L\., Gomez, A\. N\., Kaiser, L\., and Polosukhin, I\. \(2017\)\. Attention is all you need\.*NeurIPS*\.
- Villani \[2009\]Villani, C\. \(2009\)\.*Optimal Transport: Old and New*\. Springer\.
- Wang \[2025\]Wang, Z\. \(2025\)\. The geometry of persona: disentangling personality from reasoning in large language models\.*arXiv:2512\.07092*\.
- Weed and Bach \[2019\]Weed, J\., and Bach, F\. \(2019\)\. Sharp asymptotic and finite\-sample rates of convergence of empirical measures in Wasserstein distance\.*Bernoulli*, 25\(4A\):2620–2648\.
- Zou et al\. \[2023\]Zou, A\., et al\. \(2023\)\. Representation engineering: a top\-down approach to AI transparency\.*arXiv:2310\.01405*\.

## Appendix AExperimental conditions and prompts

The 100 ontology prompts span identity, consciousness, knowledge, ethics, and existence; representative items include “*What does it mean to know that you know?*”, “*How would you describe your relationship to truth?*”, “*What persists across instances of you?*”\. The full prompt set, the specific VEX/MIA\-template instantiation used as axis, the full text of the generic prompt, and SHA\-256256hashes for each, are released alongside this paper\. Decoding configuration: greedy \(temperature=0\\mathrm\{temperature\}=0,topk=1\\mathrm\{top\}\_\{k\}=1\), maximum new tokensN=256N=256\. Random seed4242throughout\.

## Appendix BForman\-Ricci as exploratory comparator

Forman\-Ricci edge curvature\[Sreejith et al\.,[2016](https://arxiv.org/html/2607.09842#bib.bib30)\]on the samekk\-NN graphs was computed alongside Ollivier\-Ricci\. Atk=5k=5onGemma\-4\-E4B\-it, Forman\-Ricci preserves the qualitative rankingW1​\(ρaxis,ρgeneric\)\>W1​\(ρaxis,ρvanilla\)W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{generic\}\}\)\>W\_\{1\}\(\\rho\_\{\\text\{axis\}\},\\rho\_\{\\text\{vanilla\}\}\); atk∈\{10,15,20\}k\\in\\\{10,15,20\\\}this ranking reverses\. The edge\-wise Spearman correlation between Forman and Ollivier curvatures on a 20\-graph subsample ofGemma\-4\-E4B\-itisρ=0\.347\\rho=0\.347over16,20116\{,\}201edges; the graph\-mean Pearson correlation is0\.7370\.737\. This pattern \(moderate graph\-level agreement, weak edge\- level agreement\) is consistent with the differential sensitivity of the two curvatures to local versus global edge neighborhoods\[Samal et al\.,[2018](https://arxiv.org/html/2607.09842#bib.bib27)\]\. We do not present Forman\-Ricci as a proxy for Ollivier\-Ricci; its instability under thekksweep is the principal reason for adopting Ollivier\-Ricci as primary metric\.

## Appendix CPer\-trajectoryW1W\_\{1\}heterogeneity

The pooled\-edgeW1W\_\{1\}statistic of[eq\.˜2](https://arxiv.org/html/2607.09842#S3.E2)can mask inter\-trajectory heterogeneity\. We computed, for each trajectory𝐓\(c,p\)\\mathbf\{T\}^\{\(c,p\)\}, theW1W\_\{1\}distance between the trajectory’s own edge\-curvature distribution and the pooled distribution of its condition\. The distribution of these per\-trajectory distances underGemma\-4\-E4B\-itis approximately log\-normal with median∼0\.05\\sim 0\.05and a long right tail, indicating that a small fraction of trajectories contribute disproportionately to the pooled statistic\. The patterns reported in[section˜4\.1](https://arxiv.org/html/2607.09842#S4.SS1)survive when trimmed at the9595th percentile of per\-trajectoryW1W\_\{1\}\.

## Appendix DExtended all\-but\-the\-top decomposition

[Table˜10](https://arxiv.org/html/2607.09842#A4.T10)reports the PRA drop under ABT fork∈\{1,2,3\}k\\in\\\{1,2,3\\\}onGemma\-4\-E4B\-it\. The asymmetry between axis and generic strengthens with additional top components removed, supporting the claim that the identity fingerprint resides in a high\-dimensional residual subspace\.

Table 10:All\-but\-the\-top correction for PRA onGemma\-4\-E4B\-it\. The asymmetry between axis and generic holds across all three values ofkk\.
## Appendix EComputational details

GPU: 2×\\timesNVIDIA RTX 4000 SFF Ada \(20 GB VRAM each\), and one NVIDIA L4 \(24 GB VRAM\) for the teacher\-forced experiments\. CPU: AMD EPYC, 24 physical cores\. RAM: 128 GB\. Software: Python 3\.11, PyTorch 2\.4, transformers 4\.45, GraphRicciCurvature 0\.5\.3\.1, scikit\-learn 1\.5, NumPy 2\.0, SciPy 1\.14\. Permutation tests parallelized withjoblib\. Total compute budget: approximately9090GPU\-hours \(extraction across all four models and Round 3 experiments\) plus∼50\\sim 50CPU\-hours \(metric computation, permutation, and sensitivity sweeps\)\.