五种架构，一个共享空间：无需权重合并的跨模型表征对齐

# Five Architectures, One Shared Space Cross-Model Representational Alignment Without Weight Merging
Source: [https://arxiv.org/html/2610.33495](https://arxiv.org/html/2610.33495)
Dimeji OlawuyiAffiliation:ÌyàwóAffiliation:Faculty of Clinical Sciences, College of Medicine, University of IbadanCorrespondence: anthoniooladimeji11@gmail\.comToba AjayiAffiliation:ÌyàwóAffiliation:Faculty of Clinical Sciences, College of Medicine, University of IbadanCorrespondence: anthoniooladimeji11@gmail\.comTemilola AderemiAffiliation:ÌyàwóAffiliation:Faculty of Clinical Sciences, College of Medicine, University of IbadanCorrespondence: anthoniooladimeji11@gmail\.com

October 2026

###### Abstract

Combining language models is usually done by averaging weights, which works only when the models share a common starting point\. We ask whether five frozen, independently trained models with different internal designs \(four transformers from four organizations and one state\-space model\) can be related to one another through their activations alone, with no weight merging\. Using permutation tests, an independent replication probe set, depth sweeps, pooling ablations and confound checks, we find statistically robust correspondence between how the five models organize the same text\. It is strongest at each model’s final layer and weaker for the state\-space model, whose weakest pairing is still3535–43×43\\timeschance\. We use this to build a shared space with Generalized Procrustes Analysis, in which no model is a fixed reference\. At the final layer its mean held\-out retrieval accuracy over the twenty ordered model pairs is0\.910\.91to0\.930\.93on two probe sets \(chance0\.030\.03\), beating pairwise linear maps \(0\.820\.82to0\.850\.85\), but it is worse than those maps in the middle layers\. The space transfers to ordinary English content of unfamiliar grammatical form and fails on mathematical and logical notation\. We then test whether the space supports a router that picks among the five models on gradeable tasks\. Across four task types we find no case where routing beats always using the strongest model, and we report this as a negative result\. A correctness signal that looked strong turned out to be category membership in disguise\. All experiments ran on one laptop without a GPU\.

## 1Introduction

The standard way to combine neural networks is to average their weights\([Wortsman et al\., 2022](https://arxiv.org/html/2610.33495#bib.bib15)\)\. This works well, but it depends on the models sharing a weight space, typically because they are fine\-tuned from one base checkpoint\. Even for independently trained networks of identical architecture, merging requires first undoing permutation symmetries of the hidden units\([Ainsworth et al\., 2023](https://arxiv.org/html/2610.33495#bib.bib1)\)\. None of this applies to models that differ in architecture, training data and parameter count\.

We ask a different question\. Rather than merging weights, can very different models be related through how they internally represent the same input, while every model stays frozen? If so, a small trainable component could in principle combine their outputs without touching their weights, preserving each model’s original behavior exactly\.

The question sits next to a growing body of evidence that independently trained networks tend toward similar representations as they scale\([Huh et al\., 2024](https://arxiv.org/html/2610.33495#bib.bib10)\), and next to the tools used to measure that similarity\([Kornblith et al\., 2019](https://arxiv.org/html/2610.33495#bib.bib11);[Bansal et al\., 2021](https://arxiv.org/html/2610.33495#bib.bib3)\)\. We study a case that is harder than most: five small open models trained by different groups, one of which \(Mamba,[Gu and Dao, 2023](https://arxiv.org/html/2610.33495#bib.bib9)\) is a state\-space model that shares no sequence\-mixing mechanism with the other four\.

#### Contributions\.

1. 1\.We show, using a series of experiments designed to falsify our own early results, that cross\-model correspondence exists among these five models, and we state where it is strong, where it is weak, and which apparent signals were artifacts \(Sections[4](https://arxiv.org/html/2610.33495#S4)–[5](https://arxiv.org/html/2610.33495#S5)\)\.
2. 2\.We build a shared space with Generalized Procrustes Analysis \(GPA\), compare it with pairwise linear maps at every depth, and measure its generalization boundary \(Sections[6](https://arxiv.org/html/2610.33495#S6)–[7](https://arxiv.org/html/2610.33495#S7)\)\.
3. 3\.We test whether the space has practical value for routing between the five models and find none, after a four\-task search that included one task chosen because the literature predicted an architectural difference \(Section[8](https://arxiv.org/html/2610.33495#S8)\)\.

Everything was run on a single consumer laptop \(Apple M1 Pro, 16 GB, no discrete GPU\)\. This limits scale, which we discuss in Section[9](https://arxiv.org/html/2610.33495#S9), but it makes every number here cheap to reproduce\.

Figure 1:Overview\. Each frozen model’s final\-layer, mean\-pooled activation is reduced by its own PCA, then rotated by its own orthogonal matrix into a space defined by the consensus of all five models\. No model is the reference\. The space is evaluated by retrieval \(Stages 2 and 3\) and used to build a category router \(Stage 4\)\. The dashed box marks the state\-space model\.

## 2Related Work

#### Model merging\.

Averaging the weights of fine\-tuned checkpoints of one base model improves accuracy and robustness\([Wortsman et al\., 2022](https://arxiv.org/html/2610.33495#bib.bib15)\)\. Merging independently trained networks of the same architecture can work after aligning hidden units up to permutation\([Ainsworth et al\., 2023](https://arxiv.org/html/2610.33495#bib.bib1)\)\. Both assume a shared parameterization\. Our models have different widths, depths and sequence\-mixing mechanisms, so there is no parameter\-level correspondence to exploit, and we work on activations instead\.

#### Representational similarity\.

Centered kernel alignment\([Kornblith et al\., 2019](https://arxiv.org/html/2610.33495#bib.bib11)\)and model stitching\([Bansal et al\., 2021](https://arxiv.org/html/2610.33495#bib.bib3)\)compare representations of different networks, with the latter asking whether a learned connector can join the early part of one network to the late part of another\. The Platonic Representation Hypothesis\([Huh et al\., 2024](https://arxiv.org/html/2610.33495#bib.bib10)\)argues that representations of independently trained models converge as they scale\. Our setting is a small\-scale test of that idea with one deliberately non\-transformer model, and with the added question of whether the correspondence can be turned into an explicit common space\.

#### Aligning separate spaces\.

Orthogonal Procrustes alignment\([Schönemann, 1966](https://arxiv.org/html/2610.33495#bib.bib14)\)maps one point configuration onto another using only a rotation \(and possibly a reflection\), and has been used to align word\-embedding spaces across languages\([Conneau et al\., 2018](https://arxiv.org/html/2610.33495#bib.bib6)\)\. Generalized Procrustes Analysis\([Gower, 1975](https://arxiv.org/html/2610.33495#bib.bib7)\)extends this to many configurations at once, aligning all of them to their own mean\. Relative representations\([Moschella et al\., 2023](https://arxiv.org/html/2610.33495#bib.bib12)\)describe each input by its similarity to a set of anchor inputs, which makes latent spaces of different models directly comparable\. We use GPA because it keeps every model on equal footing and uses only orthogonal maps, the most conservative family of transforms\.

#### State\-space models versus attention\.

Selective state\-space models such as Mamba\([Gu and Dao, 2023](https://arxiv.org/html/2610.33495#bib.bib9)\)match transformers on many language tasks at lower cost, but associative recall, retrieving the value paired with a key from context, is a documented weakness, because attention can index any position while a state\-space model must compress history into a fixed\-size state\([Arora et al\., 2023](https://arxiv.org/html/2610.33495#bib.bib2)\)\. We use this to design one targeted task in Section[8](https://arxiv.org/html/2610.33495#S8)\.

#### Routing\.

Systems such as RouteLLM\([Ong et al\., 2025](https://arxiv.org/html/2610.33495#bib.bib13)\)learn to choose between a strong and a weak model per query to save cost\. We ask a different question: whether a space built without any task labels, from activations only, carries enough information to route among five models\.

## 3Models and Experimental Setup

Table 1:The five frozen models used throughout\.All five are base \(not instruction\-tuned\) checkpoints, run in inference mode on one laptop\. Mamba ran without accelerated kernels in our software environment and was about25×25\\timesslower per probe than the transformers\.

#### Probes\.

We wrote 200 short inputs, 20 in each of ten categories: declarative facts, questions, sentence completions, syntactically ambiguous sentences, grammatical nonsense, logical reasoning, emotionally valenced text, source code, non\-English text \(six languages\), and deliberate token repetition\. A second 200\-probe set \(*set 2*\) with the same categories and entirely different content was written later and is used as an independent replication\. We refer to the first as*set 1*\. Both sets were written by the authors, which we return to in Section[9](https://arxiv.org/html/2610.33495#S9)\.

#### Representation\.

For modelmmwithLmL\_\{m\}layers and inputxxofTTtokens, lethm,t\(ℓ\)​\(x\)∈ℝdmh^\{\(\\ell\)\}\_\{m,t\}\(x\)\\in\\mathbb\{R\}^\{d\_\{m\}\}be the layer\-ℓ\\ellhidden state at tokentt\. The representation used throughout is the token\-mean

hm\(ℓ\)​\(x\)=1T​∑t=1Thm,t\(ℓ\)​\(x\)\.h^\{\(\\ell\)\}\_\{m\}\(x\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}h^\{\(\\ell\)\}\_\{m,t\}\(x\)\.\(1\)Unless stated otherwiseℓ=Lm\\ell=L\_\{m\}\(the final layer\)\. Depth comparisons use relative depths0,25,50,75,100%0,25,50,75,100\\%of each model’s own layer count\. Mean pooling was chosen after comparing it with first\-token, middle\-token, last\-token and maximum\-norm\-token pooling at each model’s middle layer\. For Qwen, Pythia and SmolLM the first\-token and maximum\-norm\-token vectors were identical for every probe \(an attention\-sink effect\), single\-token schemes collapsed onto one axis for at least one scheme in each transformer, and mean pooling showed none of these artifacts\. Mamba is the exception: its last\-token vector separated the categories better than its mean\-pooled one\. We kept mean pooling as a uniform default and note that it may understate Mamba\.

#### Agreement measures\.

LetDm∈ℝn×nD\_\{m\}\\in\\mathbb\{R\}^\{n\\times n\}be the matrix of pairwise cosine distances between thennprobe representations of modelmm\.*Distance\-matrix agreement*between modelsaaandbbis the Spearman correlationρa​b\\rho\_\{ab\}between the upper\-triangular entries ofDaD\_\{a\}andDbD\_\{b\}\.*Nearest\-neighbor match*is the fraction of probesiifor whicharg⁡minj≠i​Da​\[i,j\]=arg⁡minj≠i​Db​\[i,j\]\\arg\\min\_\{j\\neq i\}D\_\{a\}\[i,j\]=\\arg\\min\_\{j\\neq i\}D\_\{b\}\[i,j\], i\.e\. both models name the same other probe as most similar; chance is1/\(n−1\)1/\(n\-1\)\(0\.00630\.0063forn=160n=160\)\. All nulls come from2,0002\{,\}000permutations of probe identity within one model; the permutationpp\-values are therefore bounded below by5×10−45\\times 10^\{\-4\}\. Unless noted, “whole set” analyses use the 160 probes of the eight categories other than code and multilingual text, which behave as outliers \(Section[5](https://arxiv.org/html/2610.33495#S5)\)\.

## 4Stage 1: Per\-Model Geometry

We projected each model’s layer\-wise activations to two dimensions with PCA\. The four transformers each show a compressed plateau across their middle layers, where9898–99\.8%99\.8\\%of variance lies in the first two components \(Qwen: layers 5–20\)\. Mamba shows no plateau: two\-dimensional variance stays at1010–20%20\\%through the middle of the network and rises to56%56\\%only at the final layer\. All five models reorganize sharply at their own final layer, which is the layer where each prepares its output\. In the middle layers even the four transformers look unlike one another, so aligning layerkkof one model with layerkkof another is not meaningful\. This motivated analyses that treat depth as a variable \(Section[5](https://arxiv.org/html/2610.33495#S5)\)\.

## 5Stage 2: Is There Cross\-Model Correspondence?

### 5\.1A first result that did not hold up

We first asked whether the models agree on which*categories*are similar\. Using category centroids, the mean pairwise agreement over all ten categories wasρ=0\.914\\rho=0\.914\. This number was inflated by code and multilingual text, which every model places far from everything else\. On the remaining eight categories the mean agreement wasρ=0\.808\\rho=0\.808, and a permutation test that shuffled category labels within each model gave a null mean of0\.6270\.627\(sd​0\.117\\mathrm\{sd\}\\ 0\.117\), so that the observed value beat only95\.2%95\.2\\%of null draws \(p≈0\.048p\\approx 0\.048\)\. Random partitions of the same probes already agree strongly because of shared shallow statistics such as word length and word rarity\. The category\-level evidence was therefore marginal, and we did not build on it\.

### 5\.2Probe\-level agreement

Working at the level of individual probes removes the averaging problem\. At the final layer the mean agreementρ\\rhoover the ten model pairs was0\.640\.64\(set 1\) and0\.650\.65\(set 2\), against permutation nulls centered on zero with standard deviation0\.030\.03to0\.040\.04\. For nearest\-neighbor match, the ten\-pair mean was0\.3240\.324on set 1 and0\.4070\.407on set 2, against a null of0\.00630\.0063, which is50×50\\timesand65×65\\timeschance \(p<5×10−4p<5\\times 10^\{\-4\}for both\)\. The result replicates on the second probe set \(Figure[2](https://arxiv.org/html/2610.33495#S5.F2)\), is significant at every depth tested, and is not explained by probe length or lexical rarity \(the correlation between each model’s distance matrix and the length or rarity distance matrix was≤0\.10\\leq 0\.10for four of the five models and at most0\.180\.18for OLMo\)\.

Two cautions apply\. Agreement at relative depth0%0\\%\(the embedding layer\) is high \(0\.900\.90and0\.930\.93\) but is expected from shared lexical overlap in any pair of models, and we do not treat it as evidence of shared processing\. And because the probe\-level null is tight, a very smallpp\-value does not mean that the effect is equally strong at every depth: the effect size falls to0\.330\.33–0\.450\.45in the middle of the network and recovers to0\.640\.64–0\.650\.65at the final layer\.

Figure 2:Cross\-model agreement by depth\. Each point is the mean Spearmanρ\\rhobetween distance matrices over all ten model pairs, on the 160 prose probes of each set\. The grey band is±2\\pm 2standard deviations of the permutation null\. Agreement is high at the embedding layer for trivial lexical reasons, lowest at 25% depth, and rises again at the final layer\.
### 5\.3The state\-space model is weaker, in degree and not in kind

At the final layer, the four Mamba\-involving pairs were the four weakest of the ten pairs in both probe sets \(mean nearest\-neighbor match0\.2300\.230against0\.3880\.388for the six transformer\-only pairs on set 1, and0\.3060\.306against0\.4740\.474on set 2\), with no overlap between the two groups\. The weakest Mamba pair, Mamba with OLMo, still sits at3535–43×43\\timeschance\. The size of the gap depends on depth \(Figure[3](https://arxiv.org/html/2610.33495#S5.F3)\): it is near zero at75%75\\%depth \(0\.0140\.014and0\.0040\.004\) and largest at the final layer \(0\.160\.16and0\.170\.17\), which is where the all\-pairs agreement is strongest\.

Figure 3:Nearest\-neighbor match rate by depth for transformer\-only pairs and Mamba\-involving pairs, on the two probe sets\. The shaded gap is smallest in the middle of the network and largest at the final layer\. Chance is0\.00630\.0063\.To see why the gap widens at the final layer we measured each model’s own nearest\-neighbor self\-consistency between consecutive late layers\. Pythia, SmolLM2 and OLMo each reorganize substantially on entering their last layer \(self\-consistency drops by0\.310\.31–0\.590\.59\), while Mamba reorganizes mildly \(0\.090\.09–0\.110\.11\) and Qwen reorganizes least\. The category\-level changes made by the three reorganizing transformers are positively correlated with one another in both probe sets \(Spearman0\.330\.33to0\.770\.77across the three pairs\), whereas Mamba’s changes correlate weakly or negatively with every other model \(−0\.36\-0\.36to0\.340\.34\), and Qwen’s correlate inconsistently across sets\. So the transformers that reorganize at the end do so in a shared direction, and Mamba does not move that way\. We do not claim to know why\. The explanation is descriptive, not causal\.

## 6Constructing a Shared Space

### 6\.1Pairwise linear maps

As a first construction we fit ordinary least\-squares maps between every ordered pair of models \(20 maps\) and scored them with 5\-fold cross\-validated retrieval: given a held\-out probe’s mapped vector from modelaa, is the nearest neighbor \(cosine\) among the held\-out probes of modelbbthe same probe? Each held\-out fold has 32 probes, so chance is1/32=0\.0311/32=0\.031\. Accuracy at the final layer ranged from0\.700\.70to0\.970\.97over the 20 maps, with means of0\.8160\.816\(set 1\) and0\.8540\.854\(set 2\)\. Notably, Mamba pairs were not the weakest here \(the single best map, at0\.970\.97, was between Mamba and SmolLM2\), which suggests Mamba’s activations are rotated or scaled differently from the transformers’ and not incompatible with them, even though the raw nearest\-neighbor comparison of the previous section ranked them lowest\. Fitting every model onto one fixed target makes that target a privileged reference, and each map has up to2048×20482048\\times 2048free parameters estimated from 128 training probes\.

### 6\.2Generalized Procrustes Analysis

LetXm∈ℝn×DX\_\{m\}\\in\\mathbb\{R\}^\{n\\times D\}be modelmm’s centered activations reduced by PCA to a common dimensionDD\(PCA is needed becausedmd\_\{m\}differs between models\):

Xm=\(Hm−𝟏​μm⊤\)​Vm,Vm∈ℝdm×D,Vm⊤​Vm=I,X\_\{m\}=\\bigl\(H\_\{m\}\-\\mathbf\{1\}\\mu\_\{m\}^\{\\top\}\\bigr\)V\_\{m\},\\qquad V\_\{m\}\\in\\mathbb\{R\}^\{d\_\{m\}\\times D\},\\ V\_\{m\}^\{\\top\}V\_\{m\}=I,\(2\)whereHmH\_\{m\}stacks the representationshm​\(x\)h\_\{m\}\(x\)andμm\\mu\_\{m\}is their mean\. GPA\([Gower, 1975](https://arxiv.org/html/2610.33495#bib.bib7)\)finds one orthogonal matrix per model, and a consensus configurationGG, by solving

min\{Rm\},G∑m=1M∥XmRm−G∥F2s\.t\.Rm⊤Rm=I,G=1M∑m=1MXmRm\.\\min\_\{\\\{R\_\{m\}\\\},\\,G\}\\ \\sum\_\{m=1\}^\{M\}\\bigl\\lVert X\_\{m\}R\_\{m\}\-G\\bigr\\rVert\_\{F\}^\{2\}\\quad\\text\{s\.t\. \}R\_\{m\}^\{\\top\}R\_\{m\}=I,\\qquad G=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}X\_\{m\}R\_\{m\}\.\(3\)It is solved by alternating two steps until the change in the objective is below10−810^\{\-8\}\(maximum 50 iterations\): givenGG, eachRmR\_\{m\}is the orthogonal Procrustes solution\([Schönemann, 1966](https://arxiv.org/html/2610.33495#bib.bib14)\),Rm=Um​Wm⊤R\_\{m\}=U\_\{m\}W\_\{m\}^\{\\top\}withXm⊤​G=Um​Σm​Wm⊤X\_\{m\}^\{\\top\}G=U\_\{m\}\\Sigma\_\{m\}W\_\{m\}^\{\\top\}\(reflections allowed, no scaling\); given theRmR\_\{m\},GGis updated to their mean\. The iteration starts from the first model’s centered configuration, which is only an initialization, since every model is rotated toward the evolving mean\. No model is the target; the space is the consensus\. A new input from modelmmenters the space aszm​\(x\)=\(hm​\(x\)−μm\)​Vm​Rm∈ℝDz\_\{m\}\(x\)=\\bigl\(h\_\{m\}\(x\)\-\\mu\_\{m\}\\bigr\)V\_\{m\}R\_\{m\}\\in\\mathbb\{R\}^\{D\}\.

#### Evaluation\.

For held\-out probes, the PCA basisVmV\_\{m\}and rotationsRmR\_\{m\}are fit on the four training folds only \(a first version fit PCA on all probes, which inflated accuracy by0\.0070\.007to0\.0190\.019; the figures below use the corrected protocol\)\. For each ordered pair\(a,b\)\(a,b\), accuracy is the fraction of held\-out probesiiwhoseza​\(xi\)z\_\{a\}\(x\_\{i\}\)has cosine\-nearest neighborzb​\(xi\)z\_\{b\}\(x\_\{i\}\)among the held\-out\{zb​\(xj\)\}\\\{z\_\{b\}\(x\_\{j\}\)\\\}\.

#### Results at the final layer\.

WithD=100D=100, mean accuracy over the 20 ordered pairs was0\.9070\.907\(set 1\) and0\.9280\.928\(set 2\), against0\.8160\.816and0\.8540\.854for pairwise maps\. Figure[4](https://arxiv.org/html/2610.33495#S6.F4)shows every pair\. All twenty ordered accuracies lie between0\.770\.77and0\.990\.99, the two weakest \(both under0\.800\.80on set 1\) being Qwen↔\\leftrightarrowMamba\. Asymmetry between the two directions of a pair is small \(≤0\.06\\leq 0\.06\) compared with pairwise maps \(≤0\.16\\leq 0\.16\)\.

![Refer to caption](https://arxiv.org/html/2610.33495v1/fig5_heatmaps.png)Figure 4:Held\-out retrieval accuracy for every ordered pair, through the GPA space \(final layer,D=100D=100, 5\-fold cross\-validation, chance0\.0310\.031\)\. Rows are the query model and columns the model retrieved among\. Thick lines separate Mamba\. The weakest cells involve Mamba and Qwen on set 1\.
#### GPA is not better at every depth\.

Figure[5](https://arxiv.org/html/2610.33495#S6.F5)a shows that in the middle of the network the pairwise maps are far better \(0\.560\.56to0\.670\.67against0\.260\.26to0\.400\.40at2525–75%75\\%depth\), in line with the earlier finding that the middle layers of these models are individual\. A rotation inside the topDDprincipal directions is a much smaller hypothesis class than an unconstrained linear map\. It pays off only where the models have converged, and in this study that is the final layer\. The advantage at the embedding layer \(0\.990\.99\) is the lexical\-overlap artifact discussed above\. For these reasons we construct the shared space at the final layer only\.

#### Dimension\.

Accuracy grows withDDand is flat fromD=200D=200upward \(Figure[5](https://arxiv.org/html/2610.33495#S6.F5)b\)\. We initially read this as a property of the models\. It is not established: each training fold has 128 probes, so centered PCA has at most 127 informative components, and any largerDDadds only zero\-variance directions\. The plateau reflects our sample size\. We still useD=200D=200for the released space because it is fit on all 400 probes\.

Figure 5:\(a\) Mean held\-out retrieval accuracy by depth, set 1, for GPA atD=100D=100and for pairwise linear maps\. GPA wins only at the final layer and at the \(lexical\) embedding layer\. \(b\) Accuracy against the shared dimensionDDat the final layer\. The plateau coincides with the rank limit of the 128\-probe training folds\.
#### Released artifact\.

The released module fits once on all 400 probes \(both sets, all ten categories\) and exposes a function that maps a final\-layer activation of any of the five models to a point in the 200\-dimensional space\.

## 7Stage 3: Where Does the Shared Space Stop Working?

The space of the previous section was first fit on the eight prose categories\. We asked how far it generalizes\. In the content\-type tests below, accuracy is retrieval accuracy within a pool of 10 probes of one type \(chance0\.100\.10\)\.

#### Unseen categories\.

Code and multilingual probes, never used in fitting, reached0\.290\.29to0\.360\.36accuracy in pairwise retrieval \(roughly66to1414times chance in that test\), far below the in\-distribution0\.90\.9range\.

#### Content unlike any fitted category\.

Ten each of imperative commands, bare enumerated lists and mathematical expressions were written for this test, none resembling a fitted category\. Imperatives and lists reached0\.900\.90\(Figure[6](https://arxiv.org/html/2610.33495#S7.F6)a\)\. Mathematical expressions reached only0\.300\.30\. A follow\-up separated numerals from symbols: number\-dense ordinary prose scored0\.810\.81and symbol\-dense notation with fewer numerals scored0\.240\.24\. The boundary is therefore not “resembles a fitted category” but closer to “ordinary natural\-language vocabulary versus non\-standard symbolic notation”\. We stress that this is a characterization of a few content types with ten items per type, not a proof of where the boundary lies\.

#### Refitting\.

In a matched five\-fold comparison, adding code and multilingual probes to the fit nearly doubled their held\-out accuracy \(0\.45→0\.840\.45\\rightarrow 0\.84\) at a cost of0\.020\.02on the eight original categories \(0\.92→0\.900\.92\\rightarrow 0\.90; Figure[6](https://arxiv.org/html/2610.33495#S7.F6)b\)\. The released space was refit on all ten categories\.

Figure 6:\(a\) Held\-out retrieval accuracy of a space fit on eight categories, for content types it never saw \(pools of 10, chance0\.100\.10\)\. Hatched bars are the failures\. \(b\) Matched five\-fold comparison with and without code and multilingual probes in the fit\.

## 8Stage 4: From Representations to Task Performance

The previous sections concern representations only\. We now ask whether the space is useful for choosing among the models\. All five models were run with a few\-shot prompt and greedy decoding \(they are base models\) on questions with unambiguous answers, and outputs were graded by exact match\.

### 8\.1A signal that was a confound

For Pythia, a question’s position in the shared space appeared to predict whether Pythia answered it correctly\. The nearest\-neighbor same\-label rate was0\.8960\.896against a null of0\.5000\.500\(p<5×10−4p<5\\times 10^\{\-4\}\) on a 48\-question set, and a similar result held on an earlier 24\-question set\. A direct check removed it\. Pythia’s correctness followed question category almost exactly:0/240/24correct on arithmetic \(easy and hard alike\) and21/2421/24on factual questions \(Figure[7](https://arxiv.org/html/2610.33495#S8.F7)a\)\. Because the space is already known to separate categories, “predicts correctness” reduces to “predicts category”\. Restricted to factual questions, where Pythia’s answers varied, the same\-label rate was0\.8330\.833against a null of0\.7730\.773\(p=0\.25p=0\.25,n=24n=24\), and on a second, independently written and harder factual set it was0\.8250\.825against0\.7760\.776\(p=0\.29p=0\.29,n=40n=40; Figure[7](https://arxiv.org/html/2610.33495#S8.F7)b\)\. We found no fine\-grained correctness signal\. Both factual\-only tests have few incorrect answers \(33and55\), so they have low power, and we do not claim the signal is absent, only that we could not detect it\.

Figure 7:\(a\) Pythia’s accuracy by question category on the 48\-question set \(12 per cell\)\. \(b\) Nearest\-neighbor same\-correctness\-label rate in the shared space against the permutation\-null mean\. The strong result on all questions disappears when only factual questions are used\.
### 8\.2A router, and four searches for a case where it helps

We built the simplest router that the confirmed signal supports\. A nearest\-centroid classifier predicts a question’s category \(arithmetic or factual\) from its position in the shared space, and the router sends the question to the model with the highest accuracy for that category on training data\. Trained on one question set and tested on another, category classification reached0\.9790\.979accuracy, but the routing decision became a single model for every category \(Qwen, tied with SmolLM2\), the same as always using the best model\.

We then searched four task types for any case where a different model should be preferred \(Table[2](https://arxiv.org/html/2610.33495#S8.T2)\)\.

Table 2:Accuracy of each model on four task designs\. In no row does a model other than Qwen or SmolLM2 come out on top\.The fourth task was chosen deliberately\. Given the documented weakness of state\-space models at associative recall\([Gu and Dao, 2023](https://arxiv.org/html/2610.33495#bib.bib9);[Arora et al\., 2023](https://arxiv.org/html/2610.33495#bib.bib2)\), we expected Mamba to do worse than the transformers, and it did: this is the only task where it fell below Pythia\. We tagged each of the 24 key\-value questions by whether the queried item was the first, second or third mentioned \(n=7,10,7n=7,10,7\)\. Mamba was best on the first item \(57\.1%57\.1\\%\) and worst on the last \(28\.6%28\.6\\%\), while Qwen and SmolLM2 were at or near ceiling in every position and Pythia was erratic \(Figure[8](https://arxiv.org/html/2610.33495#S8.F8)\)\. We had expected the opposite direction, a recency advantage, since the latest input should be freshest in a compressed state\. We report what we saw\. With seven items per cell, the difference between4/74/7and2/72/7is well within chance variation, so this primacy pattern is a hypothesis for replication, not a finding\.

Figure 8:Associative\-recall accuracy by which of three mentioned items was queried\. OLMo \(0% everywhere\) is omitted\. Withn=7n=7to1010per position, differences within a model are not statistically distinguishable; Mamba’s decline from first to last is suggestive only\.OLMo scored0%0\\%on every task\. Before treating this as real we checked next\-token logits: this checkpoint’s greedy completions barely varied with question content and gave an unrelated generic string\. A longer few\-shot prompt and plain completion\-style prompts without question\-answer framing gave the same result, which rules out prompt formatting\. We treat it as a checkpoint\-specific limitation under greedy decoding, not as a statement about OLMo\-family models\.

### 8\.3Closing the routing question

Across four task designs \(factual recall, arithmetic, compositional reasoning, associative lookup\) we found no case in which one of the three weaker models beats the stronger generalists\. For this model set and these tasks, routing has no demonstrated value over always using the best single model\. We call this a negative result and not an underpowered search, because the final task produced an explanatory result and the four tasks differ substantially in demand\. We cannot rule out a crossover on other domains, such as code generation, multilingual tasks or long\-context tasks\.

## 9Discussion and Limitations

The central positive result is that models with different architectures, trained by different groups, organize the same input in measurably similar ways at their final layer, including a state\-space model that shares no sequence\-mixing mechanism with the rest, and that this correspondence is strong enough to support a usable shared space from orthogonal maps alone\. It is consistent with the convergence argument of[Huh et al\. \(2024\)](https://arxiv.org/html/2610.33495#bib.bib10), though we test only five small models and cannot speak to scaling\.

We caution against two readings\. First, Mamba’s lower representational agreement and its weaker associative recall are separate observations from separate experiments\. We did not test any causal link between them, and we do not claim that one explains the other\. Second, the shared space was fit without any task labels and, as Stage 4 shows, it carries category information but no detectable information about whether a given model gets a given question right\.

Limitations are as follows\.

- •Scale\.All models are11–22B\-parameter base checkpoints\. Instruction\-tuned or larger models were not tested\. Probe sets have 200 inputs and most task sets have 24 to 48 questions, so several nulls \(notably the within\-category correctness test and the position\-wise recall breakdown\) are low\-powered\.
- •Authored probes\.Both probe sets were written by the authors\. A set drawn from natural corpora would remove the possibility that our writing style makes the models agree more than they otherwise would\. The independent second set reduces but does not remove this concern\.
- •Retrieval pools\.Retrieval accuracy is measured within pools of1010or3232items, so absolute numbers are not comparable to retrieval over large databases\.
- •Dimension\.The plateau inDDis a rank limit set by our sample size, so we do not know how many dimensions carry cross\-model structure\.
- •Pooling\.Uniform mean pooling may understate Mamba\.
- •Single run\.Every experiment used one pass of one model checkpoint with greedy decoding\. We did not vary seeds beyond cross\-validation fold assignment\.
- •Task coverage\.The negative routing result is specific to these five models and four task types\.

## 10Conclusion

Five frozen language models from four organizations, including a state\-space model, share real and replicable representational structure that can be put into one space by orthogonal rotations alone, with no model privileged\. At the final layer the space recovers0\.910\.91–0\.930\.93cross\-model retrieval accuracy and generalizes to ordinary English of unfamiliar grammatical form, but not to mathematical or logical notation, and not in the middle layers\. It does not, for this model set and these tasks, provide a better way to pick which model to use\. We report both outcomes, including two apparent signals \(category\-level agreement and correctness prediction\) that did not survive controls\.

#### AI assistance\.

The experiments, analysis and drafting were carried out with substantial assistance from Claude \(Anthropic\), an AI model, which helped write code, run experiments, check statistics and draft text\. The human authors designed the study, directed each step, reviewed all results and are responsible for the content\.

## References

- Ainsworth et al\. \(2023\)Samuel K\. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa\.Git re\-basin: Merging models modulo permutation symmetries\.In*International Conference on Learning Representations*, 2023\.
- Arora et al\. \(2023\)Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re\.Zoology: Measuring and improving recall in efficient language models\.*arXiv preprint arXiv:2312\.04927*, 2023\.
- Bansal et al\. \(2021\)Yamini Bansal, Preetum Nakkiran, and Boaz Barak\.Revisiting model stitching to compare neural representations\.In*Advances in Neural Information Processing Systems*, 2021\.
- Ben Allal et al\. \(2025\)Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, et al\.Smollm2: When smol goes big – data\-centric training of a small language model\.*arXiv preprint arXiv:2502\.02737*, 2025\.
- Biderman et al\. \(2023\)Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al\.Pythia: A suite for analyzing large language models across training and scaling\.In*Proceedings of the 40th International Conference on Machine Learning*, 2023\.
- Conneau et al\. \(2018\)Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou\.Word translation without parallel data\.In*International Conference on Learning Representations*, 2018\.
- Gower \(1975\)John C Gower\.Generalized procrustes analysis\.*Psychometrika*, 40\(1\):33–51, 1975\.
- Groeneveld et al\. \(2024\)Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, et al\.Olmo: Accelerating the science of language models\.*arXiv preprint arXiv:2402\.00838*, 2024\.
- Gu and Dao \(2023\)Albert Gu and Tri Dao\.Mamba: Linear\-time sequence modeling with selective state spaces\.*arXiv preprint arXiv:2312\.00752*, 2023\.
- Huh et al\. \(2024\)Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola\.The platonic representation hypothesis\.*arXiv preprint arXiv:2405\.07987*, 2024\.
- Kornblith et al\. \(2019\)Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton\.Similarity of neural network representations revisited\.In*Proceedings of the 36th International Conference on Machine Learning*, 2019\.
- Moschella et al\. \(2023\)Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà\.Relative representations enable zero\-shot latent space communication\.In*International Conference on Learning Representations*, 2023\.
- Ong et al\. \(2025\)Isaac Ong, Amjad Almahairi, Vincent Wu, Wei\-Lin Chiang, Tianhao Wu, Joseph E\. Gonzalez, M\. Waleed Kadous, and Ion Stoica\.Routellm: Learning to route llms with preference data\.In*International Conference on Learning Representations*, 2025\.
- Schönemann \(1966\)Peter H\. Schönemann\.A generalized solution of the orthogonal procrustes problem\.*Psychometrika*, 31\(1\):1–10, 1966\.
- Wortsman et al\. \(2022\)Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo\-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt\.Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.In*Proceedings of the 39th International Conference on Machine Learning*, pages 23965–23998, 2022\.
- Yang et al\. \(2024\)An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al\.Qwen2\.5 technical report\.*arXiv preprint arXiv:2412\.15115*, 2024\.